Hi,
A few days ago we’ve updated to latest release from 1.7 and now Continua.Server.Service.exe is now consuming 100% CPU time wherever it’s busy checking/updating any repo. We have quite large repositories, so updates are no instantaneous and some kind of tight loop is slowing down server. Which in turn is not helpful to svn.exe/hg.exe child processes, since they have same exact priority, which in turn leads to dreadful 1.0 hour timeout for those child processes… I’m quite sure that previous version didn’t behave that way.
Sample stack trace from parent process:
0 mscorlib.dll!System.DateTime.GetDatePart+0x27
1 mscorlib.dll!System.TimeZoneInfo.GetUtcOffsetFromUtc+0x118
2 mscorlib.dll!System.TimeZoneInfo.ConvertUtcToTimeZone+0xca
3 mscorlib.dll!System.TimeZoneInfo.ConvertTime+0x208
4 mscorlib.dll!System.TimeZoneInfo.ConvertTimeToUtc+0x9e
5 Continua.Modules.Builds.dll!Continua.Modules.Builds.Repositories.RepositoryChecker.IsInDowntimePeriod+0xad
6 Continua.Modules.Builds.dll!Continua.Modules.Builds.Repositories.RepositoryMonitor.IsRunnableChecker+0x4d
7 System.Core.dll!<>c__DisplayClassf1.<combinepredicates>b__e+0x1a<br>8 System.Core.dll!WhereListIterator1.MoveNext+0xa5
9 Continua.Modules.Builds.dll!Continua.Modules.Builds.Repositories.RepositoryMonitor.RunRepositoryCheckers+0x497
10 Continua.Modules.Builds.dll!Continua.Modules.Builds.Repositories.RepositoryMonitor.Monitor+0x1a8
11 Continua.Shared.Application.dll!Continua.Shared.Application.BackgroundMonitor.Run+0xb3
12 mscorlib.dll!System.Threading.ExecutionContext.RunInternal+0x285
13 mscorlib.dll!System.Threading.ExecutionContext.Run+0x9
14 mscorlib.dll!System.Threading.ExecutionContext.Run+0x57
15 mscorlib.dll!System.Threading.ThreadHelper.ThreadStart+0x51
Thanks in advance,
Ilya
Hi Ilya,
Could you send a diagnostics report (downloaded from the Event Log page) to support@finalbuilder.com and also a debug log captured while this is happening? We’ll do some testing today, to try to reproduce, but this these files will help us this narrow down the possible causes of this issue.
Hi Dave, thanks for looking into this.
We’re happy to send you the diagnostic file (as generated on the link on the Event Log page), but as we look through the file, we see it has some proprietary information in plain text. Can you let us know what information is included in the file, and if there is a way to scrub the file before we submit it?
Thanks,
Daniel Zimmerman
De Novo Software
As well, it seems like many of the errors are caused by a timeout., and this may even be the root of all the problems we’re seeing. From our past conversations, it seems like there is a hard-coded hour time out for querying repositories. Can you let us know how to change that to be user-configurable?
The repository timeout is not currently configurable. We can add that option, however we should first work out why the svn commands are taking taking a full hour. This is unusual and it’s not really acceptable to be waiting so long for the build to start.
We mask passwords in the diagnostic files, but not urls and file names. I understand that you may not want to share this type of information. We’re only really interested in the settings on the repositories and any event log messages. Maybe you can extract that information from the file and/or use a search and replace function in a text editor to replace any text you do not wish to share. Alternatively you can send screenshots of the settings with any urls or other private information blanked out in a image editor.
The main information we are interested in would be in the debug file. If calls to svn.exe are timing out, then we would need to know which command lines this is happening for. It is possible that svn is prompting for input, or stuck with an error or corrupt workspace. If it is timing out because it is taking too long to list changes or download files, then we should look at your repository settings to see if we can reduce this.
We would be looking for lines such as these in the debug log which would show the svn commands that are run along with timings and outputs including any error messages:
Medium: 09:08:11 {T32} [ProcessRunner] Executing program ‘C:\PathToSvn\bin\svn.exe’ with arguments ‘log http://servername/svn/Trunk/ --non-interactive --trust-server-cert --no-auth-cache --username ContinuaUser *********************** --verbose --xml -l 1’.Medium: 09:08:11 {T32} [Debug] [ProcessRunner] Exited process ‘C:\PathToSvn\bin\svn.exe’ with process id ‘4416’. Args are 'log http://servername/trunk/ --non-interactive --trust-server-cert --no-auth-cache --username ContinuaUser *********************** --verbose --xml -l 1’Medium: 09:08:11 {T32} [Debug] SVN: RunProcess -> Returning : exitCode is 0, output is '<?xml version="1.0" encoding="UTF-8"?><snip />log>'We would also be looking at lines labelled “[Repository Monitor]” so we can see the timings and any bottlenecks when running the repository checkers.
Maybe you are able to find and identify the command lines causing the issue. Otherwise, if you can search and replace any proprietary information and send the debug log to us we’ll investigate further.
Note that any information sent to support@finalbuilder.com will not be published publicly, as with this forum.
Hi,
Thanks for the reply.
We were in a bit of a rush to get things working again last week, so we reverted to a previous version - 1.7.1.163, and everything has been working fine. So it looks like the problems are related to the changes between that version and 1.7.1.218. We had just upgraded soon before the problems really kicked in.
It’s been stable for us since we reverted, and we haven’t seen the CPU problem or SVN timeouts. If it happens again, we’ll try to get you the debug files.
Daniel