Poor reliability

Continua CI Bugs
1

Hello,

 We have now monitored our Continua installation for some months. I'm sorry to say that while the system runs at the moment a single quite simple build and testing other. The reliability is quite poor. While I was on vacation the system developed an agent communication issue out of thin air basically the next day. 

 Despite we are keeping the installation up to date, the rate of various issues is too high. At least weekly there is some issue with the SVN repository check, build hanging without explanation, communication error with agents, agent sync error or something like this.

We have number of projects that we want to migrate to Continua, but the reliability of the system even with this relatively low load scenario does not permit that.

 Just today our builds were hanging, other was synced the stage to agent and just hung. Other developed some kind of repository changeset checking issue. We reset the problematic repository and waited for it to be resync. Now when starting the build the following error terminates it:

Importance: Error

Date: Tuesday, August 20, 2013
Time: 1:01:50 PM

Message:

Stage Controller

Error Initialising Workspace
Exception: ProcessException
Message: Running C:\Program Files\VSoft Technologies\ContinuaCI Agent\hg\hg.exe with arguments "archive -r tip -I "re:^((Source.*))$" -X re:^((\^(\\|/)\.hgignore))$ --subrepos -R C:\ContinuaAgent\Repos\2cdc6a4a C:\ContinuaAgent\Ws\12706 --config ui.username=Continua" failed with return code 255 and error output: "abort: no working directory: please specify a revision
"
Stack Trace:    at Continua.Shared.Utils.Mercurial.Run(ProcessArguments args, String workingDir, Func`2 checkResult)
   at Continua.Shared.Utils.Mercurial.Archive(String repository, String destination, String revision, String includes, IEnumerable`1 excludes)
   at Continua.Modules.Builds.Agent.FileSync.AgentRepositoryCache.<>c__DisplayClassd.b__c()
   at Continua.Shared.Utils.ReadWriteLockList`1.WithReadLock(TId id, CancellationTokenSource cancelTokenSource, Action action)
   at Continua.Modules.Builds.Agent.FileSync.AgentRepositoryCache.Export(String dest, String revision, String includes, IEnumerable`1 excludes, Boolean listFiles)
   at Continua.Modules.Builds.Agent.AgentRepositoryHelper.CopySourceToWorkspace(String filePattern, IEnumerable`1 excludes, String relativeDestination, AgentWorkspaceSyncContext wsCtx, Boolean logRepoFiles)
   at Continua.Modules.Builds.Agent.AgentBuildHelper.SyncSourceFromServer(IEnumerable`1 rules, AgentWorkspaceSyncContext workspaceCtx)
   at Continua.Modules.Builds.Agent.AgentBuildHelper.InitialiseWorkspaceOnAgent(AgentCallbackProxy proxy, TransportContextDTO source, Guid callId)
   at Continua.Modules.Builds.Agent.AgentBuildRunner.OnInitialisingWorkspace(Transition`1 inState)


 

2

And the next build attempt resulted in the build being stuck at the Server To Agent Workspace Sync -phase, with status of Unknown.

The system does not provide any way to see whether it’s really syncing something, but based on the CPU load on the agent this does not appear to be the case, and there are no entries in the event log either.

3

Hi Ollie

I can only apologise for the reliability issues with SVN, it's what  we have been working on the most for the last few weeks. We have fixed number of issues with the SVN support over the last few days, the build linked below (uploaded a few hours ago for another customer) should fix the issue you are seeing  (which I believe is a side effect of another issue). Agent side cache issues usually stem from an earlier issue on the server, we do still need to find a better way to deal with these errors. 

FWIW, we are not seeing these reliability issues with any other version control system, and if I look at the code for all the repository types, SVN is by far the most complex.. it is very difficult to get svn to fit into Continua and operate with branches in the way we are trying to. Add svn externals to that and it quickly becomes a nightmare. If you just point a repository at a single branch it's very stable. We use the svn command line client, and unfortunately it has very little built in support for reporting or working with branches. 

The new build (version 1.0.0.0.2431) with this fix can be downloaded using the links below: 


4

BTW, we can only fix issue that we are aware of.

5

Hi,

We’ll try the new build. I’m aware that you obviously cannot fix issues that you are not aware of, I was just trying to communicate that it’s currently extremely frustrating to work with Continua due to these minor issues that appear and disappear only to appear later. They contribute to general uneasiness towards the product as the general reliability appears in that sense that the we get automatically the build that should come out without someone having to attend to the system.

Although it’s clear that majority appear to be related to SVN, not all of them are.

Currently the system appears to be building ok, except that it does not detect newer changes for a smallish repository. There is nothing in the event log, and the build completes ok, except that it stubbornly refuses to build newest revision.

6

hi, i’ve sent many issues about SVN repositories for Continua here and i test it for months now. nearly daily.
With the current release, i cannot find any issues regarding to SVN using branches etc. anymore. For sure, reliability is most important to such a product.
But at this time, everything seems to work, at least in my scenario. well, the agent and build server are on the same machine here, so maybe you might have issues that will not happen here.

just for your information.

7

Hi,

We have tested Continua since early betas and provided feedback especially concerning SVN usage.

Now for example our other test project is hanging. Build is started approx 10 minutes ago and is still in Build Initializing -phase with the message “[Repository Monitor] Waiting for repository ‘Suite’ to be checked.”

No entries appear in the event log and we’re using the most recent build (2448).

8

The previous error was resolved by this new build.

9

try looking at the process list on the build server, so you can see any “hg.exe” or “svn” entries. display the “command line” tab, yo you can see what will cause the repository manager to hang, hopefully.

10

btw: resetting the repository once (after the current version) might help too because a mixed up server repository checkout won’t work anymore.