We have recently seen 2 crashes of the server. It writes a lot (at least 10 / second) entries to the event log:
A call from the agent to the server failed, it will be re-tried again after the connection has been re-established.
There was no endpoint listening at net.pipe://localhost/ContinuaCI/IAgentCallbackService that could accept the message. This is often caused by an incorrect address or SOAP action. See InnerException, if present, for more details.
1 / second is enough 
And this is the reason Continua has crashed:
Application: Continua.Server.Service.exe
Framework Version: v4.0.30319
Description: The process was terminated due to an unhandled exception.
Exception Info: System.AccessViolationException
Stack:
at System.Data.SqlServerCe.SqlCeDataReader.Move(IntPtr, System.Data.SqlServerCe.DIRECTION)
at System.Data.SqlServerCe.SqlCeDataReader.Move(System.Data.SqlServerCe.DIRECTION)
at System.Data.SqlServerCe.SqlCeDataReader.Read()
at Continua.Modules.Builds.Logging.SqlCeLogReaderSession.GetMessages(System.Guid, System.Nullable1<int32>, Int32, Int32, System.Nullable1<system.datetime>)
at Continua.Modules.Builds.Logging.BuildLogReader.GetMessages(System.Guid, System.Nullable1<int32>, Int32, Int32, System.Nullable1<system.datetime>)
at Continua.Modules.Builds.Services.BuildLogService.GetUpdates(Int32, Continua.Modules.Builds.Common.Services.DataTransferObjects.LogItemsToUpdateDTO, Int32)
at DynamicClass.SyncInvokeGetUpdates(System.Object, System.Object[], System.Object[])
at System.ServiceModel.Dispatcher.SyncMethodInvoker.Invoke(System.Object, System.Object[], System.Object[] ByRef)
at System.ServiceModel.Dispatcher.DispatchOperationRuntime.InvokeBegin(System.ServiceModel.Dispatcher.MessageRpc ByRef)
at System.ServiceModel.Dispatcher.ImmutableDispatchRuntime.ProcessMessage5(System.ServiceModel.Dispatcher.MessageRpc ByRef)
at System.ServiceModel.Dispatcher.ImmutableDispatchRuntime.ProcessMessage31(System.ServiceModel.Dispatcher.MessageRpc ByRef)
at System.ServiceModel.Dispatcher.MessageRpc.Process(Boolean)
at System.ServiceModel.Dispatcher.ChannelHandler.DispatchAndReleasePump(System.ServiceModel.Channels.RequestContext, Boolean, System.ServiceModel.OperationContext)
at System.ServiceModel.Dispatcher.ChannelHandler.HandleRequest(System.ServiceModel.Channels.RequestContext, System.ServiceModel.OperationContext)
at System.ServiceModel.Dispatcher.ChannelHandler.AsyncMessagePump(System.IAsyncResult)
at System.Runtime.Fx+AsyncThunk.UnhandledExceptionFrame(System.IAsyncResult)
at System.Runtime.AsyncResult.Complete(Boolean)
at System.ServiceModel.Channels.TransportDuplexSessionChannel+TryReceiveAsyncResult.OnReceive(System.IAsyncResult)
at System.Runtime.Fx+AsyncThunk.UnhandledExceptionFrame(System.IAsyncResult)
at System.Runtime.AsyncResult.Complete(Boolean)
at System.ServiceModel.Channels.SynchronizedMessageSource+ReceiveAsyncResult.OnReceiveComplete(System.Object)
at System.ServiceModel.Channels.SessionConnectionReader.OnAsyncReadComplete(System.Object)
at System.ServiceModel.Channels.PipeConnection.OnAsyncReadComplete(Boolean, Int32, Int32)
at System.ServiceModel.Channels.OverlappedContext.CompleteCallback(UInt32, UInt32, System.Threading.NativeOverlapped*)
at System.Runtime.Fx+IOCompletionThunk.UnhandledExceptionFrame(UInt32, UInt32, System.Threading.NativeOverlapped*)
at System.Threading._IOCompletionCallback.PerformIOCompletionCallback(UInt32, UInt32, System.Threading.NativeOverlapped*)</system.datetime></system.datetime>
Ouch, will look into it.
We were not able to reproduce this on our dev machine, but I found that we were not using the latest sqlce assemblies, so updated them, put that build on our build server and… bang… same problem! So my guess is that for some reason Continua CI is picking up the newer version on your machine. Googling the error doesn’t provide any solutions. Still investigating.
Hi Geert
I think we have this solved, at least I’m no longer able to reproduce it. Please try this build (you will need to update your agents). This build has built itself several times and was tested with lots of clients opening the same log view so I’m reasonably confident it fixes the problem, but we’ll do more testing tomorrow
Please make sure you have a database backup as this build also includes some schema changes that are not backwards compatible (ie if you have to roll back to an earlier build it will not work with the new schema).
http://downloads.finalbuilder.com/downloads/continua/1.0/ContinuaCI.Server.Setup_x64_1.0.0.2890.exe
http://downloads.finalbuilder.com/downloads/continua/1.0/ContinuaCI.Server.Setup_1.0.0.2890.exe
http://downloads.finalbuilder.com/downloads/continua/1.0/ContinuaCI.Agent.Setup_x64_1.0.0.2890.exe
http://downloads.finalbuilder.com/downloads/continua/1.0/ContinuaCI.Agent.Setup_1.0.0.2890.exe
Thanks, I will try this asap and let you know!
Thanks, we just installed. If you don’t hear anything from me on this thread, assume it is fixed.
The default agent properties seems to be missing now (no properties).
Did you update your agents?
Yes, both from the installer of the server (just hit yes if you want to install the agent) and a separate download. Even rolling back to a previous version does not seem to work (both agent and server). At first I see the properties (93), then they disappear.
Unhandled Service Exception
Exception: NullReferenceException
Message: Object reference not set to an instance of an object.
Stack Trace: at Continua.Modules.Builds.Agents.ServerAgentManager.UpdateProperties(RegisteredAgent agent, IEnumerable`1 propertiesToUpdate)
Rebooting the machine solved the issue. I can now view properties with the old agents. Maybe for the new install, you also need a full reboot? We are using Windows Server 2012 R2.
Not sure why, a reboot should not be needed, the installer stops the service before doing anything. But then with wcf anything can happen… sigh!
I manually restarted the services several times. I don’t know how the agent reads its properties, but maybe there might be an issue there? Existing environment variables that no longer exists? Just popping up some ideas which might help you find out the cause.
Hi,
I’m getting the same error. I need to reboot everytime to resolve this.
The server and agent is on the v1.0.0.2859.
Which error are you seeing, there the sqlce error or the NullReferenceException?
Hi Vincent,
NullReferenceException. Then, after the error, the message: “A call from the agent to the server failed, it will be re-tried again after the connection has been re-established.
There was no endpoint listening at net.pipe://localhost/ContinuaCI/IAgentCallbackService that could accept the message. This is often caused by an incorrect address or SOAP action. See InnerException, if present, for more details.” shows up on CI Server.
Even restarting the “ContinuaCI Agent Service” a lot of times, the message continue to shows up. Then, I need to reboot the system.
By the way, even restarting all the Continua CI services, the message shows up.
Hi Lucas
I wasn’t able to reproduce the problem here so it’s not possible to say that it’s fixed, but I made some changes to the error handling in the area where the error is occurring. Please try this build. It’s been running all day on our server without any errors.
http://downloads.finalbuilder.com/downloads/continua/1.0/ContinuaCI.Server.Setup_x64_1.0.0.2911.exe
http://downloads.finalbuilder.com/downloads/continua/1.0/ContinuaCI.Server.Setup_1.0.0.2911.exe
http://downloads.finalbuilder.com/downloads/continua/1.0/ContinuaCI.Agent.Setup_x64_1.0.0.2911.exe
http://downloads.finalbuilder.com/downloads/continua/1.0/ContinuaCI.Agent.Setup_1.0.0.2911.exe
Hi,
Thanks Vincent. I installed the build.