Hi there!
We’ve recently started using Continua to build one of our smaller projects, but we’re having a big issue with one of the Git repository changed triggers. The Repository Trigger crashes every morning at 02:00AM and stops running until the project is manually started or the server rebooted. This means that manual action has to be taken each morning to ensure continuous builds throughout the day. (Just starting a normal build on the configuration seems to make the trigger start working again). I’m assuming this is caused by the connection to the repository-server being lost, but the trigger should resume working once the connection comes back up?
I’ve attempted to include all the relevant info below.
Thanks in advance,
Olemartin.
Continua Version:
v1.0.0.3205 32 bit
Operating System for Continua and Agents:
Windows Server 2003 R2 Standard (Continua Server), Windows Server 2003 R2 Standard (Agent)
Database Platform:
Bundled PostgreSQL
How much CPU / RAM / Diskspace do you have free on the Continua Server, Agents, and Database:
65-95% free cpu, 66% free RAM (2/3gig), 8% free hard disk (8/100 gig)
A detailed description of the issue:
- What is the issue, and what are the outcomes?
The Continua Repository Trigger crashes every day at 02:00 and stops running until the project is manually started or rebooted.
- When did the issue first occur?
The day after setting up said trigger.
- Does it occur all the time?
Every day between 02:00 and 02:10
- Trigger info:
Name: Changes
Build Priority: Normal
Enabled: True
Force Repository Check: True
Type: Repository
Repository: GitRepo
Quiet Period(min): 1
Associate Changesets: Latest
Trigger from: Default branch
First error message thrown:
Importance: Error
Date: Saturday, September 13, 2014
Time: 2:02:17 AM
Message:
Unhandled Service Exception
Exception: GenericADOException
Message: could not execute query
[ select project0_.Id as Id9_, project0_.Name as Name9_, project0_.NameLower as NameLower9_, project0_.Description as Descript4_9_, project0_.Archived as Archived9_, project0_.Slug as Slug9_, project0_.VariableNamespaceId as Variable7_9_ from core_project project0_ where lower(project0_.Slug)=:p0 and not (project0_.Archived=TRUE) limit 1 ]
Name:p1 - Value:tot61
[SQL: select project0_.Id as Id9_, project0_.Name as Name9_, project0_.NameLower as NameLower9_, project0_.Description as Descript4_9_, project0_.Archived as Archived9_, project0_.Slug as Slug9_, project0_.VariableNamespaceId as Variable7_9_ from core_project project0_ where lower(project0_.Slug)=:p0 and not (project0_.Archived=TRUE) limit 1]
Stack Trace: at NHibernate.Loader.Loader.DoList(ISessionImplementor session, QueryParameters queryParameters)
at NHibernate.Loader.Loader.ListIgnoreQueryCache(ISessionImplementor session, QueryParameters queryParameters)
at NHibernate.Loader.Loader.List(ISessionImplementor session, QueryParameters queryParameters, ISet1 querySpaces, IType[] resultTypes)<br> at NHibernate.Hql.Ast.ANTLR.Loader.QueryLoader.List(ISessionImplementor session, QueryParameters queryParameters)<br> at NHibernate.Hql.Ast.ANTLR.QueryTranslatorImpl.List(ISessionImplementor session, QueryParameters queryParameters)<br> at NHibernate.Engine.Query.HQLQueryPlan.PerformList(QueryParameters queryParameters, ISessionImplementor session, IList results)<br> at NHibernate.Impl.SessionImpl.List(IQueryExpression queryExpression, QueryParameters queryParameters, IList results)<br> at NHibernate.Impl.AbstractSessionImpl.List(IQueryExpression queryExpression, QueryParameters parameters)<br> at NHibernate.Impl.ExpressionQueryImpl.List()<br> at NHibernate.Linq.DefaultQueryProvider.ExecuteQuery(NhLinqExpression nhLinqExpression, IQuery query, NhLinqExpression nhQuery)<br> at NHibernate.Linq.DefaultQueryProvider.Execute(Expression expression)<br> at NHibernate.Linq.DefaultQueryProvider.Execute[TResult](Expression expression)<br> at System.Linq.Queryable.FirstOrDefault[TSource](IQueryable1 source)
at Continua.Projects.ProjectManager.TryGetProjectBySlug(String slugName, Project& project)
at Continua.Projects.ProjectManager.GetProjectBySlug(String slugName)
at Continua.Modules.Builds.Services.BuildUIService.GetConfigStatistics(String slug, String configName)
at SyncInvokeGetConfigStatistics(Object , Object[] , Object[] )
at System.ServiceModel.Dispatcher.SyncMethodInvoker.Invoke(Object instance, Object[] inputs, Object[]& outputs)
at System.ServiceModel.Dispatcher.DispatchOperationRuntime.InvokeBegin(MessageRpc& rpc)
at System.ServiceModel.Dispatcher.ImmutableDispatchRuntime.ProcessMessage5(MessageRpc& rpc)
at System.ServiceModel.Dispatcher.ImmutableDispatchRuntime.ProcessMessage41(MessageRpc& rpc)
at System.ServiceModel.Dispatcher.ImmutableDispatchRuntime.ProcessMessage4(MessageRpc& rpc)
at System.ServiceModel.Dispatcher.ImmutableDispatchRuntime.ProcessMessage31(MessageRpc& rpc)
at System.ServiceModel.Dispatcher.ImmutableDispatchRuntime.ProcessMessage3(MessageRpc& rpc)
at System.ServiceModel.Dispatcher.ImmutableDispatchRuntime.ProcessMessage2(MessageRpc& rpc)
at System.ServiceModel.Dispatcher.ImmutableDispatchRuntime.ProcessMessage11(MessageRpc& rpc)
at System.ServiceModel.Dispatcher.ImmutableDispatchRuntime.ProcessMessage1(MessageRpc& rpc)
at System.ServiceModel.Dispatcher.MessageRpc.Process(Boolean isOperationContextSet)
Exception: NpgsqlException
Message: A timeout has occured. If you were establishing a connection, increase Timeout value in ConnectionString. If you were executing a command, increase the CommandTimeout value in ConnectionString or in your NpgsqlCommand object.
Stack Trace: at Npgsql.NpgsqlState.ProcessBackendResponsesEnum(NpgsqlConnector context)
at Npgsql.NpgsqlReadyState.QueryEnum(NpgsqlConnector context, NpgsqlCommand command)
at Npgsql.NpgsqlConnector.QueryEnum(NpgsqlCommand queryCommand)
at Npgsql.NpgsqlCommand.GetReader(CommandBehavior cb)
at Npgsql.NpgsqlCommand.ExecuteReader(CommandBehavior cb)
at Npgsql.NpgsqlCommand.ExecuteDbDataReader(CommandBehavior behavior)
at System.Data.Common.DbCommand.System.Data.IDbCommand.ExecuteReader()
at NHibernate.AdoNet.AbstractBatcher.ExecuteReader(IDbCommand cmd)
at NHibernate.Loader.Loader.GetResultSet(IDbCommand st, Boolean autoDiscoverTypes, Boolean callable, RowSelection selection, ISessionImplementor session)
at NHibernate.Loader.Loader.DoQuery(ISessionImplementor session, QueryParameters queryParameters, Boolean returnProxies)
at NHibernate.Loader.Loader.DoQueryAndInitializeNonLazyCollections(ISessionImplementor session, QueryParameters queryParameters, Boolean returnProxies)
at NHibernate.Loader.Loader.DoList(ISessionImplementor session, QueryParameters queryParameters)
Hi,
Thank you for including full details with your report. I think however that we will also need a debug log to solve this one. Can you enable debug logging for the server and agent and send the log files to support@finalbuilder.com. We recommend installing version v1.5 before doing this as any fixes we do will be applied to the latest version. Also version 1.5 includes an update to the postgresql database which my be relevant in this case.
Also can you let us us know what type of scheduled tasks are running on the server at 2am? The error message implies that there is a database connectivity problem. This could be due to server overload or file locking during a back up or anti-virus scan?
Is the repository listed as in an error state? If Continua loses connectivity to a repository it will keep trying for up to 5 minutes before requiring a reset. If there are always connectivity issues a a specific time, you can set up downtime periods in the repository dialog. This will tell Continua not to check the repository during these periods.
Thank you for the quick response. I’ve updated to v1.5 and activated logging.
We got scheduled virus-scans running at 2am, so you are probably correct about that locking up the database for more than 5min causing the problem. I’ll make sure to activate the downtime setting to prevent it from connecting during those hours, which should probably prevent the error from occuring. Is there any way of having the repository trigger restart by itself, ie check if the connection is restored every 30/60min?
If you’d like I can try to reproduce the error and send you the logs before activating the downtime setting.
Hi,
It’s probably too late now (note that we are in Australia - UTC+10), however it would be good if you could produce a debug log before activating the downtime so we use this to narrow down the cause of the issue and therefore work out the best way the recover. It’s not clear just now whether the problem is due to the database connectivity issue or a repository connectivity issue.
Which anti-virus software are you using? It may help to exclude the folder %ProgramData%\VSoft\ContinuaCI\PostgreSQLDB from the anti-virus scan if the software allows you to do that.
I’ve not yet activated the downtime, but it didn’t crash last night. This could be an anomaly, so I’ll let it run without the downtime(and debug logs activated) for another few days to see if it’s reproducable. If not, it seems like upgrading to v1.5 did the trick, as I’ve done no other alterations to the current setup.
We’re using Symantec Endpoint Protection, but I’m not authorised to exclude search folders unfortunately.