Build stuck at starting stage

Continua CI Bugs
1

One of our build got stuck at starting first stage. I am guessing that the agent had some networking issue for a moment, but is not available, but Continua is stuck and not proceeding.
The timeline looks like this:

Stage initialising 	47 minutes ago 	47 minutes 	Agent 'wrpbuildce02' which is executing stage is available active. Waiting one second before rechecking. Retry 170
Stage initialising 	47 minutes ago 	10 seconds 	Agent 'wrpbuildce02' which is executing stage is available active. Waiting one second before rechecking. Retry 160
Stage initialising 	48 minutes ago 	10 seconds 	Agent 'wrpbuildce02' which is executing stage is available active. Waiting one second before rechecking. Retry 150
Stage initialising 	48 minutes ago 	10 seconds 	Agent 'wrpbuildce02' which is executing stage is available active. Waiting one second before rechecking. Retry 140
Stage initialising 	48 minutes ago 	10 seconds 	Agent 'wrpbuildce02' which is executing stage is available active. Waiting one second before rechecking. Retry 130
Stage initialising 	48 minutes ago 	10 seconds 	Agent 'wrpbuildce02' which is executing stage is available active. Waiting one second before rechecking. Retry 120
Stage initialising 	48 minutes ago 	10 seconds 	Agent 'wrpbuildce02' which is executing stage is available active. Waiting one second before rechecking. Retry 110
Stage initialising 	48 minutes ago 	10 seconds 	Agent 'wrpbuildce02' which is executing stage is available active. Waiting one second before rechecking. Retry 100
Stage initialising 	49 minutes ago 	10 seconds 	Agent 'wrpbuildce02' which is executing stage is available active. Waiting one second before rechecking. Retry 90
Stage initialising 	49 minutes ago 	10 seconds 	Agent 'wrpbuildce02' which is executing stage is available active. Waiting one second before rechecking. Retry 80
Stage initialising 	49 minutes ago 	10 seconds 	Agent 'wrpbuildce02' which is executing stage is available active. Waiting one second before rechecking. Retry 70
Stage initialising 	49 minutes ago 	10 seconds 	Agent 'wrpbuildce02' which is executing stage is available active. Waiting one second before rechecking. Retry 60
Stage initialising 	49 minutes ago 	10 seconds 	Agent 'wrpbuildce02' which is executing stage is available active. Waiting one second before rechecking. Retry 50
Stage initialising 	49 minutes ago 	5 seconds 	Agent 'wrpbuildce02' which is executing stage is available active. Waiting one second before rechecking. Retry 45
Stage initialising 	50 minutes ago 	5 seconds 	Agent 'wrpbuildce02' which is executing stage is available active. Waiting one second before rechecking. Retry 40
Stage initialising 	50 minutes ago 	5 seconds 	Agent 'wrpbuildce02' which is executing stage is available active. Waiting one second before rechecking. Retry 35
Stage initialising 	50 minutes ago 	5 seconds 	Agent 'wrpbuildce02' which is executing stage is available active. Waiting one second before rechecking. Retry 30
Stage initialising 	50 minutes ago 	5 seconds 	Agent 'wrpbuildce02' which is executing stage is available active. Waiting one second before rechecking. Retry 25
Stage initialising 	50 minutes ago 	5 seconds 	Agent 'wrpbuildce02' which is executing stage is available active. Waiting one second before rechecking. Retry 20
Stage initialising 	50 minutes ago 	5 seconds 	Agent 'wrpbuildce02' which is executing stage is available active. Waiting one second before rechecking. Retry 15
Stage initialising 	50 minutes ago 	5 seconds 	Agent 'wrpbuildce02' which is executing stage is available active. Waiting one second before rechecking. Retry 10
Stage initialising 	50 minutes ago 	1 second 	Agent 'wrpbuildce02' which is executing stage is available active. Waiting one second before rechecking. Retry 9
Stage initialising 	50 minutes ago 	1 second 	Agent 'wrpbuildce02' which is executing stage is available active. Waiting one second before rechecking. Retry 8
Stage initialising 	50 minutes ago 	1 second 	Agent 'wrpbuildce02' which is executing stage is available active. Waiting one second before rechecking. Retry 7
Stage initialising 	50 minutes ago 	1 second 	Agent 'wrpbuildce02' which is executing stage is available active. Waiting one second before rechecking. Retry 6
Stage initialising 	50 minutes ago 	1 second 	Agent 'wrpbuildce02' which is executing stage is available active. Waiting one second before rechecking. Retry 5
Stage initialising 	50 minutes ago 	1 second 	Agent 'wrpbuildce02' which is executing stage is available active. Waiting one second before rechecking. Retry 4
Stage initialising 	50 minutes ago 	1 second 	Agent 'wrpbuildce02' which is executing stage is available active. Waiting one second before rechecking. Retry 3
Stage initialising 	50 minutes ago 	1 second 	Agent 'wrpbuildce02' which is executing stage is available active. Waiting one second before rechecking. Retry 2
Stage initialising 	50 minutes ago 	1 second 	Agent 'wrpbuildce02' which is executing stage is available active. Waiting one second before rechecking. Retry 1
Stage initialising 	52 minutes ago 	2 minutes 	Agent 'wrpbuildce02' which is executing stage is available active. Waiting one second before rechecking. Retry 50
Stage initialising 	52 minutes ago 	5 seconds 	Agent 'wrpbuildce02' which is executing stage is available active. Waiting one second before rechecking. Retry 45
Stage initialising 	53 minutes ago 	5 seconds 	Agent 'wrpbuildce02' which is executing stage is available active. Waiting one second before rechecking. Retry 40
Stage initialising 	53 minutes ago 	5 seconds 	Agent 'wrpbuildce02' which is executing stage is available active. Waiting one second before rechecking. Retry 35
Stage initialising 	53 minutes ago 	5 seconds 	Agent 'wrpbuildce02' which is executing stage is available active. Waiting one second before rechecking. Retry 30
Stage initialising 	53 minutes ago 	5 seconds 	Agent 'wrpbuildce02' which is executing stage is available active. Waiting one second before rechecking. Retry 25
Stage initialising 	53 minutes ago 	5 seconds 	Agent 'wrpbuildce02' which is executing stage is available active. Waiting one second before rechecking. Retry 20
Stage initialising 	53 minutes ago 	5 seconds 	Agent 'wrpbuildce02' which is executing stage is available active. Waiting one second before rechecking. Retry 15
Stage initialising 	53 minutes ago 	5 seconds 	Agent 'wrpbuildce02' which is executing stage is available active. Waiting one second before rechecking. Retry 10
Stage initialising 	53 minutes ago 	1 second 	Agent 'wrpbuildce02' which is executing stage is available active. Waiting one second before rechecking. Retry 9
Stage initialising 	53 minutes ago 	1 second 	Agent 'wrpbuildce02' which is executing stage is available active. Waiting one second before rechecking. Retry 8
Stage initialising 	53 minutes ago 	1 second 	Agent 'wrpbuildce02' which is executing stage is available active. Waiting one second before rechecking. Retry 7
Stage initialising 	53 minutes ago 	1 second 	Agent 'wrpbuildce02' which is executing stage is available active. Waiting one second before rechecking. Retry 6
Stage initialising 	53 minutes ago 	1 second 	Agent 'wrpbuildce02' which is executing stage is available active. Waiting one second before rechecking. Retry 5
Stage initialising 	53 minutes ago 	1 second 	Agent 'wrpbuildce02' which is executing stage is available active. Waiting one second before rechecking. Retry 4
Stage initialising 	53 minutes ago 	1 second 	Agent 'wrpbuildce02' which is executing stage is available active. Waiting one second before rechecking. Retry 3
Stage initialising 	53 minutes ago 	1 second 	Agent 'wrpbuildce02' which is executing stage is available active. Waiting one second before rechecking. Retry 2
Stage initialising 	53 minutes ago 	1 second 	Agent 'wrpbuildce02' which is executing stage is available active. Waiting one second before rechecking. Retry 1
Stage initialising 	55 minutes ago 	2 minutes 	Agent 'wrpbuildce02' which is executing stage is available active. Waiting one second before rechecking. Retry 50
Stage initialising 	55 minutes ago 	5 seconds 	Agent 'wrpbuildce02' which is executing stage is available active. Waiting one second before rechecking. Retry 45
Stage initialising 	56 minutes ago 	5 seconds 	Agent 'wrpbuildce02' which is executing stage is available active. Waiting one second before rechecking. Retry 40
Stage initialising 	56 minutes ago 	5 seconds 	Agent 'wrpbuildce02' which is executing stage is available active. Waiting one second before rechecking. Retry 35
Stage initialising 	56 minutes ago 	5 seconds 	Agent 'wrpbuildce02' which is executing stage is available active. Waiting one second before rechecking. Retry 30
Stage initialising 	56 minutes ago 	5 seconds 	Agent 'wrpbuildce02' which is executing stage is available active. Waiting one second before rechecking. Retry 25
Stage initialising 	56 minutes ago 	5 seconds 	Agent 'wrpbuildce02' which is executing stage is available active. Waiting one second before rechecking. Retry 20
Stage initialising 	56 minutes ago 	5 seconds 	Agent 'wrpbuildce02' which is executing stage is available active. Waiting one second before rechecking. Retry 15
Stage initialising 	56 minutes ago 	5 seconds 	Agent 'wrpbuildce02' which is executing stage is available active. Waiting one second before rechecking. Retry 10
Stage initialising 	56 minutes ago 	1 second 	Agent 'wrpbuildce02' which is executing stage is available active. Waiting one second before rechecking. Retry 9
Stage initialising 	56 minutes ago 	1 second 	Agent 'wrpbuildce02' which is executing stage is available active. Waiting one second before rechecking. Retry 8
Stage initialising 	56 minutes ago 	1 second 	Agent 'wrpbuildce02' which is executing stage is available active. Waiting one second before rechecking. Retry 7
Stage initialising 	56 minutes ago 	1 second 	Agent 'wrpbuildce02' which is executing stage is available active. Waiting one second before rechecking. Retry 6
Stage initialising 	56 minutes ago 	1 second 	Agent 'wrpbuildce02' which is executing stage is available active. Waiting one second before rechecking. Retry 5
Stage initialising 	56 minutes ago 	1 second 	Agent 'wrpbuildce02' which is executing stage is available active. Waiting one second before rechecking. Retry 4
Stage initialising 	56 minutes ago 	1 second 	Agent 'wrpbuildce02' which is executing stage is available active. Waiting one second before rechecking. Retry 3
Stage initialising 	56 minutes ago 	1 second 	Agent 'wrpbuildce02' which is executing stage is available active. Waiting one second before rechecking. Retry 2
Stage initialising 	56 minutes ago 	1 second 	Agent 'wrpbuildce02' which is executing stage is available active. Waiting one second before rechecking. Retry 1
Stage initialising 	58 minutes ago 	2 minutes 	Agent 'wrpbuildce02' which is executing stage is available active. Waiting one second before rechecking. Retry 50
Stage initialising 	58 minutes ago 	5 seconds 	Agent 'wrpbuildce02' which is executing stage is available active. Waiting one second before rechecking. Retry 45
Stage initialising 	59 minutes ago 	5 seconds 	Agent 'wrpbuildce02' which is executing stage is available active. Waiting one second before rechecking. Retry 40
Stage initialising 	59 minutes ago 	5 seconds 	Agent 'wrpbuildce02' which is executing stage is available active. Waiting one second before rechecking. Retry 35
Stage initialising 	59 minutes ago 	5 seconds 	Agent 'wrpbuildce02' which is executing stage is available active. Waiting one second before rechecking. Retry 30
Stage initialising 	59 minutes ago 	5 seconds 	Agent 'wrpbuildce02' which is executing stage is available active. Waiting one second before rechecking. Retry 25
Stage initialising 	59 minutes ago 	5 seconds 	Agent 'wrpbuildce02' which is executing stage is available active. Waiting one second before rechecking. Retry 20
Stage initialising 	59 minutes ago 	5 seconds 	Agent 'wrpbuildce02' which is executing stage is available active. Waiting one second before rechecking. Retry 15
Stage initialising 	59 minutes ago 	5 seconds 	Agent 'wrpbuildce02' which is executing stage is available active. Waiting one second before rechecking. Retry 10
Stage initialising 	59 minutes ago 	1 second 	Agent 'wrpbuildce02' which is executing stage is available active. Waiting one second before rechecking. Retry 9
Stage initialising 	59 minutes ago 	1 second 	Agent 'wrpbuildce02' which is executing stage is available active. Waiting one second before rechecking. Retry 8
Stage initialising 	59 minutes ago 	1 second 	Agent 'wrpbuildce02' which is executing stage is available active. Waiting one second before rechecking. Retry 7
Stage initialising 	59 minutes ago 	1 second 	Agent 'wrpbuildce02' which is executing stage is available active. Waiting one second before rechecking. Retry 6
Stage initialising 	59 minutes ago 	1 second 	Agent 'wrpbuildce02' which is executing stage is available active. Waiting one second before rechecking. Retry 5
Stage initialising 	59 minutes ago 	1 second 	Agent 'wrpbuildce02' which is executing stage is available active. Waiting one second before rechecking. Retry 4
Stage initialising 	59 minutes ago 	1 second 	Agent 'wrpbuildce02' which is executing stage is available active. Waiting one second before rechecking. Retry 3
Stage initialising 	59 minutes ago 	1 second 	Agent 'wrpbuildce02' which is executing stage is available active. Waiting one second before rechecking. Retry 2
Stage initialising 	59 minutes ago 	1 second 	Agent 'wrpbuildce02' which is executing stage is available active. Waiting one second before rechecking. Retry 1
Stage initialising 	1 hour, 1 minute ago 	2 minutes 	Agent 'wrpbuildce02' which is executing stage is available active. Waiting one second before rechecking. Retry 50
Stage initialising 	1 hour, 1 minute ago 	5 seconds 	Agent 'wrpbuildce02' which is executing stage is available active. Waiting one second before rechecking. Retry 45
Stage initialising 	1 hour, 2 minutes ago 	5 seconds 	Agent 'wrpbuildce02' which is executing stage is available active. Waiting one second before rechecking. Retry 40
Stage initialising 	1 hour, 2 minutes ago 	5 seconds 	Agent 'wrpbuildce02' which is executing stage is available active. Waiting one second before rechecking. Retry 35
Stage initialising 	1 hour, 2 minutes ago 	5 seconds 	Agent 'wrpbuildce02' which is executing stage is available active. Waiting one second before rechecking. Retry 30
Stage initialising 	1 hour, 2 minutes ago 	5 seconds 	Agent 'wrpbuildce02' which is executing stage is available active. Waiting one second before rechecking. Retry 25
Stage initialising 	1 hour, 2 minutes ago 	5 seconds 	Agent 'wrpbuildce02' which is executing stage is available active. Waiting one second before rechecking. Retry 20
Stage initialising 	1 hour, 2 minutes ago 	5 seconds 	Agent 'wrpbuildce02' which is executing stage is available active. Waiting one second before rechecking. Retry 15
Stage initialising 	1 hour, 2 minutes ago 	5 seconds 	Agent 'wrpbuildce02' which is executing stage is available active. Waiting one second before rechecking. Retry 10
Stage initialising 	1 hour, 2 minutes ago 	1 second 	Agent 'wrpbuildce02' which is executing stage is available active. Waiting one second before rechecking. Retry 9
Stage initialising 	1 hour, 2 minutes ago 	1 second 	Agent 'wrpbuildce02' which is executing stage is available active. Waiting one second before rechecking. Retry 8
Stage initialising 	1 hour, 2 minutes ago 	1 second 	Agent 'wrpbuildce02' which is executing stage is available active. Waiting one second before rechecking. Retry 7
Stage initialising 	1 hour, 2 minutes ago 	1 second 	Agent 'wrpbuildce02' which is executing stage is available active. Waiting one second before rechecking. Retry 6
Stage initialising 	1 hour, 2 minutes ago 	1 second 	Agent 'wrpbuildce02' which is executing stage is available active. Waiting one second before rechecking. Retry 5
Stage initialising 	1 hour, 2 minutes ago 	1 second 	Agent 'wrpbuildce02' which is executing stage is available active. Waiting one second before rechecking. Retry 4
Stage initialising 	1 hour, 2 minutes ago 	1 second 	Agent 'wrpbuildce02' which is executing stage is available active. Waiting one second before rechecking. Retry 3
Stage initialising 	1 hour, 2 minutes ago 	1 second 	Agent 'wrpbuildce02' which is executing stage is available active. Waiting one second before rechecking. Retry 2
Stage initialising 	1 hour, 2 minutes ago 	1 second 	Agent 'wrpbuildce02' which is executing stage is available active. Waiting one second before rechecking. Retry 1
Stage initialising 	1 hour, 4 minutes ago 	2 minutes 	Sending stage 'Clone' to agent 'wrpbuildce02'
Stage initialising 	1 hour, 4 minutes ago 	313 milliseconds 	Initialising stage...
Stage ready 	1 hour, 4 minutes ago 	63 milliseconds 	Agent 'wrpbuildce02' reserved and license allocated for stage 'Clone'
Stage awaiting agent 	1 hour, 4 minutes ago 	41 milliseconds 	Agent shared resources acquired for agent 'wrpbuildce02': Stage has acquired the following shared resource locks from agent 'wrpbuildce02': a read lock on agent shared resource 'Agent.AgentRepoStore.Clone' with single lock quota for current agent
Stage awaiting agent 	1 hour, 4 minutes ago 	117 milliseconds 	Reserving agent 'wrpbuildce02' for stage 'Clone' which is at position 1 in the stage queue.
Stage awaiting agent 	1 hour, 4 minutes ago 	437 milliseconds 	Adding stage 'Clone' to stage queue. There are currently 0 stages on the queue.
Stage awaiting agent 	1 hour, 4 minutes ago 	34 milliseconds 	Queuing stage 'Clone'
Stage not run 	1 hour, 4 minutes ago 	32 milliseconds 	Skip conditions for stage 'Clone' were not met
The expression ['$Build.HasFailedStages$' equals 'True'] evaluated to False. 'False' does not equal 'True'.
Build started 	1 hour, 4 minutes ago 	35 milliseconds 	Build started
Build starting 	1 hour, 4 minutes ago 	20 milliseconds 	Preparing to start build.
Build starting 	1 hour, 4 minutes ago 	81 milliseconds 	Associating build repository cache revisions
Build queued 	1 hour, 4 minutes ago 	29 milliseconds 	Starting build.
2

After build requeue, problem persists.

3

Restart of Agents did not help, I had to restart the server for the build to proceed.

4

Hi Michal,

It’s possible that an error occurred in the build process which caused the agent service to stop responding. Can you send us details of any errors which were logged in the Windows Event log on the agent at the time - particularly with sources ContinuaCIAgent or DotNetRuntime?

Also, as this could be a server issue, send us any errors logged on the Continua CI Event Log page - ideally send us the diagnostics report which you can download from this page. To avoid sharing publicly, send to me via direct message or via email to support at finallbuilder.com.

1 Like
5

Hi Michal,

Thanks for sending the log files and report.

We can see several communication timeout errors, which imply that the server can communicate with the agent, but the agent is unable to respond to the server. If this always occurs with this agent, we would suggest that agent-to-server communication is blocked by a firewall.

However, as you’d said that you were able to run builds after restarting the server, then this implies that something else must be blocking the communication. Unfortunately there is nothing in the logs to suggest what that may be. If this happens again, check the resource usage on the server, perhaps CPU, memory or disk space were maxed out?

We see that there is a error on the server related to communicating the database. It’s worth using pgadmin4 to check the size of your database hasn’t become to unmanageable and to ensure that it is fully indexed.

6

How to verify that it DB is fully indexed ?

As of size, I don’t think so, as it is working OK for 2 days now.

BTW. I had to reboot the Continua server, then builds started to work, so it really may be some DB access problem. But why ? DB is local for server.

Edit.
I checked, in terms of space the DB size is 12 GB

Edit 2.
Maybe I should run some maintanance on the DB like reindex and vaccum ?

7

Hi Michal,

Looking at the log file again, it appears that the database error occurred while the service was being shutdown - possibly because the PostgreSQL service was shutdown first.

Nevertheless, 12GB is quite large database size, so depending on the specs of your server, it’s probably worth adjusting your build cleanup rules to reduce the amount of build information stored. Also, although the database does have some auto-vacuum rules set, some regular database maintenance, e.g. VACUUM (FULL ANALYZE) ContinuaCI, can often improve PostgreSQL database performance.

Referring to the original issue, we don’t really have enough information to diagnose the cause. If this happens again, then it may be worth enabling debug logging on the server and agent so that we can have more details of what is happening at the time.

8

If it was shutdown, then this might mean it crashed as no maintanance was done at that time.

I read that vacuum may not remove all tuples unless reindex is done, is it save to do reindex on the DB ?

Edit.
I checked with pgAdmin, it seems that biggest table is “builds_unittestv2”, it’s 5.44 GB out of 11.67 GB total.
Second biggest is “builds_changesetfile” - 281.4 MB.
There are few around 100 MB, rest is really small.

What I noticed is that in the “builds_unittestv2” we have 2 very big indexes that are rarely used:

obraz
obraz1082Ă—241 38.5 KB

9

Hi Michal,

Yes, it is safe to run REINDEX. Just note that it will take an exclusive lock on the index which will block queries that use the index, so it’s not good to reindex while Continua is running.

All the large indexes are used by the various filtering options on the unit tests page. It is possible that we could remove some of the smaller indexes, although this would require further investigation.

If you need to keep a history of builds, but don’t need to keep such a long history of the associated unit tests results, then you can add a new cleanup rule which is set to only clean up unit tests.