Yesterday I tried to create a new repository and I succeeded.
This new repository fetches the changesets successfully.
However, I think something I did broke the “official”. It has been stuck on “Getting changesets” for practically a whole day.
I tried restarting, I tried stopping and going back, I tried running “reset” and “check now”, nothing works, it keeps looking for the changesets.
When trying to run any manual command, it does not find the branches and nothing else.
I know that the other times I needed to restart the server it took a while to get the changesets, but never this long, so I think there is something wrong.
What can I do?
Please, help! I’m desperate! Almost 3 days without our builds…
Hi
You really need to provide us with much more to go on - very hard to tell what is happening with so little information.
See Troubleshooting - Continua CI - VSoft Technologies Documentation Wiki
Generate a diagnostics report from the admin\event log page to send to support @ finalbuilder.com
Also enable Debug logging Debug Logging - Continua CI - VSoft Technologies Documentation Wiki on the server and restart it, let it run for a while, then stop the service and send us the log.
Are you able to run git commands from the command line while logged in as the same user as the Continua Server service? The most common git hanging issue is it’s prompting for credentials, what setting do you have for the credentials. I’m just guessing here since you didn’t give us much to work with.
It’s a holiday here today (Australia Day) so none of our team is working - also for users of the free version (which I can only assume since I don’t see any licenses under your name) support is provided as and when we are able (ie working hours).
Sorry for the lack of information, I really don’t know where to run.
I activated the logs, in minutes it is already over 20 mb, I will send it.
The service is active, and I can run git commands there, I even activated another repository with the same token to check.
Yesterday I was even activating another build process for another repository, but it is a secondary repository with much fewer changes than the broken one.
We haven’t received any debug log file from you yet. Is your repository working now?
To diagnose this type of issue, we first need to check for any errors on the administration event log page. This is why Vincent asked you for the diagnostics report which contains a list errors in the event log. It also contains the repository settings which can help us to reproduce the issue.
Furthermore, we need to know information about your repository size, number of commits, branches and tags. Depending on your repository settings, if there are a lot of branches and/or tags which need to be checked out and added to the Continua repository cache this can take some time, although we certainly wouldn’t expect it to take several days. The debug log is large as it is quite detailed, but is very useful as it can show us want is happening during the process of checking for changesets and also provide some information about your repository. This is especially useful if you can reset the repository while the debug log is active.
One thing that often has a significant effect on the performance of repository checking and initialisation is anti-virus protection. We recommend excluding the workspace and repository cache folders from any real-time anti-virus protection.
Hi, actually i’ve sent it on friday.
This log I sent was just over 22 mb. And it was just a few minutes after it was activated.
On Sunday I logged in again and the server was working as it should.
From what I could see the first build was run on Saturday around noon.
It’s a very large repository, with a lot of history, but we only work with 6 developers.
Today we have around 90-100 simultaneous branches
The last time it was restarted it took 2 or 3 hours to download everything, now it takes 1 full day.
There is anything more I can provide you?
Hi @balena,
We haven’t received your email. I suspect that the 22mb attachment was blocked somewhere along its route. Generally we would recommend zipping up such files before emailing or sending them via a file transfer service.
Hello
I sent it again.
It was compressed, I said 22 but it’s 26 mb, gmail doesn’t let you send it that big, compressed it was a few kb.
Just for reporting.
On the day I activated the log, 01/26: 500 mb of logs.
01/27: 3.25 GB.
01/28: 366mb
01/29: 432mb
01/30: 517 mb.
It’s probably stable now, with the 27th being the one where a lot of data had to be downloaded.
Hi @balena,
Thanks for re-sending the debug log - we have received it now.
The first thing we notice is that you are using version 1.9.1.800 of Continua which is over 3 years old. Many fixes and improvements have been made since then, including some relevant ones. We recommend upgrading as soon as you are able to.
We also see that git log -n 1 takes approximately 500ms to run on your server. This is quite slow - it takes 40ms to run on a very large repository on my workstation. When a repository is initialised we use git log -n 1 to get full details about the head commit for each branch. As your repository is set up to include pull requests, there are 6132 branches in total and the git log command is run for each of them. Also, the option “Load any older changesets which have been tagged” is checked, which means that Continua lists all tags related to each branch and then runs git log on each of those tags to get changeset details.
We have been working on reducing the number of commands required to get the list of tagged changesets after previously noticing that this can be quite slow when there is a large number of branches, so expect to see some changes related to this soon.
We are also looking into the possibly of changing the logic and locking around getting changesets to allow builds to run for one branch before all changesets for all branches are retrieved.
Meanwhile, these are your options:
- Avoid resetting the repository, although this is not always possible. We’re not sure what caused your repository to get into a state where it needed to be reset. When you create a new repository it’s workspace is isolated in a new folder with a unique id so it’s not really plausible that it would interfere with an existing one.
- Switch off the option “Load any older changesets which have been tagged”, if you never build old tags.
- Reduce the number of branches, either by archiving any unused branches and pull requests (see how to close a branch in git - Stack Overflow), or adapting your branch pattern to include only the branches you use.
I updated and disabled the logs now at noon.
Let’s see how the server will behave.
However, I had this problem in one of the builds this afternoon.
The agent ‘continuaci2’ which was executing stage ‘Build’ is unavailable. Agent status is Online, Authorized, Licensed, PropertiesCollected. Agent last registered with the server at 2:40:21 PM. Agent status was checked at 2:40:50 PM. Agent communication test was successful. This implies the agent is having a problem registering with the server - check the Windows event log on the agent machine.
After running the command again, it generated the build.
As for your tips, thank you.
You mention that we have more than 6k branches, it could be PR, but we only have 50-60 open now. We have less than 100 active branches. Would that be a lot?
Regarding the configuration: “Load any older changesets which have been tagged”. Would that really be a problem? I wasn’t the one who configured the serer, I don’t know exactly what its use is.
As for slowness on the network, we will also evaluate it.
Anyway, now it’s very stable, despite the error I mentioned above. Thank you for your help.
Is there anything in the Windows event log on the agent machine? The message is a bit odd. We’ll do some investigation on this.
When the option “Fetch other remote ref” is checked for the repository, all remote references which match the Other Refs field will be fetched and treated as a branch. By default this is “+refs/pull//merge:refs/remotes/origin/pr/” and this means that all merge pull requests with be treated as branches with the name /pr/1 etc. See Building GitHub Pull Requests with Continua CI for further info on Other Refs. There’s no way with the git command line to know whether these pull requests are open or closed. I suspect the number of branches is so high because you have a lot of closed pull requests?
On initial repository load, e.g. after creation or reset, Continua adds changeset details for all the branch heads to the database and, if this option is checked, then it also adds a changeset for each tag. Continua attempts to get the branch that each tag was created on. It’s doing this by listing the tags for each branch, which is quite slow when there is a large number of branches. We have been doing some work on making this task more efficient in the last few days and hope to get an update out soon.