Hi guys,
what exactly is the purpose of change sets and storing a list of all changed files in the database?
We merged a huge feature branch into our main development branch yesterday (git repository, about 2500 commits, about 40.000 added and modified files). The Mercurial update seems to have completed within two minutes as the output of “hg log” of the Mercurial repository shows the latest commit from the feature branch as the head of the develop branch in Mercurial. Since then the repository is listed as “Saving changesets” in the Continua CI WebUI and ContinuaCI continuously writes data into “dbo.builds_changeset” and “dbo.builds_changesetfile”. This operation takes ContinuaCI about 4.5 minutes per changeset with 100% CPU usage for the Continua CI Server process. A rough calculation would assume that this step could take about 180 hours (!) which is just rediculous. The database server is at
Registering the changesets in the database sounds okay to me to be about to run a build for a specific changeset. Including a list of all updated files sounds strange to me as the Mercurial repository already stores this information.
Kind regards
Kay Zumbusch
I’d like to add that during the times with high CPU load the only disk I/O operations performed by Continua CI server are loading of type libraries and .NET framework assemblies (e.g. activeds.tlb, clr.dll, clrjit.dll) and writing the debug log file. While we usually have just one or two log files per day this increased to about eight log files per day during this sync operation. There are log entries from the Repository Monitor and Background monitor about every 200 ms.
I also recognized the following pattern on the database:
- search for changeset ID in dbo.builds_changeset (~15 ms)
- silence for the same SPID for about 5 minutes
- insert the changeset into dbo.builds_changeset (max 20 ms; usually 0 ms)
- insert associated files into dbo.builds_changesetfile (max 20 ms, usually 0ms)
There are more database request from other threads in ContinuaCI server (e.g. agent property sync, LDAP sync, etc.) but I filtered most of them out to get only requests from the repository syncing process. The CPU load on the database is always below than 5 percent and usually even below 1%. The ContinuaCI database is the only database on the server that is accessed permanently.
I would like to restart the ContinuaCI server and disable debug logging to reduce the load even more. But will the sync process continue after the restart and will the transaction for the sync be rolled back or committed?
Kind regards
Kay Zumbusch
Hi Kaye,
We’ll look into what’s happening with performance when adding changsets today and over the next few days. This area of the code has not been looked at for some time. Several minutes per changeset is not at all acceptable, we will look into possible causes for this.
The changesets are committed to the database in batches. If you shutdown the service any changesets that have not yet been saved to the database will never get added to the database, previously committed batches will not be rolled back. If you don’t care about the changeset history and only care about building the latest changesets, then it may be worth shutting down the service, then resetting the repository once the service is started up again. Resetting the repository will load only the latest changeset for each branch, which should complete much quicker.
If you can then send us the debug log, this may help us with our investigations, however there doesn’t appear to be a lot of debug logging in the relevant area of code. Can you also give us some details on the number of rows in the builds_changeset and builds_changesetfile tables?
The files are added to the database primarily for display in the UI and are not required generally for builds. It is however possible to iterate through the list changeset files in build notifications and this functionally requires that the files are in the database before the build starts.
The main issue here however appears to be a unprecedented number of changesets added in one go - maybe we don’t need to add all of them - we’ll look into ways of dealing with this.
Note that we recommend that debug logging is only turned on when diagnosing an issue. Logging will affect performance as file writes are serialised.
Hi Dave,
I’ll send you our log files and the additional information to your support mailbox.
Kind regards
Kay Zumbusch
By the way: the repository reset uses an incredible amount of resources as well. We increased the memory for the CI host VM to 8 GB and the reset still has to use the swap file. CPU load on the CI host is maxed out.
During the reset there is a reasonable load on the database host with short CPU load peaks almost up to 100% and the row count on builds_changeset and builds_changesetfile drops continuously. We started at about 80,000 changesets and 5.2 million changeset files and are down to 25,000 and 1.5 million after two hours.
Hi Kay
Something we didn’t consider yesterday is that during a reset, any changesets that do not have builds associated with them would be deleted. This would account for the high load. I suspect there is probably room for optimisation of that code as well. We did some testing yesterday with changesets that have large numbers of files (we used 50K files per changeset), and were able to reproduce the performance problem, although not quite to the extent that you are seeing. It’s going to take some time to figure out the best way to optimise this code and it’s quite complex so will need some serious testing.
We will probably also look at limiting the number of files per changeset that are recorded. The files are recorded as they are available in notification templates, however no notification is going to want to list that many files, so that will need a rethink.
Hi Vincent,
the reset reduced the number of changesets significantly. Mostly because I removed all logged builds from the history with the cleanup task in order to get rid of as many changesets as possible. After the reset we are down to 90 changesets with 160,000 files for this repository and 310 changesets with 165,000 files for all repositories. The reset took almost five hours and reduced the database size from about 4.5 GB to a mere 500 MB. We are down to 50 branches instead of 220 branches before the reset as we cleaned up feature branches during our release process that took place while the CI environment was busy saving changesets.
The removal of changesets from the database looked a bit strange. Every row in the changesetfile table is explicitly deleted via its ID. I guess that’s how NHibernate handles the deletion of the items.
Our CI environment is working at the moment so we no longer need immediate assistance with our CI environment. But even after the reset the time to save changesets seems to be off. We added a single commit to one feature branch with only a single file changed and it took Continua CI about two minutes to save the changeset. The high CPU load without I/O load was back and I could not figure out what might happen in background.
Kind regards
Kay Zumbusch
Hi Kay,
Please try version 1.8.1.655. We’ve made significant improvements to the code for saving (and deleting) changeset files to the database.
Hi Dave,
thanks a lot. I installed the latest version last week and we finally had another big merge with 19,000 updated, deleted and added files. As I expected to still take some time to save the changesets I did not check the repo state right after the merge. When I checked about five minutes later the repo was already synced and the merge was fully stored in the database. Great work, guys. I appreciate it.
Kind regards
Kay Zumbusch