Hi guys,
I’ve got a question about the workspace initialization on Continua CI agents. If all changesets are synced to a CI agent the workspace initialization is a completely local operation on the client, isn’t it? Our Git repository has almost 50,000 mostly small files. We just recently had to add over 38000 files to it for our new web project using Angular 4. After that the workspace initialization takes almost an hour compared to about two minutes before we added the files. Our build server is hosted in VMware ESXi 6.5.0 with it’s virtual disk on a SATA-SSD (RAID 1). CPU and memory seem to be okay as the CPU (2x 2.1 GHz) usage for the VM is at about 10 percent and the memory is about 50 % full (4 GB) during the initialization. Do you know something about file amount limitation in mercurial?
Kind regards
Kay Zumbusch
Hi Kay
When a build starts, the server tells the agent to sync it’s repository cache(s), effectively doing a mercuiral pull. The agent exports the changeset from it’s repocache to the workspace as per the repository rules on the stage. I’m not aware of any limitations of mercurial for dealing with lots of files, however that doesn’t mean there are none. Also, with SSD’s the export should not be that slow. Our hg repo for continua itself is around 30K files, and workspace init takes around 2 minutes (our hyper-v server has sas raid 1 drives, not ssd’s).
I’m going to guess that perhaps you have logging turned on for the repository or workspace rules on the stage? That would slow it down a lot, especially with a lot of files. It would also result in very large log files in the server’s build workspace. If that’s the case, turn logging off (should onlt be used for debugging workspace/repo rules), if that’s not the case, then we will need to investigate what is happening on the agent during that time (enable debug logging on the agent and run a build).
I just checked the log settings. We only activated logging for the workspace rules. That’s just about 50 files that are copied after the build and does not have an impact add all. The log files in the artifacts are also not to worry about. They are at about the same size for every agent (our builds always pass at least two agents).
Yesterday we had additional problems with the Web UI. My colleagues got multiple database timeout messages from NHibernate. They found a lot of locks to dbo.builds_changesetfile and disconnected the SQL Server connection that caused the locks. That seem to have improved the performance quite a lot as the next build just took 25 minutes for the workspace sync. In addition I rebuilt the indexes on that table as their fragmentation level was between 60% and 99%. The following build just took 12 minutes for the workspace sync although the debug logging was activated on the agent before I started the build.
I’ll send the debug log from the agent for further investigation. I thought that I still had debug logging enabled on the server as well but I disabled it about a year ago.
Kind regards
Kay Zumbusch