Workspace Sync takes long time

Continua CI Discussion
1

When building a project config the “Server To Agent Workspace Sync” typically takes around 8 minutes. Is there a way we can reduce this time please? Looking at the timeline the step logged as “Initialising workspace on agent ‘xxxxxx’” typically takes 7 minutes. I have enabled “Log workspace files copied” and the sync still takes about 7 minutes even on the first stage when there doesn’t seem to be any files to copy.

Thanks
Darren

2

Hi Darren,

This timeline status also involves the exporting source files from a repository cache on the agent to the build workspace. Each build has its own workspace (so that multiple builds running at the same time do not interfere which each other) .The repository rules control which folders and files are copied to this workspace.

The default repository rules will copy all the files related to the current changeset for all associated repositories to the build workspace. How long this takes depends on the number and size of the files in your repositories and the speed of your hard disk. We recommend amending the repository rules for each stage so that only the files required for the build are copied to the workspace. This can significantly reduce build time. There is an option to log the repository files copied under the repository rules for each stage.   

Note that anti-virus software can also affect performance when exporting files to the workspace, so we would recommend excluding the workspace folder from anti-virus any real-time scanning (use scheduled anti-virus scans instead). 

You can also reduce the size and number of files in the repository cache using the new Path Filtering patterns in the repository settings to exclude any files which are never used in a build e.g. documents, artifacts and archives. Depending on your repository type, you may also be able to relative folders in your branch pattern to limit files to a specific path in your repository.

Note that the link to the log on your internal build server is not accessible to us.



3

Hi Dave

Thanks for getting back to me. I have enabled the option to log the repository files copied and this is what I get for the first stage.

16:40:17   Server To Agent Workspace Sync
16:40:17   Started syncing files from the server[localhost] to the agent [FLREP001]
16:45:17   Workspace Syncing
16:45:17   Using UNC Transport.
16:45:17   /Output/** >> /Output
16:45:17   Cleaning target directory : C:\Continua\CI_WS\Ws\74835\Output
16:45:17   0 files found.
16:45:17   __variables* > __variables
16:45:17   \FLREP001\Continua\Ws\Fusion\74835__variables\FLREP001_Environment.xml > C:\Continua\CI_WS\Ws\74835__variables\FLREP001_Environment.xml
16:45:17   \FLREP001\Continua\Ws\Fusion\74835__variables\VariableNamespaces.xml > C:\Continua\CI_WS\Ws\74835__variables\VariableNamespaces.xml
16:45:17   Finished syncing files from the server [localhost] to the agent [FLREP001].


As you can see it takes 5 minutes to copy 0 files!?! As it happens the server and agent are on the same machine so I think this must be a Continua overhead rather than a file copying or network issue. I’ll email the complete log to support as this forum won’t let me upload attachments with a file extension of LOG, TXT or even PDF and I didn’t want to clutter it up by posting the entire log in a forum post.

Thanks
Darren
4

Hi Darren,

Can you turn on the "log the repository files copied" option in the Repository Rules tab of the Stage Options dialog and then run the build again? You should then see a Repository Syncing section in your build log showing a number of files being copied.

5

Hi Dave

I would have sworn I had enabled that option but apparently not!

I can now see that the Workspace Sync step performs a Repository Sync which is probably what is taking the time. Is there a way to reduce this time at all please? Beside the obvious get a faster Git host connection! :wink:

Regards
Darren

6

Hi Darren,

This stage of the process isn’t using Git at all so a faster Git connection is not relevant here. Your options are either to get a faster hard disk or to reduce the number of files exported to your workspace.

What is the total size of the files in the relevant branch for your repository? Are all the files in the repository required for the build? 

You can use the Path Filtering files in the Filtering tab in the Edit Repository dialog and the Repository Rules in the build Stage Options dialog to include specific folders required for the build and/or exclude folders and files types not required for the build.

Maybe I can clarify the process by listing the steps taken to transfer the repository files to the build workspace.

1. The server polls your Git repository at intervals to detect changesets.  
2. When a new changeset is detected, the server checks out the Git repository files to the server repository cache working folder.
3. The server then commits the files in server repository cache working folder to server repository cache (a Mercurial repository).
4. When a build stage is run, the agent transfers changes from server repository cache to agent repository cache (another Mercurial repository).
5. The agent checks out files from the agent repository cache to the build workspace.

The Path Filtering patterns can be used limit the files committed to the server repository cache at step 3.
The Repository Rules can be used to limit the files exported to the build workspace at step 5.