Hi,
I’m starting to work with continua. My problem is now that i have to work with two large svn-repositories (3 - 6 GB), which are uploaded to my agent-cache. But everytime when I build a stage all data is copied from the agent-cache to agent-workspace and this takes too much time.
Is there a way to omit this copying from cache to workspace?
My idea was:
- checkout the repositories on my agent.
- save the path to this repo in a system-variable.
- check on server with a property-collector if this variable is set on my agent.
and on each build:
- cleanup the svn-repo on path of this system-variable.
- update svn-repo to needed revision.
- work on this local svn-repo instead of the workspace.
My questions are now:
- Can i do this whit continua?
- Or is there a options so that the agent works directly on the cache and not on workspace?
- Is it a good solution or are there better ones?
- Where can i get the svn-version (e.g. which triggered the build) to which i have to update my repo on agent?
Hi Rynardald,
Continua CI is designed to run builds concurrently, we therefore use a separate clean workspace for each build. The files are often updated during the build, so we also need to make sure that these changes do not affect subsequent builds.
Our recommendations for dealing with large repositories are as follows:
1. Identify which repository folders and files are required for a build configuration. Do you really need several GB of files for your build?
2. Use relative folders and branches to narrow down your repositories to a specific project.
For example, if your Subversion repository structure is:
/Project1/trunk/Project1/branches/branch1/Project1/branches/branch2/Project2/trunk/Project1/branches/branch1…etcSet your Default Path to /Project1/trunk and Branches Path to /Project1/Branches
/trunk/Project1/trunk/Project2/Project1/branches/branch1/Project1/Project1/branches/branch1/Project2/Project1/branches/branch2/Project1/Project1/branches/branch2/Project2…etcSet your Default Path to /trunk/Project1 and Branches Path to /Branches/Project1
Note that you can use multiple Continua CI repositories per Subversion repository and per configuration as required.
3. Use Repository Rules to limit the number of files which are copied to the workspace. The default repository rule $Source$ > Source copies all repository files from the agent repository cache to the agent workspace at the start of each stage. These can be changed to include or exclude certain folders or files e.g. $Source.Repo1$\Source** >> \Source\Repo1Source. This is the key to speeding up builds.
If your build really requires a large number of files, then your solution may work. FinalBuilder or other command line or script could be used to deal with the repository check-out on the agent. You will however need to make use of Shared Resources to ensure that multiple builds do not access the folder at the same time. You should also clear the repository rules so that files don’t get copied to the workspace anyway. The build changeset revision is available using the expression $Source.CT_Source.BuiltChangeset.Id$
Alternatively, if your repository contains separate library folders with a large number of files which don’t change very often, we suggest using a separate build configuration to copy these files to a shared folder.
1. Create a new configuration e.g. “Copy Library”.
2. Set up a Continua CI repository pointing to the repository folder containing the library files and associate this repository with the configuration.
3. Add a Delete action to clear to the shared folder
4. Add a Copy action to copy the $Source.RepoName$ folder to the shared folder
5. Set up a repository trigger on the new configuration.
6. Create a Server Shared Resource of type Infinite e.g. “Shared Libraries”.
7. Add this shared resource to the Configuration Conditions for the “Copy Library” configuration with a Write lock.
In your main build configuration:
1. Set up your build actions to link to the shared folder.
2. Set up your repositories and repository rules to exclude the folder containing the library files
2. Add the “Shared Libraries” shared resource to the Stage Options for any stage which requires access to the shared libraries folder. Set it to acquire a read lock. This will allow any number of build stages to run unless a Copy Library build is running in which case the stage will wait for that build to finish. Note that we don’t recommend changing any files in the shared folder as this may impact subsequent builds - but if you need to do this acquire a write lock instead.
I hope this helps. Please let us know which options you choose.