I’m working on adding a job that builds our Android application - to our Continua CI environment.
99% of the pipeline is ran through Fastlane (Continua CI triggers these Fastlane builds for us).
Now, there’s some git commands that are supposed to run as part of the Fastlane “lane” (pipeline)
But because these git commands are actually executed on the agent server, which only holds a copy of the source directory’s contents that was synced to it from the CCI server And - because this isn’t actually a local git clone (missing the .git metadata folder) - all git commands running on it from the agent via the Fastline pipeline - obviously fail.
I was wondering, could you consider adding an optional feature flag in the repository settings scope - that also syncs the .git metadata folder from the server to the agent?
Also, I’d kindly love to hear your own opinion on this, do you think there’s a more clean/correct solution to this problem?
For example - for our iOS app builds I use a “SSH Run Script” action from Continua CI - with a few bash lines that maintain a local git repository that exists on the remote iOS server at a fixed path.
So Fastlane has no issues running git commands there, as this is an actual git repo.
But for our Android builds I’ve decided to utilize Continua CI’s “natural” way of doing things
(Source is synced from Server to Agent).
I was hoping you could help me figure out what is the best solution for this dilemma, and consider adding the flag for syncing .git metadata folder.
Stage repository rules have this rule by default: $Source$ >> Source
And there are no include/exclude filters in the repository settings.
Is there any way I could use either or both these existing mechanisms to force sync of the .git metadata folder?
Perhaps you could remove the .git sync restriction from being “hard coded” (I’m guessing here)
And instead simply allow us to control it via the repository scope include/exclude filters (with a default: of .git being excluded of course, for backwards compatibility)?
Or does this suggestion / request have more caveats than I had imagined?
I’ve rambled on enough I’d love to know your thoughts!
Thanks!
The main reason the .git folder is excluded is for performance. Continua CI creates a new clean workspace folder for each build. When a build runs, it extracts the source files from the agent repository cache to the new workspace. The .git folder contains the full history of all file changes and can therefore be quite large in both size and number of files. Extracting the files to the workspace can take some time and slow down build initialisation. The reason we have a new clean workspace for each build is so that you can run multiple concurrent builds on each agent without overwriting one build with another.
What Git information does your Fastlane process require? Generally the information that you need from Git (or from other repository types) is available via the Changeset expression object. It seems to be quite unnecessary to have to copy the full file history to the workspace to get a few details about the current changeset. Maybe you can pass the required information to the Fastlane process, rather than using its source control actions? If there is any useful information missing from Changeset object, let us know and we will consider adding it.
Another reason for excluding the .git folder is that we use Mercurial rather than Git as the transport mechanism for synchronising file differences from the server to the agent. This means that we are already copying a .hg folder to the repository cache folder (although not to each build workspace folder). We don’t really want to add a .git folder to this process as well.
We do, however, recognise that Git has now pretty much taken over as the version control of choice. Because of this, we have, for some time now, been considering adding the ability to optionally use Git as the transport mechanism. There are a few hurdles to overcome with this as Git does not have an exact equivalent to the filtering options that the Mercurial export command has. Once we do manage to implement this, it may be possible to link the build workspace to a separate shared .git directory, but there’s a significant amount of work for us to do to get to that point first.
Thank you kindly for the comprehensive response Dave!
Indeed, most of the git operations ran from our Fastlane pipeline are for retrieving some git metadata.
I also thought about the idea of sending this data from Continua CI as parameters for the Fastlane executable.
It does feel like the “more correct / clean” solution to the issue at hand -
But since modifying the Fastlane pipeline to receive parameters and use them instead of running git actions when a certain flag is enabled - is not a priority for our developers at the moment -
I have decided to run the agent’s local repository setup via some windows cmd commands instead (similar to the way we do it for iOS builds via ssh).
For now it seems to be working correctly.
A downside for this, as you mentioned - is having to keep a lock for this configuration so that no more than 1 concurrent builds are occupying the agent’s local repo with a build process, so indeed we can’t run multiple builds on the same agent simultaneously this way (unless I keep multiple copies of the local repo under different paths hehe, I might try to create this setup if multiple simultaneous builds will be something that our devs will ask for).
I’d love to know your thoughts!