Missing files in agent workspace

Hi guys,

Continua CI is screwing with me. We regularly encounter build errors due to missing folders and files in the workspace of our build agent. It’s always the same C# project that is missing. As I encountered the problem yesterday I reset the repository and the first build for a specific branch worked perfectly. After the next merge commit to the same branch our repository trigger for this branch triggered and started the next build of the branch. This time the project’s folder was missing again and the build failed.

I installed TortoiseHg on the Continua CI server to take a look at the internal HG repository. According to the log of the HG repository the project’s directory was deleted according to a specific commit in our Git repository. The thing is that not even a single line of code of the project was modified in this commit. The commit was a pull request merge in our GitHub Enterprise repository. These changes were made:

$ git show 2fe49907fc4769edfb8e30d004bf97bca31e8d0f --stat
commit 2fe49907fc4769edfb8e30d004bf97bca31e8d0f
Merge: 205957d 2ad17cd
Author: Kay Zumbusch
Date:   Tue Jun 14 10:36:46 2016 +0200
 
    Merge pull request #172 from mailstore/feature/jira-ms-45-unified-stores-manual-creation
 
    MS-45 : unified stores manual creation
 
 …/AdminTools/PageStorageLocations.Designer.cs    | 63 ++++++++++±------
 src/Client/AdminTools/PageStorageLocations.cs      | 20 ++++±
 …/CreateAttachFileGroupBaseControl.cs            | 13 +±-
 …/CreateAttachFileGroupDialog.cs                 | 26 +++++±-
 …/FirebirdPropertiesControl.cs                   | 21 ++±–
 …/PostgreSQLPropertiesControl.cs                 | 10 ±-
 …/SelectAdvancedFileGroupTypeControl.Designer.cs | 43 +±---------
 …/SelectAdvancedFileGroupTypeControl.cs          | 13 +±-
 …/SelectAdvancedFileGroupTypeControl.resx        |  6 ±
 …/SelectFileGroupTypeControl.Designer.cs         | 35 ++++++±–
 …/SelectFileGroupTypeControl.cs                  | 33 ++++++±–
 …/SelectFileGroupTypeControl.resx                |  6 ±
 …/SqlServerPropertiesControl.cs                  | 10 ±-
 src/Client/CreateAttachFilegroup/UI.de.resx        | 11 +±-
 src/Client/CreateAttachFilegroup/UI.resx           | 11 +±-
 src/Common.Tests/Common.Tests.csproj               |  1 +
 src/Common.Tests/Types/KnownFileGroupTypesTest.cs  | 77 ++++++++++++++++++++++
 src/Common.Tests/Utils/PathUtilsTests.cs           | 36 ++++++++++
 src/Common/Types/FileGroupInfo.cs                  |  9 +++
 src/Common/Types/FileGroupProperties.cs            | 49 ++++++++±----
 src/Common/Types/KnownFileGroupTypes.cs            | 63 ++++++++++++++++++
 src/Common/Types/KnownFileGroupTypesSR.cs          | 23 ++++±-
 src/Common/Types/KnownFileGroupTypesSR.de.resx     |  3 +
 src/Common/Types/KnownFileGroupTypesSR.resx        |  3 +
 src/Common/Types/KnownFileGroupTypesSR.strings     |  2 +
 src/Common/Utils/PathUtils.cs                      | 36 +++++++±-
 src/FileGroups.Firebird/FileGroups.Firebird.csproj |  1 +
 …/FirebirdClassicFileGroupBuilder.cs             |  8 ±-
 src/FileGroups.Firebird/FirebirdFileGroup.cs       | 37 ++++±-----
 …/FirebirdFileGroupBuilder.cs                    |  8 ±-
 …/FirebirdInternalFileGroupBuilder.cs            | 58 ++++++++++++++++
 src/FileGroups.Firebird/SR.cs                      | 12 +±-
 src/FileGroups.Firebird/SR.de.resx                 |  5 ±
 src/FileGroups.Firebird/SR.resx                    |  5 ±
 src/FileGroups.Firebird/SR.strings                 |  4 ±
 src/FileGroups.Firebird/ThisAssembly.cs            |  1 +
 src/FileGroups.Tests/FilegroupTest.cs              |  9 +++
 src/TestCore/FileGroupPreparer.cs                  | 12 ++±
 38 files changed, 569 insertions(+), 214 deletions(-)


And this is the corresponding changeset in ContinuaCI internal HG repository:

Commit message: Automated commit by Continua CI from ‘MailStoreGit’ repository for revision ‘2fe49907fc4769edfb8e30d004bf97bca31e8d0f’. Comment: Merge pull request #172 from mailstore/feature/jira-ms-45-unified-stores-manual-creation
 
C:\ContinuaCI\Rc\da38c763>hg status --change 54b5a104bbab
M src\Client\AdminTools\PageStorageLocations.Designer.cs
M src\Client\AdminTools\PageStorageLocations.cs
M src\Client\CreateAttachFilegroup\CreateAttachFileGroupBaseControl.cs
M src\Client\CreateAttachFilegroup\CreateAttachFileGroupDialog.cs
M src\Client\CreateAttachFilegroup\FirebirdPropertiesControl.cs
M src\Client\CreateAttachFilegroup\PostgreSQLPropertiesControl.cs
M src\Client\CreateAttachFilegroup\SelectAdvancedFileGroupTypeControl.Designer.cs
M src\Client\CreateAttachFilegroup\SelectAdvancedFileGroupTypeControl.cs
M src\Client\CreateAttachFilegroup\SelectAdvancedFileGroupTypeControl.resx
M src\Client\CreateAttachFilegroup\SelectFileGroupTypeControl.Designer.cs
M src\Client\CreateAttachFilegroup\SelectFileGroupTypeControl.cs
M src\Client\CreateAttachFilegroup\SelectFileGroupTypeControl.resx
M src\Client\CreateAttachFilegroup\SqlServerPropertiesControl.cs
M src\Client\CreateAttachFilegroup\UI.de.resx
M src\Client\CreateAttachFilegroup\UI.resx
M src\Common.Tests\Common.Tests.csproj
M src\Common.Tests\Utils\PathUtilsTests.cs
M src\Common\Types\FileGroupInfo.cs
M src\Common\Types\FileGroupProperties.cs
M src\Common\Types\KnownFileGroupTypes.cs
M src\Common\Types\KnownFileGroupTypesSR.cs
M src\Common\Types\KnownFileGroupTypesSR.de.resx
M src\Common\Types\KnownFileGroupTypesSR.resx
M src\Common\Types\KnownFileGroupTypesSR.strings
M src\Common\Utils\PathUtils.cs
M src\FileGroups.Firebird\FileGroups.Firebird.csproj
M src\FileGroups.Firebird\FirebirdClassicFileGroupBuilder.cs
M src\FileGroups.Firebird\FirebirdFileGroup.cs
M src\FileGroups.Firebird\FirebirdFileGroupBuilder.cs
M src\FileGroups.Firebird\SR.cs
M src\FileGroups.Firebird\SR.de.resx
M src\FileGroups.Firebird\SR.resx
M src\FileGroups.Firebird\SR.strings
M src\FileGroups.Firebird\ThisAssembly.cs
M src\FileGroups.Tests\FilegroupTest.cs
M src\TestCore\FileGroupPreparer.cs
A src\Common.Tests\Types\KnownFileGroupTypesTest.cs
A src\FileGroups.Firebird\FirebirdInternalFileGroupBuilder.cs
R src\FileGroups.PostgreSQL\FileGroups.PostgreSQL.csproj
R src\FileGroups.PostgreSQL\PostgreSQLContentReadStream.cs
R src\FileGroups.PostgreSQL\PostgreSQLContentSystem.cs
R src\FileGroups.PostgreSQL\PostgreSQLDatabase.cs
R src\FileGroups.PostgreSQL\PostgreSQLFileGroup.cs
R src\FileGroups.PostgreSQL\PostgreSQLFileGroupBuilder.cs
R src\FileGroups.PostgreSQL\PostgreSQLTransactionHandle.cs
R src\FileGroups.PostgreSQL\SQL\PostgreSQLMailStoreFileGroup.sql
R src\FileGroups.PostgreSQL\SQL\PostgreSQLMailStoreFn_StringListToBigIntTable.sql
R src\FileGroups.PostgreSQL\SQL\PostgreSQLMailStoreIndexTable.sql
R src\FileGroups.PostgreSQL\SQL\PostgreSQLStoredFunction_AddTermToMail.sql
R src\FileGroups.PostgreSQL\SR.cs
R src\FileGroups.PostgreSQL\SR.de.resx
R src\FileGroups.PostgreSQL\SR.resx
R src\FileGroups.PostgreSQL\SR.strings
R src\FileGroups.PostgreSQL\SearchIndex\PostgreSQLPhraseSearch.cs
R src\FileGroups.PostgreSQL\SearchIndex\PostgreSQLPostings.cs
R src\FileGroups.PostgreSQL\SearchIndex\PostgreSQLSearchCriteriasSearch.cs
R src\FileGroups.PostgreSQL\SearchIndex\PostgreSQLSearchCriteriasSqlGenerator.cs
R src\FileGroups.PostgreSQL\SearchIndex\PostgreSQLSearchIndex.cs
R src\FileGroups.PostgreSQL\SearchIndex\PostgreSQLSearchIndexManager.cs
R src\FileGroups.PostgreSQL\SearchIndex\PostgreSQLTermSearch.cs
R src\FileGroups.PostgreSQL\ThisAssembly.cs
R vendor\Rebex\Zip\Shared\Adler32.cs
R vendor\Rebex\Zip\Shared\AsynchronousOperation.cs
R vendor\Rebex\Zip\Shared\BatchTransfer.cs
R vendor\Rebex\Zip\Shared\BatchTransferInternals.cs
R vendor\Rebex\Zip\Shared\Common.cs
R vendor\Rebex\Zip\Shared\Crc32.cs
R vendor\Rebex\Zip\Shared\DeflateCompressor.cs
R vendor\Rebex\Zip\Shared\DeflateDecompressor.cs
R vendor\Rebex\Zip\Shared\DeflateMisc.cs
R vendor\Rebex\Zip\Shared\EnumHelper.cs
R vendor\Rebex\Zip\Shared\EnumeratorHelper.cs
R vendor\Rebex\Zip\Shared\FileUtil.cs
R vendor\Rebex\Zip\Shared\HuffmanTree.cs
R vendor\Rebex\Zip\Shared\LZ77HashTable.cs
R vendor\Rebex\Zip\Shared\LZ77Window.cs
R vendor\Rebex\Zip\Shared\Messages.cs
R vendor\Rebex\Zip\Shared\RebexAsyncResult.cs


As you can see the HG commit removes a lot of files that were not touched in the git commit.

Our environment:

Windows Server 2012 R2 with all updates applied
ContinuaCI 1.8.0.151
Git 2.6.1.windows.1

At the moment we could work around the problem by resetting the repository after every merge but that takes about 45 minutes each time and is not a viable solution.

If you need additional debug information I can try to extract the necessary data from the debug log files which are permanently activated on our ContinuaCI server.

Kind regards

Kay Zumbusch

Hi Kay,

This is unusual. We’ll have a look at this first thing tomorrow. If you can send the relevant debug log, covering the period of time when the changeset was committed, to support@finalbuilder.com this will help us to fill the cause of the issue quickly.



Hi Dave,

the log file is on its way.

Regards

Kay Zumbusch

Hi,

thanks to the guys at VSoft our build environment is back on track. We had some branches with different casing for the same directory which caused problems building the internal repository of Conitunua CI. We archived all historic branches in our Git repository using tags (see http://stackoverflow.com/questions/1307114/how-can-i-archive-git-branches) which left only branches with the same casing for all files and directories. Repository reset time dropped from 45 minutes to 10 minutes and were are able to build any branch still available in the repository.

Thanks a lot guys.

Kind regards

Kay Zumbusch