Hi guys,
we just updated our test suite from NUnit 2.6.4 to NUnit 3.4.1. The test results are as expected but Continua CI is not able to transfer the results from the XML file to the database and the NUnit task fails. We get the following error message from the NUnit task:
The target Nunit test file was not found or the test file could not load one of its dependencies. Ensure nunit.framework.dll is in the same directory as your test file.
We already exchanged the single instance of the nunit.framework.dll in our code respository and replaced NUnit 2.6.4 with NUnit 3.4.1 on our build server. We did not place the nunit.framework.dll in a any additional path and did not for NUnit 2.6.4 either. The XML results file is properly created and registered as an artifact in Continua CI but the tab for the unit tests remains empty.
I noticed that the XML structure has changed. In NUnit 2.6.4 the root node is “test-results” in NUnit 3.4.1 it is “test-run”.
Is there a known issue using NUnit 3?
Build server environment:
- Windows Server 2012 R2 with all latest updates
- NUnit 3.4.1 (MSI setup)
- Continua CI Agent 1.8.0.151
Continua CI Server
- Windows Server 2012 R2 with all latest updates
- Continua CI Server 1.8.0.151
Best regards
Kay Zumbusch
We just updated to 1.8.0.207 but the problem is still there.
Hi Kay,
I have tested the action against NUnit 3.4.1, and found the tests are being loaded into the database correctly. There are a few things to double check when issues of this nature occur.
- The installation version of NUnit 3 on the agent machine.
- The location of the tests dll’s in the workspace, are they being written to the location used by the NUnit action.
- Are all the other binaries associated with the test binary located in the test directory passed to NUnit.
The error that is being reported is that NUnit is reporting files missing that is requires to perform the test. Typically these are the files supplied on the command line are not present in the location given.
If your getting an xml file produced please send it to our support email address (support@finalbuilder.com). I would be surprised if this is the case as the error being reported is that the unit tests have not be run, and therefore there should be no results file.
Hi Jason,
we indeed have an XML file which is properly formatted and has all expected test results. The test run is successful but two tests fail at the moment which is expected. All test assemblies are built into a single “bin” directory ($Source$/bin) including the correct NUnit dll which is included in our code respository. To run the unit tests we supply the Visual Studio Solution file to the NUnit task and the XML file is written to the agent’s output directory ($Workspace$/Output/testresults.xml).
Do we have to copy the NUnit dll to the same directory where the XML is stored? That was not necessary with NUnit 2.6.4.
The installation of NUnit 3.4.1 seems to be correct. It’s installed into C:\Program Files (x86)\NUnit.org. Here is the part of the Continua CI log for the NUnit task.
[code]11:04:14 AMNUnit [Source\src\MailStore.sln]11:04:14 AMTesting file C:\ContinuaCI\Ws\12668\Source\src\MailStore.sln.11:04:14 AMParameters11:04:14 AMWorking Directory: C:\ContinuaCI\Ws\1266811:04:14 AMExecutable: C:\Program Files (x86)\NUnit.org\nunit-console\nunit3-console.exe11:04:14 AMArguments: C:\ContinuaCI\Ws\12668\Source\src\MailStore.sln --framework=net-4.5 --result=C:\ContinuaCI\Ws\12668\Output\testresults.xml --noheader --nocolor --config=Debug11:04:14 AMEnvironment Variables11:04:14 AMNo environment variables specified11:04:15 AMRuntime Environment11:04:15 AM OS Version: Microsoft Windows NT 6.3.9600.011:04:15 AM CLR Version: 4.0.30319.4200011:04:15 AM11:04:15 AMTest Files11:04:15 AM C:\ContinuaCI\Ws\12668\Source\src\MailStore.sln11:21:31 AM11:21:31 AM11:21:31 AMTests Not Run11:21:31 AM11:21:31 AM1) Ignored : MailStore.Connectors.Tests.ImapMailboxTests.Test01_MsgInfo11:21:31 AMOneTimeSetUp: Test IMAP Account defunct11:21:31 AM11:21:31 AM2) Ignored : MailStore.Connectors.Tests.Complex.HttpMailboxTests.TestConnectAndGetFolders(Exchange 2010, journal)11:21:31 AMUnstable Exchange 2010 environment11:21:31 AM11:21:31 AM3) Ignored : MailStore.Connectors.Tests.Complex.HttpMailboxTests.TestConnectAndGetFolders(Exchange 2010, single mailbox)11:21:31 AMUnstable Exchange 2010 environment11:21:31 AM11:21:31 AMErrors and Failures11:21:31 AM11:21:31 AM1) Failed : MailStore.Storage.Tests.AzureADApiClientTests.SetOrUpdateSkipToken (Add skiptoken)11:21:31 AM Expected string length 305 but was 307. Strings differ at index 77.11:21:31 AM Expected: "…st.com/groups?api-version=1.5&$skiptoken=X’445370740200010…"11:21:31 AM But was: "…st.com/groups?api-version=1.5&%24skiptoken=X’4453707402000…"11:21:31 AM --------------------------------------------^11:21:31 AMat NUnit.Framework.Internal.Commands.TestMethodCommand.Execute(TestExecutionContext context)11:21:31 AMat NUnit.Framework.Internal.Commands.TestActionCommand.Execute(TestExecutionContext context)11:21:31 AM11:21:31 AM2) Failed : MailStore.Storage.Tests.AzureADApiClientTests.SetOrUpdateSkipToken (Replace skiptoken)11:21:31 AM Expected string length 305 but was 307. Strings differ at index 77.11:21:31 AM Expected: "…st.com/groups?api-version=1.5&$skiptoken=X’445370740200010…"11:21:31 AM But was: "…st.com/groups?api-version=1.5&%24skiptoken=X’4453707402000…"11:21:31 AM --------------------------------------------^11:21:31 AMat NUnit.Framework.Internal.Commands.TestMethodCommand.Execute(TestExecutionContext context)11:21:31 AMat NUnit.Framework.Internal.Commands.TestActionCommand.Execute(TestExecutionContext context)11:21:31 AM11:21:31 AM3) Invalid : C:\ContinuaCI\Ws\12668\Source\bin\MailStore.Client.Tests.dll11:21:31 AMNo suitable tests found in ‘C:\ContinuaCI\Ws\12668\Source\bin\MailStore.Client.Tests.dll’.11:21:31 AMEither assembly contains no tests or proper test driver has not been found.11:21:31 AM11:21:31 AM4) Invalid : C:\ContinuaCI\Ws\12668\Source\bin\MailStore.TestCore.dll11:21:31 AMHas no TestFixtures11:21:31 AM11:21:31 AMTest Run Summary11:21:31 AM Overall result: Failed11:21:31 AM Test Count: 1991, Passed: 1986, Failed: 2, Inconclusive: 0, Skipped: 311:21:31 AM Failed Tests - Failures: 2, Errors: 0, Invalid: 011:21:31 AM Skipped Tests - Ignored: 3, Explicit: 0, Other: 011:21:31 AM Start time: 2016-07-27 09:04:35Z11:21:31 AM End time: 2016-07-27 09:21:30Z11:21:31 AM Duration: 1015.499 seconds11:21:31 AM11:21:34 AMResults (nunit3) saved as C:\ContinuaCI\Ws\12668\Output\testresults.xml11:21:34 AMAction ‘NUnit [Source\src\MailStore.sln]’ has failed due to an error.11:21:34 AMThe target Nunit test file was not found or the test file could not load one of its dependencies. Ensure nunit.framework.dll is in the same directory as your test file.[/code]
Kind regards
Hi Kay,
Thank you for replying with all the information you have. This helps greatly.
[Edited]
After talking with David it appears the following line my be the issue.
"No suitable tests found in C:\ContinuaCI\Ws\12668\Source\bin\MailStore.Client.Tests.dll. Either assembly contains no tests or proper test driver has not been found."We suggest double checking that all binaries for the testing code are located in the directory holding all testing binaries.
Hi Kay,
To add to what Jason has said, it appears that this failure is returned for two reasons.
1. If any assembly does not contain a reference to either nunit.framework.dll with a major version of 3 or 2, or nunitlite.dll with a major version of 1.
2. If any assembly does not contain any unit tests. See https://github.com/nunit/nunit/issues/1586.
With this type of failure the NUnit command line is returning an exit code of -2 which is the same exit code returned when a file is not found. We are interpreting this as a complete failure and not continuing to parse the test file. I’ll change the message to make this clearer and also look into whether we can change the code to continue to read in the xml file in this case (while still reporting the failure).
Meanwhile you should be able to get this working by specifying each assembly in the Files field of the NUnit action rather than the solution file.
Hi Jason,
hi Dave,
if there is any problem with assemblies that don’t have any unit tests we should have had this problem with NUnit 2 as well. The MailStore.Client.Tests.dll assembly does not have any tests but references the NUnit framework dll. The same applies to MailStore.TestCore.dll. There are no tests defined but some shared checks are implemented in it and therefor NUnit is referenced. That did not change between the use of NUnit 2.6.4 and 3.4.1. Maybe NUnit 3 is more restrictive in this case. I’ll remove the reference to NUnit and will try again.
Listing all test assemblies is no option for us. We build multiple branches on the CI environment during development with different sets of test assemblies as we are currently working extensively on unit testing and code coverage.
Kind regards
Kay Zumbusch
Hi Kay,
This is due to a change in NUnit 3 - see the GitHub issue link below. Removing the reference will not help. I have added a “Continue with warning if any of the test assemblies are missing or invalid” option in the new version 1.8.0.227. This will ignore the error returned from NUnit 3. Also note that you can use wildcards when specifying file paths.
Hi guys,
thanks for the hint to the open NUnit case. We tried a build with the NUnit references removed from Client.Tests.dll and TestCore.dll which worked fine.
Kind regards
Kay Zumbusch