Inconclusive Tests

Continua CI Discussion
1

It seems that inconclusive tests cause the test action to fail even if they are shelved?

Test output:
Test Run Inconclusive.
Passed 261
Inconclusive 2

2

seems to happen only if the option “fail action if any tests error” is checked. so you mean an inconclusive test is an error?
what if a “real” error happens or what is meant with “error”?

3

btw: “new failures” also shows our 8 inconclusives … slightly confusing for me

4

Hi Christian,

Is it MSTest that you are using? We’re having trouble reproducing an inconclusive test with MSTest. The command Assert.Inconclusive() produces an AssertInconclusiveException with a Failed outcome in our tests. Can you provide us with some example code which produces an inconclusive outcome?


5

Can you also let us know which version of MSTest you are using?

6

we are still using mstest boundled with vs2010. meanwhile we unshelved all tests.
if i run a build now, all tests are green and the 10 inconsluive are also covered. the test stage is green, too. so “shelving” has nothing to do with this.

if i activate “Fail action if any tests error” the test stage fails. so question for me is: what means “error”?

7

Hi Christian,

Error tests should only include tests which raise an exception. Inconclusive tests should not be classed as an error. They are interpreted as a failure in the code. It’s possible that MSTest is returning an error code along with an inconclusive state, which would cause this result.

We are trying to reproduce an inconclusive result so that we work out what is happening here, but cannot find an working example. With Nunit we would use  Assert.Inconclusive(), but this throws an exception with MSTest. If you could share an example inconclusive test, then we will then attempt to reproduce and see what we can do to fix the code to interpret an inconclusive test correctly.





8

we simply use Assert.Inconclusive(“message”), nothing special. however, these tests always throw a “AssertInconclusiveException” (as described in the documentation). the .TRX files contain a separate entry “inconclusive”.

So i am not sure how you measure the test results internally.

9

Hi Christian,

When we run an MSTest.VisualStudio.2010 action on the following test:

[TestMethod]
public void InconclusiveTest()
{
    Assert.Inconclusive(“this is inconclusive”);
}

we get the following result in the trx file:

<UnitTestResult executionId=“e2c06fa8-41ca-4ba6-ba65-27e757658369” testId=“6817d349-ecf5-36c4-231a-020d18dc26b4” testName=“InconclusiveTest” computerName=“SPARKY” duration=“00:00:00.0142090” startTime=“2016-03-24T11:38:17.1627687+11:00” endTime=“2016-03-24T11:38:17.1783835+11:00” testType=“13cdc9d9-ddb5-4fa4-a97d-d965ccfc6d4b” outcome=“Failed” testListId=“8c84fa94-04c1-424b-9868-57a2d4851a1d” relativeResultsDirectory=“e2c06fa8-41ca-4ba6-ba65-27e757658369”>
      <Output>
        <ErrorInfo>
          <Message>Test method BuildTest.Tests.InconclusiveTest threw exception:
Microsoft.VisualStudio.TestTools.UnitTesting.AssertInconclusiveException: Assert.Inconclusive failed. this is inconclusiveMessage>
          <StackTrace>   at BuildTest.Tests.InconclusiveTest() in d:\Continua\MasterAgent\Ws\1522\Source\BuildTest\Calc\Tests.cs:line 74
StackTrace>
        ErrorInfo>
      Output>
    UnitTestResult>

This is interpreted by ContinuaCI as a failure and shown under the Failed indicator of the Unit Tests view. No tests are shown under the Inconclusive indicator

The MSTest Action indicates Success in the build log however that Stage fails due the the default Stage Gate condition of "$Stage.Metrics.UnitTests.Failed$ Equals 0"

Are you sure that it is not your stage gate which is failing rather than the action? Are you seeing the tests marked as Inconclusive in ContinuaCI at all or just as Failed? Can you send a copy of your trx file to support@finalbuilder.com? 



10

its not the stage gate. in my build log the action “MSTest” is marked “Failed” and not “Success”.

the stage gate is:
$Stage.Metrics.UnitTests.Failed$ <= 0
$Stage.Metrics.UnitTests.Error$ <= 0

i agree that the single unit test result is the same as you posted. however, at the top, we can see that there is none failed and 2 inconclusive:

[code] <ResultSummary outcome=“Inconclusive”> <Counters total=“263” executed=“263” passed=“261” error=“0” failed=“0” timeout=“0” aborted=“0” inconclusive=“2” passedButRunAborted=“0” notRunnable=“0” notExecuted=“0” disconnected=“0” warning=“0” completed=“0” inProgress=“0” pending=“0” /> </ResultSummary>[/code]

continua is showing everything correct (2 inconclusive, none failed). just the action fails if i mark “any tests error”. seems a little bit inconsistent by mstest.

for sure, i could switch off “if any tests error” and everything would be fine. just want to make sure that the testing stage isn’t skipping any “errors” by accident.

11

see here: http://picpaste.de/pics/Inconclusive-oY5Srp0C.1458805625.png

12

Hi Christian,

Thanks for this additional information. We’ve now worked out what is happening. MSTest is returning an exit code of 1 when there are inconclusive tests.

The ContinuaCI MSTest action returns an error outcome in the following cases:
1. There is a error count in the result summary 
2. There is a non-zero exit code and the failed count is zero

MSTest returns an exit code of 1 for both errors and failure - we now see that this is also for an inconclusive tests. So we need to change case 2 to exclude the inconclusive tests (and possibly others). We’ll fix that for the next version.

The reason we were not seeing this in our environment, is because MSTest was returning a failure count in the ResultSummary for an inconclusive test e.g. 

<ResultSummary outcome=“Failed”>
    <Counters total=“1” executed=“1” passed=“1” error=“0” failed=“1” timeout=“0” aborted=“0” inconclusive=“0” passedButRunAborted=“0” notRunnable=“0” notExecuted=“0” disconnected=“0” warning=“0” completed=“0” inProgress=“0” pending=“0” />
ResultSummary>

We are still interested in understanding why you are getting different results to us in your trx file. Can you let us know the file version of “C:\Program Files (x86)\Microsoft Visual Studio 10.0\Common7\IDE\MSTest.exe” and also any non-default settings in your MSTest action? 

13

The version ist 10.0.30319.1. We don’t use any special. We have a test setttings file, however, running the tests without this settings file, the result is the same (and correct from my point of view).

This is how Continua is running it (copied from build log):
[code]Working Directory: C:\BuildAgent\Ws\157863 Executable: c:\Program Files (x86)\Microsoft Visual Studio 10.0\Common7\IDE\mstest.exeArguments: /testcontainer:C:\BuildAgent\Ws\157863\Source\INFOsys\Bin\Tests\INFOsys.Framework.Tests.dll /testcontainer:C:\BuildAgent\Ws\157863\Source\INFOsys\Bin\Tests\INFOsys.Objects.Test.dll /resultsfile:C:\BuildAgent\Ws\157863\Output\Tests\INFOsys.trx[/code]

14

Hi Christian,

We are now able to reproduce inconclusive test outcomes after making changes to the references in our test project.  We have made some changes so they are no longer by treated as errors. We hope to get a new version out in the next couple of days with this fix and others.