I’ve noticed that one of our unit test seems to “fail but pass” strangely.
After investigating: It looks as if the “Total” count of UnitTestResult - is not taking nested results into account, screenshot:
11 of the 40 are nested inside other UnitTestResult sections
As these aren’t counted for the “Total” - the Math doesn’t work:
29 total is not equal to 36 passed +4 failed
I’m using the following expression in order to evaluate whether or not I should fail the unit test job:
The conditional ‘$Utils.GetNumber($Build.Metrics.UnitTests.Passed$)$’ GreaterThanOrEqual ‘$Utils.GetNumber($Build.Metrics.UnitTests.CaseCount$)$’ has been evaluated to true.
This expands to:
[ If [‘36’ GreaterThanOrEqual ‘29’]]
===
===
Also (a different but possibly related issue): strangely enough:
Does not equal zero -
Both stages finish staged as “success”
The 2nd stage is staged as “success” because of the Metrics.UnitTests expression that was shown above was evaluated to “True” but the stage gate condition after this stage - fails the build (as I would expect)
The thing is, although the build is shown in the UI as failed:
We get the total count from the <ResultSummary> element in the trx file, rather that counting the unit tests. So it seems that there maybe something different with how MSTest it writing the totals in your case. Which version of MSTest are you running? If you send us a copy of the trx file (either by direct message to me her to to our support email) we’ll investigate what is happening with the counts tomorrow.
Regarding the second issue, this outcome would be expected as the stage gate expression fails the build (as there are failures) but not the stage which is already in a successful state. The stage gate runs after the stage result is complete.
Thank you Dave!
I’ve sent you the .trx file.
We’re using dotnet 6.0.202 to run the test in this case, with a command line that looks like this:
Execute Program [ cmd.exe /c dotnet test C:\…\Service.Test.csproj /p:IsTestProject=true --logger “trx;LogFileName=C:\CCIAgentShare\Ws\151043\Source\DevOps\Deployment\UnitTesting\Service.Test.26052022_145303.trx” ]
About the 2nd issue: only now I see that the event handler is operating in stage scope rather than in a build scope; So I understand why the passed/failed discrepancy occurred.
Could you tell me why the event handler works with a stage scope and not with the scope of the full build?
(Just asking out of interest)
No reason, that’s just how we it’s set up so that a stage is selected rather than a build. I guess we could add an option post at the end of the build - I’ll look into it.
What type of unit tests cause the nested UnitTestResult sections to be created? Are they parameterised tests using the [DataTestMethod] attribute?
We’ve been looking at this today and there some oddities about the content of your trx file.
The counters in the ResultSummary appear to be counting the outer UnitTestResult as well as the UnitTestResult nodes under InnerResults. This means that it counts 5 tests for the test you posted in your last comment. We think that it should be 4 and that the total for the whole file should be 37 not 40.
Also when we run dotnet test on a .Net 6 project with parameterised tests defined in a similar way to yours, all test results are listed at the same level - there are no InnerResults.
We’re currently planning to ignore the ResultSummary counters and count the results as we iterate through the file. We hope to have a patch ready by the beginning of next week.
