The 'Total' number of tests seems to be double what it should be (see screenshot below)
'Save report as artifact' doesn't seem to work, unless there is a trick?
In the build results -> DUnit tab, 'Display By' combo. No tests display unless 'Tests' is selected (Fixture, Namespace, Assembly show nothing). A bit hesitant with this one though because I've only got 1 namespace and 1 fixture running.
Thanks for the report. I’ll take a look at it again tomorrow. Getting DUnit into Continua was a bit of a mission, as the unit test architecture is very .net orientated. DUnit doesn’t report namespaces (which in Delphi are unit names), so I had to make them up by using the exe name. Also the xml runner (which is based on the original DUnit xml runner) has some serious limitations, because the nunit report format reports totals etc as attributes we had to fudge it since we don’t have a dom to work with. I may change it to use MSXML and write the output to the dom.
I actually have a half finished unit test framework for delphi (tentatively called DUnitX and mostly backwards compatible with DUnit) which will provide much better output… just need the time to finish it!
Thanks Vincent - obviously none of the items I mentioned are that important, I just figured I’d report them when I see them. With the XML fixes in 889 it tells us when we break our code, which is all we need it to do
1) I wasn’t able to reproduce that, I think I will need to see your xml file produced. 2) That should have been removed, we changed how artifacts work (see below). 3) You need to click on one of the totals… in the screenshot New Failures is selected and it has zero tests.
Artifact registration is two process thing on the Stage editor. Firstly, you need to get the artifact file back to the workspace on the server using the Stage Workspace Rules, and then also register the file as an artifact using the stage Artifacts rules. See this help topic for more info - http://wiki.finalbuilder.com/display/continua/Stage+Options
Once you have that working, send us your xml file to support[at]finalbuilder.com and I’ll run it through the parser here and see what is happening.
One small issue though, clicking 'Download' against the artifacts resulting in a 0 byte file. Also, viewing artifacts as 'raw' results in an empty browser screen. Admittedly I haven't gone delving into the MIM types on IIS yet.
Regarding the filters of the results not working, I probably should have provided a clearer screenshot. When I am in a category (passed, failed etc) that is showing unit test results when 'Tests', is selected in the combo, changing the combo to anything else results in nothing showing.
Also, when I click on an item in the 'namespace' column, 'Fixture' becomes selected in the combobox. Clicking on an entry in the 'Assembly' column selects 'Namespace' in the combo - in short, clicking any links in the results table doesn't seem to have the expected result.
I've just added a unit from another of our projects, and its totals are correct. The difference seems to be the structure depth. The tests that are showing double counts are an extra layer deep.
I have pushed an updated xml runner to github. This is pretty much a rewrite and it now uses msxml to write the xml. I have tested it with a bunch of different test suites and it seems to be working well. Note I only test the code with Delphi XE2, but it should compile ok with older versions. Let me know how it goes.
Wow, thanks Vincent! The updated DUnit writer is giving us the correct numbers in a our tests now, however I've got one suite where continua doesn't seem to like the XML output:
I've emailed the XML output in question to support in case you wanted to check it out.
I just had a look at the file, and it’s because the Assembly element is missing some attributes, however I cannot reproduce that here. Which version of Delphi and DUnit are you using? I have been testing with XE2 and the version of DUnit that shipped with it.
I have pushed an update to github which inserts default attribute values so hopefully the parser will still work with the output you are seeing. It might be worth running the tests under the debugger and see if the EndSuite method is being called for the Assembly suite.
Thanks for the info - we are using the same. XE2 with the shipping DUnit. I'll take a closer look this morning for differences across our suites and let you know what I find for future reference.
As you eluded to Vincent, EndSuite wasn't being called on the Assembly. You have to give your tests an extra level from the default structure that the DUnit wizard produces in order for everything to line up and work.
For example, the following structure results in missing attributes from Assembly:
Adding an extra 'level' (or namespace I suppose) to the test, results in valid XML output:
From my brief look, it looks like a test count inside DUnit itself not lining up, but I'd have to look deeper for that to be conclusive. As a side note, having more than 3 levels also breaks the XML (missing attributes).
Thanks for looking into it, I’ll have another look at it tomorrow. I should be able to work around this as I use a stack for the suites, I just need to check the stack is empty when TestingEnds is called and if it’s not then process the stack entries.
I just pushed another minor change, I added a ForceDirectories call on the report file folder as I had a test exe failing because the workspace subfolder I specified hadn’t been created yet.