Action to run Delphi DUnitX tests never returns

Continua CI Discussion
1

Following on from my questions last week related to MSBuild and Delphi, I got the main project working OK.

However I have some associated DUnit tests , and I have used your very useful code in the DUnit-XML GitHub repo to output the results in the xUnitX format so that I can load the XML result file in via a subsequent action and continue/stop the build based on the results of the unit tests.

So my unit tests compile fine, and the EXE runs fine when I run it interactively from the File Explorer, but when run from Continua using an Execute Program action, it creates a valid xml output file but never returns control to the stage. (As the stage has not exited I can also grab a copy of the EXE before the cleanup and run this and it terminates normally.)

I have tried enabling/disabling the options on the action - Check Program Exit Code, this makes no difference.

Any thoughts as to what I should to next apart from changing the timeout on the action ?

2

Rerunning with Check Exit code shows it exiting with code of 255.

I’ll dig around some more …

3

Make sure your unit test exe is a console application. Also DUnit has an exit behaviour setting, I forget the details exactly but one of them is to pause on failures, that’s likely the problem.

4

BTW, I recommend DUnitX rather than DUnit… but then I’m biased :wink:

1 Like
5

Thx for the fast reply Vincent.

I’ve got it working by removing the calls in unit VSoft.DUnit.XMLTestRunner.pas to writeln (in the TXMLTestListener.TestingEnds method).

I’m running Continua under a local service account on WinServer 2016 and I think that the service account cannot write to the console. They were only needed for debugging by the look of it.

I vaguely remember hitting a similar issue in FinalBuilder a few years back.

Now my next task is to get Eureka Log working after the DCC64 has done its work in the next stage.

6

P.S. the ExitBehavior option is part of DUnitX. DUnit has something similar for the TxtRunner (not being used in my application)

7

Yeah, I think it was added to keep some former DUnit users happy, I never use it.

You are correct that you cannot use writeln in a windows service.

In DUnitX, we redirect Writeln for any unit that includes the testframework (ie the test units!).

FWIW, I wrote that xml reported for dunit a long time ago, haven’t touched in 5+ years! There is a global variable in the unit - PrintReportToConsole which you can change to stop the calls to writeln.

8

I might have been celebrating too early. Despite no code changes, executing the DUnit tests seems very flaky. If all is Ok it will execute in under a minute. Otherwise it creates the output XML file and then goes into the weeds and Continua just sits there waiting for the executable to return.

Is there anyway to debug what is happening under the hood ?

9

My guess is the unit test app is putting up some sort of message or waiting for input. That’s why I said make sure it’s a console app.

In your configuration cleanup options, change it so the agent workspace is not cleaned up on failure, then you can go to that folder and run the exe with the same parameters/starting folder and see what happens.

10

Hmmm. Sometimes it runs in a few seconds other times it outputs the XML file but never returns. I can’t see any pattern and I have not made any changes to the underlying repos for 2 days.

I have changed the cleanup, it runs interactively 100% of the time OK but that is me running it as a logged in user.

I’ve just used the RUNAS command to run as the service account that Continua runs under (both server and agent on this machine) and it works fine. The console window that appears has no entries and after about 30 seconds it closes. The XML file is recreated.

11

I’ve done a work around which is to add a timeout on the run unit tests action so that there is enough time for the XML file to be created but it does not hold things up if the program does not return. That will keep my evaluation going while I work out how Continua can fit into our workflow. :slight_smile:

12

In our experience, there are a couple of reasons for applications to hang when run under Continua CI

  1. User interface - ie message boxes, prompt for inputs etc.

  2. Antivirus - if you haven’t already done so, add an exclusion for your agent workspace folder (this also speeds up builds considerably

For DUnit - make sure you are not using the GUI logger. In our DUnitX unit tests, we use ifdef’s to decide which loggers to include, for example under CI we only ever include the xml logger, there’s no point include the console logger since we’ll be importing the xml results anyway.

13

We have a unit that we include in all of the Unit tests that automagically configures the output exe depending on the target (debug/release) and whether we want DUnitX formatted output.

Regardless the strange thing is that it is an intermittent problem. We have file access mocked in classes that read from INI files for instance. I’ll take a look at the anti virus settings though.

FWIW, here is the code for the unit we include, makes configuring unit tests much, much simpler :slight_smile:

unit uRunDUnit;

// Important: copy the following lines to the top of the *.DPR file
{$IFDEF RELEASE}
{$APPTYPE CONSOLE}
{$ENDIF}


// DEFINE THE following Symbol to get NUnit XML output
//XMLOUTPUT

interface

uses
  SysUtils,
  Forms,
  TestFramework,
{$IFDEF RELEASE}
  {$IFDEF XMLOUTPUT}
    VSoft.DUnit.XMLTestRunner,
    VSoft.MSXML6;
  {$ELSE}
    FinalBuilder.XMLTestRunner;
  {$ENDIF XMLOUTPUT}
{$ELSE}
  {$IFNDEF TESTINSIGHT}
  GUITestRunner;
  {$ELSE}
  TestInsight.DUnit; // use the TestInsight runner if the project has been setup to run this way
  {$ENDIF TESTINSIGHT}
{$ENDIF RELEASE}

{$IFDEF RELEASE}
var
  OutputFile: string;
  ConfigFile: string;
{$ENDIF}

procedure RunUnitTests;

implementation

procedure RunUnitTests;
begin
{$IFDEF RELEASE}
  OutputFile := ChangeFileExt(Application.ExeName, '.xml');

  if ConfigFile <> '' then
  begin
    RegisteredTests.LoadConfiguration(ConfigFile, False, True);
  end;
  if ParamCount > 0 then
    OutputFile := ParamStr(1);
  {$IfDef XMLOUTPUT}
    PrintReportToConsole := False;
    TXMLTestListener.RunRegisteredTests(OutputFile);
  {$else}
    FinalBuilder.XMLTestRunner.RunRegisteredTests(OutputFile);
  {$endif XMLOUTPUT}
{$ELSE}
  Application.Initialize;
  {$IFNDEF TESTINSIGHT}
    TGUITestRunner.RunRegisteredTests;
  {$ELSE}
    TestInsight.DUnit.RunRegisteredTests;
  {$ENDIF TESTINSIGHT}
{$ENDIF RELEASE}
end;

end.
14

I would remove this, that unit is long ago deprecated. I recall it was used for FinalBuilder Server, were you using that?

I would also add exception handling around the running of the tests so that no exception dialogs get shown.

15

Thanks for all your help. I will refactor the test project (it has ~300 tests in 10 units) by just bringing in 1 unit at a time, needle in haystack work but it must be something in our code :frowning:
Will post an update when we track it down :slight_smile: