Testing code is something we all do. Whether it be manual usability testing, unit testing, or integration testing, knowing how much of the application is covered by the tests is important. Without knowing what parts of the application are covered, there is no way to know if key features are tested.
When performing unit testing there is an analytical way to determine what parts of the source code are covered by the tests. This is typically call source code coverage. Working with Delphi, one of the tools that performs this task is called DelphiCodeCoverage (open source). It can be located on GitHub (more recent fork) and SourceForge. Under the hood this tool simply marks each line of source code as "hit" when the application calls it at least once. From there it can generate a detailed report giving the overall coverage statistics for the project, as well as the individual lines not hit in the testing.
Code Coverage
What I will go through below is how to setup code coverage on a unit test project, and hook that into a continuous integration process using Continua CI. I will assume that if you require knowledge on how to setup a project on Continua CI you will refer to the Create your First Project wiki page.
The code that I would like to get a code coverage report on is the Core.Card.pas unit. The unit tests for this class are located in the tests folder and have a corresponding name of Core.CardTests.pas. You may have noticed that some of the code paths are not fully covered in my unit tests. This is intentional, and something that we will come to a little later on.
An extra consideration to have with a unit testing project is to make sure it can run under continuous integration. This means that it should run to completion and produce an output file that is able to be imported into the build summary. With this in mind I have created a "CI" configuration on my unit testing project. This conditionally compiles the unit testing project so that it does not wait for user input (something my debug configuration does) and generates an XML output file.
All this code and other related scripts are located in the VSoftTechnologies/DelphiCodeCoverageExample GitHub repository. Feel free to clone it to get a better sense of code coverage and the project structure I am using.
DelphiCodeCoverage
To generate a code coverage report I decided to use DelphiCodeCoverage. The tool has a number of command line options, all of which are spelt out on the GitHub page. Some of the options are a little overwhelming in the effort they require. An example of this is passing a file that contains all the source directories to search for classes to include in the code coverage report. Thankfully there is a wizard supplied with DelphiCodeCoverage that will help generate a batch file containing the correct parameters to pass to DelphiCodeCoverage.
In my project I have placed DelphiCodeCoverage into a sub-folder call "CodeCoverage" and included it into source control. There are two reasons I am doing this;
1. The code coverage executable is now available everywhere the source is pulled to.
2. It simplifies the script I will need for the CI Server.
If your uncomfortable with placing binaries into your source control, this can be altered without affecting the produced report.
Running the code coverage wizard your presented with a page to enter the executable, map file, source, and output directory locations. Below are the settings I have used:
The last option for the wizard allows for making all paths relative. This is exactly what we require to have our generated batch file run on any system, however at the time of writing it does not work correctly. This meant that I had to manually change all paths to a version that was relative to the folder in which DelphiCodeCoverage was located.
dcov_execute.bat before
"I:\Examples\DUnitX_CodeCoverage\CodeCoverage\CodeCoverage.exe"
-e "I:\Examples\DUnitX_CodeCoverage\Win32\Debug\DUnitX_And_CodeCoverage.exe"
-m "I:\Examples\DUnitX_CodeCoverage\Win32\Debug\DUnitX_And_CodeCoverage.map"
-uf dcov_units.lst -spf dcov_paths.lst
-od "I:\Examples\DUnitX_CodeCoverage\CodeCoverage\Output\" -lt -html
dcov_execute.bat after
"CodeCoverage.exe"
-e "..\Win32\CI\DUnitX_And_CodeCoverage.exe"
-m "..\Win32\CI\DUnitX_And_CodeCoverage.map"
-ife -uf dcov_units.lst -spf dcov_paths.lst -od ".\Output\" -lt -html
Note: Line breaks are only included above for readability. There are none in the resulting batch files.
Another alteration that I have made to the batch file is to include the "-ife" option. The option will include file extensions. This means that it will stop a unit like "Common.Encoding" being 'converted' to "Common". As in my project I have unit called "Core.Cards.pas" this option is required to have it included in generated code coverage report.
Next the relative path change should be applied to the two generated list files dcov_paths.lst and dcov_units.lst. The paths file should be the only one that has path in need of altering to be relative. Both however need to be checked to make sure they contain everything to be covered in the report. If there are source folders missing they need to be added to the dcov_paths.lst file. If there are unit names missing they need to be added to the dcov_units.lst file.
Now that the batch file and list files have been corrected running the dcov_executable.bat should produce summary out similar to that below. Note that the unit test project needs to be compiled as DelphiCodeCoverage runs the unit test executable.
**********************************************************************
* DUnitX - (c) 2013 Vincent Parrett *
* vincent@finalbuilder.com *
* *
* License - https://www.apache.org/licenses/LICENSE-2.0 *
**********************************************************************
Fixture : Core
-------------------------------------------------
Fixture : Core.CardTests
-------------------------------------------------
Fixture : Core.CardTests.TCardTest
-------------------------------------------------
Test : Core.CardTests.TCardTest.A_Card_FacingUp_Once_Flipped_Is_Facing_Down
-------------------------------------------------
Executing Test : A_Card_FacingUp_Once_Flipped_Is_Facing_Down
Success.
Running Fixture Teardown Method : Destroy
Done testing.
Tests Found : 1
Tests Ignored : 0
Tests Passed : 1
Tests Leaked : 0
Tests Failed : 0
Tests Errored : 0
Summary:
+-----------+-----------+-----------+
| Lines | Covered | Covered % |
+-----------+-----------+-----------+
| 15 | 11 | 73 % |
+-----------+-----------+-----------+
Continuous Integration
With the code coverage batch file we are now able to run code coverage on any system, include on a continuous integration system. Our goal with the continuous integration is to have the unit tests built and run each time a set of code is checked into source control. This will allow us to then track if any unit tests fail, and changes in the code coverage.
To achieve this I have create a Continua CI configuration that builds my unit test project, runs the unit tests under code coverage, and then import the unit test results into the build summary.
The FinalBuilder action calls the FinalBuilder project responsible for compiling the DUnitX unit test project. It uses the CI configuration so that the unit tests executable will run to completion, and will produce an NUnit XML results file in the same directory as the executable. It is important to build the unit tests each time as the source code for our project would have changed each time we run the continuous integration. Note that you do not have to use FinalBuilder, you can also use MSBuild to build your DUnitX Project - see Integrating DUnitX Unit Testing with Continua CI.
The execute program action simply runs the code coverage batch file generated above. This batch file will run the unit test project we compiled and log code coverage information as it does. The result will be a summary written out to our build log while also html files written to the report folder we specified in the batch file. It is these html files which we will attach to the continuous build report a little later.
Lastly we want to import the actual unit test results. These are written out by DUnitX as a NUnit compatible XML file which we can import with the "Import NUnit Tests" action. The results from the XML file will be attached to the build report presented by Continua CI.
As all builds for Continua CI are run on agents, and all build reports come from the server, we need to transfer the code coverage report back to the server. This is done through workspace rules on the build stage. In this example DelphiCodeCoverage writes out all html report files to a relative directory of ".\Output\". This means if we run the DelphiCodeCoverage batch file from "Source\Output\CodeCoverage\" the report should appear in "Source\Output\CodeCoverage\Output" (Note that $Source.DelphiCodeCoverage.Path$ was mapped to the \Source\ folder on the agent). Workspace rules use the greater than symbol to signal the files should be copied from the server to the agent, and the less than symbol to copy from the agent to the server. This therefore leaves use the workspace rule of "“/Output/CodeCoverage/ < \Source\CodeCoverage\Output\*.html" to get all code coverage report files back to the server.
Now that the html reports are on the server, we need to show them against the Continua CI build. To achieve this we use the reports section of our Continua CI configuration. The reports section allows us to specify a file to attach to the build as a report to be displayed or offered as a download. In this case we want to display the report summary html file. All reports work from the server point of view, and each build has it own workspace on the server. To this end the report we want to be display would have been copied to "$Workspace$\Output\CodeCoverage\CodeCoverage_summary.html".
The Code Coverage Report
The end report appearing in the report section of the Continua CI build summary.
As shown in the report the example project has some code that is not covered during unit testing. This reduces the overall coverage to 73%. If I had more than one unit each would have their own code coverage summary. In addition I could click on each file and get a line by line report to see what section of the unit is not covered.
Final Notes
It is worth mentioning that code coverage is only one arrow in a software testing quiver. In my example I purposely chose to include code that was not covered. This showed the power of code coverage in picking up where unit testing should potentially be directed to next. I also included code where the unit tests cover the code, however not fully. The code testing Core.Card.Flip only tests one path through the code, not all the possible paths. Currently the test sees if the code works when going from face up to face down, not from face down to face up. Although in this example it might be benign, it shows that other tools are needed to help cover this gap.