Hello everybody,
we are currently in the process of evaluating FinalBuilder as an alternative to the Build integrated in TFS2010. The integration with the "FinalBuilderBuild.xaml" worked and the project is correctly executed by FinalBuilder.
Setup: TFS2010 on Windows Server 2008, FinalBuilder 7.0.0.1561 (trial)
The following problems occured in the process of getting a Continuous Integration-Build (Build, Execute Tests) outsourced to FinalBuilder:
1) The Build log generated by MSBuild is logged by FinalBuilder. What's currently missing is to publish the log to the TFS - a feature available for MSTest. The switch used by TFS-Build is "/dl:WorkflowCentralLogger"
2) We are using a vsmdi-Metadata-File and Test-List for MSTest. While the File and list is found, MSTest is not able to find the Tests and logs "Test [x] cannot be found." This is due to the "storage="-tag in the vsmdi, which is set as the Binaries are produced on the local machine where the File is edited via Visual Studio. Because the Build-Output is copied to the "Binaries"-Folder, the file in the storage-tag cannot be found and therefore the tests are not executed.
The build process in TFS uses the "/searchpathroot"-switch to target a directory where the assemblies can be found. This is something missing in the current MSTest-Action.
Furthermore, instead of setting a direct Result-File via "/resultsfile", there is also the option of "/resultsfileroot" that gives the ability to set a test-directory (in the case of TFS-Build "TestResults"). This is something also missing in the Action.
3) In the TFS Build-Definition, there is an option "Label Sources". In our experience, setting this to "False" saved some time executing the build. With the FinalBuilder xaml-Template, this setting provokes "Error: Cannot find the current label ''." Has this setting to be "True", or is this an error in the FinalBuilder-Template?
These are the things we found so far. We really liked FinalBuilder on first sight and would be delighted if these options can be realized. Are these proper feature-requests or is there a way to enable this behaviour in the current Version?
Quick response to point 3: This error occurs when “Label Sources” is False and “Associate Changesets and Work Items” is True. I missed that option when I created the Build Definition for FinalBuilder.
Hi Chris,
I’m glad you like your experience with FinalBuilder so far.
On point 1: are you talking about the MSBuild log or the MSTest results? We publish the entire FinalBuilder log to TFS, which includes the MSBuild log (assuming you’ve run the MSBuild action). Is there a reason you’d want to publish the MSBuild log separately?
If you’re talking about the MSTest results, on the last tab of the action you have the option of where to publish results.
It’s likely that I’ve mis-understood what you’re trying to achieve - please feel free to correct me 
On point 2: I don’t know much about vsmdi, so forgive me if I get this wrong. As far as I’m aware, MSTest.exe doesn’t have a /searchpathroot flag, at least not according to the doco: http://msdn.microsoft.com/en-us/library/ms182489(v=VS.100).aspx (the same goes for /resultsfileroot). They may be options that MS have exposed to TFS via an API, but FinalBuilder can only use what’s available via the CLI.
That said, there should be ways to work around this. I’ll do some testing and get back to you.
On point 3: as you’ve discovered, you need to have Label Sources set to true if Associate Changesets is set to true. I believe this is true in the ‘standard’ templates that are shipped with TFS. It’s not obvious in the build definition that this is the case, but with the limited control we get of the build definition I’m not sure how we could improve it.
Cheers,
Ben
Hi again Chris,
I have TFS and FinalBuilder running MSTest with a vsdmi file, so it should be possible. Long story short: you need to build to the default location as specified in the csproj. Now the details…
I’m assuming that you’ve used the example FinalBuilder project, and you so you’re using the Build VS.Net Solution action with its output directory set to something like %SourceRoot%\Binaries. As you correctly identified, the vsmdi contains the relative path to the test assemblies, so if you build the assemblies to a different location they can’t be found.
Removing the output directory setting means that the assemblies will be built to the location specified in the csproj. This is the relative path that the vsmdi uses, so everything should work.
That’s how to get MSTest working - the problem is that you’ll still have your assemblies to the “Drop Share”. There’s two ways to do this.
If you only have a single project, it might be easiest to just change the Move Files action to get the assemblies from [project]\bin\release instead of the binaries folder.
If you have more than one project though, it would make more sense to change your projects to build to a directory other than [project]\bin\release. For example …\binaries. Then you can use the Move Files action to move from that directory to the %DropShare%.
Does that make sense?
If not, or if it doesn’t work for you, please let me know.
Cheers,
Ben
Hello Ben,
thanks for the quick response! Lets get to the issues:
1) As of now, the log produced by MSBuild is not available via the Build Summary in Visual Studio. The only output is “$/Solution.sln - 0 error(s), 0 warning(s), View Log File”. The TFS-Build is able to publish the current build-status to TFS, so the build summary shows the current warnings/errors while the build is still running.
2) The MSTest.exe switches mentioned are definitly available, but are not documented by Microsoft. With a TFS-Build, I was able to extract the exact command-line that is used for MSTest, that’s why I noticed the switch “/searchpathroot” that is set to the MSBuild output-directory. I also tried manually running the same command in a console and it worked. So it’s just a flag that is not openly documented by Microsoft.
This is the command-line that was executed by the TFS-build:
MSTest.exe /nologo /usestderr /searchpathroot:“C:*\Binaries” /resultsfileroot:“C:*\TestResults” /testmetadata:“C:*\Solution.vsmdi” /testlist:“TestListName” /publish:“CollectionURL” /publishbuild:“BuildURL” /teamproject:“TeamProject” /platform:“Platform” /flavor:“Flavor”
3) Just a remark as I experienced it with the standard xaml-Templates: When “Label Sources” is deactivated, “Associate Changesets and Work Items” is automatically (and silently) disabled and is therefore not throwing any errors.
I hope you’ll be able to reproduce the MSTest-command as I think this is a feature definitly needed for the MSTest-Action.
Best regards,
Christian
Hi Chris,
We tend to shy away from adding undocumented options to FinalBuilder actions because it can become very hard to maintain backwards compatibility. I'll talk to the boss though and see what he thinks.
Re updating the build log while the build is running - this is something that the TFS API doesn't expose. The FinalBuilder workflow activity actually sends log messages on the fly, but TFS seems to batch them up until the activity has completed.
I can see what you mean about the (post-build) summary not listing the build errors though. Because (from the TFS side of things) we don't know what the FinalBuilder project is doing as such we'd need to parse the log and check for any MSBuild for VS.NET output then populate the summary. That's certainly possible, but the amount of work means that I won't be able to get it out in a point release. I've added it to our feature list for the next version.
Cheers,
Ben
Hi Ben,
thanks for the clarification. Considering that there might be flags that somebody wants to pass to MSTest or other actions that internally use a command - how about an easy option to add “Additional Options”? Similarly to the compiler options page on C/C+±projects.
Christian
Hi Christian,
It’s a little hidden away, but a lot of the actions (MS Test included) allow you to specify extra command line parameters. If you select the action, then go to the Properties pane, there’s a section called Extra Command Line Parameters. For many of the compilation-type actions we put this in the main action dialog because it’s used so often.
Hope that helps,
Ben
Hi Ben,
‘hidden away’ is somewhat correct, it took me a few minutes to find the Properties pane. I was a little confused with the actual Action Properties (F12) and the general Properties Pane (F4).
I was able to incorporate the “searchpathroot”-flag in the ‘Extra Params at Start’ and the execution worked - thats probably a complete satisfaction of our needs.
The ‘resultsfileroot’-flag on the other hand was not applicable. The problem is that the Action is generating a ‘resultsfile’-flag which cannot be set in conjunction with the ‘resultsfileroot’-flag. Therefore we’re going with a manual definition of the results-file and are just missing the standard name-scheme used by MSTest.
Thanks a lot for the support along the way to make our CI-Build work. We are currently underway to set up our Installer-Build with FinalBuilder and have already reached the decision to buy Licenses for our team. Keep up the good work!
Best regards,
Christian
Hi Christian,
I’m glad that you’re up and running, and happy to have you on board as a customer
Please let us know if there’s anything else we can help with.
Cheers,
Ben