When one of my colleagues left for a new opportunity, I inherited the continuous build setup he built for our project. This has meant spending the past few weeks scrambling to get up to speed on CruiseControl.NET, MSTest and Subversion (among other things). Because we don't use TFS, creating a build server required us to install Visual Studio 2008 in order to run unit tests as part of the build, along with a number of other third-party tasks to make MSBuild work more like NAnt. So the first time a build failed because of tests that had passed locally, I wasn't looking forward to figuring out precisely which of these pieces triggered the problem. After re-implementing unit tests a couple of different ways and still getting the same results (tests passing locally and failing on the build server), we eventually discovered that the problem was a bug in Visual Studio 2008 SP1. Once we installed the hotfix, our unit tests passed on the build server without us having to change them. This hasn't been the last issue we've had with our "TFS-lite" build server. Build timeouts have proven to be the latest hassle. Instead of the tests passing locally and failing on the build server, they actually passed in both places. But for whatever reason, the test task didn't really complete and build timed out. Increasing the build timeout didn't address the issue either. But thanks to the Microsoft Build Sidekick editor, we narrowed the problem down to the MSTest task in our build file. The task is the creation of Nati Dobkin, and it made writing the test build target easier (at least until we couldn't get it to work consistently). So far, I haven't found (or written) an alternative task, but I did find a blog post that pointed the way to our current solution. The initial solution:
<!-- MSTest won't work if the tests weren't built in the Debug configuration –>
<Target Name="Test:MSTest" Condition=" '$(Configuration)' == 'Debug'"> <MakeDir Directories="$(TestResultsDir)" /> <MSBuild.ExtensionPack.FileSystem.Folder TaskAction="RemoveContent" Path="$(TestResultsDir)" /> <Exec Command=""$(VS90COMNTOOLS)..\IDE\mstest.exe" /testcontainer:$(TestDir)\<test assembly directory>\bin\$(Configuration)\<test assembly>.dll /testcontainer:$(TestDir)\<test assembly directory>\bin\$(Configuration)\<test assembly>.dll /testcontainer:$(TestDir)\<test assembly directory>\bin\$(Configuration)\<test assembly>.dll /runconfig:localtestrun.testrunconfig" /> </Target>
TestDir and TestResultsDir are defined in a property group at the beginning of the MSBuild file. VS90COMNTOOLS is an environment variable created during the install of Visual Studio 2008. Configuration comes from the solution file. Actual test assembly directories and names have been replaced with <test assembly> and <test assembly directory>. The only drawback to this solution so far is that we'll have to update our MSBuild file if we add a new test assembly.
A little more research revealed a way to eliminate the main shortcoming of our first solution. Using a combination of MSBuild transforms, batching, and well-known metadata, the addition of just one task eliminates the need to update the MSBuild file each time a new test assembly is added. We took advantage of the consistent naming pattern of our test DLLs to create a list of them:
<CreateItem Include="$(TestDir)\**\bin\$(Configuration)\*.Tests.dll" AdditionalMetadata="TestContainerPrefix=/testcontainer:">
<Output TaskParameter="Include" ItemName="TestAssemblies" />
</CreateItem>
The item created by the task above is a list of the test assembly names separated by semicolons. The MSTest command line requires a space between each assembly passed to it, and a “/testcontainer:”prefix (defined in the AdditionalMetadata attribute). The following command makes use of this additional metadata and replaces the semicolons with spaces.
<Exec Command=""$(VS90COMNTOOLS)..\IDE\mstest.exe" @(TestAssemblies->'%(TestContainerPrefix)%(FullPath)',' ') /runconfig:localtestrun.testrunconfig /resultsfile:$(TestResultsDir)\TestResults.trx"/>
The %(FullPath) item is a well-known metadata item that substitutes the full path to the file for each item in @(TestAssemblies). The end result of these changes is a testing target that has required no further modifications as our solution grows. This process has proven to be an interesting way to preview at least a small part of the functionality that would be part of a TFS-based continuous integration system.