NUnit results file in TeamCity - nunit

With TeamCity 8, how do I produce / find a results file for an NUnit run?
We currently also run MsTest which produces a TRX file. We then use a TRX->HTML report tool to pass a report up the management food chain. How do we do the same with NUnit in TeamCity?
Right now I'm thinking I need to execute NUnit as a CommandLine build step, but that seems crazy considering there's an NUnit add-in and the MsTest add-in offers me a "Results file:" option

TeamCity executes MSTest and NUnit differently.
NUnit is not run through the NUnit console executable but instead through TeamCity's own NUnit runner. This allows TeamCity to report NUnit test results on the fly--executing test 3...4...5...of 78--and allows instant notification of failed tests, even if all tests have not yet been executed.
MSTest, on the other hand, goes directly through the MSTest executables and does not have on-the-fly reporting. There is no progress other than "in progress". Test Results, including any failures, are only reported after every tests has been run.
TeamCity requires and parses the MSTest TRX file to do its own reporting, including on any failures, so it is also made available to you. However, the NUnit reporting files are a part of the NUnit console, and not a part of the TeamCity runner, so there is no report file to provide.
If you need the report file, you will need to run the NUnit tests through the NUnit console. There are several ways of doing this, only one of which is using a Command Line step. But be aware, you will lose the on-the-fly reporting, no matter which alternative you use.

Jay's description is correct; this is the TeamCity behaviour that makes this task impossible out of the box.
There is a known workaround though:
http://devnet.jetbrains.com/message/5218450#5218450
Essentially, you invoke the TeamCity NUnit runner manually (e.g. from MSBuild). The runner can then output a result.xml file (one per test assembly). Those result files then have to be merged back into one in order to simulate the behaviour of nunit-console.
Davy Brion has even posted the MSBuild tasks for this:
http://web.archive.org/web/20080808215345/http://davybrion.com/blog/2008/07/using-teamcitys-nunit-support-while-keeping-the-output-around/
http://web.archive.org/web/20080809002009/http://davybrion.com/blog/stuff/
He has since nuked his blog, so waybackmachine to the rescue. In case those links die too, here are the snippets:
NUnitMergeOutput
This task combines the output of multiple NUnit xml reports into one combined xml report.
The combined report will contain the results of each xml report that was fed to it, and it contains the total number of tests, failures, duration and overall success status of the entire test run.
To define the task:
<UsingTask AssemblyFile="$(MSBuildProjectDirectory)\Libs\Brion.MSBuildTasks\Brion.MSBuildTasks.dll"
TaskName="NUnitMergeOutput"/>
And to use it in a target:
<CreateItem Include="TestResults\*.xml" >
<Output TaskParameter="Include" ItemName="NUnitOutputXmlFiles"/>
</CreateItem>
<NUnitMergeOutput NUnitOutputXmlFiles="#(NUnitOutputXmlFiles)"
PathOfMergedXmlFile="TestResults\TestResults.xml" />
BuildTeamCityNUnitArguments
TeamCity doesn’t easily allow you to enable its integrated NUnit testing support while still keeping the NUnit output xml files around after the build. This task prepares an xml arguments file to pass to TeamCity’s NUnitLauncher task which does make it possible to keep the NUnit output xml in a directory you can specify. You can find more info on this problem here and more info on this workaround here.
To define the task:
<UsingTask AssemblyFile="$(MSBuildProjectDirectory)\Libs\Brion.MSBuildTasks\Brion.MSBuildTasks.dll"
TaskName="BuildTeamCityNUnitArguments"/>
And to use it in a target:
<CreateItem Include="**\Bin\Debug\*Tests*.dll" >
<Output TaskParameter="Include" ItemName="TestAssemblies" />
</CreateItem>
<BuildTeamCityNUnitArguments HaltOnError="true" HaltOnFirstTestFailure="true"
HaltOnFailureAtEnd="true" TestAssemblies="#(TestAssemblies)"
NUnitResultsOutputFolder="TestResults"
PathOfNUnitArgumentsXmlFile="nunitarguments.xml" />
<Exec Command="$(teamcity_dotnet_nunitlauncher) ## nunitarguments.xml" />

Related

Store results of unit test run into variables

I have a TeamCity build configuration that builds a C# project, runs some unit tests, and then does some extra things. My question is: Can I get information about my unit test run stored into build configuration variables (i.e. how many tests were run, how many were successful, how many failed, how many were skipped) so that I can then check these variables in a PowerShell script in later build steps and perform different actions depending on how many tests have passed?
AFAIK the best way is to ask these information directly to teamcity server using its REST API (pay attention, maybe the build locator could be a little be tricly to be found, if the build is still running).
By other hand, you can parse your NUnit test result file (or files if you run more than one NUnit test runner step in your build) inside your build agent machine.

Team City - Add Gallio test result xml to build display

Im using TeamCity with Gallio/XUnit/Specflow and trying to display our unit/acceptance test results. I believe the output is NUnit xml test result format.
Our unit tests auotmatically display using the command runner with Gallio. The acceptance tests are run through a large Powershell script which calls Gallio (Run-Gallio).
Results are output to acceptance-test-results.xml. Is there a way to display the acceptance-test-results.xml in TeamCity (6.5.5)?
To do this in a PowerShell script add the following
Write-Output "##teamcity[importData type='nunit' path='C:\SomeDirectory\YourResults.xml']"
You can use built in TeamCity feature importdata service message:
<!-- Send to TeamCity a service message using MSBuild -->
<Message Text="##teamcity[importData
type='nunit'
path='$(OutputPath)\UnitTestsReport.xml']"
Importance="High" />
For more details see: Build Script Interaction with TeamCity - Importing XML Reports
Thought this might be useful also for other people that came to this SO post with slightly different requirements. It's the public wiki for TC that covers this particular area.
http://confluence.jetbrains.com/display/TCD8/Build+Script+Interaction+with+TeamCity#BuildScriptInteractionwithTeamCity-ImportingXMLReports

Process NUnit result.xml and show it in Team City Web GUI

I remember doing this in an older version of Team City in which the NUnit Runner failed, so I tried running NUnit using the Command Line Runner instead.
I think there was an option in the Command Line Runner settings which let you specify the kind of output file: NUnit result.xml was one of them. Doing this, you could see the test result breakdown in the Web GUI, just like when the NUnit Runner worked.
But I don't see this in the latest Team City (6.5.5). Can it still be done?
PS: Before anyone asks, I can't use Team City's NUnit Runner because NUnit has a bug which will be fixed in the soon to come 2.6.0 version. Said bug does not occurr when running nunit-console
This could be done using importdata TeamCity message, this is really helpful when you are running NUnit tests manually executing nunit-console.exe from an MSBuild script, for instance using NUnit MSBuild Community Task, but keep in mind that TeamCity provides built in NUnit tests runner so you do not need this low level TeamCity scripting. Anyway you can import report in this way:
<Message
Text="##teamcity[importData type='nunit' path='...\TestResults.xml']"
Importance="High"/>
See Importing XML Reports
If you prefer to run code coverage, code inspection, test tools or
duplicate finders directly from build script, not as a build runner,
you can use the importData service messages to import generated xml
reports into TeamCity.
The functionality was still there, found it by accident when looking at another build configuration. Importing an XML is now a "Build Feature", see:

Getting TeamCity to run NUnit tests, with both NCover and TypeMock integrated?

Basically I'd like to use the NUnit plugin for TeamCity (the program, not necessarily the specific build step using it) to run my unit tests, with NCover for code coverage, and since my unit tests uses TypeMock 6, I need that too working.
So far I've tried:
Just basically pointing the TeamCity NUnit build-step to my dll's, but that fails with the following error message:
Typemock Isolator needs to be linked with Coverage Tool to run, to enable do one of the following:
link the Coverage tool through the Typemock Isolator Configuration
run tests via TMockRunner.exe -link
use TypeMockStart tasks for MSBuild or NAnt with Link
Trying to figure out the right command line, I tried this:
C:...\Isolator\6.0\TMockRunner.exe "C:\TeamCity...\JetBrains.BuildServer.NUnitLauncher.exe" v4.0 MSIL NUnit-2.5.9 MyAssembly.dll
This fails with the exact same error.
Setting the environment variables found in the mocking_on.bat file part of TypeMock, this doesn't change the outcome.
Note that the above examples doesn't contain any reference to NCover (yet), that's because I've been hacking around on the command line for a couple of hours with the above examples and still haven't gotten basic unit-tests running. NCover is extra options to the nunit-launcher of TeamCity so I hope this is as simple as just enabling that when I get that far.
Since TypeMock requires you to use their own runner program, TMockRunner, there's no way to just use the GUI options in TeamCity to get everything set up.
Instead, what I ended up doing was to first build a custom msbuild file with this contents:
<?xml version="1.0" encoding="utf-8"?>
<Project ToolsVersion="4.0" DefaultTargets="Build" xmlns="http://schemas.microsoft.com/developer/msbuild/2003">
<PropertyGroup>
<TypeMockLocation>C:\Program Files (x86)\TypeMock\Isolator\6.0</TypeMockLocation>
<NUnit>"C:\TeamCity\buildAgent\plugins\dotnetPlugin\bin\JetBrains.BuildServer.NUnitLauncher.exe"</NUnit>
<NCover>C:\Program Files (x86)\NCover\NCover.Console.exe</NCover>
</PropertyGroup>
<Import Project="$(TypeMockLocation)\TypeMock.MSBuild.Tasks"/>
<Target Name="TestWithTypeMock">
<TypeMockStart Link="NCover3.0" ProfilerLaunchedFirst="true" Target="2.0"/>
<Exec ContinueOnError="true" Command="$(NUnit) v2.0 x86 NUnit-2.5.9 SqlDatabases.Core.Tests\bin\Debug\SqlDatabases.Core.Tests.dll SqlDatabases.SqlServer.Tests\bin\Debug\SqlDatabases.SqlServer.Tests.dll /ncover:%22$(NCover)%22 /ncover-arg://ias /ncover-arg:SqlDatabases.Core /ncover-arg://ias /ncover-arg:SqlDatabases.SqlServer /ncover-arg://et /ncover-arg:.*Exception /ncover-arg://x /ncover-arg:c:\temp\coverage.xml"/>
<TypeMockStop/>
</Target>
</Project>
This file I save to a directory on my TeamCity server. Since I didn't want the test script to be part of my repository, I didn't add it to source control (I can build and right-click and run tests from within Visual Studio, if I get something not so tied to my build server I might change that decision later). Also, I only have 1 build-agent for my TeamCity server so this works for me for the time being.
In addition to the above file, I added the following batch-file:
#echo off
setlocal
set CURDIR=%CD%
copy c:\dev\sqldatabases\tests.msbuild .\
msbuild tests.msbuild /target:TestWithTypeMock
rd /s /q c:\dev\sqldatabases\codecoverage
md c:\dev\sqldatabases\codecoverage
"c:\program files\ncover\ncover.reporting.exe" c:\temp\coverage.xml //or FullCoverageReport:Html:c:\dev\sqldatabases\codecoverage
cd \dev\sqldatabases\codecoverage
del %CURDIR%\coverage.zip
7z a -r %CURDIR%\coverage.zip
And then I added the following two build-steps to my TeamCity build configuration:
Visual Studio Solution: Build the debug configuration.
Execute C:\Dev\SqlDatabases\Tests.bat (the above batch file)
I made sure that coverage.zip was listed under artifacts in my build configuration, and now I got code coverage, typemock, and unit tests.
Only thing I haven't figured out how to get so far is that if I click on the build result of a build configuration that uses the normal TeamCity GUI for setting up everything (minus TypeMock), I get a code coverage short summary listed, this is not present in the above setup, but the full report is, and the tab in the TeamCity build results.
Hopefully this can save someone else some trouble.
Lasse,
So long as you're using the TeamCity GUI, running with NCover coverage should just be a simple matter of selecting to run with it.I've never tried throwing Typemock into the mix, so I look forward to hearing how that goes.

How can I run NUnit tests in parallel?

I've got a large acceptance test (~10 seconds per test) test suite written using NUnit. I would like to make use of the fact that my machines are all multiple core boxes. Ideally, I'd be able to have one test running per core, independently of other tests.
There is PNUnit, but it's designed for testing for threading synchronization issues and things like that, and I didn't see an obvious way to accomplish this.
Is there a switch/tool/option I can use to run the tests in parallel?
If you want to run NUnit tests in parallel, there are at least 2 options:
NCrunch offers it out of the box (without changing anything, but is a commercial product)
NUnit 3 offers a Parallelizable attribute, which can be used to denote which tests can be run in parallel
NUnit version 3 will support running tests in parallel:
Adding the attribute to a class: [Parallelizable(ParallelScope.Self)] will run your tests in parallel.
• ParallelScope.None indicates that the test may not be run in parallel
with other tests.
• ParallelScope.Self indicates that the test
itself may be run in parallel with other tests.
• ParallelScope.Children indicates that the descendants of the test may
be run in parallel with respect to one another.
• ParallelScope.Fixtures indicates that fixtures may be run in parallel
with one another.
NUnit Framework-Parallel-Test-Execution
If your project contains multiple test DLLs you can run them in parallel using this MSBuild script. Obviously you'll need to tweak the paths to suit your project layout.
To run with 8 cores run with: c:\proj> msbuild /m:8 RunTests.xml
RunTests.xml
<?xml version="1.0" encoding="utf-8"?>
<Project DefaultTargets="RunTestsInParallel" xmlns="http://schemas.microsoft.com/developer/msbuild/2003">
<Import Project="$(MSBuildExtensionsPath)\MSBuildCommunityTasks\MSBuild.Community.Tasks.Targets"/>
<PropertyGroup>
<Configuration Condition=" '$(Configuration)' == '' ">Release</Configuration>
<Nunit Condition=" '$(Nunit)' == '' ">$(MSBuildProjectDirectory)\..\tools\nunit-console-x86.exe</Nunit>
</PropertyGroup>
<!-- see http://mikefourie.wordpress.com/2010/12/04/running-targets-in-parallel-in-msbuild/ -->
<Target Name="RunTestsInParallel">
<ItemGroup>
<TestDlls Include="..\bin\Tests\$(Configuration)\*.Tests.dll" />
</ItemGroup>
<ItemGroup>
<TempProjects Include="$(MSBuildProjectFile)" >
<Properties>TestDllFile=%(TestDlls.FullPath)</Properties>
</TempProjects>
</ItemGroup>
<MSBuild Projects="#(TempProjects)" BuildInParallel="true" Targets="RunOneTestDll" />
</Target>
<Target Name="RunOneTestDll">
<Message Text="$(TestDllFile)" />
<Exec Command="$(Nunit) /exclude=Integration $(TestDllFile) /labels /xml:$(TestDllFile).results.xml"
WorkingDirectory="$(MSBuildProjectDirectory)\..\bin\Tests\$(Configuration)" />
</Target>
</Project>
Update
If I were answering this question now I would highly recommend NCrunch and its command line test running tool for maximum test run performance. There's nothing like it and it'll revolutionise your code-test-debug cycle at the same time.
As an alternative to adding the Parallelizable attribute to every test class:
Add this into the test project AssemblyInfo.cs class for nunit3 or greater:
// Make all tests in the test assembly run in parallel
[assembly: Parallelizable(ParallelScope.Fixtures)]
In this article it is mentioned that in order to speed up tests the poster runs multiple instances of NUnit with command parameters specifying which tests each instance should run.
FTA:
I ran into an odd problem.
We use nunit-console to run test on
our continuous integration server.
Recently we were moving from Nunit
2.4.8 to 2.5.5 and from .Net 3.5 to 4.0. To speed up test execution we run multiple instances of Nunit in
parallel with different command line
arguments
We have two copies of our test assemblies and the nunit binaries in
folder A and B.
In folder A we execute
nunit-console-x86.exe Model.dll
Test.dll /exclude:MyCategory
/xml=TestResults.xml
/framework=net-4.0 /noshadow
In folder B we execute
nunit-console-x86.exe Model.dll
Test.dll /include:MyCategory
/xml=TestResults.xml
/framework=net-4.0 /noshadow
If we execute the commands in sequence
both run successfully. But if we
execute them in parallel only one
succeeds. As far as I can tell it's
the one that first loads the test
fixtures. The other fails with the
message "Unable to locate fixture".
Is this problem already known? I could
not find anything related in the bug
list on launchpad. BTW Our server runs
Windows Server 2008 64-bit. I could
also reproduce the problem on Windows
7 64-bit.
Assuming this bug is fixed or you are not running the newer version(s) of the software mentioned you should be able to replicate their technique.
Update
TeamCity looks like a tool you can use to automatically run NUnit tests. They have an NUnit launcher discussed here that could be used to launch multiple NUnit instances. Here is a blog post discussing the mergind of multiple NUnit XML results into a single result file.
So theoretically you could have TeamCity automatically launch multiple NUnit tests based on however you want to split up the workload and then merge the results into a single file for post test processing.
Is that automated enough for your needs?
Just because PNUnit can do synchronization inside test code doesn't mean that you actually have to use that aspect. As far as I can see there's nothing to prevent you from just spawning a set and ignoring the rest till you need it.
BTW I don't have the time to read all of their source but was curious to check out the Barrier class and it's a very simple lock counter. It just waits till N threads enter and then sends the pulse for all of them to continue running at the same time. That's all there is to it - if you don't touch it, it won't bite you.
Might be a bit counter intuitive for a normal threaded development (locks are normally used to serialize access - 1 by 1) but it is quite a spirited diversion :-)
You can now use NCrunch to parallelize your unit tests and you can even configure how many cores should be used by NCrunch and how many should be used by Visual Studio.
plus you get continuous testing as a bonus :)
It would be a bit of a hack, but you could split the unit tests into a number of categories. Then, start up a new instance of NUnit for each category.
Edit: It looks like they have added a /process option to the console app. The command-line help states this is the "Process model for tests: Single, Separate, Multiple". The test runner also appears to have this feature.
Edit 2: Unfortunately, although it does create separate processes for each assembly, the process isolation option (/process from the command line) runs the agents one at a time.
Since the project hasn't been mentioned here, I would like to bring up NUnit.Multicore. I haven't tried the project myself, but it seems to have an interesting approach to the parallel test problem with NUnit.
You can try my small tool TBox or console parallel Runner or even plugin to do distributed calulations, which also can run unit tests on the set of PCs SkyNet
TBox is created to simplify work with big solutions, which contains many projects. It supports many plugins and one of them provide ability to run NUnit tests in parallel. This plugin does not require any changes to your existing tests.
Also it support:
Cloning of the folder with unit test (if your tests changes local data),
Synchronizations of the tests (for example if your tests on
testfixtureteardown kills all dev servers or chromerunner for qunit )
x86 mode and Admin privileges to run tests
Batch run - you can run tests for many assemblies in parallel
Even for single thread run, works faster than standart nunit runner, if you have much small tests.
Also this tool supports command line tests runner (for parallel run) and you can use it with continuous integration.
I have successfully used NUnit 3.0.0 beta-4 to run tests in parallel
Runs on build server
Runs Selenium tests
Has Visual Studio support
no Resharper support yet
Thanks for peers answer.
Gotchas:
Parallelizable attribute is not inherited, so it has to be specified on the test class.
You can use following PowerShell command (for NUnit3, for NUnit2 change runner name):
PS> nunit3-console (ls -r *\bin\Debug\*.Tests.dll | % FullName | sort-object -Unique)
Presented command runs all test assemblies in single nunit instance, which allows to leverage engine built-in parallel test run.
Remarks
Remember to tweak directory search pattern. Given example runs only assemblies ending with .Tests.dll and inside \bin\Debug directories.
Be aware of Unique filtering - you may not want to have it.
To achieve level of parallelism ensure to do these two:
1)Nunit Explorer - Settings - Run tests in parallel
2)LevelOfParallelism
This is an assembly-level attribute used to specify the level of parallelism, that is, the maximum number of worker threads executing tests in the assembly.
In Assemblyinfo.cs, set
[assembly:LevelOfParallelism(N)] => here N is number