Im using TeamCity with Gallio/XUnit/Specflow and trying to display our unit/acceptance test results. I believe the output is NUnit xml test result format.
Our unit tests auotmatically display using the command runner with Gallio. The acceptance tests are run through a large Powershell script which calls Gallio (Run-Gallio).
Results are output to acceptance-test-results.xml. Is there a way to display the acceptance-test-results.xml in TeamCity (6.5.5)?
To do this in a PowerShell script add the following
Write-Output "##teamcity[importData type='nunit' path='C:\SomeDirectory\YourResults.xml']"
You can use built in TeamCity feature importdata service message:
<!-- Send to TeamCity a service message using MSBuild -->
<Message Text="##teamcity[importData
type='nunit'
path='$(OutputPath)\UnitTestsReport.xml']"
Importance="High" />
For more details see: Build Script Interaction with TeamCity - Importing XML Reports
Thought this might be useful also for other people that came to this SO post with slightly different requirements. It's the public wiki for TC that covers this particular area.
http://confluence.jetbrains.com/display/TCD8/Build+Script+Interaction+with+TeamCity#BuildScriptInteractionwithTeamCity-ImportingXMLReports
Related
I'm currently moving one of our projects to DNX (.NET Core now) and I was forced to update to nunit3. Because of other considerations, we run compile the test project as a console app with its own entry point, basically self-hosting the NUnit runner.
I now need to report the results to TeamCity via the XML Reporter, which doesn't seem to parse Nunit3 TestResults.xml files.
Any advice on how to work around this?
The NUnit 3 console has the option to produce results formatted in the NUnit 2 style.
Use the option:
--result=[filename];format=nunit2
Docs: https://github.com/nunit/nunit/wiki/Console-Command-Line
To add to the answer above:
NUnitLite inherits the --result CLI parameter which seems to do the trick.
Another option, which I went for in the end is using the --teamcity CLI parameter:
dotnetbuild --project:<path to project directory> -- --teamcity
which will integrate with TC's service messages. This will also do real-time updates.
On the TFS 2013 BuildServer, i run Post-Test PowerShellScript to Execute JavaScript Unit Test using Grunt. This produces the .xml file on the build server.
I want to write the results from .xml to the TestResult of TFS Server.
Is there a way to do this ?
Looking for a way to push the Pass and Faile results of JavaScript Test into TestResult(.trx) of TFS 2013.
You can use vstest.console.exe to publish test results. See this blog post on publishing test results using vstest: http://blogs.msdn.com/b/visualstudioalm/archive/2012/12/06/publishing-test-results-through-command-line-test-runner.aspx You should be able to invoke that exe from an InvokeProcess activity.
vstest.console.exe with TfsPublisher switch would allow you to run tests results from a unit test run executed by vstest.console.exe. Since, you are using JavaScript unit test with Grunt, that won't work.
Your best bet is to create an activity that reads information from the xml file and publish results to you team build. Have a look at this project
https://github.com/hamidshahid/TfsBuildResultPublisher
There are bits of code that you an use from this project specially around adding your own test results information.
With TeamCity 8, how do I produce / find a results file for an NUnit run?
We currently also run MsTest which produces a TRX file. We then use a TRX->HTML report tool to pass a report up the management food chain. How do we do the same with NUnit in TeamCity?
Right now I'm thinking I need to execute NUnit as a CommandLine build step, but that seems crazy considering there's an NUnit add-in and the MsTest add-in offers me a "Results file:" option
TeamCity executes MSTest and NUnit differently.
NUnit is not run through the NUnit console executable but instead through TeamCity's own NUnit runner. This allows TeamCity to report NUnit test results on the fly--executing test 3...4...5...of 78--and allows instant notification of failed tests, even if all tests have not yet been executed.
MSTest, on the other hand, goes directly through the MSTest executables and does not have on-the-fly reporting. There is no progress other than "in progress". Test Results, including any failures, are only reported after every tests has been run.
TeamCity requires and parses the MSTest TRX file to do its own reporting, including on any failures, so it is also made available to you. However, the NUnit reporting files are a part of the NUnit console, and not a part of the TeamCity runner, so there is no report file to provide.
If you need the report file, you will need to run the NUnit tests through the NUnit console. There are several ways of doing this, only one of which is using a Command Line step. But be aware, you will lose the on-the-fly reporting, no matter which alternative you use.
Jay's description is correct; this is the TeamCity behaviour that makes this task impossible out of the box.
There is a known workaround though:
http://devnet.jetbrains.com/message/5218450#5218450
Essentially, you invoke the TeamCity NUnit runner manually (e.g. from MSBuild). The runner can then output a result.xml file (one per test assembly). Those result files then have to be merged back into one in order to simulate the behaviour of nunit-console.
Davy Brion has even posted the MSBuild tasks for this:
http://web.archive.org/web/20080808215345/http://davybrion.com/blog/2008/07/using-teamcitys-nunit-support-while-keeping-the-output-around/
http://web.archive.org/web/20080809002009/http://davybrion.com/blog/stuff/
He has since nuked his blog, so waybackmachine to the rescue. In case those links die too, here are the snippets:
NUnitMergeOutput
This task combines the output of multiple NUnit xml reports into one combined xml report.
The combined report will contain the results of each xml report that was fed to it, and it contains the total number of tests, failures, duration and overall success status of the entire test run.
To define the task:
<UsingTask AssemblyFile="$(MSBuildProjectDirectory)\Libs\Brion.MSBuildTasks\Brion.MSBuildTasks.dll"
TaskName="NUnitMergeOutput"/>
And to use it in a target:
<CreateItem Include="TestResults\*.xml" >
<Output TaskParameter="Include" ItemName="NUnitOutputXmlFiles"/>
</CreateItem>
<NUnitMergeOutput NUnitOutputXmlFiles="#(NUnitOutputXmlFiles)"
PathOfMergedXmlFile="TestResults\TestResults.xml" />
BuildTeamCityNUnitArguments
TeamCity doesn’t easily allow you to enable its integrated NUnit testing support while still keeping the NUnit output xml files around after the build. This task prepares an xml arguments file to pass to TeamCity’s NUnitLauncher task which does make it possible to keep the NUnit output xml in a directory you can specify. You can find more info on this problem here and more info on this workaround here.
To define the task:
<UsingTask AssemblyFile="$(MSBuildProjectDirectory)\Libs\Brion.MSBuildTasks\Brion.MSBuildTasks.dll"
TaskName="BuildTeamCityNUnitArguments"/>
And to use it in a target:
<CreateItem Include="**\Bin\Debug\*Tests*.dll" >
<Output TaskParameter="Include" ItemName="TestAssemblies" />
</CreateItem>
<BuildTeamCityNUnitArguments HaltOnError="true" HaltOnFirstTestFailure="true"
HaltOnFailureAtEnd="true" TestAssemblies="#(TestAssemblies)"
NUnitResultsOutputFolder="TestResults"
PathOfNUnitArgumentsXmlFile="nunitarguments.xml" />
<Exec Command="$(teamcity_dotnet_nunitlauncher) ## nunitarguments.xml" />
I remember doing this in an older version of Team City in which the NUnit Runner failed, so I tried running NUnit using the Command Line Runner instead.
I think there was an option in the Command Line Runner settings which let you specify the kind of output file: NUnit result.xml was one of them. Doing this, you could see the test result breakdown in the Web GUI, just like when the NUnit Runner worked.
But I don't see this in the latest Team City (6.5.5). Can it still be done?
PS: Before anyone asks, I can't use Team City's NUnit Runner because NUnit has a bug which will be fixed in the soon to come 2.6.0 version. Said bug does not occurr when running nunit-console
This could be done using importdata TeamCity message, this is really helpful when you are running NUnit tests manually executing nunit-console.exe from an MSBuild script, for instance using NUnit MSBuild Community Task, but keep in mind that TeamCity provides built in NUnit tests runner so you do not need this low level TeamCity scripting. Anyway you can import report in this way:
<Message
Text="##teamcity[importData type='nunit' path='...\TestResults.xml']"
Importance="High"/>
See Importing XML Reports
If you prefer to run code coverage, code inspection, test tools or
duplicate finders directly from build script, not as a build runner,
you can use the importData service messages to import generated xml
reports into TeamCity.
The functionality was still there, found it by accident when looking at another build configuration. Importing an XML is now a "Build Feature", see:
Nunit works quite well with CruiseControl.NET, but there is one thing that irritates me a lot.
If there is a test that causes Nunit to crash, I would only get little information about the crash because the XML report of Nunit doesn't get a chance to be created and be merged into the CruiseControl report.
I need a way to report the progress even when Nunit crashes during the execution.
I have been tried to force each test to output some information to the console to resolve this problem. I have thought about using SetUp method, but I haven't found any good way to get the name of the current running test.
I think a better answer would be to create an NUnit Add-in that implements EventListener interface to capture the TestStarted event to output the progress to the console or a file.
The EventListener interface is documented on NUnit website: http://nunit.org/index.php?p=eventListeners&r=2.5
In addition, we can make the Dashboard report better even when NUnit crashes during its execution. We can use the following procedure to ensure that the DashBoard always shows something about the tests.
Run tests with the EventListener which outputs the progress to a separate file
After running tests, use another program to check the file
If the file does not contain a specific "end line", generate a special XML report based on the file and merge it into the CruiseControl log
If getting the name of the current running test is what you're after you could grab it with the following:
using System.Diagnostics;
...
[Test]
public void SomeTestThatWillCrash()
{
StackFrame sf = new StackFrame();
Console.WriteLine("Now running method: " + sf.GetMethod().Name);
...
}
CruiseControl.net recommends that you use NUnit through your builder (i.e. NAnt/MSBuild). See here: http://confluence.public.thoughtworks.org/display/CCNET/NUnit+Task. As they describe - it will allow you to run these tests locally first - which should give you an exception that you can clear up.
That being said - are your developers running these unit tests prior to checking in code? That could ease this issue. If its an integration issue - I would suggest grabbing the latest code base and running the tests locally to see what is out of sorts.
I don't know if NUnit is able to create the results file even when it crashes. Even if it did - you could run into problems if that file is not well formed due to the crash.
You could use #jpoh's approach but do it in the TestSetup method which would require you do it per-fixture. If really needed, you could write a base class that all your test fixtures inherit from that implement this method.
Another solution is to use MSBuild to run NUnit and use the task in the MSBuildCommunityTasks library. This allows you to continue on error and also get the error code back from NUnit. You won't get what method caused the problem, but might help some. Here is my MSBuild target:
<Target Name="UnitTest"
DependsOnTargets="BuildIt">
<NUnit Assemblies="#(TestAssemblies)"
ToolPath="$(NUnitx86Path)"
WorkingDirectory="%(TestAssemblies.RootDir)%(TestAssemblies.Directory)"
OutputXmlFile="#(TestAssemblies->'%(FullPath).$(NUnitFile)')"
Condition="'#(TestAssemblies)' != ''"
ExcludeCategory="$(ExcludeNUnitCategories)"
ContinueOnError="true">
<Output TaskParameter="ExitCode" ItemName="NUnitExitCodes"/>
</NUnit>
<!-- Copy the test results for the CCNet build before a possible build failure (see next step) -->
<CallTarget Targets="CopyTestResults" Condition="'#(TestAssemblies)' != ''"/>
<Error Text="Test error(s) occured" Code="%(NUnitExitCodes.Identity)" Condition=" '%(NUnitExitCodes.Identity)' != '0' And '#(TestAssemblies)' != ''"/>
</Target>
This probably won't fit your needs as is, but is something to try out and play with.
That said, I would agree with #rifferte that it sounds like you need to debug the problem locally and not rely on CC.NET to handle the reporting.