I want to run NUnit on TeamCity and generate report with dotCover. But for some reasons I cannot get the coverage report for all my project assemblies.
TeamCity config as below
Teamcity: 6.5.3
NUnit: 2.5.10
.NET Runtime: Platform: x86
.NET Runtime: Version: v4.0
dotCover: bundled with TC not customized
I have checked that all *.dll *.xml *.pdb files are there in the directory as expected as below (reference DLLs are not listed)
MY.PROJECT.A.dll
MY.PROJECT.A.pdb
MY.PROJECT.A.xml
MY.PROJECT.B.dll
MY.PROJECT.B.pdb
MY.PROJECT.B.xml
MY.PROJECT.C.dll
MY.PROJECT.C.pdb
MY.PROJECT.C.xml
MY.PROJECT.Test.dll
MY.PROJECT.Test.pdb
MY.PROJECT.Test.xml
MY.PROJECT.Test.dll is executed with NUnit and this assembly is excluded in the coverage report by using *Test* filter. But only MY.PROJECT.A is in the dotCover coverage report whereas MY.PROJECT.B and MY.PROJECT.C are not included.
I checked the log but not error is found.
Any thoughts is much appreciated.
Finally, I figured out what is going on, and hope this answer is useful for those who are still struggling with this or similar problem.
Basically, dotCover only includes those assemblies are actually used (more precisely, those assemblies loaded by CLR) by the tests in the code coverage report.
In my case, only MY.PROJECT.A is used by tests, MY.PROJECT.B and MY.PROJECT.C are not used by tests due to external dependencies. and even with using MY.PROJECT.B directives in the tests, it does not count as CLR's lazy loading.
One dummy workaround to show 0% coverage report for these two assemblies is that either use anything in the assembly in the tests, or force load these assemblies by calling System.Reflection.Assembly.Load("MY.PROJECT.B")
Related Question:
Visual Studio Code Coverage Not Showing All Assemblies
What constitutes full code coverage ...
Related
The SpecFlow report generation using nUnit is not working as expected.
The expected report should display the results of the execution as well, whereas the execution result is not displayed at all. Instead, only the tests are listed.
I am using the below format to generate the report.
specflow.exe nunitexecutionreport "C:\Users\Paresh\Documents\Visual Studio 2015\Projects\SpecFlowDemoNUnit\SpecFlowDemoNUnit\SpecFlowDemoNUnit.csproj" /xmlTestResult:"C:\Tests.xml" /out:"C:\Tests.html"
The file Tests.xml is generated after executing the tests using 'nunit3-console.exe'.
The versions used:
SpecFlow.2.1.0
NUnit.3.6.0
I figured out that the SpecFlow report generation with nUnit is probably broken with SpecFlow.2.1.0 and NUnit.3.6.0.
But after using a parameter 'format=nunit2' while executing the tests via nUnit console, the report generation worked. So finally, the commands used will look like these:
nUnit test execution:
nunit3-console.exe --labels=All --out=TestResult.txt "-- result=TestResult.xml;format=nunit2" bin\Debug\SpecFlowDemoNUnit.dll
SpecFlow Report Generation:
specflow.exe nunitexecutionreport SpecFlowDemoNUnit.csproj /out:MyResult.html
Here is the reference link:
https://github.com/techtalk/SpecFlow/wiki/Reporting
I am using sonar-runner to run tests and code coverage over my C# code with the help of gallio plugin. The tests are running fine, but I am not able to see any code coverage on the sonar web UI.
My Sonar settings are as follows:
sonar-project.properties
mentioning only relevant bits
sonar.gallio.coverage.tool = NCover
sonar.NCover.installDirectory = C:/Program Files/NCover
sonar.donet.visualstudio.testProjectPattern = .Test
sonar.dotnet.buildConfigurations = "Release|x86"
Any idea what coule be missing??
sonar.projectKey=XXX:XXX
sonar.projectVersion=trunk
sonar.projectName=XXX
sources=.
sonar.language=cs
sonar.dotnet.visualstudio.solution.file=Project.sln
sonar.dotnet.excludeGeneratedCode=false
sonar.dotnet.4.0.sdk.directory=C:/WIndows/Microsoft.NET/Framework/v4.0.30319
sonar.dotnet.version=4.0
# Gallio
sonar.gallio.mode=
sonar.gallio.coverage.tool=NCover
sonar.gallio.runner=IsolatedAppDomain
sonar.NCover.installDirectory=c:/Program Files/NCover
sonar.gallio.installDirectory=C:/Program Files/Gallio
sonar.dotnet.test.assemblies=$(SolutionDir)/../**/bin/**/*.Tests.Unit.dll
# FXCop
sonar.fxcop.mode=
#StyleCop
sonar.stylecop.mode=
#NDeps
sonar.ndeps.mode=skip
sonar-runner.properties
You said
sonar.dotnet.buildConfigurations = "Release|x86"
If that's true, your build likely isn't generating .pdb files, which are needed to figure out the mapping between the binaries and your source files.
Does it work if you try it with a Debug build?
I was seeing this same behavior with NCover in Sonar. I found that Sonar was generating invalid arguments for Gallio's NCover runner.
Try piping the output from Sonar's runner into a text file so that you can examine the arguments more easily (on the command line, you can just type sonar-runner > output.txt to do this).
You will likely see a line like this in your output:
INFO .u.c.CommandExecutor - Executing command: C:\Program Files\Gallio\bin\Gallio.Echo.exe /r:Local /report-directory:E:\Reports\.sonar /report-name-format:gallio-report /report-type:Xml E:\Projects\UnitTests\bin\Release\UnitTests.dll /runner-property:NCoverCoverageFile=E:\Reports\.sonar\coverage-report.xml /runner-property:NCoverArguments=//ias MyFirstAssembly;MySecondtAssembly;MyThirdAssembly
If you attempt to execute this manually via Gallio on the command line, you will get an error:
Cannot find file 'MyFirstAssembly;MySecondtAssembly;MyThirdAssembly'
If you edit this list manually down to a single entry such as MyFirstAssembly*, everything will work as expected.
This seems to indicate that Sonar is generating invalid command line arguments for Gallio. As much as I love NCover, the easiest solution was to use OpenCover instead.
I get this error when running my Moq tests through Teamcity 5
Test(s) failed.
System.IO.FileNotFoundException :
Could not load file or assembly 'Moq,
Version=3.1.416.3, Culture=neutral,
PublicKeyToken=69f491c39445e920' or
one of its dependencies. The system
cannot find the file specified. at
MyCode.Tests.SomeHandlerTests.Setup()
The tests run fine on my local; they just fail on the build server.
I made sure the assemblies are in the Bin (looking at them now over RDP just be double sure).
So the issue was to do with the Test DLL search path under the nunit settings
It was:
..\Tests\**\*Test*.dll
But is now:
..\Tests\*\bin\Debug\*Test*.dll
And things work nicely
UPDATE
http://confluence.jetbrains.com/display/TCD8/NUnit
You can use this pattern
**\*.dll
as long as you add this pattern in the "Do not run tests from" field
**\obj\**\*.dll
I had a similar issue, but found that I had different version's of Moq between my 2 Test projects.
The issue that I had was that the correct version was not available.
Just do
Update-Package Moq
From the Package Manager command line
I am having problems running tests with the command line NUnit test runner.
I am using version 2.5.4 with .NET 4 on an x64 machine.
Using the following line results in a failure "Could not load file or assembly 'bar' or one of
its dependencies. The system cannot find the file specified."
nunit-console-x86 foo.dll bar.dll /framework=4.0.30319
If I reverse the dll file names it complains about not finding 'foo' instead...
It works if I run them separately like:
nunit-console-x86 foo.dll /framework=4.0.30319
Also the tests of the second file works if I run:
nunit-console-x86 bar.dll /framework=4.0.30319
Before upgrading our projects to 4.0 we used NUnit 2.5.2 and the same command line tool options and at that point the runner worked well with multiple assemblies. It seems like the ability to run tests on multiple files at the same time is broken...
Anyone that can see the same behavior or does it work indicating that my environment is somehow broken?
/Per
Assembly loading behaviour has changed between 2.5.4 and 2.5.3. It was causing problems for us, so we reverted to 2.5.3, since that still supports the 4.0 framework.
We use nunit-console.exe 2.5.3 with multiple assemblies in our msbuild script, which looks like this:
<Exec Command="%22$(NUnit_Install_Directory)bin\net-2.0\nunit-console.exe%22
/noshadow #(TestableAssemblies, ' ') /xml $(BuildFilesPath)\NUnit-Results.xml" />
On execution, it fills out like this (edited for readability):
"c:\Program Files\NUnit 2.5.3\bin\net-2.0\nunit-console.exe" /noshadow D:\BuildAgent\GojiSoft.Application.Test\bin\Release\GojiSoft.Application.Test.dll D:\BuildAgent\GojiSoft.Common.Test\bin\Release\GojiSoft.Common.Test.dll
/xml D:\BuildAgent\work\2f016459feee51ce\Build\NUnit-Results.xml
We don't use a 64bit machine for our unit tests, so YMMV.
I have a blog post about the addtional modification your should make to the nunit-console-x86.exe.config here: Getting .Net 4.0, Team City, MSBuild and Nunit to play nice.
I has the same problem. I got around it by creating an nunit project in the GUI that includes multiple dlls. Then I can run it with nunit-console.
This question is unlikely to help any future visitors; it is only relevant to a small geographic area, a specific moment in time, or an extraordinarily narrow situation that is not generally applicable to the worldwide audience of the internet. For help making this question more broadly applicable, visit the help center.
Closed 10 years ago.
Solved: Look at the bottom of this question for the working configuration
I am attempting to set up NCover for usage in my FinalBuilder project, for a .NET 4.0 C# project, but my final coverage output file contains no coverage data.
I am using:
NCover 3.3.2
NUnit 2.5.4
FinalBuilder 6.3.0.2004
All tools are the latest official as of today.
I've finally managed to coax FB into running my unit tests under NCover for the .NET 4.0 project, so I get Tests run: 184, ..., which is correct.
However, the final Coverage.xml file output from NCover is almost empty, and looks like this:
<?xml version="1.0" encoding="utf-8"?>
<!-- saved from NCover 3.0 Export url='http://www.ncover.com/' -->
<coverage profilerVersion="3.3.2.6211" driverVersion="3.3.2" exportversion="3" viewdisplayname="" startTime="2010-04-22T08:55:33.7471316Z" measureTime="2010-04-22T08:55:35.3462915Z" projectName="" buildid="27c78ffa-c636-4002-a901-3211a0850b99" coveragenodeid="0" failed="false" satisfactorybranchthreshold="95" satisfactorycoveragethreshold="95" satisfactorycyclomaticcomplexitythreshold="20" satisfactoryfunctionthreshold="80" satisfactoryunvisitedsequencepoints="10" uiviewtype="TreeView" viewguid="C:\Dev\VS.NET\LVK.IoC\LVK.IoC.Tests\bin\Debug\Coverage.xml" viewfilterstyle="None" viewreportstyle="SequencePointCoveragePercentage" viewsortstyle="Name">
<rebasedpaths />
<filters />
<documents>
<doc id="0" excluded="false" url="None" cs="" csa="00000000-0000-0000-0000-000000000000" om="0" nid="0" />
</documents>
</coverage>
The output in FB log is:
...
***************** End Program Output *****************
Execution Time: 1,5992 s
Coverage Xml: C:\Dev\VS.NET\LVK.IoC\LVK.IoC.Tests\bin\Debug\Coverage.xml
NCover Success
My configuration of the FB step for NCover:
NCover what?: NUnit test coverage
Command: C:\Program Files (x86)\NUnit 2.5.4\bin\net-2.0\nunit-console.exe
Command arguments: LVK.IoC.Tests.dll /noshadow /framework:4.0.30319 /process=single /nothread
Note: I've tried with and without the /process and /nothread options
Working directory: %FBPROJECTDIR%\LVK.IoC.Tests\bin\Debug
List of assemblies to profile: %FBPROJECTDIR%\LVK.IoC.Tests\bin\Debug\LVK.IoC.dll
Note: I've tried just listing the name of the assembly, both with and without the extension.
The documentation for the FB step doesn't help, as it only lists minor sentences for each property, and fails to give examples or troubleshooting hints.
Since I want to pull the coverage results into NDepend to run build-time analysis, I want that file to contain the information I need.
I am also using TestDriven, and if I right-click the solution file and select "Test with NCover", NCover-explorer opens up with coverage data, and if I ask it to show me the folder with coverage files, in there is an .xml file with the same structure as the one above, just with all the data that should be there, so the tools I have is certainly capable of producing it.
Has anyone an idea of what I've configured wrong here?
Solved: The following configuration finally worked for me. It produces a code coverage basic statistics on the start results page for a build, with a link to a detailed report with everything.
Since I name my unit test projects the same as the project it tests, only with a .Tests suffix, I've excluded those from the code coverage. Also, I had to specifically include only my own assemblies (which all start with LVK.) otherwise I would get some assemblies loaded by the TeamCity runner, which slightly impacted my coverage statistics.
First, I installed latest of the tools (as of this edit):
NCover 3.4.8
TeamCity 5.1.2
NUnit 2.5.5
The configuration on the build-screen in TeamCity is then as follows:
Targets: Rebuild
Configuration: Debug
Platform: Any CPU
.NET Coverage Tool: NCover (3.x)
.NET Runtime: .NET Framework 4.0, Version: Native
Path to NCover 3: %system.ncover.v3.x64.path% (this is Windows 2008 Server R2 64-bit)
Additional NCover Arguments:
//ias LVK.*
//eas .*Tests
//eas LVK\.CodeQuality
//ea LVK\.CodeQuality\.IgnoreCodeCoverageAttribute
Additional NCover Reporting Arguments:
//or FullCoverageReport:Html:{teamcity.report.path}
NUnit Runner: NUnit 2.5.5
.NET Runtime Platform: auto (MSIL), Version: 4.0
Run tests from: **\bin\Debug*Tests.dll
As far as I know, NCover is not capable of covering tests for .NET 4.0. They announced a new version for .NET 4.0 but it is not there yet.
Find info on the subject here. Personally I have no experience with this task...
Update: They released NCover 3.4 with .NET 4.0 support. Perhaps You give it a try.
I've found your answer from this thread: NUnit isn't running Visual Studio 2010 code
To the nunit-console-x86.exe.config file, under <configuration> add:
<startup>
<requiredRuntime version="v4.0.30319" />
</startup>
Under <runtime> add
<loadFromRemoteSources enabled="true" />
Fixed it for me. I'm using TeamCity not FinalBuilder though.
I'm also using NCover 3.4.