I am generating an NUnit test report XML from solution. I am using Specflow to generate an HTML file, however the report is missing code coverage summary.
Please help me to add code coverage summary in report.
I am using this batch file:
cd C:\TF_Test\Infrastructure_Tests\NUnit-2.6.3\NUnit-2.6.3\bin
nunit-console.exe /labels /out=TestResult.txt /xml=TestResult.xml C:\TF_Test\Infrastructure_Tests\Infrstructure.1100_Tests\bin\Release\ThinkIQ.Platform.Infrastructure._1100.Tests.dll
C:\TF_Test\Infrastructure_Tests\SpecFlow.1.9.0\tools\specflow.exe nunitexecutionreport C:\TF_Test\Infrastructure_Tests\Infrstructure.1100_Tests\Infrastructure._1100.Tests.csproj /out:C:\TF_Test\Infrastructure_Tests\Report1.xml
Related
I'm working on enable java code coverage for AOSP. I added jacoco option in Android.bp ,make the module/test with coverage option:
EMMA_INSTRUMENT=true EMMA_INSTRUMENT_FRAMEWORK=false EMMA_INSTRUMENT_STATIC=true ANDROID_COMPILE_WITH_JACK=false SKIP_BOOT_JARS_CHECK=true -f build/core/main.mk MODULES-IN-external-jacoco
But my coverage.html report includes lots of org elements. How to exclude them? I only want to get the module's coverage.
The org elements are like: org.jacoco.core.tools
See screenshot of these elements
Created with JaCoCo 0.7.10.201704181138.android
I'm trying to generate a code coverage report using the build pipeline for the C# unit tests (MSTest2). Report can be generated using the Reportgenerator.exe but expects .xml file as the input. I have added the Visual Studio test task which has generated a .coverage file in the build artifact. We could use CodeCoverage.exe to convert .coverage to .xml file.
To test this locally, I have copied the .coverage file and on running:
C:\Program Files (x86)\Microsoft Visual Studio\2017\Enterprise\Team Tools\Dynamic Code Coverage Tools\amd64>CodeCoverage collect /IIS /session:WebSession /output:'C:\CoverageFiles\test.coverage'
and
C:\Program Files (x86)\Microsoft Visual Studio\2017\Enterprise\Team Tools\Dynamic Code Coverage Tools\amd64>CodeCoverage analyze /output:'c:\CoverageFiles\results.xml' 'c:\CoverageFiles\test.coverage'
the script is not throwing any error and xml file is also not generated.
Is there any other way to generate the .xml file from .coverage file? Any help on this is appreciated.
You can use coverlet to create a coverage report in different format. Coverlet will be invoked by msbuild. You can check my azure-pipelines.yml, where I use coverlet in combination with reporgenerator to create the code coverage in the build pipeline.
Parts of the yml:
- script: |
echo $(Build.SourcesDirectory)
cd src\Mwd.Exceptions.Solution
mkdir $(Build.SourcesDirectory)\results
dotnet test --logger trx /p:CollectCoverage=true /p:CoverletOutputFormat=cobertura
copy Mwd.Exceptions.Test\coverage.cobertura.xml $(Build.SourcesDirectory)\results
dotnet tool install dotnet-reportgenerator-globaltool --tool-path . --version 4.0.0-rc4
.\reportgenerator "-reports:$(Build.SourcesDirectory)\results\coverage.cobertura.xml" "-targetdir:$(Build.SourcesDirectory)\results" "-reporttypes:HTMLInline;HTMLChart"
dotnet test --logger trx /p:CollectCoverage=true /p:CoverletOutputFormat=cobertura creates the coverage in cobertura format, at the time the only format build pipelines are supporting.
.\reportgenerator "-reports:$(Build.SourcesDirectory)\results\coverage.cobertura.xml" "-targetdir:$(Build.SourcesDirectory)\results" "-reporttypes:HTMLInline;HTMLChart" generate the report.
Since coverlet adds build targets to msbuild, compiling your solution(s) via devenv.exe should also be supported.
The SpecFlow report generation using nUnit is not working as expected.
The expected report should display the results of the execution as well, whereas the execution result is not displayed at all. Instead, only the tests are listed.
I am using the below format to generate the report.
specflow.exe nunitexecutionreport "C:\Users\Paresh\Documents\Visual Studio 2015\Projects\SpecFlowDemoNUnit\SpecFlowDemoNUnit\SpecFlowDemoNUnit.csproj" /xmlTestResult:"C:\Tests.xml" /out:"C:\Tests.html"
The file Tests.xml is generated after executing the tests using 'nunit3-console.exe'.
The versions used:
SpecFlow.2.1.0
NUnit.3.6.0
I figured out that the SpecFlow report generation with nUnit is probably broken with SpecFlow.2.1.0 and NUnit.3.6.0.
But after using a parameter 'format=nunit2' while executing the tests via nUnit console, the report generation worked. So finally, the commands used will look like these:
nUnit test execution:
nunit3-console.exe --labels=All --out=TestResult.txt "-- result=TestResult.xml;format=nunit2" bin\Debug\SpecFlowDemoNUnit.dll
SpecFlow Report Generation:
specflow.exe nunitexecutionreport SpecFlowDemoNUnit.csproj /out:MyResult.html
Here is the reference link:
https://github.com/techtalk/SpecFlow/wiki/Reporting
I want to run NUnit on TeamCity and generate report with dotCover. But for some reasons I cannot get the coverage report for all my project assemblies.
TeamCity config as below
Teamcity: 6.5.3
NUnit: 2.5.10
.NET Runtime: Platform: x86
.NET Runtime: Version: v4.0
dotCover: bundled with TC not customized
I have checked that all *.dll *.xml *.pdb files are there in the directory as expected as below (reference DLLs are not listed)
MY.PROJECT.A.dll
MY.PROJECT.A.pdb
MY.PROJECT.A.xml
MY.PROJECT.B.dll
MY.PROJECT.B.pdb
MY.PROJECT.B.xml
MY.PROJECT.C.dll
MY.PROJECT.C.pdb
MY.PROJECT.C.xml
MY.PROJECT.Test.dll
MY.PROJECT.Test.pdb
MY.PROJECT.Test.xml
MY.PROJECT.Test.dll is executed with NUnit and this assembly is excluded in the coverage report by using *Test* filter. But only MY.PROJECT.A is in the dotCover coverage report whereas MY.PROJECT.B and MY.PROJECT.C are not included.
I checked the log but not error is found.
Any thoughts is much appreciated.
Finally, I figured out what is going on, and hope this answer is useful for those who are still struggling with this or similar problem.
Basically, dotCover only includes those assemblies are actually used (more precisely, those assemblies loaded by CLR) by the tests in the code coverage report.
In my case, only MY.PROJECT.A is used by tests, MY.PROJECT.B and MY.PROJECT.C are not used by tests due to external dependencies. and even with using MY.PROJECT.B directives in the tests, it does not count as CLR's lazy loading.
One dummy workaround to show 0% coverage report for these two assemblies is that either use anything in the assembly in the tests, or force load these assemblies by calling System.Reflection.Assembly.Load("MY.PROJECT.B")
Related Question:
Visual Studio Code Coverage Not Showing All Assemblies
What constitutes full code coverage ...
I am using sonar-runner to run tests and code coverage over my C# code with the help of gallio plugin. The tests are running fine, but I am not able to see any code coverage on the sonar web UI.
My Sonar settings are as follows:
sonar-project.properties
mentioning only relevant bits
sonar.gallio.coverage.tool = NCover
sonar.NCover.installDirectory = C:/Program Files/NCover
sonar.donet.visualstudio.testProjectPattern = .Test
sonar.dotnet.buildConfigurations = "Release|x86"
Any idea what coule be missing??
sonar.projectKey=XXX:XXX
sonar.projectVersion=trunk
sonar.projectName=XXX
sources=.
sonar.language=cs
sonar.dotnet.visualstudio.solution.file=Project.sln
sonar.dotnet.excludeGeneratedCode=false
sonar.dotnet.4.0.sdk.directory=C:/WIndows/Microsoft.NET/Framework/v4.0.30319
sonar.dotnet.version=4.0
# Gallio
sonar.gallio.mode=
sonar.gallio.coverage.tool=NCover
sonar.gallio.runner=IsolatedAppDomain
sonar.NCover.installDirectory=c:/Program Files/NCover
sonar.gallio.installDirectory=C:/Program Files/Gallio
sonar.dotnet.test.assemblies=$(SolutionDir)/../**/bin/**/*.Tests.Unit.dll
# FXCop
sonar.fxcop.mode=
#StyleCop
sonar.stylecop.mode=
#NDeps
sonar.ndeps.mode=skip
sonar-runner.properties
You said
sonar.dotnet.buildConfigurations = "Release|x86"
If that's true, your build likely isn't generating .pdb files, which are needed to figure out the mapping between the binaries and your source files.
Does it work if you try it with a Debug build?
I was seeing this same behavior with NCover in Sonar. I found that Sonar was generating invalid arguments for Gallio's NCover runner.
Try piping the output from Sonar's runner into a text file so that you can examine the arguments more easily (on the command line, you can just type sonar-runner > output.txt to do this).
You will likely see a line like this in your output:
INFO .u.c.CommandExecutor - Executing command: C:\Program Files\Gallio\bin\Gallio.Echo.exe /r:Local /report-directory:E:\Reports\.sonar /report-name-format:gallio-report /report-type:Xml E:\Projects\UnitTests\bin\Release\UnitTests.dll /runner-property:NCoverCoverageFile=E:\Reports\.sonar\coverage-report.xml /runner-property:NCoverArguments=//ias MyFirstAssembly;MySecondtAssembly;MyThirdAssembly
If you attempt to execute this manually via Gallio on the command line, you will get an error:
Cannot find file 'MyFirstAssembly;MySecondtAssembly;MyThirdAssembly'
If you edit this list manually down to a single entry such as MyFirstAssembly*, everything will work as expected.
This seems to indicate that Sonar is generating invalid command line arguments for Gallio. As much as I love NCover, the easiest solution was to use OpenCover instead.