I have a very simple project in my VSTS account with a few tests. I've added a VsTest step to the build and checked the code coverage box. However, after I run the build, it says "no code coverage results available" and suggests that I add a Visual Studio test build step to my build. But when I add one, it's the same as the VsTest step I already have.
How can I get code coverage from my VSTS build?
EDIT: The test logs are below. There were two portions with a lot of special characters I could not copy and paste.
2018-05-04T21:51:59.5221025Z ##[section]Starting: VsTest - testAssemblies
2018-05-04T21:51:59.5227667Z ==============================================================================
2018-05-04T21:51:59.5227804Z Task : Visual Studio Test
2018-05-04T21:51:59.5227968Z Description : Run unit and functional tests (Selenium, Appium, Coded UI test, etc.) using the Visual Studio Test runner. Test frameworks that have a Visual Studio test adapter such as xUnit, NUnit, Chutzpah, etc. can also be run. Tests can be distributed on multiple agents using this task (version 2).
2018-05-04T21:51:59.5228145Z Version : 2.5.8
2018-05-04T21:51:59.5228227Z Author : Microsoft Corporation
2018-05-04T21:51:59.5228339Z Help : [More Information](https://go.microsoft.com/fwlink/?LinkId=835764)
2018-05-04T21:51:59.5228918Z ==============================================================================
2018-05-04T21:52:00.6581473Z Run the tests locally using vstest.console.exe
2018-05-04T21:52:00.6581612Z ========================================================
2018-05-04T21:52:00.6582115Z Test selector : Test assemblies
2018-05-04T21:52:00.6582269Z Test assemblies : **\release\**\*.dll
2018-05-04T21:52:00.6582399Z Test filter criteria : null
2018-05-04T21:52:00.6582534Z Search folder : D:\a\1\s
2018-05-04T21:52:00.6582658Z Run settings file : D:\a\1\s
2018-05-04T21:52:00.6582831Z Run in parallel : false
2018-05-04T21:52:00.6582954Z Run in isolation : false
2018-05-04T21:52:00.6584318Z Path to custom adapters : null
2018-05-04T21:52:00.6584486Z Other console options : null
2018-05-04T21:52:00.6584627Z Code coverage enabled : true
2018-05-04T21:52:00.6584910Z Rerun failed tests: false
2018-05-04T21:52:00.6585116Z VisualStudio version selected for test execution : latest
2018-05-04T21:52:01.5193379Z ========================================================
2018-05-04T21:52:01.5908190Z [command]"C:\Program Files (x86)\Microsoft Visual Studio\2017\Enterprise\Common7\IDE\CommonExtensions\Microsoft\TestWindow\vstest.console.exe" #D:\a\_temp\60315131-4fe5-11e8-9269-e9217bdc2ba0.txt
2018-05-04T21:52:06.6551385Z Microsoft (R) Test Execution Command Line Tool Version 15.6.2
2018-05-04T21:52:06.6552014Z Copyright (c) Microsoft Corporation. All rights reserved.
2018-05-04T21:52:06.6552213Z
2018-05-04T21:52:06.8162937Z vstest.console.exe
2018-05-04T21:52:06.8164594Z "D:\a\1\s\TestTesting\TestTesting\bin\Release\net461\Microsoft.VisualStudio.CodeCoverage.Shim.dll"
2018-05-04T21:52:06.8165659Z "D:\a\1\s\TestTesting\TestTesting\bin\Release\net461\TestTesting.dll"
2018-05-04T21:52:06.8166614Z "D:\a\1\s\TestTesting\TestTesting\bin\Release\net461\xunit.abstractions.dll"
2018-05-04T21:52:06.8167552Z "D:\a\1\s\TestTesting\TestTesting\bin\Release\net461\xunit.assert.dll"
2018-05-04T21:52:06.8168335Z "D:\a\1\s\TestTesting\TestTesting\bin\Release\net461\xunit.core.dll"
2018-05-04T21:52:06.8169121Z "D:\a\1\s\TestTesting\TestTesting\bin\Release\net461\xunit.execution.desktop.dll"
2018-05-04T21:52:06.8169879Z "D:\a\1\s\TestTesting\TestTesting\bin\Release\net461\xunit.runner.reporters.net452.dll"
2018-05-04T21:52:06.8170678Z "D:\a\1\s\TestTesting\TestTesting\bin\Release\net461\xunit.runner.utility.net452.dll"
2018-05-04T21:52:06.8171460Z "D:\a\1\s\TestTesting\TestTesting\bin\Release\net461\xunit.runner.visualstudio.testadapter.dll"
2018-05-04T21:52:06.8172221Z "D:\a\1\s\TestTesting\TestTesting\obj\Release\net461\TestTesting.dll"
2018-05-04T21:52:06.8172936Z /EnableCodeCoverage
2018-05-04T21:52:06.8173974Z /logger:"trx"
2018-05-04T21:52:06.8174130Z /TestAdapterPath:"D:\a\1\s"
2018-05-04T21:52:10.5221260Z Starting test execution, please wait...
2018-05-04T21:52:10.9446034Z Test run will use DLL(s) built for framework .NETFramework,Version=v4.6.1 and platform X86. Following DLL(s) do not match framework/platform settings.
2018-05-04T21:52:10.9446506Z xunit.assert.dll is built for Framework 1.1 and Platform AnyCPU.
2018-05-04T21:52:10.9446876Z xunit.core.dll is built for Framework 1.1 and Platform AnyCPU.
2018-05-04T21:52:10.9447201Z xunit.execution.desktop.dll is built for Framework 4.5.2 and Platform AnyCPU.
2018-05-04T21:52:10.9449027Z xunit.runner.reporters.net452.dll is built for Framework 4.5.2 and Platform AnyCPU.
2018-05-04T21:52:10.9449230Z xunit.runner.utility.net452.dll is built for Framework 4.5.2 and Platform AnyCPU.
2018-05-04T21:52:10.9449440Z xunit.runner.visualstudio.testadapter.dll is built for Framework 4.5.2 and Platform AnyCPU.
2018-05-04T21:52:10.9449631Z Go to http://go.microsoft.com/fwlink/?LinkID=236877&clcid=0x409 for more details on managing these settings.
2018-05-04T21:52:10.9449774Z
[[Uncopyable part here. What I could read spelled out "Microsoft Coverage Collection", followed by a copyright.
2018-05-04T21:52:31.9257584Z [xUnit.net 00:00:00.9144322] Discovering: Microsoft.VisualStudio.CodeCoverage.Shim
2018-05-04T21:52:32.0551765Z [xUnit.net 00:00:01.0599884] Discovered: Microsoft.VisualStudio.CodeCoverage.Shim
2018-05-04T21:52:32.4727660Z [xUnit.net 00:00:01.4776210] Discovering: TestTesting
2018-05-04T21:52:32.7346755Z [xUnit.net 00:00:01.7395514] Discovered: TestTesting
2018-05-04T21:52:32.7417690Z [xUnit.net 00:00:01.7468703] Starting: TestTesting
2018-05-04T21:52:33.2084443Z [xUnit.net 00:00:01.9469140] TestTesting.Tests.Skipped [SKIP]
2018-05-04T21:52:33.2085046Z [xUnit.net 00:00:01.9471459] Skipping
2018-05-04T21:52:33.2085275Z [xUnit.net 00:00:01.9542762] Finished: TestTesting
2018-05-04T21:52:33.2881381Z [xUnit.net 00:00:02.2931214] Discovering: xunit.abstractions
2018-05-04T21:52:33.3305957Z [xUnit.net 00:00:02.3357501] Discovered: xunit.abstractions
2018-05-04T21:52:34.0395297Z [xUnit.net 00:00:03.0443103] Discovering: xunit.runner.reporters.net452
2018-05-04T21:52:34.1043673Z [xUnit.net 00:00:03.1090663] Discovered: xunit.runner.reporters.net452
2018-05-04T21:52:34.4684448Z Passed TestTesting.Tests.Passing
2018-05-04T21:52:34.8614665Z Skipped TestTesting.Tests.Skipped
2018-05-04T21:52:34.8620656Z Standard Output Messages:
2018-05-04T21:52:34.8620971Z Skipping
2018-05-04T21:52:34.8625877Z Passed TestTesting.Tests.NoLongerFailing
2018-05-04T21:52:36.1389527Z [xUnit.net 00:00:05.1437439] Discovering: xunit.runner.utility.net452
2018-05-04T21:52:36.2616608Z [xUnit.net 00:00:05.2660797] Discovered: xunit.runner.utility.net452
[[Uncopyable part repeated here.]]
2018-05-04T21:52:38.5800594Z Attachments:
2018-05-04T21:52:38.5800836Z D:\a\1\s\TestResults\b0d720b3-9bb9-4318-bb6b-7bfa8e404a13\VssAdministrator_factoryvm-az384 2018-05-04 21_52_18.coverage
2018-05-04T21:52:38.5801023Z
2018-05-04T21:52:38.5801196Z Total tests: 3. Passed: 2. Failed: 0. Skipped: 1.
2018-05-04T21:52:38.5801368Z Test Run Successful.
2018-05-04T21:52:38.5809597Z Test execution time: 19.0295 Seconds
2018-05-04T21:52:38.6045339Z Results File: D:\a\1\s\TestResults\VssAdministrator_factoryvm-az384_2018-05-04_21_52_34.trx
2018-05-04T21:52:38.9702367Z ##[section]Async Command Start: Publish test results
2018-05-04T21:52:39.1587014Z Publishing test results to test run '3'
2018-05-04T21:52:39.1587207Z Test results remaining: 3. Test run id: 3
2018-05-04T21:52:39.6699238Z Published Test Run : https://wjtesttesting.visualstudio.com/MyFirstProject/_TestManagement/Runs#runId=3&_a=runCharts
2018-05-04T21:52:39.6700510Z ##[section]Async Command End: Publish test results
2018-05-04T21:52:39.6701395Z ##[section]Finishing: VsTest - testAssemblies
[EDIT 2]: Starian Chen asked for the logs with System.Debug set to true. They go over SO's size limit, so I'm linking to them below:
VSTS build log
VSTS test log
EDIT 3: When I specify the framework in "Other console options" of my VsTest build step:
/framework:"Framework461"
It seems to be unhappy with my version. It gives a list of supported frameworks, the highest of which is "Framework45."
Edit 4: I tried a net452 project, with "Framework45". In that case, the tests did run via Starian Chen's instructions here, just substituting "Framework45" for "netcoreapp2.0". However, there was still no code coverage report.
With .Net framework test project, you don’t need to specify framework and vstest.console.exe path in Visual Studio Test task.
Related
I am trying create a nunit test results xml file and attach it to a test run of my build in dev ops. I have the NUnit3TestAdapter.3.13.0 package installed in my test project and reference this in console options of the VS test task (test task is version 2).
packages.config:
I have tried various combinations in the "other console options" field including "/TestAdapterPath:." or not stating the test adapter path at all. I have also tried moving the test adapter part into the "path to custom test adapters" field as "$(Build.SourcesDirectory)\packages\NUnit3TestAdapter.3.13.0\build\net35".
However, each time the tests task returns the error:
Microsoft (R) Test Execution Command Line Tool Version 16.9.1
Copyright (c) Microsoft Corporation. All rights reserved.
vstest.console.exe "C:\VSTSBuild\agentB\_work\2\s\SomeService.Tests\bin\SomeService.Tests.dll"
/Settings:"C:\VSTSBuild\agentB\_work\_temp\41oiukcwga4.tmp.runsettings"
/EnableCodeCoverage
/Logger:"trx"
/TestAdapterPath:"C:\VSTSBuild\agentB\_work\2\s"
/logger:nunit /TestAdapterPath:C:\VSTSBuild\agentB\_work\2\s\packages\NUnit3TestAdapter.3.13.0\build\net35
Starting test execution, please wait...
##[error]Could not find a test logger with AssemblyQualifiedName, URI or FriendlyName 'nunit'.
That same error was solved on my end by adding a reference to this NuGet package:
NunitXml.TestLogger
Its description is:
Xml logger for NUnit v3 compliant xml report when test is running with "dotnet test" or "dotnet vstest".
I'm transitioning a traditional C++ code base build to the Azure DevOps Pipeline. We have a bunch of gtest unit tests in a module that is just an executable named "sdktests.exe". It links to gtest 1.8.1. The user has always just manually run this exe and observed the results.
But I can't seem to make it Azure Devops detect and run these tests. I'm guessing that it needs the tests in the form of a DLL that it loads instead of an EXE
Is that the case? Do I need to convert my google tests module from an EXE to a DLL to make this work?
This is my yaml pipeline step
- task: VSTest#2
inputs:
platform: '$(buildPlatform)'
configuration: '$(buildConfiguration)'
testSelector: 'testAssemblies'
testAssemblyVer2: '**\*sdktests*.exe'
This is my Azure build output for the step:
Source filter: **\*sdktests*.exe
SystemVssConnection exists true
d:\a\_tasks\VSTest_ef087383-ee5e-42c7-9a53-ab56c98420f9\2.170.1\Modules\DTAExecutionHost.exe --inputFile d:\a\_temp\input_484b64d0-c544-11ea-bc95-251e50750ee7.json
======================================================
##########################################################################
DtaExecutionHost version 18.170.30112.1.
Starting TestExecution Model...
Result Attachments will be stored in LogStore
Run Attachments will be stored in LogStore
Result Attachments will be stored in LogStore
Result Attachments will be stored in LogStore
Run Attachments will be stored in LogStore
Updated Run Settings:
<RunSettings>
<RunConfiguration>
<BatchSize>1000</BatchSize>
<ResultsDirectory>d:\a\_temp\TestResults</ResultsDirectory>
</RunConfiguration>
</RunSettings>
**************** Starting test execution *********************
C:\Program Files (x86)\Microsoft Visual Studio\2019\Enterprise\Common7\IDE\Extensions\TestPlatform\vstest.console.exe "#d:\a\_temp\jcgckxlo13t.tmp"
Microsoft (R) Test Execution Command Line Tool Version 16.6.0
Copyright (c) Microsoft Corporation. All rights reserved.
vstest.console.exe "d:\a\1\s\x64\Release\sdktests.exe"
/Settings:"d:\a\_temp\ktasri4fewz.tmp.runsettings"
/Logger:"trx"
/TestAdapterPath:"d:\a\1\s"
Starting test execution, please wait...
A total of 1 test files matched the specified pattern.
No test is available in d:\a\1\s\x64\Release\sdktests.exe. Make sure that test discoverer & executors are registered and platform & framework version settings are appropriate and try again.
#Joe,
I have run into a similar situation in the past. Here is what I found. If you are running GTest you should be running as you would normally, no need to us VSTest task. The test result file that is generated by GTest is in a JUnit formatted XML file. You can use the publish test results v2 task to publish to the build. The setup would look something like:
- task: CmdLine#1
displayName: Run Unit Tests (GTest)
inputs:
script: 'sdktests.exe'
- task: PublishTestResults#2
displayName: Publish Unit Test Results (GTest)
inputs:
testResultsFiles: '**/SDKTestResults.xml'
testRunTitle: 'GTest Results'
Here are references that you should use to create a solution that meets your needs:
Publish Test Results - Doc
Command Line - Doc
The build runs well but can not see the unit test result When added a build pipeline which used a classic edit or for my solution.
And the code coverage also blank. I have two projects and tests project in the solution. The unit test task returns the result as below:
A total of 14 test files matched the specified pattern.
No test is available in d:\a\1\s\src.net...
Make sure that test discoverer & executors are registered and platform & framework version settings are appropriate and try again.
Results File: d:\a\_temp\TestResults\VssAdministrator_fv-az686_2019-12-26_06_10_28.trx
Attachments:
d:\a\_temp\TestResults\d301a26b-d99f-4b4e-bbe8-c95e588ee1c5\VssAdministrator_fv-az686 2019-12-26 06_10_20.coverage
Vstest.console.exe exited with code 0.
**************** Completed test execution *********************
Test results files: d:\a\_temp\TestResults\VssAdministrator_fv-az686_2019-12-26_06_10_28.trx
No Result Found to Publish 'd:\a\_temp\TestResults\VssAdministrator_fv-az686_2019-12-26_06_10_28.trx'.
Created test run: 1006840
Publishing test results: 0
Publishing test results to test run '1006840'.
TestResults To Publish 0, Test run id:1006840
Published test results: 0
Publishing Attachments: 2
Completed TestExecution Model...
And also I can not see anymore from the Tests windows.
The test YAML script as below:
steps:
- task: VSTest#2
displayName: 'Test Assemblies'
inputs:
testAssemblyVer2: |
**\$(BuildConfiguration)\*test*.dll
!**\obj\**
codeCoverageEnabled: true
platform: '$(BuildPlatform)'
configuration: '$(BuildConfiguration)'
There no any problem on your YAML, the same definition is all work for me.
A total of 14 test files matched the specified pattern.
No test is available in d:\a\1\s\src.net...
Based on these error message, the test files has been identified out from the folder which means the dlls has exists now. But it still say there's no test available, this is more relative with vstest.console.exe did not identify the test.
For example, if your test project type is NUnit, you could use one extension names NUnit test adapter to support this test type running in visual studio.
But, when the project is published into VSTS, this extension could not be published also, and this extension does not exists in VSTS hosted agent. Without this extension support, the vstest.console.exe could not identify the Nunit test, then it will prompt the message like No test is available in xxxxxxx. See this as refer to.
But, you can installed nuget packages to replace the role of that VS extension.
If it's type is Nunit test, you must ensure add the reference on packages NUnit and NUnit3TestAdapter in your csproj file:
<PackageReference Include="NUnit">
<Version>3.12.0</Version>
</PackageReference>
<PackageReference Include="NUnit3TestAdapter">
<Version>3.15.1</Version>
</PackageReference>
Similar method tp xUnit test.
When the "run code analysis" is started in the Azure DevOps the following error is shown:
==============================================================================
Task : Run Code Analysis
Description : Run scanner and upload the results to the SonarCloud server.
Version : 1.6.3
Author : sonarsource
Help : This task is not needed for Maven and Gradle projects since the scanner should be run as part of the build.
[More Information](http://redirect.sonarsource.com/doc/install-configure-scanner-tfs-ts.html)
==============================================================================
[command]D:\a\_tasks\SonarCloudPrepare_14d9cde6-c1da-4d55-aa01-2965cd301255\1.6.3\classic-sonar-scanner-msbuild\SonarScanner.MSBuild.exe end
SonarScanner for MSBuild 4.6.2
Using the .NET Framework version of the Scanner for MSBuild
Post-processing started.
17:36:00.872 Fetching code coverage report information from TFS...
17:36:00.872 Attempting to locate a test results (.trx) file...
17:36:00.92 Looking for TRX files in: D:\a\1\TestResults
17:36:00.92 No test results files found
17:36:00.951 Did not find any binary coverage files in the expected location.
17:36:00.951 Falling back on locating coverage files in the agent temp directory.
17:36:00.951 Searching for coverage files in D:\a\_temp
17:36:00.951 No coverage files found in the agent temp directory.
WARNING: The following projects do not have a valid ProjectGuid and were not built using a valid solution (.sln) thus will be skipped from analysis...
D:\a\1\s\Domain.Core\Domain.Core.csproj, D:\a\1\s\Shared\Shared.csproj, D:\a\1\s\API\API.csproj, D:\a\1\s\Domain.Core\Domain.Core.csproj, D:\a\1\s\Shared\Shared.csproj, D:\a\1\s\Domain.Wallet\Domain.Wallet.csproj, D:\a\1\s\Domain.Store\Domain.Stores.csproj, D:\a\1\s\Domain.Order\Domain.Orders.csproj, D:\a\1\s\Domain.Users\Domain.Users.csproj, D:\a\1\s\App\App.csproj, D:\a\1\s\Domain.Core\Domain.Core.csproj, D:\a\1\s\Domain.Core\Domain.Core.csproj, D:\a\1\s\Domain.Store\Domain.Stores.csproj, D:\a\1\s\Shared\Shared.csproj, D:\a\1\s\Domain.Store\Domain.Stores.csproj, D:\a\1\s\Domain.Order\Domain.Orders.csproj, D:\a\1\s\Domain.Core\Domain.Core.csproj, D:\a\1\s\Shared\Shared.csproj, D:\a\1\s\Domain.Store\Domain.Stores.csproj, D:\a\1\s\Domain.Core\Domain.Core.csproj, D:\a\1\s\Shared\Shared.csproj,
WARNING: Duplicate ProjectGuid: "00000000-0000-0000-0000-000000000000". The project will not be analyzed by SonarQube. Project file: "D:\a\1\s\Domain.Core\Domain.Core.csproj"
WARNING: Duplicate ProjectGuid: "00000000-0000-0000-0000-000000000000". The project will not be analyzed by SonarQube. Project file:
##[error]No analysable projects were found. SonarQube analysis will not be performed. Check the build summary report for details.
No analysable projects were found. SonarQube analysis will not be performed. Check the build summary report for details.
Generation of the sonar-properties file failed. Unable to complete SonarQube analysis.
##[error]17:36:01.201 Post-processing failed. Exit code: 1
17:36:01.201 Post-processing failed. Exit code: 1
##[error]D:\a\_tasks\SonarCloudPrepare_14d9cde6-c1da-4d55-aa01-2965cd301255\1.6.3\classic-sonar-scanner-msbuild\SonarScanner.MSBuild.exe failed with return code: 1
##[section]Finishing: Run Code Analysis
So I can not get the final report as expected.
In .Net core you must give to each project s special GUID.
So go to your each .csproj and inside the <PropertyGroup> add this line:
# Replace the guid here with a new one
<ProjectGuid>{13212313-gd5543-1321-3fdf1-313fdfs13}</ProjectGuid>
You can generate GUIDs to your projects here.
I was having the same issue in a pipeline, just remember to put the Sonar Prepare task before the build process and Sonar Analize at the end.
Cheers.
I am trying to implement an automated build process. After the build, the unit tests on nunit-console.exe are run. The following error is displayed:
> c:\nunit_2.5.10\nunit-console.exe c:\builds\Output\bin\TDD.nunit /framework=4.0.30319 /nologo /trace=Off
ProcessModel: Default DomainUsage: Default
Execution Runtime: v4.0.30319
Unhandled Exception:
System.ArgumentException: NUnit components for version 4.0.30319 of the CLR are not installed
Parameter name: targetRuntime
at NUnit.Util.TestAgency.LaunchAgentProcess(RuntimeFramework targetRuntime, Boolean enableDebug)
at NUnit.Util.TestAgency.CreateRemoteAgent(RuntimeFramework framework, Int32 waitTime, Boolean enableDebug)
at NUnit.Util.TestAgency.GetAgent(RuntimeFramework framework, Int32 waitTime, Boolean enableDebug)
at NUnit.Util.ProcessRunner.Load(TestPackage package)
at NUnit.ConsoleRunner.ConsoleUi.Execute(ConsoleOptions options)
at NUnit.ConsoleRunner.Runner.Main(String[] args)
There is no nunit-agent.exe on the build machine. However, on my machine it is not even called, so I suppose it is not necessary.
Why is nunit-agent.exe required in some cases but not always required? What conditions should be satisfied so nunit-agent would not need to launch?
Edit: I have found one resource, which kind of describes how it works, but not quite well: http://www.nunit.org/index.php?p=nunit-agent&r=2.5.10. It says that it is launched when program needs to run under a different framework than the one being used by NUnit (which is the case, since NUnit is compiled for 2.0). However, on my machine the nunit-agent.exe does not run although conditions seem to be the same.
I ran into this same error and it was definitely solved by including nunit-agent.exe in the folder where nunit-console.exe was launched. The complete list of .exes and .dlls necessary to run a test successfully was:
nunit.core.dll
nunit.core.interfaces.dll
nunit.framework.dll
nunit.util.dll
nunit-agent.exe
nunit-console.exe
nunit-console-runner.dll
All files are packaged in the download available from nunit.org. As of this post, 2.6.3 is the current version. Documentation for the console runner can be found here. And the direct download for the zip file is here.
For a test assembly targeting .NET 4.5.1, the following statement will execute tests:
nunit-console.exe your-assembly.dll /framework=v4.5.1
Adding a "startup/supportedRuntime" configuration tag to nunit-console.exe.config solved it for me.
<startup useLegacyV2RuntimeActivationPolicy="true">
<supportedRuntime version="v4.0.30319" />
</startup>
Try using Fusion to see what assembly might be missing and where the .exe is looking.
http://msdn.microsoft.com/en-us/library/e74a18c4(v=vs.110).aspx