I am trying create a nunit test results xml file and attach it to a test run of my build in dev ops. I have the NUnit3TestAdapter.3.13.0 package installed in my test project and reference this in console options of the VS test task (test task is version 2).
packages.config:
I have tried various combinations in the "other console options" field including "/TestAdapterPath:." or not stating the test adapter path at all. I have also tried moving the test adapter part into the "path to custom test adapters" field as "$(Build.SourcesDirectory)\packages\NUnit3TestAdapter.3.13.0\build\net35".
However, each time the tests task returns the error:
Microsoft (R) Test Execution Command Line Tool Version 16.9.1
Copyright (c) Microsoft Corporation. All rights reserved.
vstest.console.exe "C:\VSTSBuild\agentB\_work\2\s\SomeService.Tests\bin\SomeService.Tests.dll"
/Settings:"C:\VSTSBuild\agentB\_work\_temp\41oiukcwga4.tmp.runsettings"
/EnableCodeCoverage
/Logger:"trx"
/TestAdapterPath:"C:\VSTSBuild\agentB\_work\2\s"
/logger:nunit /TestAdapterPath:C:\VSTSBuild\agentB\_work\2\s\packages\NUnit3TestAdapter.3.13.0\build\net35
Starting test execution, please wait...
##[error]Could not find a test logger with AssemblyQualifiedName, URI or FriendlyName 'nunit'.
That same error was solved on my end by adding a reference to this NuGet package:
NunitXml.TestLogger
Its description is:
Xml logger for NUnit v3 compliant xml report when test is running with "dotnet test" or "dotnet vstest".
Related
When the "run code analysis" is started in the Azure DevOps the following error is shown:
==============================================================================
Task : Run Code Analysis
Description : Run scanner and upload the results to the SonarCloud server.
Version : 1.6.3
Author : sonarsource
Help : This task is not needed for Maven and Gradle projects since the scanner should be run as part of the build.
[More Information](http://redirect.sonarsource.com/doc/install-configure-scanner-tfs-ts.html)
==============================================================================
[command]D:\a\_tasks\SonarCloudPrepare_14d9cde6-c1da-4d55-aa01-2965cd301255\1.6.3\classic-sonar-scanner-msbuild\SonarScanner.MSBuild.exe end
SonarScanner for MSBuild 4.6.2
Using the .NET Framework version of the Scanner for MSBuild
Post-processing started.
17:36:00.872 Fetching code coverage report information from TFS...
17:36:00.872 Attempting to locate a test results (.trx) file...
17:36:00.92 Looking for TRX files in: D:\a\1\TestResults
17:36:00.92 No test results files found
17:36:00.951 Did not find any binary coverage files in the expected location.
17:36:00.951 Falling back on locating coverage files in the agent temp directory.
17:36:00.951 Searching for coverage files in D:\a\_temp
17:36:00.951 No coverage files found in the agent temp directory.
WARNING: The following projects do not have a valid ProjectGuid and were not built using a valid solution (.sln) thus will be skipped from analysis...
D:\a\1\s\Domain.Core\Domain.Core.csproj, D:\a\1\s\Shared\Shared.csproj, D:\a\1\s\API\API.csproj, D:\a\1\s\Domain.Core\Domain.Core.csproj, D:\a\1\s\Shared\Shared.csproj, D:\a\1\s\Domain.Wallet\Domain.Wallet.csproj, D:\a\1\s\Domain.Store\Domain.Stores.csproj, D:\a\1\s\Domain.Order\Domain.Orders.csproj, D:\a\1\s\Domain.Users\Domain.Users.csproj, D:\a\1\s\App\App.csproj, D:\a\1\s\Domain.Core\Domain.Core.csproj, D:\a\1\s\Domain.Core\Domain.Core.csproj, D:\a\1\s\Domain.Store\Domain.Stores.csproj, D:\a\1\s\Shared\Shared.csproj, D:\a\1\s\Domain.Store\Domain.Stores.csproj, D:\a\1\s\Domain.Order\Domain.Orders.csproj, D:\a\1\s\Domain.Core\Domain.Core.csproj, D:\a\1\s\Shared\Shared.csproj, D:\a\1\s\Domain.Store\Domain.Stores.csproj, D:\a\1\s\Domain.Core\Domain.Core.csproj, D:\a\1\s\Shared\Shared.csproj,
WARNING: Duplicate ProjectGuid: "00000000-0000-0000-0000-000000000000". The project will not be analyzed by SonarQube. Project file: "D:\a\1\s\Domain.Core\Domain.Core.csproj"
WARNING: Duplicate ProjectGuid: "00000000-0000-0000-0000-000000000000". The project will not be analyzed by SonarQube. Project file:
##[error]No analysable projects were found. SonarQube analysis will not be performed. Check the build summary report for details.
No analysable projects were found. SonarQube analysis will not be performed. Check the build summary report for details.
Generation of the sonar-properties file failed. Unable to complete SonarQube analysis.
##[error]17:36:01.201 Post-processing failed. Exit code: 1
17:36:01.201 Post-processing failed. Exit code: 1
##[error]D:\a\_tasks\SonarCloudPrepare_14d9cde6-c1da-4d55-aa01-2965cd301255\1.6.3\classic-sonar-scanner-msbuild\SonarScanner.MSBuild.exe failed with return code: 1
##[section]Finishing: Run Code Analysis
So I can not get the final report as expected.
In .Net core you must give to each project s special GUID.
So go to your each .csproj and inside the <PropertyGroup> add this line:
# Replace the guid here with a new one
<ProjectGuid>{13212313-gd5543-1321-3fdf1-313fdfs13}</ProjectGuid>
You can generate GUIDs to your projects here.
I was having the same issue in a pipeline, just remember to put the Sonar Prepare task before the build process and Sonar Analize at the end.
Cheers.
I have a simple NUnit test that makes a simple WebAPI call:
[TestFixture]
public class PerformanceTests
{
private const string web_api = <myapiurl>;
[Test]
public async Task PerformanceTest()
{
var response = await client.GetAsync(web_api);
Assert.AreEqual(HttpStatusCode.OK, response.StatusCode);
}
}
The test runs fine in Visual Studio when run using the normal test runner and also via ReSharper; it is highlighted as a NUnit test.
I have Taurus installed and have created a simple yml file to run my test:
execution:
- executor: nunit
iterations: 500
scenario:
script: C:\Users\...\tests.dll # assembly with tests
When I run the yml file in Taurus:
bzt my-nunit-tests.yml
the tests do not run and I get the following output:
Target: C:\Users\...\tests.dll
15:36:51 ERROR: NUnitExecutor STDERR:
Unhandled Exception: System.ArgumentException: Nothing to run, no tests were loaded
at NUnitRunner.NUnitRunner.Main(String[] args)
It would seem that the custom Taurus NUnit test runner is not picking up the tests. I can run the tests with the standard dotnet test command (the project does have the Microsoft.NET.Test.Sdk as a dependency).
As my test project is .NET Core I am publishing the project to ensure all dependencies are included. This ensures all the files specified here are in the same directory as the test assembly.
Update: I have created the exact same test project but in .NET Framework and Taurus finds the tests. To me this suggests the custom Taurus NUnit Test Runner doesn't work with .NET Core projects but I can't confirm this.
As of June 2018 Taurus' Custom NUnit runner does not support .NET Core. It would seem that this is because NUnit does not allow .NET Core tests to be run via the .NET Framework engine.
I have a very simple project in my VSTS account with a few tests. I've added a VsTest step to the build and checked the code coverage box. However, after I run the build, it says "no code coverage results available" and suggests that I add a Visual Studio test build step to my build. But when I add one, it's the same as the VsTest step I already have.
How can I get code coverage from my VSTS build?
EDIT: The test logs are below. There were two portions with a lot of special characters I could not copy and paste.
2018-05-04T21:51:59.5221025Z ##[section]Starting: VsTest - testAssemblies
2018-05-04T21:51:59.5227667Z ==============================================================================
2018-05-04T21:51:59.5227804Z Task : Visual Studio Test
2018-05-04T21:51:59.5227968Z Description : Run unit and functional tests (Selenium, Appium, Coded UI test, etc.) using the Visual Studio Test runner. Test frameworks that have a Visual Studio test adapter such as xUnit, NUnit, Chutzpah, etc. can also be run. Tests can be distributed on multiple agents using this task (version 2).
2018-05-04T21:51:59.5228145Z Version : 2.5.8
2018-05-04T21:51:59.5228227Z Author : Microsoft Corporation
2018-05-04T21:51:59.5228339Z Help : [More Information](https://go.microsoft.com/fwlink/?LinkId=835764)
2018-05-04T21:51:59.5228918Z ==============================================================================
2018-05-04T21:52:00.6581473Z Run the tests locally using vstest.console.exe
2018-05-04T21:52:00.6581612Z ========================================================
2018-05-04T21:52:00.6582115Z Test selector : Test assemblies
2018-05-04T21:52:00.6582269Z Test assemblies : **\release\**\*.dll
2018-05-04T21:52:00.6582399Z Test filter criteria : null
2018-05-04T21:52:00.6582534Z Search folder : D:\a\1\s
2018-05-04T21:52:00.6582658Z Run settings file : D:\a\1\s
2018-05-04T21:52:00.6582831Z Run in parallel : false
2018-05-04T21:52:00.6582954Z Run in isolation : false
2018-05-04T21:52:00.6584318Z Path to custom adapters : null
2018-05-04T21:52:00.6584486Z Other console options : null
2018-05-04T21:52:00.6584627Z Code coverage enabled : true
2018-05-04T21:52:00.6584910Z Rerun failed tests: false
2018-05-04T21:52:00.6585116Z VisualStudio version selected for test execution : latest
2018-05-04T21:52:01.5193379Z ========================================================
2018-05-04T21:52:01.5908190Z [command]"C:\Program Files (x86)\Microsoft Visual Studio\2017\Enterprise\Common7\IDE\CommonExtensions\Microsoft\TestWindow\vstest.console.exe" #D:\a\_temp\60315131-4fe5-11e8-9269-e9217bdc2ba0.txt
2018-05-04T21:52:06.6551385Z Microsoft (R) Test Execution Command Line Tool Version 15.6.2
2018-05-04T21:52:06.6552014Z Copyright (c) Microsoft Corporation. All rights reserved.
2018-05-04T21:52:06.6552213Z
2018-05-04T21:52:06.8162937Z vstest.console.exe
2018-05-04T21:52:06.8164594Z "D:\a\1\s\TestTesting\TestTesting\bin\Release\net461\Microsoft.VisualStudio.CodeCoverage.Shim.dll"
2018-05-04T21:52:06.8165659Z "D:\a\1\s\TestTesting\TestTesting\bin\Release\net461\TestTesting.dll"
2018-05-04T21:52:06.8166614Z "D:\a\1\s\TestTesting\TestTesting\bin\Release\net461\xunit.abstractions.dll"
2018-05-04T21:52:06.8167552Z "D:\a\1\s\TestTesting\TestTesting\bin\Release\net461\xunit.assert.dll"
2018-05-04T21:52:06.8168335Z "D:\a\1\s\TestTesting\TestTesting\bin\Release\net461\xunit.core.dll"
2018-05-04T21:52:06.8169121Z "D:\a\1\s\TestTesting\TestTesting\bin\Release\net461\xunit.execution.desktop.dll"
2018-05-04T21:52:06.8169879Z "D:\a\1\s\TestTesting\TestTesting\bin\Release\net461\xunit.runner.reporters.net452.dll"
2018-05-04T21:52:06.8170678Z "D:\a\1\s\TestTesting\TestTesting\bin\Release\net461\xunit.runner.utility.net452.dll"
2018-05-04T21:52:06.8171460Z "D:\a\1\s\TestTesting\TestTesting\bin\Release\net461\xunit.runner.visualstudio.testadapter.dll"
2018-05-04T21:52:06.8172221Z "D:\a\1\s\TestTesting\TestTesting\obj\Release\net461\TestTesting.dll"
2018-05-04T21:52:06.8172936Z /EnableCodeCoverage
2018-05-04T21:52:06.8173974Z /logger:"trx"
2018-05-04T21:52:06.8174130Z /TestAdapterPath:"D:\a\1\s"
2018-05-04T21:52:10.5221260Z Starting test execution, please wait...
2018-05-04T21:52:10.9446034Z Test run will use DLL(s) built for framework .NETFramework,Version=v4.6.1 and platform X86. Following DLL(s) do not match framework/platform settings.
2018-05-04T21:52:10.9446506Z xunit.assert.dll is built for Framework 1.1 and Platform AnyCPU.
2018-05-04T21:52:10.9446876Z xunit.core.dll is built for Framework 1.1 and Platform AnyCPU.
2018-05-04T21:52:10.9447201Z xunit.execution.desktop.dll is built for Framework 4.5.2 and Platform AnyCPU.
2018-05-04T21:52:10.9449027Z xunit.runner.reporters.net452.dll is built for Framework 4.5.2 and Platform AnyCPU.
2018-05-04T21:52:10.9449230Z xunit.runner.utility.net452.dll is built for Framework 4.5.2 and Platform AnyCPU.
2018-05-04T21:52:10.9449440Z xunit.runner.visualstudio.testadapter.dll is built for Framework 4.5.2 and Platform AnyCPU.
2018-05-04T21:52:10.9449631Z Go to http://go.microsoft.com/fwlink/?LinkID=236877&clcid=0x409 for more details on managing these settings.
2018-05-04T21:52:10.9449774Z
[[Uncopyable part here. What I could read spelled out "Microsoft Coverage Collection", followed by a copyright.
2018-05-04T21:52:31.9257584Z [xUnit.net 00:00:00.9144322] Discovering: Microsoft.VisualStudio.CodeCoverage.Shim
2018-05-04T21:52:32.0551765Z [xUnit.net 00:00:01.0599884] Discovered: Microsoft.VisualStudio.CodeCoverage.Shim
2018-05-04T21:52:32.4727660Z [xUnit.net 00:00:01.4776210] Discovering: TestTesting
2018-05-04T21:52:32.7346755Z [xUnit.net 00:00:01.7395514] Discovered: TestTesting
2018-05-04T21:52:32.7417690Z [xUnit.net 00:00:01.7468703] Starting: TestTesting
2018-05-04T21:52:33.2084443Z [xUnit.net 00:00:01.9469140] TestTesting.Tests.Skipped [SKIP]
2018-05-04T21:52:33.2085046Z [xUnit.net 00:00:01.9471459] Skipping
2018-05-04T21:52:33.2085275Z [xUnit.net 00:00:01.9542762] Finished: TestTesting
2018-05-04T21:52:33.2881381Z [xUnit.net 00:00:02.2931214] Discovering: xunit.abstractions
2018-05-04T21:52:33.3305957Z [xUnit.net 00:00:02.3357501] Discovered: xunit.abstractions
2018-05-04T21:52:34.0395297Z [xUnit.net 00:00:03.0443103] Discovering: xunit.runner.reporters.net452
2018-05-04T21:52:34.1043673Z [xUnit.net 00:00:03.1090663] Discovered: xunit.runner.reporters.net452
2018-05-04T21:52:34.4684448Z Passed TestTesting.Tests.Passing
2018-05-04T21:52:34.8614665Z Skipped TestTesting.Tests.Skipped
2018-05-04T21:52:34.8620656Z Standard Output Messages:
2018-05-04T21:52:34.8620971Z Skipping
2018-05-04T21:52:34.8625877Z Passed TestTesting.Tests.NoLongerFailing
2018-05-04T21:52:36.1389527Z [xUnit.net 00:00:05.1437439] Discovering: xunit.runner.utility.net452
2018-05-04T21:52:36.2616608Z [xUnit.net 00:00:05.2660797] Discovered: xunit.runner.utility.net452
[[Uncopyable part repeated here.]]
2018-05-04T21:52:38.5800594Z Attachments:
2018-05-04T21:52:38.5800836Z D:\a\1\s\TestResults\b0d720b3-9bb9-4318-bb6b-7bfa8e404a13\VssAdministrator_factoryvm-az384 2018-05-04 21_52_18.coverage
2018-05-04T21:52:38.5801023Z
2018-05-04T21:52:38.5801196Z Total tests: 3. Passed: 2. Failed: 0. Skipped: 1.
2018-05-04T21:52:38.5801368Z Test Run Successful.
2018-05-04T21:52:38.5809597Z Test execution time: 19.0295 Seconds
2018-05-04T21:52:38.6045339Z Results File: D:\a\1\s\TestResults\VssAdministrator_factoryvm-az384_2018-05-04_21_52_34.trx
2018-05-04T21:52:38.9702367Z ##[section]Async Command Start: Publish test results
2018-05-04T21:52:39.1587014Z Publishing test results to test run '3'
2018-05-04T21:52:39.1587207Z Test results remaining: 3. Test run id: 3
2018-05-04T21:52:39.6699238Z Published Test Run : https://wjtesttesting.visualstudio.com/MyFirstProject/_TestManagement/Runs#runId=3&_a=runCharts
2018-05-04T21:52:39.6700510Z ##[section]Async Command End: Publish test results
2018-05-04T21:52:39.6701395Z ##[section]Finishing: VsTest - testAssemblies
[EDIT 2]: Starian Chen asked for the logs with System.Debug set to true. They go over SO's size limit, so I'm linking to them below:
VSTS build log
VSTS test log
EDIT 3: When I specify the framework in "Other console options" of my VsTest build step:
/framework:"Framework461"
It seems to be unhappy with my version. It gives a list of supported frameworks, the highest of which is "Framework45."
Edit 4: I tried a net452 project, with "Framework45". In that case, the tests did run via Starian Chen's instructions here, just substituting "Framework45" for "netcoreapp2.0". However, there was still no code coverage report.
With .Net framework test project, you don’t need to specify framework and vstest.console.exe path in Visual Studio Test task.
I want to run NUnit on TeamCity and generate report with dotCover. But for some reasons I cannot get the coverage report for all my project assemblies.
TeamCity config as below
Teamcity: 6.5.3
NUnit: 2.5.10
.NET Runtime: Platform: x86
.NET Runtime: Version: v4.0
dotCover: bundled with TC not customized
I have checked that all *.dll *.xml *.pdb files are there in the directory as expected as below (reference DLLs are not listed)
MY.PROJECT.A.dll
MY.PROJECT.A.pdb
MY.PROJECT.A.xml
MY.PROJECT.B.dll
MY.PROJECT.B.pdb
MY.PROJECT.B.xml
MY.PROJECT.C.dll
MY.PROJECT.C.pdb
MY.PROJECT.C.xml
MY.PROJECT.Test.dll
MY.PROJECT.Test.pdb
MY.PROJECT.Test.xml
MY.PROJECT.Test.dll is executed with NUnit and this assembly is excluded in the coverage report by using *Test* filter. But only MY.PROJECT.A is in the dotCover coverage report whereas MY.PROJECT.B and MY.PROJECT.C are not included.
I checked the log but not error is found.
Any thoughts is much appreciated.
Finally, I figured out what is going on, and hope this answer is useful for those who are still struggling with this or similar problem.
Basically, dotCover only includes those assemblies are actually used (more precisely, those assemblies loaded by CLR) by the tests in the code coverage report.
In my case, only MY.PROJECT.A is used by tests, MY.PROJECT.B and MY.PROJECT.C are not used by tests due to external dependencies. and even with using MY.PROJECT.B directives in the tests, it does not count as CLR's lazy loading.
One dummy workaround to show 0% coverage report for these two assemblies is that either use anything in the assembly in the tests, or force load these assemblies by calling System.Reflection.Assembly.Load("MY.PROJECT.B")
Related Question:
Visual Studio Code Coverage Not Showing All Assemblies
What constitutes full code coverage ...
I have the following code in an Nunit test ...
string url = "";
url = #"http://localhost/ClientPortalDev/Account/LogOn";
ieStaticInstanceHelper = new IEStaticInstanceHelper();
ieStaticInstanceHelper.IE = new IE(url);
ieStaticInstanceHelper.IE.TextField(Find.ById("UserName")).TypeText("abc");
ieStaticInstanceHelper.IE.TextField(Find.ById("Password")).TypeText("defg");
ieStaticInstanceHelper.IE.Button(Find.ById("submit")).Click();
ieStaticInstanceHelper.IE.Close();
On right-clicking the project in Dev10(visual studio 10) and choosing [Test With][NUnit 2.5], this test code runs with no problems. I have TestDriven installed.
When opening the NUnit from C:\Program Files (x86)\NUnit 2.5.5\bin\net-2.0\nunit.exe" and then opening my test dll, the following text is reported in NUnit Errors and failures
... LoginAsWellKnownUserShouldSucceed:
System.Runtime.InteropServices.COMException : Error HRESULT E_FAIL has been returned from a call to a COM component.
As an Aside ... Right-Clicking the source cs file in Dev10 and choosing Run Test, ... works as well.
The above test is actually part of TechTalk.SpecFlow 1.3 step, I have NUnit 2.5.5.10112, installed, I have Watin 20.20 installed,
I have the following App.config for my test dll
the start angle brackets have been removed ... how do you get xml to show up in
configuration>
configSections>
sectionGroup name="NUnit">
section name="TestRunner" type="System.Configuration.NameValueSectionHandler"/>
/sectionGroup>
/configSections>
NUnit>
TestRunner>
add key="ApartmentState" value="STA" />
/TestRunner>
/NUnit>
appSettings>
add key="configCheck" value="12345" />
/appSettings>
/configuration>
Anyone hit this before ?
The NUnit test obviously runs in NUnit 2.5.5 of TestDriven but not when running NUnit 2.5.5 from outside of Dev10 and TestDriven ?
Run the test in NUnit as admin.
I was running Dev10 as admin so I could attatch and debug w3ww, which was why the TestDriven test was working.
As soon as I started running NUnit as admin the COM Server isue issue goess away.
What makes it more confusing is that running the hello-world exmaple against google from the Watin site works even though NUnit is not run as admin.
To use NUNit + Watin + against local web server, running NUnit as admin solves the com server exception issue.
Try using RequiresSTA attribute in your test code instead of the configuration file.
I am using MSTest, call Refresh to avoid cached data, this worked for me:
browser.Refresh();
browser.TextField(Find.ById("username")).TypeText("user");
browser.TextField(Find.ById("password")).TypeText("pass");
browser.Button(Find.ByName("SUBMIT")).Click();