vstest console output verbosity switch ignore by MSBuild - azure-devops

We are trying to reduce the console verbosity of our test runs in our Azure DevOps pipelines.
Due to the number of tests and complexity, we wrapped the VSTest.Console.exe run in MSBuild targets.
We configured our vstest.console.exe runs with the /logger:console;verbosity=minimal. When running the vstest command manually through command prompt this works fine, it only outputs the skipped tests and the end result summary, as intended.
When executed through MSBuild targets, even with the simplest target we confirmed, the output to the console is the full VSTest console output as if its in normal verbosity.
We are also setting the MSBuild verbosity to minimal, but this has no effect on the VSTest output.
Here is the target which we can reproduce it with
<Target Name="RunTestWorker" Outputs="$(VSTestConsoleExitCode)">
<Exec Command=""C:\Program Files (x86)\Microsoft Visual Studio\2019\Enterprise\MSBuild\..\Common7\IDE\CommonExtensions\Microsoft\TestWindow\vstest.console.exe" "ProjectA.SpecflowTest.dll" /Platform:x64 /logger:console;verbosity=q /logger:trx;LogFileName=TestTrxFile.trx"
ContinueOnError="True"
WorkingDirectory="$(TestResultsDirectory)..">
</Exec>
</Target>
Executed with this MSBuild call:
"C:\Program Files (x86)\Microsoft Visual Studio\2019\Enterprise\MSBuild\Current\Bin\amd64\MSBuild.exe" "TestBuild.targets" /verbosity:m /t:RunTestWorker
We tried additional MSBuild Exec options like ConsoleToMSBuild and EchoOff, but this didn't really change much.
Is this a known issue?

This example from Microsoft's docs wraps the arguments after /logger: in quotes:
vstest.console.exe Tests.dll /logger:"console;verbosity=minimal"
I tried this from the command line without the trx logger and it did reduce logging compared to normal.
So in your case:
/logger:"console;verbosity=quiet" /logger:"trx;LogFileName=TestTrxFile.trx"

Related

How to stop powershell script after failed dotnet command?

I see unreliable behaviour in the dotnet command when executed from a powershell script.
Executing in a powershell script:
dotnet build "$slnPath"
ignores any compilation errors and continues executing the script. I have to check $lastexitcode to see, if there are any errors.
On the other hand, the command:
dotnet test "$slnPath"
immediately terminates execution of the powershell script, if there are any failed unit tests.
Is that normal behavior? Do I need to write different error handling depending on the arguments of the dotnet command?
The dotnet test command launches the test runner console application specified for a project. The test runner executes the tests defined for a unit test framework (for example, MSTest, NUnit, or xUnit) and reports the success or failure of each test. If all tests are successful, the test runner returns 0 as an exit code; otherwise if any test fails, it returns 1.
But dotnet build building requires the project.assets.json file, which lists the dependencies of your application. The file is created when dotnet restore is executed. Without the assets file in place, the tooling can't resolve reference assemblies, which results in errors.
You can read what each command do(and may be found some -key to control they) at this adress tool_description

How to add a unit test in Azure DevOps pipeline

The build runs well but can not see the unit test result When added a build pipeline which used a classic edit or for my solution.
And the code coverage also blank. I have two projects and tests project in the solution. The unit test task returns the result as below:
A total of 14 test files matched the specified pattern.
No test is available in d:\a\1\s\src.net...
Make sure that test discoverer & executors are registered and platform & framework version settings are appropriate and try again.
Results File: d:\a\_temp\TestResults\VssAdministrator_fv-az686_2019-12-26_06_10_28.trx
Attachments:
d:\a\_temp\TestResults\d301a26b-d99f-4b4e-bbe8-c95e588ee1c5\VssAdministrator_fv-az686 2019-12-26 06_10_20.coverage
Vstest.console.exe exited with code 0.
**************** Completed test execution *********************
Test results files: d:\a\_temp\TestResults\VssAdministrator_fv-az686_2019-12-26_06_10_28.trx
No Result Found to Publish 'd:\a\_temp\TestResults\VssAdministrator_fv-az686_2019-12-26_06_10_28.trx'.
Created test run: 1006840
Publishing test results: 0
Publishing test results to test run '1006840'.
TestResults To Publish 0, Test run id:1006840
Published test results: 0
Publishing Attachments: 2
Completed TestExecution Model...
And also I can not see anymore from the Tests windows.
The test YAML script as below:
steps:
- task: VSTest#2
displayName: 'Test Assemblies'
inputs:
testAssemblyVer2: |
**\$(BuildConfiguration)\*test*.dll
!**\obj\**
codeCoverageEnabled: true
platform: '$(BuildPlatform)'
configuration: '$(BuildConfiguration)'
There no any problem on your YAML, the same definition is all work for me.
A total of 14 test files matched the specified pattern.
No test is available in d:\a\1\s\src.net...
Based on these error message, the test files has been identified out from the folder which means the dlls has exists now. But it still say there's no test available, this is more relative with vstest.console.exe did not identify the test.
For example, if your test project type is NUnit, you could use one extension names NUnit test adapter to support this test type running in visual studio.
But, when the project is published into VSTS, this extension could not be published also, and this extension does not exists in VSTS hosted agent. Without this extension support, the vstest.console.exe could not identify the Nunit test, then it will prompt the message like No test is available in xxxxxxx. See this as refer to.
But, you can installed nuget packages to replace the role of that VS extension.
If it's type is Nunit test, you must ensure add the reference on packages NUnit and NUnit3TestAdapter in your csproj file:
<PackageReference Include="NUnit">
<Version>3.12.0</Version>
</PackageReference>
<PackageReference Include="NUnit3TestAdapter">
<Version>3.15.1</Version>
</PackageReference>
Similar method tp xUnit test.

GoCD Nuget issue

I have a project and my agent does have nuget installed - Not sue what I am doign wwong
12:01:14.195 [go] Start to build DemoApp/20/second/1/secondJob on vivians-mbp-2.delta.rl.delta.com [/Users/vivianaranha/Library/Application Support/Go Agent]
12:01:14.195 [go] Current job status: passed.
12:01:14.195 [go] Start to execute task: <exec command="nuget restore packages.config -PackagesDirectory ..\..\packages" workingdir="EMApp\EMApp\EMApp.iOS" />.
12:01:14.199 Error happened while attempting to execute 'nuget restore packages.config -PackagesDirectory ..\..\packages'.
Please make sure [nuget restore packages.config -PackagesDirectory ..\..\packages] can be executed on this agent.
12:01:14.199 [Debug Information] Environment variable PATH: /usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/Library/Frameworks/Mono.framework/Versions/Current/Commands:/usr/local/bin
12:01:14.206 [go] Current job status: failed.
Have you tried setting the build details to be more verbose? Also, from when i do it I have the build use the nuget.exe saved in the source control, not on the build agent
You are making a common mistake in GoCD expecting exec task to be a shell execution.
By that build log, I can tell that your configuration in xml is
<exec command="nuget restore packages.config -PackagesDirectory ..\..\packages" workingdir="EMApp\EMApp\EMApp.iOS">
</exec>
That means GoCD would be trying to execute a file whose path is nuget restore packages.config -PackagesDirectory ..\..\packages, which does not exist.
GoCD exec task requires you to explicitly separate process executable and all arguments. So the above nuget example would have to be configured like so:
<exec command="nuget" workingdir="EMApp\EMApp\EMApp.iOS">
<arg>restore</arg>
<arg>packages.config</arg>
<arg>-PackagesDirectory</arg>
<arg>..\..\packages</arg>
</exec>
More tips on task setup:
If nuget.exe is not in the PATH, you'll need to configure full file path e.g. command="C:\nuget\nuget.exe"
Here is a nice blog post on exec tasks
If you don't like setting up process and arguments separately like this, try script executor plugin which will start a shell.
Edit 1
On mac you would need to run nuget with mono. So the process executable is mono:
<exec command="mono" workingdir="EMApp\EMApp\EMApp.iOS">
<arg>/path/to/nuget</arg>
<arg>restore</arg>
<arg>packages.config</arg>
<arg>-PackagesDirectory</arg>
<arg>..\..\packages</arg>
</exec>

Gallio with NCover shows 0% code coverage in Sonar UI

I am using sonar-runner to run tests and code coverage over my C# code with the help of gallio plugin. The tests are running fine, but I am not able to see any code coverage on the sonar web UI.
My Sonar settings are as follows:
sonar-project.properties
mentioning only relevant bits
sonar.gallio.coverage.tool = NCover
sonar.NCover.installDirectory = C:/Program Files/NCover
sonar.donet.visualstudio.testProjectPattern = .Test
sonar.dotnet.buildConfigurations = "Release|x86"
Any idea what coule be missing??
sonar.projectKey=XXX:XXX
sonar.projectVersion=trunk
sonar.projectName=XXX
sources=.
sonar.language=cs
sonar.dotnet.visualstudio.solution.file=Project.sln
sonar.dotnet.excludeGeneratedCode=false
sonar.dotnet.4.0.sdk.directory=C:/WIndows/Microsoft.NET/Framework/v4.0.30319
sonar.dotnet.version=4.0
# Gallio
sonar.gallio.mode=
sonar.gallio.coverage.tool=NCover
sonar.gallio.runner=IsolatedAppDomain
sonar.NCover.installDirectory=c:/Program Files/NCover
sonar.gallio.installDirectory=C:/Program Files/Gallio
sonar.dotnet.test.assemblies=$(SolutionDir)/../**/bin/**/*.Tests.Unit.dll
# FXCop
sonar.fxcop.mode=
#StyleCop
sonar.stylecop.mode=
#NDeps
sonar.ndeps.mode=skip
sonar-runner.properties
You said
sonar.dotnet.buildConfigurations = "Release|x86"
If that's true, your build likely isn't generating .pdb files, which are needed to figure out the mapping between the binaries and your source files.
Does it work if you try it with a Debug build?
I was seeing this same behavior with NCover in Sonar. I found that Sonar was generating invalid arguments for Gallio's NCover runner.
Try piping the output from Sonar's runner into a text file so that you can examine the arguments more easily (on the command line, you can just type sonar-runner > output.txt to do this).
You will likely see a line like this in your output:
INFO .u.c.CommandExecutor - Executing command: C:\Program Files\Gallio\bin\Gallio.Echo.exe /r:Local /report-directory:E:\Reports\.sonar /report-name-format:gallio-report /report-type:Xml E:\Projects\UnitTests\bin\Release\UnitTests.dll /runner-property:NCoverCoverageFile=E:\Reports\.sonar\coverage-report.xml /runner-property:NCoverArguments=//ias MyFirstAssembly;MySecondtAssembly;MyThirdAssembly
If you attempt to execute this manually via Gallio on the command line, you will get an error:
Cannot find file 'MyFirstAssembly;MySecondtAssembly;MyThirdAssembly'
If you edit this list manually down to a single entry such as MyFirstAssembly*, everything will work as expected.
This seems to indicate that Sonar is generating invalid command line arguments for Gallio. As much as I love NCover, the easiest solution was to use OpenCover instead.

TeamCity and running NUnit tests

In TeamCity, i need to state exact locations of assemblies that contain NUnit tests to be executed.
Is there an option to state a .SLN file so it will look up these test projects dynamically?
You can use wildcard expressions in the Run tests from box:
Source\\**\bin\\**\*Tests.dll
The above would run tests from any assembly under any bin folder under the Source folder which contains 'Tests' at the end of the assembly name.
Depending on whether you're using MSBuild or NAnt, you can add an entry to your build script like this:
<ItemGroup>
<TestAssemblies Include="tests\\test*.dll"/>
<TestAssemblies Include="tests.lib\\test*.dll"/>
</ItemGroup>
<Target Name="runTests">
<Exec Command="$(teamcity_dotnet_nunitlauncher) v2.0 x86 NUnit-2.5.0 %(TestAssemblies)" />
</Target>
In the example above, the two TestAssemblies lines point to your assemblies.
You can read more about this here: http://blogs.jetbrains.com/teamcity/2008/09/24/using-teamcity-nunit-launcher/