I am trying to run xUnit tests using the Azure CI pipeline, despite the unit tests running successfully in Visual Studio 2017, but on Azure DevOps no results. Below are screenshots for the vs solution and build and test tasks in the Azure pipeline: '
Test project
The solution in the visual studio
The solution in the pipeline
Build task
Test task
Test task results
What is the problem with my setup?
Please change your test files and replace this line:
**\bin\$(BuildConfiguration)\**\*test*.dll
with this:
**\bin\$(BuildConfiguration)\**\UnitTesting.dll
or this:
**\bin\$(BuildConfiguration)\UnitTesting.dll
Please try to apply this pattern to your folder structure in you bin folder.
Related
I have an .Net Core application along with unit test cases. For that I have configured the Build pipeline in Azure DevOps. In that pipeline I have integrated SonarQube tasks (prepare analysis, run code analysis, and publish quality gate results).
I can see the report in SonarQube server after successful run. But in that report, I didn’t see the Code Coverage Results and Unit test results. Even though I used Cobertura for unit tests.
Your pipeline seems to contain the right steps, so there can be two issues:
1. Code coverage file is not generated (correctly)
The easiest way to validate if the code coverage file is generated correctly, is by publishing it as an artifact. Now check what format the output file is. If there is no output file, please check if you did include /p:CollectCoverage=true --logger trx to the test command. If you are running the build pipeline on Linux, you should also add /p:CoverletOutputFormat=opencover and install the coverlet.collector NuGet package in the .NET Test Project.
2. Code coverage file is not sent to Sonarqube
If you configured step 1 correctly, it is still possible that the generated files are not sent to Sonarqube. The best way to see what is going wrong, is by checking the build logs of the Run Code Analysis and Publish Quality Gate Results steps.
The most common issue, is that Sonarscanner is checking the wrong directory. In the prepare step, please specify where the files are located, like:
sonar.cs.opencover.reportsPaths=$(Build.SourcesDirectory)/**/coverage.opencover.xml
sonar.cs.vstest.reportsPaths=$(Agent.TempDirectory)/*.trx
Build pipeline generates code coverage files would it be coverlet or opencover, the problem is that Azure Devops pipeline creates the reports outside the working directory on the build agent, it uses the _temp folder inside _work whereas soanrquube searches inside the working directory. I am facing this issue with coverlet as well as Visual studio coverage tool. You can read the following 2 threads
https://github.com/coverlet-coverage/coverlet/issues/1399
https://github.com/microsoft/azure-pipelines-tasks/issues/11536.
I posted an issue on snoarqube community site but did not get any response so far. The solution I think is either make the test step in the build pipeline change the directory or configure somewher sonarqube to search the agent build directory _temp which is above the working directory
This seems a pretty straightforward thing to do, but I cannot find relevant information. In Visual Studio is very easy to calculate code metrics for all projects and I would like to do the same during a build pipeline in Azure DevOps.
Has anyone done something like this?
In azure devops, you can review code coverage results. The results can be viewed and downloaded on the Code coverage tab.
Publish Code Coverage Results publishes code coverage results to
Azure Pipelines or TFS, which were produced by a build in Cobertura
or JaCoCo format.
Built-in tasks such as Visual Studio Test, .NET Core, Ant, Maven,
Gulp, Grunt, and Gradle provide the option to publish code coverage
data to the pipeline.
Here are some documents you can refer to:
Review code coverage results
Azure DevOps and the Code Coverage
In addition, you can get code analysis through SonarCloud integrated with Azure devops. SonarCloud is a cloud-based code quality and security service.
Here is the lab you can follow.
You can generate code metrics by adding the Microsoft.CodeAnalysis.Metrics nuget and building with the parameter that target the "Metrics" build, this will store the results in an xml file.
msbuild /t:Metrics
If you're building a solution with multiple projects, you have to add the nuget to every project otherwise the build will fail because the Metrics target won't be present for the other projects.
The build will produce one xml file per project.
You can also use the metrics command line tool from the roslyn analyzer repository.
See https://learn.microsoft.com/en-us/visualstudio/code-quality/how-to-generate-code-metrics-data?view=vs-2022
The image above shows my Hello World project structure. I am trying to build the Setup-HelloWorld-x86.vdproj in Azure Pipeline in Self-Hosted Windows Agent (which is nothing but my local machine) by using the command line script task like shown in below image.
But I get the following error.
Kindly help me to build this Setup-HelloWorld-x86.vdproj
If you want to run the project via cmd devenv, you can refer to this doc.
As a workaround, you can install the extension Build VS Installer and use the task DutchWorkz - Build VS Installer(s) to build Visual Studio Installer Project in Azure Pipelines.
You can specify to build .sln or .vdproj to generate .msi file(s) in Task-mode option.
Update1
Thanks Tharunavignesh J for sharing.
The solution was to run the azure agent under proper account. Earlier my azure agent was running under some Network Service account, then I change it to the local account, then this cmd worked properly.
This other stack overflow link helped me get my visual studio installer project compiled with Azure DevOps pipelines. It is using yaml configuration instead of classic, but same steps / commands apply to both:
YAML Script for building Visual Studio Installer Projects using Azure DevOps
Basically the steps are:
Setup your variables to point to the tools
Download Nuget packages for your solution
Disable Out Of Process Builds command executed
Script task with the command line arguments to build your solution
We are trying to Run tests in parallel using the Visual Studio Test task based on below link but we noticed "Run Unit tests" step failing due to packages directory was not found. Trying to contact Azure DevOps team to get some help here.
https://learn.microsoft.com/en-us/azure/devops/pipelines/test/parallel-testing-vstest?view=azure-devops
We tried to publish the test files as artifacts from the first job. And download the test artifacts to the second job. But we found the test files were located separately and there were many test data files needed to be published too. It was hard to publish all the test files to azure devops sever. And the vstest task failed with either assemblies not found error or some testcases just failed.
We also tried configuring a single job running in parallel with both build tasks and vstest task. It turned out the publish build artifacts task didnot work properly in parallel jobs.
After many times try and working together with #NAGAJU. The workaround we found is to configure two agent jobs to separate the publish artifacts task and Vstest task.
The first agent job with build tasks and publish artifacts tasks runs in an single agent. The second job with build tasks and Vstest task to run the tests in multiple agents in parallel.
I have a staged release pipeline that deploys to Development and then to Staging.
I want my Integration Test project to run during release pipeline only before deployment to staging. How do I do that? is there a specific test task that I will be able to hook into before the staging deployment? like a pre-deploy task or something?
In order to run Integration Tests in your release pipeline. You should include your Test projects or test assembly dll files in the artifacts published in your build pipeline. So that your Integration test projects are accessible to the test tasks in your release pipeline.
To Include your test files in the artifacts. You can add a second publish build artifacts task in your Build Pipeline. Specify Path to publish to the location of your test files. Fore below example.
I add a another publish build artifacts task to publish all the test files that needed to run the tests.
Then I specify the Path to publish field to folder that contains all the test files.
After the build pipeline completed. You will see the Test folder being added to the build artifacts and published to the azure devops pipeline server.
Now you can run your tests in your release pipeline by adding the VsTest task or other test tasks in your release pipeline. The release pipeline will download your artifacts to
folder $(System.DefaultWorkingDirectory)(eg. C:\agent_work\r1\a).
For above example you will find the test files in folder $(System.DefaultWorkingDirectory)\artifact_alias\Test and the artifacts to be deployed in $(System.DefaultWorkingDirectory)\artifact_alias\drop.
Then you can either create a new stage with vstest task before deployment stage as #Kontekst mentioned, or just add the vstest task on the top of your deployment stage before your deployment task.
In above screenshot I add the Vstest task before the Azure deployment task and point the Search folder to the Test folder where the test files reside.
Update:
You can find the artifacts-alias in the field shown in below screenshot.
The main idea is to publish the test files in the artifacts in build pipeline and then add the test tasks in the release pipeline to run the tests.
I suppose you are using C# unit tests.
Add new stage "Integration tests" right before stage "Deployment to Staging".
Add task installing VSTest Platform. It will be cached in agent folder.
https://learn.microsoft.com/en-us/azure/devops/pipelines/tasks/tool/vstest-platform-tool-installer?view=azure-devops
Add to your tests projects proper unit tests adapter NuGet package, so that VSTest Platform will be able to find unit tests in dll assemblies, for example for NUnit tests it will be "NUnit 3 Test Adapter".
Add VSTest task specyfing:
a) Test files -> for example "***IntegrationTests.dll"
b) Test platform version -> "Installed by Tools Installer"
https://learn.microsoft.com/en-us/azure/devops/pipelines/tasks/test/vstest?view=azure-devops
All test results returned by VSTest task will be integrated with Azure DevOps services.
I Love both answers by Levi Lu-MSFT and Kontekst!
My team used to do it like that and it got the job done.
However if you'd like more "fancier" reporting and if like me, you were looking for something out of the box that Microsoft was offering specifically for integration tests or automated tests then Azure DevOps' Test Plan feature might be what you're looking for.
Test Plans really seem to be catering for actual web automated tests but you can basically run any tests that you can fit in a release pipeline.
It integrates well with Visual Studio Test Manager.
Here's the link to the documentation: https://learn.microsoft.com/en-us/azure/devops/test/run-automated-tests-from-test-hub?view=azure-devops
And here's some screenshots in case the link fails in the future: