I have a dot net core version: '3.0.100', building on 'ubuntu-latest' and am trying to get code coverage reports.
I have been using Coverlet to generate Cobertura files which can then use the PublishCodeCoverageResults#1 to publish to the Devops pipelines code coverage viewer.
I have used one stage name build for building the project and generating artifacts and published them.
In another stage named unit tests i will be downloading the artifacts and using the artifacts of previous build.
But when I run unit tests by specifying the working directory where these artifacts were downloaded
Everything went fine up to this point,but when it is generating the code coverage reports in the unit test job it is generating NaN% value.
enter image description here
I have tried all possible ways playing with working path directory and restoring the nuget packages.But No Luck None worked for me. I have even searched for this issue,I found one but that didn't worked me
When i tried to run the build and unit tests in one stage itself it successfully generated the code coverage reports,but placing that in another stage and downloading the previous build artifacts made me run into the issue.
How would I achieve this ?
Related
I have an .Net Core application along with unit test cases. For that I have configured the Build pipeline in Azure DevOps. In that pipeline I have integrated SonarQube tasks (prepare analysis, run code analysis, and publish quality gate results).
I can see the report in SonarQube server after successful run. But in that report, I didn’t see the Code Coverage Results and Unit test results. Even though I used Cobertura for unit tests.
Your pipeline seems to contain the right steps, so there can be two issues:
1. Code coverage file is not generated (correctly)
The easiest way to validate if the code coverage file is generated correctly, is by publishing it as an artifact. Now check what format the output file is. If there is no output file, please check if you did include /p:CollectCoverage=true --logger trx to the test command. If you are running the build pipeline on Linux, you should also add /p:CoverletOutputFormat=opencover and install the coverlet.collector NuGet package in the .NET Test Project.
2. Code coverage file is not sent to Sonarqube
If you configured step 1 correctly, it is still possible that the generated files are not sent to Sonarqube. The best way to see what is going wrong, is by checking the build logs of the Run Code Analysis and Publish Quality Gate Results steps.
The most common issue, is that Sonarscanner is checking the wrong directory. In the prepare step, please specify where the files are located, like:
sonar.cs.opencover.reportsPaths=$(Build.SourcesDirectory)/**/coverage.opencover.xml
sonar.cs.vstest.reportsPaths=$(Agent.TempDirectory)/*.trx
Build pipeline generates code coverage files would it be coverlet or opencover, the problem is that Azure Devops pipeline creates the reports outside the working directory on the build agent, it uses the _temp folder inside _work whereas soanrquube searches inside the working directory. I am facing this issue with coverlet as well as Visual studio coverage tool. You can read the following 2 threads
https://github.com/coverlet-coverage/coverlet/issues/1399
https://github.com/microsoft/azure-pipelines-tasks/issues/11536.
I posted an issue on snoarqube community site but did not get any response so far. The solution I think is either make the test step in the build pipeline change the directory or configure somewher sonarqube to search the agent build directory _temp which is above the working directory
I have a test-complete project which is run via azure-devops, setup for CI. This project is linked to cucumber for the steps, as well as pushing results back up for reporting (we already have this setup and working for selenium, visual studio). The problem I am having is that I have multiple test runs in cucumber that are run using this single test-complete project.
When running manually there is no problem because we can manually switch between what test runs on cucumber our project currently pushes too. But with azure-devops that is not an option.
We currently create a .trx report that we use to bring results back into azure-devops but this report is not compatible with cucumber. I ran /ListLoggers to show all loggers available but none of these seem useable for pushing to cucumber using hiptest-publisher:
/ListLoggers
The following Test Logger Add-Ins are available:
Microsoft.TestPlatform.Extensions.BlameDataCollector.BlameLogger
Uri: logger://Microsoft/TestPlatform/Extensions/Blame/v1
FriendlyName: Blame
Microsoft.VisualStudio.TestPlatform.Extensions.HtmlLogger.HtmlLogger
Uri: logger://Microsoft/TestPlatform/HtmlLogger/v1
FriendlyName: Html
Microsoft.VisualStudio.TestPlatform.Extensions.TrxLogger.TrxLogger
Uri: logger://Microsoft/TestPlatform/TrxLogger/v1
FriendlyName: Trx
This is the part of the release pipeline that runs the tests:
I filter out tests to-be-run like so FullyQualifiedName~SmokeTest. So the question remains, how can I send results back up to cucumber, to specific test runs, from the single project via azure-devops without changing the test-run linked to in the test-complete project manually?
I am trying to publish code coverage from my create-react-app in Azure devOps even though the coverage is generated when I run test locally I get the following error In azure pipeline when trying to publish the coverage "##[error]No code coverage results were found to publish." find the screenshots of my pipeline config and the error. have being on this for two days now a working solution would greatly be appreciated.
files in the coverage folder:
Coverage
lcov-report(folder)
clover.xml
cobertura-coverage.xml
coverage-final.json
lcov.info
https://drive.google.com/file/d/1J6tRN0RfYWtfc-cJ64Itxwnmlqe8c-BH/view?usp=sharing, https://drive.google.com/file/d/1g-IKWETw2tuXexbk36VPOVpYRD6bCVYf/view?usp=sharing, https://drive.google.com/file/d/1sY6qBc1xEroCfOF3yt6ORbAXPPkr04JE/view?usp=sharing
We are trying to Run tests in parallel using the Visual Studio Test task based on below link but we noticed "Run Unit tests" step failing due to packages directory was not found. Trying to contact Azure DevOps team to get some help here.
https://learn.microsoft.com/en-us/azure/devops/pipelines/test/parallel-testing-vstest?view=azure-devops
We tried to publish the test files as artifacts from the first job. And download the test artifacts to the second job. But we found the test files were located separately and there were many test data files needed to be published too. It was hard to publish all the test files to azure devops sever. And the vstest task failed with either assemblies not found error or some testcases just failed.
We also tried configuring a single job running in parallel with both build tasks and vstest task. It turned out the publish build artifacts task didnot work properly in parallel jobs.
After many times try and working together with #NAGAJU. The workaround we found is to configure two agent jobs to separate the publish artifacts task and Vstest task.
The first agent job with build tasks and publish artifacts tasks runs in an single agent. The second job with build tasks and Vstest task to run the tests in multiple agents in parallel.
I have a staged release pipeline that deploys to Development and then to Staging.
I want my Integration Test project to run during release pipeline only before deployment to staging. How do I do that? is there a specific test task that I will be able to hook into before the staging deployment? like a pre-deploy task or something?
In order to run Integration Tests in your release pipeline. You should include your Test projects or test assembly dll files in the artifacts published in your build pipeline. So that your Integration test projects are accessible to the test tasks in your release pipeline.
To Include your test files in the artifacts. You can add a second publish build artifacts task in your Build Pipeline. Specify Path to publish to the location of your test files. Fore below example.
I add a another publish build artifacts task to publish all the test files that needed to run the tests.
Then I specify the Path to publish field to folder that contains all the test files.
After the build pipeline completed. You will see the Test folder being added to the build artifacts and published to the azure devops pipeline server.
Now you can run your tests in your release pipeline by adding the VsTest task or other test tasks in your release pipeline. The release pipeline will download your artifacts to
folder $(System.DefaultWorkingDirectory)(eg. C:\agent_work\r1\a).
For above example you will find the test files in folder $(System.DefaultWorkingDirectory)\artifact_alias\Test and the artifacts to be deployed in $(System.DefaultWorkingDirectory)\artifact_alias\drop.
Then you can either create a new stage with vstest task before deployment stage as #Kontekst mentioned, or just add the vstest task on the top of your deployment stage before your deployment task.
In above screenshot I add the Vstest task before the Azure deployment task and point the Search folder to the Test folder where the test files reside.
Update:
You can find the artifacts-alias in the field shown in below screenshot.
The main idea is to publish the test files in the artifacts in build pipeline and then add the test tasks in the release pipeline to run the tests.
I suppose you are using C# unit tests.
Add new stage "Integration tests" right before stage "Deployment to Staging".
Add task installing VSTest Platform. It will be cached in agent folder.
https://learn.microsoft.com/en-us/azure/devops/pipelines/tasks/tool/vstest-platform-tool-installer?view=azure-devops
Add to your tests projects proper unit tests adapter NuGet package, so that VSTest Platform will be able to find unit tests in dll assemblies, for example for NUnit tests it will be "NUnit 3 Test Adapter".
Add VSTest task specyfing:
a) Test files -> for example "***IntegrationTests.dll"
b) Test platform version -> "Installed by Tools Installer"
https://learn.microsoft.com/en-us/azure/devops/pipelines/tasks/test/vstest?view=azure-devops
All test results returned by VSTest task will be integrated with Azure DevOps services.
I Love both answers by Levi Lu-MSFT and Kontekst!
My team used to do it like that and it got the job done.
However if you'd like more "fancier" reporting and if like me, you were looking for something out of the box that Microsoft was offering specifically for integration tests or automated tests then Azure DevOps' Test Plan feature might be what you're looking for.
Test Plans really seem to be catering for actual web automated tests but you can basically run any tests that you can fit in a release pipeline.
It integrates well with Visual Studio Test Manager.
Here's the link to the documentation: https://learn.microsoft.com/en-us/azure/devops/test/run-automated-tests-from-test-hub?view=azure-devops
And here's some screenshots in case the link fails in the future: