I have a Python project hosted on GitHub, which I test using Azure Pipelines (here is a link to the configuration).
I am running unit tests with pytest and get coverage results, which I upload in my pipeline. I get an overall code coverage percentage in the pipeline execution summary (example), but I don't see a "Code Coverage" tab that contains the details of which lines are covered. The build artifacts contain the HTML code coverage report, which contains this information, but it'd be nicer to see it in the pipeline run page itself.
Here is the invocation of pytest:
pytest tests --doctest-modules --junitxml=junit/test-results.xml
--cov=callgraph --cov-report=xml --cov-report=html
What am I missing?
that doesnt work like that, unfortunately. at least that was the case 6 months ago.
https://github.com/Microsoft/azure-pipelines-tasks/issues/7703
Related
I have an .Net Core application along with unit test cases. For that I have configured the Build pipeline in Azure DevOps. In that pipeline I have integrated SonarQube tasks (prepare analysis, run code analysis, and publish quality gate results).
I can see the report in SonarQube server after successful run. But in that report, I didn’t see the Code Coverage Results and Unit test results. Even though I used Cobertura for unit tests.
Your pipeline seems to contain the right steps, so there can be two issues:
1. Code coverage file is not generated (correctly)
The easiest way to validate if the code coverage file is generated correctly, is by publishing it as an artifact. Now check what format the output file is. If there is no output file, please check if you did include /p:CollectCoverage=true --logger trx to the test command. If you are running the build pipeline on Linux, you should also add /p:CoverletOutputFormat=opencover and install the coverlet.collector NuGet package in the .NET Test Project.
2. Code coverage file is not sent to Sonarqube
If you configured step 1 correctly, it is still possible that the generated files are not sent to Sonarqube. The best way to see what is going wrong, is by checking the build logs of the Run Code Analysis and Publish Quality Gate Results steps.
The most common issue, is that Sonarscanner is checking the wrong directory. In the prepare step, please specify where the files are located, like:
sonar.cs.opencover.reportsPaths=$(Build.SourcesDirectory)/**/coverage.opencover.xml
sonar.cs.vstest.reportsPaths=$(Agent.TempDirectory)/*.trx
Build pipeline generates code coverage files would it be coverlet or opencover, the problem is that Azure Devops pipeline creates the reports outside the working directory on the build agent, it uses the _temp folder inside _work whereas soanrquube searches inside the working directory. I am facing this issue with coverlet as well as Visual studio coverage tool. You can read the following 2 threads
https://github.com/coverlet-coverage/coverlet/issues/1399
https://github.com/microsoft/azure-pipelines-tasks/issues/11536.
I posted an issue on snoarqube community site but did not get any response so far. The solution I think is either make the test step in the build pipeline change the directory or configure somewher sonarqube to search the agent build directory _temp which is above the working directory
I have a pipeline that runs on pull requests and executes tests , it also produces an xml execution report (it is a node project and tests run via jest) , and coverage report that I use for sonarcloud. However , the developers want to be able to see the tests results on each pr and see what tests failed in particular, so my question is whether it is possible to somehow export and display those results in github or sonarcloud
Please refer to the following two methods to get the test cases metric in sonarcloud:
You can add sonar.testExecutionReportPaths in Additional Properties option in Prepare Analysis Configuration task. About sonar.coverageReportPaths
For SonarQube to pick up and analyse the content of the report file, you need to inform the sonar scanner. About the location of the XML file. To do so, edit the sonarqube.properties file. Please refer to this.
coverage reporting:
sonar.testExecutionReportPaths=reports/xxx/xxx.xml
In addition, you can also refer to this similar case.
I have a YAML task that runs tests on .NET solutions in Azure pipeline.
It is meant to run after a build step and execute Unit-tests on that assembly.
The output is a simple XML file with test results that needs to be shown after each build run in the summary tab.
How can I make it recognizable by azure?
For example: MsBuild step is recognized and shown in the summary menu as Build Artifacts and have the option to download them from Azure UI. How can I make Azure recognize my task and show it's artifacts and info too? How many tests ran and info in the title and when I click drop it will show artifacts
Summary menu after build run on Azure DevOps server
You can use ##vso[task.uploadsummary]local file path to upload summary to the build summary. However you may need to write the contents of xml file into md file. please check here to learn more upload tasks.
- powershell: '##vso[task.uploadsummary]path to test result'
However test results isnot intended to be displayed on the summary page.
You must have noticed that there is a Test tab beside Summary tab. Usually the test results will be automatically published by Vstest tasks and the test results will be displayed in Test tab.
You can also publish your test results using task PublishTestResults as mentioned by #4c74356b41. You test results will then be displayed in Test tab.
If you are asking how to make your test results visible in the build summary, there is task meant for that specifically.
- task: PublishTestResults#2
inputs:
testResultsFormat: 'JUnit' # Options: JUnit, NUnit, VSTest, xUnit, cTest
testResultsFiles: '**/TEST-*.xml'
your test results should be compatible with one of the supported versions
I have a azure pipeline build that runs the following command.
pytest tests --junitxml=junit/test-results.xml --cov --cov-report=xml --cov-report=html
I know this generates the correct coverage report as azure produces it as an artifact which i can download. The problem is that azure seems to have an issue with the coverage report. i get the following warnings:
##[warning]Invalid results file. Make sure the result format of the file '/home/vsts/work/1/s/test-cov.xml' matches 'JUnit' test results format.
##[warning]No code coverage results were found to publish.
In the code coverage tab after the pipeline has finished the html output is there but is is incorrectly displayed.
How do i fix this?
I have a requirement of settings code coverage percentage for my unit test cases running in VSTS through the build.yml file.
Right now the build is passing for any code coverage percentage that's coming. I want to restrict this and fail the build if the percentage is not 70%.
Can you help me with this?
You need to publish the test results (and code coverage is in that) and then use another task or release gate to check those results. You can do this with SonarQube Quality gate (needs a SonarQube server) or you can try the marketplace.
For example this task: https://marketplace.visualstudio.com/items?itemName=mspremier.BuildQualityChecks
You can also check it yourself (using PowerShell for example) via the API: https://learn.microsoft.com/en-us/rest/api/vsts/test/code%20coverage?view=vsts-rest-5.0