I have a requirement of settings code coverage percentage for my unit test cases running in VSTS through the build.yml file.
Right now the build is passing for any code coverage percentage that's coming. I want to restrict this and fail the build if the percentage is not 70%.
Can you help me with this?
You need to publish the test results (and code coverage is in that) and then use another task or release gate to check those results. You can do this with SonarQube Quality gate (needs a SonarQube server) or you can try the marketplace.
For example this task: https://marketplace.visualstudio.com/items?itemName=mspremier.BuildQualityChecks
You can also check it yourself (using PowerShell for example) via the API: https://learn.microsoft.com/en-us/rest/api/vsts/test/code%20coverage?view=vsts-rest-5.0
Related
I have an .Net Core application along with unit test cases. For that I have configured the Build pipeline in Azure DevOps. In that pipeline I have integrated SonarQube tasks (prepare analysis, run code analysis, and publish quality gate results).
I can see the report in SonarQube server after successful run. But in that report, I didn’t see the Code Coverage Results and Unit test results. Even though I used Cobertura for unit tests.
Your pipeline seems to contain the right steps, so there can be two issues:
1. Code coverage file is not generated (correctly)
The easiest way to validate if the code coverage file is generated correctly, is by publishing it as an artifact. Now check what format the output file is. If there is no output file, please check if you did include /p:CollectCoverage=true --logger trx to the test command. If you are running the build pipeline on Linux, you should also add /p:CoverletOutputFormat=opencover and install the coverlet.collector NuGet package in the .NET Test Project.
2. Code coverage file is not sent to Sonarqube
If you configured step 1 correctly, it is still possible that the generated files are not sent to Sonarqube. The best way to see what is going wrong, is by checking the build logs of the Run Code Analysis and Publish Quality Gate Results steps.
The most common issue, is that Sonarscanner is checking the wrong directory. In the prepare step, please specify where the files are located, like:
sonar.cs.opencover.reportsPaths=$(Build.SourcesDirectory)/**/coverage.opencover.xml
sonar.cs.vstest.reportsPaths=$(Agent.TempDirectory)/*.trx
Build pipeline generates code coverage files would it be coverlet or opencover, the problem is that Azure Devops pipeline creates the reports outside the working directory on the build agent, it uses the _temp folder inside _work whereas soanrquube searches inside the working directory. I am facing this issue with coverlet as well as Visual studio coverage tool. You can read the following 2 threads
https://github.com/coverlet-coverage/coverlet/issues/1399
https://github.com/microsoft/azure-pipelines-tasks/issues/11536.
I posted an issue on snoarqube community site but did not get any response so far. The solution I think is either make the test step in the build pipeline change the directory or configure somewher sonarqube to search the agent build directory _temp which is above the working directory
This seems a pretty straightforward thing to do, but I cannot find relevant information. In Visual Studio is very easy to calculate code metrics for all projects and I would like to do the same during a build pipeline in Azure DevOps.
Has anyone done something like this?
In azure devops, you can review code coverage results. The results can be viewed and downloaded on the Code coverage tab.
Publish Code Coverage Results publishes code coverage results to
Azure Pipelines or TFS, which were produced by a build in Cobertura
or JaCoCo format.
Built-in tasks such as Visual Studio Test, .NET Core, Ant, Maven,
Gulp, Grunt, and Gradle provide the option to publish code coverage
data to the pipeline.
Here are some documents you can refer to:
Review code coverage results
Azure DevOps and the Code Coverage
In addition, you can get code analysis through SonarCloud integrated with Azure devops. SonarCloud is a cloud-based code quality and security service.
Here is the lab you can follow.
You can generate code metrics by adding the Microsoft.CodeAnalysis.Metrics nuget and building with the parameter that target the "Metrics" build, this will store the results in an xml file.
msbuild /t:Metrics
If you're building a solution with multiple projects, you have to add the nuget to every project otherwise the build will fail because the Metrics target won't be present for the other projects.
The build will produce one xml file per project.
You can also use the metrics command line tool from the roslyn analyzer repository.
See https://learn.microsoft.com/en-us/visualstudio/code-quality/how-to-generate-code-metrics-data?view=vs-2022
I have a repo in azure DevOps with only folder as test.
Now, I have given the task structure in this way in azure DevOps. But I cannot see the code getting analyzed in sonarqube. The code tab shows blank. Could someone help me with where I am going wrong?? I do not want to give folder name in sources..I want whatever code I add in the branch to be analyzed.
edit: Just realized this is happening only for feature short lived branch..My sonarqube version is 8.0
steps:
task: SonarQubePrepare#4
inputs:
SonarQube: 'connection name'
scannerMode: 'CLI'
configMode: 'manual'
cliProjectKey: 'pipeline-sonar-demo'
cliProjectName: 'pipeline-sonar-demo'
cliSources: "."
extraProperties: |
# Additional properties that will be passed to the scanner,
# Put one key=value per line, example:
sonar.exclusions=**/*.xml
SonarQube extension provides three tasks you will use in your build definitions to analyze your projects:
Prepare Analysis Configuration task, to configure all the required
settings before executing the build.
This task is mandatory.
In case of .NET solutions or Java projects, it helps to integrate
seamlessly with MSBuild, Maven and Gradle tasks.
Run Code Analysis task, to actually execute the analysis of the
source code.
This task is not required for Maven or Gradle projects, because
scanner will be run as part of the Maven/Gradle build.
Publish Quality Gate Result task, to display the Quality Gate status
in the build summary and give you a sense of whether the application
is ready for production "quality-wise".
This task is optional.
It can significantly increase the overall build time because it
will poll SonarQube until the analysis is complete. Omitting this
task will not affect the analysis results on SonarQube - it simply
means the Azure DevOps Build Summary page will not show the status
of the analysis or a link to the project dashboard on SonarQube.
It seems you still need add Run Code Analysis task. Regarding how to use SonarScanner for Azure DevOps, please refer to the following documentation:
https://docs.sonarqube.org/latest/analysis/scan/sonarscanner-for-azure-devops/
I have a pipeline that runs on pull requests and executes tests , it also produces an xml execution report (it is a node project and tests run via jest) , and coverage report that I use for sonarcloud. However , the developers want to be able to see the tests results on each pr and see what tests failed in particular, so my question is whether it is possible to somehow export and display those results in github or sonarcloud
Please refer to the following two methods to get the test cases metric in sonarcloud:
You can add sonar.testExecutionReportPaths in Additional Properties option in Prepare Analysis Configuration task. About sonar.coverageReportPaths
For SonarQube to pick up and analyse the content of the report file, you need to inform the sonar scanner. About the location of the XML file. To do so, edit the sonarqube.properties file. Please refer to this.
coverage reporting:
sonar.testExecutionReportPaths=reports/xxx/xxx.xml
In addition, you can also refer to this similar case.
I use Gitlab CI t report Code Coverage. The CI script prints the output to the console which is then fetched by GitLab via regular expression.
As the result, I get the following coverage description in every merge request:
The question is - how can I track the trend (or change) of this value over a period of time.
I'd like to show the chart with coverage change for e.g. last month, so I'd see, whether the project's code quality is improved or decreased. Looking to something similar to this image:
Which GitLab plugins or open source tools allow to get such a picture / report? Is it possible to achieve a similar result by using just gitlab alone?