I am trying to get code coverage results to show up on the pipeline run summary tab. To debug this, I used a sample project with the following pipeline.yaml file:
trigger:
- master
pool:
vmImage: ubuntu-latest
steps:
- task: UsePythonVersion#0
inputs:
versionSpec: '3.7'
displayName: 'Use Python 3.7'
- script: |
pip install -r requirements.txt
displayName: 'Install requirements'
- script: |
pip install pytest pytest-azurepipelines
pip install pytest-cov
pytest --doctest-modules --junitxml=junit/test-results.xml --cov=. --cov-report=xml
displayName: 'pytest'
- task: PublishTestResults#2
condition: succeededOrFailed()
inputs:
testResultsFiles: '**/test-*.xml'
testRunTitle: 'Publish test results'
The output of the PublishTestResults task is (Edit: As pointed out by the answers, this has nothing to do with the problem since it merely reports the test upload but not the code coverage report upload):
Starting: PublishTestResults
==============================================================================
Task : Publish Test Results
Description : Publish test results to Azure Pipelines
Version : 2.180.0
Author : Microsoft Corporation
Help : https://learn.microsoft.com/azure/devops/pipelines/tasks/test/publish-test-results
==============================================================================
/usr/bin/dotnet --version
5.0.301
Result Attachments will be stored in LogStore
Run Attachments will be stored in LogStore
Async Command Start: Publish test results
Publishing test results to test run '324'.
TestResults To Publish 1, Test run id:324
Test results publishing 1, remaining: 0. Test run id: 324
Published Test Run : https://dev.azure.com/.../Runs?runId=324&_a=runCharts
Async Command End: Publish test results
Finishing: PublishTestResults
So, it seems that everything works as expected. However, I do not get the results to display on the summary tab (and no code coverage tab appears):
Screenshot of pipeline summary
Since I am running out of ideas what to try: Is this feature broken or did I do something wrong?
Edit:
It has been suggested to add a PublishCodeCoverageResults-task. However, with the following at the end of the above yaml-file, it still does not work.
- task: PublishCodeCoverageResults#1
inputs:
codeCoverageTool: Cobertura
summaryFileLocation: '$(System.DefaultWorkingDirectory)/**/coverage.xml'
The output of the task is:
Starting: PublishCodeCoverageResults
==============================================================================
Task : Publish code coverage results
Description : Publish Cobertura or JaCoCo code coverage results from a build
2021-07-21T04:26:34: -reports:/home/vsts/work/1/s/**/coverage.xml
2021-07-21T04:26:34: -targetdir:/home/vsts/work/_temp/cchtml
2021-07-21T04:26:34: -reporttypes:HtmlInline_AzurePipelines
2021-07-21T04:26:35: Writing report file '/home/vsts/work/_temp/cchtml/index.html'
2021-07-21T04:26:35: Report generation took 0.5 seconds
Generated code coverage html report: /home/vsts/work/_temp/cchtml
Reading code coverage summary from '/home/vsts/work/1/s/coverage.xml'
Async Command Start: Publish code coverage
Publishing coverage summary data to TFS server.
Lines- 17 of 22 covered.
Branches- 0 of 0 covered.
Modifying Cobertura Index file
Publishing code coverage files to TFS server.
Uploading 8 files
Building file tree
Uploaded 0 out of 1,426,385 bytes.
Uploaded 1,426,385 out of 1,426,385 bytes.
Associating files
Total files: 8 ---- Associated files: 0 (0%)
File upload succeed.
Published '/home/vsts/work/_temp/cchtml' as artifact 'Code Coverage Report_1263'
Async Command End: Publish code coverage
Finishing: PublishCodeCoverageResults
To see code coverage I think you need to add task PublishCodeCoverageResults#1
https://learn.microsoft.com/en-us/azure/devops/pipelines/ecosystems/python?view=azure-devops#run-tests
Add the Publish Code Coverage Results task.
- task: PublishCodeCoverageResults#1
inputs:
codeCoverageTool: Cobertura
summaryFileLocation: '$(System.DefaultWorkingDirectory)/**/coverage.xml'
Estension https://marketplace.visualstudio.com/items?itemName=dazfuller.pyunittest-task
Related
I've got a build pipe for an Azure Function using .Net Core 3.1.x. All the steps until the publishing are doing fine. I can get the publish step working by using script, but not through the yaml task. What am I missing?
Script (works)
- script: dotnet publish --configuration Release .\af-process-mds-vehicle-output-to-deviation\af-process-mds-vehicle-output-to-deviation.csproj
Task (does not work)
- task: DotNetCoreCLI#2
displayName: 'Publish Project'
inputs:
command: 'publish'
configuration: 'Release'
projects: '.\af-process-mds-vehicle-output-to-deviation\af-process-mds-vehicle-output-to-deviation.csproj'
zipAfterPublish: true
It doesn't find the project.
Here's the error message.
2021-10-29T05:21:44.3024816Z ##[section]Starting: dotnet publish
2021-10-29T05:21:44.3150367Z ==============================================================================
2021-10-29T05:21:44.3150726Z Task : .NET Core
2021-10-29T05:21:44.3151190Z Description : Build, test, package, or publish a dotnet application, or run a custom dotnet command
2021-10-29T05:21:44.3151475Z Version : 2.187.0
2021-10-29T05:21:44.3151733Z Author : Microsoft Corporation
2021-10-29T05:21:44.3152035Z Help : https://learn.microsoft.com/azure/devops/pipelines/tasks/build/dotnet-core-cli
2021-10-29T05:21:44.3152373Z ==============================================================================
2021-10-29T05:21:44.7797987Z [command]C:\Windows\system32\chcp.com 65001
2021-10-29T05:21:44.7903026Z Active code page: 65001
2021-10-29T05:21:44.7927221Z Info: .NET Core SDK/runtime 2.2 and 3.0 are now End of Life(EOL) and have been removed from all hosted agents. If you're using these SDK/runtimes on hosted agents, kindly upgrade to newer versions which are not EOL, or else use UseDotNet task to install the required version.
2021-10-29T05:21:44.8938257Z ##[error]No web project was found in the repository. Web projects are identified by presence of either a web.config file, wwwroot folder in the directory, or by the usage of Microsoft.Net.Web.Sdk in your project file. You can set Publish web projects property to false (publishWebProjects: false in yml) if your project doesn't follow this convention or if you want to publish projects other than web projects.
2021-10-29T05:21:44.9001249Z Info: Azure Pipelines hosted agents have been updated and now contain .Net 5.x SDK/Runtime along with the older .Net Core version which are currently lts. Unless you have locked down a SDK version for your project(s), 5.x SDK might be picked up which might have breaking behavior as compared to previous versions. You can learn more about the breaking changes here: https://learn.microsoft.com/en-us/dotnet/core/tools/ and https://learn.microsoft.com/en-us/dotnet/core/compatibility/ . To learn about more such changes and troubleshoot, refer here: https://learn.microsoft.com/en-us/azure/devops/pipelines/tasks/build/dotnet-core-cli?view=azure-devops#troubleshooting
2021-10-29T05:21:44.9003648Z ##[error]Project file(s) matching the specified pattern were not found.
2021-10-29T05:21:44.9182124Z ##[section]Finishing: dotnet publish
After the tips from the answer I got the pipe working. Here's the full working pipe. (Still don't know why it didn't work earlier.)
Working pipe:
name : af-vehicle-sync-to-deviation
## if there is a change is the deviation folder for the main branch. Then trigger.
trigger:
branches:
include:
- main
paths:
include:
- af-process-mds-vehicle-output-to-deviation/*
pool:
vmImage: 'windows-latest'
variables:
buildConfiguration: 'Release'
SolutionPath: '**\*.sln'
stages:
- stage: Build
displayName: Build solution
jobs:
- job: Build
displayName: Build and publish solution
steps:
- checkout: self
- task: NuGetToolInstaller#1
- task: NuGetCommand#2
inputs:
restoreSolution: $(SolutionPath)
- task: UseDotNet#2
inputs:
packageType: 'sdk'
version: '3.1.x'
displayName: 'Use .NET Core SDK 3.1.x'
- task: DotNetCoreCLI#2
inputs:
command: 'build'
configuration: $(buildConfiguration)
projects: '$(SolutionPath)'
displayName: 'Build solution'
- task: DotNetCoreCLI#2
displayName: 'Publish Project'
inputs:
command: 'publish'
configuration: 'Release'
projects: '**\*.csproj'
publishWebProjects: false
zipAfterPublish: true
arguments: '--output $(Build.ArtifactStagingDirectory)/$(buildConfiguration)'
- task: PublishBuildArtifacts#1
inputs:
PathtoPublish: '$(Build.ArtifactStagingDirectory)'
ArtifactName: 'drop'
publishLocation: 'Container'
You could use '**/*.csproj' but honestly, I would do something like this answer and add a script to list out all the files and folders recursively before this step that fails.
Assuming that you have a restore or build step before this publish you could add it after those, or just as the first step after your checkout one.
You can also inspect the logs of earlier steps to see the file path/s., instructions on doing this are available here.
Using $(System.DefaultWorkingDirectory) as your root is also recommended, rather than .\, so you would have '$(System.DefaultWorkingDirectory)\af-process-mds-vehicle-output-to-deviation...'.
Edit
If you look at the logs for your build step you will see entries like /home/vsts/work/1/s/XXX.YYY.ZZZ/XXX.YYY.ZZZ.csproj that refer to the different projects inside your solution. By default most commands will be run in $(System.DefaultWorkingDirectory) which would equate to /home/vsts/work/1/s/ in this instance, you can think of it as the root of your repository - there is more information on this structure here.
The error you were encountering is actually about the lack of a web project, rather than a path issue though, for the build step it is best practice to use the --output <output-directory-here> flag to output the compile files into a specific folder, that way you can easily publish that folder.
I have an Azure DevOps pipeline that validates pull requests. I have configured dotnet test to collect code coverage metrics using the --collect "Code coverage" argument:
- task: DotNetCoreCLI#2
displayName: dotnet test
inputs:
command: 'test'
arguments: '--configuration $(BuildConfiguration) --collect "Code coverage" /p:CollectCoverage=true /p:CoverletOutputFormat=cobertura'
workingDirectory: $(baseWorkingDirectory)
projects: 'tests/**/*.csproj'
nobuild: true
As you can see, I'm also passing /p:CollectCoverage=true /p:CoverletOutputFormat=cobertura. This I have to do in order to generete a coverage report:
- task: CmdLine#2
inputs:
script: dotnet tool install -g dotnet-reportgenerator-globaltool
- task: CmdLine#2
inputs:
script: reportgenerator -reports:$(Build.SourcesDirectory)/tests/**/coverage.cobertura.xml -targetdir:$(Build.SourcesDirectory)/CodeCoverage -reporttypes:HtmlInline_AzurePipelines;Cobertura
- task: PublishCodeCoverageResults#1
inputs:
codeCoverageTool: Cobertura
summaryFileLocation: '$(Build.SourcesDirectory)/CodeCoverage/Cobertura.xml'
reportDirectory: '$(Build.SourcesDirectory)/CodeCoverage'
Documentation states that code coverage for pull requests is available only using the Visual Studio code coverage results format (file extension .coverage)
So:
I have to use Visual Studio code coverage --collect:"Code Coverage" to get code coverage for pull requests, because Cobertura format won't work.
I need to use Cobertura format in order to get a readable report on the Code Coverage tab in the pipeline, but the Cobertura report won't show up if I use --collect:"Code Coverage" at the same time.
It seems I can't get both code coverage for PRs and a full report in Cobertura format at the same time.
Other people seem to have the same problem, but the issue wasn't resolved in that thread.
Am I missing something?
Can you try use this?
You have to install coverlet.collector package in all your test projects. Please notice that I used argument --collect:"XPlat Code Coverage".
# You just added coverlet.collector to use 'XPlat Code Coverage'
- task: DotNetCoreCLI#2
displayName: Test
inputs:
command: test
projects: '**/*Tests/*.csproj'
arguments: '--configuration $(buildConfiguration) --collect:"XPlat Code Coverage" -- RunConfiguration.DisableAppDomain=true'
workingDirectory: $(Build.SourcesDirectory)
- task: DotNetCoreCLI#2
inputs:
command: custom
custom: tool
arguments: install --tool-path . dotnet-reportgenerator-globaltool
displayName: Install ReportGenerator tool
- script: ./reportgenerator -reports:$(Agent.TempDirectory)/**/coverage.cobertura.xml -targetdir:$(Build.SourcesDirectory)/coverlet/reports -reporttypes:"Cobertura"
displayName: Create reports
- task: PublishCodeCoverageResults#1
displayName: 'Publish code coverage'
inputs:
codeCoverageTool: Cobertura
summaryFileLocation: $(Build.SourcesDirectory)/coverlet/reports/Cobertura.xml
Please be aware that you may have different folder strcture.
If you want to use Code coverage for pull requests feature you should not use Cobertura:
Code coverage for pull requests capability is currently only available for Visual Studio code coverage (.coverage) formats. This can be used if you publish code coverage using the Visual Studio Test task, the test verb of dotnet core task and the TRX option of the publish test results task. Support for other coverage tools and result formats will be added in future milestones.
In the pipeline definition:
- task: DotNetCoreCLI#2
displayName: Test
inputs:
command: test
projects: 'path/to/test.csproj' # if multiple test projects are to be executed, create a dirs.proj and use it here. That will ensure the tests are executed in parallel
arguments: '--collect "Code Coverage" '
publishTestResults: true
In test project (or Packages.props if using https://github.com/microsoft/MSBuildSdks/tree/main/src/CentralPackageVersions)
...
<PackageReference Include="Microsoft.NET.Test.Sdk" Version="17.0.0" />
...
According to https://github.com/microsoft/vstest/issues/981 and with Microsoft.NET.Test.Sdk 17.0.0+ coverlet is no longer needed as all coverage processing can be done using .coverage files on other-than Windows platforms using --collect:"Code Coverage". coverlet is no longer needed.
Unfortunately Azure DevOps is not supporting both code coverage PR support and readable reports at the same time.
My recommendation is to use dynamic code coverage (--collect "Code Coverage") and get *.coverage files. On PR build you can publish test results with publishRunAttachments: true. By this you should get code coverage PR support.
Additionally you can use dotnet-coverage to convert your *.coverage reports to cobertura. Then you can use report-generator to generate readable coverage report. This coverage report can be uploaded as zip into artifacts of build. In this case using PublishCodeCoverageResults#1 will not work as Azure DevOps is not supporting it.
On non-PR builds you can again use dynamic code coverage but when publishing test results specify publishRunAttachments: false. Then again use dotnet-coverage to merge and convert your reports to cobertura. Finally you can add
PublishCodeCoverageResults#1 task to get readable coverage report.
Both dotnet-coverage and report-generator are cross-plat dotnet global tools. You can easily install it in your pipeline.
Example script to generate html report and zip it:
dotnet-coverage merge -r -f cobertura -o merged.cobertura.xml *.coverage
reportgenerator -reports:merged.cobertura.xml -targetDir coverageReportHtml -reporttypes:HtmlInline
zip -r report.zip coverageReportHtml/*
There is one more workaround but I didn't try this. You could theoretically trigger 1 more build from your PR build. The second build could download *.coverage artifacts from first one and then use tooling like above to finally perform PublishCodeCoverageResults#1 task.
I am using Azure DevOps build pipeline to send a artifact over to a release pipeline. My build pipeline is as follows:
Install Node JS with NodeTool#0
Run a Script Command that runs npm install and then runs the test suite.
Post the code coverage results via PublishCodeCoverageResults#1
Publish artifact with PublishPipelineArtifact#1 and targetPath: '$(Pipeline.Workspace)' as the artifact directory.
When I originally did this without using the code coverage part, IE not installing node, the files sent over were 257 and was a little over 25MB. Completely workable and quick.
However when I installed node and ran code coverage the files exploded to 750MB, which is understandable - but is way too much to transfer as an artifact IMHO.
So, looking at a solution I found this: https://learn.microsoft.com/en-us/azure/devops/artifacts/reference/artifactignore?view=azure-devops
And, I added this into the root of the repo.
However, this file seems to have not made a difference using the PublishPipelineArtifact#1 at all. No matter what I did, all the files from the Workspace were used as the artifact.
What did work though, was physically deleting the files in the build process.
I realize that is a lot of detail for a question, but I would like to know if the .artifactignore file actually works with this build pipeline?
Is there a way I can use the artifact ignore file properly, and not worry about deleting files before creating the artifact?
Tried to change the target directory in the PublishPipelineArtifact#1 task.
Here is the current working YAML:
# Node.js with Angular
# Build a Node.js project that uses Angular.
# Add steps that analyze code, save build artifacts, deploy, and more:
# https://learn.microsoft.com/azure/devops/pipelines/languages/javascript
trigger:
- master
pool:
vmImage: 'ubuntu-latest'
steps:
- task: NodeTool#0
inputs:
versionSpec: '10.x'
displayName: 'Install Node.js'
- script: |
npm install
npm run test:devops
displayName: 'npm install and test'
- task: PublishCodeCoverageResults#1
displayName: 'publish ng code coverage results'
condition: succeededOrFailed()
inputs:
codeCoverageTool: 'Cobertura'
summaryFileLocation: 'coverage/cobertura-coverage.xml'
reportDirectory: coverage
failIfCoverageEmpty: true
- task: DeleteFiles#1
inputs:
SourceFolder: '$(Pipeline.Workspace)/s'
Contents: |
node_modules
coverage
- task: PublishPipelineArtifact#1
inputs:
targetPath: '$(Pipeline.Workspace)'
artifact: 'my-artifact'
The expected results would be a smaller artifact file that still posts code coverage and installs node.
The .artifactignore should be in the same directory as defined as target path of the publish pipelines artifact task (not only in the root of the repo like azure-pipelines.yml).
Is not in the docs and there is an issue about it here. also check here.
Ok, so I used the information from Shayki Abramczyk - which is the correct answer - but wanted to put a working yaml together for everyone. I had to copy my .artifactignore from the repo, which is: $(Pipeline.Workspace)/s to the root which is $(Pipeline.Workspace)
As soon as I did this - it showed results. As soon as I adjusted the artifact ignore file, the output reduced from 750+MB to 2.3MB and processed in 8 seconds.
Working YAML:
- task: NodeTool#0
inputs:
versionSpec: '10.x'
displayName: 'Install Node.js'
- script: |
npm install
npm run test
displayName: 'npm install and test'
- task: PublishCodeCoverageResults#1
displayName: 'publish ng code coverage results'
condition: succeededOrFailed()
inputs:
codeCoverageTool: 'Cobertura'
summaryFileLocation: 'coverage/cobertura-coverage.xml'
reportDirectory: coverage
failIfCoverageEmpty: true
- task: CopyFiles#2
inputs:
SourceFolder: '$(Pipeline.Workspace)/s'
Contents: '**/.artifactignore'
TargetFolder: '$(Pipeline.Workspace)'
- task: PublishPipelineArtifact#1
inputs:
targetPath: '$(Pipeline.Workspace)'
artifact: 'my-artifact'
FYI - this is being built for angular, so this is the artifact ignore file I used. Your mileage will vary based on your project.
**/dist
**/node_modules
**/coverage
**/.git
I have a build configuration defined here:
https://github.com/cpoDesign/APIFramework/blob/master/azure-pipelines.yml
I have managed to generate a nuget package using the following command
- task: DotNetCoreCLI#2
inputs:
command: pack
projects: '**/*ApiFramework.csproj'
A subset of the task output of the script is
Task "PackTask"
2018-11-27T23:02:32.4459067Z Successfully created package '/home/vsts/work/1/a/CPODesign.ApiFramework.1.0.0.nupkg'.
Next step resolved:
I do not want to create a build with release into nuget as those steps have to be logically separated. Therefore I have created a new step Create a drop.
Configuration:
My drop task definition:
- task: PublishBuildArtifacts#1
inputs:
pathtoPublish: '$(Build.ArtifactStagingDirectory)'
artifactName: drop
contents: '**/$(BuildConfiguration)/**/?(*.nupkg)'
Build output:
2018-11-27T23:04:24.6351310Z ##[section]Starting: PublishBuildArtifacts
2018-11-27T23:04:24.6353582Z ==============================================================================
2018-11-27T23:04:24.6353896Z Task : Publish Build Artifacts
2018-11-27T23:04:24.6353944Z Description : Publish build artifacts to Azure Pipelines/TFS or a file share
2018-11-27T23:04:24.6354007Z Version : 1.142.2
2018-11-27T23:04:24.6354046Z Author : Microsoft Corporation
2018-11-27T23:04:24.6354091Z Help : [More Information](https://go.microsoft.com/fwlink/?LinkID=708390)
2018-11-27T23:04:24.6354156Z ==============================================================================
2018-11-27T23:04:26.1357631Z ##[section]Async Command Start: Upload Artifact
2018-11-27T23:04:26.1357755Z Uploading 1 files
2018-11-27T23:04:26.6373194Z File upload succeed.
2018-11-27T23:04:26.6373313Z Upload '/home/vsts/work/1/a' to file container: '#/1558454/drop'
2018-11-27T23:04:27.9231805Z Associated artifact 91 with build 806
2018-11-27T23:04:27.9231947Z ##[section]Async Command End: Upload Artifact
2018-11-27T23:04:27.9232436Z ##[section]Finishing: PublishBuildArtifacts
Note: The UI for azure-devops has changed and the artefacts (artifacts) are no longer created as a new tab but rather badly added into the report summary
The question:
How do I generate a specific version of nuget package IE: 1.0.%(Build.BuildId)?
My last attempt is
- task: DotNetCoreCLI#2
inputs:
command: pack
projects: '**/*ApiFramework.csproj'
# packageVersion:'1.0.$(Build.BuildId)'
where
packageVersion:'1.0.$(Build.BuildId)'
Will cause the build to fail
(the current branch is published here:https://github.com/cpoDesign/APIFramework/blob/cpoDesign-build-mods-1/azure-pipelines.yml)
after frustrating couple hours I have found the answer
Publish the generated nuget package into build artefacts
Generate a release to publish the nuget package
Build configuration
pool:
vmImage: 'Ubuntu 16.04'
variables:
buildConfiguration: 'Release'
steps:
- script: dotnet build --configuration $(buildConfiguration)
displayName: 'dotnet build $(buildConfiguration)'
- task: DotNetCoreCLI#2
inputs:
command: restore
projects: '**/*.csproj'
- script: dotnet test **/*.Tests.Unit.csproj --logger trx
- task: PublishTestResults#2
inputs:
testRunner: VSTest
testResultsFiles: '**/*.trx'
- script: dotnet pack /p:PackageVersion='1.0.$(Build.BuildId)' --configuration $(buildConfiguration) -o $(Build.ArtifactStagingDirectory)
- task: PublishBuildArtifacts#1
inputs:
pathtoPublish: '$(Build.ArtifactStagingDirectory)'
artifactName: drop
contents: '**/$(BuildConfiguration)/**/?(*.nupkg)'
Release section
I have updated the release to trigger after each successful build
dotnet nuget push artefactName.APIFramework/drop/CPODesign.ApiFramework.1.0.$(Build.BuildId).nupkg -k $(myapiKey) -s https://api.nuget.org/v3/index.json
I can't see the contents as a valid input in the YAML for PublishBuildArtifacts#1
Do you mean to do a copy task first before you do the publish task. Like shown in the publish documentation?
https://learn.microsoft.com/en-us/azure/devops/pipelines/tasks/utility/publish-build-artifacts?view=vsts
Im currently setting up an azure pipeline for my repository. Currently it builds correctly and runs the unit tests. However the code coverage tab just spins infinitely. Any idea on what would cause this?
Details:
The artifact directory looks like this:
The console shows this error:
Error: Could not find route for route id
ms.vss-tfs-web.project-overview-route. Ensure that the requested route
is added to routes shared data.
This is how the test results are ran and generated:
dotnet tool install dotnet-reportgenerator-globaltool --tool-path .
dotnet test $(Build.SourcesDirectory)\RulesMadeEasy.Tests -c debug --logger trx --no-restore /p:CollectCoverage=true /p:CoverletOutputFormat=cobertura --results-directory $(Build.SourcesDirectory)\TestResults\ /p:CoverletOutput=$(Build.SourcesDirectory)\TestResults\
.\reportgenerator -reports:$(Build.SourcesDirectory)\TestResults\coverage.cobertura.xml -targetdir:$(Build.SourcesDirectory)\TestResults\ -reporttypes:"HTMLInline_AzurePipelines;Badges" --version 4.0.0-rc4
The code coverage results are published using the PublishCodeCoverageResults#1 task with the following inputs
inputs:
codeCoverageTool: Cobertura
summaryFileLocation: '$(Build.SourcesDirectory)\TestResults\coverage.cobertura.xml'
reportDirectory: '$(Build.SourcesDirectory)\TestResults'
You have to enable the Boards service in you Azure DevOps project to let the error disappear.