I want to fail the build pipeline if a single test failed with azure pipelines.
Azure can successfully detect that my tests entered a failed state however it gives a success state to the entire build pipeline:
The question is how to make azure give a failed build state if the tests stage failed?
Here's my azure-pipelines.yml :
# Build ASP.NET Core project using Azure Pipelines
# https://learn.microsoft.com/azure/devops/pipelines/languages/dotnet-core?view=vsts
pool:
vmImage: 'Ubuntu 16.04'
variables:
buildConfiguration: 'Release'
steps:
- script: |
dotnet build --configuration $(buildConfiguration)
dotnet test dotnetcore-tests --configuration $(buildConfiguration) --logger trx
dotnet publish --configuration $(buildConfiguration) --output $BUILD_ARTIFACTSTAGINGDIRECTORY
- task: PublishTestResults#2
inputs:
testRunner: VSTest
testResultsFiles: '**/*.trx'
- task: PublishBuildArtifacts#1
The original answer didn't work for me, but it looks like there was a lot of discussion on this, and there's now a failTaskOnFailedTests param for the task. That seems to work.
- task: PublishTestResults#2
inputs:
testRunner: VSTest
testResultsFiles: '**/*.trx'
failTaskOnFailedTests: true
I'm still surprised this wasn't default behavior.
Try to add failOnStandardError: 'true' in the task inputs:
- task: PublishTestResults#2
inputs:
testRunner: VSTest
testResultsFiles: '**/*.trx'
failOnStandardError: 'true'
Untick the below-highlighted option
Related
I am trying to integrate SonarQube in a CI-pipeline in Azure DevOps. I used the exact configuration that I used in other pipelines where everything runs smoothly. However, this pipeline can't find the analysis in the SonarQube Publish step as shown in the logs below.
##[debug] /home/vsts/work/_temp/sonar (directory)
##[debug] /home/vsts/work/_temp/sonar/Company.Service-CI[sonarqube-test][121226][2022-01-19] (directory)
##[debug] /home/vsts/work/_temp/sonar/Company.Service-CI[sonarqube-test][121226][2022-01-19]/9731b11b-6af8-d211-752d-e8fe0770156c (directory)
##[debug] /home/vsts/work/_temp/sonar/Company.Service-CI[sonarqube-test][121226][2022-01-19]/9731b11b-6af8-d211-752d-e8fe0770156c/report-task.txt (file)
##[debug]4 results
##[debug]found 4 paths
##[debug]applying include pattern
##[debug]adjustedPattern: '/home/vsts/work/_temp/sonar/Company.Service-CI[sonarqube-test][121226][2022-01-19]/**/report-task.txt'
##[debug]0 matches
##[debug]0 final results
##[debug][SQ] Searching for sonar/Company.Service-CI[sonarqube-test][121226][2022-01-19]/**/report-task.txt - found 0 file(s)
##[warning]No analyses found in this build! Please check your build configuration.
I find this problem hard to solve because it seems like the analysis report is there but after applying the include pattern there are zero matches. The pipeline contains the following steps:
stages:
- stage:
jobs:
- job: Build_Restore_and_Run_Unit_Tests
pool:
vmImage: ubuntu-latest
steps:
- task: SonarQubePrepare#5
displayName: SonarQube
inputs:
SonarQube: SonarQube
scannerMode: MSBuild
projectKey: $(sonarQubeProjectKey)
projectName: $(sonarQubeProjectName)
extraProperties: |
sonar.cs.opencover.reportsPaths=$(Agent.TempDirectory)/*/coverage.opencover.xml
- task: DotNetCoreCLI#2
displayName: Restore
inputs:
command: 'restore'
projects: 'src/**/*.csproj'
feedsToUse: 'select'
vstsFeed: '989127ec-b6ef-4c8c-ae54-90dec743281d'
- task: DotNetCoreCLI#2
displayName: Build
inputs:
command: build
projects: |
**/*.Test.csproj
arguments: '--configuration $(buildConfiguration) --no-restore'
- task: DotNetCoreCLI#2
displayName: Test
inputs:
command: test
projects: '**/*.Test.csproj'
nobuild: true
arguments: '--no-restore --collect:"XPlat Code Coverage" -- DataCollectionRunSettings.DataCollectors.DataCollector.Configuration.Format=cobertura,opencover'
publishTestResults: true
- task: PublishCodeCoverageResults#1
displayName: Publish code coverage
inputs:
codeCoverageTool: Cobertura
summaryFileLocation: $(Agent.TempDirectory)/*/coverage.cobertura.xml
- task: SonarQubeAnalyze#5
displayName: Run code analysis
- task: SonarQubePublish#5
displayName: Publish quality gate result
inputs:
pollingTimeoutSec: '300'
My attempts to fix this issue:
Currently I am using ubuntu-latest. I have tried a Windows image.
Applied time-outs.
Upgraded the SonarQube steps from major version 4 to 5.
Added an unique GUID to each projects csproj.
As Mickaël Caro stated on my similar post in the SonarSource community forum:
Unfortunately some characters are not recognized nor parsed correctly when we try to find the report path (We do have this issue filed here : [VSTS-240] Escape regex characters when finding report path file. - SonarSource).
I changed the [ ] in the name of the build to ( ), and the analysis runs smoothly.
In a pipeline I have two different steps. The first one generates some files, the second should take these files as an input.
the Yaml for that pipeline is the following:
name: myscript
stages:
- stage: Tes/t
displayName: owasp-test
jobs:
- job: owasp_test
displayName: run beasic checks for site
pool:
name: default
demands: Agent.OS -equals Windows_NT
steps:
- task: DotNetCoreCLI#2
inputs:
command: 'build'
projects: '**/*.sln'
- task: dependency-check-build-task#5
inputs:
projectName: 'DependencyCheck'
scanPath: '**/*.dll'
format: 'JUNIT'
- task: PublishTestResults#2
inputs:
testResultsFormat: 'JUnit'
testResultsFiles: '**/*-junit.xml'
the dependency-check-build-task returns an XML-File:
File upload succeed.
Upload 'P:\Azure-Pipelines-Agent\_work\2\TestResults\dependency-check\dependency-check-junit.xml' to file container: '#/11589616/dependency-check'
Associated artifact 53031 with build 21497
The following step (PublishTestResults) SHOULD take that file but returns
##[warning]No test result files matching **/*-junit.xml were found.
instead. I can see that file in the artifact after the pipeline is run.
This is because your report is written to Common.TestResultsDirectory which is c:\agent_work\1\TestResults (for Microsoft Hosted agents), and publish test task looks in System.DefaultWorkingDirectory which is c:\agent_work\1\s.
Please try:
- task: PublishTestResults#2
inputs:
testResultsFormat: 'JUnit'
testResultsFiles: '**/*-junit.xml'
searchFolder: '$(Common.TestResultsDirectory)'
I had the same trouble:
I fixed changing the Agent Specification
I have the following three steps to publish a function app to artifact in Azure Pipeline:
- task: DotNetCoreCLI#2
displayName: 'dotnet publish function app'
inputs:
command: publish
arguments: '--configuration Release --output updater_publish_output'
projects: 'Service/XYZ/Hosts.FA/*.csproj'
publishWebProjects: false
modifyOutputPath: false
zipAfterPublish: false
- task: ArchiveFiles#2
displayName: 'archive function app files'
inputs:
rootFolderOrFile: "$(System.DefaultWorkingDirectory)/updater_publish_output"
includeRootFolder: false
archiveFile: "$(System.DefaultWorkingDirectory)/Hosts.FA.zip"
- task: PublishBuildArtifacts#1
displayName: 'publish function app files'
inputs:
PathtoPublish: '$(System.DefaultWorkingDirectory)/Hosts.FA.zip'
ArtifactName: '$(Build.BuildNumber)'
Here is the project structure:
I have updated the first step to:
- name: dotnet publish function app
run: dotnet publish Service/XYZ/Hosts.FA/Hosts.FA.csproj --configuration Release --output updater_publish_output
How do I convert the tasks ArchiveFiles#2 & PublishBuildArtifacts#1 to GitHub Actions?
Use the Upload-Artifact task from here: https://github.com/actions/upload-artifact. It will replace both ArchiveFiles#2 (zipping) and PublishBuildArtifacts#1 (uploading).
- uses: actions/upload-artifact#v2
with:
name: ${{github.run_number}}
path: |
updater_publish_output
As per https://docs.github.com/en/actions/reference/context-and-expression-syntax-for-github-actions#github-context github.run_number is
A unique number for each run of a particular workflow in a repository. This number begins at 1 for the workflow's first run, and increments with each new run. This number does not change if you re-run the workflow run.
You could also use github.run_id:
A unique number for each run within a repository. This number does not change if you re-run the workflow run.
I follow the steps of Computing code coverage for a .NET Core project with Azure DevOps and Coverlet.
Build run like expected and every step ends successfully.
The Artefact-Explorer shown the uploaded report and In summary I get the Code Coverage result.
But I missing the Code Coverage tab next to Tests tab to take a look to the detailed report.
Configuration YAML:
- task: NuGetToolInstaller#0
displayName: 'Use NuGet 5.0.2'
inputs:
versionSpec: 5.0.2
checkLatest: true
- task: NuGetCommand#2
displayName: 'NuGet restore'
inputs:
restoreSolution: '$(Parameters.solution)'
- task: VSBuild#1
displayName: 'Projektmappe **\*.sln erstellen'
inputs:
solution: '$(Parameters.solution)'
platform: '$(BuildPlatform)'
configuration: '$(BuildConfiguration)'
- task: VisualStudioTestPlatformInstaller#1
displayName: 'Installer für Visual Studio Test-Plattform'
enabled: false
- task: VSTest#2
displayName: 'VsTest - testAssemblies'
inputs:
testAssemblyVer2: |
**\$(BuildConfiguration)\*test*.dll
!**\obj\**
codeCoverageEnabled: true
platform: '$(BuildPlatform)'
configuration: '$(BuildConfiguration)'
- powershell: ./'D:\AzureDevOpsData\Skripte\PowerShell\CodeCoverage.ps1'
displayName: 'PowerShell Test Code Coverage'
- task: Palmmedia.reportgenerator.reportgenerator-build-release-task.reportgenerator#4
displayName: ReportGenerator
inputs:
reports: coverage.cobertura.xml
targetdir: '$(Build.SourcesDirectory)/CodeCoverage'
- task: PublishCodeCoverageResults#1
displayName: 'Code Coverage veröffentlichen von $(Build.SourcesDirectory)/CodeCoverage/Cobertura.xml'
inputs:
codeCoverageTool: Cobertura
summaryFileLocation: '$(Build.SourcesDirectory)/CodeCoverage/Cobertura.xml'
reportDirectory: '$(Build.SourcesDirectory)/CodeCoverage'
The PowerShell Script contains:
#TEST CSPROJ
$csproj = "FrameworkA_Tests"
#SEARCH TEST CSPROJ.DLL
"`nrun tests:"
$unitTestFile = gci -Recurse | ?{ $_.FullName -like "*bin\*$csproj.dll" }
Write-Host "`$unitTestFile value: $unitTestFile"
#GET COVERLET.EXE
$coverlet = "D:\AzureDevOpsData\Tools\coverlet\coverlet.exe"
#RUN COVERLET.EXE
"calling $coverlet for $($unitTestFile.FullName)"
&$coverlet $unitTestFile.FullName --target "dotnet" --targetargs "vstest $($unitTestFile.FullName) --logger:trx" --format "cobertura"
Do I forget something?
Please try to enable "Boards" in the project settings:
This issue was fixed for Azure DevOps but may still exist in Azure DevOps Server.
See:
https://developercommunity.visualstudio.com/content/problem/385331/code-coverage-results-not-rendered-in-build-result.html
https://developercommunity.visualstudio.com/content/problem/398209/build-results-tab-code-coverage-does-not-show-up.html
ReportGenerator missing Code Coverage tab (Azure DevOps Server 2019.0.1)
This should be a known issue on the Developer Community:
Code coverage tab missing in Azure DevOps Server
MS team reply: A fix for this issue has been internally implemented and is being prepared for release.
As workaround, you can try the method provided by jingzhu yan:
you can add copy files and publish build results steps , then you can
download coverage result file from Artifacts.
Hope this helps.
Added a Publish test results task in Azure DevOpsCI/CD pipeline, test were successfull, but after running the test it complaints about ##[warning]No test result files matching **/test-*.xml were found. Could someone please advise on how can we resolve similar problem ?
Publish Test Results task : configuration
Test result format= JUnit
Test results files= **/test-*.xml
Search folder = $(System.DefaultWorkingDirectory)
Test results title = Cypress Test Results
note: I have try adding the search folder path as follows: C:\agent_work\r5\a\drop\ui-tests\cypress
package.json to run the tests
"scripts": {
"test": "cypress run --record --key <key value here>"
}
My directory path in server:
C:\agent_work\r5\a\drop\ui-tests\cypress
My friend, I was facing the same issue on Azure DevOps.
In my case, the folder where the xml files were generated was reports on the root of the repo, that depends on how you got configured Junit on your cypress.json file
So In my case, the solution was changing this on azure-pipelines.yml
testResultsFiles: "results/*.xml"
searchFolder: $(System.DefaultWorkingDirectory)
So that's the entire setup of the testing pipeline
# Node.js
# Build a general Node.js project with npm.
# Add steps that analyze code, save build artifacts, deploy, and more:
# https://learn.microsoft.com/azure/devops/pipelines/languages/javascript
trigger:
- master
pool:
vmImage: 'ubuntu-latest'
steps:
- task: NodeTool#0
inputs:
versionSpec: '12.x'
displayName: 'Install Node.js'
- script: "npm i"
displayName: "Install project dependencies"
- script: "npm run cy:verify"
displayName: "Cypress Verify"
- script: "source cypress.env" # comment this script to run tests against production
displayName: "Using env variables to change url to test against development branch"
- script: "npm run cy:run-report"
displayName: "Run Cypress Tests"
- task: PublishBuildArtifacts#1
displayName: "Publish Artifact: cypress-azure-devops screenshots"
inputs:
PathtoPublish: cypress/screenshots
ArtifactName: "CypressAzureDevopsTestRunScreenshots"
condition: failed()
- task: PublishTestResults#2
displayName: "Publish Test Results"
condition: succeededOrFailed()
inputs:
testResultsFormat: "JUnit"
testResultsFiles: "results/*.xml"
searchFolder: $(System.DefaultWorkingDirectory)
mergeTestResults: true
testRunTitle: 'Test Results'
continueOnError: true
Saludos desde Argentina 🇦🇷