I have some postman collections and I would like to run them in azure devops,
for whatever the reason my tests are not published.
What Am I missing?
If I remove "reporters: junit i can see the result in the step. I was expecting a tab "test" next to summary
resources:
- repo: self
clean: true
queue:
name: Hosted VS2017
demands: npm
steps:
- task: Npm#1
displayName: 'npm install'
inputs:
verbose: false
customCommand: 'install newman -g'
- task: NewmanPostman#4
displayName: 'Postman tests'
inputs:
collectionFileSource: 'my.postman_collection.json'
environmentSourceType: none
reporters: junit
ignoreRedirect: false
bail: false
sslInsecure: false
htmlExtraDarkTheme: false
htmlExtraLogs: false
htmlExtraTestPaging: false
- task: PublishTestResults#2
inputs:
testResultsFormat: 'JUnit' # Options: JUnit, NUnit, VSTest, xUnit
testResultsFiles: '**/*.xml'
What Am I missing? If I remove "reporters: junit i can see the result in the step. I was expecting a tab "test" next to summary
Based on my test, I could reproduce your issue.
Here are the following points, you need to check:
The XML file path in Publish Test Result task.
By default, the Junit xml file will be saved at newman folder.
You could set the file path with the following path:
- task: PublishTestResults#2
displayName: 'Publish Test Results **/*/*.xml'
inputs:
testResultsFiles: '**/*/*.xml'
You can also change the Junit file output path: reporterJUnitExport field in NewmanPostman task
Example:
- task: NewmanPostman#4
displayName: 'Newman - Postman'
inputs:
collectionFileSource: '$(build.sourcesdirectory)'
Contents: 'kevintest123.postman_collection.json'
environmentSourceType: none
ignoreRedirect: false
bail: false
sslInsecure: false
reporters: junit
htmlExtraDarkTheme: false
htmlExtraLogs: false
htmlExtraTestPaging: false
reporterJUnitExport: '$(Build.sourcesdirectory)\Results\junitReport.xml'
You need to make sure that the Json file contain the test. When you Export the files in Postman, you need to add the content in test tab:
Here is a doc about Add test in Postman.
Then you could publish the test result to the test tab successfully via Publish test Restlt task.
Or you will get the following issue and the test result Junit xml file doesn't contain the test result:
Update:
Cannot find "continue on error" anywhere.anyideas
Yaml Editor
- task: PublishTestResults#2
displayName: 'Publish Test Results **/*/*.xml'
inputs:
testResultsFiles: '**/*/*.xml'
continueOnError: true
Classic editor
Junit report is generated in the newman folder in the current directory. You change this location as:
--reporter-junit-export a.xml
--reporter-junit-export folder/a.xml
this will generate report as a.xml or a.xml in the folder/directory you specified.
you have to publish that using publishtest result , if you are not using --reporter-junit-export , then make sure workingdirectory for all the steps are same.
else give full path as :
--reporter-junit-export c:/folder/a.xml
and
in step give the full path of report:
- task: PublishTestResults#2
inputs:
testResultsFormat: 'JUnit' # Options: JUnit, NUnit, VSTest, xUnit
testResultsFiles: 'c:/folder/a.xml'
Related
I am trying to integrate SonarQube in a CI-pipeline in Azure DevOps. I used the exact configuration that I used in other pipelines where everything runs smoothly. However, this pipeline can't find the analysis in the SonarQube Publish step as shown in the logs below.
##[debug] /home/vsts/work/_temp/sonar (directory)
##[debug] /home/vsts/work/_temp/sonar/Company.Service-CI[sonarqube-test][121226][2022-01-19] (directory)
##[debug] /home/vsts/work/_temp/sonar/Company.Service-CI[sonarqube-test][121226][2022-01-19]/9731b11b-6af8-d211-752d-e8fe0770156c (directory)
##[debug] /home/vsts/work/_temp/sonar/Company.Service-CI[sonarqube-test][121226][2022-01-19]/9731b11b-6af8-d211-752d-e8fe0770156c/report-task.txt (file)
##[debug]4 results
##[debug]found 4 paths
##[debug]applying include pattern
##[debug]adjustedPattern: '/home/vsts/work/_temp/sonar/Company.Service-CI[sonarqube-test][121226][2022-01-19]/**/report-task.txt'
##[debug]0 matches
##[debug]0 final results
##[debug][SQ] Searching for sonar/Company.Service-CI[sonarqube-test][121226][2022-01-19]/**/report-task.txt - found 0 file(s)
##[warning]No analyses found in this build! Please check your build configuration.
I find this problem hard to solve because it seems like the analysis report is there but after applying the include pattern there are zero matches. The pipeline contains the following steps:
stages:
- stage:
jobs:
- job: Build_Restore_and_Run_Unit_Tests
pool:
vmImage: ubuntu-latest
steps:
- task: SonarQubePrepare#5
displayName: SonarQube
inputs:
SonarQube: SonarQube
scannerMode: MSBuild
projectKey: $(sonarQubeProjectKey)
projectName: $(sonarQubeProjectName)
extraProperties: |
sonar.cs.opencover.reportsPaths=$(Agent.TempDirectory)/*/coverage.opencover.xml
- task: DotNetCoreCLI#2
displayName: Restore
inputs:
command: 'restore'
projects: 'src/**/*.csproj'
feedsToUse: 'select'
vstsFeed: '989127ec-b6ef-4c8c-ae54-90dec743281d'
- task: DotNetCoreCLI#2
displayName: Build
inputs:
command: build
projects: |
**/*.Test.csproj
arguments: '--configuration $(buildConfiguration) --no-restore'
- task: DotNetCoreCLI#2
displayName: Test
inputs:
command: test
projects: '**/*.Test.csproj'
nobuild: true
arguments: '--no-restore --collect:"XPlat Code Coverage" -- DataCollectionRunSettings.DataCollectors.DataCollector.Configuration.Format=cobertura,opencover'
publishTestResults: true
- task: PublishCodeCoverageResults#1
displayName: Publish code coverage
inputs:
codeCoverageTool: Cobertura
summaryFileLocation: $(Agent.TempDirectory)/*/coverage.cobertura.xml
- task: SonarQubeAnalyze#5
displayName: Run code analysis
- task: SonarQubePublish#5
displayName: Publish quality gate result
inputs:
pollingTimeoutSec: '300'
My attempts to fix this issue:
Currently I am using ubuntu-latest. I have tried a Windows image.
Applied time-outs.
Upgraded the SonarQube steps from major version 4 to 5.
Added an unique GUID to each projects csproj.
As Mickaël Caro stated on my similar post in the SonarSource community forum:
Unfortunately some characters are not recognized nor parsed correctly when we try to find the report path (We do have this issue filed here : [VSTS-240] Escape regex characters when finding report path file. - SonarSource).
I changed the [ ] in the name of the build to ( ), and the analysis runs smoothly.
Our team is implementing an Azure DevOps testing pipeline. After our initial commit to create the pipeline .yml file this error message was displayed. After looking into it, I realized I forgot to include the trigger in the .yml. However after adding, it this error message hasn't gone away. The pipeline is working as expected though, we are just using a manual trigger which is shown below. The only listed issue is from the our original commit. Is there a way I can acknowledge this error to make it go away or am I potentially missing a different error that I just haven't noticed yet? Thanks for any help in advance, please let me know if I can provide any additional information.
Here are the error messages that I am seeing when I view the runs of that pipeline. I also included a screen shot of how I'm setting up my trigger.
Edit: As request I included the actual .yml file code below with slight naming modifications. We do have some custom plugins such as creating files for files that are untracked but still needed to be created. So you might need to remove those to test this.
trigger:
- none
pool:
name: myPool
demands:
- msbuild
- visualstudio
steps:
- task: NuGetToolInstaller#0
displayName: 'Use NuGet 4.4.1'
inputs:
versionSpec: 4.4.1
- task: NuGetCommand#2
displayName: 'NuGet restore'
inputs:
restoreSolution: '$(Parameters.solution)'
- task: eliostruyf.build-task.custom-build-task.file-creator#6
displayName: 'Create Connection Strings file'
inputs:
filepath: '$(System.DefaultWorkingDirectory)/ID_Web/config/ConnectionStrings.config'
filecontent: |
<connectionStrings>
</connectionStrings>
endWithNewLine: true
- task: eliostruyf.build-task.custom-build-task.file-creator#6
displayName: 'Create Developer Settings File'
inputs:
filepath: '$(System.DefaultWorkingDirectory)/ID_Web/config/developerAppSettings.config'
filecontent: |
<appSettings>
</appSettings>
endWithNewLine: true
- task: eliostruyf.build-task.custom-build-task.file-creator#6
condition: contains(variables['Agent.Name'], '1')
displayName: 'Create Developer Integration Setting for agent 1'
inputs:
filepath: '$(System.DefaultWorkingDirectory)/ID_Test/config/developerIntegrationSettings.config'
filecontent: |
<developerIntegrationSettings>
<add key="ModelsIntegrationTestDb" value="Models_IntegrationTest_BuildAgent1"/>
<add key="ErrorsIntegrationTestDb" value="Errors_IntegrationTest_BuildAgent1"/>
</developerIntegrationSettings>
endWithNewLine: true
- task: VisualStudioTestPlatformInstaller#1
displayName: 'Visual Studio Test Platform Installer'
inputs:
versionSelector: latestStable
# Build the solution.
- task: VSBuild#1
displayName: 'Build solution'
inputs:
solution: '$(Parameters.solution)'
msbuildArgs: '/p:DeployOnBuild=true /p:WebPublishMethod=Package /p:PackageAsSingleFile=true /p:SkipInvalidConfigurations=true /p:PackageLocation="$(build.artifactstagingdirectory)\\"'
platform: '$(BuildPlatform)'
configuration: '$(BuildConfiguration)'
clean: true
# Run all unit tests in parallel
- task: VSTest#2
displayName: 'Run Unit Tests'
inputs:
testAssemblyVer2: |
**\*ID_Test*.dll
!**\*TestAdapter.dll
!**\obj\**
searchFolder: '$(System.DefaultWorkingDirectory)/ID_Test'
testFiltercriteria: '(FullyQualifiedName!~Integration & FullyQualifiedName!~Ioc)'
runOnlyImpactedTests: false
vsTestVersion: toolsInstaller
runSettingsFile: 'ID_Test/.runsettings'
runInParallel: true
runTestsInIsolation: false
codeCoverageEnabled: false
testRunTitle: 'Unit Tests'
platform: '$(BuildPlatform)'
configuration: '$(BuildConfiguration)'
diagnosticsEnabled: true
rerunFailedTests: true
# Run integration tests serially
- task: VSTest#2
displayName: 'Run Integration Tests'
inputs:
testAssemblyVer2: |
**\*ID_Test*.dll
!**\*TestAdapter.dll
!**\obj\**
searchFolder: '$(System.DefaultWorkingDirectory)/ID_Test'
testFiltercriteria: '(FullyQualifiedName~Integration | FullyQualifiedName~Ioc)'
runOnlyImpactedTests: false
vsTestVersion: toolsInstaller
runSettingsFile: 'ID_Test/.runsettings'
runTestsInIsolation: true
codeCoverageEnabled: false
testRunTitle: 'Integration Tests'
platform: '$(BuildPlatform)'
configuration: '$(BuildConfiguration)'
diagnosticsEnabled: true
rerunFailedTests: true
# Clean agent directories
- task: mspremier.PostBuildCleanup.PostBuildCleanup-task.PostBuildCleanup#3
displayName: 'Clean Agent Directories'
Edit (2): Included below is a screen shot of what I am using for trigger settings now, originally it was unchecked. Checking it doesn't seem to have any affect though.
I had the same and was about to curl up in a ball and cry when I found out the real issue. As the wonderful message is saying, it has absolutely nothing to do with trigger :)
I assume you created a new branch with your YAML file for testing purpose before merging it to master. So you need to setup your build to point to this branch because the file doesn't exist on your main branch.
Here the steps :
Edit your pipeline
Click the 3 dots on top right > Triggers
Click YAML tab > Get sources
Change the 'Default branch for manual and scheduled builds' to point to the branch where your .yml file is
So we gave up on this problem since it wasn't having any effect and we couldn't find the problem. After about a week or two it just stopped showing up. So I assume this was just some quirk with the Azure DevOps and not a problem with the pipeline itself.
According to your description, this issue looks more like an episodic issue. In YAML files, you don't have to include triggers. YAML pipelines are configured by default with a CI trigger on all branches. You can create a new pipeline and copy your YAML file to see if there are still any error messages.
Or, the issue could come from Classic UI triggers. On the pipeline editing page, select More actions-> Triggers.
Then you can check if there is anything illegal. If you want to use the trigger in the YAML file, leave the '
Override the YAML continuous integration trigger from here' check box off.
In a pipeline I have two different steps. The first one generates some files, the second should take these files as an input.
the Yaml for that pipeline is the following:
name: myscript
stages:
- stage: Tes/t
displayName: owasp-test
jobs:
- job: owasp_test
displayName: run beasic checks for site
pool:
name: default
demands: Agent.OS -equals Windows_NT
steps:
- task: DotNetCoreCLI#2
inputs:
command: 'build'
projects: '**/*.sln'
- task: dependency-check-build-task#5
inputs:
projectName: 'DependencyCheck'
scanPath: '**/*.dll'
format: 'JUNIT'
- task: PublishTestResults#2
inputs:
testResultsFormat: 'JUnit'
testResultsFiles: '**/*-junit.xml'
the dependency-check-build-task returns an XML-File:
File upload succeed.
Upload 'P:\Azure-Pipelines-Agent\_work\2\TestResults\dependency-check\dependency-check-junit.xml' to file container: '#/11589616/dependency-check'
Associated artifact 53031 with build 21497
The following step (PublishTestResults) SHOULD take that file but returns
##[warning]No test result files matching **/*-junit.xml were found.
instead. I can see that file in the artifact after the pipeline is run.
This is because your report is written to Common.TestResultsDirectory which is c:\agent_work\1\TestResults (for Microsoft Hosted agents), and publish test task looks in System.DefaultWorkingDirectory which is c:\agent_work\1\s.
Please try:
- task: PublishTestResults#2
inputs:
testResultsFormat: 'JUnit'
testResultsFiles: '**/*-junit.xml'
searchFolder: '$(Common.TestResultsDirectory)'
I had the same trouble:
I fixed changing the Agent Specification
I follow the steps of Computing code coverage for a .NET Core project with Azure DevOps and Coverlet.
Build run like expected and every step ends successfully.
The Artefact-Explorer shown the uploaded report and In summary I get the Code Coverage result.
But I missing the Code Coverage tab next to Tests tab to take a look to the detailed report.
Configuration YAML:
- task: NuGetToolInstaller#0
displayName: 'Use NuGet 5.0.2'
inputs:
versionSpec: 5.0.2
checkLatest: true
- task: NuGetCommand#2
displayName: 'NuGet restore'
inputs:
restoreSolution: '$(Parameters.solution)'
- task: VSBuild#1
displayName: 'Projektmappe **\*.sln erstellen'
inputs:
solution: '$(Parameters.solution)'
platform: '$(BuildPlatform)'
configuration: '$(BuildConfiguration)'
- task: VisualStudioTestPlatformInstaller#1
displayName: 'Installer für Visual Studio Test-Plattform'
enabled: false
- task: VSTest#2
displayName: 'VsTest - testAssemblies'
inputs:
testAssemblyVer2: |
**\$(BuildConfiguration)\*test*.dll
!**\obj\**
codeCoverageEnabled: true
platform: '$(BuildPlatform)'
configuration: '$(BuildConfiguration)'
- powershell: ./'D:\AzureDevOpsData\Skripte\PowerShell\CodeCoverage.ps1'
displayName: 'PowerShell Test Code Coverage'
- task: Palmmedia.reportgenerator.reportgenerator-build-release-task.reportgenerator#4
displayName: ReportGenerator
inputs:
reports: coverage.cobertura.xml
targetdir: '$(Build.SourcesDirectory)/CodeCoverage'
- task: PublishCodeCoverageResults#1
displayName: 'Code Coverage veröffentlichen von $(Build.SourcesDirectory)/CodeCoverage/Cobertura.xml'
inputs:
codeCoverageTool: Cobertura
summaryFileLocation: '$(Build.SourcesDirectory)/CodeCoverage/Cobertura.xml'
reportDirectory: '$(Build.SourcesDirectory)/CodeCoverage'
The PowerShell Script contains:
#TEST CSPROJ
$csproj = "FrameworkA_Tests"
#SEARCH TEST CSPROJ.DLL
"`nrun tests:"
$unitTestFile = gci -Recurse | ?{ $_.FullName -like "*bin\*$csproj.dll" }
Write-Host "`$unitTestFile value: $unitTestFile"
#GET COVERLET.EXE
$coverlet = "D:\AzureDevOpsData\Tools\coverlet\coverlet.exe"
#RUN COVERLET.EXE
"calling $coverlet for $($unitTestFile.FullName)"
&$coverlet $unitTestFile.FullName --target "dotnet" --targetargs "vstest $($unitTestFile.FullName) --logger:trx" --format "cobertura"
Do I forget something?
Please try to enable "Boards" in the project settings:
This issue was fixed for Azure DevOps but may still exist in Azure DevOps Server.
See:
https://developercommunity.visualstudio.com/content/problem/385331/code-coverage-results-not-rendered-in-build-result.html
https://developercommunity.visualstudio.com/content/problem/398209/build-results-tab-code-coverage-does-not-show-up.html
ReportGenerator missing Code Coverage tab (Azure DevOps Server 2019.0.1)
This should be a known issue on the Developer Community:
Code coverage tab missing in Azure DevOps Server
MS team reply: A fix for this issue has been internally implemented and is being prepared for release.
As workaround, you can try the method provided by jingzhu yan:
you can add copy files and publish build results steps , then you can
download coverage result file from Artifacts.
Hope this helps.
Added a Publish test results task in Azure DevOpsCI/CD pipeline, test were successfull, but after running the test it complaints about ##[warning]No test result files matching **/test-*.xml were found. Could someone please advise on how can we resolve similar problem ?
Publish Test Results task : configuration
Test result format= JUnit
Test results files= **/test-*.xml
Search folder = $(System.DefaultWorkingDirectory)
Test results title = Cypress Test Results
note: I have try adding the search folder path as follows: C:\agent_work\r5\a\drop\ui-tests\cypress
package.json to run the tests
"scripts": {
"test": "cypress run --record --key <key value here>"
}
My directory path in server:
C:\agent_work\r5\a\drop\ui-tests\cypress
My friend, I was facing the same issue on Azure DevOps.
In my case, the folder where the xml files were generated was reports on the root of the repo, that depends on how you got configured Junit on your cypress.json file
So In my case, the solution was changing this on azure-pipelines.yml
testResultsFiles: "results/*.xml"
searchFolder: $(System.DefaultWorkingDirectory)
So that's the entire setup of the testing pipeline
# Node.js
# Build a general Node.js project with npm.
# Add steps that analyze code, save build artifacts, deploy, and more:
# https://learn.microsoft.com/azure/devops/pipelines/languages/javascript
trigger:
- master
pool:
vmImage: 'ubuntu-latest'
steps:
- task: NodeTool#0
inputs:
versionSpec: '12.x'
displayName: 'Install Node.js'
- script: "npm i"
displayName: "Install project dependencies"
- script: "npm run cy:verify"
displayName: "Cypress Verify"
- script: "source cypress.env" # comment this script to run tests against production
displayName: "Using env variables to change url to test against development branch"
- script: "npm run cy:run-report"
displayName: "Run Cypress Tests"
- task: PublishBuildArtifacts#1
displayName: "Publish Artifact: cypress-azure-devops screenshots"
inputs:
PathtoPublish: cypress/screenshots
ArtifactName: "CypressAzureDevopsTestRunScreenshots"
condition: failed()
- task: PublishTestResults#2
displayName: "Publish Test Results"
condition: succeededOrFailed()
inputs:
testResultsFormat: "JUnit"
testResultsFiles: "results/*.xml"
searchFolder: $(System.DefaultWorkingDirectory)
mergeTestResults: true
testRunTitle: 'Test Results'
continueOnError: true
Saludos desde Argentina 🇦🇷