File from previous step cannot be found in Azure DevOps-Pipeline - azure-devops

In a pipeline I have two different steps. The first one generates some files, the second should take these files as an input.
the Yaml for that pipeline is the following:
name: myscript
stages:
- stage: Tes/t
displayName: owasp-test
jobs:
- job: owasp_test
displayName: run beasic checks for site
pool:
name: default
demands: Agent.OS -equals Windows_NT
steps:
- task: DotNetCoreCLI#2
inputs:
command: 'build'
projects: '**/*.sln'
- task: dependency-check-build-task#5
inputs:
projectName: 'DependencyCheck'
scanPath: '**/*.dll'
format: 'JUNIT'
- task: PublishTestResults#2
inputs:
testResultsFormat: 'JUnit'
testResultsFiles: '**/*-junit.xml'
the dependency-check-build-task returns an XML-File:
File upload succeed.
Upload 'P:\Azure-Pipelines-Agent\_work\2\TestResults\dependency-check\dependency-check-junit.xml' to file container: '#/11589616/dependency-check'
Associated artifact 53031 with build 21497
The following step (PublishTestResults) SHOULD take that file but returns
##[warning]No test result files matching **/*-junit.xml were found.
instead. I can see that file in the artifact after the pipeline is run.

This is because your report is written to Common.TestResultsDirectory which is c:\agent_work\1\TestResults (for Microsoft Hosted agents), and publish test task looks in System.DefaultWorkingDirectory which is c:\agent_work\1\s.
Please try:
- task: PublishTestResults#2
inputs:
testResultsFormat: 'JUnit'
testResultsFiles: '**/*-junit.xml'
searchFolder: '$(Common.TestResultsDirectory)'

I had the same trouble:
I fixed changing the Agent Specification

Related

Azure Devops - Can not retrieve artifact

I am publishing a artifact in a job (which completes successfully). In a separate job, I am trying to retrieve the artifact. However, nothings shows in the ls $(Build.ArtifactStagingDirectory). Why is this?
- task: PublishPipelineArtifact#1
displayName: 'Publish Artifact'
inputs:
targetpath: 'anatomy/dist'
artifact: 'anatomy-artifact'
publishLocation: 'pipeline'
- job: deployBucket
displayName: 'Deploy Bucket'
dependsOn: buildAnatomy
condition: eq(dependencies.buildAnatomy.result,'Succeeded')
pool:
vmImage: "ubuntu-20.04"
steps:
- task: Bash#3
displayName: "Upload contents to bucket"
inputs:
targetType: "inline"
script: |
ls -al '$(Build.ArtifactStagingDirectory)'
Each job runs in its own space and the agent cleans up after itself by default. If you want the contents of the artifact to be available in the 2nd job, you need to fetch it again.
Use the DownloadPipelineArtifact#1 task at the start of your 2nd job.
Or move the deploy command into the same job.

SonarQube: No analysis found in this build after applying include pattern

I am trying to integrate SonarQube in a CI-pipeline in Azure DevOps. I used the exact configuration that I used in other pipelines where everything runs smoothly. However, this pipeline can't find the analysis in the SonarQube Publish step as shown in the logs below.
##[debug] /home/vsts/work/_temp/sonar (directory)
##[debug] /home/vsts/work/_temp/sonar/Company.Service-CI[sonarqube-test][121226][2022-01-19] (directory)
##[debug] /home/vsts/work/_temp/sonar/Company.Service-CI[sonarqube-test][121226][2022-01-19]/9731b11b-6af8-d211-752d-e8fe0770156c (directory)
##[debug] /home/vsts/work/_temp/sonar/Company.Service-CI[sonarqube-test][121226][2022-01-19]/9731b11b-6af8-d211-752d-e8fe0770156c/report-task.txt (file)
##[debug]4 results
##[debug]found 4 paths
##[debug]applying include pattern
##[debug]adjustedPattern: '/home/vsts/work/_temp/sonar/Company.Service-CI[sonarqube-test][121226][2022-01-19]/**/report-task.txt'
##[debug]0 matches
##[debug]0 final results
##[debug][SQ] Searching for sonar/Company.Service-CI[sonarqube-test][121226][2022-01-19]/**/report-task.txt - found 0 file(s)
##[warning]No analyses found in this build! Please check your build configuration.
I find this problem hard to solve because it seems like the analysis report is there but after applying the include pattern there are zero matches. The pipeline contains the following steps:
stages:
- stage:
jobs:
- job: Build_Restore_and_Run_Unit_Tests
pool:
vmImage: ubuntu-latest
steps:
- task: SonarQubePrepare#5
displayName: SonarQube
inputs:
SonarQube: SonarQube
scannerMode: MSBuild
projectKey: $(sonarQubeProjectKey)
projectName: $(sonarQubeProjectName)
extraProperties: |
sonar.cs.opencover.reportsPaths=$(Agent.TempDirectory)/*/coverage.opencover.xml
- task: DotNetCoreCLI#2
displayName: Restore
inputs:
command: 'restore'
projects: 'src/**/*.csproj'
feedsToUse: 'select'
vstsFeed: '989127ec-b6ef-4c8c-ae54-90dec743281d'
- task: DotNetCoreCLI#2
displayName: Build
inputs:
command: build
projects: |
**/*.Test.csproj
arguments: '--configuration $(buildConfiguration) --no-restore'
- task: DotNetCoreCLI#2
displayName: Test
inputs:
command: test
projects: '**/*.Test.csproj'
nobuild: true
arguments: '--no-restore --collect:"XPlat Code Coverage" -- DataCollectionRunSettings.DataCollectors.DataCollector.Configuration.Format=cobertura,opencover'
publishTestResults: true
- task: PublishCodeCoverageResults#1
displayName: Publish code coverage
inputs:
codeCoverageTool: Cobertura
summaryFileLocation: $(Agent.TempDirectory)/*/coverage.cobertura.xml
- task: SonarQubeAnalyze#5
displayName: Run code analysis
- task: SonarQubePublish#5
displayName: Publish quality gate result
inputs:
pollingTimeoutSec: '300'
My attempts to fix this issue:
Currently I am using ubuntu-latest. I have tried a Windows image.
Applied time-outs.
Upgraded the SonarQube steps from major version 4 to 5.
Added an unique GUID to each projects csproj.
As Mickaël Caro stated on my similar post in the SonarSource community forum:
Unfortunately some characters are not recognized nor parsed correctly when we try to find the report path (We do have this issue filed here : [VSTS-240] Escape regex characters when finding report path file. - SonarSource).
I changed the [ ] in the name of the build to ( ), and the analysis runs smoothly.

Azure pipeline - unzip artefact, copy one directory into Azure blob store YAML file

I am getting stuck with Azure pipelines.
I have an existing node SPA project that needs built for each environment (TEST and PRODUCTION). This i can do, but need to have a manual step when pushing to PROD. I am using Azure Dev-op pipeline environments with Approval and Checks to mandate this.
The issue is using a 'deploy job' to take an artefact from a previous step I am unable to find the right directory. This is my YAML file have so far:
variables:
# Agent VM image name
vmImageName: 'ubuntu-latest'
trigger:
- master
# Don't run against PRs
pr: none
stages:
- stage: Development
displayName: Devlopment stage
jobs:
- job: install
displayName: Install and test
pool:
vmImage: $(vmImageName)
steps:
- task: NodeTool#0
inputs:
versionSpec: '12.x'
displayName: 'Install Node.js'
- script: |
npm install
displayName: Install node modules
- script: |
npm run build
displayName: 'Build it'
# Build creates a ./dist folder. The contents will need to be copied to blob store
- task: ArchiveFiles#2
inputs:
rootFolderOrFile: '$(Build.BinariesDirectory)'
includeRootFolder: true
archiveType: 'zip'
archiveFile: '$(Build.ArtifactStagingDirectory)/$(Build.BuildId).zip'
replaceExistingArchive: true
verbose: true
- deployment: ToDev
environment: development
dependsOn: install
strategy:
runOnce:
deploy:
steps:
- task: DownloadPipelineArtifact#2
inputs:
buildType: 'current'
targetPath: '$(Pipeline.Workspace)'
- task: ExtractFiles#1
inputs:
archiveFilePatterns: '**/*.zip'
cleanDestinationFolder: true
destinationFolder: './cpDist/'
# Somehow within a deploy job retrieve the .zip artefact, unzip, copy the ./dist folder into the blob store
- task: AzureCLI#2
inputs:
azureSubscription: MYTEST-Development
scriptLocation: "inlineScript"
scriptType: "bash"
inlineScript: |
az storage blob upload-batch -d \$web --account-name davey -s dist --connection-string 'DefaultEndpointsProtocol=https;AccountName=davey;AccountKey=xxxxxxx.yyyyyyyyy.zzzzzzzzzz;EndpointSuffix=core.windows.net'
displayName: "Copy build files to Development blob storage davey"
- script: |
pwd
ls
cd cpDist/
pwd
ls -al
displayName: 'list'
- bash: echo "Done"
If you are confused with the folder path, you could add few debug steps to check the location of know system variables to understand what was going on using a powershell script as below:
- task: PowerShell#2
displayName: 'Degug parameters'
inputs:
targetType: Inline
script: |
Write-Host "$(Build.ArtifactStagingDirectory)"
Write-Host "$(System.DefaultWorkingDirectory)"
Write-Host "$(System.ArtifactsDirectory)"
Write-Host "$(Pipeline.Workspace)"
Write-Host "$(System.ArtifactsDirectory)"
You should simply publish the build generated artifacts to drop folder.
Kindly check this official doc -- Artifact selection , in there is explaining that you can define the path which to download the artifacts to with the following task:
steps:
- download: none
- task: DownloadPipelineArtifact#2
displayName: 'Download Build Artifacts'
inputs:
patterns: '**/*.zip'
path: '$(Build.ArtifactStagingDirectory)'
Please be aware that the download happens automatically to $(Pipeline.Workspace), so if you don’t want you deployment to download the files twice, you need to specify the “download: none” in your steps.

Azure DevOps Multi Stage Pipeline Error: No package found with specified pattern: /home/vsts/work/1/s/**/*.zip - How do I fix?

I have an Azure DevOps Build (yaml) and Release Pipeline (Classic) successfully deploying to Azure.
I am trying to convert these 2 separate steps in a Multi Stage Yaml Pipeline.
On the Azure App Service Deploy task (AzureRmWebAppDeployment#4), I am getting the following error:
No package found with specified pattern: /home/vsts/work/1/a/*.zip
Below is my Multi Stage Yaml Pipeline
stages:
- stage: Build
jobs:
- job: 'Build'
pool:
vmImage: 'windows-latest'
variables:
buildConfiguration: 'Release'
steps:
- task: DotNetCoreCLI#2
displayName: Restore
inputs:
command: restore
projects: '**/*.csproj'
vstsFeed: 'dd55642d-8943-411f-8856-9714dd0da8af'
- task: DotNetCoreCLI#2
displayName: Build
inputs:
projects: '**/*.csproj'
arguments: '--configuration $(buildConfiguration)'
- task: DotNetCoreCLI#2
displayName: Test
inputs:
command: test
projects: '**/*[Tt]ests/*.csproj'
arguments: '--configuration $(buildConfiguration)'
- task: DotNetCoreCLI#2
displayName: Publish
inputs:
command: publish
publishWebProjects: false
projects: '**/Tools.Client.Blazor.ServerApp.csproj'
arguments: '--configuration $(buildConfiguration) --output $(build.artifactstagingdirectory)'
- task: PublishSymbols#2
displayName: 'Publish symbols path'
inputs:
SearchPattern: '**\bin\**\*.pdb'
PublishSymbols: false
continueOnError: true
- task: CopyFiles#2
displayName: 'Copy Files to: $(build.artifactstagingdirectory)\AzureDeploy'
inputs:
SourceFolder: AzureDeploy
TargetFolder: '$(build.artifactstagingdirectory)\AzureDeploy'
- task: PublishBuildArtifacts#1
displayName: 'Publish Artifact: drop'
inputs:
PathtoPublish: '$(build.artifactstagingdirectory)'
condition: succeededOrFailed()
- stage: Systest
jobs:
- job: 'Systest'
variables:
resourceGroupName: '$(appName)-rg-$(environment)'
location: 'East US'
appServiceName: '$(appName)-svc-$(environment)'
appInsightsName: '$(appName)-ins-$(environment)'
appServicePlanName: '$(appName)-asp-$(environment)'
appName: 'tools'
owner: 'Pod'
environment: 'systest'
steps:
- task: AzureResourceManagerTemplateDeployment#3
displayName: 'ARM Template deployment: Resource Group scope'
inputs:
azureResourceManagerConnection: 'Dev/Test Connection'
subscriptionId: ''
resourceGroupName: '$(resourceGroupName)'
location: '$(location)'
csmFile: '$(System.DefaultWorkingDirectory)/AzureDeploy/Tools.azureDeploy.json'
csmParametersFile: '$(System.DefaultWorkingDirectory)/AzureDeploy/Tools.azureDeploy.parameter.json'
overrideParameters: '-appServiceName "$(appServiceName)" -appInsightsName "$(appInsightsName)" -appServicePlanName "$(appServicePlanName)" -owner "$(owner)" -environment "$(environment)" -location "$(location)"'
- task: AzureRmWebAppDeployment#4
displayName: 'Azure App Service Deploy: $(appServiceName)'
inputs:
ConnectionType: 'AzureRM'
azureSubscription: ''
appType: 'webApp'
WebAppName: '$(appServiceName)'
packageForLinux: '$(Build.ArtifactStagingDirectory)/*.zip'
Any help / suggestions would be appreciated.
Because it's 2 stages the second stage doesn't have the file you published in the first stage, you need to download it.
You can use Pipeline artifacts instead of build artifacts.
Pipeline artifacts provide a way to share files between stages in a
pipeline or between different pipelines. They are typically the output
of a build process that needs to be consumed by another job or be
deployed. Artifacts are associated with the run they were produced in
and remain available after the run has completed.
To publish (upload) an artifact for the current run:
steps:
- publish: $(build.artifactstagingdirectory)
artifact: drop
And in the second stage, you download the artifact:
steps:
- download: current
artifact: drop
You can also achieve it with build artifacts and download with DownloadBuildArtifacts#0 task.
During Publish it will not work like this. Instead of using path "/home/vsts/work/1/a/.zip", this path can be used "$(System.DefaultWorkingDirectory)/_Releasepipelinename/drop/.zip"

Issue in YAMEL file

I am using Azure DevOps to Build and deploy my Azure Synapse
I have two YAML files. First one works well and I need to separate Build and Release steps. because of that I wrote second YAML file. Second YAML file has this error :
##[error]No files were found to deploy with search pattern
d:\a\1\s\SQL_ASynapse\bin\Release\SQL_ASynapse.dacpacCheck out how to troubleshoot failures at
https://aka.ms/sqlazuredeployreadme#troubleshooting-
First YAML file that works well:
trigger:
- master
pool:
vmImage: 'windows-latest'
variables:
solution: '**/*.sln'
buildPlatform: 'Any CPU'
buildConfiguration: 'Release'
steps:
- task: NuGetToolInstaller#1
- task: NuGetCommand#2
inputs:
restoreSolution: '$(solution)'
- task: AzureKeyVault#1
inputs:
azureSubscription: 'XXX'
KeyVaultName: 'XXX-Dev'
SecretsFilter: '*'
- task: VSBuild#1
inputs:
solution: '$(solution)'
platform: '$(buildPlatform)'
configuration: '$(buildConfiguration)'
- task: VSTest#2
inputs:
platform: '$(buildPlatform)'
configuration: '$(buildConfiguration)'
- task: SqlAzureDataWarehouseDacpacDeployment#1
inputs:
azureSubscription: 'XXX'
AuthenticationType: 'server'
ServerName: 'XXX'
DataWarehouse: '$(SynapseName)'
SqlUsername: '$(SynapseSQLUsername)'
SqlPassword: '$(SynapseSQLPassword)'
deployType: 'DacpacTask'
DeploymentAction: 'Publish'
DacpacFile: 'SQL_ASynapse\bin\Release\SQL_ASynapse.dacpac'
IpDetectionMethod: 'AutoDetect'
Second YAML file who has error:
trigger:
- master
variables:
solution: '**/*.sln'
buildPlatform: 'Any CPU'
buildConfiguration: 'Release'
# Agent VM image name
vmImageName: 'windows-latest'
stages:
- stage: Build
displayName: Build stage
jobs:
- job: Build
displayName: Build
pool:
vmImage: $(vmImageName)
steps:
- task: NuGetToolInstaller#1
- task: NuGetCommand#2
inputs:
restoreSolution: '$(solution)'
- task: VSBuild#1
inputs:
solution: '$(solution)'
platform: '$(buildPlatform)'
configuration: '$(buildConfiguration)'
- task: VSTest#2
inputs:
platform: '$(buildPlatform)'
configuration: '$(buildConfiguration)'
- stage: Deploy
displayName: Deploy stage
dependsOn: Build
condition: succeeded()
jobs:
- deployment: Deploy
displayName: Deploy
environment: 'development'
pool:
vmImage: $(vmImageName)
strategy:
runOnce:
deploy:
steps:
- task: AzureKeyVault#1
inputs:
azureSubscription: 'XXX-Dev'
KeyVaultName: 'XXX-Dev'
SecretsFilter: '*'
- task: SqlAzureDataWarehouseDacpacDeployment#1
inputs:
azureSubscription: 'XXX-Dev'
AuthenticationType: 'server'
ServerName: 'XXX'
DataWarehouse: '$(SynapseName)'
SqlUsername: '$(SynapseSQLUsername)'
SqlPassword: '$(SynapseSQLPassword)'
deployType: 'DacpacTask'
DeploymentAction: 'Publish'
DacpacFile: 'SQL_ASynapse\bin\Release\SQL_ASynapse.dacpac'
IpDetectionMethod: 'AutoDetect'
UPDATE #1 FOR #Levi Lu-MSFT ANSWER
I have changed my second script like below code :
...
- task: VSBuild#1
inputs:
solution: '$(solution)'
platform: '$(buildPlatform)'
configuration: '$(buildConfiguration)'
- publish: $(system.defaultworkingdirectory)\SQL_ASynapse\bin\Release ## path to a file or folder
artifact: MyBuildOutputs
...
- download: current # refers to artifacts published by current pipeline
artifact: MyBuildOutputs
- task: SqlAzureDataWarehouseDacpacDeployment#1
inputs:
azureSubscription: 'xxx-Dev'
AuthenticationType: 'server'
ServerName: 'xxx.database.windows.net'
DataWarehouse: '$(SynapseName)'
SqlUsername: '$(SynapseSQLUsername)'
SqlPassword: '$(SynapseSQLPassword)'
deployType: 'DacpacTask'
DeploymentAction: 'Publish'
DacpacFile: 'SQL_ASynapse\bin\Release\SQL_ASynapse.dacpac'
IpDetectionMethod: 'AutoDetect'
But I have still same error.
Upload task result
Download task result
SqlAzureDataWarehouseDacpacDeployment error :
##[error]No files were found to deploy with search pattern
d:\a\1\s\SQL_ASynapse\bin\Release\SQL_ASynapse.dacpacCheck out how to
troubleshoot failures at https://aka.ms/sqlazuredeployreadme#troubleshooting-
If you use Microsoft-hosted agent. Every job defined in the yaml pipeline will run on a fresh new virtual machine. The virtual machine is discarded after one use.
So that the build artifacts from Build job of Build stage doesnot exist on the agent machine of the deploy job of Deploy stage. That's why you got above error.
You need to add a publish task after Vsbuild task in Build job to publish the build results to azure devops server. You should specify the correct path(where the build artifacts reside) to publish.
- publish: $(system.defaultworkingdirectory)\SQL_ASynapse\bin\Release ## path to a file or folder
artifact: MyBuildOutputs
Then add a download task before qlAzureDataWarehouseDacpacDeployment task in Deploy job to download the build results to deploy job agent. So that the build results will be available to the deployment tasks.
- download: current # refers to artifacts published by current pipeline
artifact: MyBuildOutputs
To publish and download build artifacts, You can also use Publish Build Artifacts task, Download Build Artifacts task.
Update:
When you use download task: See here for more information.
By default, files are downloaded to $(Pipeline.Workspace)/{artifact}, where artifact is the name of the artifact. The folder structure of the artifact is always preserved.
And you can see from above updated screenshot. The artifacts is downloaded to folder D:\a\1\MyBuildOutputs
So you should set DacpacFile attribute as below:
DacpacFile : $(Pipeline.Workspace)/MyBuildOutputs/SQL_ASynapse.dacpac