Azure Pipeline with multiple drop folders in YAML - azure-devops

I have create a YAML pipeline for Azure deployment. There are many templates, but I will only show the master pipeline to illustrate my issue.
Basically
the first stage is to build from repository source.
the next stage is pre-deployment followed by deployment
The build drops the output files to a drop folder. During pre-deployment some of these files go through some transformations (replacing tokens with values according to target environment).
The problem is that currently there is only one drop folder, so you can see the problem coming .... If I deploy to DEV, the files are transformed using the DEV values. But then if I deploy to INT, the files are already transformed and I end up deploying to INT files with DEV values.
It get worse if DEV and INT deployment run at the same time...
So how can I use separate drop folder per environment ? In predeployment, should I copy the drop folder to another location before transformation. In which case, how do I specify the new location in the deployment stages ?
Here's the master pipeline for reference :
trigger:
- master
pool:
name: Default
demands:
- npm
- msbuild
- visualstudio
stages:
- stage: build
displayName: 'Build & Test stage'
jobs:
- template: templates/pipeline-build/master.yml
parameters:
buildConfiguration: 'Release'
dropFolder: '\\srvbuild\DROP'
- stage: deployDev
displayName: 'Deploy Dev Stage'
dependsOn: build
condition: succeeded()
jobs:
- deployment: deploymentjob
displayName: deployment job
environment: dev
variables:
- template: variables/dev.yml
strategy:
runOnce:
preDeploy:
steps:
- template: templates/pipeline-predeploy/master.yml
deploy:
steps:
- template: templates/pipeline-deploy/master.yml
- stage: deployInt
displayName: 'Deploy Int Stage'
dependsOn: build
condition: succeeded()
jobs:
- deployment: deploymentjob
displayName: deployment job
environment: int
variables:
- template: variables/int.yml
strategy:
runOnce:
preDeploy:
steps:
- template: templates/pipeline-predeploy/master.yml
deploy:
steps:
- template: templates/pipeline-deploy/master.yml

As workaround, you can publish the build artifact to A file share, and then download the build artifact through the Download Fileshare Artifacts task in each stage to transform
it separately.
- task: PublishPipelineArtifact#1
displayName: 'Publish Pipeline Artifact'
inputs:
artifact: drop
publishLocation: filepath
fileSharePath: '***'
Use this task to download fileshare artifacts:
- task: DownloadFileshareArtifacts#1
inputs:
filesharePath:
artifactName:
#itemPattern: '**' # Optional
#downloadPath: '$(System.ArtifactsDirectory)'
#parallelizationLimit: '8' # Optional

Related

Azure DevOps Pipeline with environments get stuck

I created an environment and registered a virtual machine (on prem) as a resource
Whenever I try to run the deployment in the resources, the pipeline gets stuck in the deployment stage. Inside of the job, the only log that I see is JOb is pending...
This is the relevant section of the pipeline:
- stage: deployInTest
displayName: Deploy in Test Envs
dependsOn: build
jobs:
- deployment: Deploy
displayName: "Deploy in Test"
environment:
name: 'Development'
resourceType: VirtualMachine
strategy:
runOnce:
deploy:
steps:
- task: DownloadBuildArtifacts#1
inputs:
buildType: 'current'
downloadType: 'single'
artifactName: 'frontEnd'
downloadPath: '$(System.ArtifactsDirectory)'
Note that if I change this to the following yaml, the stage runs, but it tries to execute the task IISWebAppManagementOnMachineGroup#0 on the deployment server (OnPrem too) where IIS is not installed.
- stage: deployInTest
displayName: Deploy in Test Envs
dependsOn: build
jobs:
- deployment: Deploy
displayName: "Deploy in dev3"
environment: "Development"
strategy:
runOnce:
deploy:
steps:
- task: DownloadBuildArtifacts#1
inputs:
buildType: 'current'
downloadType: 'single'
artifactName: 'frontEnd'
downloadPath: '$(System.ArtifactsDirectory)'
Ok, so after trying something that was in the back on my mind since yesterday it worked. I saw another post in SO (I can't find it right now) that had the same issue, but they had 2 deployment jobs.
Their solution was to give them unique names i.e. 'DeployInTest' instead of deploy.
For me changing
- stage: deployInTest
displayName: Deploy in Test Envs
dependsOn: build
jobs:
- deployment: Deploy
displayName: "Deploy in Test"
Into
- stage: deployInTest
displayName: Deploy in Test Envs
dependsOn: build
jobs:
- deployment: DeployInTest #<-- this is what changed
displayName: "Deploy in Test"
Did the trick. I just realized that I was even trying that before writing the question. I'll edit the question to show the actual status with which it was not working

How to dynamically reference previous jobs in Azure Pipelines if there are any in the current stage

I am trying to set up an azure yaml pipeline that uses two deployment templates for two respective jobs (test and deploy). The jobs should be run in sequence for each stage, as the test-job creates an artifact that the deploy-job consumes. This works nicely.
However, for one environment I have split the deployment into two stages, one stage that only runs tests and one stage that runs the actual deployment. The problem I run into is that my deploy-job has a "dependson" which references the test-job. Because of this, my pipeline is invalid when I add the these two stages for my environment.
My question is if there is some sort of dynamic "dependson" reference for a job, something like "depends on any previous jobs in this stage, if there are any"?
Snippets of code from the yaml-files below for reference:
Main.yaml
#This first stage would work as it uses both templates in succession
stages:
- stage: BothTemplates
displayName: 'Run both templates'
jobs:
- template: TemplateTest.yml
- template: TemplateDeploy.yml #depends on the job in TemplateTest.yml
# The pipeline is invalid because of OnlyDeploy, as the TemplateDeploy.yml depends in the job "Test", which does not exist in the OnlyDeploy-context
- stage: OnlyTest
dependsOn: BothTemplates
displayName: 'Run only test template'
jobs:
- template: TemplateTest.yml
- stage: OnlyDeploy
dependsOn: OnlyTest
displayName: 'Run only deploy template'
jobs:
- template: TemplateDeploy.yml
TemplateTest.yaml
jobs:
- deployment: Test
dependsOn:
displayName: Test
continueOnError: false
strategy:
runOnce:
deploy:
steps:
#Here the steps for the tests are
Current TemplateDeploy.yaml
jobs:
- deployment: Deploy
dependsOn: Test
displayName: Deploy
continueOnError: false
strategy:
runOnce:
deploy:
steps:
#Here the steps for the deployment are
My idea would be to change TemplateDeploy.yaml to something like this:
jobs:
- deployment: Deploy
dependsOn: previousJob() #Wait until previous job in stage has finished, if there are any
displayName: Deploy
continueOnError: false
strategy:
runOnce:
deploy:
steps:
#Here the steps for the deployment are
Since you already have dependencies between stages
- stage: OnlyTest
dependsOn: BothTemplates
displayName: 'Run only test template'
jobs:
- template: TemplateTest.yml
- stage: OnlyDeploy
dependsOn: OnlyTest
displayName: 'Run only deploy template'
jobs:
- template: TemplateDeploy.yml
I think that you actually don't need here dependency to a job on TemplateDeploy.yml but if you want to make this dependent on previous jobs you can achieve this using parameters
parameters:
- name: makeExplicitDependency
displayName: 'Make excplicit job dependency'
type: boolean
default: true
jobs:
- deployment: Deploy
${{ if eq(parameters.makeExplicitDependency, true) }}:
dependsOn: Test
displayName: Deploy
continueOnError: false
strategy:
runOnce:
deploy:
steps:
#Here the steps for the deployment are
and then:
stages:
- stage: BothTemplates
displayName: 'Run both templates'
jobs:
- template: TemplateTest.yaml
- template: TemplateDeploy.yaml #depends on the job in TemplateTest.yml
- stage: OnlyTest
dependsOn: BothTemplates
displayName: 'Run only test template'
jobs:
- template: TemplateTest.yaml
- stage: OnlyDeploy
dependsOn: OnlyTest
displayName: 'Run only deploy template'
jobs:
- template: TemplateDeploy.yaml
parameters:
makeExplicitDependency: false
So removing dependsOn works like your expected dependsOn: previousJob() #Wait until previous job in stage has finished, if there are any

Azure DevOps yaml pipeline : how to download only specific artifacts of a referenced pipeline?

By referencing another pipeline in a YAML pipeline's resources all artifacts published by the referenced pipeline get automatically downloaded. I'm not sure how to stop this behavior and download only the needed artifacts. Adding a download task for only the needed artifacts does not stop the initial download of the full set of artifacts.
So what you need is disabling default behavior as
Artifacts are only downloaded automatically in deployment jobs. In a regular build job, you need to explicitly use the download step keyword or Download Pipeline Artifact task.
To stop artifacts from being downloaded automatically, add a download step and set its value to none:
steps:
- download: none
and then add additional step to download specific artifact.
Here is an example:
resources:
pipelines:
- pipeline: MultipleArtifact
project: 'DevOps Manual'
source: 'kmadof.devops-manual (64)'
jobs:
- job: Build
pool:
vmImage: 'ubuntu-latest'
steps:
- script: echo Hello, world!
displayName: 'Run a one-line script'
- script: |
echo Add other tasks to build, test, and deploy your project.
echo See https://aka.ms/yaml
displayName: 'Run a multi-line script'
# Track deployments on the environment.
- deployment: DeployWeb
displayName: deploy Web App
pool:
vmImage: 'Ubuntu-16.04'
# Creates an environment if it doesn't exist.
environment: 'smarthotel-dev'
strategy:
# Default deployment strategy, more coming...
runOnce:
deploy:
steps:
- download: none
- download: MultipleArtifact
artifact: art-1
- checkout: self
- script: echo my first deployment
To download a specific artifact rather than all artifacts, you can include the following:
steps:
- download: current
artifact: 'Artifact-Name'
Below is a complete example comparing the default behaviour with download: current and download: none.
Here is a screenshot from Azure pipelines showing correct number of Download steps in each case.
jobs:
- job: Create_Two_Artifacts
steps:
- bash: echo "test" >> file.txt
- task: PublishBuildArtifacts#1
inputs:
pathToPublish: 'file.txt'
artifactName: 'Artifact1'
- task: PublishBuildArtifacts#1
inputs:
pathToPublish: 'file.txt'
artifactName: 'Artifact2'
- deployment: Download_All_Deployment
environment: MyEnvironment
dependsOn: [Create_Two_Artifacts]
strategy:
runOnce:
deploy:
steps: []
- deployment: Download_SpecificArtefact_Deployment
environment: MyEnvironment
dependsOn: [Create_Two_Artifacts]
strategy:
runOnce:
deploy:
steps:
- download: current
artifact: 'Artifact1'
- deployment: Download_None_Deployment
environment: MyEnvironment
dependsOn: [Create_Two_Artifacts]
strategy:
runOnce:
deploy:
steps:
- download: none

Azure Pipelines Deploy Stage Failing without Error

The deploy stage of the pipeline fails without error after build stage completes successfully.
Enabling system diagnostics does not give in any additional information (see the screenshot below).
The following pipelines yaml file was used:
trigger:
- master
resources:
- repo: self
variables:
vmImageName: 'ubuntu-latest'
stages:
- stage: Build
displayName: Build stage
jobs:
- job: Build
displayName: Build
pool:
vmImage: $(vmImageName)
steps:
- task: CmdLine#2
inputs:
script: |
ls -la
- stage: Deploy
displayName: Deploy Notebook Instance Stage
dependsOn: Build
jobs:
- deployment: Deploy
displayName: Deploy
pool:
vmImage: $(vmImageName)
environment: 'myenv.default'
strategy:
runOnce:
deploy:
steps:
- task: CmdLine#2
inputs:
script: echo Some debug text!
I used your script and I change only environment as I don't have myenv.default and all is fine.
Please double check your environment setting.

How do you download *all* pipeline resources in a job?

The deployment job automatically downloads all the pipeline resources. However, standard job does not. I tried to use - download: current, but that does not download the pipeline resources.
The reason I want to do this is to simulate a deployment for GitOps. The simulation will include a step that does a git diff that shows the differences for review.
However, I don't see an all or * option in https://learn.microsoft.com/en-us/azure/devops/pipelines/yaml-schema?view=azure-devops&tabs=schema%2Cparameter-schema#download
My current workaround is to do a deployment to a "Temp" environment prior to the real deployment.
UPDATE:
Here's an example of what I have tried
resources:
pipelines:
- pipeline: auth
project: myproj
source: auth CI
trigger:
branches:
include:
- master
...
jobs:
- job: diagnostics
displayName: Job Diagnostics
steps:
- checkout: self
# - download:
- task: DownloadPipelineArtifact#2
displayName: 'Download Pipeline Artifact'
inputs:
path: $(Build.SourcesDirectory)
- bash: |
env | sort
displayName: Display environment variables
- bash: |
pwd
displayName: Present working directory
- bash: |
find $(Pipeline.Workspace) -type f -print
displayName: Display files
UPDATE:
Another approach I was mulling is to create a pipeline that creates another pipeline. That way the list of pipeline resources does not need to be Azure Pipeline YAML, it could be a CSV or a simplified YAML that I transform in which case I can generate
resources:
pipelines:
- pipeline: pipelineAlias1
project: myproj
source: auth CI
branch: master
trigger:
branches:
include:
- master
- pipeline: pipelineAlias2
project: myproj
source: auth CI
branch: master
trigger:
branches:
include:
- master
...
job:
steps:
- download: pipelineAlias1
- download: pipelineAlias2
and then set that up as another Azure pipeline to execute when updated.
How do you download all pipeline resources in a job?
Indeed, this is a known issue about using the download keyword to download the Pipeline Artifacts.
You could track this issue from below ticket:
Artifacts do not download on multi stage yaml build using DownloadPipelineArtifactV2
To resolve this issue, please try to use the DownloadPipelineArtifact task instead of the download keyword:
- task: DownloadPipelineArtifact#2
displayName: 'Download Pipeline Artifact'
inputs:
path: $(Build.SourcesDirectory)
Update:
I have noticed your yml file, it seems you do not add build job in your pipeline. The task DownloadPipelineArtifact is used to:
download pipeline artifacts from earlier stages in this pipeline,
or from another pipeline
So, we need to add stage to build the pipeline to generate the artifact, otherwise, no pipeline artifacts were downloaded.
Check my test YAML file:
variables:
ArtifactName: drop
stages:
- stage: Build
jobs:
- job: Build
displayName: Build
pool:
name: MyPrivateAgent
steps:
- task: CopyFiles#2
displayName: 'Copy Files to: $(build.artifactstagingdirectory)'
inputs:
SourceFolder: '$(System.DefaultWorkingDirectory)\LibmanTest'
targetFolder: '$(build.artifactstagingdirectory)'
- task: PublishBuildArtifacts#1
displayName: 'Publish Artifact: LibmanTest'
inputs:
ArtifactName: $(ArtifactName)
- stage: Dev
dependsOn: Build
jobs:
- job: Dev
displayName: Dev
pool:
name: MyPrivateAgent
steps:
- task: CopyFiles#2
displayName: 'Copy Files to: $(build.artifactstagingdirectory)'
inputs:
SourceFolder: '$(System.DefaultWorkingDirectory)\TestSample'
targetFolder: '$(build.artifactstagingdirectory)'
- task: PublishBuildArtifacts#1
displayName: 'Publish Artifact: TestSample'
inputs:
ArtifactName: $(ArtifactName)
- stage: Deployment
dependsOn: Dev
pool:
name: MyPrivateAgent
jobs:
- deployment: Deployment
displayName: DeployA
environment: 7-1-0
strategy:
runOnce:
deploy:
steps:
- task: DownloadPipelineArtifact#2
displayName: 'Download Pipeline Artifact'
inputs:
path: $(Build.SourcesDirectory)
As you can see, I use two stages Build and Dev to copy the project LibmanTest and TestSample as artifact, then use the task DownloadPipelineArtifact to download those two artifacts.
The test result:
Update2:
your example still does not show resources.pripelines
So, now you want to download the artifact from other pipelines, you need not use the default configuration for the task DownloadPipelineArtifact:
resources:
pipelines:
- pipeline: xxx
project: MyTestProject
source: xxx
trigger:
branches:
include:
- master
jobs:
- deployment: Deployment
displayName: DeployA
environment: 7-1-0
strategy:
runOnce:
deploy:
steps:
- task: DownloadPipelineArtifact#2
displayName: 'Download Pipeline Artifact For Test'
inputs:
buildType: specific
project: MyTestProject
definition: 13
The test result:
Besides, you could check the configuration from the classic editor, then get the YAML file:
Hope this hellps.