I'm trying to figure how to dynamically create a dependency and run the job based on the condition.
Here is the structure of my pipeline:
main.yaml:
stages:
- stage: build
jobs:
- template: build.yaml
- stage: deployDev
dependsOn: build
jobs:
- template: deployApp1.yaml
parameters:
environmentName: Dev
- template: deployApp2.yaml
parameters:
environmentName: Dev
- stage: deployQA
dependsOn: deployDev
jobs:
- template: promote.yaml
parameters:
environmentName: QA
- template: deployApp1.yaml
parameters:
environmentName: QA
- template: deployApp2.yaml
parameters:
environmentName: QA
promote.yaml
jobs:
- job: copy
steps:
- task:
deployApp1.yaml
jobs:
-job: deployApp1
steps:
- task:
deployApp2.yaml
jobs:
- job: deployApp2
steps:
- task:
In deployQA i have a separate job which copies the build artifacts and the next two jobs (deployApp1 and deployApp2) will fail without the copy step in deployQA.
I would like to create a conditional dependency on job: copy for job: deployApp1 so that it should be able to skip if i'm deploying to Dev which doesn't have this dependency. I already tried different solutions from different posts without any luck.
I know if i can add additional stage for the copy that would solve my problem but i would like to have the copy as part of the QA stage.
You want deployApp1 to depend on copy when running in stage deployQA and not depend on anything when running in stage deployDev?
You could add a dependsOn parameter to your template and use it to control job's dependencies:
stages:
- stage: build
jobs:
- template: build.yaml
- stage: deployDev
dependsOn: build
jobs:
- template: deployApp1.yaml
parameters:
environmentName: Dev
- template: deployApp2.yaml
parameters:
environmentName: Dev
- stage: deployQA
dependsOn: deployDev
jobs:
- template: promote.yaml
parameters:
environmentName: QA
- template: deployApp1.yaml
parameters:
environmentName: QA
dependsOn: copy
- template: deployApp2.yaml
parameters:
environmentName: QA
dependsOn: copy
# promote.yaml
jobs:
- job: copy
steps:
- task:
# deployApp1.yaml
parameters:
- name: environmentName
- name: dependsOn
default: []
jobs:
- job: deployApp1
dependsOn: ${{ parameters.dependsOn }}
steps:
- task:
# deployApp2.yaml
parameters:
- name: environmentName
- name: dependsOn
default: []
jobs:
- job: deployApp2
dependsOn: ${{ parameters.dependsOn }}
steps:
- task:
I was able to figure out the solution for my scenario from below post however to be able to achieve the answer to my specific use-case i had to set the makeExplicitDependency parameter to false.
${{ if eq(parameters.makeExplicitDependency, true) }}:
dependsOn: Test
thanks to below post:
How to dynamically reference previous jobs in Azure Pipelines if there are any in the current stage
Related
How can i loop over an array or through an object to create stages?
Below is a yml file that works. You can see the build stage loops over the parameters environments for jobs. IS it possible to achieve the same thing for the publishing stages?
The publishing stages require manual approval, must run in order and only when the previous stage is successfully complete?
parameters:
- name: 'environments'
type: object
default:
- environment: development
variableGroup: strata2-admin-spa-vg
dependsOn: 'build'
- environment: test
variableGroup: strata2-test-admin-spa-vg
dependsOn: 'development'
- environment: production
variableGroup: strata2-development-variables
dependsOn: 'development'
- name: 'buildTemplate'
type: string
default: buildTemplate.yml
- name: 'publishTemplate'
type: string
default: publishTemplate.yml
trigger:
- main
pool:
vmImage: ubuntu-latest
stages:
- stage: build
displayName: Build stage
jobs:
# Can I do this for stages?
- ${{each build in parameters.environments}}:
- template: ${{parameters.buildTemplate}}
parameters:
environment: ${{build.environment}}
variableGroup: ${{build.variableGroup}}
# How to loop over parameters.environments to dynamically create stages
- stage: Publish_Development
displayName: Publish development environment
dependsOn: build
jobs:
- template: ${{parameters.publishTemplate}}
parameters:
environment: Development_websites
variableGroup: strata2-admin-spa-vg
- stage: Publish_Test
displayName: Publish test environment
dependsOn: Publish_Dev
jobs:
- template: ${{parameters.publishTemplate}}
parameters:
environment: Test_websites
variableGroup: strata2-test-admin-spa-vg
- stage: Publish_Production
displayName: Publish production environment
dependsOn: Publish_Test
jobs:
- template: ${{parameters.publishTemplate}}
parameters:
environment: Production_websites
variableGroup: strata2-development-variables
You can create a stages object the same way you created the environments object.
stages:
Publish_Development:
- stage: Publish_Development
- displayName: Publish development environment
- dependsOn:
- ...
Publish_Test
- stage: Publish_Development
- ...
Then you can loop over the stages object like you did with environments.
- ${{each stage in parameters.stages}}:
- stage: ${{ stage.stage }}
displayName: ${{ stage.displayName}}
dependsOn: ${{ stage.dependsOn}}
...
Managed to get this working for myself. Stages automatically generated based on numerical batch numbers, that run in parallel. Hope it helps someone out there.
- name: batches
displayName: BATCH
type: object
default:
- 1
- 2
- 3
stages:
- ${{ each stage in parameters.batches }}:
- stage: BATCH_${{ stage }}
dependsOn: []
jobs:
- job: PREP
steps:
- template: install.yml
- job: RUN
dependsOn: PREP
timeoutInMinutes: 300
steps:
- template: run.yml
parameters:
batch: ${{ stage }}
Would be nice if the batch numbers weren't displayed as an editable box when running the pipeline from Azure DevOps. I tried setting them as fixed values, but couldn't get that to work, so this is what I went with in the end.
My problem
I try to create templated yaml Azure Devops pipeline:
parameters:
- name: envs
type: object
default:
- QUAL
- PROD
- PREPROD
stages:
- template: Orchestrator.yml
parameters:
name: envs
type: object
default:
- QUAL
- PROD
- PREPROD
This is my template:
parameters:
envs: {}
stages:
- ${{ each env in parameters.envs }}:
- stage: ${{ env }}
jobs:
- job: Deploy
steps:
- script: echo Deploy project
displayName: 'Deploy'
- job: Tests
steps:
- script: echo Unit tests
displayName: Test 1
But I get this error message:
An error occurred while loading the YAML build pipeline. The array
must contain at least one element. Parameter name: stages
OK, I modify my main script like this:
parameters:
- name: envs
type: object
default:
- QUAL
- PROD
- PREPROD
stages:
- stage: Build
jobs:
- job: Build
steps:
- script: echo Compilation completed...
displayName: 'Compile'
- template: Orchestrator.yml
parameters:
name: envs
type: object
default:
- QUAL
- PROD
- PREPROD
This time pipeline runs, but only first Job. The Template is not loaded.
What I need
I was able to make this scenario working with a single file script, but I would like to make it working with a templated script.
Is this scenario supported? How I can do?
thanks
Here I have a sample as reference:
In the template YAML file (here I name it template.yaml), write as this.
parameters:
- name: envs
type: object
default:
- QUAL
- PROD
- PREPROD
stages:
- ${{ each env in parameters.envs }}:
- stage: ${{ env }}
displayName: 'Stage ${{ env }}'
jobs:
- job: job1
displayName: 'Job 1'
steps:
- bash: echo "Current job is job1 in ${{ env }}"
- job: job2
displayName: 'Job 2'
steps:
- bash: echo "Current job is job2 in ${{ env }}"
In the pipeline YAML (here I name it pipeline.yaml), write as this.
trigger:
- main
extends:
template: template.yaml
Result
If you do not want to hardcode the value of the parameter 'envs' in template.yaml, you can write like as below.
In template.yaml write as this.
parameters:
- name: envs
type: object
default: []
stages:
- ${{ each env in parameters.envs }}:
- stage: ${{ env }}
displayName: 'Stage ${{ env }}'
jobs:
- job: job1
displayName: 'Job 1'
steps:
- bash: echo "Current job is job1 in ${{ env }}"
- job: job2
displayName: 'Job 2'
steps:
- bash: echo "Current job is job2 in ${{ env }}"
In pipeline.yaml write as this.
trigger:
- main
stages:
- template: template.yaml
parameters:
envs:
- QUAL
- PROD
- PREPROD
Result. Same as above.
To view more details, you can see "YAML schema reference".
I am trying to set up an azure yaml pipeline that uses two deployment templates for two respective jobs (test and deploy). The jobs should be run in sequence for each stage, as the test-job creates an artifact that the deploy-job consumes. This works nicely.
However, for one environment I have split the deployment into two stages, one stage that only runs tests and one stage that runs the actual deployment. The problem I run into is that my deploy-job has a "dependson" which references the test-job. Because of this, my pipeline is invalid when I add the these two stages for my environment.
My question is if there is some sort of dynamic "dependson" reference for a job, something like "depends on any previous jobs in this stage, if there are any"?
Snippets of code from the yaml-files below for reference:
Main.yaml
#This first stage would work as it uses both templates in succession
stages:
- stage: BothTemplates
displayName: 'Run both templates'
jobs:
- template: TemplateTest.yml
- template: TemplateDeploy.yml #depends on the job in TemplateTest.yml
# The pipeline is invalid because of OnlyDeploy, as the TemplateDeploy.yml depends in the job "Test", which does not exist in the OnlyDeploy-context
- stage: OnlyTest
dependsOn: BothTemplates
displayName: 'Run only test template'
jobs:
- template: TemplateTest.yml
- stage: OnlyDeploy
dependsOn: OnlyTest
displayName: 'Run only deploy template'
jobs:
- template: TemplateDeploy.yml
TemplateTest.yaml
jobs:
- deployment: Test
dependsOn:
displayName: Test
continueOnError: false
strategy:
runOnce:
deploy:
steps:
#Here the steps for the tests are
Current TemplateDeploy.yaml
jobs:
- deployment: Deploy
dependsOn: Test
displayName: Deploy
continueOnError: false
strategy:
runOnce:
deploy:
steps:
#Here the steps for the deployment are
My idea would be to change TemplateDeploy.yaml to something like this:
jobs:
- deployment: Deploy
dependsOn: previousJob() #Wait until previous job in stage has finished, if there are any
displayName: Deploy
continueOnError: false
strategy:
runOnce:
deploy:
steps:
#Here the steps for the deployment are
Since you already have dependencies between stages
- stage: OnlyTest
dependsOn: BothTemplates
displayName: 'Run only test template'
jobs:
- template: TemplateTest.yml
- stage: OnlyDeploy
dependsOn: OnlyTest
displayName: 'Run only deploy template'
jobs:
- template: TemplateDeploy.yml
I think that you actually don't need here dependency to a job on TemplateDeploy.yml but if you want to make this dependent on previous jobs you can achieve this using parameters
parameters:
- name: makeExplicitDependency
displayName: 'Make excplicit job dependency'
type: boolean
default: true
jobs:
- deployment: Deploy
${{ if eq(parameters.makeExplicitDependency, true) }}:
dependsOn: Test
displayName: Deploy
continueOnError: false
strategy:
runOnce:
deploy:
steps:
#Here the steps for the deployment are
and then:
stages:
- stage: BothTemplates
displayName: 'Run both templates'
jobs:
- template: TemplateTest.yaml
- template: TemplateDeploy.yaml #depends on the job in TemplateTest.yml
- stage: OnlyTest
dependsOn: BothTemplates
displayName: 'Run only test template'
jobs:
- template: TemplateTest.yaml
- stage: OnlyDeploy
dependsOn: OnlyTest
displayName: 'Run only deploy template'
jobs:
- template: TemplateDeploy.yaml
parameters:
makeExplicitDependency: false
So removing dependsOn works like your expected dependsOn: previousJob() #Wait until previous job in stage has finished, if there are any
How to trigger by branch to use specific template under "stages"?
trigger:
branches
include:
- ci
- prod
stages:
template: ci.yml
condition: and(eq(['build.sourceBranch'], 'ci'))
template: prod.yml
condition: and(eq(['build.sourceBranch'], 'prod'))
Tried above condition but didn't work. I was getting "unexpected value condition". Any help is appreciated
***** Tried one of the solution as by passing condition as parameter to the template:
stages:
template: ci.yml
parameters:
condition: and(succeeded(), eq(variables['Build.SourceBranch'], 'refs/heads/ci'))
template: prod.yml
parameters:
condition: and(succeeded(), eq(variables['Build.SourceBranch'], 'refs/heads/prod'))
Getting "unexpected parameter condition"
Pipeline structure:
master.yml (contains runtime parameters)
stages:
template: ci.yml
parameters:
condition: and(succeeded(), eq(variables['Build.SourceBranch'], 'refs/heads/ci'))
template: prod.yml
parameters:
condition: and(succeeded(), eq(variables['Build.SourceBranch'], 'refs/heads/prod'))
ci.yml
stages:
stage: BuildApp
stage: BuildWeb
stage: DeployLocal
prod.yml
stages:
stage: BuildApp
stage: BuildWeb
stage: DeployProd
How to trigger by branch to use specific template under "stages"?
To resolve this issue, we could add the condition on the job level, like:
stages:
- stage: Test1
jobs:
- job: ci
displayName: ci
pool:
name: MyPrivateAgent
condition: and(succeeded(), eq(variables['Build.SourceBranch'], 'refs/heads/ci'))
steps:
- template: ci.yml
- job: prod
displayName: prod
pool:
name: MyPrivateAgent
condition: and(succeeded(), eq(variables['Build.SourceBranch'], 'refs/heads/prod'))
steps:
- template: prod.yml
Check the document Specify conditions for some more details.
On the other hand, we could also set the condition as parameter to the template yml, like:
- template: ci.yml
parameters:
doTheThing: and(succeeded(), eq(variables['Build.SourceBranch'], 'refs/heads/ci'))
The template yml file:
# template.yml
parameters:
doTheThing: 'false'
steps:
- script: echo This always happens!
displayName: Always
- script: echo Sometimes this happens!
condition: ${{ parameters.doTheThing }}
displayName: Only if true
You could check the thread YAML - Support conditions for templates for some more details.
Hope this helps.
I use Azure DevOps Templates in Stage and I want some job to start only when Job from template is completed (dependsOn):
- stage: stage1
jobs:
- job: job1
steps:
- bash: |
...
- template: template1.yml
parameters:
param1: 'val1'
- job: job2
**dependsOn: how to put `template: template1.yml` here?**
steps:
- bash: |
...
How could it be done?
Building on Eric Smith's answer you can pass in the name of the job that the template will depend on as a parameter.
#template1.yml
jobs:
- job: mytemplateJob1
steps:
- script: npm install
#template2.yml
parameters:
DependsOn: []
jobs:
- job: mytemplateJob2
dependsOn: ${{ parameters.DependsOn }}
steps:
- bash: pwd
By setting the default value for DependsOn to [] you ensure that the template will run if no value is passed in for DependsOn but you can optionally create a dependency like this:
stages:
- stage: stage1
jobs:
- template: template1.yml # Template reference
- template: template2.yml
parameters:
DependsOn: 'mytemplateJob1'
You can accomplish this by using the name of the job, as it is defined in your template in the dependsOn.
#template1.yml
jobs:
- job: mytemplateJob
steps:
- script: npm install
and
stages:
- stage: stage1
jobs:
- job: job1
steps:
- bash: pwd
- template: template1.yml # Template reference
parameters:
param: 'val1'
- job: job2
dependsOn: mytemplateJob
steps:
- bash: pwd