I have a number of jobs for different platforms I'd like to run in parallel. I'd like to build a different set of platforms for different situations (i.e. full build, smoke, pull request, etc.). How can I make a list of jobs dynamic based on variables?
For example, if this is one of the "hard-coded" implementations:
jobs:
- job: Platform1
pool: Pool1
steps:
- template: minimal_template.yml
parameters:
BuildTarget: Platform1
- job: Platform2
pool: Pool1
steps:
- template: minimal_template.yml
parameters:
BuildTarget: Platform2
- job: Platform3
pool: Pool2
steps:
- template: minimal_template.yml
parameters:
BuildTarget: Platform3
How could I instead extract out a collection of variable sets, i.e.
[[Platform1, Pool1], [Platform2, Pool1], [Platform3, Pool2]]
And execute that on a pipeline like:
jobs:
??(Foreach platform in platforms)??
- job: $(platform[0])
pool: $(platform[1])
steps:
- template: minimal_template.yml
parameters:
BuildTarget: $(platform[0])
You can define it in the parameters and loop it:
parameters:
- name: Platforms
type: object
default:
- name: 'Platform1'
pool: 'Platform1Pool'
- name: 'Platform2'
pool: 'Platform2Pool'
jobs:
- ${{ each platform in parameters.Platforms}}:
- job: ${{ platform.name }}
pool: ${{ platform.pool }}
steps:
- template: minimal_template.yml
You may alos use 'jobList' type for template parameters:
parameters:
- name: 'testsJobsList'
type: jobList
default: []
jobs:
- ${{ each job in parameters.testsJobsList }}: # Each job
- ${{ each pair in job }}: # Insert all properties other than "steps"
${{ if ne(pair.key, 'steps') }}:
${{ pair.key }}: ${{ pair.value }}
steps: # Wrap the steps
- ${{ job.steps }} # Users steps
And then:
trigger:
- none
pool:
vmImage: 'windows-latest'
jobs:
- template: deployment-template.yml
parameters:
testsJobsList:
- job: Platform1
pool: Platform1Pool
steps:
- template: minimal_template.yml
- job: Platform2
pool: Platform2Pool
steps:
- template: minimal_template.yml
You’re looking for conditions: https://learn.microsoft.com/en-us/azure/devops/pipelines/process/conditions?view=azure-devops&tabs=yaml
Stages, jobs, and steps can all have a condition defined.
For example, running a job if a variable is set to true:
condition: eq(variables['System.debug'], 'true')
Related
How do I do the condition part in the code below under "Approval_Test" stage? I have to generate the condition dynamically , can't hardcode it due to some requirements. The stage will also be generated dynamically based on the parameters param.
"Approval_Test" stage can only be run after all solution1, 2 and 3 are finished and/or skipped
#azure-pipeline.yml
trigger: none
#Package Parameter
parameters:
- name: "params"
type: object
default:
Solution1:
name: "Solution1"
Solution2:
name: "Solution2"
Solution3:
name: "Solution3"
stages:
- ${{ each param in parameters.params }}:
- stage: Deploy_dev_${{ param.value.name }}
jobs:
- template: deploy-dev.yml
- stage: Approval_Test
dependsOn:
- ${{ each param2 in parameters.params }}:
- Deploy_dev_${{ param2.value.name }}
condition: |
#The "each" below would not work and throw errors.
and
(
- ${{ each param2 in parameters.params }}:
in(dependencies.Deploy_dev_${{ param2.value.name }}.result, 'Succeeded', 'SucceededWithIssues', 'Skipped'),\
)
jobs:
- deployment: Approval
environment: 'sandbox'
This was a tough nut to crack and I wasn't able to solve it with the each keyword, but please let me present a feasible alternative.
Alternate solution: Job status check functions
Without your requirement of also having the skipped stages the solution would be adding condition: succeeded() to the stage: Approval_Test :
But this doesn't work for a skipped stage.
Not Failed?
Unfortunately there is no 'Skipped' to check from the job status functions, but what if we take the opposite of failed(), not(failed())!?
The works out fine:
#azure-pipeline.yml
trigger: none
#Package Parameter
parameters:
- name: "params"
type: object
default:
Solution1:
name: "Solution1"
Solution2:
name: "Solution2"
Solution3:
name: "Solution3"
stages:
- ${{ each param in parameters.params }}:
- stage: Deploy_dev_${{ param.value.name }}
jobs:
- template: deploy-dev.yml
- stage: skipped # test to simulate a skip from the template
condition: failed()
jobs:
- job: concat
steps:
- ${{ each parameter in parameters.params }}:
- script: echo Deploy_dev_${{ parameter.value.name }}.result
- stage: Approval_Test
condition: not(failed())
dependsOn:
- skipped # test to simulate a skip from the template
- ${{ each param2 in parameters.params }}:
- Deploy_dev_${{ param2.value.name }}
jobs:
- deployment: Approval
environment: 'sandbox'
(Don't) use the equivalent
According to the docs failed() is a equivalent for eq(variables['Agent.JobStatus'], 'Failed').
Would eq(variables['Agent.JobStatus'], 'Succeeded', 'SucceededWithIssues', 'Skipped') not be the solution then?
No!
Why I don't understand, but in my tests (above) with the yaml (below) it wouldn't fly.
#azure-pipeline.yml
trigger: none
#Package Parameter
parameters:
- name: "params"
type: object
default:
Solution1:
name: "Solution1"
Solution2:
name: "Solution2"
Solution3:
name: "Solution3"
stages:
- ${{ each param in parameters.params }}:
- stage: Deploy_dev_${{ param.value.name }}
jobs:
- template: deploy-dev.yml
- stage: skipped # test to simulate a skip from the template
condition: failed()
jobs:
- job: concat
steps:
- ${{ each parameter in parameters.params }}:
- script: echo Deploy_dev_${{ parameter.value.name }}.result
- stage: Approval_Test
condition: in(variables['Agent.JobStatus'], 'Succeeded', 'SucceededWithIssues', 'Skipped')
dependsOn:
- skipped # test to simulate a skip from the template
- ${{ each param2 in parameters.params }}:
- Deploy_dev_${{ param2.value.name }}
jobs:
- deployment: Approval
environment: 'sandbox'
Conclusion
If not(failed()) works for you, I advice you to use that.
Can anybody point me in the right direction here?
When I attempt to kick off the pipeline from the main yaml template the reference template gives me the unexpected 'steps' value.
I attempted to add a stage before the defined job and then receive the unexpected 'stage' message. Looking at some similar previously asked questions I understand a step can't be place directly under a 'stage' but this is not the case here.
parameters:
- name: baseEnvironmentName
type: string
- name: environmentSuffix
type: string
- name: postDeploySteps
type: stepList
default: []
- name: publishedPackageName
type: string
- name: serviceName
type: string
- name: dnsHostName
type: string
jobs:
- deployment: ${{ parameters.environmentSuffix }}Deployment
displayName: '${{ parameters.environmentSuffix }} Deployment'
pool:
vmImage: 'windows-2019'
environment: ${{ parameters.baseEnvironmentName }}-${{ parameters.environmentSuffix }}
variables:
- template: variables/variables.common.yaml
- template: variables/variables.${{ parameters.environmentSuffix }}.yaml
- name: MonitorExceptionAlert_Name
value: '${{ variables.IdPrefix }}-${{ parameters.environmentSuffix }}-${{ parameters.dnsHostName }}-ExceptionAlert'
steps:
- checkout: self
- checkout: common_iac
- task: Random Task Name
displayName: 'Create Environment Resource Group'
inputs:
ConnectedServiceName: '$(ADOS.ServiceConnectionName)'
resourceGroupName: '$(ResourceGroup.Name)'
location: '$(ResourceGroup.Location)'
condition: and(succeeded(), ne(variables['CreateResourceGroupInTaskGroup'], 'false'))
- task: Random Task Name 2
displayName: 'Create Application Insights'
inputs:
ConnectedServiceName: '$(ADOS.ServiceConnectionName)'
ResourceGroupName: '$(ResourceGroup.Name)'
Location: '$(ResourceGroup.Location)'
instanceName: '$(ApplicationInsights.Name)'
You are using deployment jobs. Try defining a strategy. Try replacing steps: inline with deployment with:
- deployment: ....
strategy:
runOnce:
deploy:
steps:
From the example at the Azure docs templates page, jobs: should contain a - job: which contains steps:. I don't know that jobs: can directly contain steps:.
Is there a way to return a variable from a template so I can use that as a parameter for other templates?
I have a yml file that looks something like this (actual names redacted).
- stage: A
jobs:
- template: a.yml
parameters:
environment: foo
group: bar
- template: b.yml
parameters:
dependsOn: a
environment: foo
group: bar
- template: c.yml
parameters:
dependsOn: a
environment: foo
group: bar
Each template is a deployment.
I want a.yml to set a variable, then for that to be returned so that I could use that variable as a parameter for the other 2 templates. How can I return a variable using 'a.yml'?
You can use depands to pass parameters between two templates just as passing parameters between stages. Here is an example. I use environment as the parameters to be passed between a.yml and b.yml.
a.yml:
parameters:
environment: foo
group: bar
jobs:
- job: deployA
variables:
env: ${{ parameters.environment }}
steps:
- pwsh: |
echo "##vso[task.setvariable variable=env;isOutput=true]$($env:ENV)"
name: outputVar
b.yml:
parameters:
dependsOn: a
environment: foo
group: bar
jobs:
- job: deployB
dependsOn: deployA
steps:
...
pipeline.yml:
trigger: none
pool:
vmImage: ubuntu-latest
stages:
- stage: A
jobs:
- template: a.yml
parameters:
environment: Test
- template: templateB.yml
parameters:
environment: $[dependencies.deployA.outputs['outputVar.env']]
I am trying to execute an Azure Devops Job on a specific pool based on a condition.
The goal is to switch between self-hosted agent and microsoft agent.
Here is the configuration:
parameters:
custom_agent: true
jobs:
- job: Test
displayName: Test job
- ${{ if eq(parameters.custom_agent, true) }}:
- pool:
name: mypool
demands:
- agent.os -equals Linux
- ${{ if eq(parameters.custom_agent, false) }}:
- pool:
vmImage: 'ubuntu-latest'
steps:
- task: npmAuthenticate#0
Any ideas ?
The below example solved my requirement
parameters:
- name: 'vmImage'
type: string
default: 'ubuntu-latest'
- name: 'agentPool'
type: string
default: ''
jobs:
- job: 'Example'
pool:
${{ if ne(parameters.agentPool, '') }}:
name: ${{ parameters.agentPool }}
${{ if eq(parameters.agentPool, '') }}:
vmImage: ${{ parameters.vmImage }}
steps:
- script: example
Another apporach to conditionally select pools if you use non-vm pools:
variables:
- ${{ if eq(parameters.custom_agent, true) }}:
- name: testJobPool
value: mypool
- ${{ if eq(parameters.custom_agent, false) }}:
- name: testJobPool
value: mypool_second
jobs:
- job: Test
displayName: Test job
pool:
name: $(testJobPool)
steps:
- task: npmAuthenticate#0
This has proved working.
We can specify conditions under which a step, job, or stage will run. We can configure the jobs in the pipeline with different condition entries, and set demands based on those conditions.
A skeleton version looks like this:
parameters:
- name: custom_agent
displayName: Pool Image
type: boolean
default: True
jobs:
- job: selfhostedagent
condition: eq(${{ parameters.custom_agent }}, True)
displayName: 'self_hosted agent'
pool:
name: Default
demands:
- Agent.Name -equals WS-VITOL-01
steps:
- script: echo self_hosted agent
- job: hostedagent
condition: eq(${{ parameters.custom_agent }}, False)
displayName: 'hosted agent'
pool:
vmImage: 'ubuntu-latest'
steps:
- script: echo hosted agent
Update1
In addition, we can configure task template, then use the template in the steps.
Result:
It looks like pool is not a valid property of a job type
Try switching your job to a deployment type:
jobs:
- deployment: Test
- ${{ if eq(parameters.custom_agent, true) }}:
pool:
name: mypool
demands:
agent.os -equals Linux
strategy:
runOnce:
deploy:
steps:
Currently in Azure pipelines, we can pass the list of jobs to be executed to a child template with a parameter of jobList type as shown in the doco.
https://learn.microsoft.com/en-us/azure/devops/pipelines/process/templates?view=azure-devops#iterative-insertion
Is there a way I can encapsulate these jobs from pipeline.yml that are being passed to the jobList param inside another job template and pass that template to a jobList param. I tried to structure my pipeline as follows:
pipeline.yml
deployment-template.yml
post-deploy-tests-dev.yml
post-deploy-smoke-tests-prod.yml
I would like to dynamically insert different tests' jobs to the end of the deployment template depending on the environment. I tried the jobList type parameter in the deployment-template.yml as the following but it throws an error saying mapping not expected.
#post-deploy-tests-dev.yml
jobs:
- job: Test1
steps:
- script: execute test1
#post-deploy-tests-smoke-tests-prod.yml
jobs:
- job: Test2
steps:
- script: execute test2
#pipeline.yml
...
- template: deployment-template.yml
parameters:
environment: dev
testsJobsList:
template: post-deploy-tests-dev.yml
- template: deployment-template.yml
parameters:
environment: prod
testsJobsList:
template: post-deploy-smoke-tests-prod.yml
#deployment-template.yml
parameters:
- name: testsJobsList
type: jobList
default: []
#All deployment jobs here
jobs:
...
...
#Tests as the end
- ${{ parameters.testsJobsList }}
Is there a way to dynamically pass the jobList ?
it throws an error saying mapping not expected.
Test with the YAML sample, the cause of this issue: you are missing - before the template field in pipeline.yml file (- template: post-deploy-tests-dev.yml and - template: post-deploy-smoke-tests-prod.yml). In this position, template is equivalent to a job, and you need to add -.
Here is my Sample:
pipeline.yml
trigger:
- none
pool:
vmImage: 'windows-latest'
jobs:
- template: deployment-template.yml
parameters:
testsJobsList:
- template: post-deploy-tests-dev.yml
- template: deployment-template.yml
parameters:
testsJobsList:
- template: post-deploy-smoke-tests-prod.yml
deployment-template.yml
parameters:
- name: 'testsJobsList'
type: jobList
default: []
jobs:
- ${{ each job in parameters.testsJobsList }}: # Each job
- ${{ each pair in job }}: # Insert all properties other than "steps"
${{ if ne(pair.key, 'steps') }}:
${{ pair.key }}: ${{ pair.value }}
steps: # Wrap the steps
- ${{ job.steps }} # Users steps
post-deploy-tests-dev.yml
jobs:
- job: Test1
steps:
- script: echo test1
post-deploy-smoke-tests-prod.yml
jobs:
- job: Test2
steps:
- script: echo test2
Result: