Can't run a templated yaml Azure Devops pipeline with dynamically created stage - azure-devops

My problem
I try to create templated yaml Azure Devops pipeline:
parameters:
- name: envs
type: object
default:
- QUAL
- PROD
- PREPROD
stages:
- template: Orchestrator.yml
parameters:
name: envs
type: object
default:
- QUAL
- PROD
- PREPROD
This is my template:
parameters:
envs: {}
stages:
- ${{ each env in parameters.envs }}:
- stage: ${{ env }}
jobs:
- job: Deploy
steps:
- script: echo Deploy project
displayName: 'Deploy'
- job: Tests
steps:
- script: echo Unit tests
displayName: Test 1
But I get this error message:
An error occurred while loading the YAML build pipeline. The array
must contain at least one element. Parameter name: stages
OK, I modify my main script like this:
parameters:
- name: envs
type: object
default:
- QUAL
- PROD
- PREPROD
stages:
- stage: Build
jobs:
- job: Build
steps:
- script: echo Compilation completed...
displayName: 'Compile'
- template: Orchestrator.yml
parameters:
name: envs
type: object
default:
- QUAL
- PROD
- PREPROD
This time pipeline runs, but only first Job. The Template is not loaded.
What I need
I was able to make this scenario working with a single file script, but I would like to make it working with a templated script.
Is this scenario supported? How I can do?
thanks

Here I have a sample as reference:
In the template YAML file (here I name it template.yaml), write as this.
parameters:
- name: envs
type: object
default:
- QUAL
- PROD
- PREPROD
stages:
- ${{ each env in parameters.envs }}:
- stage: ${{ env }}
displayName: 'Stage ${{ env }}'
jobs:
- job: job1
displayName: 'Job 1'
steps:
- bash: echo "Current job is job1 in ${{ env }}"
- job: job2
displayName: 'Job 2'
steps:
- bash: echo "Current job is job2 in ${{ env }}"
In the pipeline YAML (here I name it pipeline.yaml), write as this.
trigger:
- main
extends:
template: template.yaml
Result
If you do not want to hardcode the value of the parameter 'envs' in template.yaml, you can write like as below.
In template.yaml write as this.
parameters:
- name: envs
type: object
default: []
stages:
- ${{ each env in parameters.envs }}:
- stage: ${{ env }}
displayName: 'Stage ${{ env }}'
jobs:
- job: job1
displayName: 'Job 1'
steps:
- bash: echo "Current job is job1 in ${{ env }}"
- job: job2
displayName: 'Job 2'
steps:
- bash: echo "Current job is job2 in ${{ env }}"
In pipeline.yaml write as this.
trigger:
- main
stages:
- template: template.yaml
parameters:
envs:
- QUAL
- PROD
- PREPROD
Result. Same as above.
To view more details, you can see "YAML schema reference".

Related

YAML Extend Stage Template

Looking to try and get some help figuring out a solution to what seems like a simple task. I am trying to develop some extend YAML templates for security within our ADO pipelines but first I have to get passed this error I am experiencing.
The error being returned by ADO when I try to validate or run the pipeline is
/ADO_Stage_Restrictions_Dev.yml#AdoRestrictions (Line: 7, Col: 3): Unexpected value job
Working Extend Template YAML
This template validates and executes without issue, which to me means I am successfully passing the stages object into the extends template
parameters:
- name: stageObjs
type: stageList
default: []
stages:
- ${{ each stage in parameters.stageObjs }}:
${{ stage }}
Broken Extend Template YAML
This template does not validate and throws the 'Unexpected value job' exception. based on the stage schema I would assume that I would be able to loop the jobs property within the stage.
parameters:
- name: stageObjs
type: stageList
default: []
stages:
- ${{ each stage in parameters.stageObjs }}:
- ${{ each job in stage.jobs }}:
${{ job }}
Build YAML
The main yaml file that extends stages
resources:
repositories:
- repository: self
type: git
ref: refs/heads/Development
- repository: AdoRestrictions
type: git
name: utl-yaml-templates
ref: refs/heads/main
trigger: none
pool:
name: PROD
extends:
template: ADO_Stage_Restrictions_Dev.yml#AdoRestrictions
parameters:
stageObjs:
- stage: 'BuildStage'
displayName: 'Build Test'
jobs:
- job: 'BuildJob'
displayName: 'Build'
steps:
- task: PowerShell#2
displayName: 'Hello World'
inputs:
targetType: inline
script: |
Write-Host "Hello World"
There's nothing in your yaml which defines the start of the stage, or declares the start of the list of jobs; that's why it wasn't expecting to find 'job' there. You can add those parts in, like this:
parameters:
- name: stageObjs
type: stageList
default: []
stages:
- ${{ each stage in parameters.stageObjs }}:
- stage: ${{ stage.stage }}
displayName: ${{ stage.displayName }}
jobs:
- ${{ each job in stage.jobs }}:
${{ job }}

Azure Devops Pipeline dynamic created stages to run in order/sequence rather parallel

I am currently working on azure pipeline with multiple stages created dynamically based on the input parameter(s), and want to run the stage2_stg_x in sequence rather than parallel(currently it runs in parallel). I couldn't found any possible solution to get that achieved.
Could someone suggest here.
main pipeline : test.yml
trigger: none
pool:
name: 'linuxagent'
parameters:
- name: appComponents
displayName: YAML list of Components to Build and Deploy
type: object
default:
- stg_a
- stg_b
- stg_c
- stg_d
- stg_e
stages:
- template: pipeline/stages/stage1.yml
- ${{ each appComponents in parameters.appComponents }}:
- template: pipeline/stages/stage2.yml
parameters:
appComponents: ${{ appComponents }}
stage1.yml
stages:
- stage: stage1
dependsOn: []
displayName: stage1
jobs:
- job:
steps:
- bash: |
# Write your commands here
echo "Hello world"
echo "i am here"
stage2.yml
parameters:
- name: appComponents
displayName: "Component name"
type: object
stages:
- stage: stage2_${{ replace(parameters.appComponents, '-', '_') }}
dependsOn: stage1
displayName: stage2 For ${{ parameters.appComponents }}
jobs:
- job:
steps:
- bash: |
# Write your commands here
echo "Hello world"
echo "i am here"
Note : Here i have just using the basic echo for testing purpose. But my actual pipeline has different logic.
Try removing "dependsOn" from all your stages.
I have a similar setup and the stages all just depend on the stage before them.

Selecting Azure DevOps service connection based on if else logic for tenant selection

I am trying to select a service connection based on if else logic.
I have following two pipelines azure-caller.yml and template.yaml.
azure-caller.yml
parameters:
- name: directory
displayName: 'select appropriate environment directory'
values:
- Dev
- Stage
stages:
- template: job-templates/template.yml
parameters:
selectdirectory: $(select-directory)
template.yml
parameters:
selectdirectory: default
stages:
- stage: A
variables:
- name: service-connection
${{ if eq(parameters.selectdirectory, 'Dev') }}:
value: Dev-Service-connection
${{ elseif eq(parameters.selectdirectory, 'Stage') }}:
value: Stage-Service-connection
jobs:
- job: Install_terraform
displayName: Intialize Terraform
continueOnError: true
steps:
- checkout: self
- task: TerraformInstaller#0
inputs:
terraformVersion: '1.0.5'
- job: Validate_terraform_Plan
displayName: Validate Terraform
continueOnError: true
dependsOn: Install_terraform
steps:
- checkout: self
- task: TerraformTaskV2#2
inputs:
provider: 'azurerm'
command: 'init'
backendServiceArm: ${{ variables.service-connection }}
backendAzureRmResourceGroupName: 'any-rg'
backendAzureRmStorageAccountName: 'any-storage'
backendAzureRmContainerName: 'statefile'
backendAzureRmKey: abcd.tfstate
- task: TerraformTaskV2#2
displayName: 'plan'
inputs:
provider: 'azurerm'
command: 'plan'
commandOptions: '-out="plan.out"'
environmentServiceNameAzureRM: ${{ variables.service-connection }}
My tasks are failing because it fetches the value of the service connection at compile time, which is empty.
Azure Caller is a wrapper/main pipeline, and the template is the only one that is allowed to use Dev or Stage service connections for deployment purposes to achieve security. Is there a way to accomplish this?
Your input will be helpful. thx
It is not supported to dynamically specify value of pipeline variable conditionally.
In the YMAL pipeline, you can use the if conditional to judge using or not using a pipeline variable. You can not use it to judge specifying a value or another one to a pipeline variable.
However, you can dynamically set the input conditionally on a task. See "Conditionally set a task input".
You also can dynamically specify value of parameter conditionally.
Below is an example to dynamically specify value of parameter conditionally. You can reference it to change your YAML files.
template.yml
parameters:
- name: ServiceConnection
type: string
stages:
- stage: A
displayName: 'Stage A'
jobs:
- job: A1
displayName: 'Job A1'
pool:
vmImage: ubuntu-latest
steps:
- task: Bash#3
displayName: 'Show parameter value'
inputs:
targetType: inline
script: echo "The value of ServiceConnection is ${{ parameters.ServiceConnection }}"
azure-pipelines.yml
parameters:
- name: directory
displayName: 'Select appropriate environment directory:'
type: string
values:
- Dev
- Stage
stages:
- template: template.yml
parameters:
${{ if eq(parameters.directory, 'Dev') }}:
ServiceConnection: Dev-Service-connection
${{ if eq(parameters.directory, 'Stage') }}:
ServiceConnection: Stage-Service-connection
Result
[UPDATE]
Dynamically set the input conditionally on the task.
template.yml
parameters:
- name: selectdirectory
type: string
stages:
- stage: A
displayName: 'Stage A'
jobs:
- job: Validate_terraform_Plan
displayName: 'Validate Terraform'
. . .
steps:
. . .
- task: TerraformTaskV2#2
inputs:
provider: 'azurerm'
command: 'init'
${{ if eq(parameters.selectdirectory, 'Dev') }}:
backendServiceArm: Dev-Service-connection
${{ if eq(parameters.selectdirectory, 'Stage') }}:
backendServiceArm: Stage-Service-connection
backendAzureRmResourceGroupName: 'any-rg'
backendAzureRmStorageAccountName: 'any-storage'
backendAzureRmContainerName: 'statefile'
backendAzureRmKey: abcd.tfstate
. . .
azure-pipelines.yml
parameters:
- name: directory
displayName: 'Select appropriate environment directory:'
type: string
values:
- Dev
- Stage
stages:
- template: template.yml
parameters:
selectdirectory: ${{ parameters.directory }}
Have you checked this answer out: AzureDevops: Can we pass dynamic value as service connection in yaml
Essentially rather then pass in your service connection name via an if statement have the service connection name stored in a variable.yml file that is determined off your selected directory and will load the appropriate service connection name.

How to use stages from 1 ymal pipeline to 2 pipeline Azure Devops

I'm working on a project where we have 2 pipelines. 1 pipeline is currently working as expected having 8 stages in it.
Now I want to write code for 2 pipelines where I want to utilize few stages (approx. 4 stages) from the 1 pipeline because code and functionality are similar.
Is there any way I can achieve this in the Azure DevOps YAML pipeline?
Sure, you can export similar stages to a template, then you can use it in other pipelines with extends:
# File: azure-pipelines.yml
trigger:
- master
extends:
template: start.yml
parameters:
buildSteps:
- bash: echo Test #Passes
displayName: succeed
- bash: echo "Test"
displayName: succeed
- task: CmdLine#2
displayName: Test 3 - Will Fail
inputs:
script: echo "Script Test"
The template will be (for example):
# File: start.yml
parameters:
- name: buildSteps # the name of the parameter is buildSteps
type: stepList # data type is StepList
default: [] # default value of buildSteps
stages:
- stage: secure_buildstage
pool: Hosted VS2017
jobs:
- job: secure_buildjob
steps:
- script: echo This happens before code
displayName: 'Base: Pre-build'
- script: echo Building
displayName: 'Base: Build'
- ${{ each step in parameters.buildSteps }}:
- ${{ each pair in step }}:
${{ if ne(pair.value, 'CmdLine#2') }}:
${{ pair.key }}: ${{ pair.value }}
${{ if eq(pair.value, 'CmdLine#2') }}:
'${{ pair.value }}': error
- script: echo This happens after code
displayName: 'Base: Signing'

Azure DevOps template as a dependency for a job

I use Azure DevOps Templates in Stage and I want some job to start only when Job from template is completed (dependsOn):
- stage: stage1
jobs:
- job: job1
steps:
- bash: |
...
- template: template1.yml
parameters:
param1: 'val1'
- job: job2
**dependsOn: how to put `template: template1.yml` here?**
steps:
- bash: |
...
How could it be done?
Building on Eric Smith's answer you can pass in the name of the job that the template will depend on as a parameter.
#template1.yml
jobs:
- job: mytemplateJob1
steps:
- script: npm install
#template2.yml
parameters:
DependsOn: []
jobs:
- job: mytemplateJob2
dependsOn: ${{ parameters.DependsOn }}
steps:
- bash: pwd
By setting the default value for DependsOn to [] you ensure that the template will run if no value is passed in for DependsOn but you can optionally create a dependency like this:
stages:
- stage: stage1
jobs:
- template: template1.yml # Template reference
- template: template2.yml
parameters:
DependsOn: 'mytemplateJob1'
You can accomplish this by using the name of the job, as it is defined in your template in the dependsOn.
#template1.yml
jobs:
- job: mytemplateJob
steps:
- script: npm install
and
stages:
- stage: stage1
jobs:
- job: job1
steps:
- bash: pwd
- template: template1.yml # Template reference
parameters:
param: 'val1'
- job: job2
dependsOn: mytemplateJob
steps:
- bash: pwd