Azure Devops: Template use sequence typed parameter to generate jobs and stages - azure-devops

It seems the template variable does not support sequence type but the template parameter does support it(called object type in Microsoft doc), together with JobList type and StageList type, but they seem to be just sequences with defined schema.
When iterating through object-typed parameter to generate step I didn't meet any problem, but I got error called 'A sequence was not expected' when I was trying to do such to generate jobs and stages.
So the result have to be like this:
# main.yml:
trigger: none
extends:
template: template.yml
parameters:
PROJECTNAME: foo.bar
DEPLOYMENTTARGETS:
- stage:
jobs:
- deployment:
environment:
name: TEST
tags: Web # tag must be comma separated string which is also kinda weird
- stage:
jobs:
- deployment:
environment:
name: PROD
tags: Web
# template.yml:
parameters:
- name: PROJECTNAME
type: string
default: ""
- name: DEPLOYMENTTARGETS
type: stageList # I was expecting object to be working
default: []
variables:
- group: LoginSecrets
stages:
- ${{ each deploymentTarget in parameters.DEPLOYMENTTARGETS }}: # iterating through an object here will result in error: 'A sequence was not expected', iterating through a StageList is OK
- stage: Deploy_${{replace(parameters.PROJECTNAME,'.','_')}}_${{replace(deploymentTarget.jobs[0].environment.name,'-','')}}_${{replace(replace(deploymentTarget.jobs[0].environment.tags,',','_'),'-','')}}_Stage # assuming each stage contains only one deployment job. We also iterate through jobs here if required.
dependsOn: BuildAndPublish_${{replace(parameters.PROJECTNAME,'.','_')}}_Stage
jobs:
- deployment: Deploy_${{replace(parameters.PROJECTNAME,'.','_')}}_Job
environment:
name: ${{deploymentTarget.jobs[0].environment.name}}
resourceType: VirtualMachine
tags: ${{deploymentTarget.jobs[0].environment.tags}}
strategy:
runOnce:
deploy:
steps:
- pwsh: $(Pipeline.Workspace)/${{parameters.PROJECTNAME}}/pipelineRelease.ps1
env:
LOGINNAME: $(loginName)
LOGINPASSWORD: $(loginPassword)
PROJECTNAME: ${{parameters.PROJECTNAME}}
But I was expecting something like this to be working:
# main.yml:
trigger: none
extends:
template: template.yml
parameters:
PROJECTNAME: foo.bar
DEPLOYMENTTARGETS:
- EnvName: TEST
Tags: Web
- EnvName: PROD
Tags: Web
# template.yml:
parameters:
- name: PROJECTNAME
type: string
default: ""
- name: DEPLOYMENTTARGETS
type: object # instead of StageList, I passed an object here
default: []
variables:
- group: LoginSecrets
stages:
- ${{ each deploymentTarget in parameters.DEPLOYMENTTARGETS }}: # error here
- stage: Deploy_${{replace(parameters.PROJECTNAME,'.','_')}}_${{replace(deploymentTarget.EnvName,'-','')}}_${{replace(replace(deploymentTargetTags,',','_'),'-','')}}_Stage
dependsOn: BuildAndPublish_${{replace(parameters.PROJECTNAME,'.','_')}}_Stage
jobs:
- deployment: Deploy_${{replace(parameters.PROJECTNAME,'.','_')}}_Job
environment:
name: ${{deploymentTarget.EnvName}}
resourceType: VirtualMachine
tags: ${{deploymentTarget.Tags}}
strategy:
runOnce:
deploy:
steps:
- pwsh: $(Pipeline.Workspace)/${{parameters.PROJECTNAME}}/pipelineRelease.ps1
env:
LOGINNAME: $(loginName)
LOGINPASSWORD: $(loginPassword)
PROJECTNAME: ${{parameters.PROJECTNAME}}

Azure Devops: Template use sequence typed parameter to generate jobs and stages
I am afraid we have to use the stageList instead of object to transfer two-dimensional arrays.
That because the parameters is defined as two-dimensional arrays in the main.yml, But if we define this arrays as object in the template.yml file. In this case, we could not loop the object in the template file.
You could check the document Loop through parameters for some more details.
Also, you could check the similar thread for some details.

Related

Azure yaml same key issue when checking out same repository in multiple jobs

I have reusable build template job. In my pipeline, I'm reusing this template every time I want to build new application. Inside this build template, it has inline checkout step. Problem occurs if I checkout same repository and branch.
Template structure
# **pipeline.yaml**
stages:
- template: templates/group-build.yaml
parameters:
branchRef: ${{ variables.branchRef}}
# -----------------------------------------
# **templates/group-build.yaml**
parameters:
- name: branchRef
type: string
stages:
- stage: 'Build'
displayName: 'Build'
variables:
jobs:
- template: template/functionapp-build-job.yaml
parameters:
branchRef: ${{ parameters.branchRef}}
- template: template/webapp-build-job.yaml
parameters:
branchRef: ${{ parameters.branchRef}}
# -----------------------------------------
# **template/functionapp-build-job.yaml**
parameters:
- name: branchRef
type: string
jobs:
- template: templates/function/build-job.yaml
parameters:
repository: repo
branchRef: ${{ parameters.branchRef}}
project: functionapp/function.csproj'
# -----------------------------------------
# **templates/function/build-job.yaml**
parameters:
- name: repository
type: string
- name: project
type: string
- name: componentName
type: string
- name: branchRef
type: string
jobs:
- job: JOB_${{ parameters.componentName}}_BUILD # <- Component is unique
displayName: Build ${{ parameters.componentName}} Job
steps:
// Checking out same repo is not allowed here
- checkout: git://project/${{ parameters.repository }}#${{ parameters.branchRef}}
- Other Task here...
# -----------------------------------------
# **template/webapp-build-job.yaml**
# Same structure as function app...
This is how it looks like structurally
Pipeline
Stages:
- Stage:
Jobs:
- Job: Web App Build
Steps:
- checkout: 'Here I call same repo and branch but different build steps'
- Job: Function App Build
Steps:
- checkout: 'Here I call same repo and branch but different build steps'
Validation issue image
Adding resources in pipeline is not my option because refs is being stored in variable group, which is not support to resolve due to limitation of azure yaml as stated here.
Any suggestion or idea to circumvent this issue is highly appreciated.
From your sample, the template webapp-build-job.yaml and functionapp-build-job.yaml are reference the same yaml:build-job.yaml.
In this case, the job names (- job: JOB_BUILD) are the same key and this can be the root cause of this issue.
To solve this issue, you can define job name as a parameter and assign value in webapp-build-job.yaml and functionapp-build-job.yaml.
For example:
build-job.yaml
parameters:
- name: repository
type: string
- name: project
type: string
- name: jobname
type: string
- name: branchRef
type: string
jobs:
- job: ${{ parameters.jobname }}
displayName: Build Job
steps:
- checkout: git://Artifacts/${{ parameters.repository }}#${{ parameters.branchRef}}
functionapp-build-job.yaml
parameters:
- name: branchRef
type: string
jobs:
- template: build-job.yaml
parameters:
repository: Repo11
branchRef: ${{ parameters.branchRef}}
jobname: test1
project: functionapp/function.csproj'
webapp-build-job.yaml
parameters:
- name: branchRef
type: string
jobs:
- template: build-job.yaml
parameters:
repository: Repo11
branchRef: ${{ parameters.branchRef}}
jobname: test2
project: functionapp/function.csproj'
group-build.yaml
parameters:
- name: branchRef
type: string
stages:
- stage: 'Build'
displayName: 'Build'
jobs:
- template: functionapp-build-job.yaml
parameters:
branchRef: ${{ parameters.branchRef}}
- template: webapp-build-job.yaml
parameters:
branchRef: ${{ parameters.branchRef}}
Pipeline.yaml
variables:
- group: test
stages:
- template: group-build.yaml
parameters:
branchRef: ${{ variables.branchRef}}

Dynamic parameters/variables does not work for deployment job in checkout step in template but work directly in pipeline

The dynamic variables with the checkout step do not work with the template.
The same works without a template.
#- checkout: git://MY_PROJ/MY_REPO#refs/tags/${{parameters.tag_name}} ## DOES NOT WORK
Getting below error while running pipeline:
"ERROR: An item with the same key has already been added"
Working:
my_pipeline.yml without template works
parameters:
- name: RELEASE_TAG
displayName: Enter The Master Release Tag Name Example 1.0.0-RELEASE
default: 1.0.0-RELEASE
type: string
trigger:
- none
variables:
db_resource_path: '$(System.DefaultWorkingDirectory)/resources/db'
pipeline_environment_name: 'PROD_ENV'
db_env_name: 'prod'
stages:
- stage: "PROD_DB_DEPLOYMENT"
displayName: "PROD DB Deployment"
pool:
name: $(param.agent.pool.name)
variables:
- group: PROD_VG
jobs:
- job:
steps:
- script: |
echo param release tag: ${{parameters.RELEASE_TAG}}
- deployment: Deploy
environment: PROD_ENV
strategy:
runOnce:
deploy:
steps:
- checkout: git://MY_PROJ/MY_REPO#refs/tags/${{parameters.RELEASE_TAG}} ## WORKS
#- checkout: git://MY_PROJ/MY_REPO#refs/tags/${{variables.tag_name}} ## WORKS WITH VAR ALSO
Not Working:
Gives ERROR: An item with the same key has already been added.
my azure-pipelines.yml is:
parameters:
- name: RELEASE_TAG
displayName: Enter The Master Release Tag Name Example 1.0.0-RELEASE
default: 1.0.0-RELEASE
type: string
resources:
repositories:
- repository: templates
type: git
name: MY_PROJECT/MY_TEMPLATE_REPO
trigger:
- none
variables:
tagName: '${{ parameters.RELEASE_TAG }}'
stages:
- stage: "PROD_DB_DEPLOYMENT"
displayName: "PROD DB Deployment"
variables:
- group: PROD_VG
jobs:
- template: my_template.yml#templates
parameters:
tag_name: $(tagName)
db_env_name: 'prod'
agent_pool_name: $(param.agent.pool.name)
db_resource_path: $(System.DefaultWorkingDirectory)/resources/db
pipeline_environment_name: PROD_ENV
is_release: 'true'
My Template is:
parameters:
- name: 'agent_pool_name'
type: string
- name: 'db_resource_path'
type: string
- name: 'pipeline_environment_name'
type: string
- name: 'db_env_name'
type: string
- name: 'is_release'
type: string
default: 'false'
- name: 'tag_name'
type: string
default: '1.0.0-RELEASE'
jobs:
- job:
pool:
name: ${{parameters.agent_pool_name}}
steps:
- script: |
echo param tag_name: ${{parameters.tag_name}}
echo var tag_name: $(tagName)
- deployment: Deploy
pool:
name: ${{parameters.agent_pool_name}}
environment: ${{parameters.pipeline_environment_name}}
strategy:
runOnce:
deploy:
steps:
- checkout: self
- ${{ if eq(parameters.is_release, true) }}:
#- checkout: git://MY_PROJ/MY_REPO#refs/tags/1.0.0-RELEASE ## WORKS
#- checkout: git://MY_PROJ/MY_REPO#refs/tags/$(tag_name) ## DOES NOT WORK
#- checkout: git://MY_PROJ/MY_REPO#refs/tags/${{parameters.tag_name}} ## DOES NOT WORK
Tried below variable option also but get error:Unexpected value 'variables'
Any suggestion, please.
This is due to that you passed $(var) as parameters of the template.
You can pass ${{variables.tagName}} instead.
checked on my side screenshot
In a pipeline, template expression variables (${{ variables.var }}) get processed at compile time, before runtime starts. Macro syntax variables ($(var)) get processed during runtime before a task runs.
Because templates are expanded before the pipeline execution gets planned, so you cannot pass $(var) as parameter to the template.
Please check official doc for the variable syntax.
https://learn.microsoft.com/en-us/azure/devops/pipelines/process/variables?view=azure-devops&tabs=yaml%2Cbatch#understand-variable-syntax
Edit: for error "Unexpected value 'variables'":
Move variables to stage scope: my yaml here
My template here

AzureDevops set variables based on parameter and pass to template at next stage

In Azure DevOps, I'm trying to create a pipeline which offers a simple selection of pre-set options to the user running it. These options will be converted into different combinations of parameters as specified by a templated stage (the definition of which, I have no control over). The idea of my pipeline is that frequently-used build configurations are easy to select correctly, rather than having to manually set 3 or 4 different parameters.
I need the "Build.Setup" from immutable_pipeline to print config_one, profile_one when the first radio is selected (buildType=type1), config_two, profile_two when buildType=type2, and so on.
Unfortunately I'm really struggling to get any variable value into the templated stage other than the defaults. Are ADO variables even mutable variables at all - or just constants?
I've read the MS docs extensively and understand the meaings of the different macro declaration types. I've tried many different combinations of syntaxes ${{...}}, $(...) and $[...], all behave differently but none seems able to deliver what's needed. Is this even possible? Is there a simple solution someone can suggest?
Pipeline:
name: $(Date:yyyyMMdd).$(Rev:r)
parameters:
- name: buildType
displayName: 'Type of build'
type: string
default: 'type3'
values: ['type1', 'type2', 'type3']
pool:
name: default
variables:
- name: config
value: 'defaultConfig'
- name: profile
value: 'defaultProfile'
stages:
- stage: Stage1
displayName: Prepare build config
jobs:
- job: Job1_1
steps:
- checkout: none
- task: Bash#3
name: SetVariables
inputs:
targetType: inline
script: |
p1='${{ parameters.buildType }}'
v1='$(config)'
v2='$(profile)'
echo -e "BEFORE: p1='${p1}'\n v1='${v1}'\n v2='${v2}'"
case ${p1} in
type1)
v1='config_one'
v2='profile_one'
;;
type2)
v1='config_two'
v2='profile_two'
;;
type3)
v1='config_three'
v2='profile_three'
;;
esac
echo -e "AFTER: p1='${p1}'\n v1='${v1}'\n v2='${v2}'"
echo "##vso[task.setvariable variable=config]${v1}"
echo "##vso[task.setvariable variable=profile;isOutput=True]${v2}"
- job: Job1_2
dependsOn: Job1_1
variables:
- name: variable1
value: $(config)
- name: variable2
value: $[ dependencies.Job1_1.outputs['SetVariables.profile']]
steps:
- task: Bash#3
name: GetVariables2
inputs:
targetType: inline
script: |
echo -e 'SAME STAGE: v1="$(variable1)"\n v2="$(variable2)"'
# Next stage - use computed values for "config" and "profile"
- template: templates/immutable_pipeline.yml
parameters:
config: $(config)
profile: ${{ variables.profile }}
templates/immutable_pipeline.yml:
Note that I don't have access to change this, I can't make it dependsOn: Stage1.Job1_1.
parameters:
- name: config
displayName: 'Config'
type: string
default: 'unset'
- name: profile
displayName: 'Profile'
type: string
default: 'unset'
stages:
- stage: Build
displayName: Templated build
jobs:
- job: Setup
pool:
name: default
demands:
- Agent.OS -equals Linux
steps:
- checkout: none
- script: |
echo '##[info] parameters.config=${{ parameters.config }}'
echo '##[info] parameters.profile=${{ parameters.profile }}'
I just found one solution (which is arguably simpler than using variables) using the ${{ if eq(...) }}: syntax:
name: $(Date:yyyyMMdd).$(Rev:r)
parameters:
- name: buildType
displayName: 'Type of build'
type: string
default: 'type3'
values: ['type1', 'type2', 'type3']
pool:
name: default
stages:
- template: templates/immutable_pipeline.yml
${{ if eq(parameters.buildType, 'type1') }}:
parameters:
config: config_one
profile: profile_one
${{ if eq(parameters.buildType, 'type2') }}:
parameters:
config: config_two
profile: profile_two
${{ if eq(parameters.buildType, 'type3') }}:
parameters:
config: config_three
profile: profile_three
Still interested in whether the original approach of setting variables is even possible, if only beause I've spent so much time on it.

Applying tags to a CFN stack using Azure DevOps pipeline - list?

Following on from my question the other day...
Passing an object into a template in Azure DevOps pipeline
I am now facing the problem where I'm trying to use the pipeline to apply tags to the AWS CloudFormation stack that gets created from the Azure DevOps pipeline.
Here's the pipeline (some bits of it) :-
# azure-pipelines.yml
parameters:
- name: stackparams
type: object
default:
- displayname: AWS Test User Account
awscredentials: TESTAWS-BuildUser
templatefile: ./Test/TestApp/ConsoleSwitchRolesGroupsPolicies.yml
templateparametersfile: ./Test/TestApp/demoparams.json
tags:
- tag1
- tag2
- Managed=Cloudformation
stages:
- stage: ProdDeploy # name of the stage (A-Z, a-z, 0-9, and underscore)
displayName: Production Deploy # friendly name to display in the UI
jobs:
- deployment: DeveloperRoles # name of the deployment job (A-Z, a-z, 0-9, and underscore)
displayName: Manage Developer Roles using Cloudformation # friendly name to display in the UI
pool:
vmImage: 'ubuntu-latest'
environment: aws-test-appaccount-structure # target environment name and optionally a resource name to record the deployment history; format: <environment-name>.<resource-name>
strategy:
runOnce: #rolling, canary are the other strategies that are supported
deploy:
steps:
- template : ./Templates/CfnUpdateStack/CfnUpdateStack.yml
parameters:
stackparams: ${{ parameters.stackparams }}
runRolesGroupsPolicies: ${{ parameters.runRolesGroupsPolicies }}
Here's the template :-
# CfnUpdateStack.yml
parameters:
- name: stackparams
type: object
- name: runRolesGroupsPolicies
type: boolean
steps:
- ${{ each stackparam in parameters.stackparams }}:
- task: CloudFormationCreateOrUpdateStack#1
displayName: ${{ stackparam.displayname }}
inputs:
awsCredentials: ${{ stackparam.awscredentials }}
regionName: 'eu-west-1'
stackName: 'ConsoleSwitchRolesGroupsPolicies'
templateSource: 'file'
templateFile: ${{ stackparam.templatefile }}
templateParametersFile: ${{ stackparam.templateparametersfile }}
tags: |
Component=RoleUserAccess
${{ stackparam.tags }}
condition: eq('${{ parameters.runRolesGroupsPolicies }}', true)
When I run this I only get the first tag Component=RoleUserAccess. I have managed to get it working, but using this horrendous ugly method of using a variable (not a param) and putting a blank line between each tag. It does work though!
Versions
variables:
- name: tags
value: "tag1
tag2
Managed=Cloudformation"
We have around 20 tags for each resource, so it gets pretty messy. I have cut it down to make the question simpler to read.
Any suggestions?
Thanks,
It seems you are trying to add an array (tags) to an array (stackparams), I don't think it's correct. You may try the following syntax:
# CfnUpdateStack.yml
parameters:
- name: 'tags'
type: object
default: {}
- name: 'displayname'
type: string
default: ''
- name: 'awscredentials'
type: string
default: ''
- name: 'templatefile'
type: string
default: ''
- name: 'templateparametersfile'
type: string
default: ''
steps:
# azure-pipelines.yml
- template: CfnUpdateStack.yml
parameters:
tags:
- tag1
- tag2
- Managed=Cloudformation
displayname: AWS Test User Account
awscredentials: TESTAWS-BuildUser
templatefile: ./Test/TestApp/ConsoleSwitchRolesGroupsPolicies.yml
templateparametersfile: ./Test/TestApp/demoparams.json
- template: CfnUpdateStack.yml
parameters:
tags:
- tag1
- tag2
- Managed=Cloudformation
displayname: AWS Test User Account2
awscredentials: TESTAWS-BuildUser2
templatefile: ./Test/TestApp/ConsoleSwitchRolesGroupsPolicies2.yml
templateparametersfile: ./Test/TestApp/demoparams2.json

azure devOps yml pipeline: pass Boolean value to task prisma-cloud-compute-scan#2 written in template

I want to send Boolean value to task 'prisma-cloud-compute-scan#2' written in a template file.
It always gives below error.
Error: The 'prisma_continue_on_error' parameter value '$(prismaContinueOnError)' is not a valid Boolean.
Main pipeline abc.yml
resources:
repositories:
- repository: templates
type: git
name: my_projects/my-build-templates
ref: refs/heads/features/add-build-template
variables:
name: prismaContinueOnError
value: false
isMainBranch: $[eq(variables['Build.SourceBranch'], 'refs/heads/master')]
stages:
- stage: "Build"
displayName: Build
jobs:
- template: my_build_stage/my_template.yml#templates
parameters:
prisma_continue_on_error: $(prismaContinueOnError)
Template my_template.yml
parameters:
- name: prisma_continue_on_error
type: boolean
default: false
- name: pool_name
type: string
default: abc_pool
jobs:
- job: Build
pool:
name: ${{parameters.pool_name}}
steps:
- task: prisma-cloud-compute-scan#2
inputs:
scanType: 'images'
twistlockService: 'SERVICE_CONNECTIONM_NAME'
artifact: ...
continueOnError: ${{parameters.prisma_continue_on_error}}
You mixed syntaxes here
variables:
name: prismaContinueOnError
value: false
isMainBranch: $[eq(variables['Build.SourceBranch'], 'refs/heads/master')]
it should be:
variables:
prismaContinueOnError: false
isMainBranch: $[eq(variables['Build.SourceBranch'], 'refs/heads/master')]
But this will not solve the issue, because variables are just string. You can't have variable of type boolean. You need to pass there runtime expression which delay type evaluation:
stages:
- stage: "Build"
displayName: Build
jobs:
- template: my_build_stage/my_template.yml#templates
parameters:
prisma_continue_on_error: ${{ variables.prismaContinueOnError }}