I have multiple resusable yaml templates to build my project using Azure DevOps. I want to let the default template values when I don't want to override it. The problem I have is when I nest multiple templates, it's not working as I expected, here is an example of my issue:
task_template1.yml:
parameters:
foo: 'data'
steps:
- task: Bash#3
inputs:
targetType: 'inline'
script: echo ${{ parameters.foo }}
Inside a job_template.yml I call this task_template1.yml like this:
job_template.yml:
parameters:
fooTemplate: 'data'
jobs:
- job:
steps:
- template: task_template1.yml
parameters:
${{ if variables[parameters.fooTemplate] }}:
foo: ${{ parameters.fooTemplate }}
And then my global template:
stage_build.yml:
stages:
- stage: Build
jobs:
- template: job_template.yml
parameters:
fooTemplate: 'hello'
So in this case, by default I expect that my variable fooTemplate will set the value of the variable foo inside my task_template1.yml to hello because I specify it.
But it's not the case, because it looks like ${{ if variables[parameters.fooTemplate] }}: is evaluated before the build start so foo is set to data instead of hello. Conversely if I would like to keep the default value of fooTemplate (data) I just want to not specify it like this:
stages:
- stage: Build
jobs:
- template: job_template.yml
It looks like a problem of nested templates. How can I do to fit this two rules? Feel free to ask me some more details if needed.
EDIT:
I always have this message : 'foo has no value: undefined'
Other case that I have, instead of a simple parameter:
parameters:
foo:'data'
foo1:'data1'
steps:
- task: Bash#3
inputs:
targetType: 'inline'
script: echo ${{ parameters.foo }}
So for the job I have:
parameters:
fooTemplate:
foo:'data'
foo1:'data1'
jobs:
- job:
steps:
- template: task_template1.yml
parameters:
${{ if ne(parameters.fooTemplate.foo, ' ') }}:
foo: ${{ parameters.fooTemplate.foo }}
${{ if ne(parameters.fooTemplate.foo1, ' ') }}:
foo: ${{ parameters.fooTemplate.foo1 }}
And then same case if I don't specify the foo and foo1 value I have the undefined error, instead of taking the default values:
stages:
- stage: Build
jobs:
- template: job_template.yml
I think something is wrong with your syntax here:
parameters:
${{ if variables[parameters.fooTemplate] }}:
foo: ${{ parameters.fooTemplate }}
Try this way(according to this):
parameters:
${{ if eq(parameters.fooTemplate, true) }}:
foo: ${{ parameters.fooTemplate }}
It works on my side.
Notes:
You can have something like this:
parameters:
fooTemplate: 'data'
jobs:
- job:
steps:
- template: task_template1.yml
parameters:
${{ if ne(parameters.fooTemplate, ' ') }}:
foo: ${{ parameters.fooTemplate }}
Now if you specify 'hello' in stage_build.yml, the task_template1.yml will output hello. And if you don't specify the parameter in stage_build.yml like this:
pool:
vmImage: 'ubuntu-latest'
jobs:
- template: job_template.yml
It will output the default data. Here're more details about Expressions and Conditions.
Related
I'm trying to build a new yaml file that reads keyvault secrets based on the parameters at the runtime and declared variables with the condition as per the parameters, but this isn't working.
- name: azure_subscription
displayName: " Select subscription "
type: string
default: "service-connection-dev"
values:
- 'service-connection-dev'
- 'service-connection-sit'
- 'service-connection-tes'
- 'service-connection-prd'
variables:
- ${{ if eq('${{ parameters.azure_subscription }}', 'service-connection-sit') }}:
name: key_vault
value: 'core-kv-sit'
- ${{ if eq('${{ parameters.azure_subscription }}', 'service-connection-dev') }}:
name: key_vault
value: 'core-kv-dev'
stages:
- stage: Validate
${{ if eq(parameters.azure_subscription, 'service-connection-dev') }}:
pool:
name: agent-pool-win-dev
${{ if eq(parameters.azure_subscription, 'service-connection-sit') }}:
pool:
name: agent-pool-win-sit
jobs:
- job: Validate
steps:
- task: AzureKeyVault#2
inputs:
KeyVaultName: "${{variables.key_vault}}"
SecretsFilter: "*"
RunAsPreJob: false
azureSubscription: ${{ parameters.azure_subscription }}
I've tried using variables inside jobs, but that is also not working. Can someone please help?
Also, I'll have to declare 2 more variables as per the parameters input, Is it possible ?
Thanks in advance
- ${{ if eq('${{ parameters.azure_subscription }}', 'service-connection-sit') }}:
You're using a literal value of "${{ parameters.azure_subscription }}" for the left side of the comparison. The comparison should just be parameters.azure_subscription.
So: - ${{ if eq(parameters.azure_subscription, 'service-connection-sit') }}:
I need to do an each from a variable obtained with stageDependencies previously.
It doesn't work Pass variable from one stage to another and iterate with an each loop using a split function
What am I doing wrong?
Code:
task.yml
steps:
- task: Bash#3
name: env_string
inputs:
targetType: 'inline'
script: |
environmentsStr='dev,prd'
echo "##vso[task.setvariable variable=environmentsString;isOutput=true]$environmentsStr"
pipeline
- stage: stage1
jobs:
- job: job_stage1
steps:
- template: tasks.yml
- stage: stage2
dependsOn:
- stage1
variables:
- name: envFromVar
value: 'dev,prd'
- name: envFromStageDependencies
value: $[ stageDependencies.stage1.job_stage1.outputs['env_string.environmentsString'] ]
jobs:
- job: job_stage2
steps:
- template: stage3.yml
parameters:
envFromStageDependencies: $(envFromStageDependencies)
environmentsFromVar: ${{ variables.envFromVar }}
stage3.yml
parameters:
- name: envFromStageDependencies
type: string
- name: environmentsFromVar
type: string
steps:
- ${{ each environment in split(parameters.envFromStageDependencies, ',') }}:
- bash: |
echo "env ${{ environment }}" # OUTPUT FAIL; (env dev,prd)
- ${{ each environment in split(parameters.environmentsFromVar, ',') }}:
- bash: |
echo "envVar ${{ environment }}" # OUTPUT OK, 2 iterations envVar [dev,prd]
OUTPUT
BASH -> env dev,prd
BASH -> envVar dev
BASH -> envVar prd
You should change your pipeline YAML like as below:
. . .
jobs:
- job: job_stage2
steps:
- template: stage3.yml
parameters:
environments: ${{ variables.envFromStageDependencies }}
environmentsFromVar: ${{ variables.envFromVar }}
As #Daniel Mann has mentioned, the template is processed at compile time which is before runtime. So, to pass the variable values into the template, you should use the compile time variable syntax (${{ variables.varName }}) instead of runtime variable syntax ($(varName)).
For more details, you can reference the document "Understand variable syntax".
I have two Azure Devops pipelines: 'Starter' and 'Processing'. 'Starter' triggers 'Processing' and passes some parameters to it.
Starter:
trigger: none
pool:
vmImage: 'windows-2019'
stages:
- stage: A
jobs:
- template: Processing.yml
parameters:
products: $(Products)
creds: $(Creds)
Processing:
parameters:
- name: products
type: object
default: []
- name: creds
default: ''
jobs:
- ${{ each product in parameters.products }}:
- task: PowerShell#2
displayName: Importing ${{ product }} solution
inputs:
targetType: 'inline'
script: |
#Code
Key detail here is opportunity to loop through 'products' variable (each product in parameters.products), which must be setted in 'Starter' variables:
In other words, starting my pipeline I must pass list of products as 'string' and then loop through this list in second pipeline. 'Is it generally possible? Maybe products should be another type? I tried some work around but didn't get appropriate solution:
- job: Prepare_Products_Array
steps:
- task: PowerShell#2
inputs:
targetType: 'inline'
script: |
# Write your PowerShell commands here.
$productsArray = []
$productsArray = $(Products)
$productsArray = $productsArray.Split(',')
Write-Host ("##vso[task.setvariable variable=productsArray;]$productsArray")
- template: Processing.yml
parameters:
products: $env:productsArray
Exception:
From your yaml sample, you are defining the variable in YAML Pipeline UI and using parameters in YAML Template.
In this case, the variables defined on the UI will be assigned at runtime, but the parameters and expressions in the YAML template will be expanded at compile time.
Therefore, YAML UI variables cannot be passed to the Pipeline YAML Template.
And it will show the error:
Expected a...... Actual value $(Product)
This means that the pipeline variable not expanded at compile time.
I am afraid that there is no such method can pass the UI Pipeline variable to YAML Template.
Here are the workarounds:
Method 1 : You can use parameters in Starter yaml to pass the Object type parameters to YAML template.
Starter:
trigger: none
parameters:
- name: products
type: object
default: []
- name: creds
default: ''
pool:
vmImage: 'windows-2019'
stages:
- stage: A
jobs:
- template: Processing.yml
parameters:
products: ${{ parameters.products }}
creds: ${{ parameters.creds }}
Processing:
parameters:
- name: products
type: object
default: []
- name: creds
default: ''
jobs:
- job: test
steps:
- ${{ each product in parameters.products }}:
- task: PowerShell#2
displayName: Importing ${{ product }} solution
inputs:
targetType: 'inline'
script: |
echo ${{ product }}
Result: You can input the value when you run the pipeline.
Method2: You need to define the variable in Starter pipeline and change the products parameters as String type. Then you can use the expression - ${{ each product in split(parameters.products, ',')}}: to split the string.
Starter:
pool:
vmImage: 'windows-2019'
variables:
products: '1,2,3'
creds: test
stages:
- stage: A
jobs:
- template: Processing.yml
parameters:
products: ${{ variables.products }}
creds: ${{ variables.creds }}
Processing:
parameters:
- name: products
type: string
default: ''
- name: creds
default: ''
jobs:
- job: test
steps:
- ${{ each product in split(parameters.products, ',')}}:
- task: PowerShell#2
displayName: Importing ${{ product }} solution
inputs:
targetType: 'inline'
script: |
echo ${{ product }}
I want to be able to use a variable group in Azure DevOps Pipeline based upon a generated name.
Like so: (where dtapName is a letter like d or t.)
variables:
- group: 'project-${{ dtapName }}'
I've seen it work in other pipelines but creating one myself has been a pain.
My current pipeline is as follows:
The variable group "global-d" contains a variable 'dtapName' with value 'd'.
And the variable group "project-t" contains URLs and such specific for our 'develop' landscape.
File "pipeline-infra.yaml":
trigger:
branches:
include:
- main
- develop
paths:
include:
- Project/pipelines
stages:
- stage: Develop
displayName: Develop deployment of Project
variables:
- name: environmentName
value: 'develop-o'
- group: 'global-o'
jobs:
- deployment: 'Debug_001'
environment: ${{ variables.environmentName }}
strategy:
runOnce:
deploy:
steps:
- script: echo [Testing job]
- script: echo v.environment = ${{ variables.environmentName }} # Does work
- script: echo v.dtapName = ${{ variables.dtapName }} #❗Fails, why? It is defined in the variable group 'global-o'
# script: echo environment = ${{ environmentName }} #❗Failed, why does it work in the template call below?
# script: echo dtapName = ${{ dtapName }} #❗Failed, why does it work in the template call below?
- script: echo env ENVIRONMENTNAME = $ENVIRONMENTNAME # Does work
- script: echo env DTAPNAME = $DTAPNAME # Does work
- template: pipeline-infra.extra.yaml
parameters:
environment: $(environmentName)
dtapName: $(dtapName)
File "pipeline-infra.extra.yaml"
parameters:
- name: environment
type: string
- name: dtapName
type: string
jobs:
- deployment: 'Debug_002'
# environment: ${{ parameters.environment }} #❗Fails with: Environment $(environmentName) could not be found. WHY?
environment: 'develop-o'
pool:
vmImage: 'ubuntu-latest'
variables:
- name: placeholder
value: 'none'
# - group: 'project-${{ parameters.dtapName }}' #❗Fails with: Variable group 'project-$(dtapName)' could not be found. How to do this?
strategy:
runOnce:
deploy:
steps:
- script: echo [ pipeline-infra.extra.yaml ]
- script: echo Environment = ${{ parameters.environment }} # WORKS
- script: echo "Environment (v) = ${{ variables.environmentName }}" #❗FAILS, this variable is declare on the stage of this job... why can't it be accessed?
- script: echo Parameter = ${{ parameters.dtapName }} # WORKS
- script: echo PARAMETER = $DTAPNAME # WORKS
- script: echo Variable = ${{ variables.dtapName }} #❗FAILS, would think this comes available when the problem in the calling stage is fixed
- task: Bash#3
displayName: VariableListing 2
inputs:
targetType: inline
script: 'env | sort'
Can anyone explain me (or point me to a description) why these problems occur?
I've read (most of) Define variables and documents linked from here.
It's probably a little thing, but for the past couple of days I'm not seeing it.
You have two errors in your main yaml:
You cannot invoke variables from Variable group with template expression ${{ variables.var }}. Template variables are processed at compile time, and are replaced before runtime starts. It cannot get variable value from Group, it can get value from predefined variable.
Template expression should have format as ${{ variables.var }} not ${{ var }}.
Please check doc: Understand variable syntax for the details. I add the comment on your yaml below:
trigger:
branches:
include:
- main
- develop
paths:
include:
- Project/pipelines
stages:
- stage: Develop
displayName: Develop deployment of Project
variables:
- name: environmentName
value: 'develop-o'
- group: 'global-o'
jobs:
- deployment: 'Debug_001'
environment: ${{ variables.environmentName }}
strategy:
runOnce:
deploy:
steps:
- script: echo [Testing job]
- script: echo v.environment = ${{ variables.environmentName }} # Does work, it use template expression syntax and can get the value as it's predefined variable.
- script: echo v.dtapName = ${{ variables.dtapName }} #❗Not work, Template variables are processed at compile time, and are replaced before runtime starts, it cannot get the Test Group variable value, should use macro syntax $(dtapName).
# script: echo environment = ${{ environmentName }} #❗Failed, syntax error, should be ${{ variables.environmentName }} or $(environmentName)
# script: echo dtapName = ${{ dtapName }} #❗Failed, syntax error, should be $(dtapName)
- script: echo env ENVIRONMENTNAME = $ENVIRONMENTNAME # Does work, the variable is set as environment value, you can use it with syntax $NAME.
- script: echo env DTAPNAME = $DTAPNAME # Does work,the variable is set as environment value, you can use it with syntax $NAME.
- template: pipeline-infra.extra.yaml
parameters:
environment: $(environmentName) # Environment creation happens at compile time, if you pass marco syntax to template, it will cause the error, should use template expreesion ${{ variables.environmentName }}.
dtapName: $(dtapName) # not work here, should define it out of variable group, and transfer with parameter expression same as above.
For your template yaml:
Environment creation happens at compile time, so if you define $(environmentName) in main yaml for the template, it cannot get the value, you should use template expression ${{ variables.environmentName }} in main yaml to transfer the value.
No variable defined in template, you cannot use ${{ variables.var }}, you should use parameters expression ${{ parameters.var }}.
parameters:
- name: environment
type: string
- name: dtapName
type: string
jobs:
- deployment: 'Debug_002'
# environment: ${{ parameters.environment }} #❗Fails, should use ${{ variables.environmentName }} in main yaml for template.
environment: 'develop-o'
pool:
vmImage: 'ubuntu-latest'
variables:
- name: placeholder
value: 'none'
# - group: 'project-${{ parameters.dtapName }}' #❗Fails, same as above, it cannot get the value
strategy:
runOnce:
deploy:
steps:
- script: echo [ pipeline-infra.extra.yaml ]
- script: echo Environment = ${{ parameters.environment }} # WORKS
- script: echo "Environment (v) = ${{ variables.environmentName }}" #❗FAILS, should use parameter expression as no variable defined.
- script: echo Parameter = ${{ parameters.dtapName }} # WORKS
- script: echo PARAMETER = $DTAPNAME # WORKS
- script: echo Variable = ${{ variables.dtapName }} #❗FAILS, same as above.
- task: Bash#3
displayName: VariableListing 2
inputs:
targetType: inline
script: 'env | sort'
I have the following pipeline template that I want to use to conditionally execute stages based on the input parameter stages.
parameters:
- name: dryRun
default: false
type: boolean
- name: path
type: string
default: terraform
- name: stages
type: object
default:
- test
- prod
stages:
- stage:
pool:
vmImage: 'ubuntu-latest'
displayName: "Deploy to test"
condition: in('test', ${{ parameters.stages }})
jobs:
- template: terraform-job.yaml
parameters:
stage: test
path: ${{ parameters.path }}
dryRun: ${{ parameters.dryRun }}
- stage:
pool:
vmImage: 'ubuntu-latest'
displayName: "Deploy to production"
condition: in('prod', '${{ join(', ', parameters.stages) }}')
jobs:
- template: terraform-job.yaml
parameters:
stage: production
path: ${{ parameters.path }}
dryRun: ${{ parameters.dryRun }}
In the example you can see two of the approached I tried (I tried a lot...). The last one (in('prod', '${{ join(', ', parameters.stages) }}')) actually compiles but then the check only works partially as the array gets converted into a single string: 'test,prod' which will fail the in('test', 'test,prod') check.
And the first example (in('test', ${{ parameters.stages }})) is the one I think should work with logical thinking but I get the following error when compiling the template: /terraform-deployment.yml (Line: 19, Col: 16): Unable to convert from Array to String. Value: Array.
So now the question:
How do I check if a string is part of an array that was defined as a parameter?
2022 update
You can now use containsValue:
condition: ${{ containsValue(parameters.stages, 'test') }}
Try contains instead:
condition: contains('${{ join(';',parameters.stages) }}', 'test')