I am trying to use variables defined at the root level on a YAML pipeline inside Azure DevOps inside templates via the template syntax, but it seems that the variables are not available inside the templates, but when adding the steps directly to the pipeline, the exact same thing works perfectly.
So with a pipeline snippet like that
variables:
- name: test
value: asdf
stages:
- stage:
jobs:
- job: test_job
steps:
- script: echo "${{ variables.test }}"
- template: ./test.yaml
And a test.yaml like that
jobs:
- job: test
steps:
- script: echo "${{ variables.test }}"
The script inside the test_job job writes out asdf while the job inside the template just resolves to echo "".
Since my understanding of pipeline templates is, that those basically get inserted into the main pipeline, this seems like a bug. Any ideas on how to use the root variables in a template syntax inside templates or why this is not working? (Macro synatx is not an option as I need the variable inside a templated condition like ${{ if eq(variables['test'], 'asdf') }})
For security reasons, we only allow you to pass information into
templated code via explicit parameters.
https://docs.microsoft.com/en-us/azure/devops/pipelines/process/templates?view=azure-devops
The means the author of the pipeline using your template needs to
commit changes where they explicitly pass the needed info into your
template code.
There are some exceptions to this, where the variable is statically
defined in the same file or at pipeline compile time, but generally
speaking, it’s probably better to use parameters for everything that
does not involve system-defined read-only dynamic variable and
custom-defined dynamic output variables.
This behavior is by design, check this thread in the developer community
So you can either pass the variable as a parameter to the template or define a centralized variables file to include in the template like here
Related
Is it possible to have different variable template files that would be used based on the branch being built? I was trying to do something like this:
jobs:
- job: BuildandPublish
variables:
- template: /env/$(Build.SourceBranchName).vars.yml
But that doesn't work, I'm guessing do to the order that those variables are replaced.
Is it possible to have different variable template files that would be used based on the branch being built?
From your YAML sample, you are using the format: $(Build.SourceBranchName). The variable value will be expanded at runtime.
But the template will read the variable at Compile time.
To solve this issue, you need to change the format : ${{ variables['Build.SourceBranchName'] }}
Here is an example:
jobs:
- job: BuildandPublish
variables:
- template: /env/${{ variables['Build.SourceBranchName'] }}.vars.yml
So I'm working on a bunch of pipelines and I've set everything up using yml templates. However I struggle with getting protected variable expanded inside of my template steps. I've tried passing in the protected variables by normal means, but they seem to not get expanded. Then I tried using a variable group, which I supposedly can directly reference inside of templates. I say supposedly, because Microsoft says so on their website https://learn.microsoft.com/en-us/azure/devops/pipelines/library/variable-groups?view=azure-devops&tabs=yaml:
"You can also reference a variable group in a template. In the template variables.yml, the group my-variable-group is referenced. The variable group includes a variable named myhello."
variables:
- group: my-variable-group
However, whenever I include a variables section into my template code, Azure DevOps immediately complains about it when parsing the yml, before running the pipeline. It spits out the following message:
/ymls/my-template#my-repo (Line: 1, Col: 1): Unexpected value 'variables'
I don't insist on using variable groups, I just want to have my protected variables expanded in my yml template. Does anybody know how to do that???
Any help greatly appreciated!
You should define your variable group on your main pipeline and not in the template. Then you can call your template and use the variable that you defined.
For example lets say that you have your main.yml which calls template.yml
You should define the below variable group on main.yml
variables:
- group: my-variable-group
And call the variable on your template.yml
$(MY_VARIABLE)
https://thomasthornton.cloud/2021/09/02/referencing-variable-groups-in-azure-devops-pipeline-templates/
Finally figured it out. Thanks to #GeralexGR. Turns out, when you reference a variable group in the main pipeline yml, you automatically have access to it in the template yml. But for normal script steps you still have to pass it in explicitly.
It then looks s.th. like this:
main-pipeline.yml:
variables:
- group: my-variable-group
...
jobs:
- job: my-job
steps:
- template: ymls/my-template.yml#my-repo
# no need to pass in the variable group s parameter
my-template.yml:
steps:
- task: ShellScript#2
inputs:
scriptPath: 'my-script.sh'
args: '$(my-secret-variable)'
Hi I have a pipeline dependency challenge and have thought up a number of possible solutions. I can try them all out in a lab but the problem I am wondering if they work well "in the field" and so would like to know if anyone has tried them?
I have 3 stages, each in their own YML file. Each one is called from a main YML which is called from a main pipeline.
- template: 'build.yml'
- template: 'deploy.yml'
- template: 'test.yml'
The 'deploy.yml' generates a large number of output environment variables and 4 of these are consumed by the 'test.yml' using the "stageDependencies" syntax:
stages:
- stage: 'Test_Stage'
dependsOn: Deploy_Stage
jobs:
job: 'Test_Job'
variables:
MyWebSite: [ stageDependencies.Deploy_Stage.Variables_Job.outputs['Variables_Job.Variables.MyWebSite'] ]
This works nicely.
But, I'd like to be able to create a pipeline that just does the test stage (to test a pre-existing web site). That doesn't work of course because of the dependency dependsOn: Deploy_Stage.
I can think of a few possible solutions:
Instead of having a dependency and using the [ stageDependencies... ] syntax, send the MyWebSite as a pipeline parameter between stages. (Note that there are actually parameters not 1, I just simplified to demonstrate the challenge.) If I do that, the tester gets prompted to fill out (or choose from a list) the various parameters. But, it does create linkage between Deploy_Stage and Test_Stage - I don't know if that's a bad thing?
Pass a Boolean parameter from Deploy_Stage to Test_Stage such as "CalledFromDeployStage" and then in Test_Stage, do this:
stages:
- stage: 'Test_Stage'
${{ if eq(parameters.CalledFromDeployStage, true) }}:
dependsOn: Deploy_Stage
jobs:
job: 'Test_Job'
variables:
MyWebSite: [ stageDependencies.Deploy_Stage.Variables_Job.outputs['Variables_Job.Variables.MyWebSite'] ]
This feels a bit clunky.
Create a new YML called "Test_Stage_Manual" and get it to prompt for the various parameters and leave the rest as-is. (If I do this, I would probably put the jobs into their own YML file and call that YML from both Test stages.)
Something else?
You can try like as below:
Create an individual YAML pipeline to only run the test.
In the "Deploy_Stage" of the main pipeline, add a step or job at the end of this stage to execute the API "Runs - Run Pipeline" to trigger the pipeline for test after all the previous steps and jobs in this stage are completed successfully.
When calling the "Runs - Run Pipeline" API, you can pass the variables and parameters generated in the "Deploy_Stage" to the pipeline for test via the Request Body (JSON type) of the API.
Due to the test is in an individual pipeline, you can manually trigger this pipeline if you like. When manually trigger, you can manually set the value of the required variables and parameters in the pipeline.
With this way, you can trigger the pipeline for test both via the "Deploy_Stage" and manually.
We looking to create a pipeline to update our multi-tenant azure environment. We need to perform some actions during the update per tenant. To accomplish this, we would like to create a job per tenant, so we can process tenants in parallel. To accomplish this, I want to use a runtime parameter to pass the tenants to update to my pipeline as follows:
parameters:
- name: tenants
type: object
the value of the tenants parameter might look like something like this:
- Name: "customer1"
Someotherproperty: "some value"
- Name: "customer2"
Someotherproperty: "some other value"
to generate the jobs, we do something like this:
stages:
- stage:
jobs:
- job: Update_Tenant
strategy:
matrix:
${{ each tenant in parameters.Tenants }}:
${{ tenant.tenantName }}:
name: ${{ tenant.tenantName }}
someproperty: ${{ tenant.otherProperty }}
maxParallel: 2
steps:
- checkout: none
- script: echo $(name).$(someproperty)
Now what we need, is some way to fill this tenants parameter. Now I tried a few solutions:
Ideally I would like to put a build stage before the Update_Tenants stage to call a REST api to get the tenants, and expand the tenants parameter when the Update_Tenants stage starts, but this is not supported AFAIK, since parameter expansion is done when the pipeline starts.
A less ideal but still workable option would have been to create a variable group yaml file containing the tenants, and include this variable group in my pipeline, and use the ${{ variables.Tenants }} syntax to reference them. However, for some reason, variables can only be strings.
The only solution I can currently think of, is to create a pipeline that calls a REST api to get the tenants to update, and then uses the azure devops api to queue the actual update process with the correct parameter value. But this feels like a bit of a clunky workaround to accomplish this.
Now my question is, are there any (better?) alternatives to accomplish what I want to do?
Maybe this can help. I was able to use external source (.txt file) to fill array variable in azure pipelines.
Working example
# Create a variable
- bash: |
arrVar=()
for images in `cat my_images.txt`;do
arrVar+=$images
arrVar+=","
done;
echo "##vso[task.setvariable variable=list_images]$arrVar"
# Use the variable
# "$(list_images)" is replaced by the contents of the `list_images` variable by Azure Pipelines
# before handing the body of the script to the shell.
- bash: |
echo my pipeline variable is $(list_images)
Sources (there is also example for matrix)
https://learn.microsoft.com/en-us/azure/devops/pipelines/process/variables?view=azure-devops&tabs=yaml%2Cbatch#set-a-job-scoped-variable-from-a-script
Other sources
https://learn.microsoft.com/en-us/azure/devops/pipelines/process/runtime-parameters?view=azure-devops&tabs=script
To accomplish this, we would like to create a job per tenant, so we
can process tenants in parallel.
Apart from rolling deployment strategy, you can also check Strategies and Matrix.
You can try something like this unless you have to use Runtime parameters:
jobs:
- job: Update
strategy:
matrix:
tenant1:
Someotherproperty1: '1.1'
Someotherproperty2: '1.2'
tenant2:
Someotherproperty1: '2.1'
Someotherproperty2: '2.2'
tenant3:
Someotherproperty1: '3.1'
Someotherproperty2: '3.2'
maxParallel: 3
steps:
- checkout: none
- script: echo $(Someotherproperty1).$(Someotherproperty2)
displayName: 'Echo something'
I'm attempting to create multiple pipelines in Azure DevOps but I would like to reuse the same pipeline YAML file with the differences per environment being loaded from a separate template variables file.
For that purpose I've created two variable files, which are located in the same folder as the pipeline definition:
# vars.dev.yml
variables:
- name: EnvironmentName
value: Development
# vars.prd.yml
variables:
- name: EnvironmentName
value: Production
And the definition of the pipeline is the following:
trigger: none
pr: none
variables:
- name: EnvironmentCode
value: dev
- name: EnvironmentFileName
value: vars.$('EnvironmentCode').yml
stages:
- stage: LoadVariablesPerEnvironment
displayName: Load Variables Per Environment
variables:
- template: $(EnvironmentFileName)
jobs:
- job: ShowcaseLoadedVariables
steps:
- pwsh: Write-Host "Variables have been loaded for the '$ENV:ENVIRONMENTNAME' environment"
displayName: Output Environment Variables
After importing the pipelines using the Azure DevOps UI, I can go to settings of each and set the Environment Code variable to whatever desired environment code:
However I'm always getting the same error when I try to run the pipeline, regardless of the code I fill in the variable value:
So the question here is: Is this kind of variable expansion not supported or is there a different way that I should use to accomplish this?
Thanks!
EDIT
I was able to expand the variables using another method. The new version of the pipeline is as such:
variables:
- name: EnvironmentCode
value: dev
- name: EnvironmentFileName
value: vars.${{ variables.EnvironmentCode }}.yml
stages:
- stage: LoadVariablesPerEnvironment
displayName: Load Variables Per Environment
variables:
- template: ${{ variables.EnvironmentFileName }}
jobs:
- job: ShowcaseLoadedVariables
steps:
- pwsh: Write-Host "Variables have been loaded for the '$ENV:ENVIRONMENTNAME' environment"
displayName: Output Environment Variables
However there is yet the issue of loading different files. I made different attempts and verified the following:
If you give a different environment code using the UI, when running
the pipeline, the value it assumes is still the one that's on the
pipeline definition;
If you remove from the pipeline definition the
default value or the variable entirely the expression
${{variables.EnvironmentCode}} will return an empty string
assuming the filename to be vars..yml which doesn't exist.
Is this kind of variable expansion not supported or is there a
different way that I should use to accomplish this?
If I am not misunderstand, at first, you want to use $() to get the variable you defined using the UI but failed. But later, ${{ }} can give you the value of the variable EnvironmentCode.
In fact, while you change to use ${{ }}, it just accessing the variable you predefined in the YAML files instead of the one you defined with UI. Just see this doc: Variable templates.
For the variable you defined with UI, it can be get and used with the format $()(Note: ${{ }} is the format of get the variables which defined in YAML file). But also, there some things you need to pay attention is for the variables you defined in UI, it can only be get/accessed after the build begin to run, because the variable which defined with UI exists in environment only after the build compiled begin. In one word, they are the agent-scope variable. That's why the value it used is still the one that's on the pipeline definition instead of on the UI.
If you remove from the pipeline definition the default value or the
variable entirely the expression ${{variables.EnvironmentCode}} will
return an empty string assuming the filename to be vars..yml which
doesn't exist.
As the document defined and I mentioned before, ${{}} is format which used to get the value of variable which defined in YAML files rather than the one which defined using UI.
In the steps of job, the variable that defined in the UI or defined in the YAML file can all be get/accessed with the format $(). Still, for the variable which defined in the YAML file can also get with ${{variables.xxxx}}. But at this time, if the variable name which defined in YAML file is same with the one defined with UI, the server can only get the one defined in YAML file.