So I'm working on a bunch of pipelines and I've set everything up using yml templates. However I struggle with getting protected variable expanded inside of my template steps. I've tried passing in the protected variables by normal means, but they seem to not get expanded. Then I tried using a variable group, which I supposedly can directly reference inside of templates. I say supposedly, because Microsoft says so on their website https://learn.microsoft.com/en-us/azure/devops/pipelines/library/variable-groups?view=azure-devops&tabs=yaml:
"You can also reference a variable group in a template. In the template variables.yml, the group my-variable-group is referenced. The variable group includes a variable named myhello."
variables:
- group: my-variable-group
However, whenever I include a variables section into my template code, Azure DevOps immediately complains about it when parsing the yml, before running the pipeline. It spits out the following message:
/ymls/my-template#my-repo (Line: 1, Col: 1): Unexpected value 'variables'
I don't insist on using variable groups, I just want to have my protected variables expanded in my yml template. Does anybody know how to do that???
Any help greatly appreciated!
You should define your variable group on your main pipeline and not in the template. Then you can call your template and use the variable that you defined.
For example lets say that you have your main.yml which calls template.yml
You should define the below variable group on main.yml
variables:
- group: my-variable-group
And call the variable on your template.yml
$(MY_VARIABLE)
https://thomasthornton.cloud/2021/09/02/referencing-variable-groups-in-azure-devops-pipeline-templates/
Finally figured it out. Thanks to #GeralexGR. Turns out, when you reference a variable group in the main pipeline yml, you automatically have access to it in the template yml. But for normal script steps you still have to pass it in explicitly.
It then looks s.th. like this:
main-pipeline.yml:
variables:
- group: my-variable-group
...
jobs:
- job: my-job
steps:
- template: ymls/my-template.yml#my-repo
# no need to pass in the variable group s parameter
my-template.yml:
steps:
- task: ShellScript#2
inputs:
scriptPath: 'my-script.sh'
args: '$(my-secret-variable)'
Related
I am trying to use variables defined at the root level on a YAML pipeline inside Azure DevOps inside templates via the template syntax, but it seems that the variables are not available inside the templates, but when adding the steps directly to the pipeline, the exact same thing works perfectly.
So with a pipeline snippet like that
variables:
- name: test
value: asdf
stages:
- stage:
jobs:
- job: test_job
steps:
- script: echo "${{ variables.test }}"
- template: ./test.yaml
And a test.yaml like that
jobs:
- job: test
steps:
- script: echo "${{ variables.test }}"
The script inside the test_job job writes out asdf while the job inside the template just resolves to echo "".
Since my understanding of pipeline templates is, that those basically get inserted into the main pipeline, this seems like a bug. Any ideas on how to use the root variables in a template syntax inside templates or why this is not working? (Macro synatx is not an option as I need the variable inside a templated condition like ${{ if eq(variables['test'], 'asdf') }})
For security reasons, we only allow you to pass information into
templated code via explicit parameters.
https://docs.microsoft.com/en-us/azure/devops/pipelines/process/templates?view=azure-devops
The means the author of the pipeline using your template needs to
commit changes where they explicitly pass the needed info into your
template code.
There are some exceptions to this, where the variable is statically
defined in the same file or at pipeline compile time, but generally
speaking, it’s probably better to use parameters for everything that
does not involve system-defined read-only dynamic variable and
custom-defined dynamic output variables.
This behavior is by design, check this thread in the developer community
So you can either pass the variable as a parameter to the template or define a centralized variables file to include in the template like here
I have a DevOps build script that looks like this:
variables:
MyConnectionString: ''
stages:
- stage: BuildApp
. . .
# build app
. . .
- job: RunApp
steps:
- script: |
echo ${{variables.MyConnectionString}}
The issue that I have is that the connection string is not coming through to the script - I've set it in the edit variables, but it comes through as blank. I tried setting it directly in the script, but when I did that, it appeared to truncate the value at the semi-colon.
I feel like I'm missing something fundamental around how these variables work. Please can someone point me in the right direction?
The way you are calling the variable is actually for pipeline parameters or varibles template expressions
Calling variables or group variables into tasks should be just using the macro syntax interpolation $(MyConnectionString) syntax.
So change it to:
echo $(MyConnectionString)
Or use the other format of defining variables (usually used when you're also referring to variable groups)
variables:
- group: myVarGroup
- name: MyConnectionString
value: connectioncreds
And also make sure to check this article for more info about devops variables.
I'm trying to get Azure DevOps pipelines to prompt for a version number when manually launching a pipeline (defined using the new YAML syntax).
Even when I define variables in the template, the launch screen says "This pipeline has no defined variables". How can I define variables so that they show up in the pipeline launch?
Current YAML definition contains:
variables:
- name: versionName
value: ''
These are not shown when launching the pipeline:
While Shayki's answer is correct for defining variables, what I was really looking for is runtime parameters.
With the following YAML definition:
parameters:
- name: myParameter
displayName: Description of myParameter
default: defaultMyParameter
type: string
it prompts for the parameter value when launching the pipeline:
The parameter must be referenced in the template using ${{ parameters.myParameter }}, the other variable syntaxes don't work.
From the docs:
If a variable appears in the variables block of a YAML file, its
value is fixed and can't be overridden at queue time. Best practice is
to define your variables in a YAML file but there are times when this
doesn't make sense. For example, you may want to define a secret
variable and not have the variable exposed in your YAML. Or, you may
need to manually set a variable value during the pipeline run.
You have two options for defining queue-time values. You can define a
variable in the UI and select the option to Let users override this
value when running this pipeline or you can use runtime parameters
instead. If your variable is not a secret, the best practice is to use
runtime parameters.
To set a variable at queue time, add a new variable within your
pipeline and select the override option.
To allow a variable to be set at queue time, make sure the variable
doesn't also appear in the variables block of a pipeline or job. If
you define a variable in both the variables block of a YAML and in the
UI, the value in the YAML will have priority.
I have three build pipelines, one for each environment: Dev, QA, Prod. I've defined group variables for each environment - group-Dev, group-QA, group-Prod. Each build pipeline has a variable TestEnvironment set to its respective environment. I am trying to expand the group variable name using the pipeline variable but I cannot get it to work. At compile time it cannot evaluate the group variable name, I assume because it doesn't know the runtime variable value yet.
What I want is something like this:
variables:
env: $(TestEnvironment)
stages:
- stage: Deploy
jobs:
- job: TestVariable
variables:
- group: 'group-${{variables.env}}'
steps:
- script: echo $(TestEnvironment), $(Database.Name), ${{variables.env}}
displayName: 'Printing the variable'
But the only way i can get it to work is if I hard-code the variable, ie: env: QA. I also tried removing variables section and setting group: 'group-$(TestEnvironment), but that didn't work either.
This is not supported today. You need to declare the variable group you want to consume in a YAML pipeline. Please check the following issue:
https://github.com/MicrosoftDocs/vsts-docs/issues/3702
I'm attempting to create multiple pipelines in Azure DevOps but I would like to reuse the same pipeline YAML file with the differences per environment being loaded from a separate template variables file.
For that purpose I've created two variable files, which are located in the same folder as the pipeline definition:
# vars.dev.yml
variables:
- name: EnvironmentName
value: Development
# vars.prd.yml
variables:
- name: EnvironmentName
value: Production
And the definition of the pipeline is the following:
trigger: none
pr: none
variables:
- name: EnvironmentCode
value: dev
- name: EnvironmentFileName
value: vars.$('EnvironmentCode').yml
stages:
- stage: LoadVariablesPerEnvironment
displayName: Load Variables Per Environment
variables:
- template: $(EnvironmentFileName)
jobs:
- job: ShowcaseLoadedVariables
steps:
- pwsh: Write-Host "Variables have been loaded for the '$ENV:ENVIRONMENTNAME' environment"
displayName: Output Environment Variables
After importing the pipelines using the Azure DevOps UI, I can go to settings of each and set the Environment Code variable to whatever desired environment code:
However I'm always getting the same error when I try to run the pipeline, regardless of the code I fill in the variable value:
So the question here is: Is this kind of variable expansion not supported or is there a different way that I should use to accomplish this?
Thanks!
EDIT
I was able to expand the variables using another method. The new version of the pipeline is as such:
variables:
- name: EnvironmentCode
value: dev
- name: EnvironmentFileName
value: vars.${{ variables.EnvironmentCode }}.yml
stages:
- stage: LoadVariablesPerEnvironment
displayName: Load Variables Per Environment
variables:
- template: ${{ variables.EnvironmentFileName }}
jobs:
- job: ShowcaseLoadedVariables
steps:
- pwsh: Write-Host "Variables have been loaded for the '$ENV:ENVIRONMENTNAME' environment"
displayName: Output Environment Variables
However there is yet the issue of loading different files. I made different attempts and verified the following:
If you give a different environment code using the UI, when running
the pipeline, the value it assumes is still the one that's on the
pipeline definition;
If you remove from the pipeline definition the
default value or the variable entirely the expression
${{variables.EnvironmentCode}} will return an empty string
assuming the filename to be vars..yml which doesn't exist.
Is this kind of variable expansion not supported or is there a
different way that I should use to accomplish this?
If I am not misunderstand, at first, you want to use $() to get the variable you defined using the UI but failed. But later, ${{ }} can give you the value of the variable EnvironmentCode.
In fact, while you change to use ${{ }}, it just accessing the variable you predefined in the YAML files instead of the one you defined with UI. Just see this doc: Variable templates.
For the variable you defined with UI, it can be get and used with the format $()(Note: ${{ }} is the format of get the variables which defined in YAML file). But also, there some things you need to pay attention is for the variables you defined in UI, it can only be get/accessed after the build begin to run, because the variable which defined with UI exists in environment only after the build compiled begin. In one word, they are the agent-scope variable. That's why the value it used is still the one that's on the pipeline definition instead of on the UI.
If you remove from the pipeline definition the default value or the
variable entirely the expression ${{variables.EnvironmentCode}} will
return an empty string assuming the filename to be vars..yml which
doesn't exist.
As the document defined and I mentioned before, ${{}} is format which used to get the value of variable which defined in YAML files rather than the one which defined using UI.
In the steps of job, the variable that defined in the UI or defined in the YAML file can all be get/accessed with the format $(). Still, for the variable which defined in the YAML file can also get with ${{variables.xxxx}}. But at this time, if the variable name which defined in YAML file is same with the one defined with UI, the server can only get the one defined in YAML file.