I'm trying to get Azure DevOps pipelines to prompt for a version number when manually launching a pipeline (defined using the new YAML syntax).
Even when I define variables in the template, the launch screen says "This pipeline has no defined variables". How can I define variables so that they show up in the pipeline launch?
Current YAML definition contains:
variables:
- name: versionName
value: ''
These are not shown when launching the pipeline:
While Shayki's answer is correct for defining variables, what I was really looking for is runtime parameters.
With the following YAML definition:
parameters:
- name: myParameter
displayName: Description of myParameter
default: defaultMyParameter
type: string
it prompts for the parameter value when launching the pipeline:
The parameter must be referenced in the template using ${{ parameters.myParameter }}, the other variable syntaxes don't work.
From the docs:
If a variable appears in the variables block of a YAML file, its
value is fixed and can't be overridden at queue time. Best practice is
to define your variables in a YAML file but there are times when this
doesn't make sense. For example, you may want to define a secret
variable and not have the variable exposed in your YAML. Or, you may
need to manually set a variable value during the pipeline run.
You have two options for defining queue-time values. You can define a
variable in the UI and select the option to Let users override this
value when running this pipeline or you can use runtime parameters
instead. If your variable is not a secret, the best practice is to use
runtime parameters.
To set a variable at queue time, add a new variable within your
pipeline and select the override option.
To allow a variable to be set at queue time, make sure the variable
doesn't also appear in the variables block of a pipeline or job. If
you define a variable in both the variables block of a YAML and in the
UI, the value in the YAML will have priority.
Related
I have setup a map(string) variable in terraform. and defined them in Azure YAML pipeline. i want to get the value for those YAML variable from variable group in Azure DevOps. if i give the direct value for the key, it works as expected. but when trying to access the value from variable groups i receive errors.
Defined Variable in Terraform:
Variables.tf
variable "secrets" {
type = map(string)
}
variables.tfvars
secrets = $(secrets)
in YAML pipeline:
displayName: DEV
variables:
- group: 'Environment - Dev'
- name: secrets
value: '{"testAPIKey1" = $(testAPIKey1) , "testAPIKey2" = $(testAPIKey2) }'
I was getting the error like below
Expected a closing parenthesis to terminate the expression. ##\[error\]Terraform command 'plan' failed with exit code '1'.: Unbalanced parentheses
\##\[error\]
Error: Unbalanced parentheses
Can some one suggest please how i can access the values from variable groups to YAML variables.
The values of the variable group are not available yet when the 2nd variable is set. The yaml pipelines follow a pretty opaque multi-stage parsing, transformation and execution order.
You can use a script task or my set variable task to set the value of the 2nd variable at runtime in the job.
https://marketplace.visualstudio.com/items?itemName=jessehouwing.jessehouwing-vsts-variable-tasks
See also: https://stackoverflow.com/a/74788506/736079
if I have steps
steps:
- bash: |
echo "versionType: $(versionType)"
and I define versionType in a variable group which is used by pipeline all works fine. If it is not defined it blows up. I'd like to use a default value in case if not defined to avoid blow up. How I can specify default value if not defined? I tried ${{ coalesce(variables.versionType,someType)}} and I always get someType.
Unfortunately variables do not directly allow for a default value the way a parameter would. If possible, consider using a parameter in which case you can use something like this:
parameters:
- name: versionType
type: string
default: ''
Otherwise, if you need it to be a variable group, you can perform a check before that step something like this:
- ${{ if not(eq(variables['versionType'], '')) }}:
- bash: |
echo "versionType: $(versionType)"
We have no method to specify a default value for undefined variables on Azure DevOps.
In Azure Pipelines, if you want to use a variable, you should define it with a specified value at first. You can define the variable directly on the current pipeline, in a variable group or in the variable template.
You also can use the logging command "SetVariable" to define a new variable or update the existing variable with the specified value in a running job. Then in the subsequent tasks in the job you can use the variable with the new value.
Is there a way to pass the Azure DevOps pipeline secrets without passing them as task env variables
As per the documentation, we can only pass the secrets like below
- powershell: |
Write-Host "recommended: $env:MY_MAPPED_ENV_VAR"
env:
MY_MAPPED_ENV_VAR: $(mySecret) # the recommended way to map to an env variable
This works okay when you have 1 or 2 variables, but managing 20 or more than that becomes so tedious.
Any comments or suggestions would be greatly appreciated
Update:
Is there a way to pass the Azure DevOps pipeline secrets without passing them as task env variables
I'm afraid there is no method to use secret variables directly without mapping them to the task environment.
From the document you mentioned：
Don't set secret variables in your YAML file. Operating systems often
log commands for the processes that they run, and you wouldn't want
the log to include a secret that you passed in as an input. Use the
script's environment or map the variable within the variables block to
pass secrets to your pipeline.
We don't recommend passing the secret variables directly into pipeline. So we could use the env variable to map the secret variable.
Generally, we only place a small number of secret variables in a single task.
Therefore, it is valuable to use environment variables to map secret variables.
I have a pipeline in Azure DevOps somewhat like this:
parameters:
- name: Scenario
displayName: Scenario suite
type: string
default: 'Default'
variables:
Scenario: ${{ parameters.Scenario }}
...
steps:
- script: echo Scenario is $(Scenario)
And I'm executing the pipeline via the VSTS CLI like this:
vsts build queue ... --variables Scenario=Test
When I run my pipeline, it seems that the parameter default value overwrites my cmd line specified variable value and I get the step output Scenario is Default. I tried something like Scenario: $[coalesce(variables['Scenario'], ${{ parameters.Scenario }})] but I think I got the syntax wrong because that caused a parsing issue.
What would be the best way to only use the parameter value if the Scenario variable has not already been set?
What would be the best way to only use the parameter value if the
Scenario variable has not already been set?
Sorry but as I know your scenario is not supported by design. The Note here has stated that:
When you set a variable in the YAML file, don't define it in the web editor as settable at queue time. You can't currently change variables that are set in the YAML file at queue time. If you need a variable to be settable at queue time, don't set it in the YAML file.
The --variables switch in command can only be used to overwrite the variables which are marked as Settable at queue time. Since yaml pipeline doesn't support Settable variables by design, your --variables Scenario=Test won't actually be passed when queuing the yaml pipeline.
Here're my several tests to prove that:
1.Yaml pipeline which doesn't support Settable variable at Queue time:
pool:
vmImage: 'windows-latest'
variables:
Scenario: Test
steps:
- script: echo Scenario is $(Scenario)
I ran the command vsts build queue ... --variables Scenario=Test123, the pipeline run started but the output log would always be Scenario is Test instead of expected Scenario is Test123. It proves that it's not Pipeline parameter overwrites variable value, instead the --variables Scenario=xxx doesn't get passed cause yaml pipeline doesn't support Settable variables.
2.Create Classic UI build pipeline with pipeline variable Scenario:
Queuing it via command az pipelines build queue ... --variables Scenario=Test12345(It has the same function like vsts build queue ... --variables Scenario=Test) only gives this error:
Could not queue the build because there were validation errors or warnings.
3.Then enable the Settable at queue time option of this variable:
Run the same command again and now it works to queue the build. Also it succeeds to overwrite the original pipeline variable with the new value set in command-line.
You can do similar tests like what I did to figure out the cause of the behavior you met.
In addition:
VSTS CLI has been deprecated and replaced by Azure CLI with the Azure DevOps extension for a long time. So now it's more recommend to use az pipelines build queue
instead.
Lance had a great suggestion, but here is how I ended up solving it:
- name: Scenario
displayName: Scenario suite
type: string
default: 'Default'
variables:
ScenarioFinal: $[coalesce(variables['Scenario'], '${{ parameters.Scenario }}')]
...
steps:
- script: echo Scenario is $(ScenarioFinal)
In this case we use the coalesce expression to assign the value of a new variable, ScenarioFinal. This way we can still use --variables Scenario=Test via the CLI or use the parameter via the pipeline UI. coalesce will take the first non-null value and effectively "reorder" the precedence Lance linked to here: https://learn.microsoft.com/en-us/azure/devops/pipelines/process/variables?view=azure-devops&tabs=yaml%2Cbatch#expansion-of-variables
(Note that there need to be single quotes around the parameter reference '${{}}' because the ${{}} is simply converted to to the value, but then the coalesce expression doesn't know how to interpret the raw value unless it has the single quotes around it to denote it as a string)
Note that the ability to set parameters via the CLI is a current feature suggestion here: https://github.com/Azure/azure-devops-cli-extension/issues/972
I'm attempting to create multiple pipelines in Azure DevOps but I would like to reuse the same pipeline YAML file with the differences per environment being loaded from a separate template variables file.
For that purpose I've created two variable files, which are located in the same folder as the pipeline definition:
# vars.dev.yml
variables:
- name: EnvironmentName
value: Development
# vars.prd.yml
variables:
- name: EnvironmentName
value: Production
And the definition of the pipeline is the following:
trigger: none
pr: none
variables:
- name: EnvironmentCode
value: dev
- name: EnvironmentFileName
value: vars.$('EnvironmentCode').yml
stages:
- stage: LoadVariablesPerEnvironment
displayName: Load Variables Per Environment
variables:
- template: $(EnvironmentFileName)
jobs:
- job: ShowcaseLoadedVariables
steps:
- pwsh: Write-Host "Variables have been loaded for the '$ENV:ENVIRONMENTNAME' environment"
displayName: Output Environment Variables
After importing the pipelines using the Azure DevOps UI, I can go to settings of each and set the Environment Code variable to whatever desired environment code:
However I'm always getting the same error when I try to run the pipeline, regardless of the code I fill in the variable value:
So the question here is: Is this kind of variable expansion not supported or is there a different way that I should use to accomplish this?
Thanks!
EDIT
I was able to expand the variables using another method. The new version of the pipeline is as such:
variables:
- name: EnvironmentCode
value: dev
- name: EnvironmentFileName
value: vars.${{ variables.EnvironmentCode }}.yml
stages:
- stage: LoadVariablesPerEnvironment
displayName: Load Variables Per Environment
variables:
- template: ${{ variables.EnvironmentFileName }}
jobs:
- job: ShowcaseLoadedVariables
steps:
- pwsh: Write-Host "Variables have been loaded for the '$ENV:ENVIRONMENTNAME' environment"
displayName: Output Environment Variables
However there is yet the issue of loading different files. I made different attempts and verified the following:
If you give a different environment code using the UI, when running
the pipeline, the value it assumes is still the one that's on the
pipeline definition;
If you remove from the pipeline definition the
default value or the variable entirely the expression
${{variables.EnvironmentCode}} will return an empty string
assuming the filename to be vars..yml which doesn't exist.
Is this kind of variable expansion not supported or is there a
different way that I should use to accomplish this?
If I am not misunderstand, at first, you want to use $() to get the variable you defined using the UI but failed. But later, ${{ }} can give you the value of the variable EnvironmentCode.
In fact, while you change to use ${{ }}, it just accessing the variable you predefined in the YAML files instead of the one you defined with UI. Just see this doc: Variable templates.
For the variable you defined with UI, it can be get and used with the format $()(Note: ${{ }} is the format of get the variables which defined in YAML file). But also, there some things you need to pay attention is for the variables you defined in UI, it can only be get/accessed after the build begin to run, because the variable which defined with UI exists in environment only after the build compiled begin. In one word, they are the agent-scope variable. That's why the value it used is still the one that's on the pipeline definition instead of on the UI.
If you remove from the pipeline definition the default value or the
variable entirely the expression ${{variables.EnvironmentCode}} will
return an empty string assuming the filename to be vars..yml which
doesn't exist.
As the document defined and I mentioned before, ${{}} is format which used to get the value of variable which defined in YAML files rather than the one which defined using UI.
In the steps of job, the variable that defined in the UI or defined in the YAML file can all be get/accessed with the format $(). Still, for the variable which defined in the YAML file can also get with ${{variables.xxxx}}. But at this time, if the variable name which defined in YAML file is same with the one defined with UI, the server can only get the one defined in YAML file.