I have setup a map(string) variable in terraform. and defined them in Azure YAML pipeline. i want to get the value for those YAML variable from variable group in Azure DevOps. if i give the direct value for the key, it works as expected. but when trying to access the value from variable groups i receive errors.
Defined Variable in Terraform:
Variables.tf
variable "secrets" {
type = map(string)
}
variables.tfvars
secrets = $(secrets)
in YAML pipeline:
displayName: DEV
variables:
- group: 'Environment - Dev'
- name: secrets
value: '{"testAPIKey1" = $(testAPIKey1) , "testAPIKey2" = $(testAPIKey2) }'
I was getting the error like below
Expected a closing parenthesis to terminate the expression. ##\[error\]Terraform command 'plan' failed with exit code '1'.: Unbalanced parentheses
\##\[error\]
Error: Unbalanced parentheses
Can some one suggest please how i can access the values from variable groups to YAML variables.
The values of the variable group are not available yet when the 2nd variable is set. The yaml pipelines follow a pretty opaque multi-stage parsing, transformation and execution order.
You can use a script task or my set variable task to set the value of the 2nd variable at runtime in the job.
https://marketplace.visualstudio.com/items?itemName=jessehouwing.jessehouwing-vsts-variable-tasks
See also: https://stackoverflow.com/a/74788506/736079
Related
So I'm working on a bunch of pipelines and I've set everything up using yml templates. However I struggle with getting protected variable expanded inside of my template steps. I've tried passing in the protected variables by normal means, but they seem to not get expanded. Then I tried using a variable group, which I supposedly can directly reference inside of templates. I say supposedly, because Microsoft says so on their website https://learn.microsoft.com/en-us/azure/devops/pipelines/library/variable-groups?view=azure-devops&tabs=yaml:
"You can also reference a variable group in a template. In the template variables.yml, the group my-variable-group is referenced. The variable group includes a variable named myhello."
variables:
- group: my-variable-group
However, whenever I include a variables section into my template code, Azure DevOps immediately complains about it when parsing the yml, before running the pipeline. It spits out the following message:
/ymls/my-template#my-repo (Line: 1, Col: 1): Unexpected value 'variables'
I don't insist on using variable groups, I just want to have my protected variables expanded in my yml template. Does anybody know how to do that???
Any help greatly appreciated!
You should define your variable group on your main pipeline and not in the template. Then you can call your template and use the variable that you defined.
For example lets say that you have your main.yml which calls template.yml
You should define the below variable group on main.yml
variables:
- group: my-variable-group
And call the variable on your template.yml
$(MY_VARIABLE)
https://thomasthornton.cloud/2021/09/02/referencing-variable-groups-in-azure-devops-pipeline-templates/
Finally figured it out. Thanks to #GeralexGR. Turns out, when you reference a variable group in the main pipeline yml, you automatically have access to it in the template yml. But for normal script steps you still have to pass it in explicitly.
It then looks s.th. like this:
main-pipeline.yml:
variables:
- group: my-variable-group
...
jobs:
- job: my-job
steps:
- template: ymls/my-template.yml#my-repo
# no need to pass in the variable group s parameter
my-template.yml:
steps:
- task: ShellScript#2
inputs:
scriptPath: 'my-script.sh'
args: '$(my-secret-variable)'
if I have steps
steps:
- bash: |
echo "versionType: $(versionType)"
and I define versionType in a variable group which is used by pipeline all works fine. If it is not defined it blows up. I'd like to use a default value in case if not defined to avoid blow up. How I can specify default value if not defined? I tried ${{ coalesce(variables.versionType,someType)}} and I always get someType.
Unfortunately variables do not directly allow for a default value the way a parameter would. If possible, consider using a parameter in which case you can use something like this:
parameters:
- name: versionType
type: string
default: ''
Otherwise, if you need it to be a variable group, you can perform a check before that step something like this:
- ${{ if not(eq(variables['versionType'], '')) }}:
- bash: |
echo "versionType: $(versionType)"
We have no method to specify a default value for undefined variables on Azure DevOps.
In Azure Pipelines, if you want to use a variable, you should define it with a specified value at first. You can define the variable directly on the current pipeline, in a variable group or in the variable template.
You also can use the logging command "SetVariable" to define a new variable or update the existing variable with the specified value in a running job. Then in the subsequent tasks in the job you can use the variable with the new value.
I have setup a in my azure pipeline which create a Redis cache instance through Azure CLI.
There is another task which runs afterwards that pick set the values in my application config file from a pipeline variable named "CacheConnectionKey".
I have to set the variable value manually in the pipeline. I want to automate this process by adding a new task in between both described above. The new task should get PrimaryKey from Redis cache instance and set the value in the pipeline variable (i.e. CacheConnectionKey).
There is a command I have tried in the power shell which is giving me the access keys:
Get-AzRedisCacheKey -ResourceGroupName "MyResourceGroup" -Name "MyCacheKey"
PrimaryKey : pJ+jruGKPHDKsEC8kmoybobH3TZx2njBR3ipEsquZFo=
SecondaryKey : sJ+jruGKPHDKsEC8kmoybobH3TZx2njBR3ipEsquZFo=
Now I want to set PrimaryKey resulting from this command to be set in the pipeline variable CacheConnectionKey so that the next process could use the value properly.
The "process" referred in the question could be anything like a run/job/stage/pipeline I suppose. Regardless, in YAML pipelines, you can set variables at the root, stage, and job level. You can also use a variable group to make variables available across multiple pipelines. Some tasks define output variables, which you can consume in downstream steps, jobs, and stages.
In YAML, you can access variables across jobs and stages by using dependencies. By default, each stage in a pipeline depends on the one just before it in the YAML file. So if you need to refer to a stage that isn't immediately prior to the current one, you can override this automatic default by adding a dependsOn section to the stage.
For example, let's assume you have a task called MyTask, which sets an output variable called MyVar.
To use outputs in the same job:
steps:
- task: MyTask#1 # this step generates the output variable
name: ProduceVar # because we're going to depend on it, we need to name the step
- script: echo $(ProduceVar.MyVar) # this step uses the output variable
To use outputs in a different job:
jobs:
- job: A
steps:
# assume that MyTask generates an output variable called "MyVar"
- task: MyTask#1
name: ProduceVar # because we're going to depend on it, we need to name the step
- job: B
dependsOn: A
variables:
# map the output variable from A into this job
varFromA: $[ dependencies.A.outputs['ProduceVar.MyVar'] ]
steps:
- script: echo $(varFromA) # this step uses the mapped-in variable
For more details on the syntax and examples, check the following articles:
Use output variables from tasks
Dependencies
I'm trying to get Azure DevOps pipelines to prompt for a version number when manually launching a pipeline (defined using the new YAML syntax).
Even when I define variables in the template, the launch screen says "This pipeline has no defined variables". How can I define variables so that they show up in the pipeline launch?
Current YAML definition contains:
variables:
- name: versionName
value: ''
These are not shown when launching the pipeline:
While Shayki's answer is correct for defining variables, what I was really looking for is runtime parameters.
With the following YAML definition:
parameters:
- name: myParameter
displayName: Description of myParameter
default: defaultMyParameter
type: string
it prompts for the parameter value when launching the pipeline:
The parameter must be referenced in the template using ${{ parameters.myParameter }}, the other variable syntaxes don't work.
From the docs:
If a variable appears in the variables block of a YAML file, its
value is fixed and can't be overridden at queue time. Best practice is
to define your variables in a YAML file but there are times when this
doesn't make sense. For example, you may want to define a secret
variable and not have the variable exposed in your YAML. Or, you may
need to manually set a variable value during the pipeline run.
You have two options for defining queue-time values. You can define a
variable in the UI and select the option to Let users override this
value when running this pipeline or you can use runtime parameters
instead. If your variable is not a secret, the best practice is to use
runtime parameters.
To set a variable at queue time, add a new variable within your
pipeline and select the override option.
To allow a variable to be set at queue time, make sure the variable
doesn't also appear in the variables block of a pipeline or job. If
you define a variable in both the variables block of a YAML and in the
UI, the value in the YAML will have priority.
I have a pipeline in Azure DevOps somewhat like this:
parameters:
- name: Scenario
displayName: Scenario suite
type: string
default: 'Default'
variables:
Scenario: ${{ parameters.Scenario }}
...
steps:
- script: echo Scenario is $(Scenario)
And I'm executing the pipeline via the VSTS CLI like this:
vsts build queue ... --variables Scenario=Test
When I run my pipeline, it seems that the parameter default value overwrites my cmd line specified variable value and I get the step output Scenario is Default. I tried something like Scenario: $[coalesce(variables['Scenario'], ${{ parameters.Scenario }})] but I think I got the syntax wrong because that caused a parsing issue.
What would be the best way to only use the parameter value if the Scenario variable has not already been set?
What would be the best way to only use the parameter value if the
Scenario variable has not already been set?
Sorry but as I know your scenario is not supported by design. The Note here has stated that:
When you set a variable in the YAML file, don't define it in the web editor as settable at queue time. You can't currently change variables that are set in the YAML file at queue time. If you need a variable to be settable at queue time, don't set it in the YAML file.
The --variables switch in command can only be used to overwrite the variables which are marked as Settable at queue time. Since yaml pipeline doesn't support Settable variables by design, your --variables Scenario=Test won't actually be passed when queuing the yaml pipeline.
Here're my several tests to prove that:
1.Yaml pipeline which doesn't support Settable variable at Queue time:
pool:
vmImage: 'windows-latest'
variables:
Scenario: Test
steps:
- script: echo Scenario is $(Scenario)
I ran the command vsts build queue ... --variables Scenario=Test123, the pipeline run started but the output log would always be Scenario is Test instead of expected Scenario is Test123. It proves that it's not Pipeline parameter overwrites variable value, instead the --variables Scenario=xxx doesn't get passed cause yaml pipeline doesn't support Settable variables.
2.Create Classic UI build pipeline with pipeline variable Scenario:
Queuing it via command az pipelines build queue ... --variables Scenario=Test12345(It has the same function like vsts build queue ... --variables Scenario=Test) only gives this error:
Could not queue the build because there were validation errors or warnings.
3.Then enable the Settable at queue time option of this variable:
Run the same command again and now it works to queue the build. Also it succeeds to overwrite the original pipeline variable with the new value set in command-line.
You can do similar tests like what I did to figure out the cause of the behavior you met.
In addition:
VSTS CLI has been deprecated and replaced by Azure CLI with the Azure DevOps extension for a long time. So now it's more recommend to use az pipelines build queue
instead.
Lance had a great suggestion, but here is how I ended up solving it:
- name: Scenario
displayName: Scenario suite
type: string
default: 'Default'
variables:
ScenarioFinal: $[coalesce(variables['Scenario'], '${{ parameters.Scenario }}')]
...
steps:
- script: echo Scenario is $(ScenarioFinal)
In this case we use the coalesce expression to assign the value of a new variable, ScenarioFinal. This way we can still use --variables Scenario=Test via the CLI or use the parameter via the pipeline UI. coalesce will take the first non-null value and effectively "reorder" the precedence Lance linked to here: https://learn.microsoft.com/en-us/azure/devops/pipelines/process/variables?view=azure-devops&tabs=yaml%2Cbatch#expansion-of-variables
(Note that there need to be single quotes around the parameter reference '${{}}' because the ${{}} is simply converted to to the value, but then the coalesce expression doesn't know how to interpret the raw value unless it has the single quotes around it to denote it as a string)
Note that the ability to set parameters via the CLI is a current feature suggestion here: https://github.com/Azure/azure-devops-cli-extension/issues/972