I know that it is possible to conditionally set a variable in azure pipelines yml.
See https://learn.microsoft.com/en-us/azure/devops/pipelines/process/expressions?view=azure-devops#conditional-insertion
Is it also somehow possible to conditionally use a variable group?
Let's say if a pipeline run variable is set or has a certain value then the pipeline shall use a variable group. If not then the group shall not be used.
Thank you
Is it possible like here for instance:
trigger: none
pr: none
variables:
isProd: true
stages:
- stage: Test
displayName: Build and restore
variables:
- ${{ if eq(variables['isProd'], 'false') }}:
- group: QA
- ${{ if eq(variables['isProd'], 'true') }}:
- group: PROD
jobs:
- job: A
steps:
- bash: echo $(name)
Related
I have a pretty complex setup of my Pipelines in Azure DevOps for various reasons but I'm kind of stuck in a special scenario now. Let me explain a bit.
There is a Stage_A with Job_A setting a Variable_A. Now there is a Stage_B with Job_B, need to use the Variable_A from Stage_A.Job_A.
The variable in Job_A is set by this:
echo ##vso[task.setvariable variable=Variable_A;isOutput=true]$value
Now, Job_B in Stage_B can access the variable in a condition with
variables:
Variable_A_FromStageA: $[stageDependencies.Stage_A.Job_A.outputs['task_A.Variable_A']]
I can also do an echo on the variable by using
echo $(Variable_A_FromStageA)
the Question is now, how can I use this in an if-statement? I tried different approaches:
- ${{ if eq($(Variable_A_FromStageA), 'True') }}:
- ${{ if eq(variables.Variable_A_FromStageA, 'True') }}:
- ${{ if eq(variables['Variable_A_FromStageA'], 'True') }}:
- ${{ if eq(stageDependencies.Stage_A.Job_A.outputs['task_A.Variable_A'], "True") }}:
Nothing actually works. Either the system complains about syntax issues or it doesn't evaluate it correctly. I don't really know how to use the information in my if statement in the yaml file. The documentation is not really clear about it. It only mentions the usage of a stage dependency in a condition and that's it.
Hope anyone can help me here!
Cheers,
Frank
use stageDependencies in if-statements in Azure DevOps yaml
If you mean you want to use conditional insertion to use the variables output from the logging command, then answer is NO.
The reason is the conditional insertion needs compile time value(you must provide them before pipeline run.), but the variable that the logging command output is runtime. Conditional Insertion will be unable to get it.
The right way is to use "condition" instead of "Conditional Insertion". Using condition can achieve your situation.
I write a demo for you as below:
trigger:
- none
pool:
vmImage: ubuntu-latest
stages:
- stage: A
jobs:
- job: A1
steps:
- bash: echo "##vso[task.setvariable variable=shouldrun;isOutput=true]true"
# or on Windows:
# - script: echo ##vso[task.setvariable variable=shouldrun;isOutput=true]true
name: printvar
- stage: B
condition: and(succeeded(), eq(dependencies.A.outputs['A1.printvar.shouldrun'], 'true'))
variables:
myStageAVar: $[stageDependencies.A.A1.outputs['printvar.shouldrun']]
dependsOn: A
jobs:
- job: B1
steps:
- script: echo $(myStageAVar)
I would like to pass the variable group as a runtime parameter so that whenever I run the pipeline, it should allow me to provide the input variable group name, and based on the input value for the variable group during runtime my pipeline should proceed.
I want to achieve this when we click on the run button, then there's a variable section also. So, I want you to accept the variable group names from there.
Pipeline.yml:
stages:
- stage: VMBackupandValidate
displayName: 'VM Backup and Validate using RSV'
jobs:
- job: VMBackupValidate
displayName: 'Azure VM Backup'
steps:
- task: AzurePowerShell#5
inputs:
azureSubscription: $(azure_sc)
ScriptType: 'FilePath'
ScriptPath: 'pipelines/automation/scripts/vmbackup.ps1'
ScriptArguments: '-ResourceGroupName $(ResourceGroupName) -Storagetype $(Storagetype) -SourceVMname $(SourceVMname) -RSVaultname $(RSVaultname) -Location $(Location) -WorkLoadType $(WorkLoadType) -Policyname $(Policyname) -Verbose'
azurePowerShellVersion: 'LatestVersion'
pwsh: true
Based on comments communication with OP.
I suggest using a parameter with a default value. It will ask you for input if want other values, before you hit run then make a condition to select the right variable based on input.
Here is a minified sample of the pipeline:
parameters:
- name: environment
displayName: Deploy Environment
type: string
default: TEST
values:
- TEST
- PROD
trigger:
- 'none'
variables:
- name: environment
${{ if contains(parameters.environment, 'TEST') }}:
value: TEST
${{ if contains(parameters.environment, 'PROD') }}:
value: PROD
stages:
- stage: TEST
displayName: Build
condition: ${{ eq(variables.environment, 'TEST') }}
jobs:
- job:
pool:
vmImage: 'ubuntu-20.04'
steps:
- script: |
echo ${{ variables.environment}}
displayName: 'Print environment info'
You can extend the logic, or replace it with other values and consume it in code later. You can create multiple stages with conditions as well as shown.
Lets say you have two variable groups with names prod and test. You could use the below pipeline:
trigger:
- main
parameters:
- name: environment
displayName: Where to deploy?
type: string
default: test
values:
- prod
- test
pool:
vmImage: ubuntu-latest
variables:
- group: ${{parameters.environment}}
steps:
- script: |
echo $(ENV)
echo $(VERSION)
displayName: Step 1 - print version and environment
- script: pwd ENV ${{parameters.environment}}
displayName: Step 2 - print parameter
You should define ENV, VERSION values on both variable groups.
Your stage should stay as is. In your case you will delete the steps I provided and use only the first part of the pipeline
Adding a reference article.
https://blog.geralexgr.com/azure/deploy-between-different-environments-with-variable-groups-azure-devops?msclkid=002b01eab02f11ec8dffa95dc3a34094
I am trying to find a way to define a variable group at stage level and then access it in below jobs through a template? How would I go about doing this?
# Template file getchangedfilesandvariables.yaml
parameters:
- name: "previouscommitid"
type: string
steps:
- task: PowerShell#2
displayName: 'Get the changed files'
name: CommitIds
inputs:
targetType: 'filePath'
filePath: '$(Build.SourcesDirectory)\AzureDevOpsPipelines\Get-COChangedfiles.ps1'
arguments: >
-old_commit_id ${{ previouscommitid }}
- task: PowerShell#2
name: PassOutput
displayName: 'Getting Variables for Packaging'
inputs:
targetType: 'filepath'
filepath: '$(System.DefaultWorkingDirectory)\AzureDevOpsPipelines\Get-COADOvariables.ps1'
And below is my yaml file.
trigger: none
name: $(BuildID)
variables:
system.debug: true
CodeSigningCertThumbprint: "somethumbprint"
# Triggering builds on a branch itself.
${{ if startsWith(variables['Build.SourceBranch'], 'refs/heads/') }}:
branchName: $[ replace(variables['Build.SourceBranch'], 'refs/heads/', '') ]
# Triggering builds from a Pull Request.
${{ if startsWith(variables['Build.SourceBranch'], 'refs/pull/') }}:
branchName: $[ replace(variables['System.PullRequest.SourceBranch'], 'refs/heads/', '') ]
## it will create pipeline package and it will push it private or public feed artifacts
stages:
- stage: Stage1
variables:
- group: Cloudops
- name: oldcommitid
value: $[variables.lastcommitid]
jobs:
- job: IdentifyChangedFilesAndGetADOVariables
pool:
name: OnPrem
workspace:
clean: all # Ensure the agent's directories are wiped clean before building.
steps:
- powershell: |
[System.Version]$PlatformVersion = ((Get-Content "$(Build.SourcesDirectory)\AzureDevOpsPipelines\PlatformVersion.json") | ConvertFrom-Json).PlatformVersion
Write-Output "The repository's PlatformVersion is: $($PlatformVersion.ToString())"
$NewPackageVersion = New-Object -TypeName "System.Version" -ArgumentList #($PlatformVersion.Major, $PlatformVersion.Minor, $(Build.BuildId))
Write-Output "This run's package version is $($NewPackageVersion.ToString())"
echo "##vso[task.setvariable variable=NewPackageVersion]$($NewPackageVersion.ToString())"
echo "##vso[task.setvariable variable=commitidold;isOutput=true]$(oldcommitid)"
displayName: 'Define package version.'
name: commitidorpackageversion
errorActionPreference: stop
- template: getchangedfilesandvariables.yaml
parameters:
previouscommitid:
- $(commitidorpackageversion.commitidold)
# - $(oldcommitid)
I get the error at the second last line of the code that
/AzureDevOpsPipelines/azure-pipelines.yml (Line: 49, Col: 13): The 'previouscommitid' parameter is not a valid String.
I tried different combinations but I am still getting the errors.
Any ideas?
Thanks for your response. I already had the variable group setup in my library. I was just not able to use it.
The way I was able to achieve this I created another template file and supplied it to variables section under my stage. After doing this I was able to actually able to use the variables from my variable group in my successive jobs.
For more information you can review this doc : https://learn.microsoft.com/en-us/azure/devops/pipelines/library/variable-groups?view=azure-devops&tabs=yaml
stagevariables.yaml
variables:
- group: Cloudops
azure-pipelines.yml
stages:
- stage: Stage1
variables:
- template: stagevariables.yaml
jobs:
- job: CheckwhichfeedsAreAvailable
In YAML pipeline, you can't define a new variable group under the variables key.
Actually, we do not have the syntax can be available to create new variable group when running the YAML pipeline.
Under the variables key, you can:
Define new variables with the specified values.
Override the existing variables with new values.
Reference the variables from the existing variable groups and variable templates.
So, if you want to use a variable group with some variables in the pipeline, you should manually define the variable group on the Pipelines > Library page, then reference it in the pipeline.
Using the Azure Devops gui, it is very easy to scope variable groups to pipeline stages. I need to replicate this functionality in a yaml build/release pipeline but I cannot find a way to do it. Anyone found a way to do this yet?
Scope variable groups to pipeline stages functionality in a yaml
build/release pipeline
With YAML, the way to achieve that is just specify the Variable group at the stage level to let it available only to this specific stage.
For example, I have a variable group names 1122. And 2 stages: one and two. Now, I want this variable group only available for stage one, which means the stage two should not access its content.
See below simple sample:
stages:
- stage: one
displayName: one
variables:
- group: 1122
jobs:
- job: A
steps:
- bash: echo $(a)
- stage: two
displayName: two
jobs:
- job: A1
steps:
- bash: echo $(a)
parameters:
- name: stage
displayName: Stage
type: string
default: Development
values:
- Development
- Staging
- NonProd
variables:
serviceConnection: 'Your-Favourite-SC'
stages:
- stage: Deploy_Resources
displayName: Deploy Resources
variables:
- group: "Somevargroupname - Release"
- group: "Somevargroupname - ${{ parameters.stage }}"
Quick and simple!
I'm trying to assign one of 2 values to a variable in addition to variable group and can't find the reference that how to use IF ELSE.
Basically I need to convert this jerkins logic to azure DevOps.
Jenkins
if (branch = 'master') {
env = 'a'
} else if (branch = 'dev'){
env ='b'
}
I found 1 reference from the following one, but this one seems to work if the variables section doesn't have variable groups.
https://stackoverflow.com/a/57532526/5862540
But in my pipeline, I already have a variable group for secrets, so I have to use name/value convention and the example doesn't work with the errors like expected a mapping or A mapping was not expected or Unexpected value 'env'
variables:
- group: my-global
- name: env
value:
${{ if eq(variables['Build.SourceBranchName'], 'master') }}:
env: a
${{ if eq(variables['Build.SourceBranchName'], 'dev') }}:
env: b
or
variables:
- group: my-global
- name: env
value:
${{ if eq(variables['Build.SourceBranchName'], 'master') }}: a
${{ if eq(variables['Build.SourceBranchName'], 'dev') }}: b
This code works.
I'm doing similar with parameters.
variables:
- name: var1
${{ if eq(parameters.var1, 'custom') }}:
value: $(var1.manual.custom)
${{ if ne(parameters.var1, 'custom') }}:
value: ${{ parameters.var1 }}
Update 09/09/2021
We have now natively if else expression and we can write it like
variables:
- group: PROD
- name: env
${{ if eq(variables['Build.SourceBranchName'], 'master') }}:
value: a
${{ else }}:
value: b
steps:
- script: |
echo '$(name)'
echo '$(env)'
Original reply
Syntax with template expressions ${{ if ...... }} is not limited only to job/stage level. Both below pipeline does the same and produce the same output:
stages:
- stage: One
displayName: Build and restore
variables:
- group: PROD
- name: env
${{ if eq(variables['Build.SourceBranchName'], 'master') }}:
value: a
${{ if eq(variables['Build.SourceBranchName'], 'dev') }}:
value: b
jobs:
- job: A
steps:
- script: |
echo '$(name)'
echo '$(env)'
variables:
- group: PROD
- name: env
${{ if eq(variables['Build.SourceBranchName'], 'master') }}:
value: a
${{ if eq(variables['Build.SourceBranchName'], 'dev') }}:
value: b
steps:
- script: |
echo '$(name)'
echo '$(env)'
Microsoft a few weeks ago released a new feature for YAML pipeliens that just lets you do that: IF ELSE notation.
https://learn.microsoft.com/en-us/azure/devops/release-notes/2021/sprint-192-update#new-yaml-conditional-expressions
Writing conditional expressions in YAML files just got easier with the use of ${{ else }} and ${{ elseif }} expressions. Below are examples of how to use these expressions in YAML pipelines files.
steps:
- script: tool
env:
${{ if parameters.debug }}:
TOOL_DEBUG: true
TOOL_DEBUG_DIR: _dbg
${{ else }}:
TOOL_DEBUG: false
TOOL_DEBUG_DIR: _dbg
variables:
${{ if eq(parameters.os, 'win') }}:
testsFolder: windows
${{ elseif eq(parameters.os, 'linux' }}:
testsFolder: linux
${{ else }}:
testsFolder: mac
I wanted to have runtime condition evaluation, something similar to compile time:
variables:
VERBOSE_FLAG:
${{if variables['System.Debug']}}:
value: '--verbose'
${{else}}:
value: ''
but unfortunately Azure devops does not supports special kind of functions like if(condition, then case, else case) - so I've played around and find out that it's possible do double string replacement using replace function. It does looks bit hacky of course.
For example, one may want to tweak task inputs depending on whether system debugging is enabled or not. This cannot be done using "standard conditional insertion" (${{ if … }}:), because System.Debug isn't in scope in template expressions. So, runtime expressions to the rescue:
- job:
variables:
VERBOSE_FLAG: $[
replace(
replace(
eq(lower(variables['System.Debug']), 'true'),
True,
'--verbose'
),
False,
''
)
]
steps:
- task: cURLUploader#2
inputs:
# …
options: --fail --more-curl-flags $(VERBOSE_FLAG)
Note that using eq to check the value of System.Debug before calling replace is not redundant: Since eq always returns either True or False, we can then safely use replace to map those values to '--verbose' and '', respectively.
In general, I highly recommend sticking to boolean expressions (for example the application of a boolean-valued function like eq, gt or in) as the first argument of the inner replace application. Had we not done so and instead just written for example
replace(
replace(
lower(variables['System.Debug']),
'true',
'--verbose'
),
'false',
''
)
then, if System.Debug were set to e.g. footruebar, the value of VERBOSE_FLAG would have become foo--verbosebar.
I think for now you're going to need to use a task to customize with name/value syntax variables and conditional variable values. It looks like the object structure for name/value syntax breaks the parsing of expressions, as you have pointed out.
For me, the following is a reasonably clean implementation, and if you want to abstract it away from the pipeline, it seems that a simple template for your many pipelines to use should satisfy the desire for a central "global" location.
variables:
- group: FakeVarGroup
- name: env
value: dev
steps:
- powershell: |
if ($env:Build_SourceBranchName -eq 'master') {
Write-Host ##vso[task.setvariable variable=env;isOutput=true]a
return
} else {
Write-Host ##vso[task.setvariable variable=env;isOutput=true]b
}
displayName: "Set Env Value"
As far as I know, the best way to have conditional branch build is using "trigger" in your YAML, instead of implementing complex "if-else". It is also much safer, and you have more explicit controls on the branch triggers instead of relying on CI variables.
Example:
# specific branch build
jobs:
- job: buildmaster
pool:
vmImage: 'vs2017-win2016'
trigger:
- master
steps:
- script: |
echo "trigger for master branch"
- job: buildfeature
pool:
vmImage: 'vs2017-win2016'
trigger:
- feature
steps:
- script: |
echo "trigger for feature branch"
To have trigger with branches inclusion and exclusion, you could use more complex syntax of trigger with branches include and exclude.
Example:
# specific branch build
trigger:
branches:
include:
- master
- releases/*
exclude:
- releases/1.*
The official documentation of Azure DevOps Pipelines trigger in YAML is:
Azure Pipelines YAML trigger documentation
UPDATE 1:
I repost my comment here with additional notes:
I was thinking to have different pipelines because having the complexity of juggling between CI variables is not more maintainable than having multi jobs in one YAML with triggers. Having multijobs with triggers is also enforcing us to have clear distinction and provision on branch management. Triggers and conditional branches inclusions have been used for a year by my team because of these maintainability advantages.
Feel free to disagree, but to me having an embedded logic in any scripted in any steps to check which branch is currently in session and then does any further actions, are more like ad-hoc solutions. And this has given my team and me maintenance problems before.
Especially if the embedded logic tends to grow by checking other branches, the complexity is more complex later than having clear separations between branches. Also if the YAML file is going to be maintained for long time, it should have clear provisions and roadmaps across different branches. Redundancy is unavoidable, but the intention to separate specific logic will pay more in the long run for maintainability.
This is why I also emphasize branches inclusions and exclusions in my answer :)
Azure YAML if-else solution (when you have a group defined which required name/value notation use thereafter.
variables:
- group: my-global
- name: env
value: a # set by default
- name: env
${{ if eq(variables['Build.SourceBranchName'], 'master') }}:
value: b # will override default
Of if you don't have a group defined:
variables:
env: a # set by default
${{ if eq(variables['Build.SourceBranchName'], 'master') }}:
env: b # will override default