I have a sub phase in my Azure pipeline build called integrationSteps , which I think is a built-in, and typical way of running post-deploy tests.
This code fails to expand the ENVIRONMENT variable in the if-condition. The bash step prints out all the variables, and ENVIRONMENT=dev is one of the variables that gets printed out. After the bash step, the if-condition fails, but the azure pipeline doesn't tell me it isn't parsable :
integrationSteps:
- bash: |
env | sort
displayName: 'Debug $(ENVIRONMENT)'
- ${{ if eq(variables['ENVIRONMENT'], 'dev') }}:
- checkout: self
path: it
Related
I have a pretty complex setup of my Pipelines in Azure DevOps for various reasons but I'm kind of stuck in a special scenario now. Let me explain a bit.
There is a Stage_A with Job_A setting a Variable_A. Now there is a Stage_B with Job_B, need to use the Variable_A from Stage_A.Job_A.
The variable in Job_A is set by this:
echo ##vso[task.setvariable variable=Variable_A;isOutput=true]$value
Now, Job_B in Stage_B can access the variable in a condition with
variables:
Variable_A_FromStageA: $[stageDependencies.Stage_A.Job_A.outputs['task_A.Variable_A']]
I can also do an echo on the variable by using
echo $(Variable_A_FromStageA)
the Question is now, how can I use this in an if-statement? I tried different approaches:
- ${{ if eq($(Variable_A_FromStageA), 'True') }}:
- ${{ if eq(variables.Variable_A_FromStageA, 'True') }}:
- ${{ if eq(variables['Variable_A_FromStageA'], 'True') }}:
- ${{ if eq(stageDependencies.Stage_A.Job_A.outputs['task_A.Variable_A'], "True") }}:
Nothing actually works. Either the system complains about syntax issues or it doesn't evaluate it correctly. I don't really know how to use the information in my if statement in the yaml file. The documentation is not really clear about it. It only mentions the usage of a stage dependency in a condition and that's it.
Hope anyone can help me here!
Cheers,
Frank
use stageDependencies in if-statements in Azure DevOps yaml
If you mean you want to use conditional insertion to use the variables output from the logging command, then answer is NO.
The reason is the conditional insertion needs compile time value(you must provide them before pipeline run.), but the variable that the logging command output is runtime. Conditional Insertion will be unable to get it.
The right way is to use "condition" instead of "Conditional Insertion". Using condition can achieve your situation.
I write a demo for you as below:
trigger:
- none
pool:
vmImage: ubuntu-latest
stages:
- stage: A
jobs:
- job: A1
steps:
- bash: echo "##vso[task.setvariable variable=shouldrun;isOutput=true]true"
# or on Windows:
# - script: echo ##vso[task.setvariable variable=shouldrun;isOutput=true]true
name: printvar
- stage: B
condition: and(succeeded(), eq(dependencies.A.outputs['A1.printvar.shouldrun'], 'true'))
variables:
myStageAVar: $[stageDependencies.A.A1.outputs['printvar.shouldrun']]
dependsOn: A
jobs:
- job: B1
steps:
- script: echo $(myStageAVar)
How to create a multi-stage pipeline depending on a stage/job-name derived from a parameter whereas stages run firstly in parallel and eventually one stage that waits for all previous stages?
Here's what I've tried so far:
A multi-stage pipeline runs for several stages depending on a tool parameter in parallel, whereas dependsOn is passed as parameter. Running it in parallel for each tool waiting for the previous stage for the said tool works smoothly.
Main template: all wait for for all
- ${{ each tool in parameters.Tools }}:
- template: ../stages/all-wait-for-all.yml
parameters:
Tool: ${{ tool }}
stages/all-wait-for-all.yml
parameters:
- name: Tool
type: string
stages:
- stage: ALL_WAIT_${{ parameters.Tool}}
dependsOn:
- PREPARE_STAGE
- OTHER_TEMPLATE_EXECUTED_FOR_ALL_TOOLS_${{ parameters.Tool }}
Now there should be one stage that should only run once and not per tool, but it should only run after the individual tool stages are done. It can't be hardcoded as there are various tools. So I hoped defining the individual wait-stages in a prepare job would work out:
Main template: prepare-stage
- script: |
toolJson=$(echo '${{ convertToJson(parameters.Tools) }}')
tools=$(echo "$toolJson" | jq '.[]' | xargs)
stage="ALL_WAIT"
for tool in $tools; do
stageName="${stage}_${tool }"
stageWaitArray+=($stageName)
done
echo "##vso[task.setvariable variable=WAIT_ON_STAGES]${stageWaitArray}"
echo "##vso[task.setvariable variable=WAIT_ON_STAGES;isOutput=true]${stageWaitArray}"
displayName: "Define wait stages"
name: WaitStage
stages/one-waits-for-all.yml
stages:
- stage: ONE_WAITS
dependsOn:
- $[ stageDependencies.PREPARE_STAGE.PREPARE_JOB.outputs['waitStage.WAIT_ON_STAGES'] ]
whereas below error is shown:
Stage ONE_WAITS depends on unknown stage $[ stageDependencies.PREPARE_STAGE.PREPARE_JOB.outputs['WaitStage.WAIT_ON_STAGES'] ].
As I understand depends on can not have dynamic $[] or macro $() expressions evaluated at runtime. You can use template expressions ${{}} which are evaluated at queue time.
Guess I was overthinking the solution as eventually it was pretty obvious.
So first template can be called within a loop from the main template whereas it's executed as many times as tools we got. Second template shall be called once waiting on previous stages for all tools, whereas the job/stage prefix is known, only the tool name as postfix was unknown. So just add them in a loop directly in dependsOn..
Here you go:
stages:
- stage: ONE_WAITS
dependsOn:
- PREPARE_STAGE
- ${{ each tool in parameters.Tools }}:
- OTHER_TEMPLATE_EXECUTED_FOR_ALL_TOOLS_${{ tool}}
- ALL_WAIT_${{ tool }}
I am trying to use if else conditions in Azure Devops yml pipeline with variable groups. I am trying to implement it as per latest Azure Devops yaml pipeline build.
Following is the sample code for the if else condition in my scenario. test is a variable inside my-global variable group.
variables:
- group: my-global
- name: fileName
${{ if eq(variables['test'], 'true') }}:
value: 'product.js'
${{ elseif eq(variables['test'], false) }}:
value: 'productCost.js'
jobs:
- job:
steps:
- bash:
echo test variable value $(fileName)
When the above code is executed, in echo statement we don't see any value for filename, i.e. it empty, meaning none of the above if else condition was executed, however when I test the if else condition with the following condition.
- name: fileName
${{ if eq('true', 'true') }}:
value: 'product.js'
Filename did echo the correct value, i.e. product.js. So my conclusion is that I am not able to refer the variables from the variable group correctly. So any suggestion will be helpful and appreciated.
Thanks!
Unfortunately there is no ternary operator in Azure DevOps Pipelines. And it seems unlikely considering the state of https://github.com/microsoft/azure-pipelines-yaml/issues/256 and https://github.com/microsoft/azure-pipelines-yaml/issues/278. So for the time being the only choices are :
conditional insertion : it works with parameters, and should work with variables according to the documentation (but it is difficult to use properly),
or the hacks you can find in this Stack Overflow question.
Another work-around has been posted by Simon Alling on GitHub (https://github.com/microsoft/azure-pipelines-yaml/issues/256#issuecomment-1077684972) :
format(
replace(replace(condition, True, '{0}'), False, '{1}'),
valueIfTrue,
valueIfFalse
)
It is similar to the solution provided by Tejas Nagchandi, but I find it a little bit better because the syntax looks closer to what it would be if there was a ternary operator.
I was able to achieve the goal using some dirty work-around, but I do agree that using parameters would be much better way unless ternary operators are available for Azure DevOps YAML pipeline.
The issue is that ${{ if condition }}: is compile time expression, thus the variables under variable group are not available.
I was able to use runtime expressions $[<expression>]
Reference: https://learn.microsoft.com/en-us/azure/devops/pipelines/process/expressions?view=azure-devops
My pipeline:
trigger:
- none
variables:
- group: Temp-group-for-testing
- name: fileName
value: $[replace(replace('True',eq(variables['test'], 'True'), 'value1'),'True','value2')]
stages:
- stage: test
jobs:
- job: testvar
continueOnError: false
steps:
- bash: echo $(fileName)
displayName: "echo variable"
Results are available on github
After detailed investigation I realized that if else doesnt work with variables in Az Devop yaml pipelines, it only works with parameters. However the solution posted by #Tejas Nagchandi is a workaround and might be able to accomplish the same logic of if else setting variable value with replace commands. Hats off to TN.
I am creating YAML pipeline in Azure DevOps that consists of two stages.
The first stage (Prerequisites) is responsible for reading the git commit and creates a comma separated variable containing the list of services that has been affected by the commit.
The second stage (Build) is responsible for building and unit testing the project. This Stage consists of many templates, one for each Service. In the template script, the job will check if the relevant Service in in the variable created in the previous stage. If the job finds the Service it will continue to build and test the service. However if it cannot find the service, it will skip that job.
Run.yml:
stages:
- stage: Prerequisites
jobs:
- job: SetBuildQueue
steps:
- task: powershell#2
name: SetBuildQueue
displayName: 'Set.Build.Queue'
inputs:
targetType: inline
script: |
## ... PowerShell script to get changes - working as expected
Write-Host "Build Queue Auto: $global:buildQueueVariable"
Write-Host "##vso[task.setvariable variable=buildQueue;isOutput=true]$global:buildQueueVariable"
- stage: Build
jobs:
- job: StageInitialization
- template: Build.yml
parameters:
projectName: Service001
projectLocation: src/Service001
- template: Build.yml
parameters:
projectName: Service002
projectLocation: src/Service002
Build.yml:
parameters:
projectName: ''
projectLocation: ''
jobs:
- job:
displayName: '${{ parameters.projectName }} - Build'
dependsOn: SetBuildQueue
continueOnError: true
condition: and(succeeded(), contains(dependencies.SetBuildQueue.outputs['SetBuildQueue.buildQueue'], '${{ parameters.projectName }}'))
steps:
- task: NuGetToolInstaller#1
displayName: 'Install Nuget'
Issue:
When the first stages runs it will create a variable called buildQueue which is populated as seen in the console output of the PowerShell script task:
Service001 Changed
Build Queue Auto: Service001;
However when it gets to stage two and it tries to run the build template, when it checks the conditions it returns the following output:
Started: Today at 12:05 PM
Duration: 16m 7s
Evaluating: and(succeeded(), contains(dependencies['SetBuildQueue']['outputs']['SetBuildQueue.buildQueue'], 'STARS.API.Customer.Assessment'))
Expanded: and(True, contains(Null, 'service001'))
Result: False
So my question is how do I set the dependsOn and condition to get the information from the previous stage?
It because you want to access the variable in a different stage from where you defined them. currently, it's impossible, each stage it's a new instance of a fresh agent.
In this blog you can find a workaround that involves writing the variable to disk and then passing it as a file, leveraging pipeline artifacts.
To pass the variable FOO from a job to another one in a different stage:
Create a folder that will contain all variables you want to pass; any folder could work, but something like mkdir -p $(Pipeline.Workspace)/variables might be a good idea.
Write the contents of the variable to a file, for example echo "$FOO" > $(Pipeline.Workspace)/variables/FOO. Even though the name could be anything you’d like, giving the file the same name as the variable might be a good idea.
Publish the $(Pipeline.Workspace)/variables folder as a pipeline artifact named variables
In the second stage, download the variables pipeline artifact
Read each file into a variable, for example FOO=$(cat $(Pipeline.Workspace)/variables/FOO)
Expose the variable in the current job, just like we did in the first example: echo "##vso[task.setvariable variable=FOO]$FOO"
You can then access the variable by expanding it within Azure Pipelines ($(FOO)) or use it as an environmental variable inside a bash script ($FOO).
I'm currently using Build in Visual Studio Team Services (was Visual Studio Online), and would like to be able to set a Build Variable in a Build Step so that the new value can be used in a subsequent Build Step.
Obviously you can set it before the Build starts but I'm looking to late bind the variable during a subsequent Build Step.
Is this possible?
When inside of a script you can update a variable by emitting the following in your ps1
"##vso[task.setvariable variable=testvar;]testvalue"
You can then pass the variable into the next script using $(testvar)
This doc from the API talks about what ##vso commands you can use.
Don't forget to set system.debug to true. It seems there is a bug that muted stdout and thus, all ##vso are not working.
https://github.com/Microsoft/vso-agent-tasks/blob/master/docs/authoring/commands.md
You can create a powershell script an reference it as a build task.
Then inside your powershell scripts add this:
"##vso[task.setvariable variable=key]value"
After that on all your tasks you can read the variable as $(key).
If you want to protect your variable, use:
"##vso[task.setvariable variable=secretVar;issecret=true]value"
And then use it as $(secretVar) in your next tasks.
I found this link helpful: https://learn.microsoft.com/en-us/azure/devops/pipelines/scripts/logging-commands?view=azure-devops&tabs=powershell
This has the complete options of what you can do: https://learn.microsoft.com/en-us/azure/devops/pipelines/process/variables?view=azure-devops&tabs=yaml%2Cbatch
You can reuse set variable from task to task, and also job to job. I couldn't find anything on stage to stage.
In summary:
jobs:
# Set an output variable from job A
- job: A
pool:
vmImage: 'vs2017-win2016'
steps:
- powershell: echo "##vso[task.setvariable variable=myOutputVar;isOutput=true]this is the value"
name: setvarStep
- script: echo $(setvarStep.myOutputVar)
name: echovar
# Map the variable into job B
- job: B
dependsOn: A
pool:
vmImage: 'ubuntu-16.04'
variables:
myVarFromJobA: $[ dependencies.A.outputs['setvarStep.myOutputVar'] ] # map in the variable
# remember, expressions require single quotes
steps:
- script: echo $(myVarFromJobA)
name: echovar