I am building Azure Pipelines for deployments of intranet, dotnet core projects from our Windows Self hosted agent.
I have a request from the team where if a PowerShell script initialize a variable to some value, the next step should include a specific environment.
The goal is, for some project, we compare a file from the agent to the DEV server and if different we request an approval.
I'm trying to make this work with global variables and get confused with different syntax.
Here's what I have:
YAML
variables:
- name: MYVAR
value: '0'
stages:
- stage: Init
jobs:
- job: DisplayBefore
steps:
- checkout: none
- task: CmdLine#2
inputs:
script: |
echo Varname before :
echo $(MYVAR)
===> Displays: 0
- stage: RunScript
dependsOn: Init
jobs:
- job: CallScript
steps:
- checkout: none
- task: PowerShell#2
name: SetVarname
inputs:
targetType: filePath
filePath: '{agent}\compareFileContent.ps1'
- job: DisplayInSameStage
dependsOn: CallScript
steps:
- checkout: none
- task: CmdLine#2
inputs:
script: |
echo Varname after :
echo ${{ variables.MYVAR }}
===> Displays: 0
echo $(MYVAR)
===> Displays: 0
- stage: DeploymentStage
dependsOn: RunScript
condition: eq(variables.MYVAR, '1')
jobs:
- deployment: DeploymentJob
environment: 'ATest'
strategy:
runOnce:
deploy:
steps:
- checkout: none
- task: CmdLine#2
inputs:
script: |
echo Varname after :
echo ${{ variables.MYVAR }}
{agent}\compareFileContent.ps1
[CmdletBinding()]
param ()
echo $env:MYVAR
===> Displays: 0
$env:MYVAR = 2
echo $env:MYVAR
===> Displays: 2
echo "##vso[task.setvariable variable=env:MYVAR;]3"
exit
My plan is to have another stage for the opposite condition, maybe you would see a better approach or guidance on what I am missing ? Thanks !!!
*********** Update 1
I used environment variables as I understood it was the way to access them from script, if there is another way, I'd be glad to know !
I used Stages because I need to use an Environment and they seem to need their own stage and deployment block otherwise the environment is tested before the condition is evaluated.
Here is my latest test:
variables:
- name: MYVAR
value: '0'
stages:
- stage: SingleStage
jobs:
- job: DisplayBefore
steps:
- checkout: none
- task: CmdLine#2
inputs:
script: |
echo $(MYVAR)
- job: CallScript
dependsOn: DisplayBefore
steps:
- checkout: none
- task: PowerShell#2
name: SetVarname
inputs:
targetType: filePath
filePath: '..\compareFileContent.ps1'
- deployment: DeploymentJob
dependsOn: DisplayInSameStage
condition: eq(variables.MYVAR, '1')
environment: 'ATest'
strategy:
runOnce:
deploy:
steps:
- checkout: none
- task: CmdLine#2
inputs:
script: |
echo EnvName Finally: $(EnvName)
This YAML doesn't do much as it request the approval from the environment at the start.
My scenario ask that this approval is triggered only based on a value calculated in the PowerShell script.
Any ideas ? Thanks.
Jobs and stages usually run on different agents so you cannot expect environmental variables from different jobs to keep their values.
If you want to work with globals, you can have steps within one job. Otherwise, to keep some logical job/stage structure, you can just write and read values to/from an external source. This depends on your scenario.
Related
my goal is to define a variable in the parent script in Azure DevOps for my pipeline, that I can read it within the template script.
I tried this:
stages:
- stage: Download_Suppliers_Artifacts
variables:
buildVersion2: test2
jobs:
- template: /Pipelines/Templates/download-components-hcp5.yml
Within my script I tried to access it but it is empty:
It is not showed here:
- task: Bash#3
inputs:
targetType: 'inline'
script: 'env | sort'
Nor here:
- task: Bash#3
inputs:
targetType: 'inline'
script: '{{ variables.buildVersion2}}'
This is related to the Runtime expression syntax for variables: https://learn.microsoft.com/en-us/azure/devops/pipelines/process/variables?view=azure-devops&tabs=yaml%2Cbatch#runtime-expression-syntax
In your case, you could directly use macro syntax $(var) instead of template expression ${{ variables.var }}
Test sample (Parent script)
# Starter pipeline
# Start with a minimal pipeline that you can customize to build and deploy your code.
# Add steps that build, run tests, deploy, and more:
# https://aka.ms/yaml
trigger:
- none
pool:
vmImage: windows-latest
stages:
- stage: test
variables:
buildVersion2: test2
jobs:
- template: download-components-hcp5.yml
- job:
steps:
- script: echo $(buildVersion2)
Template:
# download-components-hcp5.yml
jobs:
- job: job1
steps:
- script: env | sort
- powershell: echo $(buildVersion2)
If I do the same thing but don't use a template so passing variables between 2 stages then it works and I can echo "$(varFromStageA)" but when I implement the same using a template the variable is blank.
azure-pipeline.yaml
stages:
- stage: A
jobs:
- job: JA
steps:
- task: AzureKeyVault#1
displayName: 'Get Secret'
inputs:
azureSubscription: "***********"
KeyVaultName: "*****"
SecretsFilter: '*'
RunAsPreJob: true
- script: |
echo "This is job Foo."
echo "##vso[task.setvariable variable=doThing;isOutput=true]$(MySecret)"
name: DetermineResult
- script: echo $(DetermineResult.doThing)
name: echovar
my-pipeline-template.yaml
parameters:
#source
sourceAccountEndpoint: '******'
sourceDatabaseName: '*****'
sourceAccount: '****'
sourceEnvironment: '******'
- stage: DownloadScript
displayName: migrata data
dependsOn: A
jobs:
- job: Pull
variables:
varFromStageA: $[stageDependencies.A.DetermineResult.outputs['DetermineResult.doThing'] ]
pool:
vmImage: windows-latest
displayName: migrate
steps:
- checkout: self
- task: Powershell#2
inputs:
targetType: 'inline'
script: |
echo "$(varFromStageA)"
Refer to this doc: Jobs can access output variables from previous stages
The format of referencing variables between stages is as follows:
stageDependencies.stageName.jobName.outputs['stepName.variableName']
From your YAML Sample, stageName: A, JobName : JA.
So you need to define the variable in template with the following format:
varFromStageA: $[stageDependencies.A.JA.outputs['DetermineResult.doThing']
Then the value of the variable can be passed.
Template example:
stages:
- stage: DownloadScript
displayName: migrata data
dependsOn: A
jobs:
- job: Pull
variables:
varFromStageA: $[stageDependencies.A.JA.outputs['DetermineResult.doThing'] ]
pool:
vmImage: windows-latest
displayName: migrate
steps:
- checkout: self
- task: Powershell#2
inputs:
targetType: 'inline'
script: |
echo "$(varFromStageA)"
Have a look at this:
https://learn.microsoft.com/en-us/azure/devops/pipelines/process/variables?view=azure-devops&tabs=yaml%2Cbatch
There's some documented examples - it might help. However, why not just set a global variable in the yaml file (i.e. above the stages)? It's not clear from your question why you can't do this.
If you can, something like this:
variables:
var1: newitem
stages:
- stage: one
etc
If that won't work, try these approaches:
https://arunksingh16.medium.com/azure-devops-share-variable-across-stages-9bca85abfe8a
https://stefanstranger.github.io/2019/06/26/PassingVariablesfromStagetoStage/
Problem Description
I was having some problems trying to use variables created in one stage in another stage and managed to find various articles old and new describing how this can be done. The more recent articles/posts identifying the new syntax
$[stageDependencies.{stageName}.{jobName}.outputs['{stepName}.{variableName}']
Used like this:
variables:
myVariable: $[stagedependencies.CreateStageVarStage.CreateStageVarJob.outputs['SetValueStep.VariableFromFirstStage']]
This works great until you needed to use job templates.
None of the samples I found online covered the situation of templates. They just demonstrated how multiple stages in the same yaml file could obtain the value.
The syntax depends on being able to put the expression into a variable. Unfortunately, when you use a template for a job, it's not possible to declare variables and passing it as a parameter results in it being unevaluated.
- stage: UseJobTemplateStage
displayName: 'Use Job Template Stage'
dependsOn: CreateStageVarStage
jobs:
- template: templates/job-showstagevars.yml
parameters:
ValueToOutput: $[ stagedependencies.CreateStageVarStage.CreateStageVarJob.outputs['SetValueStep.VariableFromFirstStage'] ]
In this snippet, it comes through as-is. The value does not get substituted in.
Theoretically, you could set your job to have the expression present in the variables block but that sort of hard-coding undermines one of the main benefits of templates.
Related Articles
Share variables across stages in Azure DevOps Pipelines
Azure DevOps Release Notes - sprint 168
I know the question asks about template jobs, but for future reference I want to describe how it can be achieved with template stages as well.
It is done with a variable workaround as in the accepted answer, and with a reference to stagedependencies when no dependsOn exists. (Templates don't allow dependsOn). Somehow, this still works.
Example YAML using stage template (I have modified the code from the accepted answer):
stages:
- stage: CreateStageVarStage
displayName: 'Create StageVar Stage'
jobs:
- job: CreateStageVarJob
displayName: 'Create StageVar Job'
timeoutInMinutes: 5
pool:
name: 'Azure Pipelines'
vmImage: 'windows-2019'
steps:
- checkout: none
- pwsh: |
[string]$message = 'This is the value from the first stage'
Write-Host "Setting output variable 'VariableFromFirstStage' to '$message'"
Write-Output "##vso[task.setvariable variable=VariableFromFirstStage;isOutput=$true]$message"
name: SetValueStep
# stage template cannot use dependsOn, but is still allowed to refer to stagedependencies...
- template: templates/stage-showstagevars.yml
parameters:
ValueToOutput: $[ stagedependencies.CreateStageVarStage.CreateStageVarJob.outputs['SetValueStep.VariableFromFirstStage'] ]
Stage template:
parameters:
- name: ValueToOutput
type: string
stages:
- stage: ShowStageVarStage
variables:
- name: LocalVarOfValueToOutputParam
value: ${{ parameters.ValueToOutput }}
jobs:
- job: ShowStageVarJob
displayName: 'Show stage var'
pool:
name: 'Azure Pipelines'
vmImage: 'windows-2019'
steps:
- checkout: none
- pwsh: |
Write-Host "ValueToOutput parameter=${{ parameters.ValueToOutput }}"
Write-Host "LocalVarOfValueToOutputParam (pre-processor syntax)=${{ variables.LocalVarOfValueToOutputParam }}"
Write-Host "LocalVarOfValueToOutputParam (macro syntax)=$(LocalVarOfValueToOutputParam)"
displayName: 'Show StageVariable'
Solution
The answer isn't actually far away. The original expression just need to be passed through a variable in the template job. Basically, set a variable to be the value of the parameter and use the macro syntax to evaluate the variable.
parameters:
- name: ValueToOutput
type: string
...
variables:
- name: LocalVarOfValueToOutputParam
value: ${{ parameters.ValueToOutput }}
Using the macro syntax of $(LocalVarOfValueToOutputParam) will result in the value making its way into the template job correctly.
Example
If we have a yaml file for the build definition:
stages:
- stage: CreateStageVarStage
displayName: 'Create StageVar Stage'
jobs:
- job: CreateStageVarJob
displayName: 'Create StageVar Job'
timeoutInMinutes: 5
pool:
name: 'Azure Pipelines'
vmImage: 'windows-2019'
steps:
- checkout: none
- pwsh: |
[string]$message = 'This is the value from the first stage'
Write-Host "Setting output variable 'VariableFromFirstStage' to '$message'"
Write-Output "##vso[task.setvariable variable=VariableFromFirstStage;isOutput=$true]$message"
name: SetValueStep
- stage: UseJobTemplateStage
displayName: 'Use Job Template Stage'
dependsOn: CreateStageVarStage
jobs:
- template: templates/job-showstagevars.yml
parameters:
ValueToOutput: $[ stagedependencies.CreateStageVarStage.CreateStageVarJob.outputs['SetValueStep.VariableFromFirstStage'] ]
That uses the job template templates/job-showstagevars.yml
parameters:
- name: ValueToOutput
type: string
jobs:
- job: ShowStageVarJob
displayName: 'Show stage var'
timeoutInMinutes: 5
pool:
name: 'Azure Pipelines'
vmImage: 'windows-2019'
variables:
- name: LocalVarOfValueToOutputParam
value: ${{ parameters.ValueToOutput }}
steps:
- checkout: none
- pwsh: |
Write-Host "ValueToOutput parameter=${{ parameters.ValueToOutput }}"
Write-Host "LocalVarOfValueToOutputParam (pre-processor syntax)=${{ variables.LocalVarOfValueToOutputParam }}"
Write-Host "LocalVarOfValueToOutputParam (macro syntax)=$(LocalVarOfValueToOutputParam)"
displayName: 'Show StageVariable'
What we get in our output of the second stage is:
ValueToOutput parameter=$[ stagedependencies.CreateStageVarStage.CreateStageVarJob.outputs['SetValueStep.VariableFromFirstStage'] ]
LocalVarOfValueToOutputParam (pre-processor syntax)=$[ stagedependencies.CreateStageVarStage.CreateStageVarJob.outputs['SetValueStep.VariableFromFirstStage'] ]
LocalVarOfValueToOutputParam (macro syntax)=This is the value from the first stage
we are trying to create a DevOps pipeline where we generate a string in the first stage and we would like to store into a variable which can be used in subsequent stages, or in subsequent tasks within the same stage, is there a way to do this? For clarity the string is generated by querying an external api which returns a string value.
Hope that makes sense 🥳
Thanks in advance
Yes, you shoudl use logging command and mark is as output. Here you have an example
## azure-pipelines.yml
stages:
- stage: one
jobs:
- job: A
steps:
- task: Bash#3
inputs:
filePath: 'script-a.sh'
name: setvar
- bash: |
echo "##vso[task.setvariable variable=skipsubsequent;isOutput=true]true"
name: skipstep
- stage: two
jobs:
- job: B
variables:
- name: StageSauce
value: $[ stageDependencies.one.A.outputs['setvar.sauce'] ]
- name: skipMe
value: $[ stageDependencies.one.A.outputs['skipstep.skipsubsequent'] ]
steps:
- task: Bash#3
inputs:
filePath: 'script-b.sh'
name: fileversion
env:
StageSauce: $(StageSauce) # predefined in variables section
skipMe: $(skipMe) # predefined in variables section
- task: Bash#3
inputs:
targetType: 'inline'
script: |
echo 'Hello inline version'
echo $(skipMe)
echo $(StageSauce)
In Azure Devops multistage YAML pipeline we got multiple environments.
In stages to run normally we do a build and deploy only in QA, so we need to deselect each stage manually. By default all stages are selected is it possible to have exact opposite, where all stages are deselected by default???
trigger: none
pr: none
stages:
- stage: 'Build'
jobs:
- deployment: 'Build'
pool:
name: Default
# testing
environment: INT
strategy:
runOnce:
deploy:
steps:
- checkout: none
- powershell: |
echo "Hello Testing"
Start-Sleep -Seconds 10
- stage: 'Sandbox'
jobs:
- job: 'Sandbox'
pool:
name: Default
steps:
- checkout: none
# testing
- powershell: |
echo "Hello Testing"
- stage: 'Test'
jobs:
- job: 'DEV'
pool:
name: Default
steps:
- checkout: none
- powershell: |
echo "Hello Testing"
- stage: 'QA'
dependsOn: ['Test','Test1','Test2']
jobs:
- job: 'QA'
pool:
name: Default
steps:
- checkout: none
# Testing
- powershell: |
echo "Hello Testing"
I am afraid that there is no UI (like stage to run) method that can meet your needs.
You could try to add parameters to your Yaml Sample.
Here is an example:
trigger: none
pr: none
parameters:
- name: stageTest
displayName: Run Stage test
type: boolean
default: false
- name: stageBuild
displayName: Run Stage build
type: boolean
default: false
stages:
- ${{ if eq(parameters.stageBuild, true) }}:
- stage: 'Build'
jobs:
- deployment: 'Build'
pool:
name: Default
environment: INT
strategy:
runOnce:
deploy:
steps:
- checkout: none
- powershell: |
echo "Hello Testing"
Start-Sleep -Seconds 10
- ${{ if eq(parameters.stageTest, true) }}:
- stage: Test
dependsOn: []
jobs:
- job: B1
steps:
- script: echo "B1"
The parameters are used to determine whether to run these stages. You could add expressions before the stage to check if the parameter value could meet expression.
The default value is false. This means that the stage will not run by default.
Here is the result:
You can select the stage you need to run by clicking the selection box.
Update
Workaround has some limitations. When the select stage has depenencies, you need to select all dependent stages to run.
For example:
- stage: 'QA'
dependsOn: ['Test','Test1','Test2']
On the other hand, I have created a suggestion ticket to report this feature request. Here is the suggestion ticket link: Pipeline Deselect Stages By Default You could vote and add comment in it .
I've used this solution to build a nuget-package, and:
always push packages from master
conditionally push packages from other branches
Using GitVersion ensures that the packages from other branches get prerelease version numbers, e.g. 2.2.12-my-branch-name.3 or 2.2.12-PullRequest7803.4. The main branch simply gets 2.2.12, so the master branch is recognized as a "regular" version.
The reason I'm repeating the answer above, is that I chose to make the stage conditional instead of using an if:
trigger:
- master
parameters:
- name: pushPackage
displayName: Push the NuGet package
type: boolean
default: false
stages:
- stage: Build
jobs:
- job: DoBuild
steps:
- script: echo "I'm building a NuGet package (versioned with GitVersion)"
- stage: Push
condition: and(succeeded('build'), or(eq('${{ parameters.pushPackage }}', true), eq(variables['build.sourceBranch'], 'refs/heads/master')))
jobs:
- job: DoPush
steps:
- script: echo "I'm pushing the NuGet package"
Like the other answer, this results in a dialog:
But what's different from the (equally valid) solution with '${{ if }}', is that the stage is always shown (even if it's skipped):