How does one overwrite a pipeline variable or how does one create a pipeline variable from a job?
I'm running a prepare job where I extract the current git tag into a variable that I need in following jobs so I decided to create a pipeline variable and overwrite its value in the first job:
variables:
GIT_TAG: v0.0.1
jobs:
- job: job1
pool:
vmImage: 'ubuntu-16.04'
steps:
- powershell: |
Write-Host "##vso[task.setvariable variable=GIT_TAG]$(git describe --tags --always)"
But, in the next job GIT_TAG has the initial value of v0.0.1.
By default, if you overwrite a variable the value is available only to his job, not to the sequence jobs.
Passing variables between jobs in the same stage is a bit more complex, as it requires working with output variables.
Similarly to the example above, to pass the FOO variable:
Make sure you give a name to the job, for example job: firstjob
Likewise, make sure you give a name to the step as well, for example: name: mystep
Set the variable with the same command as before, but adding ;isOutput=true, like: echo "##vso[task.setvariable variable=FOO;isOutput=true]some value"
In the second job, define a variable at the job level, giving it the value $[ dependencies.firstjob.outputs['mystep.FOO'] ] (remember to use single quotes for expressions)
A full example:
jobs:
- job: firstjob
pool:
vmImage: 'Ubuntu-16.04'
steps:
# Sets FOO to "some value", then mark it as output variable
- bash: |
FOO="some value"
echo "##vso[task.setvariable variable=FOO;isOutput=true]$FOO"
name: mystep
# Show output variable in the same job
- bash: |
echo "$(mystep.FOO)"
- job: secondjob
# Need to explicitly mark the dependency
dependsOn: firstjob
variables:
# Define the variable FOO from the previous job
# Note the use of single quotes!
FOO: $[ dependencies.firstjob.outputs['mystep.FOO'] ]
pool:
vmImage: 'Ubuntu-16.04'
steps:
# The variable is now available for expansion within the job
- bash: |
echo "$(FOO)"
# To send the variable to the script as environmental variable, it needs to be set in the env dictionary
- bash: |
echo "$FOO"
env:
FOO: $(FOO)
More info you can find here.
Related
I am new to azure pipelines and am currently experimenting with the passing variables to the later jobs. Here is the current snippet whereby I am trying to extract the project version from the pom file using maven help: evaluate plugin. The pomVersion variable is populated but is not available in the same step or later steps in the second job via the projectVersionDynamic variable.
stages:
- stage: FirstStage
jobs:
- job: FirstJob
pool:
vmImage: 'ubuntu-latest'
steps:
- task: Bash#3
inputs:
targetType: 'inline'
script: |
pomVersion=`mvn help:evaluate -Dexpression=project.version -q -DforceStdout`
echo $pomVersion ##Prints semantic version 2.27.0-SNAPSHOT
echo '##vso[task.setvariable variable=projectVersionDynamic;isOutput=true]$pomVersion'
echo '##vso[task.setvariable variable=projectVersionStatic;isOutput=true]2.27.0'
echo $(Task1.projectVersionDynamic) ##Error message
echo $projectVersionDynamic ##Is empty
name: Task1
displayName: Task1 in JobOne of FirstStage
- job: SecondJob
dependsOn: FirstJob
variables:
DynVar: $[ dependencies.FirstJob.outputs['Task1.projectVersionDynamic'] ]
StaVar: $[ dependencies.FirstJob.outputs['Task1.projectVersionStatic'] ]
pool:
vmImage: 'ubuntu-latest'
steps:
- task: Bash#3
inputs:
targetType: 'inline'
script: |
echo 'SecondJob'
echo $(DynVar) ##Is empty
echo $DynVar ##Is empty
echo $(StaVar) ##Prints 2.27.0
echo $StaVar ##Is empty
displayName: Task in JobTwo of FirstStage
Observation: projectVersionDynamic value does not get populated and is not available in the same task or subsequent tasks within or in later jobs / stages. However, the static variable gets populated in projectVersionStatic without any issues.
Is it possible to set dynamic values for user-defined variables in azure pipelines or is it that I am doing something wrong? I see an example here under the Stages section where it seems to be working.
Variables in Azure Pipelines can be really tricky sometimes. The documentation isn't always crystal-clear on how it's supposed to work. Looking at your example, a cpuple of observations:
echo '##vso[task.setvariable variable=projectVersionDynamic;isOutput=true]$pomVersion' - your single quotes need to be double quotes to expand the value of $pomVersion into the echo statement
(and this is where things get fuzzy): The purpose of task.setvariable is to communicate values to other tasks or jobs. From the documentation: "This doesn't update the environment variables, but it does make the new variable available to downstream steps within the same job."
$variable syntax won't work because the task.setVariable doesn't inject into the running environment - rather, it's a signal to the pipeline to capture the output and store it away for later use.
$(variable) syntax won't work because it's expanded just before the job starts, so it's too late to capture here.
If you use my suggestion in point 1) about double-quoting the task.setVariable, you should see the value available in the second job.
For anybody having a use case where variables defined in one azure pipeline stage needs to be used in another stage, this is how it can be done:
Following snipett of azure pipeline that defines a variable and make it available to be used in a job in the same stage and in a job in another stage:
stages:
- stage: stage1
jobs:
- job: job1
steps:
- task: "AzureCLI#2"
name: step1
inputs:
inlineScript: |
my_variable=a-value
echo "##vso[task.setvariable variable=var_name;isOutput=true]$my_variable"
- job: job2
variables:
stage1_var: $[ dependencies.job1.outputs['step1.var_name'] ]
steps:
- bash: |
echo "print variable value"
echo $(stage1_var)
- stage: stage2
jobs:
- job: job1
variables:
stage2_var: $[ stageDependencies.stage1.job1.outputs['step1.var_name'] ]
steps:
- bash: |
echo "print variable value"
echo $(stage2_var)
I want to perform some actions (publish) on an array of string which are passed as parameter.
The thing is, this parameter is dynamic :
# pipeline.yml
- job: MyJob
pool:
[...]
steps:
- pwsh: |
$affected = ['app-one', 'app-two'] # Here I hardcoded the array but in my real code this is set dynamically
Write-Host "##vso[task.setvariable variable=affected;isOutput=true]$affected"
name: setAffected
displayName: 'Settings affected'
- template: ./build.yml
parameters:
affected: $[ dependencies.Affected.outputs['setAffected.affected'] ] # Here I pass the array of string to the template
# build.yml
parameters:
affected: ''
jobs:
- job: Build
condition: succeeded('Affected')
dependsOn: Affected
pool:
[...]
variables:
affected: ${{ parameters.affected }}
steps:
- ${{each app in $(affected)}}:
- pwsh: |
Write-Host "${{app}}"
- ${{each app in parameters.affected}}:
- pwsh: |
Write-Host "${{app}}"
Neither of ${{each app in $(affected)}} or ${{each app in parameters.affected}} work...
How can I manage to execute some actions on each of my array item?
Thanks
Within a template expression, you have access to the parameters
context that contains the values of parameters passed in.
Additionally, you have access to the variables context that contains
all the variables specified in the YAML file plus the system
variables. Importantly, it doesn't have runtime variables such
as those stored on the pipeline or given when you start a run.
Template expansion happens very early in the run, so those variables
aren't available.
https://learn.microsoft.com/en-us/azure/devops/pipelines/process/templates?view=azure-devops
I am trying to build an automatic Environment Approval with Invoke Azure Function. I need to pass a variable that is created in a previous build stage. I have added the variable to the function body, but the variable is not evaluated:
Task that creates the output variable (in previous stage)
- task: CmdLine#2
inputs:
script: |
# Debug output (value visible in logs)
pulumi stack output resourceGroupName
echo "##vso[task.setvariable variable=resourceGroupName;isoutput=true]$(pulumi stack output resourceGroupName)"
workingDirectory: 'infra'
Body for the Azure function:
{ "ResourceGroup": "$(resourceGroupName)" }
Log:
2020-03-26T15:57:01.2590351Z POST https://azure-function-url
Request body: {
"ResourceGroup": "$(resourceGroupName)"
}
Added the variable to the function body, but the variable is not
evaluated
This is a expected action. Here what you are doing is share variable across stages, which does not supported directly until now.
Output variable just used to share values between steps instead of stages.
Work around:
If you want to use the generated variable in next stage, a workaround you can consider to use is writing the variable to a file, leveraging pipeline artifacts.
Sample steps:
Here I will pass one variable which name is resourceGroupName to next stage.
1) Create a folder which will contain the variables you want to pass
mkdir -p $(Pipeline.Workspace)/variables
2) Write the contents of the variable to file StageUsed:
echo "$resourceGroupName" > $(Pipeline.Workspace)/variables/StageUsed
3) In next stage, add one job before your InvokeAzureFunction job. Download the variables pipeline artifact that previous stage published.
4) Transfer each file into a variable:
resourceGroupName=$(cat $(Pipeline.Workspace)/variables/StageUsed)
5) Make the variable exposed in the current job, and set it reference name as Out:
echo "##vso[task.setvariable variable=resourceGroupName;isoutput=true]$resourceGroupName"
6) Now, you can access the variable in your InvokeAzureFunction job by calling dependencies.secondjob.outputs['output.resourceGroupName']
Sample Script:
trigger:
- master
pool:
vmImage: 'ubuntu-latest'
stages:
- stage: FirstStage
jobs:
- job: firstjob
pool:
vmImage: 'Ubuntu-16.04'
steps:
- bash: |
resourceGroupName="value"
mkdir -p $(Pipeline.Workspace)/variables
echo "$resourceGroupName" > $(Pipeline.Workspace)/variables/resourceGroupName
- publish: $(Pipeline.Workspace)/variables
artifact: variables
- stage: SecondStage
jobs:
- job: secondjob
pool:
vmImage: 'Ubuntu-16.04'
steps:
- download: current
artifact: variables
- bash: |
resourceGroupName=$(cat $(Pipeline.Workspace)/variables/resourceGroupName)
echo "##vso[task.setvariable variable=resourceGroupName;isoutput=true]$resourceGroupName"
name: output
- bash: |
echo "$(output.resourceGroupName)"
- job: ServerJob
dependsOn: secondjob
pool: server
variables:
resourceGroupName: $[dependencies.secondjob.outputs['output.resourceGroupName']]
steps:
- task: AzureFunction#1
inputs:
function:
method: 'POST'
body: '$(sdf)'
waitForCompletion: 'false'
When running the following task in a pipeline
- task: AzureRmWebAppDeployment#3
displayName: 'Azure App Service Deploy: Web App'
inputs:
azureSubscription: ${{ parameters.azureSubscription }}
WebAppName: $[dependencies.preDeploy.outputs['webAppName']]
Package: '$(Pipeline.Workspace)/**/PackageTmp'
WebAppUri: WebAppUrl
UseWebDeploy: true
AdditionalArguments: '-useChecksum'
RenameFilesFlag: true
enableXmlVariableSubstitution: true
it complains about the variable I am assigning to the WebAppName
##[error]Error: Failed to get resource ID for resource type 'Microsoft.Web/Sites' and resource name '$[dependencies.preDeploy.outputs['webAppName']]'. Error: {"error":{"code":"InvalidFilterInQueryString","message":"Invalid $filter 'resourceType EQ 'Microsoft.Web/Sites' AND name EQ '$[dependencies.preDeploy.outputs['webAppName']]'' specified in the query string."}} (CODE: 400)
Why is the variable not being replaced with its actual value?
Your variable needs to be addressed by its reference name, which is usually the step name the variable came from.
If you want to make a variable available to future jobs, you must mark it as an output variable using isOutput=true. Then you can map it into future jobs using $[] syntax and including the step name which set the variable.
More reading about Multi-job Variables
See myVarFromJobA: $[ dependencies.A.outputs['setvarStep.myOutputVar'] ]
in the below example.
jobs:
# Set an output variable from job A
- job: A
pool:
vmImage: 'vs2017-win2016'
steps:
- powershell: echo "##vso[task.setvariable variable=myOutputVar;isOutput=true]this is the value"
name: setvarStep
- script: echo $(setvarStep.myOutputVar)
name: echovar
# Map the variable into job B
- job: B
dependsOn: A
pool:
vmImage: 'ubuntu-16.04'
variables:
myVarFromJobA: $[ dependencies.A.outputs['setvarStep.myOutputVar'] ] # map in the variable
# remember, expressions require single quotes
steps:
- script: echo $(myVarFromJobA)
name: echovar
Let's say I'm starting with this configuration that works:
jobs:
- job: ARelease
pool:
vmImage: 'Ubuntu-16.04'
steps:
- script: |
echo "##vso[task.setvariable variable=ReleaseVar;isOutput=true]1"
name: JobResult
- job: C
pool:
vmImage: 'Ubuntu-16.04'
dependsOn: ARelease
variables:
AVar: $[ dependencies.A.outputs['JobResult.ReleaseVar'] ]
steps:
- script: |
echo $(AVar)
As expected job C outputs 1.
Now let's say that I have to add a new job ADebug, which is almost identical to ARelease, so I use a strategy:
jobs:
- job: A
strategy:
matrix:
Release:
BUILD_TYPE: Release
Debug:
BUILD_TYPE: Debug
pool:
vmImage: 'Ubuntu-16.04'
steps:
- script: |
echo "##vso[task.setvariable variable=$(BUILD_TYPE)Var;isOutput=true]1"
name: JobResult
- job: C
pool:
vmImage: 'Ubuntu-16.04'
dependsOn: A
variables:
AReleaseVar: $[ dependencies.A.outputs['JobResult.ReleaseVar'] ]
ADebugVar: $[ dependencies.A.outputs['JobResult.DebugVar'] ]
steps:
- script: |
echo $(AReleaseVar)
echo $(ADebugVar)
I would expect that everything works and that I can see the outputs.. but the output is empty.
Queuing the job with diagnostics it seems that $[ dependencies.A.outputs['JobResult.ReleaseVar'] ] and $[ dependencies.A.outputs['JobResult.DebugVar'] ] evaluate to Null.
I've tried different variations to access those variables, but it always evaluates to Null.
Any idea what is the correct way?
https://learn.microsoft.com/en-us/azure/devops/pipelines/process/variables?view=azure-devops&tabs=yaml%2Cbatch#set-a-multi-job-output-variable
If you're setting a variable from a matrix or slice, then to reference
the variable, you have to include the name of the job as well as the
step when you access it from a downstream job.
The format is as follows: dependencies.{job}.outputs['{matrix/slice key}.{step name}.{variable name}']
In your scenario, you have Job A run with a matrix strategy (Release, Debug), and you respectively set the variable names to be ReleaseVar and DebugVar. The way to accurately access these variables is:
dependencies.A.outputs['Release.JobResult.ReleaseVar']
dependencies.A.outputs['Debug.JobResult.DebugVar']
On a side note, perhaps just use the same variable name since it you are already able to distinguish between the values based on the matrix name.