Azure DevOps Pipeline define variable in deployment and reuse in subsequent job - azure-devops

I'm using an Azure DevOps pipeline to deploy my code and now I'm in need of passing a variable value from a deployment job to a subsequent job that depends on it. I've read up on this example but it does not seem to work at all.
What I'm trying to do is run an Azure ARM Deployment that provisions a Key Vault. The name of the key vault is outputted from the ARM deployment job and I'm then trying to pass that name to another job which needs to add specific secrets. Access control is taken care of, but I still need to pass the name.
I've boiled the problem down to the basics of passing a variable from a deployment to a job. Here is my complete test pipeline (almost entirely copied from here):
trigger: none
stages:
- stage: X
jobs:
- deployment: A
pool:
vmImage: "ubuntu-16.04"
environment: test
strategy:
runOnce:
deploy:
steps:
- script: echo "##vso[task.setvariable variable=myOutputVar;isOutput=true]this is the deployment variable value"
name: setvarStep
- script: echo $(setvarStep.myOutputVar)
name: echovar
- job: B
dependsOn: A
pool:
vmImage: "ubuntu-16.04"
variables:
myVarFromDeploymentJob: $[ dependencies.A.outputs['deploy.setvarStep.myOutputVar'] ]
steps:
- script: "echo $(myVarFromDeploymentJob)"
name: echovar
Once I run this the echoed value is blank in job B, but defined in deployment A. Why is this? And is there a way to dum everything in dependencies.A.outputs so that I can see what I have to work with?
How can I pass a variable from a runOnce deployment job to a regular job?

I've solved it. The problem is that the documentation here specifies this schema for fetching the variable for a runOnce deployment:
$[dependencies.<job-name>.outputs['<lifecycle-hookname>.<step-name>.<variable-name>']]
This is in fact WRONG. The <lifecycle-hookname> parameter should be replaced with the name of the deployment like this:
$[dependencies.<job-name>.outputs['<job-name>.<step-name>.<variable-name>']]
The example from this documentation (scroll down a bit) is correct.
A full example pipeline that I've tested and works:
trigger: none
stages:
- stage: X
jobs:
- deployment: A # This name is important
pool:
vmImage: 'ubuntu-16.04'
environment: staging
strategy:
runOnce:
deploy:
steps:
- script: echo "##vso[task.setvariable variable=myOutputVar;isOutput=true]this is the deployment variable value"
name: setvarStep # This name is also important
- script: echo $(setvarStep.myOutputVar)
name: echovar
- job: B
dependsOn: A
pool:
vmImage: 'ubuntu-16.04'
variables:
myVarFromDeploymentJob: $[ dependencies.A.outputs['A.setvarStep.myOutputVar'] ]
steps:
- script: "echo $(myVarFromDeploymentJob)"
name: echovar

Related

Why we need AZure Devops Yaml template extends?

My issue
Yaml Devops propose the extends bloc. This is the kind of sample one use:
# main.yaml
trigger: none
extends:
template: ./extends-pipeline1.yaml
And:
# extends-pipeline1.yaml
stages:
- stage: Build
jobs:
- job:
steps:
- script: echo Build solution
- stage: Deploy
jobs:
- job:
steps:
- script: echo Deploy solution
Works fine, but this works as well without extends:
# main.yaml
trigger: none
stages:
- template: extends-pipeline1.yaml
What I did
I read the release note:
Release Note
My question
What is the benefit to use extends?
Thanks
What is the benefit to use extends?
There are two benefits of this feature.
First, 'extends' is more free than 'stages'.
When you use stages, then the pipeline will expect stages in the template, you must write the stages part in the template, otherwise the validation step will not be passed.
For example, when you use stages, your pipeline must be like this:
azure-pipelines.yml
trigger:
- none
pool:
vmImage: ubuntu-latest
stages:
- template: extends-pipeline1.yaml
extends-pipeline1.yaml
stages: #You must have this section in the template.
- stage: Build
jobs:
- job:
steps:
- script: echo Build solution
- stage: Deploy
jobs:
- job:
steps:
- script: echo Deploy solution
But if you use extends, your pipeline could be like this instead of must writing stages section in template:
azure-pipelines.yml
trigger:
- none
pool:
vmImage: ubuntu-latest
extends:
template: extends-pipeline1.yaml
extends-pipeline1.yaml
jobs:
- job:
steps:
- script: echo Build solution
Second, 'extends' has a feature named Required template.(Daniel had already mentioned this.)
If you set the 'required template' of the service connection, then the service connection will only been able to use in specific YAML template file which been 'extends'.
No matter directly use the service connection via main YAML file or use the service connection in the YAML file referred via 'stages'-template will both make the pipeline failed.
This feature can help you increase the security and this feature is exclusive to the 'extends' feature.

Set environment variables into azure devops pipelines step from template

Given I have following step in Azure Devops pipeline:
- task: DotNetCoreCLI#2
displayName: Run tests
inputs:
command: "test"
env:
"ASPNETCORE_ENVIRONMENT": "test"
"Case_sensitive_environment_variable_key": "This is the value"
I'd like to move this step into the template and enable users of that template to pass in env key values making sure it keeps case sensitivity of keys and values.
How can I do that?
According to your description and to transfer the parameter as the environmental varailbles on the level of a task, you may try Passing parameters into the template and using the parameter type of object.
Kindly take the following main YAML pipeline and template.yml for a reference.
Main
trigger: none
parameters:
- name: pEnvironment
type: object
default:
ASPNETCORE_ENVIRONMENT: "TEST 1"
Case_sensitive_environment_variable_key: "This is the value 1"
pool:
vmImage: windows-latest
stages:
- stage:
jobs:
- template: Templates/template1.yml
parameters:
transferedEnv: ${{ parameters.pEnvironment }}
Template
parameters:
- name: transferedEnv
type: object
jobs:
- job:
steps:
- task: DotNetCoreCLI#2
displayName: Run tests
inputs:
command: "test"
env:
${{ parameters.transferedEnv }}
When we run the main pipeline, we can input as well the values of the env to transferred into the template.

How to create Azure Pipeline Template to run a jobList all on the same agent?

I am trying to make a pipeline template that takes a JobList a parameter and runs all the jobs, while ensuring that they run on the same agent every time. Basically the approach I've been taking is to try to adapt this answer into a genericized template format.
This is what I have so far, and I've tried a lot of slight tweaks of this with nothing passing the Validate test on the pipeline that calls it.
parameters:
- name: jobsToRun
type: jobList
- name: pool
type: string
default: Default
- name: demands
type: object
default: []
jobs:
- job:
steps:
- script: echo "##vso[task.setvariable variable=agentName;isOutput=true;]$(Agent.Name)"
pool:
name: ${{ parameters.pool }}
demands:
- ${{ each demand in parameters.demands }}:
${{ demand }}
- ${{ each j in parameters.jobsToRun }}:
${{ each pair in j }}:
${{ pair.key }} : ${{ pair.value }}
pool:
name: Default
demands:
- Agent.Name -equals $(agentName)
What am I doing wrong here? It seems like it should be possible if that answer I reference is correct, but it seems like I'm just a bit off.
Name missing on the job.., example below.
- job: 'test-Name'
Steps need a associated job and pool to run is declared inside
jobsToRun:
- job: sample_job1
displayName: "sample_job1"
pool:
name: "your_PoolName"
steps:
- script: |
echo "Hi"
On this bottom pool declaration...
pool:
name: Default
demands:
- Agent.Name -equals $(agentName)
i am not sure but i have tried this many times but i think this can't be included separate to the job since each individual job is passed as parameter. pool definition needs to be inside the job or inside the job template if you are using templates..
example:
jobsToRun:
- job: output_message_job1
displayName: "in pipe Output Message Job"
pool:
name: "your_PoolName"
steps:
- script: |
echo "Hi"

Why can't I use a variable to define the environment property in the Azure Pipeline YAML config file?

I'm trying to create a deploy pipeline YAML template for all environments/stages. I've set up the Environments on Azure DevOps so that I can add checks and approvals on the Test and Prod environments before they get deployed. I've set up a library group for each stage and each one of them has a variable called 'env' which defines the current stage running in the pipeline. For some reason, the environment property under the deployment job (see code snippet below) doesn't read that variable.
Has anyone faced this issue before, or is there a reason why the variable won't be read for that specific property?
Note: I've tested the variables and they do work, for example, the stage property outputs as 'deploy-dev/test/prod' (depending on the environment)
- stage: deploy-$(env)
jobs:
- deployment: DeployWeb
displayName: deploy Web App
pool:
vmImage: 'Ubuntu-latest'
# creates an environment if it doesn't exist
environment: 'smarthotel-$(env)'
strategy:
runOnce:
deploy:
steps:
- script: echo Hello world
You can't do this because it have to be know at compilation phase.
But you can try this (lets name this file deploy.yml):
parameters:
- name: env
type: string
default: 'dev'
stages:
- stage: deploy-${{ parameters.env }}
jobs:
- deployment: DeployWeb
displayName: deploy Web App
pool:
vmImage: 'Ubuntu-latest'
# creates an environment if it doesn't exist
environment: 'smarthotel-${{ parameters.env }}'
strategy:
runOnce:
deploy:
steps:
- script: echo Hello world
and then you need to run as follows (in build.yml file):
stages:
- template: deploy.yml
parameters:
env: dev
- template: deploy.yml
parameters:
env: qa
- template: deploy.yml
parameters:
env: prod

Using pipeline variables across stages with template jobs

Problem Description
I was having some problems trying to use variables created in one stage in another stage and managed to find various articles old and new describing how this can be done. The more recent articles/posts identifying the new syntax
$[stageDependencies.{stageName}.{jobName}.outputs['{stepName}.{variableName}']
Used like this:
variables:
myVariable: $[stagedependencies.CreateStageVarStage.CreateStageVarJob.outputs['SetValueStep.VariableFromFirstStage']]
This works great until you needed to use job templates.
None of the samples I found online covered the situation of templates. They just demonstrated how multiple stages in the same yaml file could obtain the value.
The syntax depends on being able to put the expression into a variable. Unfortunately, when you use a template for a job, it's not possible to declare variables and passing it as a parameter results in it being unevaluated.
- stage: UseJobTemplateStage
displayName: 'Use Job Template Stage'
dependsOn: CreateStageVarStage
jobs:
- template: templates/job-showstagevars.yml
parameters:
ValueToOutput: $[ stagedependencies.CreateStageVarStage.CreateStageVarJob.outputs['SetValueStep.VariableFromFirstStage'] ]
In this snippet, it comes through as-is. The value does not get substituted in.
Theoretically, you could set your job to have the expression present in the variables block but that sort of hard-coding undermines one of the main benefits of templates.
Related Articles
Share variables across stages in Azure DevOps Pipelines
Azure DevOps Release Notes - sprint 168
I know the question asks about template jobs, but for future reference I want to describe how it can be achieved with template stages as well.
It is done with a variable workaround as in the accepted answer, and with a reference to stagedependencies when no dependsOn exists. (Templates don't allow dependsOn). Somehow, this still works.
Example YAML using stage template (I have modified the code from the accepted answer):
stages:
- stage: CreateStageVarStage
displayName: 'Create StageVar Stage'
jobs:
- job: CreateStageVarJob
displayName: 'Create StageVar Job'
timeoutInMinutes: 5
pool:
name: 'Azure Pipelines'
vmImage: 'windows-2019'
steps:
- checkout: none
- pwsh: |
[string]$message = 'This is the value from the first stage'
Write-Host "Setting output variable 'VariableFromFirstStage' to '$message'"
Write-Output "##vso[task.setvariable variable=VariableFromFirstStage;isOutput=$true]$message"
name: SetValueStep
# stage template cannot use dependsOn, but is still allowed to refer to stagedependencies...
- template: templates/stage-showstagevars.yml
parameters:
ValueToOutput: $[ stagedependencies.CreateStageVarStage.CreateStageVarJob.outputs['SetValueStep.VariableFromFirstStage'] ]
Stage template:
parameters:
- name: ValueToOutput
type: string
stages:
- stage: ShowStageVarStage
variables:
- name: LocalVarOfValueToOutputParam
value: ${{ parameters.ValueToOutput }}
jobs:
- job: ShowStageVarJob
displayName: 'Show stage var'
pool:
name: 'Azure Pipelines'
vmImage: 'windows-2019'
steps:
- checkout: none
- pwsh: |
Write-Host "ValueToOutput parameter=${{ parameters.ValueToOutput }}"
Write-Host "LocalVarOfValueToOutputParam (pre-processor syntax)=${{ variables.LocalVarOfValueToOutputParam }}"
Write-Host "LocalVarOfValueToOutputParam (macro syntax)=$(LocalVarOfValueToOutputParam)"
displayName: 'Show StageVariable'
Solution
The answer isn't actually far away. The original expression just need to be passed through a variable in the template job. Basically, set a variable to be the value of the parameter and use the macro syntax to evaluate the variable.
parameters:
- name: ValueToOutput
type: string
...
variables:
- name: LocalVarOfValueToOutputParam
value: ${{ parameters.ValueToOutput }}
Using the macro syntax of $(LocalVarOfValueToOutputParam) will result in the value making its way into the template job correctly.
Example
If we have a yaml file for the build definition:
stages:
- stage: CreateStageVarStage
displayName: 'Create StageVar Stage'
jobs:
- job: CreateStageVarJob
displayName: 'Create StageVar Job'
timeoutInMinutes: 5
pool:
name: 'Azure Pipelines'
vmImage: 'windows-2019'
steps:
- checkout: none
- pwsh: |
[string]$message = 'This is the value from the first stage'
Write-Host "Setting output variable 'VariableFromFirstStage' to '$message'"
Write-Output "##vso[task.setvariable variable=VariableFromFirstStage;isOutput=$true]$message"
name: SetValueStep
- stage: UseJobTemplateStage
displayName: 'Use Job Template Stage'
dependsOn: CreateStageVarStage
jobs:
- template: templates/job-showstagevars.yml
parameters:
ValueToOutput: $[ stagedependencies.CreateStageVarStage.CreateStageVarJob.outputs['SetValueStep.VariableFromFirstStage'] ]
That uses the job template templates/job-showstagevars.yml
parameters:
- name: ValueToOutput
type: string
jobs:
- job: ShowStageVarJob
displayName: 'Show stage var'
timeoutInMinutes: 5
pool:
name: 'Azure Pipelines'
vmImage: 'windows-2019'
variables:
- name: LocalVarOfValueToOutputParam
value: ${{ parameters.ValueToOutput }}
steps:
- checkout: none
- pwsh: |
Write-Host "ValueToOutput parameter=${{ parameters.ValueToOutput }}"
Write-Host "LocalVarOfValueToOutputParam (pre-processor syntax)=${{ variables.LocalVarOfValueToOutputParam }}"
Write-Host "LocalVarOfValueToOutputParam (macro syntax)=$(LocalVarOfValueToOutputParam)"
displayName: 'Show StageVariable'
What we get in our output of the second stage is:
ValueToOutput parameter=$[ stagedependencies.CreateStageVarStage.CreateStageVarJob.outputs['SetValueStep.VariableFromFirstStage'] ]
LocalVarOfValueToOutputParam (pre-processor syntax)=$[ stagedependencies.CreateStageVarStage.CreateStageVarJob.outputs['SetValueStep.VariableFromFirstStage'] ]
LocalVarOfValueToOutputParam (macro syntax)=This is the value from the first stage