I'm trying to learn how to use the new yaml configured pipeline system for Azure Devops, and I'm having a bit of trouble getting my head around the way the variables are supposed to work.
When I setup the pipeline, it created a file azure-pipelines.yml and committed this to the master branch.
By default, this file looks like so...
trigger:
- master
pool:
vmImage: 'windows-latest'
variables:
solution: '**/*.sln'
buildPlatform: 'Any CPU'
buildConfiguration: 'Release'
My project is setup with the following build configurations... "prod", "staging", "develop".
What I'm confused about, is where am I supposed to override these default variables for the actual pipelines?
I can modify the values directly in this file, but that's not really going to work. When I merge the changes back from "master" to "staging" etc, then presumably the pipelines for these lower environments will then be trying to build with "prod" configuration.
Surely there must be some way to configure variables independent of the source code.
There are 2 places where I can see an option to add Variables...
When I choose "Edit" for the pipeline, up in the top right, there is a "Variables" button next to run.
I can add variables there, but they don't appear to do anything. They are not applied when I run the pipeline.
Also, to make things more confusing, when I choose to "Run pipeline", there is also an option to define variables, but likewise, these don't seem to do anything. The build still just runs with the pre-defined values from the yaml file.
Agree with Shayki Abramczyk. This method could manually override the variable value on the UI interface.
I would like to share the method of automatically appending values to variables.
You could use Expressions to judge different situations(e.g. build branch). Then you could set the value for different situations.
Here is an example:
trigger:
- '*'
pool:
vmImage: 'windows-latest'
variables:
${{ if eq(variables['Build.SourceBranchName'], 'master') }}:
buildConfiguration: Prod
${{ if eq(variables['Build.SourceBranchName'], 'staging') }}:
buildConfiguration: Staging
steps:
- task: PowerShell#2
inputs:
targetType: 'inline'
script: |
# Write your PowerShell commands here.
Write-Host $(buildConfiguration)
This sample code can select the corresponding value according to different trigger branch names. (master: Prod , staging: staging)
Hope this helps.
You can define the variables with Let users override this value when running this pipeline:
Use the variable in the build step with $(BuildConfiguration ).
When you run the build you can override the value:
May be you need not a variables but a parameters?
parameters:
- name: STAND_NAME
displayName: Select Stand to deploy
type: string
default: none
values:
- dev
- stage
- prod
variables:
- group: global-variables # use global variable from library
- name: STAND_NAME
value: ${{ parameters.STAND_NAME }}
- ${{ if eq(parameters['STAND_NAME'], 'prod') }}:
- name: variable_depends_on_stand
value: "prod_value" #
- ${{ if eq(parameters['STAND_NAME'], 'stage') }}:
- name: variable_depends_on_stand
value: "stage_value"
- ${{ if eq(parameters['STAND_NAME'], 'dev') }}:
- name: variable_depends_on_stand
value: "dev_value"
- name: SOME_OTHER_GLOBAL_VARIABLE
value: some_other_value
It would be display like this in pipeline:
screenshot of the pipeline WUI
Related
I know that it is possible to conditionally set a variable in azure pipelines yml.
See https://learn.microsoft.com/en-us/azure/devops/pipelines/process/expressions?view=azure-devops#conditional-insertion
Is it also somehow possible to conditionally use a variable group?
Let's say if a pipeline run variable is set or has a certain value then the pipeline shall use a variable group. If not then the group shall not be used.
Thank you
Is it possible like here for instance:
trigger: none
pr: none
variables:
isProd: true
stages:
- stage: Test
displayName: Build and restore
variables:
- ${{ if eq(variables['isProd'], 'false') }}:
- group: QA
- ${{ if eq(variables['isProd'], 'true') }}:
- group: PROD
jobs:
- job: A
steps:
- bash: echo $(name)
I have a solution where a git branch is directly related to an environment (this has to be this way, so please do not discuss whether this is good or bad, I know it is not best practice).
We have the option to run a verification deployment (including automatic tests) towards an environment, without actually deploying the solution to the environment. Because of this, I would like to set up a pipeline that runs this verification for an environment, whenever a pull request is opened towards that environment's branch. Moreover, I am using a template for the majority of the pipeline. The actual pipeline in the main repository is just a tiny solution that points towards the template pipeline in another repository. This template, in turn, has stages for each respective environment.
I have, in the main pipeline, successfully added a solution that identifies the current branch, which for pull requests should be the target branch:
variables:
- name: currentBranch
${{ if eq(variables['Build.Reason'], 'PullRequest') }}:
value: $(System.PullRequest.TargetBranch)
${{ if ne(variables['Build.Reason'], 'PullRequest') }}:
value: $(Build.SourceBranch)
I would like to send this variable currentBranch down to the template through a parameter, as my template pipeline has different stages depending on the branch. My solution was to use the pipeline like this:
extends:
template: <template-reference>
parameters:
branch: $(currentBranch)
...and then for a stage in my pipeline do this:
- stage: TestAndDeployBranchName
condition: eq('${{ parameters.branch }}', 'refs/heads/branchName')
jobs:
- job1... etc.
Basically, the stage should run if the current branch is either "branchName", or (for pull requests) when the target branch is "branchName", which comes from the "branch" parameters that is sent to the template.
However, I see here that System.PullRequest.TargetBranch is not available for templates and further here that the parameters are not available for templates (the variable is empty) when the template is expanded. Thus my pipeline does not work as expected (the condition does not trigger when it should, ie. when there is a match on the branch name).
Is there any way that I can use System.PullRequest.TargetBranch in a condition within a template, or should I look for another solution?
After investigating this further I concluded that what I am trying to do is not possible.
In short, System.PullRequest.TargetBranch (and I assume at least some other variables within System.PullRequest are not available in compile time for template, which is when conditions are evaluated. Thus, using these variables in a condition in a template is not possible.
As my goal was to have certain steps run for pull requests only, based on the target branch of the pull request, I solved this by creating duplicate pipelines. Each pipeline is the same and references the same template, except for that the input parameter for the template is different. I then added each "PR pipelines" to run as part of the branch policy each respective branch this was applicable.
This works great, however it requires me to create a new pipeline if I have the same requirement for another branch. Moreover, I have to maintain each PR pipeline separately (which can be both good and bad).
Not an ideal solution, but it works.
Reference PR pipeline:
trigger: none # no trigger as PR triggers are set by branch policies
#This references the template repository to reuse the basic pipeline
resources:
repositories:
- repository: <template repo>
type: git # "git" means azure devops repository
name: <template name> # Syntax: <project>/<repo>
ref: refs/heads/master # Grab latest pipeline template from the master branch
stages:
- stage: VerifyPullRequest
condition: |
and(
not(failed()),
not(canceled()),
eq(variables['Build.Reason'], 'PullRequest')
)
displayName: 'Verify Pull Request'
jobs:
- template: <template reference> # Template reference
parameters:
image: <image>
targetBranch: <targetBranch> # Adjust this to match each respective relevant branch
The targetBranch parameter is the used in relevant places in the template to run PR verification.
Example of branch policy:
(Set this up for each relevant branch)
Picture of branch policy set up
After checking your script, we find we can not use the
variables:
- name: currentBranch
${{ if eq(variables['Build.Reason'], 'PullRequest') }}:
value: $(System.PullRequest.TargetBranch)
${{ if ne(variables['Build.Reason'], 'PullRequest') }}:
value: $(Build.SourceBranch)
in the variables.
The variables will duplicate the second value to first one.
This will cause your issue.
So, on my side, I create a work around and hope this will help you. Here is my main yaml:
parameters:
- name: custom_agent
displayName: Use Custom Agent
type: boolean
default: true
- name: image
type: string
default: default
resources:
repositories:
- repository: templates
type: git
name: Tech-Talk/template
trigger: none
pool:
vmImage: windows-latest
# vmImage: ubuntu-20.04
stages:
- stage: A
jobs:
- job: A1
steps:
- task: PowerShell#2
name: printvar
inputs:
targetType: 'inline'
script: |
If("$(Build.Reason)" -eq "PullRequest"){
Write-Host "##vso[task.setvariable variable=currentBranch;isOutput=true]$(System.PullRequest.TargetBranch)"
}
else{
Write-Host "##vso[task.setvariable variable=currentBranch;isOutput=true]$(Build.SourceBranch)"
}
- stage: B
condition: eq(dependencies.A.outputs['A1.printvar.currentBranch'], 'refs/heads/master')
dependsOn: A
jobs:
- job: B1
variables:
varFromA: $[ stageDependencies.A.A1.outputs['printvar.currentBranch'] ]
steps:
- task: PowerShell#2
inputs:
targetType: 'inline'
script: |
# Write your PowerShell commands here.
Write-Host "$(varFromA)"
- template: temp.yaml#templates
parameters:
branchName: $(varFromA)
agent_pool_name: ''
db_resource_path: $(System.DefaultWorkingDirectory)
Please Note:
If we use this, we need to modified your temp yaml.
We need to move the condition to the main yaml and make the temp yaml only steps is left.
I'm new to Azure DevOps and am looking to create the correct pipeline/release structures for my projects. I'm struggling to get to the point of how I'd build my code passing in the correct value for my build configs.
I have created my repo and branches:
main (main branch for released/to be released code)
feature/add_new_customer (feature branches per piece of work)
uat (a branch merging in multiple features commits specifically for testing prior to releases)
I created a build pipeline and this has created me an "azure-pipelines.yaml" file that I have used to build and publish the build files. This is triggered from my branches:
trigger:
- main
- feature/*
- uat
All good up to now. But this was by specifying the build config statically/hardcoded in yaml
variables:
solution: '**/*.sln'
buildPlatform: 'Any CPU'
**buildConfiguration: 'Release'**
I have now added a variable to my pipeline, BuildConfig, and set to "UAT" so I can reference in my yaml like so:
variables:
solution: '**/*.sln'
buildPlatform: 'Any CPU'
**buildConfiguration: '$(BuildConfig)'**
and this lets me set the build config "dynamically", but if I create another pipeline, say to build my release build from the main branch this creates an azure-pipelines-1.yaml which must be wrong as I'd then have to duplicate these files just to change the trigger branch?
Is there a "proper" way to create builds for different environments based on the branch I am checking in? I've seen the Environments but they just seem to offer me VM's or Kubernetes? I'm simply looking to build a .net framework legacy web form application so don't need anything fancy.
I've not started on the deployment process yet!!! :D
Sure, you can have a single pipeline that changes its behavior based upon variables within the release pipelines themselves. You'll want to do this at the time you run the build, instead of at the root variable level.
There are two ways you can accomplish this:
The condition: parameter within the build task
YAML If Statements that selectively set the variables.
Either would work in your scenario, I believe using an if statement would be best in your case. Here's what it would look like in a simple example:
name: Stackoverflow-Example-Variables
trigger:
- main
- feature/*
- uat
stages:
- stage: your_build_stage
variables:
- name: solution
value: '**/*.sln'
- name: buildPlatform
value: 'Any CPU'
- name: buildConfiguration
${{ if eq(variables['Build.SourceBranch'], 'refs/heads/main') }}:
value: 'Release'
${{ if ne(variables['Build.SourceBranch'], 'refs/heads/main') }}:
value: 'UAT'
displayName: "Build Solution"
jobs:
- job: output_message_job
displayName: "Output Message Job"
pool:
vmImage: "ubuntu-latest"
steps:
- powershell: |
Write-Host ${{ variables.buildConfiguration }}
Write-Host $(Build.SourceBranch)
Normal (non-template) jobs in Azure DevOps yaml support inter-job variable passing as follows:
jobs:
- job: A
steps:
- script: "echo ##vso[task.setvariable variable=skipsubsequent;isOutput=true]false"
name: printvar
- job: B
condition: and(succeeded(), ne(dependencies.A.outputs['printvar.skipsubsequent'], 'true'))
dependsOn: A
steps:
- script: echo hello from B
How do I do something similar in the following, given that templates don't support the dependsOn syntax? I need to get an output from the first template and pass it as 'environmentSlice' to the second template.
- stage: Deploy
displayName: Deploy stage
jobs:
- template: build-templates/get-environment-slice.yml#templates
parameters:
configFileLocation: 'config/config.json'
- template: build-templates/node-app-deploy.yml#templates
parameters:
# Build agent VM image name
vmImageName: $(Common.BuildVmImage)
environmentPrefix: 'Dev'
environmentSlice: '-$(dependencies.GetEnvironmentSlice.outputs['getEnvironmentSlice.environmentSlice'])'
The reason I want the separation between the two templates is the second one is a deployment template and I would like input from the first template in naming the environment in the second template. I.e. initial part of node-app-deploy.yml (2nd template) is:
jobs:
- deployment: Deploy
displayName: Deploy
# Because we use the environmentSlice to name the environment, we have to have it passed in rather than
# extracting it from the config file in steps below
environment: ${{ parameters.environmentPrefix }}${{ parameters.environmentSlice }}
Update:
The accepted solution does allow you to pass variables between separate templates, but won't work for my particular use case. I wanted to be able to name the 'environment' section of the 2nd template dynamically, i.e. environment: ${{ parameters.environmentPrefix }}${{ parameters.environmentSlice }}, but this can only be named statically since templates are compiled on pipeline startup.
The downside of the solution is that it introduces a hidden coupling between the templates. I would have preferred the calling pipeline to orchestrate the parameter passing between templates.
You can apply the depend on and dependency variable into templates.
See below sample:
To make sample more clear, here has 2 template files, one is azure-pipelines-1.yml, and another is azure-pipeline-1-copy.yml.
In azure-pipelines-1.yml, specify the environment value as output variable:
parameters:
environment: ''
jobs:
- job: preDeploy
variables:
EnvironmentName: preDeploy-${{ parameters.environment }}
steps:
- checkout: none
- pwsh: |
echo "##vso[task.setvariable variable=EnvironmentName;isOutput=true]$($env:ENVIRONMENTNAME)"
name: outputVars
And then, in azure-pipeline-1-copy.yml use dependency to get this output variable:
jobs:
- job: deployment
dependsOn: preDeploy
variables:
EnvironmentNameCopy: $[dependencies.preDeploy.outputs['outputVars.EnvironmentName']]
steps:
- checkout: none
- pwsh: |
Write-Host "$(EnvironmentNameCopy)"
name: outputVars
At last, in YAML pipeline, just need to pass the environment value
stages:
- stage: deployQA
jobs:
- template: azure-pipelines-1.yml
parameters:
environment: FromTemplate1
- template: azure-pipeline-1-copy.yml
Now, you can see the value get successfully in the second template job:
It is possible to avoid the dependency in the called template. However, as the OP says, the environment name cannot be created dynamically.
Here is an example of the "calling" template, which firstly calls another template (devops-variables.yml) that sets some environment variables that we wish to consume in a later template (devops-callee.yml):
stages:
- stage: 'Caller_Stage'
displayName: 'Caller Stage'
jobs:
- template: 'devops-variables.yml'
parameters:
InitialEnvironment: "Development"
- template: 'devops-callee.yml'
parameters:
SomeParameter: $[dependencies.Variables_Job.outputs['Variables_Job.Variables.SomeParameter']]
In the devops-variables.yml file, I have this:
"##vso[task.setvariable variable=SomeParameter;isOutput=true;]Wibble"
Then, in the "devops-callee.yml", I just consume it something like this:
parameters:
- name: SomeParameter
default: ''
jobs:
- deployment: 'Called_Job'
condition: succeeded()
displayName: 'Called Job'
environment: "Development"
pool:
vmImage: 'windows-2019'
dependsOn:
- Variables_Job
variables:
SomeParameter: ${{parameters.SomeParameter}}
strategy:
runOnce:
deploy:
steps:
- download: none
- task: AzureCLI#2
condition: succeeded()
displayName: 'An Echo Task'
inputs:
azureSubscription: "$(TheServiceConnection)"
scriptType: pscore
scriptLocation: inlineScript
inlineScript: |
echo "Before"
echo "$(SomeParameter)"
echo "After"
Output:
2021-04-10T09:22:29.6188535Z Before
2021-04-10T09:22:29.6196620Z Wibble
2021-04-10T09:22:29.6197124Z After
This way, the callee doesn't reference the caller. Unfortunately, setting the environment in the callee thus:
environment: "$(SomeParameter)"
doesn't work - you'll just get an environment with the literal characters '$(SomeParameter)'.
I'm trying to assign one of 2 values to a variable in addition to variable group and can't find the reference that how to use IF ELSE.
Basically I need to convert this jerkins logic to azure DevOps.
Jenkins
if (branch = 'master') {
env = 'a'
} else if (branch = 'dev'){
env ='b'
}
I found 1 reference from the following one, but this one seems to work if the variables section doesn't have variable groups.
https://stackoverflow.com/a/57532526/5862540
But in my pipeline, I already have a variable group for secrets, so I have to use name/value convention and the example doesn't work with the errors like expected a mapping or A mapping was not expected or Unexpected value 'env'
variables:
- group: my-global
- name: env
value:
${{ if eq(variables['Build.SourceBranchName'], 'master') }}:
env: a
${{ if eq(variables['Build.SourceBranchName'], 'dev') }}:
env: b
or
variables:
- group: my-global
- name: env
value:
${{ if eq(variables['Build.SourceBranchName'], 'master') }}: a
${{ if eq(variables['Build.SourceBranchName'], 'dev') }}: b
This code works.
I'm doing similar with parameters.
variables:
- name: var1
${{ if eq(parameters.var1, 'custom') }}:
value: $(var1.manual.custom)
${{ if ne(parameters.var1, 'custom') }}:
value: ${{ parameters.var1 }}
Update 09/09/2021
We have now natively if else expression and we can write it like
variables:
- group: PROD
- name: env
${{ if eq(variables['Build.SourceBranchName'], 'master') }}:
value: a
${{ else }}:
value: b
steps:
- script: |
echo '$(name)'
echo '$(env)'
Original reply
Syntax with template expressions ${{ if ...... }} is not limited only to job/stage level. Both below pipeline does the same and produce the same output:
stages:
- stage: One
displayName: Build and restore
variables:
- group: PROD
- name: env
${{ if eq(variables['Build.SourceBranchName'], 'master') }}:
value: a
${{ if eq(variables['Build.SourceBranchName'], 'dev') }}:
value: b
jobs:
- job: A
steps:
- script: |
echo '$(name)'
echo '$(env)'
variables:
- group: PROD
- name: env
${{ if eq(variables['Build.SourceBranchName'], 'master') }}:
value: a
${{ if eq(variables['Build.SourceBranchName'], 'dev') }}:
value: b
steps:
- script: |
echo '$(name)'
echo '$(env)'
Microsoft a few weeks ago released a new feature for YAML pipeliens that just lets you do that: IF ELSE notation.
https://learn.microsoft.com/en-us/azure/devops/release-notes/2021/sprint-192-update#new-yaml-conditional-expressions
Writing conditional expressions in YAML files just got easier with the use of ${{ else }} and ${{ elseif }} expressions. Below are examples of how to use these expressions in YAML pipelines files.
steps:
- script: tool
env:
${{ if parameters.debug }}:
TOOL_DEBUG: true
TOOL_DEBUG_DIR: _dbg
${{ else }}:
TOOL_DEBUG: false
TOOL_DEBUG_DIR: _dbg
variables:
${{ if eq(parameters.os, 'win') }}:
testsFolder: windows
${{ elseif eq(parameters.os, 'linux' }}:
testsFolder: linux
${{ else }}:
testsFolder: mac
I wanted to have runtime condition evaluation, something similar to compile time:
variables:
VERBOSE_FLAG:
${{if variables['System.Debug']}}:
value: '--verbose'
${{else}}:
value: ''
but unfortunately Azure devops does not supports special kind of functions like if(condition, then case, else case) - so I've played around and find out that it's possible do double string replacement using replace function. It does looks bit hacky of course.
For example, one may want to tweak task inputs depending on whether system debugging is enabled or not. This cannot be done using "standard conditional insertion" (${{ if … }}:), because System.Debug isn't in scope in template expressions. So, runtime expressions to the rescue:
- job:
variables:
VERBOSE_FLAG: $[
replace(
replace(
eq(lower(variables['System.Debug']), 'true'),
True,
'--verbose'
),
False,
''
)
]
steps:
- task: cURLUploader#2
inputs:
# …
options: --fail --more-curl-flags $(VERBOSE_FLAG)
Note that using eq to check the value of System.Debug before calling replace is not redundant: Since eq always returns either True or False, we can then safely use replace to map those values to '--verbose' and '', respectively.
In general, I highly recommend sticking to boolean expressions (for example the application of a boolean-valued function like eq, gt or in) as the first argument of the inner replace application. Had we not done so and instead just written for example
replace(
replace(
lower(variables['System.Debug']),
'true',
'--verbose'
),
'false',
''
)
then, if System.Debug were set to e.g. footruebar, the value of VERBOSE_FLAG would have become foo--verbosebar.
I think for now you're going to need to use a task to customize with name/value syntax variables and conditional variable values. It looks like the object structure for name/value syntax breaks the parsing of expressions, as you have pointed out.
For me, the following is a reasonably clean implementation, and if you want to abstract it away from the pipeline, it seems that a simple template for your many pipelines to use should satisfy the desire for a central "global" location.
variables:
- group: FakeVarGroup
- name: env
value: dev
steps:
- powershell: |
if ($env:Build_SourceBranchName -eq 'master') {
Write-Host ##vso[task.setvariable variable=env;isOutput=true]a
return
} else {
Write-Host ##vso[task.setvariable variable=env;isOutput=true]b
}
displayName: "Set Env Value"
As far as I know, the best way to have conditional branch build is using "trigger" in your YAML, instead of implementing complex "if-else". It is also much safer, and you have more explicit controls on the branch triggers instead of relying on CI variables.
Example:
# specific branch build
jobs:
- job: buildmaster
pool:
vmImage: 'vs2017-win2016'
trigger:
- master
steps:
- script: |
echo "trigger for master branch"
- job: buildfeature
pool:
vmImage: 'vs2017-win2016'
trigger:
- feature
steps:
- script: |
echo "trigger for feature branch"
To have trigger with branches inclusion and exclusion, you could use more complex syntax of trigger with branches include and exclude.
Example:
# specific branch build
trigger:
branches:
include:
- master
- releases/*
exclude:
- releases/1.*
The official documentation of Azure DevOps Pipelines trigger in YAML is:
Azure Pipelines YAML trigger documentation
UPDATE 1:
I repost my comment here with additional notes:
I was thinking to have different pipelines because having the complexity of juggling between CI variables is not more maintainable than having multi jobs in one YAML with triggers. Having multijobs with triggers is also enforcing us to have clear distinction and provision on branch management. Triggers and conditional branches inclusions have been used for a year by my team because of these maintainability advantages.
Feel free to disagree, but to me having an embedded logic in any scripted in any steps to check which branch is currently in session and then does any further actions, are more like ad-hoc solutions. And this has given my team and me maintenance problems before.
Especially if the embedded logic tends to grow by checking other branches, the complexity is more complex later than having clear separations between branches. Also if the YAML file is going to be maintained for long time, it should have clear provisions and roadmaps across different branches. Redundancy is unavoidable, but the intention to separate specific logic will pay more in the long run for maintainability.
This is why I also emphasize branches inclusions and exclusions in my answer :)
Azure YAML if-else solution (when you have a group defined which required name/value notation use thereafter.
variables:
- group: my-global
- name: env
value: a # set by default
- name: env
${{ if eq(variables['Build.SourceBranchName'], 'master') }}:
value: b # will override default
Of if you don't have a group defined:
variables:
env: a # set by default
${{ if eq(variables['Build.SourceBranchName'], 'master') }}:
env: b # will override default