I have a set of jobs that needs to be triggered only when branches are named by certain pattern. (Eg: ctl-sw-v01.01.01-vbeta, ctl-sw-v01.11.01-vbeta, ctl-sw-v11.01.01-vbeta, ctl-sw-v01.01.21-vbeta). To make this generic, I have a pattern developed using regex '~ /^ctl-sw-v\d+\.\d+.\d+(-vbeta)?$/'. But I am finding it confusing as to how to specify this in the variables section in the yml file.
I was using as below:
trigger:
tags:
include:
- '*'
branches:
include:
- ctl-sw-v01.11.01-vbeta
pool:
vmImage: ubuntu-latest
variables:
isBranch: $[startsWith(variables['Build.SourceBranch'], 'refs/head/~ /^ctl-sw-v\d+\.\d+.\d+(-vbeta)?$/')]
jobs:
- job: A
condition: and(succeeded(), eq(variables.isBranch, 'true'))
steps:
- script: |
echo "hello"
- job: B
steps:
- script: |
echo "howdy"
The job A is being skipped continuously, I checked my regex, it produces a match.
What am I doing wrong here?
I don't think you can use regular expressions in the startsWith expression.
A solution would be to create a separate step (in a job before job A) where you evaluate the variable isBranch with a script. This creates more code, but the benefit is that you can independently test the regex in the future which might save maintenance cost in the long run.
Moreover, condition: and(succeeded(), eq(variables.isBranch, 'true')) booleans are written simply as True, not 'true'.
As Bast said, it is not supported to use regular expressions as part of condition expressions in Azure DevOps.
In addition,
The branches should begins with refs/heads/, but you are using refs/head/.
To avoid mistake like this, I suggest you to use Build.SourceBranchName instead of Build.SourceBranch. Build.SourceBranchName ignores the file structure of the branch and just search for the branch name.
Here is the example:
variables['Build.SourceBranch'], 'refs/heads/main'
variables['Build.SourceBranchName'], 'main'
Related
Is it possible to have a single file azure-pipelines.yaml that can :
Trigger a job A on a push from any branch BUT main
Trigger a job B on PR to main and all subsequent commit on that PR
Trigger a job C when main is merged
I have tried to play arround with trigger, pr keywords and even with condition(), variables['Build.Reason'], or System.PullRequest.TargetBranch but I didn't manage to reach the expected result.
I start thinking it cannot be done with a single file - am I wrong?
You can set conditions on your stages to run depending on a variable but I am not pretty sure this will work with your conditions. Maybe you could also combine some variable values.
For example source branch main and pr is created.
and(eq(variables['Build.Reason'], 'PullRequest'), eq(variables['Build.SourceBranch'], 'refs/heads/main'))
Azure documentation sample:
variables:
isMain: $[eq(variables['Build.SourceBranch'], 'refs/heads/main')]
stages:
- stage: A
jobs:
- job: A1
steps:
- script: echo Hello Stage A!
- stage: B
condition: and(succeeded(), eq(variables.isMain, 'true'))
jobs:
- job: B1
steps:
- script: echo Hello Stage B!
- script: echo $(isMain)
Keep in mind that triggers are appending resources. This means that if you specify a trigger like the below, it will be triggered whether the branch filter is triggered OR the pr is created.
trigger:
branches:
include:
- '*'
pr:
branches:
include:
- current
As you said this can be accomplished for sure with separate files for the pipelines.
https://learn.microsoft.com/en-us/azure/devops/pipelines/process/conditions?view=azure-devops&tabs=yaml
Your first question is possible by using trigger with include and exclude branches as below
trigger:
branches:
include:
- stage
- releases/*
exclude:
- master
Refer CI triggers in Azure Repos Git documentation to have more understanding.
I'm trying to set up a pipeline in Azure.
Actual behavior
On pull-request a build validation triggers the pipeline and all stages and jobs are triggered.
After the merge, all jobs are skipped.
Expected behavior
After the merge of the pull request, I would expect stage Bto be triggered
Question
What am I missing so pipeline triggers correctly on merge?
azure.pipelines.yml
trigger:
branches:
include:
- master
stages:
- template: 'main.yml'
main.yml
stages:
- stage: 'A'
condition: startsWith(variables['build.reason'], 'PullRequest')
jobs:
- job: A
steps:
- script: echo A
- stage: 'B'
jobs:
- job: 'B'
steps:
- script: echo B
The trigger feature only works for the whole pipeline and not for an individual stage/job in the pipeline.
Normally, we use the different trigger types (CI/PR, resources) and filters (branches, paths, tags) to define when the pipeline should be triggered.
In the pipeline, we generally specify conditions to a stage, job or step to define when the this stage, job or step should be run or not. The conditions will be verified after the pipeline has been triggered.
To specify conditions, you can use one of the following ways:
use the condition key.
use the if expression.
I found the issue and applied suggestions from #bright-ran-msft
Stage A and B are implicitly linked. Since stage A was not triggered on the merge, stage B would not start either.
Solution
Instead of using condition, it is required to use if
Example
stages:
- ${{ if eq( variables['build.reason'], 'PullRequest') }}:
- stage: 'A'
jobs:
- job: A
steps:
- script: echo A
- stage: 'B'
jobs:
- job: 'B'
steps:
- script: echo B
I am trying to use if else conditions in Azure Devops yml pipeline with variable groups. I am trying to implement it as per latest Azure Devops yaml pipeline build.
Following is the sample code for the if else condition in my scenario. test is a variable inside my-global variable group.
variables:
- group: my-global
- name: fileName
${{ if eq(variables['test'], 'true') }}:
value: 'product.js'
${{ elseif eq(variables['test'], false) }}:
value: 'productCost.js'
jobs:
- job:
steps:
- bash:
echo test variable value $(fileName)
When the above code is executed, in echo statement we don't see any value for filename, i.e. it empty, meaning none of the above if else condition was executed, however when I test the if else condition with the following condition.
- name: fileName
${{ if eq('true', 'true') }}:
value: 'product.js'
Filename did echo the correct value, i.e. product.js. So my conclusion is that I am not able to refer the variables from the variable group correctly. So any suggestion will be helpful and appreciated.
Thanks!
Unfortunately there is no ternary operator in Azure DevOps Pipelines. And it seems unlikely considering the state of https://github.com/microsoft/azure-pipelines-yaml/issues/256 and https://github.com/microsoft/azure-pipelines-yaml/issues/278. So for the time being the only choices are :
conditional insertion : it works with parameters, and should work with variables according to the documentation (but it is difficult to use properly),
or the hacks you can find in this Stack Overflow question.
Another work-around has been posted by Simon Alling on GitHub (https://github.com/microsoft/azure-pipelines-yaml/issues/256#issuecomment-1077684972) :
format(
replace(replace(condition, True, '{0}'), False, '{1}'),
valueIfTrue,
valueIfFalse
)
It is similar to the solution provided by Tejas Nagchandi, but I find it a little bit better because the syntax looks closer to what it would be if there was a ternary operator.
I was able to achieve the goal using some dirty work-around, but I do agree that using parameters would be much better way unless ternary operators are available for Azure DevOps YAML pipeline.
The issue is that ${{ if condition }}: is compile time expression, thus the variables under variable group are not available.
I was able to use runtime expressions $[<expression>]
Reference: https://learn.microsoft.com/en-us/azure/devops/pipelines/process/expressions?view=azure-devops
My pipeline:
trigger:
- none
variables:
- group: Temp-group-for-testing
- name: fileName
value: $[replace(replace('True',eq(variables['test'], 'True'), 'value1'),'True','value2')]
stages:
- stage: test
jobs:
- job: testvar
continueOnError: false
steps:
- bash: echo $(fileName)
displayName: "echo variable"
Results are available on github
After detailed investigation I realized that if else doesnt work with variables in Az Devop yaml pipelines, it only works with parameters. However the solution posted by #Tejas Nagchandi is a workaround and might be able to accomplish the same logic of if else setting variable value with replace commands. Hats off to TN.
I'm looking for a way to define a variable in my azure-pipelines.yml file where I can substring the 'Build.SourceVersion' -> Use the first 7 characters only.
Seems like there is no build-in function that can do such string operations in the documentation. Is there something I'm missing?
My other approach would be to use a bash task and overwrite the variable there but finding something build-in that can do this would be way better solution.
My other approach would be to use a bash task and overwrite the variable there but finding something build-in that can do this would be way better solution.
I agree with Lucas. There is no such built-in task to get the first 7 characters of $(Build.SourceVersion) in Azure DevOps.
We could use the command line/powershell task to split long sha into short sha:
echo $(Build.SourceVersion)
set TestVar=$(Build.SourceVersion)
set MyCustomVar=%TestVar:~0,7%
echo %MyCustomVar%
echo ##vso[task.setvariable variable=ShortSourceVersion]%MyCustomVar%
In this case, we could get the short versions of Build.SourceVersion and set it as environment variable.
Then we could set this command line task as a task group:
So, we could use this task to set the ShortSourceVersion directly.
Hope this helps.
Here is the shortest version that I use for this job;
- bash: |
longcommithash=$(Build.SourceVersion)
echo "##vso[task.setvariable variable=shorthash;]$(echo ${longcommithash::9})"
It gives you the output as shown below;
On Azure devops it's possible to be done using batch technique indeed - like other answers propose in here - however I wanted to remove string from the end of string - and with batch this gets more complex ( See following link ).
Then I've concluded to use Azure's built-in expressions, for example like this:
- name: sourceBranchName
value: ${{ replace(variables['Build.SourceBranch'], 'refs/heads/') }}
replace will have similar effect to substring - so it will remove unnecessary part from original string.
If however you want to concatenate some string and then remove it from original name - it's possible to be done using for instance format function:
- name: originBranchName
value: ${{ replace(variables.sourceBranchName, format('-to-{0}', variables.dynamicvar ) ) }}
If dynamicvar is a - then function will remove -to-a from branch name.
You are right, there is no native way to do it. You will have to write a script to transform the variable.
Here is an example:
trigger:
- master
resources:
- repo: self
stages:
- stage: Build
displayName: Build image
jobs:
- job: Build
displayName: Build
pool:
vmImage: 'ubuntu-latest'
steps:
- task: CmdLine#2
inputs:
script: ' x=`echo "$(Build.SourceVersion)" | head -c 7`; echo "##vso[task.setvariable variable=MyVar]$x"'
- task: CmdLine#2
inputs:
script: 'echo "$(MyVar)"'
Sure you can! If you absolutely must. Here is a runtime computation taking the first 7 characters of the Build.SourceVersion variable.
variables:
example: ${{ format('{0}{1}{2}{3}{4}{5}{6}', variables['Build.SourceVersion'][0], variables['Build.SourceVersion'][1], variables['Build.SourceVersion'][2], variables['Build.SourceVersion'][3], variables['Build.SourceVersion'][4], variables['Build.SourceVersion'][5], variables['Build.SourceVersion'][6]) }}
NB: I can't get it to work with $[...] syntax as the variable is apparently empty at the initial.
I am trying to set Azure DevOps to skip a stage on a multi-stage pipeline if a message does not start with a given text.
From the examples documentation, I think it is just
- stage: t1
condition: not(startsWith(variables['Build.SourceVersionMessage'], '[maven-release-plugin]'))
jobs:
- job: ReleasePrepare
displayName: Prepare release
pool:
vmImage: 'ubuntu-16.04'
steps:
- script: |
env | sort
However, this gets executed regardless. Here's an example of where I expect the t1 task to not be run based on the commit message https://dev.azure.com/trajano/experiments/_build/results?buildId=110&view=results
The output of env shows that the message was passed in correctly
Just in case it is a bug I reported it here as well https://developercommunity.visualstudio.com/content/problem/697290/startswith-buildsourceversionmessage-variable-not.html
It appears that Build.SourceVersionMessage at the time of this post is only resolvable on the steps.
Here's a working example that stores the value in a variable in one step and use it in the next job (which can be a deployment)
trigger:
batch: true
branches:
include:
- master
stages:
- stage: ci
displayName: Continuous Integration
jobs:
- job: Build
pool:
vmImage: 'ubuntu-16.04'
steps:
- script: |
env | sort
echo "$(Build.SourceVersionMessage)"
- stage: t1
displayName: Release
condition: eq(variables['Build.SourceBranch'],'refs/heads/master')
jobs:
- job: GetCommitMessage
displayName: Get commit message
steps:
- bash: |
echo "##vso[task.setvariable variable=commitMessage;isOutput=true]$(Build.SourceVersionMessage)"
echo "Message is '$(Build.SourceVersionMessage)''"
name: SetVarStep
displayName: Store commit message in variable
- job: ReleasePrepare
displayName: Prepare release
dependsOn: GetCommitMessage
pool:
vmImage: 'ubuntu-16.04'
condition: not(startsWith(dependencies.GetCommitMessage.outputs['SetVarStep.commitMessage'], '[maven-release-plugin]'))
steps:
- script: |
echo this would be a candidate for release
env | sort
displayName: Don't do it if maven release
- job: NotReleasePrepare
displayName: Don't Prepare Release
dependsOn: GetCommitMessage
pool:
vmImage: 'ubuntu-16.04'
condition: startsWith(dependencies.GetCommitMessage.outputs['SetVarStep.commitMessage'], '[maven-release-plugin]')
steps:
- script: |
echo this not be a candidate for release because it was created by the plugin
env | sort
condition: startsWith(variables.commitMessage, '[maven-release-plugin]')
displayName: Do it if maven release
The build can be found in https://dev.azure.com/trajano/experiments/_build/results?buildId=133&view=logs&s=6fc7e65a-555d-5fab-c78f-9502ae9436c4&j=b5187b8c-216e-5267-fcdb-c2c33d846e05
I am trying to set Azure DevOps to skip a stage if a message does not
start with a given text.
If I am not misunderstand, the condition you want is if the message match the start with maven-release-plugin, the current stage will be queued.
If this, the condition you write is not correct, I think you should specified it:
startsWith(variables['Build.SourceVersionMessage'], '[maven-release-plugin]')
As I tested on my pipeline:
And in fact, the value of this variable is Deleted 121321. Here is the result:
As you can see, that it is successful to skip the stage. My logic is, the value of Build.SourceVersionMessage should start with othermessage. But in fact, in my pipeline, it's value is Deleted 121321. Not match, so skip this stage.
(Delete 121321 is not only my PR name, I just set the commit message as the default PR name.)
Updated 2:
Though my test logic is not incorrect, but after I reproduced with YAML and many other method tested, such as use Build.SourceVersion which can only got after the source pulled.
Yes, you are right about that Build.SourceVersionMessage does not has value in Job level. As my tested, it indeed null in job level.
But, unfortunately, this is not a bug. This is as designed in fact.
We can think that the source repos be pulled locally only the stage job begin to execute, right? You can see the Checkout log, it record the process about pull source file down.
If the stage does not be executed, the source will not be pulled down. But also, if no source pulled, the server will also could not get the value of Build.SourceVersionMessage because of no source history. That's why I also tested with the variable Build.SourceVersion in the Job level.
We can not use these two variable at the agent job level because it hasn't pulled sources yet so Build.SourceVersionMessage is null. You'll need to copy it to each step in your pipeline. This is what confirmed by our Product Group team.
But, I still need to say sorry. Sorry about our doc is not too clear to announce that this could not used in agent job level.