I have one pipeline created with two stages.
Lets say,
Stage A
Stage B
I would like to run Stage A when PR is merged into qa branch and run Stage B when release/* or hotfix/* branch is created.
Can this be achieved in one pipeline? If so how?
You can try to use CI trigger:
trigger:
- qa
- releases/*
- hotfix/*
Then detect them for each stage: Specify conditions
variables:
isQA: $[eq(variables['Build.SourceBranch'], 'refs/heads/qa')]
- stage: B
condition: and(succeeded(), eq(variables.isQA, 'true'))
jobs:
- job: B1
steps:
- script: echo Hello Stage B!
- script: echo $(isQA)
Check additional conditions: contains
Related
I have 3 environments in Azure , Sandbox, Test and Prod.
I have yaml pipeline in azure devops which builds the infrastructure. The environment built depends on the variables in the terraform code
The same pipleline is used to deploy to all environments depending on conditions in the yaml file.I want Dev to trigger on a merge to master but only want test and prod to deploy manually. How can i set this up in the yaml file?
For Dev, you can set up build validation of branch 'Master':
And in the Master branch, the YML file should be like this:
trigger:
- master
pool:
vmImage: ubuntu-latest
steps:
- script: echo Hello, world!
displayName: 'Run a one-line script'
This can make sure the pipeline of the master branch(Dev environment) only triggers after the PR merge is completed.
For test and prod environment, you can create a branch with the same name YML that the pipeline is looking for. And use the below YML definition:
trigger:
- none
pool:
vmImage: ubuntu-latest
steps:
- script: echo Hello, world!
displayName: 'Run a one-line script'
This can make sure the test and prod only can be triggered manually.
Above solution just need one pipeline.
Based on your requirement, you can add the condition in your YAML pipeline.
Dev to trigger on a merge to master
You can set the variable: Build.Reason and System.PullRequest.TargetBranch.
For example:
condition: and(eq(variables['Build.Reason'], 'PullRequest'), eq(variables['System.PullRequest.TargetBranch'], 'refs/heads/master'))
want test and prod to deploy manually
You can set the variable: Build.Reason
condition: eq(variables['Build.Reason'], 'Manual')
YAML example:
stages:
- stage: Dev
condition: and(eq(variables['Build.Reason'], 'PullRequest'), eq(variables['System.PullRequest.TargetBranch'], 'refs/heads/master'))
jobs:
- job: A
steps:
- xx
- stage: Prod
condition: eq(variables['Build.Reason'], 'Manual')
jobs:
- job: B
steps:
- xx
- stage: Test
condition: eq(variables['Build.Reason'], 'Manual')
jobs:
- job: C
steps:
- xx
When the pipeline is triggered by Pull Request and the target branch is master, the Dev stage will run.
When the pipeline is triggered manually, the Test and Prod stage will run.
Refer to this doc: Condition and Predefined variables.
Is it possible to have a single file azure-pipelines.yaml that can :
Trigger a job A on a push from any branch BUT main
Trigger a job B on PR to main and all subsequent commit on that PR
Trigger a job C when main is merged
I have tried to play arround with trigger, pr keywords and even with condition(), variables['Build.Reason'], or System.PullRequest.TargetBranch but I didn't manage to reach the expected result.
I start thinking it cannot be done with a single file - am I wrong?
You can set conditions on your stages to run depending on a variable but I am not pretty sure this will work with your conditions. Maybe you could also combine some variable values.
For example source branch main and pr is created.
and(eq(variables['Build.Reason'], 'PullRequest'), eq(variables['Build.SourceBranch'], 'refs/heads/main'))
Azure documentation sample:
variables:
isMain: $[eq(variables['Build.SourceBranch'], 'refs/heads/main')]
stages:
- stage: A
jobs:
- job: A1
steps:
- script: echo Hello Stage A!
- stage: B
condition: and(succeeded(), eq(variables.isMain, 'true'))
jobs:
- job: B1
steps:
- script: echo Hello Stage B!
- script: echo $(isMain)
Keep in mind that triggers are appending resources. This means that if you specify a trigger like the below, it will be triggered whether the branch filter is triggered OR the pr is created.
trigger:
branches:
include:
- '*'
pr:
branches:
include:
- current
As you said this can be accomplished for sure with separate files for the pipelines.
https://learn.microsoft.com/en-us/azure/devops/pipelines/process/conditions?view=azure-devops&tabs=yaml
Your first question is possible by using trigger with include and exclude branches as below
trigger:
branches:
include:
- stage
- releases/*
exclude:
- master
Refer CI triggers in Azure Repos Git documentation to have more understanding.
I'm trying to set up a pipeline in Azure.
Actual behavior
On pull-request a build validation triggers the pipeline and all stages and jobs are triggered.
After the merge, all jobs are skipped.
Expected behavior
After the merge of the pull request, I would expect stage Bto be triggered
Question
What am I missing so pipeline triggers correctly on merge?
azure.pipelines.yml
trigger:
branches:
include:
- master
stages:
- template: 'main.yml'
main.yml
stages:
- stage: 'A'
condition: startsWith(variables['build.reason'], 'PullRequest')
jobs:
- job: A
steps:
- script: echo A
- stage: 'B'
jobs:
- job: 'B'
steps:
- script: echo B
The trigger feature only works for the whole pipeline and not for an individual stage/job in the pipeline.
Normally, we use the different trigger types (CI/PR, resources) and filters (branches, paths, tags) to define when the pipeline should be triggered.
In the pipeline, we generally specify conditions to a stage, job or step to define when the this stage, job or step should be run or not. The conditions will be verified after the pipeline has been triggered.
To specify conditions, you can use one of the following ways:
use the condition key.
use the if expression.
I found the issue and applied suggestions from #bright-ran-msft
Stage A and B are implicitly linked. Since stage A was not triggered on the merge, stage B would not start either.
Solution
Instead of using condition, it is required to use if
Example
stages:
- ${{ if eq( variables['build.reason'], 'PullRequest') }}:
- stage: 'A'
jobs:
- job: A
steps:
- script: echo A
- stage: 'B'
jobs:
- job: 'B'
steps:
- script: echo B
My team is responsible for 10 microservices and it would be useful to have a single pipeline that triggers their individual CI/CD pipelines.
I know it is possible to use pipeline triggers, like
resources:
pipelines:
- pipeline: MasterPipeline
source: DeployAllMicroservices
trigger: true
and I can add this to the pipelines and create a very simple DeployAllMicroservices pipeline. This works, but the pipelines will be triggered in a random order.
The thing is, two services need to be rolled out first before the other 8 can be deployed. Is there a way to first trigger pipeline A & B, where pipelines C-J are triggered after their completion?
Something else I've tried is to load the pipeline files A.yml, B.yml as templates from the master pipeline.
steps:
- template: /CmcBs/Pipelines/A.yml
- template: /CmcBs/Pipelines/B.yml
but that doesn't work with full-fledged pipelines (starting with trigger, pool, parameters, et cetera).
Currently DevOps does not support multiple pipelines as the triggering pipeline of one pipeline at the same time.
There is a workaround you can refer to:
Set pipelineA as the triggering pipeline of the pipelineB.
Set pipelineB as the triggering pipeline of the other pipelines(pipelines c-j).
For more info about the triggering pipeline, please see Trigger one pipeline after another.
Another approach is to use stages in order to execute first pipeline A,B and then C to J.
An example .yml for this approach would be the below.
trigger:
- none
pool:
vmImage: ubuntu-latest
stages:
- stage: FirstBatch
displayName: Build First Batch
jobs:
- job: pipelineA
displayName: Build pipelineA
steps:
- script: echo pipelineA
displayName: pipelineA
- job: pipelineB
displayName: Build pipelineB
steps:
- script: echo pipelineB
displayName: pipelineB
- stage: SecondBatch
displayName: Build Second Batch
jobs:
- job: pipelineC
displayName: Build pipelineC
steps:
- checkout: none
- script: echo Build pipelineC
displayName: Build pipelineC
- job: pipelineD
displayName: Build pipelineD
steps:
- checkout: none
- script: echo Build pipelineD
displayName: Build pipelineD
- job: pipelineE
displayName: Build pipelineE
steps:
- checkout: none
- script: echo Build pipelineE
displayName: Build pipelineE
The drawback of this approach would be to have a single pipeline and not separated pipelines for your microservices. In order to decouple more this solution you could use templates.
I have a pipeline.yaml that looks like this
pool:
vmImage: image
stages:
-stage: A
jobs:
-job: a
steps:
- script: |
echo "This is stage build"
echo "##vso[task.setvariable variable=doThing;isOutput=true]Yes"
name: BuildStageRun
-stage: B
jobs:
-job: b
steps: #do something in steps
-job: c
dependsOn: a
condition: eq(dependencies.build.outputs['BuildStageRun.doThing'], 'Yes')
steps:
- script: echo "I am scripting"
So, there are 2 stages, A with one job a, and B with 2 jobs b and c. I would like to have job c executed only when job a has executed. I tried to do so by setting the variable doThing in job a to Yes and then check this variable in job c.
But I get an error:
Stage plan job c depends on unknown job a.
The varible definition and the definition of condition was taken from Azure documentation
Do you have any suggestion on how to get this working?
While Shayki is correct that it is not supported - there is a workaround that I am currently using. That I used with the help of this blog https://medium.com/microsoftazure/how-to-pass-variables-in-azure-pipelines-yaml-tasks-5c81c5d31763
Basically you create your output as normal, and then publish the variables as pipeline artifacts. In the next stage, you read the artifact in the first job, and use that to construct your conditionals e.g.
stages:
- stage: firststage
jobs:
- job: setup_variables
pool:
vmImage: 'Ubuntu-16.04'
steps:
- powershell: |
$ShouldBuildThing1 = $true
# Write to normal output for other jobs in this stage
Write-Output "##vso[task.setvariable variable=BuildThing1;isOutput=true]$ShouldBuildThing1"
# Write to file to publish later
mkdir -p $(PipelineWorkspace)/variables
Write-Output "$ShouldBuildThing1" > $PipelineWorkspace/variables/BuildThing1
name: variablesStep
# Publish the folder as pipeline artifact
- publish: $(Pipeline.Workspace)/variables
artifact: VariablesArtifact
- job: build_something
pool:
vmImage: 'Ubuntu-16.04'
dependsOn: setup_variables
condition: eq(dependencies.setup_variables.outputs['variablesStep.BuildThing1'], 'true')
variables:
BuildThing1: $[dependencies.setup_variables.outputs['variablesStep.BuildThing1']]
steps:
- powershell: |
Write-Host "Look at me I can Read $env:BuildThing1"
- somethingElse:
someInputArgs: $(BuildThing1)
- stage: secondstage
jobs:
- job: read_variables
pool:
vmImage: 'Ubuntu-16.04'
steps:
# If you download all artifacts then foldername will be same as artifact name under $(Pipeline.Workspace). Artifacts are also auto downloaded on deployment jobs.
- task: DownloadPipelineArtifact#2
inputs:
artifact: "VariablesArtifact"
path: $(Pipeline.Workspace)/VariablesArtifact
- powershell: |
$ShouldBuildThing1 = $(Get-Content $(Pipeline.Workspace)/VariablesArtifact/BuildThing1)
Write-Output "##vso[task.setvariable variable=BuildThing1;isOutput=true]$ShouldBuildThing1"
name: variablesStep
- job: secondjob
pool:
vmImage: 'Ubuntu-16.04'
dependsOn: read_variables
condition: eq(dependencies.read_variables.outputs['variablesStep.BuildThing1'], 'true')
variables:
BuildThing1: $[dependencies.setup_variables.outputs['variablesStep.BuildThing1']]
steps:
- powershell: |
Write-Host "Look at me I can Read $env:BuildThing1"
- somethingElse:
someInputArgs: $(BuildThing1)
Looks like a few options available from Microsoft now.
First is job-to-job dependencies across stages
From Microsoft:
In this example, job B1 will run whether job A1 is successful or skipped. Job B2 will check the value of the output variable from job A1 to determine whether it should run.
trigger: none
pool:
vmImage: 'ubuntu-latest'
stages:
- stage: A
jobs:
- job: A1
steps:
- bash: echo "##vso[task.setvariable variable=shouldrun;isOutput=true]true"
# or on Windows:
# - script: echo ##vso[task.setvariable variable=shouldrun;isOutput=true]true
name: printvar
- stage: B
dependsOn: A
jobs:
- job: B1
condition: in(stageDependencies.A.A1.result, 'Succeeded', 'SucceededWithIssues', 'Skipped')
steps:
- script: echo hello from Job B1
- job: B2
condition: eq(stageDependencies.A.A1.outputs['printvar.shouldrun'], 'true')
steps:
- script: echo hello from Job B2
Also, there's another option where you could consume output variables across stages.
From Microsoft site:
Stages can also use output variables from another stage. In this example, Stage B depends on a variable in Stage A.
stages:
- stage: A
jobs:
- job: A1
steps:
- bash: echo "##vso[task.setvariable variable=shouldrun;isOutput=true]true"
# or on Windows:
# - script: echo ##vso[task.setvariable variable=shouldrun;isOutput=true]true
name: printvar
- stage: B
condition: and(succeeded(), eq(dependencies.A.outputs['A1.printvar.shouldrun'], 'true'))
dependsOn: A
jobs:
- job: B1
steps:
- script: echo hello from Stage B
It's because you can't depend on a job from another stage, you can depend stage B on stage A or job c on job b.
You can't achieve your goal with YAML conditions because you want to use a variable that you declared it in the first stage, the second stage doesn't know this variable, Azure DevOps don't support it yet:
You cannot currently specify that a stage run based on the value of an
output variable set in a previous stage.
You can depend stage B on A, so if in stage A there is only one job you depend stage B on stage A:
- stage: B
dependsOn: A