The environment which I'm deploying to has 3 stages
dev
uat
production
The git repository for the project has dev and main branches.
On push to dev, the CI pipeline runs, creates an image and pushes the image to a registry.
Next the CD release pipeline is triggered and auto deploys to the dev environment.
I want the next stage to be triggered by the merge from dev to main,
so I can't see how I can use the same CD pipeline - I'd love to be shown that this is actually possible.
So for now, I'm going with 2 CI pipelines and 2 cd pipelines as follows:
dev-branch
CI (auto triggered by push to dev)
CD
dev-stage (auto triggered by CI above)
main-branch
CI (auto triggered by push to main)
CD
uat-stage (auto triggered by CI above)
prod-stage
The problem that I'd like to solve with this setup is :
can this be done with a single CI pipeline, suppressing the CD pipeline runs based on the source branch in git which triggered the build?
My CI pipeline yaml is as follows, I'd like to use this one file for both main and dev,
rather than having to duplicate it with the only difference being the individual triggers for dev and main.
trigger:
branches:
include:
- dev
- main
variables:
- group: devops-project-group
- template: app-variables.yml
pool:
vmImage: ubuntu-latest
steps:
- checkout: self
- bash: |
echo 'starting build ...'
you can use conditions:
- bash: |
echo 'starting MASTER build ...'
condition: and(succeeded(), variables['Build.SourceBranch'], 'refs/heads/master')
alternatively you can do ifs:
steps:
- ${{ if eq(variables['Build.SourceBranch'], 'refs/heads/master') }}:
- bash: master step1
- bash: master step2
etc
${{ else }}:
- bash: dev step1
- bash: dev step2
etc
-
or you can determine branch in the bash step and do something accordingly
Related
I have 3 environments in Azure , Sandbox, Test and Prod.
I have yaml pipeline in azure devops which builds the infrastructure. The environment built depends on the variables in the terraform code
The same pipleline is used to deploy to all environments depending on conditions in the yaml file.I want Dev to trigger on a merge to master but only want test and prod to deploy manually. How can i set this up in the yaml file?
For Dev, you can set up build validation of branch 'Master':
And in the Master branch, the YML file should be like this:
trigger:
- master
pool:
vmImage: ubuntu-latest
steps:
- script: echo Hello, world!
displayName: 'Run a one-line script'
This can make sure the pipeline of the master branch(Dev environment) only triggers after the PR merge is completed.
For test and prod environment, you can create a branch with the same name YML that the pipeline is looking for. And use the below YML definition:
trigger:
- none
pool:
vmImage: ubuntu-latest
steps:
- script: echo Hello, world!
displayName: 'Run a one-line script'
This can make sure the test and prod only can be triggered manually.
Above solution just need one pipeline.
Based on your requirement, you can add the condition in your YAML pipeline.
Dev to trigger on a merge to master
You can set the variable: Build.Reason and System.PullRequest.TargetBranch.
For example:
condition: and(eq(variables['Build.Reason'], 'PullRequest'), eq(variables['System.PullRequest.TargetBranch'], 'refs/heads/master'))
want test and prod to deploy manually
You can set the variable: Build.Reason
condition: eq(variables['Build.Reason'], 'Manual')
YAML example:
stages:
- stage: Dev
condition: and(eq(variables['Build.Reason'], 'PullRequest'), eq(variables['System.PullRequest.TargetBranch'], 'refs/heads/master'))
jobs:
- job: A
steps:
- xx
- stage: Prod
condition: eq(variables['Build.Reason'], 'Manual')
jobs:
- job: B
steps:
- xx
- stage: Test
condition: eq(variables['Build.Reason'], 'Manual')
jobs:
- job: C
steps:
- xx
When the pipeline is triggered by Pull Request and the target branch is master, the Dev stage will run.
When the pipeline is triggered manually, the Test and Prod stage will run.
Refer to this doc: Condition and Predefined variables.
I need to run some tasks if a pull request is opened. Im new to this so apologies .
eg A pull request is rasied on github.
If this happens I want to build some review apps based on whether the above condition is true
I need to do this using YAML
This is how pr trigger works. The below example defines two triggers. When a new pr is created for the develop branch then the pipeline will be triggered. This means that pr is open for develop branch until it merges. The second trigger will be for main branch. This means that when you merge or commit code on your main branch then the pipeline will be also triggered.
trigger:
branches:
include:
- 'main'
pr:
branches:
include:
- develop
https://learn.microsoft.com/en-us/azure/devops/pipelines/repos/azure-repos-git?view=azure-devops&tabs=yaml#pr-triggers
This worked for me
Starter pipeline
Start with a minimal pipeline that you can customize to build and deploy your code.
Add steps that build, run tests, deploy, and more:
https://aka.ms/yaml
pr:
branches:
include:
- '*'
#trigger:
#- main
pool:
vmImage: ubuntu-latest
steps:
- script: echo Hello, world!
displayName: 'Run a one-line script'
- script: |
echo Add other tasks to build, test, and deploy your project.
echo See https://aka.ms/yaml
displayName: 'Run a multi-line script'
- task: PowerShell#2
inputs:
targetType: 'inline'
script: |
# Write your PowerShell commands here.
Write-Host "pullrequest"
**Write-Host "PR Number is:-" $(System.PullRequest.PullRequestId)"
Write-Host "PR Number is:-" $(System.PullRequest.PullRequestNumber)"**
I defined the following stage in my azure devops release :
steps:
- bash: |
# Write your commands here
echo 'Hello world'
curl -X POST -H "Authorization: Bearer dapiXXXXXXXX" -d #conf/dbfs_api.json https://adb-YYYYYYYY.X.azuredatabricks.net/api/2.0/jobs/create > file.json
displayName: 'Bash Script'
my repo has a folder called conf with the file dbfs_api.json inside of it , unfortunately this file is not found during the deployment of this stage and I get the following error:
Couldn't read data from file "D:ar1a/conf/dbfs_api.json", this makes an empty POST.
The release stages of an Azure Pipelines workflow don't perform a checkout by default. You can manually add a checkout task to the release stage to perform a checkout:
A deployment job doesn't automatically clone the source repo. You can checkout the source repo within your job with checkout: self. Deployment jobs only support one checkout step.
jobs:
- deployment:
environment: 'prod'
strategy:
runOnce:
deploy:
steps:
- checkout: self
Or you can create a pipeline artifact in the build stage and consume the artifact in a later artifact.
stages:
- stage: build
jobs:
- job:
steps:
- publish: '$(Build.ArtifactStagingDirectory)/scripts'
artifact: drop
- stage: test
dependsOn: build
jobs:
- job:
steps:
- download: current
artifact: drop
See:
Use Artifacts across stages
Deployment jobs
When a pipeline is triggered by a repository resource trigger I need to be able to determine metadata about the triggering repository (such as repo name and branch).
It sounds like these variables should be set during a pipeline run that was triggered by a repository resource; however, their values are blank when I do echo $(resources.triggeringAlias) or env | sort in a pipeline run that was triggered by a repository resource.
UPDATE: the predefined variables for Build.Repository.Name and Build.SourceBranchName now work as expected when used with a repository trigger. So, while I'm no longer in need of resources.triggeringAlias and resources.triggeringCategory, they still aren't working.
I need to be able to determine metadata about the triggering
repository (such as repo name and branch)
We can use $(Build.Repository.Name) and $(Build.Repository.Uri) to get repo name and repo uri.
And $(Build.SourceVersion) can be used to get CommitId, $(Build.SourceBranch) or $(Build.SourceBranchName) can be used to get branch info.
Just note we must also checkout the triggering repo to make above variables work to fetch the info of triggering repo, otherwise those variables will always represents the value of triggered repo:
steps:
- checkout: self
- checkout: TheTriggeringRepo
Some details:
I have a triggering repo PipelineA, and the triggered repo PipelineB. PipelineB's azure-pipelines.yml file:
resources:
repositories:
- repository: PipelineA
type: git
name: PipelineA
trigger:
- master
steps:
- checkout: self
- checkout: PipelineA
- task: Bash#3
inputs:
targetType: 'inline'
script: |
echo $(Build.Repository.Name)
echo $(Build.SourceBranch)
So this pipeline will be triggered by both PipelineA repo and PipelineB repo.
When PipelineB repo has changes:
When PipelineA repo has changes:
It's clear the $(Build.Repository.Name) variable can work well to output the real trigger repo if we checkout both these two repos. So just make sure you checkout those triggering repos, then my variables above would work for you.
resources:
pipelines:
- pipeline: client-qa-deployment ## Resources.TriggeringAlias will be 'client-qa-deployment' if this resource gets triggered.
source: benjose.dev-code
trigger:
stages:
- client_deploy_qa
- stage: qa_smoketest
displayName: Smoke Test on QA Environment
condition: eq(variables['Resources.TriggeringAlias'], 'client-qa-deployment')
pool:
vmImage: 'windows-latest'
jobs:
- job: TestQA
steps:
- script: echo Hello, $(Resources.TriggeringAlias)!
displayName: 'Running Smoke Test on QA Environment'
Hello, client-staging-deployment!
I believe the above snippet will help you to understand the syntax and usage of Resources.TriggeringAlias. This variable will be helpful if we have multiple resources in the same pipeline and to find out which resource got trigged.
Problem
I have two repositories A and B in one project, each with their own pipeline A-CI and B-CI. The repos are Azure Repos Git (so not external ones). I got it working to trigger pipeline B-CI whenever A-CI has completed. If A-CI got triggered by a commit to branch develop, then B-CI is triggered to build master although B also has a develop branch.
I want to build a new release of B for the dev environment, when a new dev version of A has been built.
Is it possible to let a pipeline-resource trigger the B-CI pipeline to build the branch with the same name as the branch which was just built by the pipeline-resource? It would be fine for me if it would fallback to master in case a matching branch is not available in B.
This scenario is working however if A-C and B-CI would both refer to different pipeline yamls of the same repository.
Pipeline YAMLs
A-CI
trigger:
- '*'
stages:
- stage: Build
jobs:
- job: BuildJob
pool:
name: 'MyBuildPool'
steps:
- powershell: |
Write-Host "Building A"
B-CI
resources:
pipelines:
- pipeline: Pipeline_A
source: 'A-CI'
trigger:
branches:
- master
- develop
- feature/*
trigger:
- '*'
stages:
- stage: Build
jobs:
- job: BuildJob
pool:
name: MyBuildPool
steps:
- powershell: |
Write-Host $(Build.SourceBranch) # is always refs/heads/master
Write-Host $(Build.Reason) # is ResourceTrigger
Background Info
The main idea behind that is, that A contains the IaC project and whenever the infrastructure of the project changes, then all apps should be deployed, too.
I do not want to put the IaC into the app repo because we have multiple apps, so I would have to split the IaC code into several chunks.
And then I would probably still have the same problem because some resources, like Azure KeyVault, are shared among the apps so A would still include the common stuff used by all apps and changes to it would require re-deployments of all apps.
Please check pipeline triggers:
If the triggering pipeline and the triggered pipeline use the same repository, then both the pipelines will run using the same commit when one triggers the other. This is helpful if your first pipeline builds the code, and the second pipeline tests it.
However, if the two pipelines use different repositories, then the triggered pipeline will use the latest version of the code from its default branch.
In this case, since master if the default branch of your B-CI, $(Build.SourceBranch) is always refs/heads/master.
As a workaround:
You can create a new yaml pipeline for repository B. You can use similar content of the yaml file for B-CI. And you only need to change something in it to:
resources:
pipelines:
- pipeline: Pipeline_A
source: 'A-CI'
trigger:
branches:
- develop
When we create the new yaml file, it's always placed in master branch. For me, I created a file with same name in dev branch, and copy the same content in it. Then i delete the new yaml file in master branch, now when dev of A-CI pipeline is built, dev of B repos will be used.
I think I have found a nicely working workaround as the built-in pipeline-triggers are not addressing our specific problem (though I can't say if we have an odd approach and there are better ways).
What I am doing now it to use the Azure CLI DevOps extension based on this docs entry and trigger the pipelines manually.
Pipeline YAMLs
A-CI
trigger:
- '*'
stages:
- stage: Build
displayName: Build something and create a pipeline artifact
jobs:
- job: BuildJob
pool:
name: 'MyBuildPool'
steps:
- powershell: |
Write-Host "Building A"
# steps to publish artifact omitted
- stage: TriggerAppPipelines
displayName: Trigger App Pipeline
jobs:
- job: TriggerAppPipelinesJob
displayName: Trigger App Pipeline
steps:
- bash: az extension list | grep azure-devops
displayName: 'Ensure Azure CLI DevOps extension is installed'
- bash: |
echo ${AZURE_DEVOPS_CLI_PAT} | az devops login
az devops configure --defaults organization=https://dev.azure.com/MyOrg project="MyProject" --use-git-aliases true
displayName: 'Login Azure CLI'
env:
AZURE_DEVOPS_CLI_PAT: $(System.AccessToken)
# By passing the build Id of this A-CI run, I can use that in B-CI to download the matching artifact of A-CI.
# If there is no matching branch, then the command fails.
- bash: |
az pipelines run --branch $(Build.SourceBranch) --name "B-CI" --variables a_Version="$(Build.BuildId)" -o none
displayName: 'Trigger pipeline'
B-CI
trigger:
- '*'
stages:
- stage: Build
jobs:
- job: BuildJob
pool:
name: MyBuildPool
steps:
- powershell: |
Write-Host $(Build.SourceBranch) # is same as the the triggering A-CI branch
Write-Host $(Build.Reason) # B-CI is triggered manually but the user is Project Collection Build Service, so automated runs can be distinguished
As B-CI is triggered manually now, there is no need for a resource node anymore.