I have 2 basic pipelines.
pipeline 1 and pipeline 2.
I have read the following article below
https://learn.microsoft.com/en-us/azure/devops/pipelines/process/resources?view=azure-devops&tabs=schema#change-default-branch-for-triggers-optional
In a summary, I put the following in my the following in my pipeline2 to listen to pipeline1 execution.
# azure-pipelines-trigger.yml
name: Pipeline2
trigger: none
# this pipeline will be triggered by another pipeline
resources:
pipelines:
- pipeline: Pipeline1
project: DevOpsProject
source: Pipeline1
trigger:
branches:
include:
- releases/*
- main
exclude:
- topic/*
tags:
- Verified
- Signed
stages:
- Production
- PreProduction
pool:
vmImage: ubuntu-latest
stages:
- stage: Dev
displayName: 'Build_Deploy'
jobs:
- job: 'Build_Deploy'
steps:
- script: echo This pipeline was set to be triggered after first pipeline completes.
- stage: Prod
displayName: 'Build_Deploy'
jobs:
- job: 'Build_Deploy'
steps:
- script: echo This pipeline was set to be triggered after first pipeline completes.
Therefore, the default branch of pipeline 2 would trigger when Pipeline1 production and preproduction has launched and finished. The requirement is more complex than that. The requirements are:
Pipeline 2 would only listen to the trigger if it's in release/xxxx branch, not the default one/ master branch.
We don't want the trigger to create another pipeline run. The trigger is only meant to listen to kick off the Prod stage of pipeline 2.
any thoughts? Thanks.
Related
My issue
Yaml Devops propose the extends bloc. This is the kind of sample one use:
# main.yaml
trigger: none
extends:
template: ./extends-pipeline1.yaml
And:
# extends-pipeline1.yaml
stages:
- stage: Build
jobs:
- job:
steps:
- script: echo Build solution
- stage: Deploy
jobs:
- job:
steps:
- script: echo Deploy solution
Works fine, but this works as well without extends:
# main.yaml
trigger: none
stages:
- template: extends-pipeline1.yaml
What I did
I read the release note:
Release Note
My question
What is the benefit to use extends?
Thanks
What is the benefit to use extends?
There are two benefits of this feature.
First, 'extends' is more free than 'stages'.
When you use stages, then the pipeline will expect stages in the template, you must write the stages part in the template, otherwise the validation step will not be passed.
For example, when you use stages, your pipeline must be like this:
azure-pipelines.yml
trigger:
- none
pool:
vmImage: ubuntu-latest
stages:
- template: extends-pipeline1.yaml
extends-pipeline1.yaml
stages: #You must have this section in the template.
- stage: Build
jobs:
- job:
steps:
- script: echo Build solution
- stage: Deploy
jobs:
- job:
steps:
- script: echo Deploy solution
But if you use extends, your pipeline could be like this instead of must writing stages section in template:
azure-pipelines.yml
trigger:
- none
pool:
vmImage: ubuntu-latest
extends:
template: extends-pipeline1.yaml
extends-pipeline1.yaml
jobs:
- job:
steps:
- script: echo Build solution
Second, 'extends' has a feature named Required template.(Daniel had already mentioned this.)
If you set the 'required template' of the service connection, then the service connection will only been able to use in specific YAML template file which been 'extends'.
No matter directly use the service connection via main YAML file or use the service connection in the YAML file referred via 'stages'-template will both make the pipeline failed.
This feature can help you increase the security and this feature is exclusive to the 'extends' feature.
How can i ensure that all stages of my pipelines are performed in the same working directory.
I have pipeline that looks like this:
resources:
repositories:
- repository: AzureRepoDatagovernance
type: git
name: DIF_data_governance
ref: develop
trigger:
branches:
include:
- main
paths:
include:
- terraform/DIF
variables:
- group: PRD_new_resources
- name: initial_deployment
value: false
pool: $(agent_pool_name)
stages:
- stage: VariableCheck
jobs:
- job: VariableMerge
steps:
- checkout: self
- checkout: AzureRepoDatagovernance
- ${{ if eq(variables.initial_deployment, 'false') }}:
- task: PythonScript#0
inputs:
scriptSource: filePath
scriptPath: DIF-devops/config/dynamic_containers.py
pythonInterpreter: /usr/bin/python3
arguments: --automount-path $(System.DefaultWorkingDirectory)/DIF_data_governance/data_ingestion_framework/$(env)/AutoMount_Config.json --variables-path $(System.DefaultWorkingDirectory)/DIF-devops/terraform/DIF/DIF.tfvars.json
displayName: "Adjust container names in variables.tf.json"
- stage: Plan
jobs:
- job: Plan
steps:
- checkout: self
- checkout: AzureRepoDatagovernance
- script: |
cd $(System.DefaultWorkingDirectory)$(terraform_folder_name) && ls -lah
terraform init
terraform plan -out=outfile -var-file=DIF.tfvars.json
displayName: "Plan infrastructure changes to $(terraform_folder_name) environment"
- stage: ManualCheck
jobs:
- job: ManualCheck
pool: server
steps:
- task: ManualValidation#0
timeoutInMinutes: 5
displayName: "Validate the configuration changes"
- stage: Apply
jobs:
- job: Apply
steps:
- checkout: self
- checkout: AzureRepoDatagovernance
- script: |
cd $(System.DefaultWorkingDirectory)$(terraform_folder_name) && ls -lah
terraform apply -auto-approve "outfile"
displayName: "Apply infrastructure changes to $(terraform_folder_name) environment"
How can I make sure that all 4 stages are inside this same working directory so I can check out just once and all stages have access to work done by previous jobs? I know that this
I know that my pipeline has some flaws that will need to be polished.
This is not possible. Each azure devops stage has its own working directory and it is considered a different devops agent job. The jobs inside the stage will use the same working directory for the steps that are included on them.
If you need to pass code or artifacts between stages you should use publish pipeline artifacts and download pipeline artifacts native devops tasks.
I'm trying to implement Approval gates in an Azure Devops YML pipeline and following these steps. An issue I am seeing is that since configuring the pipeline to use a blank environment (so it is only used for the approval process as per the article) when the pipeline runs I am no longer seeing the "Checkout" step where my repo contents are loaded into the virtual machine runner used by the pipeline. How can I use an environment and load in the repo contents?
This code has the checkout step:
trigger: none
pool:
vmImage: ubuntu-latest
stages:
- stage: tempname
jobs:
- job: tempname
steps:
- bash: |
echo hello world
This code does not have the checkout step:
trigger: none
pool:
vmImage: ubuntu-latest
stages:
- stage: tempname
jobs:
- deployment: tempname
environment: test
strategy:
runOnce:
deploy:
steps:
- bash: |
echo hello world
checkout is a step, so should go in the steps section - see https://learn.microsoft.com/en-us/azure/devops/pipelines/yaml-schema/steps-checkout?view=azure-pipelines.
Example of how it might look:
trigger: none
pool:
vmImage: ubuntu-latest
stages:
- stage: tempname
jobs:
- deployment: tempname
environment: test
strategy:
runOnce:
deploy:
steps:
- checkout: self # Use none to avoid checking out code.
- bash: |
echo hello world
In Azure Devops multistage YAML pipeline we got multiple environments.
In stages to run normally we do a build and deploy only in QA, so we need to deselect each stage manually. By default all stages are selected is it possible to have exact opposite, where all stages are deselected by default???
trigger: none
pr: none
stages:
- stage: 'Build'
jobs:
- deployment: 'Build'
pool:
name: Default
# testing
environment: INT
strategy:
runOnce:
deploy:
steps:
- checkout: none
- powershell: |
echo "Hello Testing"
Start-Sleep -Seconds 10
- stage: 'Sandbox'
jobs:
- job: 'Sandbox'
pool:
name: Default
steps:
- checkout: none
# testing
- powershell: |
echo "Hello Testing"
- stage: 'Test'
jobs:
- job: 'DEV'
pool:
name: Default
steps:
- checkout: none
- powershell: |
echo "Hello Testing"
- stage: 'QA'
dependsOn: ['Test','Test1','Test2']
jobs:
- job: 'QA'
pool:
name: Default
steps:
- checkout: none
# Testing
- powershell: |
echo "Hello Testing"
I am afraid that there is no UI (like stage to run) method that can meet your needs.
You could try to add parameters to your Yaml Sample.
Here is an example:
trigger: none
pr: none
parameters:
- name: stageTest
displayName: Run Stage test
type: boolean
default: false
- name: stageBuild
displayName: Run Stage build
type: boolean
default: false
stages:
- ${{ if eq(parameters.stageBuild, true) }}:
- stage: 'Build'
jobs:
- deployment: 'Build'
pool:
name: Default
environment: INT
strategy:
runOnce:
deploy:
steps:
- checkout: none
- powershell: |
echo "Hello Testing"
Start-Sleep -Seconds 10
- ${{ if eq(parameters.stageTest, true) }}:
- stage: Test
dependsOn: []
jobs:
- job: B1
steps:
- script: echo "B1"
The parameters are used to determine whether to run these stages. You could add expressions before the stage to check if the parameter value could meet expression.
The default value is false. This means that the stage will not run by default.
Here is the result:
You can select the stage you need to run by clicking the selection box.
Update
Workaround has some limitations. When the select stage has depenencies, you need to select all dependent stages to run.
For example:
- stage: 'QA'
dependsOn: ['Test','Test1','Test2']
On the other hand, I have created a suggestion ticket to report this feature request. Here is the suggestion ticket link: Pipeline Deselect Stages By Default You could vote and add comment in it .
I've used this solution to build a nuget-package, and:
always push packages from master
conditionally push packages from other branches
Using GitVersion ensures that the packages from other branches get prerelease version numbers, e.g. 2.2.12-my-branch-name.3 or 2.2.12-PullRequest7803.4. The main branch simply gets 2.2.12, so the master branch is recognized as a "regular" version.
The reason I'm repeating the answer above, is that I chose to make the stage conditional instead of using an if:
trigger:
- master
parameters:
- name: pushPackage
displayName: Push the NuGet package
type: boolean
default: false
stages:
- stage: Build
jobs:
- job: DoBuild
steps:
- script: echo "I'm building a NuGet package (versioned with GitVersion)"
- stage: Push
condition: and(succeeded('build'), or(eq('${{ parameters.pushPackage }}', true), eq(variables['build.sourceBranch'], 'refs/heads/master')))
jobs:
- job: DoPush
steps:
- script: echo "I'm pushing the NuGet package"
Like the other answer, this results in a dialog:
But what's different from the (equally valid) solution with '${{ if }}', is that the stage is always shown (even if it's skipped):
Question: how do you setup CI/CD in YAML pipelines for following context.
branches
master
release/{ALPHABETICAL NAME} ex. release/Albert next release is release/Bertrand and so on.
environments
accept: everything that's pushed on master
test: latest release ex. release/Bertrand
sandbox: latest release -1 (here we can test hotfixes) ex. release/Albert
live: latest release -1 (with hotfixes)
Closest solution
build: creates project artifacts
build.yml
trigger:
- master
- release/*
pool:
vmImage: 'ubuntu-latest'
steps:
- powershell: |
New-Item -Path . -Name "testfile1.txt" -ItemType "file" -Value "This is a text string."
- publish: $(Pipeline.workspace)
artifact: testArtifact
release-phase1: deploys master branch to accept
release-phase1.yml
trigger: none
resources:
pipelines:
- pipeline: pipelineId
source: build
trigger:
branches:
- master
pool:
vmImage: 'ubuntu-latest'
jobs:
- deployment: DeployWeb
environment: 'testenvironment'
strategy:
runOnce:
deploy:
steps:
- script: echo FOO
release-phase2: deploys release branch to test
release-phase2.yml
trigger: none
resources:
pipelines:
- pipeline: pipelineId
source: build
trigger:
branches:
- release/current
pool:
vmImage: 'ubuntu-latest'
jobs:
- deployment: DeployWeb
environment: 'testenvironment'
strategy:
runOnce:
deploy:
steps:
- script: echo FOO
release-phase3: deploys release-1 branch to sandbox and after manual approval to live
release-phase3.yml
trigger: none
resources:
pipelines:
- pipeline: pipelineId
source: build
trigger:
branches:
- release/previous
pool:
vmImage: 'ubuntu-latest'
jobs:
- deployment: DeployWeb
environment: 'testenvironment'
strategy:
runOnce:
deploy:
steps:
- script: echo FOO
Reasons why this solutions doesn't fulfill our needs:
the names of the release branches aren't static.
we should be able to run release-phase3.yml pipeline without running a build on this branch firts. It should download artifacts from the latest build of that branch. Which is not the case.
SHORT ON PURPOSE
Since you have multiple branches (master and releases branches), different branch is built and deploy to different environment. So you can try having the CI build yaml pipeline in each branch and put the CD deployment yaml pipeline in a template yaml in master branch.(You have to have the build yaml file in each branch to get the code in this branch built. You can check this thread).
Below is a simple example.
In master branch
There are azure-pipelines.yml and a template-deploy.yml. In azure-pipelines.yml the Environment value will be passed as a parameter to template-deploy.yml. So that the build will be deployed to its corresponding environment.
azure-pipelines.yml:
trigger:
- master
- release/*
pool:
vmImage: 'windows-latest'
resources:
repositories:
- repository: deploy
type: git
name: {project name}
jobs:
- job: Build
steps:
- script: echo "start build job"
- template: template-deploy.yml#deploy
parameters:
envir: "prod"
template-deploy.yml:
parameters:
envir: ""
jobs:
- deployment: DeployWeb
environment: '${{parameters.envir}}'
strategy:
runOnce:
deploy:
steps:
- script: echo FOO
In the release branches
You can define its individual ci build yaml like below example:
azure-pipelines.yml in release-phase2 branch:
pool:
vmImage: 'windows-latest'
resources:
repositories:
- repository: deploy
type: git
name: {project name}
jobs:
- job: Build
steps:
- script: echo "start build job"
- template: template-deploy.yml#deploy
parameters:
envir: "test"