I am trying to build a YAML release pipeline in Azure DevOps
for that I have created multiple branches to keep environment-specific files
I created 4 pipelines for release as well:
Problem: Whenever I am making any changes in any branch, all the branch pipeline starts running. If I run an individual pipeline it works fine.
# Starter pipeline
# Start with a minimal pipeline that you can customize to build and deploy your code.
# Add steps that build, run tests, deploy, and more:
# https://aka.ms/yaml
trigger:
- acc
pool:
name: 'Agent'
steps:
- task: Kubernetes#1
displayName: 'Deploy on K8s Cluster'
inputs:
connectionType: 'Azure Resource Manager'
azureSubscriptionEndpoint: 'vs-aks-sc'
azureResourceGroup: 'azwe-rg-01'
kubernetesCluster: 'azwe-aks-01'
command: 'apply'
arguments: '-f $(System.DefaultWorkingDirectory)/kubernetes/acc.yaml'
Problem: Whenever I am making any changes in any branch, all the branch pipeline starts running.
If you just want to run the corresponding pipeline of the branch you modified, you need to make sure set the pipeline with the YAML file in the corresponding branch and set the correct branch trigger.
For example, for the Acc branch:
We need create a YAML file under the branch Feature/TestSample-acc with the branch trigger in the YAML file:
trigger:
branches:
include:
- Feature/TestSample-acc
Then create a pipeline with existing Azure pipeline YAML file:
New Pipeline-> Azure Repos Git(YAML)-> Select your repository-> Select existing Azure pipeline YAML file:
Now, this pipeline only triggered by the modification on the branch Feature/TestSample-acc:
You could set the same this for other branches, like
trigger:
branches:
include:
- Feature/TestSample-dev
Besides, if you do not want to control the trigger by the YAML file, you could override it by the UI trigger settings:
This option is disable by default, so that we could control the trigger in the YAML file directly.
If you enable it, you just need to add the Branch filters for just one branch:
If I am not understand your question correctly, please let me know for free that what you want to achieve.
You should check the CI trigger setting of the pipeline to only allow it to trigger on your wanted branches
Change CI Trigger
The answer by "Leo Liu-MSFT" is correct.
BUT you also need to make sure that on each branch the YAML file has a different name, otherwise a commit to a single branch triggers more than one pipeline build.
Related
I would like to have a master pipeline capable of running the pipelines of our system's individual components. I'd also like to be able to run any of those components' pipelines individually. Additionally, some of the component pipelines are configured using yaml, while others are using the classic approach. (I'm not sure if that figures into any possible solutions to this problem.) Those that are configured using yaml typically contain multiple jobs, and I'd need all of the jobs to run in those cases.
Using approach #2 recommended here, I tried the following:
jobs:
- job: build_and_deploy
displayName: Build and Deploy
cancelTimeoutInMinutes: 1
pool:
name: some-pool
steps:
- checkout: self
- template: component_one_pipeline.yml
- template: component_two_pipeline.yml
I receive an error for the following "unexpected values": trigger, resources, name, variables, and jobs. I'm guessing these aren't allowed in any yaml file referenced in the template step of another pipeline yaml file. As I mentioned above, though, I need these values in their files because we need to run the pipelines individually.
If possible, could someone point me in the direction of how to get this done?
EDIT: I have also tried the approach given here. I was thinking I'd have a master pipeline that essentially did nothing except serve as a trigger for all of the child pipelines that are supposed to run sequentially. Essentially, the child pipelines should subscribe to the master pipeline and run when it's done. I ended up with the following 2 files:
# master-pipeline.yml
trigger: none
pool:
name: some agent pool
steps:
- script: echo Running MASTER PIPELINE
displayName: 'Run master pipeline'
#child-pipeline.yml
trigger: none
#- testing-branch (tried these combinations trying to pick up master run)
#- main
pool:
name: some agent pool
resources:
pipelines:
- pipeline: testing_master_pipeline
source: TestingMasterPipeline
trigger: true
steps:
- script: echo Running CHILD PIPELINE 1
displayName: 'Run Child Pipeline 1'
Unfortunately, it's not working. I don't get any exceptions, but the child pipeline isn't running when I manually run the master pipeline. Any thoughts?
Thanks in advance.
The way that those approaches you linked work, and Azure DevOps build triggering works in general, is that a build completion can trigger another build, and you have to have the trigger in build to be triggered. So:
Yaml templates can't have things like triggers, so they won't really help you here (though you can of course split any of the individual pipelines to templates). Triggers are in the main yaml pipeline fail, which references the template-files. So you can't have a individual component pipelines as templates.
Yaml pipelines can be chained with the resources-declaration mentioned in the first link. The way this works is that the resource declaration is in the pipeline to be triggered, and you configure the conditions (like branch filters: https://learn.microsoft.com/en-us/azure/devops/pipelines/process/pipeline-triggers?view=azure-devops#branch-filters) to the pipeline to be triggered. For example, in your component pipeline you declare the master pipeline as resource, and set the conditions when the component pipeline will be triggered, like when the master pipeline is run against /release/* -branch. Or just set the trigger to true in order to trigger the component pipeline from any succesful run of the master pipeline. The component pipeline can still have its own pipeline triggers at the start of the pipeline declaration.
The classic build definitions can also be chained via edit build definition -> triggers -> build completion (see, for example, here: https://jpearson.blog/2019/03/27/chaining-builds-in-azure-devops/). This works the same way as with yaml pipelines; you configure the conditions for this the classic pipeline to trigger, so add the master pipeline as trigger to the component pipelines. Again, you can also set pipeline triggers for the component pipeline.
The limitation here is, that a classic pipeline can be triggered by an yaml pipeline, but not vice versa. A similar limitation in the yaml resources-declaration; they can't be triggered by a classic pipeline. If you need such triggering, or otherwise find the "native" triggers not to be enough, you can of course shoot an Azure DevOps API call in either type of pipeline to trigger any pipeline. See: https://blog.geralexgr.com/cloud/trigger-azure-devops-build-pipelines-using-rest-api, or just search for the azure devops rest api and associated blog posts that trigger the api with powershell, the rest api -task or by some other means.
As it turns out, I needed to set the pipelines' default branch to the one I was testing on for things to work correctly. The code in my original post works fine.
If you run into this problem and you're using a testing branch to test your pipelines, check out the information here on how to configure your pipeline to listen for triggers on your branch. Apparently a pipeline only listens for them on its default branch. Note: The example in the link uses the "classic" approach to pipeline configuration as an example, but you can reach the same page from a yaml configuration's edit screen by clicking the 3 dots on the right and selecting "Triggers."
Hope this helps someone.
I've two Azure DevOps CI pipelines
DataPipeline\Windows - Build
DataPipeline\TestPipeline
The second pipeline is a pipeline resource trigger. I've used this doc as a reference.
When a pull request is created (e.g. with dev branch as source branch and main branch as target branch, not merged yet) both these pipelines are triggered (by default configuration). The second pipeline may pick up old artefacts (this is a different part altogether). A second run for pipeline DataPipeline\TestPipeline is triggered automatically as soon as DataPipeline\Windows - Build completes for a commit in the dev branch. Is this possible? I've specified the branch name in the resources trigger. Is something wrong with the YAML configuration?
My expected behaviour is when a pull request is created for branch name: xyz (not main), DataPipeline\TestPipeline shouldn't be triggered again after DataPipeline\Windows - Build completes.
DataPipeline\TestPipeline.yml
trigger: none
resources:
pipelines:
- pipeline: DataPipeline
source: 'DataPipeline\Windows - Build'
branch: main
trigger:
branches:
include:
- main
name: TestPipeline_$(variable)
jobs:
// template based job
With the classic Azure DevOps release pipeline our release flow was very easy to setup.
We had a build pipeline running many times during the day. On success it deployed to our development environment. Every night the latest successful deployment to dev was released to our test environment (running automated tests for hours), before it deployed to UAT. But often we also need to deploy to test during the day, if we have a new change which needs to go directly into test or UAT. The classic pipelines allowed us to skip a stage, or deploy if the previous was only partly successful.
1) Development - automatic
2) Test - nightly or manually
3) UAT - nightly or manually
4) Staging - manual approval
5) Production - manual approval
With the multi-stage pipelines the same flow seems to be very difficult to do. At least when it comes to making it as a single deployment pipeline. The first part is fine. We can have our build trigger the development deployment. But how can we hold back the release to the test environment until 0:30am, while still retain the ability to also release it manually? If I created a separate test environment pipeline, then it could work if it had no triggers, but a schedule.
Same with the UAT, as we also need the flexibility to manually run UAT deployments, then it would also need to go into its own pipeline. Releases to our staging and production environment we "gate" with manual approvals, which is fine.
While this technically could work, if we split the deployment pipeline into multiple pipelines it really gets difficult to manage "a release". Not to say that it kind of goes against the whole multi-stage pipeline principle if we create a separate pipeline per stage.
But with this being so easy to setup like this in the classic pipelines, then I cannot really imaging that other companies have not run into the same limitations. Is it just me who cannot see the light, or cannot this really not be done with multi-stage pipelines?
manually run UAT deployments
We could add Azure DevOps Multi-Stage Pipelines Approval Strategies in the yaml build.
Steps:
Open the tab Environments and click the button New environment-> Click the button approvals and checks-> My environment name is TEST.
Then use it in the yaml pipeline(just a sample):
trigger: none
pool:
vmImage: 'ubuntu-latest'
stages:
- stage: A
jobs:
- deployment: 'MyDeployment'
displayName: MyDeployment
environment: 'TEST'
- job: A1
steps:
- script: echo "##vso[task.setvariable variable=skipsubsequent;isOutput=true]false"
name: printvar
- stage: B
condition: and(succeeded(), ne(stageDependencies.A.A1.outputs['printvar.skipsubsequent'], 'true'))
dependsOn: A
jobs:
- job: B1
steps:
- script: echo hello from Stage B
Result:
We could also configure schedule Trigger and use them in the multi-stage pipelines.
Note: The schedule trigger and Approval Strategies are using in the stage level.
For scheduled jobs: you can use something like this in your YAML:
(Copied from Microsoft documentation)
schedules:
- cron: string # cron syntax defining a schedule
displayName: string # friendly name given to a specific schedule
branches:
include: [ string ] # which branches the schedule applies to
exclude: [ string ] # which branches to exclude from the schedule
always: boolean # whether to always run the pipeline or only if there have been source code changes since the last successful scheduled run. The default is false.
For manual jobs, you can use the Create Release button to create and deploy a release manually. Do note that sometimes this can create a conflict with the schedule. Also, to "hold back a release" put an approver on the release, and then when approving, defer the release:
noting that it's in UTC, and it defaults to tomorrow - you can change it to any time after now.
In template pipelines you can't place any trigger statement such as trigger: none as specified in Microsoft's docs to disable ci trigger, so I wonder how do you prevent these pipelines from being executed every time you update them or any other yaml file in the same branch?
The pipeline yaml definition now supports disabling all trigers with
trigger: none
Reference
So in the end in a template pipeline you can't state something like trigger: none (to set only manual triggering) and you cannot specify stages or jobs, only steps are allowed (so you can't define any condition to prevent pipeline execution on the job or on the stage).
You have an option to disable the CI trigger by going in the triggers section for the template pipeline and select the following:
I don't like much this option because it means having pipeline configuration that is not captured in the yaml pipeline definition but I found no other way to disable the template pipeline from being triggered every time something (including the pipeline itself) is updated in its branch.
Other answers are fine.
Here is another approach I found. I have a yaml based build pipeline. I did not want to touch the yaml just for disabling the trigger. So I disabled the pipeline as follows.
Pick your pipeline and double click to edit it.
Go to the settings.
Now you should see an option to disable. Note here we are disabling the pipeline, not just disabling trigger to the pipeline.
First you need to go to your Pipelines dashboard, then select the pipeline that you want to disable CI for it
then Click on Edit
You will be redirected to the pipeline yaml file, from there you will click on the vertical ellipsis ⋮ => Triggers
and here you go, you can disable CI for your pipeline that easily
Save it, and you are done.
If you want to update your template without affecting pipelines which uses this template make a change on separate branch and merge it to master (or whatever you use) once you are sure that you have what you want.
The same applies if you use template from different repo:
# Repo: Contoso/WindowsProduct
# File: azure-pipelines.yml
resources:
repositories:
- repository: templates
type: github
name: Contoso/BuildTemplates
ref: refs/tags/v1.0 # optional ref to pin to
jobs:
- template: common.yml#templates # Template reference
parameters:
vmImage: 'vs2017-win2016'
By default you use template from your main branch, but you can point to any branch you want. Thus you can also test that your changes on limited pipelines as you need to point directly to a branch where you modified your template.
Let's say you have that branching:
master
|_ feature/add-extra-step
And you make a change in template but on feature/add-extra-step by adding additional step.
Now hen you trigger pipeline which uses that template:
trigger goes from master - your additional step in template won't be run
trigger goes from feature/add-extra-step - your additional step will be run
I made a change in template on feature/extra-step branch:
And this is change is not avialable when I run pipeline (even the same pipeline) on master:
If you for instance don't want to trigger ci build making changes on template, pleace commit those changes with phrase [skip ci] in th git message. Check here for more details.
I'm a bit confused with how to set this workflow up using pull requests.
I have in place an existing multi-stage YAML build pipeline that in summary does the following:
Runs a build from any branch
Runs a deployment job to a Development Environment/Resource, if the source branch is feature/* or release/* and the build succeeds
Runs a deployment job to a UAT Environment/Resource, if the source branch is release/*
Runs a deployment job to Live/Production Environment/Resource, if the source branch is master
So off the back of CI this workflow seems to work fine, the correct stages are run depending on the branch etc.
I then decided that branch policies and pull requests might be a better option for code quality rather than let any of the main branches be direct committed to, so I started to rework the YAML to account - mainly by removing the trigger to
trigger: none
This now works correctly in line with a branch policy, the build only gets kicked off when a pull request on to develop or master is opened.
However this is then where I'm a bit confused with how this is supposed to work and how I think it works ....
Firstly - is it not possible to trigger the multi-stage YAML off the back of pull requests (using Azure Repos) ? In my head all I want to do is introduce the pull request and branch policies but keep the multi-stage deployments to environments as is.
However, the deployment job stages all get skipped now - but this might be to do with my conditions within the YAML, which are as follows:
- stage: 'Dev'
displayName: 'Development Deployment'
dependsOn: 'Build'
condition: |
and
(
succeeded()
eq(variables['Build.Reason'], 'PullRequest'),
ne(variables['System.PullRequest.PullRequestId'], 'Null'),
or
(
startsWith(variables['Build.SourceBranch'], 'refs/heads/feature/'),
startsWith(variables['Build.SourceBranch'], 'refs/heads/release/')
)
)
jobs:
- deployment: Deploy
pool:
name: 'Development Server Agent Pool'
variables:
Parameters.WebsitePhysicalPath: '%SystemDrive%\inetpub\wwwroot\App'
Parameters.VirtualPathForApplication: ''
Parameters.VirtualApplication: ''
environment: 'Development.Resource-Name'
.....
Is there something I am missing?
Or do I have to remove the multi-stage deployments from YAML and revert back to using Release Pipelines for pull requests (maybe with approval gates??)
Thanks in advance!
It looks like the issue of the conditions within the YAML.
If the pipeline is triggered by a PR. the value variables['Build.SourceBranch'] will be refs/pull/<PR id>/merge. The express in above condtion startsWith(variables['Build.SourceBranch'], 'refs/heads/feature/') will be false, which caused the stage to be skipped. See build variables for more information.
You can try using variables['System.PullRequest.SourceBranch'], which will be evaluated to the value of the source Branch of the PR. Check System variables for more information. See below:
condition: |
and
(
succeeded(),
eq(variables['Build.Reason'], 'PullRequest'),
ne(variables['System.PullRequest.PullRequestId'], ''),
or
(
startsWith(variables['System.PullRequest.SourceBranch'], 'refs/heads/feature/'),
startsWith(variables['System.PullRequest.SourceBranch'], 'refs/heads/release/')
)
)