We have a YAML based pipeline that Unit Tests and build an ASP NET Core website then if everything is OK it deploys to DEV, TEST and eventually Live Azure Resources.
Our source control is Git within Azure Dev Ops.
Our process has us working in a branch for each feature, once those branches are ready we merge them into a "release" branch for an integration test before being PR'ed to MAIN. An example of our release branch would be "release_3_1_5".
The start of the YAML pipeline looks like this
pool:
vmImage: 'windows-latest'
# Why would I want 'resources'
# resources:
# pipelines:
# - pipeline:
variables:
azureSubscriptionEndpoint: 'ARM Service Connection'
webAppKind: webApp
solution: '**/*.sln'
buildPlatform: 'Any CPU'
buildConfiguration: 'Release'
# Change this when adding functionality that is a breaking change
majorVersion: 3
# Change this when adding functionality that is backwards compatible
minorVersion: 1
# Change this when making fixes that are backwards compatible and not adding functionality
buildVersion: 0
# Concatenate version parts and buildId to get a full build version string
fullBuildVersionString: $(majorVersion).$(minorVersion).$(buildVersion).$(Build.BuildID)
name: $(MajorVersion).$(MinorVersion).$(buildVersion).$(Build.BuildID)
stages:
- stage: Build
jobs:
- job: Build_Job
steps:
- bash: |
echo $(fullBuildVersionString)
We don't specify any explicit triggers so the build runs everytime we push to a branch.
The "MAIN" branch has some branch policies set, those policies include "Build Validation" and currently the Build Validation build policy is configured to run the same YAML pipeline.
The CI pipeline works just fine when pushing changes to our branches, except when the branch in question is the subject\source of a PR to MAIN. In this situation the pipeline starts twice. Once for the push to the "release" branch and once by the branch policy because of the PR into MAIN.
Is there a better way to configure the pipeline so it does not kick off twice? I basically do not want the CI truigger to fire when the branch is the source of a PR to MAIN but that looks like an impossible condition
This is something we struggle with as well. We have just accepted the double builds for now. However, I am starting to consider not having a build trigger for feature/ branches and only trigger for PRs.
The only other option are double manifests. One manifest for branches that are not MAIN, and the other being a manifest that includes only PRs and the MAIN branch.
If you want builds to run for branches, you could consider pre-receive hooks that requires builds to run locally.
Related
Like many before me I'm struggling hard with configuring pipeline triggers in Azure DevOps.
Background:
single project in the Organization
three branches: main, Infrastructure, Application
The branches are kind of independent of each other. They are never merged into main either.
I have a pipeline which deploys two App Services. The YAML file for this pipeline is in the Infrastructure branch. The Default branch for manual and scheduled builds is set to Infrastructure.
Then I have 2 pipelines, each to deploy a different App to the App Service. The YAMLs for those pipelines are in the Application branch. The Default branch for manual and scheduled builds is set to Application.
By themselves, the pipelines work perfectly fine. However what I am trying to achieve is to trigger the App pipelines after the App Service pipeline finishes. And no matter what combination of settings I try, I can't get it to work.
This is currently how it looks like in the n-th version of the YAML:
name: 'deploy-webapp-002'
pool:
vmImage: windows-latest
resources:
pipelines:
- pipeline: 'Deploy App Services' # Internal name of the source pipeline, used elsewhere within this YAML
# e.g. to reference published artifacts
source: deploy-appservices # Azure Pipelines name of the source pipeline referenced
project: HomeLab # Required only if the source pipeline is in another project
trigger:
branches:
include:
- Infrastructure
- Application
pr: none
trigger: none
Is it even possible to do what I'm trying to do?
If yes, what settings should be specified in the Resources/Pipelines section in the YAML, and how should the Default branch for manual and scheduled builds look like for each of those pipelines?
I can reproduce the same issue when I put the YAML files separately in two branches and set the default branch.
Refer to this doc: Define a pipelines resource
When you define a resource trigger, if its pipeline resource is from the same repository (say self) as the current pipeline, triggering follows the same branch and commit on which the event is raised. But, if the pipeline resource is from a different repository, the current pipeline triggers on the default branch of the self repository.
In your caseļ¼ you are using the same repo. So triggering follows the same branch and commit on which the event is raised.
To solve this issue, you need to copy the YAML file in Application Branch to Infrastructure Branch.
On the other hand, you can also try to set Build completion trigger manually on UI.
For example:
I have 3 Yaml Pipelines:
CI1 that should be build on any commit to services/* branches
CI2 that should be build on any commit to services-release/* branches
CD that should do the deployment of the artifacts created by CI2
CD is setup the following way:
YAML Settings
Triggers Settings
As you can see, I've tried different formats of the branches to branch filters. Even if I add non-wildcard filter, I still see the following behavior:
CD pipeline is triggered after CI2, triggered by commit into services-release/* branch (which is correct), but it releases latest build from a branch, specified in "default branch for manual and scheduled builds" dropdown - which is develop in my case.
What should I change to deploy the artifact that was generated by CI2 build from services-release/* branch?
Judging by the pictures, you are using yaml-pipelines but classic pipeline triggers. While this works for triggering the pipelines, you might want to consider implementing the triggers in yaml files for C1- and C2-pipelines.
https://learn.microsoft.com/en-us/azure/devops/pipelines/repos/azure-repos-git?view=azure-devops&tabs=yaml#ci-triggers
As for the yaml-based CD pipeline, in order for the pipeline to trigger upon completion of CI2 and for it to download artifacts from the triggering run (instead of latest from the default pipeline), you should reference the CI2-pipeline as resource:
https://learn.microsoft.com/en-us/azure/devops/pipelines/process/pipeline-triggers?view=azure-devops
So something like:
resources:
pipelines:
- pipeline: ci2_pipeline #this is used to reference this resource in CD pipeline
source: CI2 #Rename this to match your build pipeline name
trigger:
branches:
- services-release/*
For the artifacts, you want to ensure that you are using Pipeline Artifacts instead of classic build artifacts and use the Download Pipeline Artifacts -task in CD-pipeline (https://learn.microsoft.com/en-us/azure/devops/pipelines/tasks/utility/download-pipeline-artifact?view=azure-devops). So something like:
steps:
- download: none # Disable downloading default artifacts
- task: DownloadPipelineArtifact#2
inputs:
buildType: "specific"
project: "$(resources.pipeline.ci2_pipeline.projectID)"
definition: "$(resources.pipeline.ci2_pipeline.pipelineID)"
preferTriggeringPipeline: true
buildVersionToDownload: "latestFromBranch"
branchName: "$(Build.SourceBranch)"
targetPath: "$(Pipeline.Workspace)"
In ADO, you can create a "repository resource" per this documentation. The "trigger" section allows you to define a CI trigger for any Azure repo in your space. Therefore, given the following config:
Repos:
AzureRepo1 - Contains project files that should be built
AzureRepo2 - Contains pipeline file 'pipeline.yml'
resources:
repositories:
- repository: "Azure_Repo_1"
type: git
name: AzureRepo1
ref: development
trigger:
branches:
include:
- development
- staging
- production
pool:
vmImage: 'windows-latest'
jobs:
- template: Template.yml
parameters:
service: "development"
run_tests: true
When I make a change to AzureRepo1, the pipeline triggers. At runtime, how would I determine which branch ("production", "staging", or "development") of the target repo (AzureRepo1) triggered the build? Ideally, the "service" parameter being fed into the example template would dynamically reflect which branch triggered the build.
Note: "Build.SourceBranch" and "Build.SourceBranchName" seem to pull the branch from the repo that hosts the YML file (in this case, AzureRepo2).
I was wrong. These function as intended. Use the below solution.
According to documentation here:
When an update to one of the repositories triggers a pipeline, then the following variables are set based on triggering repository:
Build.Repository.ID
Build.Repository.Name
Build.Repository.Provider
Build.Repository.Uri
Build.SourceBranch
Build.SourceBranchName
Build.SourceVersion
Build.SourceVersionMessage
For the triggering repository, the commit that triggered the pipeline determines the version of the code that is checked out. For other repositories, the ref defined in the YAML for that repository resource determines the default version that is checked out.
If triggers happens on AzureRepos1 you should have correct branch name in Build.SourceBranchName
I have a release pipeline with the usual stages
Dev -> Test -> Pre Production -> Production
I would like to know if there is a way to ensure that only the master branch can be promoted all the way to production to avoid releases from feature and develop branches ending on a production environment.
YAML Pipelines
If you're using a YAML-based pipeline, you can use the Checks + Approval Gates feature to add a branch policy check.
Create an Environment for each deployment environment
Structure your YAML pipeline to use a Deployment Job that specifies the Environment
stages:
- stage: build
- stage: dev
- stage: test
- stage: preprod
- stage: prod
You can put conditions on the stages to ensure the branch conditions are met.
- stage: production
dependsOn: preprod
condition: |
and(
succeeded('preprod'),
or(
eq(variables['Build.SourceBranch'], 'refs/heads/main'),
startsWith(variables['Build.SourceBranch'], 'refs/heads/release/'),
startsWith(variables['Build.SourceBranch'], 'refs/heads/hotfix/')
)
)
Ensure each stage that performs the deployment activities has a Deployment Job:
- deployment: deploy
displayName: 'Deploy'
environment: 'Production'
strategy:
runonce:
deploy:
steps:
- pwsh: Write-Host 'Deployment Steps go here!'
In your Production Environment, add the Branch Control check:
Classic Release Pipelines
If you're using a Classic Release pipeline, you can add artifact filters to prevent the pipeline from being queued, but these do not enforce that the Stage cannot be run manually or override these filters.
If you truly want to prevent the release, even if someone manually overrides the artifact filter, you could fail the build by checking that it is the correct branch in a script.
$currentBranch = "$(Release.Artifacts.MyAlias.SourceBranch)"
if ( $currentBranch -ne "refs/heads/main" )
{
Write-Host "##vso[task.logissue type=error]This release must originate from the main branch. Current branch: $currentBranch"
Exit 1
}
You can set a branch filter on the Continuous deployment Trigger, See below:
Click the highlighted trigger icon in the Artifacts section to open the trigger panel--> Enabled Continuous deployment Trigger--> Set the branch filter to only include master branch.
When you set your release pipeline Continuous deployment Trigger as above. Only the artifacts generated from master branch can trigger the release pipeline.
You can also set an artifact filter for each stage to make sure only artifacts come from a specific branch can be deployed to this stage. See below:
You can check out this document to learn more about classic CD pipeline.
#paddingtonMike,
This is a core feature of ADO Release Pipelines, you can complete this many ways; however, the most common is by modifying the source of your artifact.
I highly recommend that you invest some time in learning ADO Release Pipelines. Here is a great resource to get started with:
Learn - Create Release Pipeline
I don't know of a way to prevent a deployment from being created from a non-main branch. That said, Levi's answer will prevent a release from being created or deployed automatically. In addition to that, you can set up required reviewers on deployments which will guarantee no release can go out without the approval of a person who is authorized to deploy.
(Specific section of the link Levi shared) https://learn.microsoft.com/en-us/azure/devops/pipelines/release/define-multistage-release-process?view=azure-devops#add-pre-deployment-approvals
Problem
Azure DevOps has a feature (documented here) to trigger a pipeline on completion from another pipeline.
This works fine in a test organization, but it won't work in our main organization.
There could be something on the organization, project, repository or even branching level, but I'm currently stuck and any help would be appreciated!
Pipelines
Pipeline Pipeline B should run automatically when pipeline Pipeline A completes.
File pipeline-a.yaml for Pipeline A:
pool:
vmImage: 'ubuntu-latest'
steps:
- script: echo Hello, world!
displayName: 'Do something'
File pipeline-b.yaml for Pipeline B:
trigger: none
pool:
vmImage: 'ubuntu-latest'
resources:
pipelines:
- pipeline: pipeline-a
source: 'Pipeline A'
branch: master
trigger:
branches:
- master
steps:
- script: echo Hello, world!
displayName: 'Do something'
Organizations
In my test organization the above pipelines run like a charm. This means that Pipeline A runs on a commit, and after completion, Pipeline B runs automatically.
Yet in our production organization, Pipeline B does not run automatically.
Discovery
Both pipelines run fine when started manually, in both organizations
All Preview features are equal on organization and personal level for both organizations, including the Multi-stage pipelines feature.
The production organization has branch policies on master, while the test organization does not have policies. I don't see a connection with pipeline triggers and did not investigate this.
Installing extensions to have them equal on test and production does not make a difference.
The test organization seems to be in the slow ring and was still on Sprint 161. EDIT: The issue persists after the organization was updated to Sprint 162.
It works when I use the classic editor and manually create a build completion trigger. But this overrides the YAML pipeline trigger and I don't want to do this (I want to generate the pipeline and it's triggers)
Deleting and re-adding the pipeline did the trick. So keep the YAML file but delete the pipeline and add it again.
The Azure DevOps backend seems to miss a relationship between pipelines now and then.
We troubleshot a similar problem today. With Pipeline-A defined as a resource that is meant to be consumed by Pipeline-B.
The consuming pipeline was never being triggered. Deleting and recreating the pipeline did not work for us. This work\pipeline was net new and on a feature branch. That ended up being important.
The ultimate fix was defining that feature branch as the Default branch for manual and scheduled builds in Pipeline-B. You can find that setting tucked away in Pipeline -> Edit -> triggers -> yaml-> Get Sources. Expect that as we promote this code to the main branch we will need to update the setting.
So it seems like the Default branch for manual and scheduled builds would be better named
Default branch for manual and scheduled builds and Pipeline Completion Triggers
In my case it was as simple as the source pipeline completing with an error. Testing with a very simple non erroring pipeline and the code worked fine.