Azure DevOps pipeline repository trigger doesn't fire - azure-devops

Context
I'm creating a CI/CD configuration for an application having this repository configuration (each repository in the same Organization and Project):
Frontend repository (r1)
API Service repository (r2)
Infrastructure As Code repo (r3)
Within the repository r3 there are the solution's Azure DevOps Pipelines, each one of them has been configured for Manual & Scheduled trigger on develop branch:
Frontend CI Pipeline p1
Backend CI Pipeline p2
Deployment Pipeline p3
The behavior I want is
Git commit on r1 repo
Pipeline p1 on repo r3 triggered (this will create artifacts, apply a tag and notify)
Pipeline p3 triggered by p1 completion (this will deploy the artifacts)
Pipeline p1 looks like the following
trigger: none
resources:
containers:
- container: running-image
image: ubuntu:latest
options: "-v /usr/bin/sudo:/usr/bin/sudo -v /usr/lib/sudo/libsudo_util.so.0:/usr/lib/sudo/libsudo_util.so.0 -v /usr/lib/sudo/sudoers.so:/usr/lib/sudo/sudoers.so -v /etc/sudoers:/etc/sudoers"
repositories:
- repository: frontend
name: r1
type: git
ref: develop
trigger:
branches:
include:
- develop
exclude:
- main
name: $(SourceBranchName)_$(date:yyyyMMdd)$(rev:.r) - Frontend App [CI]
variables:
- name: imageName
value: fronted-app
- name: containerRegistryConnection
value: apps-registry-connection
pool:
vmImage: "ubuntu-latest"
stages:
- stage: Build
displayName: Build and push
jobs:
- job: JobBuild
displayName: Build job
container: running-image
steps:
- checkout: frontend
displayName: Checkout Frontend repository
path: fe
persistCredentials: true
...
Pipeline p3 looks like the following
name: $(SourceBranchName)_$(date:yyyyMMdd)$(rev:.r) - App [CD]
trigger: none
resources:
containers:
- container: running-image
image: ubuntu:latest
options: "-v /usr/bin/sudo:/usr/bin/sudo -v /usr/lib/sudo/libsudo_util.so.0:/usr/lib/sudo/libsudo_util.so.0 -v /usr/lib/sudo/sudoers.so:/usr/lib/sudo/sudoers.so -v /etc/sudoers:/etc/sudoers"
pipelines:
- pipeline: app-fe-delivery
source: "p1"
trigger:
stages:
- Build
branches:
include:
- develop
pool:
vmImage: "ubuntu-latest"
stages:
- stage: Delivery
jobs:
- job: JobDevelopment
steps:
- template: ../templates/template-setup.yaml # Template reference
parameters:
serviceDisplayName: ${{ variables.serviceDisplayName }}
serviceName: ${{ variables.serviceName }}
...
Issue
Even if followed step by step all the rules exposed in the official documentation:
Pipeline p1 is never triggered by any commit on develop branch in r1 repository
Even if manually run Pipeline p1, Pipeline p3 is never triggered
Remarks
As stated in the pipelines YAML reference, Triggers are enabled by default
in the same documentation, if no branch include filter is expressed, the trigger will happen on all branches
As stated in the triggers for Checkout Multiple repositories in pipelines triggers happens only for repos in Azure DevOps repositories
is it possible to disable pipeline CI triggers (trigger: none) and have resource's repositories triggers happening
Build agent user has been authorized to access and queue new builds

Couple possible solutions.
First off believe your issue is with:
trigger: none
This means the pipeline will only work manually. In the documentation you referenced :
Triggers are enabled by default on all the resources. However, you can choose to override/disable triggers for each resource.
The way this is configured all push triggers are disabled.
One possible way to achieve what you are attempting I believe is to remove the trigger:none from p1 and p3
If I read your question correctly you are trying to do a CI/CD build deployment on the repository. If so, may I suggest if the scenario is appropriate (i.e. a Build will always trigger a deployment) then combine these pipelines into one and put an if statement around the deployment stage similar to:
- ${{ if eq(variables['Build.SourceBranch'], 'refs/heads/master')}}:
Also, if deploying to multiple environments this can be followed up with a loop indented in one line:
- ${{ each environmentNames in parameters.environmentNames }}:
I noticed you are already using template so this would be just moving the template call up from the job to the stage and have it act as a wrapper. Feel free to provide feedback. If this answer isn't appropriate, I can update it accordingly.

Related

Losing the source branch of triggering build pipeline in Azure DevOps deployment pipeline

In Azure DevOps, I have two Git repositories in the MyProject project:
MyProject.Web is the .NET Core server-side code for an Angular application.
MyProject.UI is the associated Angular project.
Pull requests to specified branches in both of those projects trigger a build pipeline. The pipeline YAML files are in the MyProject.Web repo. Successful completion of the build pipeline triggers a deployment pipeline. The build is working fine; the deployment is fine when a MyProject.Web branch is the trigger, but it isn't always doing what's expected when a MyProject.UI branch is the trigger.
Details
We currently have three deployment environments: Dev, QA, Test.
I have a build pipeline that pulls the code from each of these repos together and builds the application (into a Docker image, for what it's worth).
That pipeline is triggered by completed pull requests to any master or main or release/* branch in either the Web or UI repo. (In the Web project, it's master;
in the UI project, it's main). Whether the trigger comes from the Web repo or the UI repo, the pipeline knows to combine it with the corresponding branch
from the other repo (i.e., master with main, main with master, release/1.0 with release/1.0, release/1.2 with release/1.2, etc.).
I have a deployment pipeline that is triggered by completion of the build pipeline. It has one stage per deployment environment It is supposed to know whether the source branch was master or main or whether it was a release/* branch.
If it was master or main, it should deploy to Dev and, after manager approval, to QA. It should never deploy to Test.
If it was a release/* branch, it should deploy, after manager approval, to Test. It should never deploy to Dev or QA.
Expectations in Each Triggering Scenario
Here is a summary of my expectations/requirements for each possible type of build trigger, and the reality in the deployment in the fourth scenario.
TriggeringProject
TriggeringBranch
MyProject.Web Branchto Build
MyProject.UIBranchto Build
MyProject.WebBranch toDeploy From
Environment(s) toDeploy To
MyProject.Web
master
master
main
master
Dev, QA
MyProject.Web
release/1.0
release/1.0
release/1.0
release/1.0
Test
MyProject.UI
main
master
main
master
Dev, QA
MyProject.UI
release/1.0
release/1.0
release/1.0
release/1.0 ❌(reality: seems to pullfrom master)
Test ❌(reality: deploysto Dev, QA)
The Problem
My setup performs as expected, as set out in the table, except for the deployment process for the final case. While I can tell from the DevOps pipeline logs that the builds are
correctly choosing the branches to get the code from in all four scenarios, the deployment in the final scenario seems to be taking a build associated
with the MyProject.Web master branch (the log shows that the pipeline was run against master) and it is deploying whatever build it's finding on master to Dev and QA instead of to Test.
Any thoughts about what's going on here? Guidance for a solution? Details below.
The Pipeline Files
Three pipeline files come into play here:
docker-build.yml, the YAML file behind a pipeline named Docker-Build.
It resides in MyProject.Web.
It is triggered by completed pull requests to master or release/* branches on MyProject.Web.
It is also triggered by completed PRs to main or release/* branches on MyProject.UI.
After setting things up and setting template parameter values, the meat of its work is performed by an invoked pipeline template, but there's no need to go into that here.
kubernetes-deploy.yml, the YAML file behind a pipeline named Kubernetes-Deploy.
It resides in MyProject.Web.
It is triggered by successful completion of Docker-Build.
After setting things up and setting one template parameter value (the pipeline ID of the associated Docker-Build pipeline), it invokes the pipeline template deploy-pipeline-template.yml, discussed next.
deploy-pipeline-template.yml, a pipeline template.
It is shared by multiple applications/repos in this same project.
It resides in a separate repository called MyProject.Pipeline.
The content of these files, with extraneous omissions noted:
docker-build.yml:
# This is the YAML for the "Docker-Build" pipeline. It resides in the MyProject.Web repository.
resources:
repositories:
- repository: self
type: git
name: MyProject.Web
trigger:
- master
- release/*
- repository: UiRepo
type: git
name: MyProject.UI
trigger:
- main
- release/*
- repository: PipelineRepo
type: git
name: MyProject.Pipeline
# [variables and pool omitted]
steps:
- ${{ if in(variables['Build.SourceBranchName'], 'master', 'main') }}:
- checkout: git://MyProject/MyProject.Web#refs/heads/master
- checkout: git://MyProject/MyProject.UI#refs/heads/main
- ${{ if not(in(variables['Build.SourceBranchName'], 'master', 'main')) }}:
- checkout: git://MyProject/MyProject.Web#${{ variables['Build.SourceBranch'] }}
- checkout: git://MyProject/MyProject.UI#${{ variables['Build.SourceBranch'] }}
# [some details omitted]
- template: build-pipeline-template.yml#PipelineRepo
parameters:
relativeSolutionPath: MyProject.Web
relativeProjectPath: MyProject.Web/MyProject.Web
# [other parameters omitted]
kubernetes-deploy.yml:
repositories:
- repository: PipelineRepo
type: git
name: MyProject.Pipeline
pipelines:
- pipeline: buildPipeline
source: 'Docker-Build'
trigger: true
trigger:
- none
# [pool omitted]
stages:
- template: deploy-pipeline-template.yml#PipelineRepo
parameters:
buildPipelineId: '123'
# I can probably replace '123' with variables['resources.pipeline.buildPipeline.PipelineID'] or the
# same thing with another one of Azure DevOps' multitudinous syntaxes, but I haven't tested it yet.
build-pipeline-template.yml
# This is a pipeline template that resides in the MyProject.Pipeline repository.
parameters:
- name: buildPipelineId
displayName: ID of the pipeline that produced the artifacts to download
type: string
stages:
- template: deploy-pipeline-job-template.yml
parameters:
stageName: Development
canRun: and(not(or(failed(), canceled())), in(variables['resources.pipeline.buildPipeline.sourceBranch'], 'refs/heads/master', 'refs/heads/main'))
variableGroup: myproject-web-variables-dev
buildPipelineId: ${{ parameters.buildPipelineId }}
devOpsEnvironment: myproject-dev
kubernetesServiceConnection: myproject-dev-kubeconfig
- template: deploy-pipeline-job-template.yml
parameters:
stageName: QA
canRun: and(not(or(failed(), canceled())), in(variables['resources.pipeline.buildPipeline.sourceBranch'], 'refs/heads/master', 'refs/heads/main'))
variableGroup: myproject-web-variables-qa
buildPipelineId: ${{ parameters.buildPipelineId }}
devOpsEnvironment: myproject-qa
kubernetesServiceConnection: myproject-dev-kubeconfig
- template: deploy-pipeline-job-template.yml
parameters:
stageName: Test
canRun: and(not(or(failed(), canceled())), startsWith(variables['resources.pipeline.buildPipeline.sourceBranch'], 'refs/heads/release/'))
variableGroup: myproject-web-variables-test
buildPipelineId: ${{ parameters.buildPipelineId }}
devOpsEnvironment: myproject-test
kubernetesServiceConnection: myproject-dev-kubeconfig

Pipeline resource triggers doesn't consider branch

I've two Azure DevOps CI pipelines
DataPipeline\Windows - Build
DataPipeline\TestPipeline
The second pipeline is a pipeline resource trigger. I've used this doc as a reference.
When a pull request is created (e.g. with dev branch as source branch and main branch as target branch, not merged yet) both these pipelines are triggered (by default configuration). The second pipeline may pick up old artefacts (this is a different part altogether). A second run for pipeline DataPipeline\TestPipeline is triggered automatically as soon as DataPipeline\Windows - Build completes for a commit in the dev branch. Is this possible? I've specified the branch name in the resources trigger. Is something wrong with the YAML configuration?
My expected behaviour is when a pull request is created for branch name: xyz (not main), DataPipeline\TestPipeline shouldn't be triggered again after DataPipeline\Windows - Build completes.
DataPipeline\TestPipeline.yml
trigger: none
resources:
pipelines:
- pipeline: DataPipeline
source: 'DataPipeline\Windows - Build'
branch: main
trigger:
branches:
include:
- main
name: TestPipeline_$(variable)
jobs:
// template based job

How to trigger GitHub Action workflow based on several workflows

Suppose i have three workflows: build_backend, build_frontend and deploy. First two should trigger in parallel, but the third should only trigger when both of those workflows are finished.
Currently the deploy workflow triggers twice -- i suspect that's for each of the two workflows completed.
# .github/workflows/build-xxx.yml
name: Build and Test - Backend
on:
push:
branches:
- master
pull_request:
branches:
- master
jobs:
build:
runs-on: ubuntu-latest
steps:
- name: Checkout
uses: actions/checkout#v2
# ...
# .github/workflows/deploy.yml
name: Deploy
on:
workflow_run:
workflows:
- "Build and Test - Backend"
- "Build and Test - Frontend"
types:
- completed
branch: master
jobs:
deploy:
# ...
I haven't found the solution in docs:
https://docs.github.com/en/actions/reference/events-that-trigger-workflows#workflow_run
https://docs.github.com/en/actions/reference/workflow-syntax-for-github-actions
GitHub actions does not support specific trigger definitions like this for whole workflows.
However, you can use the needs keyword on a job level. So if you could consolidate all of these workflows into one workflow file. It seems like like this could work for you since these workflows all have the same (branch) trigger and the buildxxx are only a single job each.
There is also a GitHub Roadmap item describing, that they are working on adding workflow partials. That would enable you to separte these parts out in the future if you want to, but that is not avilable yet it seems like.

Azure Devops Release Jobs/Pipeline

We are using Azure devops for our CI/CD. Typically, all the CI are written as azure yaml files and the release jobs have to be created on devops portal (using GUI). One of the general principle which we want to follow is to have everything as code.
Questions:
Can Azure release pipelines be created as code (yaml , etc) ?
I spent some time on it and it seems it is limited. Please correct me if i am wrong here.
Release pipelines have numerous things like approvals, auto trigger, release trigger, etc . Is it possible with release pipelines in yaml ?
Azure deployments can be configured with code. You can add multiple release triggers (pipeline, pull request etc.) Approvals can be configured per environment (https://www.programmingwithwolfgang.com/deployment-approvals-yaml-pipeline/), then reference the environment in your pipeline.
The example below is triggered when its own yaml code changes and when the Build pipeline completes.
trigger:
branches:
include:
- myBranch
paths:
include:
- '/Deployment/azure-deploy.yml'
resources:
pipelines:
- pipeline: BuildPipeline
project: myProjectName
source: 'myBuildPipeline'
trigger:
enabled: true
jobs:
- deployment: Deploy
displayName: Deploy
environment: $(environment)
pool:
vmImage: 'windows-latest'
strategy:
runOnce:
deploy:
steps:
- task: AzureRmWebAppDeployment#4
displayName: Deploy Web App
inputs:
ConnectionType: 'AzureRM'
azureSubscription: $(azureSubscription)
appType: 'webApp'
appSettings:
-SETTING-1 "$(mySetting1)"
WebAppName: '$(myAppName)'
package: '$(Pipeline.Workspace)/**/*.zip'
[1]: https://www.programmingwithwolfgang.com/deployment-approvals-yaml-pipeline/

Conditional Approval gate in deployment jobs in azure pipelines

Since conditional approval doesn't work in azure yaml pipeline i've been trying a workaround using 2 environment in deployment stage, shown in yaml.
using a conditions in job and a variable i want to check if approval required or not
but when i run the pipeline , i see its still asking for approval even though the condition is not satisfied for the deployment job that requires approval. Post approval though the job that required approval skips as expected. I dont understand why its asking for approval.
Are approval executed first for a stage before jobs conditions are evaluated?
Did i miss something in the yaml?
trigger:
- none
variables:
- group: pipelinevariables
# Agent VM image name
- name: vmImageName
value: 'ubuntu-latest'
stages:
- stage: Deploy
displayName: Deploy stage
jobs:
- deployment: DeployWebWithoutApprval
displayName: deploy Web App without approval
condition: and(succeeded(),ne(variables.DEV_APPROVAL_REQUIRED,'true'))
pool:
vmImage: $(vmImageName)
# creates an environment if it doesn't exist
environment: 'app-dev'
strategy:
runOnce:
deploy:
steps:
- script: echo No approval
- deployment: DeployWebWithApprval
displayName: deploy Web App with approval
dependsOn: DeployWebWithoutApprval
condition: and(eq(dependencies.DeployWebWithoutApprval.result,'Skipped'),eq(variables.DEV_APPROVAL_REQUIRED,'true'))
pool:
vmImage: $(vmImageName)
# creates an environment if it doesn't exist
environment: 'app-dev-with-approval'
strategy:
runOnce:
deploy:
steps:
- script: echo requires approval
update :
this works if i define 2 stages and and same set of conditions but that would show 2 stages in build details page which we don't want
Another question is
Can we conditionally insert stage template based on variable value from variable group
stages
​${{ifeq(variables['Policy_Approval_Required'],'true')}}:
Insert template conditionally is supported, you can check the following link: https://github.com/microsoft/azure-pipelines-agent/issues/1749. Check the following example:
- ${{ if eq(variables['Build.Reason'], 'PullRequest') }}:
- template: sharedstep.yml#templates
parameters:
value: true
I have had the exact same issue with approval gates and conditions. It is unfortunately not supported as of yet, but it's reported to Microsoft (here). There is also this issue.
Seems like an issue with the order of evaluating approvals vs conditions.