When a pipeline is triggered by a repository resource trigger I need to be able to determine metadata about the triggering repository (such as repo name and branch).
It sounds like these variables should be set during a pipeline run that was triggered by a repository resource; however, their values are blank when I do echo $(resources.triggeringAlias) or env | sort in a pipeline run that was triggered by a repository resource.
UPDATE: the predefined variables for Build.Repository.Name and Build.SourceBranchName now work as expected when used with a repository trigger. So, while I'm no longer in need of resources.triggeringAlias and resources.triggeringCategory, they still aren't working.
I need to be able to determine metadata about the triggering
repository (such as repo name and branch)
We can use $(Build.Repository.Name) and $(Build.Repository.Uri) to get repo name and repo uri.
And $(Build.SourceVersion) can be used to get CommitId, $(Build.SourceBranch) or $(Build.SourceBranchName) can be used to get branch info.
Just note we must also checkout the triggering repo to make above variables work to fetch the info of triggering repo, otherwise those variables will always represents the value of triggered repo:
steps:
- checkout: self
- checkout: TheTriggeringRepo
Some details:
I have a triggering repo PipelineA, and the triggered repo PipelineB. PipelineB's azure-pipelines.yml file:
resources:
repositories:
- repository: PipelineA
type: git
name: PipelineA
trigger:
- master
steps:
- checkout: self
- checkout: PipelineA
- task: Bash#3
inputs:
targetType: 'inline'
script: |
echo $(Build.Repository.Name)
echo $(Build.SourceBranch)
So this pipeline will be triggered by both PipelineA repo and PipelineB repo.
When PipelineB repo has changes:
When PipelineA repo has changes:
It's clear the $(Build.Repository.Name) variable can work well to output the real trigger repo if we checkout both these two repos. So just make sure you checkout those triggering repos, then my variables above would work for you.
resources:
pipelines:
- pipeline: client-qa-deployment ## Resources.TriggeringAlias will be 'client-qa-deployment' if this resource gets triggered.
source: benjose.dev-code
trigger:
stages:
- client_deploy_qa
- stage: qa_smoketest
displayName: Smoke Test on QA Environment
condition: eq(variables['Resources.TriggeringAlias'], 'client-qa-deployment')
pool:
vmImage: 'windows-latest'
jobs:
- job: TestQA
steps:
- script: echo Hello, $(Resources.TriggeringAlias)!
displayName: 'Running Smoke Test on QA Environment'
Hello, client-staging-deployment!
I believe the above snippet will help you to understand the syntax and usage of Resources.TriggeringAlias. This variable will be helpful if we have multiple resources in the same pipeline and to find out which resource got trigged.
Related
I am testing a POC build pipeline and I am trying to use templates as a sort of reference library, so I can reuse the same code in multiple pipelines, while also allowing for simpler editing/updating, if changes are required.
I want to be able to run the pipeline from a DevOps branch and refer to the same branch (or tag) when validating/running all the template pipelines in this build pipeline.
The ultimate goal is to pass multiple variables to the template (via a loop/for each) to create multiple build artifacts referencing public repositories (in this test a public terraform github repo). We could then maybe only need to maintain the variable data and not the source/reference code.
I thought DevOps did this 'OOTB', but realised that this was probably not the case. Something like this:
Repo: InfraAsData (all pipelines)
Branch: feature/sparse-clone-repo
Updated azure-pipeline.yaml and added template templates/sparse-clone.yaml (FYI - this template is untested as well)
azure-pipeline.yaml:
trigger: none
resources:
repositories:
- repository: InfraAsData #resource name to be used in the build pipeline
type: git #Azure git
name: 'IAC/InfraAsData'
refs: 'refs/heads/$(branch_name)'
parameters:
- name: branch
displayName: branch
type: string
default: $(Build.SourceBranch)
variables:
- name: branch_name
value: ${{ parameters.branch }}
- template: ./templates/variables/resource-groups.yaml#InfraAsData #variables template
pool:
vmImage: 'ubuntu-latest'
stages:
#sparse clone the target public repo blobs to build pipeline artifacts directory
- stage: Template-Sparse-Clone-To-Artifacts
displayName: 'Test Sparse Clone to Artifacts'
jobs:
- deployment: sparseCloneTfModule
displayName: 'Sparse Clone Terraform module to Artifacts'
environment: Test
strategy:
runOnce:
deploy:
steps:
- template: $(Build.SourceBranch)/templates/sparse-clone.yaml#InfraAsData #Create artifacts based on template variables
parameters: #module specific var from var template
repoUrl: '${{ variables[template.repoUrl] }}'
repoPath: '${{ variables[template.repoPath] }}'
artifactPath: '${{ variables[template.artifactPath] }}'
sparse-clone.yaml
parameters:
repoUrl: ''
repoPath: ''
artifactPath: ''
#testing
steps:
- script: |
md $(Build.BinariesDirectory)/$(parameters.artifactPath)
git clone --filter=blob:none --sparse $(parameters.repoUrl) $(parameters.artifactPath)
cd $(Build.BinariesDirectory)/$(parameters.artifactPath)
git sparse-checkout init --cone
git sparse-checkout set $(parameters.repoPath)
git checkout main
dir $(Build.BinariesDirectory)/$(parameters.artifactPath)
displayName: 'Clone Github Repo Subdirectory - filter blob none'
When I run this from the feature/sparse-clone-repo branch in DevOps I get an error that the new template does not exist in the main branch (it doesn't of course).
/build/test/azure-pipeline.yaml: File /templates/variables/resource-groups.yaml not found in repository https://dev.azure.com//IAC/_git/InfraAsData branch refs/heads/main version
I have also tried using no resources.repository, using $(Build.SourceBranch) $(Build.SourceBranchName) as refs value and as the template path reference (like in the 2nd template example).
EDIT:
If i remove the resource.repositories reference, then the correct branch is identified, but the azure-pipeline.yaml relative path is appended to the template path:
/build/test/azure-pipeline.yaml: File /build/test/$(Build.SourceBranch)/templates/variables/resource-groups.yaml not found in repository https://dev.azure.com//IAC/_git/InfraAsData branch refs/heads/feature/pipelines
So correct branch, but not the correct relative path.
EDIT 2:
I can remove the resources section and use a relative path
../../templates/variables/resource-groups.yaml
but this is not dynamic, to a 'root' reference, so would not work if the folder structure were to change (EG I moved the azure-pipeline.yaml up a level)
I'm sure I am missing something obvious or misunderstanding the docs (or pipeline templates). Any pointers gratefully received!
When I run this from the feature/sparse-clone-repo branch in DevOps I
get an error that the new template does not exist in the main branch
(it doesn't of course).
From your description, seems you thought the 'refs' of the resources determine the branch of the alias of the template part.
But it is not, the 'resources.repositories.repository' doesn't have a section named 'refs', only 'ref' is allowed.
In your situation, you used a section named 'refs' which doesn't exist. Pipeline will use the default branch of the repository to looking for the template YAML. So change the default branch of the repository or change the 'refs' to 'ref' will solve the first issue.
And I notice you were using variable in resources section, you have two mistakes.
One is '$()' can't use in compile time part. Another is even compile time variables are also not allowed in 'ref' section.
ref
string
ref name to checkout; defaults to 'refs/heads/main'. The branch
checked out by default whenever the resource trigger fires. Does not
accept variables.
You can use your pipeline like this:
trigger: none
resources:
repositories:
- repository: InfraAsData #resource name to be used in the build pipeline
type: git #Azure git
name: 'BowmanCP/template_branch'
ref: 'refs/heads/${{parameters.branch}}'
parameters:
- name: branch
displayName: branch
type: string
default: main2
pool:
vmImage: 'ubuntu-latest'
stages:
#sparse clone the target public repo blobs to build pipeline artifacts directory
- stage: Template_Sparse_Clone_To_Artifacts
displayName: 'Test Sparse Clone to Artifacts'
jobs:
- deployment: sparseCloneTfModule
displayName: 'Sparse Clone Terraform module to Artifacts'
environment: Test
strategy:
runOnce:
deploy:
steps:
- template: ./templates/sparse-clone.yaml#InfraAsData #Create artifacts based on template variables
parameters: #module specific var from var template
repoUrl: 'xxx'
repoPath: 'xxx'
artifactPath: 'xxx'
Basically, variables can't be used in your situation and there doesn't have a parameter reuse feature. That's everything.
I have a yaml pipeline which I want to make it to run for more branches. Therefore I am trying to checkout one branch, specified in pipeline variables. I'm unable to do so, the error being Unexpected value 'ref'.
The pipeline file is:
trigger:
batch: true
branches:
include:
- 'release/dev'
- 'release/test'
- 'release/prod'
paths:
include:
- '*'
resources:
- repo: self
ref: $(branch)
variables:
dockerRegistryServiceConnection: 'conn'
containerRegistry: 'conn.reg'
tag: '$(Build.BuildId)'
vmImageName: 'ubuntu-latest'
stages:
- stage: Build
displayName: Build and push stage
jobs:
- job: Build
displayName: Build
pool:
vmImage: $(vmImageName)
steps:
- template: docker-build-template.yml
parameters:
dockerfilePath: $(Build.SourcesDirectory)/Dockerfile.release
imageName: conn.reg/container
imageRepository: container
pushToRegistry: true
How can I checkout different branches for building the container out of them?
Later edit: I want the pipeline to automatically run after a PR or a commit is pushed on any of the braches. (Manually it can be run with specifiyng a branch.)
Azure DevOps Pipelines: how to check out branch of the self repo?
You could specify the name of the self repo in the resource with a specific ref,like:
resources:
repositories:
- repository: MyTestProject
type: git
name: MyTestProject
ref: $(branch)
Then checkout with another path:
steps:
- checkout: MyTestProject
path: Another path/xxxx/xxx
You don't define the self repo as a resource. If you want to run your build on another branch, just choose your branch in the "Run pipeline" screen:
As for running automatically after completing a PR, you already have the triggers.branches.include set, so merges (or pushes) to all these branches will trigger a build in which the relevant branch will be checked out.
I'm somewhat confused as to what goes in Build.SourceBranch and Build.SourceBranchName. When you run manually a YAML pipeline those predefined variables end up tracking the branch for the pipeline's definition itself, not the source code branch that is being checked out. I suppose since those variables need to be set when a pipeline is started, it would not be possible to have any one predefined variable return the checked out source code branch(es).
When you run manually a YAML pipeline those predefined variables end
up tracking the branch for the pipeline's definition itself, not the
source code branch that is being checked out.
As stated in this document: Build.SourceBranch represents the branch of the triggering repo the build was queued for and Build.SourceBranchName is the name of the branch in the triggering repo the build was queued for.
These two variables are only used to track the triggering repo, they have nothing to do with source branches that are checked out. When manually running YAML pipeline, these variables only represent the branch selected here:
I suppose since those variables need to be set when a pipeline is
started, it would not be possible to have any one predefined variable
return the checked out source code branch(es).
We don't have predefined variables for checked out branches. For pipelines that check out multiple repos with different branched, such variable(not supported) would have to contain more than
one branch names like "master:RepoA dev:RepoB ...".
Possible workaround:
Here's my working script for YAML pipeline that checks out self repo(Test repo) and another two repos(Agile repo and RepoB repo):
pool:
VmImage: "windows-latest"
resources:
repositories:
- repository: RepoA
type: git
name: Agile
ref: Test2.0
trigger:
- Test2.0
- repository: RepoB
type: git
name: RepoB
ref: main
trigger:
- main
- dev
steps:
- checkout: self
- checkout: RepoA
- checkout: RepoB
- script: echo branch:$(Build.SourceBranch) from repo:$(Build.Repository.Name).
displayName: 'Run a one-line script in master'
- task: PowerShell#2
inputs:
targetType: 'inline'
script: |
if ("$(Build.Repository.Name)" -eq "Test") {
$BranchOfYamlRepo = "$(Build.SourceBranchName)"
$BranchOfRepoA = "Test2.0"
$BranchOfRepoB = "main"
}elseif ("$(Build.Repository.Name)" -eq "Agile") {
$BranchOfYamlRepo = "main"
$BranchOfRepoA = "$(Build.SourceBranchName)"
$BranchOfRepoB = "main"
}else {
$BranchOfYamlRepo = "main"
$BranchOfRepoA = "Test2.0"
$BranchOfRepoB = "$(Build.SourceBranchName)"
}
Write-Host "$BranchOfYamlRepo,$BranchOfRepoA,$BranchOfRepoB"
When you manually trigger a YAML pipeline, by default the pipeline will run on the default branch (master or main), if the YAML file also is existing on the default branch. So the source branch will be set as the default branch by default.
However, when you manually trigger the pipeline, you can specify the pipeline runs on another branch or a tag, and this also will set the source branch as the branch or tag you specify.
My pipeline trigger in Azure DevOps does not fire.
Enviroment:
PipelineA (branch dev)
PipelineB (branch dev)
PipelineB should be fired if PipelineB was running successfully
Here is my current code of PipelineB.yaml
trigger: none
resources:
pipelines:
- pipeline: build_pipeline
source: PipelineA
branch: dev
trigger:
branches:
- dev
steps:
- task: Bash#3
inputs:
targetType: 'inline'
script: |
echo 'Hello world'
It worked in the past, but suddenly it stopped
Here's the document of Pipeline Triggers, please check this statement:
However, if the two pipelines use different repositories, then the triggered pipeline will use the latest version of the code from its default branch.
Cause of your issue:
The triggered pipeline is PipelineB in your scenario, and the default branch is always the master. Since the pipeline trigger for dev branch of PipelineA is defined in your dev branch instead of the default master branch, it's expected behavior that the pipeline trigger won't fire.
To make the pipeline trigger work:
1.You can choose to define the pipeline trigger for dev of PipelineA in default master of PipelineB.
2.Or you can change the Default branch for manual and scheduled builds of PipelineB (Change it from master to dev). You can find detailed steps about how to find this setting from #2 of this answer.
Both two choices above can resolve your issue and make the pipeline trigger work.
I just created almost identical pipeline
trigger: none
resources:
pipelines:
- pipeline: build_pipeline
source: kmadof.devops-manual (14)
branch: master
trigger:
branches:
- master
steps:
- task: Bash#3
inputs:
targetType: 'inline'
script: |
echo 'Hello world'
and all works fine. Are you sure you run PipelineA on dev branch?
Problem
I have two repositories A and B in one project, each with their own pipeline A-CI and B-CI. The repos are Azure Repos Git (so not external ones). I got it working to trigger pipeline B-CI whenever A-CI has completed. If A-CI got triggered by a commit to branch develop, then B-CI is triggered to build master although B also has a develop branch.
I want to build a new release of B for the dev environment, when a new dev version of A has been built.
Is it possible to let a pipeline-resource trigger the B-CI pipeline to build the branch with the same name as the branch which was just built by the pipeline-resource? It would be fine for me if it would fallback to master in case a matching branch is not available in B.
This scenario is working however if A-C and B-CI would both refer to different pipeline yamls of the same repository.
Pipeline YAMLs
A-CI
trigger:
- '*'
stages:
- stage: Build
jobs:
- job: BuildJob
pool:
name: 'MyBuildPool'
steps:
- powershell: |
Write-Host "Building A"
B-CI
resources:
pipelines:
- pipeline: Pipeline_A
source: 'A-CI'
trigger:
branches:
- master
- develop
- feature/*
trigger:
- '*'
stages:
- stage: Build
jobs:
- job: BuildJob
pool:
name: MyBuildPool
steps:
- powershell: |
Write-Host $(Build.SourceBranch) # is always refs/heads/master
Write-Host $(Build.Reason) # is ResourceTrigger
Background Info
The main idea behind that is, that A contains the IaC project and whenever the infrastructure of the project changes, then all apps should be deployed, too.
I do not want to put the IaC into the app repo because we have multiple apps, so I would have to split the IaC code into several chunks.
And then I would probably still have the same problem because some resources, like Azure KeyVault, are shared among the apps so A would still include the common stuff used by all apps and changes to it would require re-deployments of all apps.
Please check pipeline triggers:
If the triggering pipeline and the triggered pipeline use the same repository, then both the pipelines will run using the same commit when one triggers the other. This is helpful if your first pipeline builds the code, and the second pipeline tests it.
However, if the two pipelines use different repositories, then the triggered pipeline will use the latest version of the code from its default branch.
In this case, since master if the default branch of your B-CI, $(Build.SourceBranch) is always refs/heads/master.
As a workaround:
You can create a new yaml pipeline for repository B. You can use similar content of the yaml file for B-CI. And you only need to change something in it to:
resources:
pipelines:
- pipeline: Pipeline_A
source: 'A-CI'
trigger:
branches:
- develop
When we create the new yaml file, it's always placed in master branch. For me, I created a file with same name in dev branch, and copy the same content in it. Then i delete the new yaml file in master branch, now when dev of A-CI pipeline is built, dev of B repos will be used.
I think I have found a nicely working workaround as the built-in pipeline-triggers are not addressing our specific problem (though I can't say if we have an odd approach and there are better ways).
What I am doing now it to use the Azure CLI DevOps extension based on this docs entry and trigger the pipelines manually.
Pipeline YAMLs
A-CI
trigger:
- '*'
stages:
- stage: Build
displayName: Build something and create a pipeline artifact
jobs:
- job: BuildJob
pool:
name: 'MyBuildPool'
steps:
- powershell: |
Write-Host "Building A"
# steps to publish artifact omitted
- stage: TriggerAppPipelines
displayName: Trigger App Pipeline
jobs:
- job: TriggerAppPipelinesJob
displayName: Trigger App Pipeline
steps:
- bash: az extension list | grep azure-devops
displayName: 'Ensure Azure CLI DevOps extension is installed'
- bash: |
echo ${AZURE_DEVOPS_CLI_PAT} | az devops login
az devops configure --defaults organization=https://dev.azure.com/MyOrg project="MyProject" --use-git-aliases true
displayName: 'Login Azure CLI'
env:
AZURE_DEVOPS_CLI_PAT: $(System.AccessToken)
# By passing the build Id of this A-CI run, I can use that in B-CI to download the matching artifact of A-CI.
# If there is no matching branch, then the command fails.
- bash: |
az pipelines run --branch $(Build.SourceBranch) --name "B-CI" --variables a_Version="$(Build.BuildId)" -o none
displayName: 'Trigger pipeline'
B-CI
trigger:
- '*'
stages:
- stage: Build
jobs:
- job: BuildJob
pool:
name: MyBuildPool
steps:
- powershell: |
Write-Host $(Build.SourceBranch) # is same as the the triggering A-CI branch
Write-Host $(Build.Reason) # B-CI is triggered manually but the user is Project Collection Build Service, so automated runs can be distinguished
As B-CI is triggered manually now, there is no need for a resource node anymore.