Consider a yaml based pipeline
resources:
repositories:
- repository: templates
type: git
name: My-Proj/azure-build-templates
...
stages:
- template: test_pipeline/include-build-java-sonarqube.yml#templates
parameters:
agent_pool_name: $(agentPoolName)
maven_goal: 'mvn clean package'
...
This will refer to repository azure-build-templates in branch master. Is there a way I can declare to refer a tag instead of master? like name: My-Proj/azure-build-templates#release-20211215-better-tag
Context of its utility: Recently we pushed something in master which consequently broke all the pipelines referring. As a fix we had to rollback with git operations on master branch. I wonder if there would be a way to refer from tag like jenkins and rollback would be just point from one git tag to another. (Downside on every release you have update all the pipelines, so I am up for any suggestions for templates release management). Thanks.
ref: https://learn.microsoft.com/en-us/azure/devops/pipelines/process/resources?view=azure-devops&tabs=schema#define-a-repositories-resource
You can link to any ref.
Tags are represented in git as refs/tags/myTag
resources:
repositories:
- repository: string # identifier (A-Z, a-z, 0-9, and underscore)
type: enum # see the following "Type" topic
name: string # repository name (format depends on `type`)
ref: string # ref name to use; defaults to 'refs/heads/main'
You can list your tags with: git show-ref --tags
Related
I have three Azure Repos, all in the same ADO Project - repo-0001, repo-0002, repo-0003
repo-0001 has my yaml pipeline which has a manual trigger, and the pipeline needs to be run on demand by either branch or tag (eg release/branch001, release/tag001)
The Pipeline checkouts out repo-0001, repo-0002 & repo-0003 at whatever ref (branch or tag) is used to start the pipeline in repo-0001 (these branches & tags are created by a different process and are always present in all repos)
I've been using $(Build.SourceBranch) to extract:
branch - refs/heads/release/branch001
tag - refs/tags/release/tag001
This yaml works for a pipeline started using a branch, but not for a tag, and it results in Could not get the latest source version for repository repo-0001 hosted on Azure Repos using refs/heads/refs/tags/release/tag001 so it appears to append the whole ref path of the tag to refs/heads
variables:
REPOSITORY_SOURCE_BRANCH: $(Build.SourceBranch)
resources:
repositories:
- repository: repo-0001
type: git
name: aks/repo-0001
ref: $(REPOSITORY_SOURCE_BRANCH)
- repository: repo-0002
type: git
name: aks/repo-0002
ref: $(REPOSITORY_SOURCE_BRANCH)
- repository: repo-0003
type: git
name: aks/repo-0003
ref: $(REPOSITORY_SOURCE_BRANCH)
I can make it work with a tag by doing something like this, or passing in as a parameter
variables:
tag: "release/tag001"
resources:
repositories:
- repository: repo-0001
type: git
name: aks/repo-0001
ref: 'refs/tags/$(tag)'
etc
But that's not what I need...
Can a tag ref be passed in from pipeline variables to make this work, and is there a way to make it flexible so that I can use the same pipeline code for tags and branches?
Thanks!
The ref field in repositories Resources doesn't support defining Pipeline variable. It only supports hardcode the ref value.
Can a tag ref be passed in from pipeline variables to make this work, and is there a way to make it flexible so that I can use the same pipeline code for tags and branches?
To meet your requirement, you can change to use the following format to checkout the repo.
- checkout: git://MyProject/MyRepo#refs/tags/$(tag)
- checkout: git://MyProject/MyRepo#$(REPOSITORY_SOURCE_BRANCH)
Here is an example:
variables:
REPOSITORY_SOURCE_BRANCH: test
steps:
- checkout: git://aks/repo-0001#$(REPOSITORY_SOURCE_BRANCH)
For more detailed info, you can refer to this doc: Checking out a specific ref
I am testing a POC build pipeline and I am trying to use templates as a sort of reference library, so I can reuse the same code in multiple pipelines, while also allowing for simpler editing/updating, if changes are required.
I want to be able to run the pipeline from a DevOps branch and refer to the same branch (or tag) when validating/running all the template pipelines in this build pipeline.
The ultimate goal is to pass multiple variables to the template (via a loop/for each) to create multiple build artifacts referencing public repositories (in this test a public terraform github repo). We could then maybe only need to maintain the variable data and not the source/reference code.
I thought DevOps did this 'OOTB', but realised that this was probably not the case. Something like this:
Repo: InfraAsData (all pipelines)
Branch: feature/sparse-clone-repo
Updated azure-pipeline.yaml and added template templates/sparse-clone.yaml (FYI - this template is untested as well)
azure-pipeline.yaml:
trigger: none
resources:
repositories:
- repository: InfraAsData #resource name to be used in the build pipeline
type: git #Azure git
name: 'IAC/InfraAsData'
refs: 'refs/heads/$(branch_name)'
parameters:
- name: branch
displayName: branch
type: string
default: $(Build.SourceBranch)
variables:
- name: branch_name
value: ${{ parameters.branch }}
- template: ./templates/variables/resource-groups.yaml#InfraAsData #variables template
pool:
vmImage: 'ubuntu-latest'
stages:
#sparse clone the target public repo blobs to build pipeline artifacts directory
- stage: Template-Sparse-Clone-To-Artifacts
displayName: 'Test Sparse Clone to Artifacts'
jobs:
- deployment: sparseCloneTfModule
displayName: 'Sparse Clone Terraform module to Artifacts'
environment: Test
strategy:
runOnce:
deploy:
steps:
- template: $(Build.SourceBranch)/templates/sparse-clone.yaml#InfraAsData #Create artifacts based on template variables
parameters: #module specific var from var template
repoUrl: '${{ variables[template.repoUrl] }}'
repoPath: '${{ variables[template.repoPath] }}'
artifactPath: '${{ variables[template.artifactPath] }}'
sparse-clone.yaml
parameters:
repoUrl: ''
repoPath: ''
artifactPath: ''
#testing
steps:
- script: |
md $(Build.BinariesDirectory)/$(parameters.artifactPath)
git clone --filter=blob:none --sparse $(parameters.repoUrl) $(parameters.artifactPath)
cd $(Build.BinariesDirectory)/$(parameters.artifactPath)
git sparse-checkout init --cone
git sparse-checkout set $(parameters.repoPath)
git checkout main
dir $(Build.BinariesDirectory)/$(parameters.artifactPath)
displayName: 'Clone Github Repo Subdirectory - filter blob none'
When I run this from the feature/sparse-clone-repo branch in DevOps I get an error that the new template does not exist in the main branch (it doesn't of course).
/build/test/azure-pipeline.yaml: File /templates/variables/resource-groups.yaml not found in repository https://dev.azure.com//IAC/_git/InfraAsData branch refs/heads/main version
I have also tried using no resources.repository, using $(Build.SourceBranch) $(Build.SourceBranchName) as refs value and as the template path reference (like in the 2nd template example).
EDIT:
If i remove the resource.repositories reference, then the correct branch is identified, but the azure-pipeline.yaml relative path is appended to the template path:
/build/test/azure-pipeline.yaml: File /build/test/$(Build.SourceBranch)/templates/variables/resource-groups.yaml not found in repository https://dev.azure.com//IAC/_git/InfraAsData branch refs/heads/feature/pipelines
So correct branch, but not the correct relative path.
EDIT 2:
I can remove the resources section and use a relative path
../../templates/variables/resource-groups.yaml
but this is not dynamic, to a 'root' reference, so would not work if the folder structure were to change (EG I moved the azure-pipeline.yaml up a level)
I'm sure I am missing something obvious or misunderstanding the docs (or pipeline templates). Any pointers gratefully received!
When I run this from the feature/sparse-clone-repo branch in DevOps I
get an error that the new template does not exist in the main branch
(it doesn't of course).
From your description, seems you thought the 'refs' of the resources determine the branch of the alias of the template part.
But it is not, the 'resources.repositories.repository' doesn't have a section named 'refs', only 'ref' is allowed.
In your situation, you used a section named 'refs' which doesn't exist. Pipeline will use the default branch of the repository to looking for the template YAML. So change the default branch of the repository or change the 'refs' to 'ref' will solve the first issue.
And I notice you were using variable in resources section, you have two mistakes.
One is '$()' can't use in compile time part. Another is even compile time variables are also not allowed in 'ref' section.
ref
string
ref name to checkout; defaults to 'refs/heads/main'. The branch
checked out by default whenever the resource trigger fires. Does not
accept variables.
You can use your pipeline like this:
trigger: none
resources:
repositories:
- repository: InfraAsData #resource name to be used in the build pipeline
type: git #Azure git
name: 'BowmanCP/template_branch'
ref: 'refs/heads/${{parameters.branch}}'
parameters:
- name: branch
displayName: branch
type: string
default: main2
pool:
vmImage: 'ubuntu-latest'
stages:
#sparse clone the target public repo blobs to build pipeline artifacts directory
- stage: Template_Sparse_Clone_To_Artifacts
displayName: 'Test Sparse Clone to Artifacts'
jobs:
- deployment: sparseCloneTfModule
displayName: 'Sparse Clone Terraform module to Artifacts'
environment: Test
strategy:
runOnce:
deploy:
steps:
- template: ./templates/sparse-clone.yaml#InfraAsData #Create artifacts based on template variables
parameters: #module specific var from var template
repoUrl: 'xxx'
repoPath: 'xxx'
artifactPath: 'xxx'
Basically, variables can't be used in your situation and there doesn't have a parameter reuse feature. That's everything.
I have two repositories:
devops - with definitions of pipelines
src_project - with sources of project
Default when I run a Azure pipeline I can choice between branch/tag of repository with pipeline definition. I would like to add a parameter with which I can select a branch from the source project.
Is this possible in a simple way in Azure? I don't want, I cannot keep definitions of pipelines in src_project.
In Jenkins, we used an additional method to fetch these branches, using the shared libraries plugin.
You can do it with resources from type repositories:
resources:
repositories:
- repository: string # identifier (A-Z, a-z, 0-9, and underscore)
type: enum # see the following "Type" topic
name: string # repository name (format depends on `type`)
ref: string # ref name to use; defaults to 'refs/heads/master'
endpoint: string # name of the service connection to use (for types that aren't Azure Repos)
trigger: # CI trigger for this repository, no CI trigger if skipped (only works for Azure Repos)
branches:
include: [ string ] # branch names which will trigger a build
exclude: [ string ] # branch names which will not
tags:
include: [ string ] # tag names which will trigger a build
exclude: [ string ] # tag names which will not
paths:
include: [ string ] # file paths which must match to trigger a build
exclude: [ string ] # file paths which will not trigger a build
And with checkout step, for example:
resources:
repositories:
- repository: MyAzureReposGitRepository # In a different organization
endpoint: MyAzureReposGitServiceConnection
type: git
name: OtherProject/MyAzureReposGitRepo
trigger:
- main
pool:
vmImage: 'ubuntu-latest'
steps:
- checkout: self
- checkout: MyAzureReposGitRepository
#Shayki
Not exactly what I want. I use checkout step but with parameter:
- checkout: git://project/repo#{{parameters.BRANCH_NAME}}
Because of this, I cannot use the resources. And for parameters I have:
parameters:
- name: BRANCH_NAME
type: string
default: develop
This is static method. I don't know how to dynamically fetch branches for a repository and pass them as values for the parameter.
I currently have my YAML pipeline and my application's source code in two different branches.
I was trying find evidence that what is being checked out is indeed the source code's branch and not my pipeline's branch.
I see that the checkout call at the end of the git fetch is to a specific commit, not to the specified branch name.
This is my resources definition:
resources:
repositories:
- repository: RepoName
type: git
name: 'MyRepository' # repository in Azure DevOps
trigger:
branches:
include:
- UAT
and in one of my steps I do a checkout: RepoName.
I was expecting a git checkout UAT after pulling the source code, but as said I see a checkout of a specific commit.
How can I be sure about which branch is being checked out?
By default, the build pipeline will check out the single commit that triggered the current pipeline run. When you manual, or via other methods, trigger a pipeline run, the run will check out the latest commit on default branch or the branch you specify.
However, no matter what methods triggers the pipeline run, you can use the predefined build variable Build.SourceBranch or Build.SourceBranchName to get the branch name where the commit is checked out from.
To view more details, you can see "Use predefined variables".
Typical syntax to display is:
steps:
- bash: echo $(Build.SourceBranch)
In addition, you also can go to the root of the local repository, and execute the git branch command. This command will output the name of the current branch.
For example:
git branch --show-current
If current branch is master, the output:
master
Note that a number of people have reported that this produces no output
You need to set ref
resources:
repositories:
- repository: string # identifier (A-Z, a-z, 0-9, and underscore)
type: enum # see the following "Type" topic
name: string # repository name (format depends on `type`)
ref: string # ref name to use; defaults to 'refs/heads/master'
endpoint: string # name of the service connection to use (for types that aren't Azure Repos)
trigger: # CI trigger for this repository, no CI trigger if skipped (only works for Azure Repos)
branches:
include: [ string ] # branch names which will trigger a build
exclude: [ string ] # branch names which will not
tags:
include: [ string ] # tag names which will trigger a build
exclude: [ string ] # tag names which will not
paths:
include: [ string ] # file paths which must match to trigger a build
exclude: [ string ] # file paths which will not trigger a build
so it would be
resources:
repositories:
- repository: RepoName
type: git
name: 'MyRepository' # repository in Azure DevOps
ref: 'UAT'
trigger:
branches:
include:
- UAT
In ref property you can set like this: ref: 'refs/heads/UAT'
resources:
repositories:
- repository: RepoName
type: git
name: 'MyRepository' # repository in Azure DevOps
ref: 'refs/heads/UAT'
trigger:
branches:
include:
- UAT
Then in steps, you need to add - checkout: RepoName
Here is my sample you can refer to:
pool:
vmImage: 'windows-2019'
trigger: none
resources:
repositories:
- repository: RepoName
type: git
name: '{proName}/{repoName}' # repository in Azure DevOps
ref: 'refs/heads/dev'
steps:
- checkout: RepoName
In build summary page:
In log :
I have repo A that holds pipeline templates. Repo A has the following azure-template.yml:
# Repo A / azure-template.yml
stages:
- stage: ${{ variables.stageName }}
jobs:
- job:
steps:
- task:
A lot of the code in repo A that has the templates refers to variables in the following format:
${{ variables.variableName }}. The variable file is in a different folder in repo A. (e.g. variables/variables.yaml)
Now let's move to repo B. Repo B has my azure-pipeline.yml that needs to build from repo A:
# Repo B / azure-pipeline.yml
resources:
repositories:
- repository: templates
type: git
name: repoA
ref: refs/heads/develop
variables:
- template: variables/variables.yml#templates
stages:
- template: azure-template.yml#templates # Template reference
When I run azure-pipeline.yml, I get the following error:
An error occurred while loading the YAML build pipeline. The string must have at least one character. Parameter name: environmentName
That parameter is not one of mine. I don't have it declared or set anywhere. This tells me it is Azure specific but I have no idea where/why it's there or where it is even set.
How can I run a pipeline with a yaml template that refers to a
variables file in a different repo with Azure DevOps?
You're in correct direction, at least a working direction. It's supported to do that like what you've done above. About the error you got, I assume there could be something wrong with your yaml syntax. You can try following steps to locate the issue:
Copy the content of azure-template.yml and variables.yaml directly into azure-pipeline.yml file, and run the pipeline again to check if the issue persists.
In your Azure-pipeline.yml, specify the trigger and pool.
In azure-template.yml, try replacing the ${{ variables.stageName }} with hard-code value.
This is my first time to see this error message, but according to Parameter name: environmentName.You can also check if Release.EnvironmentName has valid value in one PS task. Hope it helps :)