As a continuation to Azure DevOps - Handling single release for separate code repositories for UI and Dotnet API layer I am trying to proceed with checking out multiple repos in my yaml by defining it under repositories as shows below. I am unable to understand where to change the branch to make sure, my tasks such as checkout does not happen on master, but on my custom branch.
resources:
repositories:
- repository: mybitbucketrepo
type: bitbucket
endpoint: myserviceconnection
trigger: # CI trigger for this repository, no CI trigger if skipped (only works for Azure Repos)
branches:
include: [ custom-branch ]
name: orgname/reponame
What I tried so far is below for which I get the error - Only 'self', 'none' or a repository alias are supported. Any help will be appreciated.
- checkout: mybitbucketrepo#custom-branch
You need to define ref settings:
When using a repository resource, specify the ref using the ref property. The following example checks out the features/tools/ branch of the designated repository.
resources:
repositories:
- repository: MyGitHubRepo
type: github
endpoint: MyGitHubServiceConnection
name: MyGitHubOrgOrUser/MyGitHubRepo
ref: features/tools
and then just
steps:
- checkout: MyGitHubRepo
IT's not support to use #, please use inline ref syntax
steps:
- checkout: self
- checkout: git://MyProject/MyToolsRepo#features/mytools
By appending #<ref>, the agent can be instructed to check out a different ref. In this case, it's assumed to be a branch called features/mytools. Branches can also be prefixed with refs/heads/ to make it unambiguous that it's a branch.
Other valid refs (and ref-like things):
refs/tags/
commit ID
For more details, please take a look at this link: Multi-checkout: checking out multiple repos
You could also define it in resource, check our official doc --Checking out a specific ref A sample as below:
resources:
repositories:
- repository: app
type: github
name: org1/repoA
ref: master
endpoint: 'GitHub endpoint'
trigger:
- master
- release/*
Related
You think is possible to reference the same BitBucket Repository under 2 different names with 2 different paths in order to achieve and trigger 2 different stages ? I want to build and publish 7 different microservices on K8s but the repos are only 3 (divided in subfolders) you this this can be achievable? The idea is to create different blocks of template one for each microservice. But during the template checkout the 3 macrorepository only (for dev purposes). Let me show you my idea.
resources:
repositories:
############## 3 BITBUCKET BIG REPOS ##############
- repository: omni-omsf-api
type: bitbucket
endpoint: OMNI-OMSF-BitBucket-SC
name: ovsdev/omni-omsf-api
trigger: none
- repository: omni-omsf-extension
type: bitbucket
endpoint: OMNI-OMSF-BitBucket-SC
name: ovsdev/omni-omsf-extension
trigger: none
- repository: omni-omsf-core
type: bitbucket
endpoint: OMNI-OMSF-BitBucket-SC
name: ovsdev/omni-omsf-core
trigger: none
###################################################
############# 7 SUB-REPOS ONLY FOR TRIGGERING #########
- repository: ovs-api-service
type: bitbucket
endpoint: OMNI-OMSF-BitBucket-SC
name: ovsdev/omni-omsf-api
trigger:
branches:
include:
- release_qa
- master
paths:
include:
- ovs-api-service/*
###################################################
stages:
- ${{ if or( and( eq( parameters.ovsapiservice, true), eq( variables['Build.Reason'], 'Manual') ), eq( variables['Build.Repository.Name'], 'ovs-api-service') ) }}:
- template: microservice-buildRelease.template.yml
parameters:
dockerFilePath: omni-omsf-api/ovs-api-service/Dockerfile
dockerImageName: ovs-api-service
tag: $(Build.BuildId)
microservicename: ovs-api-service
- ${{ else }}:
- stage:
jobs:
- job:
steps:
- task: Bash#3
displayName: Showing folder hierarchy
inputs:
targetType: 'inline'
script: |
tree $(Pipeline.Workspace)
Can I reference the same BitBucket Repository under two different
names
The answer is Yes. Different alias can refer from the same repository.
enable custom triggering on Azure DevOps Services YAML
I read the YAML definition you provided, if you are talking about trigger of the resources section, then the answer is NO.
It should be pointed out that the usage you are using does not exist.
Please check these official articles, both of them mentioned this point:
resources.repositories.repository definition
trigger: trigger # CI trigger for this repository, no CI trigger if
skipped (only works for Azure Repos).
Triggers Usage in resources
Repository resource triggers only work for Azure Repos Git
repositories in the same organization at present. They do not work for
GitHub or Bitbucket repository resources.
So trigger via resources section of YAML is unable to achieve, you can only set YAML on bitbucket side and the condition should based on common trigger on bitbucket side.
I have a build that uses the Azure DevOps REST api to do analysis across a collection of repositories in a single Azure DevOps project.
To speed up the build, it only checks out a single repository containing certain build utilities, such as powershell scripts - the rest of the analysis is done via querying specific information via the REST api.
NOTE: This build is running on Azure DevOps Server 2020, which still calls the setting "Limit job authorization scope to referenced Azure DevOps repositories". Looking at doc history, I believe this is equivalent to "Protect access to repositories in YAML pipeline", I use the more recent term below.
This all worked fine until "Protect access to repositories in YAML pipelines" was turned on. With that setting turned on, the REST api only returns information about the repository containing the build utilities. This is due to the reduced scope of the Job Access Token (see Protect access to repositories in YAML pipelines)
I've attempted to create a template containing the list of all repositories, so that a few select builds can continue to easily access all repositories. Previously, no explicit list of repositories was needed, but now it appears they will have to manually be listed, and I'd like to do that in a single file.
Both yaml files below are in the same repository.
Template allRepos.yaml:
parameters:
- name: steps
type: stepList
default: []
jobs:
- job:
pool: 'swimming'
uses:
repositories:
- R1
- R2
- R3
- Rnnn
steps:
- ${{ parameters.steps }}
Yaml for pipeline:
extends:
template: allRepos.yaml
parameters:
steps:
- checkout: self
- task: PowerShell#2
displayName: multiRepoAnalysis
inputs:
filePath: analysis.ps1
env:
SYSTEM_ACCESSTOKEN: $(System.AccessToken)
It seems like this should work according to:
access tokens - Protect access to repositories in YAML pipelines)
repos - Protect access to repositories in YAML pipelines
jobs.job definition
However, attempting to run this pipeline results in the errors:
Job 'Job1' references the repository 'R1' which is not defined by the pipeline.
Job 'Job1' references the repository 'R2' which is not defined by the pipeline.
Job 'Job1' references the repository 'R3' which is not defined by the pipeline.
Job 'Job1' references the repository 'Rnnn' which is not defined by the pipeline.
How can I create a template that:
allows the access token for specific builds to access all repositories when "Protect access to repositories in YAML pipelines" is turned on
do so without checking out each repository
Limitations
This solution only works for a list of at most 20 repositories
- checkout: self counts toward the 20 repository limit
You must include submodule repositories in the list, which also counts toward the 20 repository limit.
Exceeding 20 repositories will still result in an error:
##[error]Tokens may be scoped to a maximum of 20 repositories. Reference fewer repositories in YAML or disable Scoped Tokens.
This limitation could be worked around by splitting the build into multiple definitions - but note if your analysis includes accessing submodule repositories, those submodule repositories must be referenced in any list of the primary repositories.
To my current knowledge, there is not a way to scope the access token for a particular pipeline with a generic "all repositories" permission.
Solution
The original templates were on the right track - but missed the fact that any uses: repositories must previously have been declared as a resources: repository.
There is a github issue to clarify the Microsoft documentation: uses: keyword is not explained, does not work as described
The working solution is:
Template allRepos.yaml:
parameters:
- name: steps
type: stepList
default: []
resources:
repositories:
- repository: R1
type: git
name: MyDevOpsProject/R1
- repository: R2
type: git
name: MyDevOpsProject/R2
- repository: R3
type: git
name: MyDevOpsProject/R3
- repository: Rnnn
type: git
name: MyDevOpsProject/Rnnn
jobs:
- job:
pool: 'swimming'
uses:
repositories:
- R1
- R2
- R3
- Rnnn
steps:
- ${{ parameters.steps }}
Yaml for pipeline:
extends:
template: allRepos.yaml
parameters:
steps:
- checkout: self
- task: PowerShell#2
displayName: multiRepoAnalysis
inputs:
filePath: analysis.ps1
env:
SYSTEM_ACCESSTOKEN: $(System.AccessToken)
Scenario:
I have a azure git repo for all my infrastructure code (basically all my yaml files are there and are triggered from there for any other jobs needed).
I want to be able to get all branches from repository x and use them as a value list in a parameter.
Basically:
resources:
repositories:
- repository: repo_x
type: git
name: Proj/repo_x
"""
Something should happen here and give me a list of branches from repo_x
in a variable 'branch_list'
"""
parameters:
- name: branch
displayName: Target branch
type: string
default: 'main'
values: [branch_list]
when we trigger a pipeline, the branch list is already available, this defines in which version the pipeline will run, is this what you want ?
I have a pipeline that has a section like this that lists the pipelines that would trigger the pipeline.
resources:
# List all the microservice pipelines to be watched plus infrastructure, the pipeline name is the name
# of the stack. Note template-maven and template-gradle are not to be part of this build.
pipelines:
- pipeline: auth
project: services
source: auth
branch: master
trigger:
branches:
include:
- master
- pipeline: ai
project: services
source: artificial-intelligence
branch: master
trigger:
branches:
include:
- master
- pipeline: ui
project: frontend
source: ui CI
branch: master
trigger:
branches:
include:
- master
I then have a job with the following steps (because deployment pulls all files, I just need one folder from each pipeline
- job: publishDeploymentPipelineFiles
condition: not(canceled())
steps:
- checkout: none
- download: auth
artifact: drop
- download: ai
artifact: drop
- download: ui
artifact: drop
What I am hoping for is some form of template that does
steps:
- checkout: none
- template: pull-deployment-manifests.yml
parameters:
sources:
- project: services
source: auth
stackName: auth
- project: services
source: artificial-intelligence
stackName: ai
- project: frontend
source: ui CI
stackName: ui
Which only lists the project and CI pipeline and create the appropriate pipeline ID from stackName and create the resources and the steps.
My workaround right now is to create a project that takes a CSV containing those items and generating the azure-pipelines.yml
As far as I know you can't dynamically create resources. So you create this
steps:
- checkout: none
- template: pull-deployment-manifests.yml
parameters:
sources:
- project: services
source: auth
stackName: auth
- project: services
source: artificial-intelligence
stackName: ai
- project: frontend
source: ui CI
stackName: ui
and run checkout inside the template unless you defined resources with those names on root level.
As documentation says here:
Resources are defined at one place and can be consumed anywhere in your pipeline.
Sure you can set up a template with resources, and use this template in a YAML pipeline. You can reference "Extend from a template with resources".
However, please note that if you have defined resources and steps in the template, you can't use it under the steps key in the YAML pipeline. You should use the extends key to extend the resources from the template, just like as the example shows in the document.
You may need to defined all the required steps in the template, or use the steps from other step template into the template.
Problem: We use github actions workflow for CI and we have many github repositories. I need to be able change everything repeatable for every repository at once.
Is it possible to use in github action workflow yml file some snippet that located mb in different repository.
You can include other public and local actions in your workflow, which lets you reuse common steps. Using versioned actions with {owner}/{repo}#{ref}:
steps:
- uses: actions/setup-node#74bc508 # Reference a specific commit
- uses: actions/setup-node#v1 # Reference the major version of a release
- uses: actions/setup-node#v1.2 # Reference a minor version of a release
- uses: actions/setup-node#master # Reference a branch
..or local actions with ./path/to/dir:
jobs:
my_first_job:
steps:
- name: Check out repository
uses: actions/checkout#v2
- name: Use local my-action
uses: ./.github/actions/my-action
https://help.github.com/en/actions/reference/workflow-syntax-for-github-actions#jobsjob_idstepsuses
One way of doing this is having a central CICD / GitHub actions repository with shared workflows which are triggered on repository_dispatch events.
on:
repository_dispatch:
types:
- your_event
jobs:
job1:
name: Do something
runs-on: ubuntu-latest
env:
SOURCE_BRANCH: ${{ github.event.client_payload.source_branch }}
SOURCE_REPO: ${{ github.event.client_payload.source_repo }}
# do all your stuff
Then in each github repo you write a small workflow file which outlines the triggers for the local repo, pushing to master / opening a PR etc. That action simply dispatches a repository_dispatch event to your central CICD repo with the repo and branchname it came from.
name: Trigger external CICD
on:
push:
branches:
- master
jobs:
trigger_cicd:
name: Trigger external CICD
runs-on: ubuntu-latest
steps:
- name: Send repository_dispatch event
uses: peter-evans/repository-dispatch#v1
with:
token: ${{ secrets.CICD_GITHUB_TOKEN }}
repository: yourorg/centralcicdrepo
event-type: ${{ env.EVENT_TYPE }}
client-payload: '{"source_branch": "${{ github.ref }}", "source_repo": "${{ github.repository }}" }'
One gotcha is that you need an access token to talk between repos, in the above example it's added as a secret called CICD_GITHUB_TOKEN. The easiest is to just use your own account but this will label all your central CICD runs as 'triggered by you'. You can also create a bot account or you can have each developer add their access tokens as secrets then map the right author to the right access token.
There is currently (Feb. 3, 2021) no supported method for reusing workflows or snippets from a centralized repository. There are hacks, as Michael Parker has cleverly demonstrated, but these come with significant downsides (eg. observability, opacity, etc.).
I've written this blog post that describes the problem you have in more detail, along with an open-source solution.
––
Similar topics:
DRYing GH Actions workflows
External workflow configuration
Bringing this issue to GH's attention:
Raise this issue with GH
GH Roadmap item