I work in a platform team that builds and supports Azure Pipelines YAML templates and various custom scripting for these pipelines. People in my organisation use these YAML templates like:
resources:
repositories:
- repository: templates
type: git
name: PROJECT/cicd_stuff
ref: refs/heads/releases/v0.21
extends:
template: our_awesome_template.yml#templates
parameters:
...
In order to reliably match the version of our scripts to the version of the YAML templates, we now include an extra build stage in our pipeline templates that checks out the repo templates repo and puts all our scripts in an artifact to be used in that run. Using "release branches" allows us to safely put out and test new features and fixes to our pipelines: Teams can upgrade their pipeline on their own pace, with older version remaining supported for a while.
I'd like to start using Azure Artifacts for our script artifacts, but now I'm wondering "how can I determine which version of my scripts I should be downloading from Azure Artifacts?" The information included in the resources block would work for me, but I can't seem to access that with an expression or with a predefined variable. The only solution I can currently think of is to use the az pipelines cli. Any thoughts?
how can I determine which version of my scripts I should be downloading from Azure Artifacts?
If the feature verion is the target version, you could try the following yaml to get its value. See: Repository details for details.
resources:
repositories:
- repository: templates
type: git
name: PROJECT/cicd_stuff
ref: releases/v0.21
variables:
tools.ref: $[ resources.repositories['templates'].ref ]
pool:
vmImage: ubuntu-latest
steps:
- bash: |
echo "Tools version: $TOOLS_REF"
Also you could try Daniel's solution using tagref: refs/tags/v1.0 # optional ref to pin to by reference to this doc: Use other repositories and Checking out a specific ref.
Update>>Currently there is no available predefined variables regarding to the name that was used to include the repository resource ('templates' in this case). If we know the repository alias, the repository name can be parsed via $[ resources.repositories['templates'].name].
Another finding is that the repository resource will be added as the build artifacts, and we can get it from API:GET https://dev.azure.com/{organization}/{project}/_build/results?buildId={buildId}&view=artifacts&__rt=fps&type=consumedArtifacts(I grab this API from browser developer tool). And buildId can be got using variable Build.BuildId. See: Build variables (DevOps Services) for details. From the response, search consumedSources field to find below json segment, which will return all artifacts, you could find repository resource and all its detailed information.
Related
I have an Azure Pipeline which
is triggered manually
downloads the build artifacts of 3 other pipelines, all of them based on different (git) repositories
allows the user to input the BuildId (Run number) of these pipeline-artifacts, to choose which runs to take them from
creates a package with them
is written in YAML
I'm looking for a way to show in the "Source Code" tab the Source code link for the commit relative to each of pipeline builds, which artifacts have been downloaded during this run.
To do this, I have to name each of the repositories and checkout the proper version.
What I can't get my head around is how to exploit the BuildId variable to get the SourceVersion variable.
I know that Build.BuildId variable is the one defining the run id of the pipeline, and we use this to choose which run to take the specific artifact from.
At the same time, Build.SourceVersion contains the commit Id used for the pipeline run. But normally, Build is the current Build.
How can I reference Build_x, starting from Build_x.BuildId, so to be able to recover the Build_x.SourceVersion?
Thank you
Based on your requirement, I suggest that you can change use the Pipelines Resource in YAML pipeline.
Here is an example:
resources:
pipelines:
- pipeline: PipelineAlisa
project: project
source: Pipelinename
- pipeline: PipelineAlisa
project: proejct
source: Pipelinename
pool:
vmImage: windows-latest
steps:
- download: MyCIAlias
- download: MyCIAlias1
In this case, you can still select the Build runs when your run the pipeline(Resources option).
This should also be able to achieve the same function as your existing pipeline.
And this method can more conveniently obtain the corresponding build id and source version of pipeline artifacts.
You can directly use the variables:
Source version : RESOURCES.PIPELINE.Aliasname.SOURCECOMMIT
Build ID: RESOURCES.PIPELINE.Aliasname.RUNID
For example:
I would like to run a yaml pipeline from one project. I have a task in my yaml to scan all the source code. Using this Yaml I would like to scan all the source code in master branch for all the project and all the repository inside the same Org.
How can I get all the repo for all the project and iterate? Can someone help me ?
test.yaml
repositories:
- repository: justAnotherName
type: github
name: myGitRepo
endpoint: myGitServiceConnection
trigger:
branches:
include:
- master
steps:
- task: CredScan#2
inputs:
toolMajorVersion: 'V2'
outputFormat: 'tsv'
scanFolder: '$(Build.SourcesDirectory)'
If you're looking to pull every repo within a project, you have one of two options (see below). However, I'd advise caution before attempting this on a Microsoft-hosted agent, they have a 60-minute timeout by default. If you're using a self-hosted agent, you need not worry. I'd still advise breaking this up to avoid creating a long-running release that also consumes a large amount of disk space with each run.
https://learn.microsoft.com/en-us/azure/devops/pipelines/process/phases?view=azure-devops&tabs=yaml#timeouts
That being said, here are the options you have:
Option 1 (Not the best)
Manually add a repository: dependency for every project and a checkout: task for every repo within the projects.
This is heavily manual and would require maintenance every time a report is added.
Option 2
You can write a custom PowerShell/bash script that uses the Azure DevOps API and git to automatically scan all projects and repos within the org and pull them onto the machine.
Start by issuing a request to get all of the projects within the org:
https://learn.microsoft.com/en-us/rest/api/azure/devops/core/projects/list?view=azure-devops-rest-6.0
Then, iterate through every project and get all repos:
https://learn.microsoft.com/en-us/rest/api/azure/devops/git/repositories/list?view=azure-devops-rest-6.0
Finally, iterate through each repo and run git clone [repository URL] to clone it onto the build agent.
NOTE: You will want to ensure to have a lot of free disk space on the agent machine and that you clean up the build space after this operation.
We are having multiple projects in Azure devops.
We would like to access the pipelines created across multiple projects.
Is that possible to access pipeline across projects or the pipelines are project specific/restricted?
Thanks
Naresh Ede
If you use yaml pipelines and rely on multiple repositories which are stored in multi-projects, please refer to this doc: Check out multiple repositories in your pipeline to implement this requirement.
In addition, please disable “Limit job authorization scope to referenced Azure DevOps repositories” option in Project settings page by reference to this doc. Please note that there are also other limitation settings listed here: Job authorization scope.
If you use classic editor pipelines, please specify the authorization scope for a build job as “Project Collection”, see: Build job authorization scope for details. And you need to manually checkout other repositories in pipeline tasks.
For YAML Pipelines here is the code you'd need:
resources:
repositories:
- repository: repositoryReferenceName
type: git
name: ProjectName/RepositoryName
And to reference a template it would be
- template: templateName.yml#repositoryReferenceName
I've read the official documents to put the yaml file in the root of a project. I'm thinking to create a some sort of pipeline repo that contains several yaml files in charge of different pipeline workflow for different project. But Azure pipeline only recognise the azure-pipeline.yaml file name.
Issue:
It is obviously not possible to create several yaml files with the same azure-pipeline.yaml name under the same folder. What's the best practice to organise the azure pipeline yaml files? Shall it be just put in the root of the project?
It sounds like templates might be what you're looking for. This assumes you have a single project/repo and a large pipeline that you'd like to split up so it's easier to read or reason about individual parts.
Taking an example from the linked documentation page, you can define a template yaml file like this (ex: include-npm-steps.yml):
steps:
- script: npm install
- script: yarn install
- script: npm run compile
And then include it as a "module" in the main azure-pipelines.yml file like this:
jobs:
- job: Linux
pool:
vmImage: 'ubuntu-latest'
steps:
- template: templates/include-npm-steps.yml # Template reference
- job: Windows
pool:
vmImage: 'windows-latest'
steps:
- template: templates/include-npm-steps.yml # Template reference
It is obviously not possible to create several yaml files with the
same azure-pipeline.yaml name under the same folder.
Yes, it's not possible to create several yaml pipelines with same name under same folder. Cause the yaml pipeline is under version control and Azure Devops git doesn't support two files with same name in same folder...
What we can do is to create several pipelines with different names in same folder, like azure-pipeline.yaml,azure-pipelines-1.yml,azure-pipelines-2.yml and so on.
Not sure if you know this option when editing yaml pipeline:
We can easily change the yaml file's name in source control, and we just need to modify the path here:
What's the best practice to organise the azure pipeline yaml files?
Shall it be just put in the root of the project?
Assuming you own one Team Project with two repos A and B:
If A and B both represent the module of one final product, then you should have corresponding pipelines for A and B. It means in most scenarios, you should have at least one pipeline in RepoA and one in RepoB. They all need corresponding azure-pipeline.yaml file.
Now if azure-pipeline.yaml in RepoA and azure-pipeline.yaml in RepoB have many same variables/tasks/jobs, we can consider moving the duplicate contents into templates. We can create a RepoC in same project to store the templates, and in this templates repo, we don't need to create yaml pipeline here.
About how to reference templates in RepoC in RepoA's pipeline, see this document. If the source is in github, you can check Krzysztof's link. And if the RepoC is in Azure Devops Repos and same project with your RepoA and RepoB, you can should this format:
resources:
repositories:
- repository: templates
type: git
name: RepoC
ref: refs/heads/master
To sum up, functional repos (those with source code) should have corresponding yaml pipeline in it. And if you want to monitor the changes in one repo (without source code) for some purpose, you can also have one yaml pipeline in that. For templates repo, yaml pipelines are not necessary.
Also, apart from yaml pipelines you may sometimes use Classic Build/Release pipelines which are not under Version Control. See this.
I have a project on Azure DevOps containing multiple forks of the same main repository. I created a build pipeline for that repository which unfortunately cannot be reused for the present forks since a pipeline can only be configured for a single repository.
This solution is not ideal because leads to multiple identical pipelines, one for each fork, and maintaining all of them can be difficult.
Is there a way to use one pipeline for multiple repositories?
you can create a template file and reference that file from each pipeline, that way you can edit a single file and every pipeline will change.
example how to reuse a step file from different repo
resources:
repositories:
- repository: DevOps
type: git
name: DevOps
trigger: none
jobs:
- template: vsts/yaml/build.yaml#DevOps
parameters:
solutionName: xxx
registryName: yyy
You can take a look at the official docs for more examples
https://learn.microsoft.com/en-us/azure/devops/pipelines/process/templates?view=azure-devops
It's on the roadmap for 2019 Q3:
Multi-repository support for YAML pipelines
https://dev.azure.com/mseng/AzureDevOpsRoadmap/_workitems/edit/1454026
Update: this is now implemented:
https://learn.microsoft.com/en-us/azure/devops/pipelines/repos/multi-repo-checkout?view=azure-devops#triggers
"Triggers
You can trigger a pipeline when an update is pushed to the self repository or to any of the repositories declared as resources."