I have an Azure Pipeline which
is triggered manually
downloads the build artifacts of 3 other pipelines, all of them based on different (git) repositories
allows the user to input the BuildId (Run number) of these pipeline-artifacts, to choose which runs to take them from
creates a package with them
is written in YAML
I'm looking for a way to show in the "Source Code" tab the Source code link for the commit relative to each of pipeline builds, which artifacts have been downloaded during this run.
To do this, I have to name each of the repositories and checkout the proper version.
What I can't get my head around is how to exploit the BuildId variable to get the SourceVersion variable.
I know that Build.BuildId variable is the one defining the run id of the pipeline, and we use this to choose which run to take the specific artifact from.
At the same time, Build.SourceVersion contains the commit Id used for the pipeline run. But normally, Build is the current Build.
How can I reference Build_x, starting from Build_x.BuildId, so to be able to recover the Build_x.SourceVersion?
Thank you
Based on your requirement, I suggest that you can change use the Pipelines Resource in YAML pipeline.
Here is an example:
resources:
pipelines:
- pipeline: PipelineAlisa
project: project
source: Pipelinename
- pipeline: PipelineAlisa
project: proejct
source: Pipelinename
pool:
vmImage: windows-latest
steps:
- download: MyCIAlias
- download: MyCIAlias1
In this case, you can still select the Build runs when your run the pipeline(Resources option).
This should also be able to achieve the same function as your existing pipeline.
And this method can more conveniently obtain the corresponding build id and source version of pipeline artifacts.
You can directly use the variables:
Source version : RESOURCES.PIPELINE.Aliasname.SOURCECOMMIT
Build ID: RESOURCES.PIPELINE.Aliasname.RUNID
For example:
Related
I am having an azure build pipeline that builds an app - let's call it App2 - and includes the output of another build that has build another app - App1 - where App2 is depending on. But I do not want to include App1 builds based on a pull-request (PR), only want a build based on the code in the branch the PR merges to.
So, how do I exclude a PR build as resource to a dependent pipeline?
Update
It appears to be fixed by setting the branch parameter to development on the pipeline resource section at the start of our build pipeline yaml. For example:
- pipeline: pipeline_reference_name
source: pipeline_name
branch: development
I have a YAML build pipeline which resides on the repo PipelineRepo, branch master.
It contains a reference to a second repo AppRepo with a trigger on branch dev.
When a commit is made on AppRepo/dev, the build triggers and produces an artifact. The Build.SourceBranch predefined variable available during the build is of course dev as this was the triggering branch on AppRepo. This is as per the docs here:
https://learn.microsoft.com/en-us/azure/devops/pipelines/repos/multi-repo-checkout?view=azure-devops#triggers
The specific part of the docs is this:
When an update to one of the repositories triggers a pipeline, then
the following variables are set based on triggering repository:
Build.Repository.ID
Build.Repository.Name
Build.Repository.Provider
Build.Repository.Uri
Build.SourceBranch
Build.SourceBranchName
Build.SourceVersion
Build.SourceVersionMessage
I consume the artifact in a Release pipeline, where I have a "Dev" stage with an artifact filter for branch dev:
I have a continuous deployment trigger on the artifact for branch dev:
Now we come to the problem
Every time a new build artifact is produced from the AppRepo/dev branch, the continuous deployment trigger doesn't fire because it thinks the build branch of the artifact was PipelineRepo/master. This is also true of the artifact filter on the stage - I have tested it by changing the continuous deployment trigger to master.
Note: you can see on the screenshot the build name contains the word "dev". This is because I use the Build.SourceBranch var in my custom build name. This proves the artifact is definitely produced by the AppRepo/dev triggering branch.
How can I get the DevOps Release Pipeline to pick up the triggering branch?
Based on your description, I could reproduce the similar issue in my organization.
When the pipeline is triggered by another repo branch, it still show the master branch in Release artifacts.
At the same time, the display of the trigger branch is also inconsistent in Pipelines.
For example:
Overview:
Detail view:
I suggest that you could create a feedback ticket in Our feedback Site.
For a workaround:
You could add a build tag in Pipeline to distinguish artifacts.
Yaml sample:
resources:
repositories:
- repository: test
type: git
name: 123/ARM
trigger:
- test
steps:
- checkout: test
- script: echo "##vso[build.addbuildtag]$(Build.SourceBranch)"
Release Pipeline:
You could set the master branch and add the build tag.
I have a pipeline with with some stages like
Build -> Dev -> Test -> Prod
The build stage generate some nugets for different parts of the system. As this happen in different jobs, a couple of artifact names is published like:
server\server.nupkg
client\client.nupkg
auth\auth.nupkg
Those are then consumed in the following stages that dowload, apply variable substitution to some configs and publish a new nuget configured to an environment as a drop.
For example the test stage has one job that transform the config in the needed nuget and publish that as an pipeline artifact. So at that point the artifacts for the hole pipeline looks like:
server\server.nupkg
client\client.nupkg
auth\auth.nupkg
Test
server.nupkg
client.nupkg
auth.nupkg
When it comes to the prod stage i want to download all *.nupkg from all artifact names except from the Test artifact name.
I've tried to use an exclude patterns in the 'DownloadPipelineArtifact' task like this but no success:
- task: DownloadPipelineArtifact#2
displayName: Download nugets
inputs:
buildType: 'current'
itemPattern: |
'*/*.nupkg'
'!Test'
'!Test/*.nupkg'
targetPath: '$(MyDirectory)/nugets'
Any ideas?
If server, client, auth and Test are different artifacts then you should be able to either download the first three separately or do it in a single task invocation by using "exclude" file matching pattern with first segment being the artifact name. See Publish and download artifacts in Azure Pipelines | Multiple artifacts.
Example #1, download artifacts separately:
- download: current
artifact: server
- download: current
artifact: client
- download: current
artifact: auth
Although, based on the current docs, the following example is supposed to work, it doesn't. It looks like the docs are incorrect. See this issue for details:
https://github.com/microsoft/azure-pipelines-agent/issues/3416
Example #2, download artifacts using file matching patterns:~~
Note double asterisk for recursive wildcard and the absence of the quotes that would be included literally in the multiline YAML string otherwise.
- download: current
patterns: |
**/*.nupkg
!Test/**
I work in a platform team that builds and supports Azure Pipelines YAML templates and various custom scripting for these pipelines. People in my organisation use these YAML templates like:
resources:
repositories:
- repository: templates
type: git
name: PROJECT/cicd_stuff
ref: refs/heads/releases/v0.21
extends:
template: our_awesome_template.yml#templates
parameters:
...
In order to reliably match the version of our scripts to the version of the YAML templates, we now include an extra build stage in our pipeline templates that checks out the repo templates repo and puts all our scripts in an artifact to be used in that run. Using "release branches" allows us to safely put out and test new features and fixes to our pipelines: Teams can upgrade their pipeline on their own pace, with older version remaining supported for a while.
I'd like to start using Azure Artifacts for our script artifacts, but now I'm wondering "how can I determine which version of my scripts I should be downloading from Azure Artifacts?" The information included in the resources block would work for me, but I can't seem to access that with an expression or with a predefined variable. The only solution I can currently think of is to use the az pipelines cli. Any thoughts?
how can I determine which version of my scripts I should be downloading from Azure Artifacts?
If the feature verion is the target version, you could try the following yaml to get its value. See: Repository details for details.
resources:
repositories:
- repository: templates
type: git
name: PROJECT/cicd_stuff
ref: releases/v0.21
variables:
tools.ref: $[ resources.repositories['templates'].ref ]
pool:
vmImage: ubuntu-latest
steps:
- bash: |
echo "Tools version: $TOOLS_REF"
Also you could try Daniel's solution using tagref: refs/tags/v1.0 # optional ref to pin to by reference to this doc: Use other repositories and Checking out a specific ref.
Update>>Currently there is no available predefined variables regarding to the name that was used to include the repository resource ('templates' in this case). If we know the repository alias, the repository name can be parsed via $[ resources.repositories['templates'].name].
Another finding is that the repository resource will be added as the build artifacts, and we can get it from API:GET https://dev.azure.com/{organization}/{project}/_build/results?buildId={buildId}&view=artifacts&__rt=fps&type=consumedArtifacts(I grab this API from browser developer tool). And buildId can be got using variable Build.BuildId. See: Build variables (DevOps Services) for details. From the response, search consumedSources field to find below json segment, which will return all artifacts, you could find repository resource and all its detailed information.
I would like to run a yaml pipeline from one project. I have a task in my yaml to scan all the source code. Using this Yaml I would like to scan all the source code in master branch for all the project and all the repository inside the same Org.
How can I get all the repo for all the project and iterate? Can someone help me ?
test.yaml
repositories:
- repository: justAnotherName
type: github
name: myGitRepo
endpoint: myGitServiceConnection
trigger:
branches:
include:
- master
steps:
- task: CredScan#2
inputs:
toolMajorVersion: 'V2'
outputFormat: 'tsv'
scanFolder: '$(Build.SourcesDirectory)'
If you're looking to pull every repo within a project, you have one of two options (see below). However, I'd advise caution before attempting this on a Microsoft-hosted agent, they have a 60-minute timeout by default. If you're using a self-hosted agent, you need not worry. I'd still advise breaking this up to avoid creating a long-running release that also consumes a large amount of disk space with each run.
https://learn.microsoft.com/en-us/azure/devops/pipelines/process/phases?view=azure-devops&tabs=yaml#timeouts
That being said, here are the options you have:
Option 1 (Not the best)
Manually add a repository: dependency for every project and a checkout: task for every repo within the projects.
This is heavily manual and would require maintenance every time a report is added.
Option 2
You can write a custom PowerShell/bash script that uses the Azure DevOps API and git to automatically scan all projects and repos within the org and pull them onto the machine.
Start by issuing a request to get all of the projects within the org:
https://learn.microsoft.com/en-us/rest/api/azure/devops/core/projects/list?view=azure-devops-rest-6.0
Then, iterate through every project and get all repos:
https://learn.microsoft.com/en-us/rest/api/azure/devops/git/repositories/list?view=azure-devops-rest-6.0
Finally, iterate through each repo and run git clone [repository URL] to clone it onto the build agent.
NOTE: You will want to ensure to have a lot of free disk space on the agent machine and that you clean up the build space after this operation.