I need to modify an existing YAML pipeline so that it downloads an artifact published from another existing ADO pipeline. The other pipeline is a Classic one in case it matters.
In the current setup, a daily Release pipeline takes the artifact from the Classic pipeline and pushes it to a company repository external to ADO.
Now, the YAML pipeline is only run occasionally and it is run manually. Currently it downloads the artifact from the external repo to which the Release pipeline pushed. This hasn't been a problem generally. However a recent issue highlighted that it would be desirable to be able to avoid the delays built in to the current approach and essentially just grab the artifact directly from the latest run of the Classic pipeline.
When I set out to do this, I assumed that it would be straightforward but I seem to have run into a brick wall. The only information I have found describes using DownloadPipelineArtifact#2 but this depends on various IDs like the pipeline ID and run ID which it seems are not easily obtainable.
I am pretty new to ADO. I'm not a devops person at all really but I've had this put on my plate. So before I spend too much time on this, am I missing something or is this just something that one should not really be doing in ADO? If it is possible, is there a guide somewhere?
UPDATE
Thanks to a useful answer from daniel-mann, I was able to get this working but there were some quirks that I thought I should mention in the event that they might be useful to anyone else.
When I started adding the DownloadPipelineArtifact#2 task (this was editing directly in ADO on a browser), ADO was hinting field names to me that seemed to be different from the documented ones. Possibly these were aliases but I had a hard time knowing what to trust with respect to documentation.
I also noticed a Settings "link" had appeared above the first line of the task definition. When I clicked on this it opened up an editor to the right of the page that helped fill in the fields. It provided dropdowns for things like the project and the pipeline ID.
This what I ended up with:
- task: DownloadPipelineArtifact#2
displayName: "my task description"
inputs:
buildType: 'specific'
project: <long "UID" string identifying project>
definition: <numeric id for pipeline>
buildVersionToDownload: 'latest'
artifactName: <name of artifact as defined in upstream pipeline>
targetPath: '$(Pipeline.Workspace)'
Note that the editor tool added a definition field, but apparently this is an alias for pipeline. I am not sure why it thinks this is more helpful.
Unfortunately the above did not work. I saw this error:
##[error]Pipeline Artifact Task is not supported in on-premises. Please use Build Artifact Task instead.
I don't know what caused this - perhaps the ADO setup in my organization? As I understand it the Build Artifact Task is deprecated in favour of the Pipeline Artifact Task but I did not have any choice but to try it and this time it did work for me.
This time I used the "Settings" editor from the outset and ended up with this:
- task: DownloadBuildArtifacts#0
displayName: "my task description"
inputs:
buildType: 'specific'
project: $(System.TeamProjectId)
pipeline: <numeric ID as above>
buildVersionToDownload: 'latest'
downloadType: 'single'
artifactName: '$(ARTNAME)'
downloadPath: '$(System.ArtifactsDirectory)'
The fields that I manually edited here were:
using our own ARTNAME variable that is we define to be the artifact name in one of our variable groups. The relevant variable group is imported to this pipeline.
using the builtin System.TeamProjectId for the project name. This seemed prefereable to having the "UID" string in there. (Though I also found that the normal name string for our project worked here too.)
but this depends on various IDs like the pipeline ID and run ID
Not for your use case.
You said
just grab the artifact directly from the latest run of the Classic pipeline.
In which case, referring to the parameters explained in the documentation,
# Download pipeline artifacts
# Download build and pipeline artifacts
- task: DownloadPipelineArtifact#2
inputs:
#source: 'current' # Options: current, specific
#project: # Required when source == Specific
#pipeline: # Required when source == Specific
#preferTriggeringPipeline: false # Optional
#runVersion: 'latest' # Required when source == Specific# Options: latest, latestFromBranch, specific
#runBranch: 'refs/heads/master' # Required when source == Specific && RunVersion == LatestFromBranch
#runId: # Required when source == Specific && RunVersion == Specific
#tags: # Optional
#artifact: # Optional
#patterns: '**' # Optional
#path: '$(Pipeline.Workspace)'
You would just need to set the project, pipeline, source: specific, and runVersion: latest parameters.
Or you could use the download alias, which is a little bit simpler but can achieve the same thing
Related
The problem i am running into is perfectly explained here, but at the time the user was pointed to another forum and no solution is written, if even available...
https://github.com/microsoft/azure-pipelines-yaml/issues/459
In short, my problem is that I have defined a pipeline resource in a YAML Pipeline.
pipelines:
- pipeline: MainRebuild
project: ProjectName
source: 'Main - Rebuild'
branch: feature/DummySmokeTest
trigger:
branches:
- master
- feature/DummySmokeTest
The trigger works great, if the MainRebuild completes as Successful or PartiallySucceeded, a new pipeline is triggered that picks up the right version and can download the right artifacts.
The problem is when queueing the pipeline manually. The default resource configured is "Last successful run", which is indeed what i am looking for.
Only my last run has a result of PartiallySucceeded.
When I do not touch the resource and trigger the build, the build will pick up the latest build with state successful, not the very latest PartiallySucceeded build.
One workaround I found is that when triggering the build manually, I can choose a "different" resource. The pipeline nicely shows me all the Successful and PartiallySucceeded builds and I can choose one. I can choose the latest and say Use the selected run, the pipeline will then also use that properly.
So that is a workaround, but I would like to fix the default behavior of the latest successful to include the partially succeeded builds.
Does anyone know if this is possible?
Update 1:
This my Download build task
# Download build and pipeline artifacts
- task: DownloadBuildArtifacts#0
inputs:
buildType: 'specific'
project: ProjectName
pipeline: '$(RESOURCES.PIPELINE.MAINREBUILD.PIPELINEID)'
specificBuildWithTriggering: true
buildVersionToDownload: specific
buildId: '$(RESOURCES.PIPELINE.MAINREBUILD.RUNID)'
allowPartiallySucceededBuilds: true
downloadType: specific
itemPattern: |
**\Installers\*.exe
downloadPath: '$(System.ArtifactsDirectory)'
The docs for the DownloadPipelineArtifact#2 says you can use a option in the task to disable download of partially succeeded builds. I assume this is the task you are using to grab the artifact?
https://learn.microsoft.com/en-us/azure/devops/pipelines/tasks/utility/download-pipeline-artifact?view=azure-devops#arguments
allowPartiallySucceededBuilds
Download artifacts from partially succeeded builds (Optional) If checked, this build task will try to download artifacts whether the build is succeeded or partially succeeded
Default value: false
I am new to Azure pipelines. My team has inherited an existing pipeline, and I am just trying to figure out what these specific settings are doing.
We have a stage with a job with a task:
task: DownloadBuildArtifacts#0
inputs:
buildType: 'specific'
specificBuildWithTriggering: true
buildVersionToDownload: 'latestFromBranch'
branchName: '<path-to-branch>'
There are more settings of course, but I'm really trying to wrap my head around the "buildType: specific" one. The docs say "Download artifacts produced by the current build, or from a specific build." But I'm not sure how we're specifying "specific". Is it the specificBuildWithTriggering setting? Or is the the buildVersionToDownload setting?
Trying to read this (and ignoring "specific"), it looks like we're saying "build with artifacts from the triggering commit, but also get the latest". To me, those could be different, if 2 builds are triggered by rapid commits. Am I reading that right? What happens if there is a conflict?
Finally, the default for buildType is 'current'. To me this sounds like, "just download the current from the branch" ... which to me also sounds like I could skip 3 of those settings, right?
Thanks for any clarity the community can provide!
I have an Azure Pipeline which
is triggered manually
downloads the build artifacts of 3 other pipelines, all of them based on different (git) repositories
allows the user to input the BuildId (Run number) of these pipeline-artifacts, to choose which runs to take them from
creates a package with them
is written in YAML
I'm looking for a way to show in the "Source Code" tab the Source code link for the commit relative to each of pipeline builds, which artifacts have been downloaded during this run.
To do this, I have to name each of the repositories and checkout the proper version.
What I can't get my head around is how to exploit the BuildId variable to get the SourceVersion variable.
I know that Build.BuildId variable is the one defining the run id of the pipeline, and we use this to choose which run to take the specific artifact from.
At the same time, Build.SourceVersion contains the commit Id used for the pipeline run. But normally, Build is the current Build.
How can I reference Build_x, starting from Build_x.BuildId, so to be able to recover the Build_x.SourceVersion?
Thank you
Based on your requirement, I suggest that you can change use the Pipelines Resource in YAML pipeline.
Here is an example:
resources:
pipelines:
- pipeline: PipelineAlisa
project: project
source: Pipelinename
- pipeline: PipelineAlisa
project: proejct
source: Pipelinename
pool:
vmImage: windows-latest
steps:
- download: MyCIAlias
- download: MyCIAlias1
In this case, you can still select the Build runs when your run the pipeline(Resources option).
This should also be able to achieve the same function as your existing pipeline.
And this method can more conveniently obtain the corresponding build id and source version of pipeline artifacts.
You can directly use the variables:
Source version : RESOURCES.PIPELINE.Aliasname.SOURCECOMMIT
Build ID: RESOURCES.PIPELINE.Aliasname.RUNID
For example:
I have a pipeline with with some stages like
Build -> Dev -> Test -> Prod
The build stage generate some nugets for different parts of the system. As this happen in different jobs, a couple of artifact names is published like:
server\server.nupkg
client\client.nupkg
auth\auth.nupkg
Those are then consumed in the following stages that dowload, apply variable substitution to some configs and publish a new nuget configured to an environment as a drop.
For example the test stage has one job that transform the config in the needed nuget and publish that as an pipeline artifact. So at that point the artifacts for the hole pipeline looks like:
server\server.nupkg
client\client.nupkg
auth\auth.nupkg
Test
server.nupkg
client.nupkg
auth.nupkg
When it comes to the prod stage i want to download all *.nupkg from all artifact names except from the Test artifact name.
I've tried to use an exclude patterns in the 'DownloadPipelineArtifact' task like this but no success:
- task: DownloadPipelineArtifact#2
displayName: Download nugets
inputs:
buildType: 'current'
itemPattern: |
'*/*.nupkg'
'!Test'
'!Test/*.nupkg'
targetPath: '$(MyDirectory)/nugets'
Any ideas?
If server, client, auth and Test are different artifacts then you should be able to either download the first three separately or do it in a single task invocation by using "exclude" file matching pattern with first segment being the artifact name. See Publish and download artifacts in Azure Pipelines | Multiple artifacts.
Example #1, download artifacts separately:
- download: current
artifact: server
- download: current
artifact: client
- download: current
artifact: auth
Although, based on the current docs, the following example is supposed to work, it doesn't. It looks like the docs are incorrect. See this issue for details:
https://github.com/microsoft/azure-pipelines-agent/issues/3416
Example #2, download artifacts using file matching patterns:~~
Note double asterisk for recursive wildcard and the absence of the quotes that would be included literally in the multiline YAML string otherwise.
- download: current
patterns: |
**/*.nupkg
!Test/**
This is an ASP.NET Core 3.0 project that builds with no errors, but when it triggers the pipeline to release to Azure App Service, it fails with the following error:
2019-11-10T23:09:23.8008460Z ##[error]Error: No package found with specified pattern: D:\a\r1\a***.zip
What needs to be done to fix the release pipeline? The pipeline release is pulling the latest build as its artifact.
Assumptions
The following info assumes that you are appropriately publishing your build artifact from your Build pipeline, and that you have added the correct build artifact into you release pipeline.
In your release pipeline you have specified a build artifact in the Artifacts area
When adding your build artifact to your release pipeline, you chose to give it an alias of Build Artifact. This means that at the very lease (with default settings) your .zip file will be in some sub-directory of $(system.DefaultWorkingDirectory)/Build Artifact/
A new unique folder in the agent is created for every release pipeline when you initiate a release, and the artifacts are downloaded into that folder. The $(System.DefaultWorkingDirectory) variable maps to this folder.
To ensure the uniqueness of every artifact download, each artifact source linked to a release pipeline is automatically provided with a specific download location known as the source alias. This location can be accessed through the variable:
$(System.DefaultWorkingDirectory)\[source alias]
This uniqueness also ensures that, if you later rename a linked artifact source in its original location (for example, rename a build pipeline in Azure Pipelines or a project in Jenkins), you don't need to edit the task properties because the download location defined in the agent does not change.
The source alias is, by default, the name of the source selected when you linked the artifact source, prefixed with an underscore; depending on the type of the artifact source this will be the name of the build pipeline, job, project, or repository. You can edit the source alias from the artifacts tab of a release pipeline; for example, when you change the name of the build pipeline and you want to use a source alias that reflects the name of the build pipeline.
(from some of the abundant documentation
Instead of searching for your package using ***.zip (which isn't proper wildcard syntax) use Build Artifact/**/*.zip
** is for recursively searching into directories
(I don't know what folder)
* is for searching a part of a given level of the path
any file/folder that
starts with (SomeFile.*)
ends with (*File.zip)
and i think contains (*meFi*)
The pipeline YAML was missing the following tasks. Not sure why this isn't included in the ASP.NET Core template, very confusing for developers new to Azure DevOps.
- task: DotNetCoreCLI#2
inputs:
command: 'publish'
publishWebProjects: true
- task: CopyFiles#2
inputs:
targetFolder: '$(Build.ArtifactStagingDirectory)'
- task: PublishBuildArtifacts#1
inputs:
PathtoPublish: '$(Build.ArtifactStagingDirectory)'
ArtifactName: 'drop'
publishLocation: 'Container'
When adding an Artifact of Source type "Build", select "Default version" as "Latest from the build pipeline default branch with tags", as follows: