I'm new to Aure DevOps. Trying to create build and release pipelines there's one thing I don't understand:
Commonly, every kind of build finally results in some output, called artifacts.
With Azure DevOps it seems like there is always a final copy or publish task necessary to copy the created artifact from A to B, so the release task may then access the compiled artifacts.
Why aren't these artifacts plain accessible to a release pipeline right from the location where they have been built? Why don't the build tasks automatically set a variable pointing to the right folder, so the release pipeline may access the files right from there?
Or is this already happening and I'm just missing something from the tutorials I watched?
There are so many reasons.
Two easy ones:
There is no guarantee that the agent's working folder still contains the files. Agents are reused from build to build and release to release, and a given build or release will always use the same working folder. The working folder is cleaned up between builds.
Releases may run on different agents. On different machines. In different domains. Or any combination. There's no guarantee that the agent where the build ran is accessible by the agent where the release is running. Publishing the artifact allows a guarantee: As long as the machine the release is running on has the ability to talk to Azure DevOps (which is a requirement for the agent to function in the first place), it can get the artifacts it needs.
Why aren't these artifacts plain accessible to a release pipeline
right from the location where they have been built?
Agree with Daniel.
The main reason for me is because we can't hold the hosted agent all the time. Since MS wants to protect resources efficiently, it is not occupied for a long time.
When we queue a build, MS will assign us a brand new clean agent to execute our task, and after the build is complete, the MS will reclaim the agent assigned to our build and restore the agent to its initial state in preparation for accepting the assignment of the next task.
So, we could not keep hold the hosted agent to use it in next release pipeline. We have to store the artifacts in the cloud/server, then we could download it in the release pipeline. Otherwise, we could not get the artifact we need from an agent that has been restored.
Besides, MS is randomly assigned to the agent, and we cannot guarantee that the same agent will be allocated and built during the release pipeline.
That is the main reason why we need to copy or publish the artifacts.
If you do not want to copy or publish the artifacts, you could setup your own private agent, and do not clean the agent before you execute the release pipeline.
Update:
why is the user, well, bothered to find a place for the artifacts
manually? I would have expected every build pipeline to come with a
personal space to store the latest build artifacts. A space where
Azure DevOps automatically copies the build artifacts to. To me it
looks like things have to be manually copied from A to B and then
later from B to C.
That because not all output is needed, for example, the test project, what we need is test result/Code coverage for the test project, not the output for the solution. In this case, we do not need to copy the output to the artifacts. On the other hand, we need to copy some special files to artfacts, then automatically copy the build artifacts will not meet your requirements.
That is also the reason why we provide the task to copy files to artifacts, so that we could customize our personality needs.
Of course, if you think that manual copying is superfluous, you can use the MSBuild parameter /p:output=$(build.artifactstagingdirectory) to set the output directly to artifacts.
If I need to copy things from A to B in the Build pipeline, then what
should keep me from copying it to C right away? Then a separate
Release pipeline would be, well, rather optional, if not redundant.
If you are in the build pipeline, there is another task Download build artifacts, which could download the build artifacts.
if you are in the release pipeline, you just need select the build artifacts as source, release pipeline will download that artifact automatically:
Check this document for some more details.
Hope this helps.
Related
Can anybody tell me where the Release Artifact comes from when you use it in a Release Definition?
I’m currently using TFS on-prem (v15 Update 1) to handle undertake Build & Release. I believe this version is very close to what the current Azure DevOps documentation describes as ‘Classic’.
I have a Build Definition that outputs an Artifact to the default local path on the agent (c:\agent_work\1\a) using the Build.ArtifactStagingDirectory. I also publish/copy the artifact to a network share.
I have a Release Definition which references the Build Definition as a Linked Artifact Source. Therefore, when I create a new release where does the Release Definition get the Artifact Does it get it from my published network share? Does it get it from Source Control? How do I know it’s getting it from TFS or Network Share?
I know the Build.ArtifactStagingDirectory gets deleted on every build, so I know it can’t be from that location.
You tied your release definition to a build definition. Build definitions can publish artifacts when they run.
Your release definition looks at a build run, looks at the artifacts it published, and then retrieves them from whatever location to which they were published. This can be a file share, or it can be within the TFS/Azure DevOps instance itself. Not source control or Azure Artifacts, but rather a completely internal store just for builds.
Now to directly answer your question:
How do you know where the artifact came from? By looking at the build run. How do you know where the artifacts are going to be published in the future? By looking at the build definition.
I have a DACPAC deployment task which is failing, because for some reason the value of the $(Build.ArtifactStagingDirectory) pipeline variable is changing between the build pipeline and the release pipeline. In the build pipeline, the variable is set to C:\agent\_work\2\a, but in the release pipeline it's C:\agent\_work\r2\a. This is causing the release pipeline to fail when it tries to deploy the DACPAC artifact, because the folder it's looking in is empty; the folder where the artifact actually is is ignored. How do I make these variables consistent between the build and release pipelines so that the artifact is retrieved from the same folder it's generated in? These variables appear to be built in, so I don't see any way to change them. I could always hardcode a path, but that seems a bit kludgy...
In releases you have System.ArtifactsDirectory which is
The directory to which artifacts are downloaded during deployment of a release. The directory is cleared before every deployment if it requires artifacts to be downloaded to the agent. Same as Agent.ReleaseDirectory and System.DefaultWorkingDirectory.
Example: C:\agent_work\r1\a
and in pipelines/builds Build.ArtifactStagingDirectory
The local path on the agent where any artifacts are copied to before being pushed to their destination. For example: c:\agent_work\1\a
A typical way to use this folder is to publish your build artifacts with the Copy files and Publish build artifacts tasks.
Note: Build.ArtifactStagingDirectory and Build.StagingDirectory are interchangeable. This directory is purged before each new build, so you don't have to clean it up yourself.
See Artifacts in Azure Pipelines.
This variable is agent-scoped, and can be used as an environment variable in a script and as a parameter in a build task, but not as part of the build number or as a version control tag.
This is in line with your experience. And you cannot change it, as they are predefined. But can you clarify why this is a problem for you?
In the release pipeline, you can't directly access the files in the build pipeline, not only because the working directory is different, but also because they do not use the same agent. You need to download the artifacts first, and then use them in the release pipeline.
You can use the following ways to download artifacts:
Use the Download Build Artifacts task.
Go to the edit release pipeline page -> Select Add artifact -> Select Build -> Fill in the information related to the build pipeline (Notice the value of Source alias)-> Add it. You will find your artifacts are downloaded in $(System.ArtifactsDirectory)/${Source alias}
For more information about consuming artifacts in release pipelines, you can click this document.
Oh, I think I figured it out. Each release pipeline stage has an option called "Artifact download" which lets you specify which artifacts out of all those linked in the pipeline are actually used by that stage. I needed to check the appropriate checkboxes there in order to use the artifacts in the stage's tasks.
Azure DevOps release pipelines let me specify multiple input artifacts but automated triggering only happens for the primary artifact. Is there a way to do a 'wait all' type operation on N input artifacts so that my release pipeline only starts when each artifact has a version available that matches some specified value?
For instance, say I make a commit M in a repo which has multiple different CI jobs A and B which take very different amounts of time. One CI job (A) creates a NuGet package as an artifact in minutes, the other CI job (B) takes hours to create VM images. I want a release pipeline which will use the NuGet package and VM image as inputs. But if I simply use the latest version of A and B as an input artifact in a release pipeline with a CI trigger on the primary artifact (either A or B) then when the release triggers it could be using mismatched versions of artifacts as 10 NuGet packages from 10 different commits are complete before I even get 2 VM images made by the other CI pipeline. I thought maybe build/artifact tags could solve this so that my release waits for artifacts which all have the same tag but I can't use variables in build/artifact tag filters which makes this not really amendable to any sort of automated scheduled/trigger.
Is this some good pattern for this type of workflow?
For this issue , based on your workflow, I am afraid that it is impossible to achieve a fully automated release. Enabling the Continuous deployment trigger will create a new release every time when a new build is available. Since the two artifacts are not generated at the same time, the two matching artifacts cannot be used as the source in automatic release.
To use specific artifacts as source in a release, we usually specify the version in Artifact. But this can't achieve continuous deployment. One way is to merge jobs A and B to generate an artifact, so that you don’t need to worry about the corresponding issues of artifacts, but according to your actual situation, I ’m not sure about the feasibility of this.
do a 'wait all' type operation on N input artifacts so that my release
pipeline only starts when each artifact has a version available that
matches some specified value
Apart from the negative answer, I think what you want is a good idea! So I post a feature request here in DC forum. You could vote that suggestion ticket and share your comment there.The product team would provide the updates if they view it. Thank you for helping us build a better Azure DevOps.
I need to deploy many similar apps to the same environments with Azure DevOps.
What are some ways to share and maintain the release process for these apps....
to avoid updating every app's Release process when it changes?
Background
I have dozens of similar apps with identical release processes.
Each app (in its own repo) will have its own Azure Pipeline.
I'm fine with a custom solution to this problem.
Options
Make a Release for every Pipeline -- not a fan!
CON: Azure seems to want a 1-to-1 relationship between Pipelines and Releases.
CON: I want to avoid many Releases at all costs, since changes would be nearly unmaintainable.
Use custom stage templates -- doesn't work for what I need
CON: Releases do not share custom stage templates.
Custom templates that you create are scoped to the project that you created them in. [Azure Documentation]
CON: Custom stage templates cannot be updated (as of this post).
To update an stage template, delete the existing template in a release pipeline and then save the stage as a template with the same name. [Azure Documentation]
Put the release process in the Pipeline .yml file -- seems possible, but....
PRO: The process would be stored in a shared "common" repo.
PRO: Each app will have a minimal Pipeline .yml file, setting some parameters before running the shared process.
CON: How would I track and manually deploy releases without the Release portal UI? Am I missing something?
Trigger the same Release with artifacts from different apps -- not sure this will work....
PRO: One release process, easy to track and deploy as needed via the Release portal UI.
CON: How would I track releases for different apps? Would I name Releases with the app name?
CON: I don't see how to set it up, since a Release is tied to a primary Pipeline.
At the time of linking an artifact source to a release pipeline.... [Azure Documentation]
CON: Though multiple artifact sources can be used, Releases need a default artifact Pipeline or source repo. Would I need to setup the artifact source programmatically?
When you link multiple artifact sources to a release pipeline, one of them is designated as the primary artifact source. The primary artifact source is used to set a number of pre-defined variables. It can also be used in naming releases. [Azure Documentation]
Use multiple artifact sources and artifact variables -- not sure this will work....
When there are multiple artifact sources linked to a release pipeline, you can access information about each of these. [Azure Documentation]
Bottom Line
I was able to solve many of these problems in Octopus Deploy. However, I'm having significant trouble seeing how to move my DevOps process into Azure DevOps. How would you handle this situation?
There is no perfect solution to meet your requirements.
Generally deploy multiple apps to multiple environments, we recommend that Make a Release for every Pipeline. But just as you mentioned it has 1-to-1 relationship between build pipelines and Releases and it has many releases.
We can deploy multiple apps to multiple environments through ONE release with customized Template as you mentioned, we can also use the Task Groups to combine a set of tasks together, then reuse them in different stages. But we still need to update the definitions accordingly (change the Task Group version) once the specific task group is updated.
Another way is Clone Stage, you mentioned they are the same environments, so you can config one stage, then clone stages from the existing one, just name the stages with the specific App names accordingly. But you still need to change the settings accordingly, for example the Artifact filters (see Release triggers) to determine the condition to trigger the specific stage. But in this way it will download all the artifacts first when you add multiple artifact sources, this will take long time to get sources...
I have an ADO build pipeline that builds three artifacts. However, while I am developing, I only want one of the artifacts to be built since that is the artifact on which I am working and it takes longer for all of them to build and I don't want to wait. Can I disable the other artifacts from building?
As I know, you could not use UI to disable some of artifacts which build from one build pipeline. Because we does not provide this feature in VSTS now.
But as workaround, you can use Copy File task in your release to achieve this.
Specify the artifact what you want to work with in release pipeline, and copy it from File A(the source file where these build artifacts is) to File B(any file you want). And then, you can do other thing with this specific build.
Hope this could help -)