I use azure pipeline for build my solution.
Aftere build I need to generate and upload multiple packages with different assets. Packing implemented as a number of additional tasks in my agent job.
But I need ability to generate only selected packages or all packages based on specified arguments.
What is the best way to achieve this?
Ideally, it would have 2 pipelines. The first is automatic project build. And the second should use the result of the first and be able to repeatedly start manually with the desired parameters to exclude project rebuild. But I do not know how this can be implemented.
Not sure if I understand the question correctly, but two possible answers would be:
Classic pipelines, with build pipeline for project build and release pipeline for uploading the artifact(s) that build generates, if using release pipeline is applicable. Release pipeline can have a cd-trigger for the first run, and redeployed manually after that. If need to change release variables for subsequent deployments, you can create a new release with the same build artifact.
Multi-staged pipeline, with build and upload as different stages, manually redeploy/rerun the upload stage when needed. Build phase generates deployable pipeline artifact(s).
Somehow I think you're looking for more elaborate solution, as you state that you're already using pipelines. So how about creative use of conditional tasks (https://learn.microsoft.com/en-us/azure/devops/pipelines/process/conditions?view=azure-devops&tabs=yaml) using pre-defined variables like Build.Reason (with value 'Manual') to exclude the execution of certain tasks in some runs. Then group tasks you want to re-use into either build templates or task groups.
Related
Need your inputs as I am new to Azure DevOps. Is it possible to have multiple release pipeline & single build pipeline in Azure? As per the best practices, if within a single solution, there are multiple projects, it's best to have single build pipeline per project & then create corresponding release pipelines. Now information I have received from developers, their application is so integrated that it is very difficult to seperate out projects to their individual solutions. They are suggesting to create multiple release pipelines for windows services & web services for various projects.
is it still possible?
Is it possible to have multiple release pipeline & single build pipeline in Azure?
The answer is yes.
There is no one-to-one correspondence between the build pipeline and the release pipeline. We can have multiple release pipeline & single build pipeline, or have multiple build pipeline & single release pipeline. It all depends on your needs.
But when we are dealing with these special situations, we need to be very clear about the output of the build pipeline and the generation of artifacts or the input of the release pipeline and the deployment of artifacts, otherwise it is easy to cause confusion.
For your situation, I have create a simple sample to explain the process.
In the build pipeline, I have two projects in my build pipeline. Build the solution (or all projects) by Visual Studio build task, then we need create two artifacts for each projects:
Then publish the build artifacts with different artifact name:
So we get two artifact for one build pipeline:
Last but not least, we create multiple release pipeline (or we could create multiple stages) for windows services & web services. Release pipeline will download two artifacts from the build pipeline, We only need to deploy the artifacts of the corresponding project to the corresponding server instead of deploying all the artifacts to the server.
For example, if we only deploy the artifact from the project1, we just need to select the artifact, like:
The artifact source should be:
$(System.ArtifactsDirectory)\_TestBuild\drop1
I need to deploy many similar apps to the same environments with Azure DevOps.
What are some ways to share and maintain the release process for these apps....
to avoid updating every app's Release process when it changes?
Background
I have dozens of similar apps with identical release processes.
Each app (in its own repo) will have its own Azure Pipeline.
I'm fine with a custom solution to this problem.
Options
Make a Release for every Pipeline -- not a fan!
CON: Azure seems to want a 1-to-1 relationship between Pipelines and Releases.
CON: I want to avoid many Releases at all costs, since changes would be nearly unmaintainable.
Use custom stage templates -- doesn't work for what I need
CON: Releases do not share custom stage templates.
Custom templates that you create are scoped to the project that you created them in. [Azure Documentation]
CON: Custom stage templates cannot be updated (as of this post).
To update an stage template, delete the existing template in a release pipeline and then save the stage as a template with the same name. [Azure Documentation]
Put the release process in the Pipeline .yml file -- seems possible, but....
PRO: The process would be stored in a shared "common" repo.
PRO: Each app will have a minimal Pipeline .yml file, setting some parameters before running the shared process.
CON: How would I track and manually deploy releases without the Release portal UI? Am I missing something?
Trigger the same Release with artifacts from different apps -- not sure this will work....
PRO: One release process, easy to track and deploy as needed via the Release portal UI.
CON: How would I track releases for different apps? Would I name Releases with the app name?
CON: I don't see how to set it up, since a Release is tied to a primary Pipeline.
At the time of linking an artifact source to a release pipeline.... [Azure Documentation]
CON: Though multiple artifact sources can be used, Releases need a default artifact Pipeline or source repo. Would I need to setup the artifact source programmatically?
When you link multiple artifact sources to a release pipeline, one of them is designated as the primary artifact source. The primary artifact source is used to set a number of pre-defined variables. It can also be used in naming releases. [Azure Documentation]
Use multiple artifact sources and artifact variables -- not sure this will work....
When there are multiple artifact sources linked to a release pipeline, you can access information about each of these. [Azure Documentation]
Bottom Line
I was able to solve many of these problems in Octopus Deploy. However, I'm having significant trouble seeing how to move my DevOps process into Azure DevOps. How would you handle this situation?
There is no perfect solution to meet your requirements.
Generally deploy multiple apps to multiple environments, we recommend that Make a Release for every Pipeline. But just as you mentioned it has 1-to-1 relationship between build pipelines and Releases and it has many releases.
We can deploy multiple apps to multiple environments through ONE release with customized Template as you mentioned, we can also use the Task Groups to combine a set of tasks together, then reuse them in different stages. But we still need to update the definitions accordingly (change the Task Group version) once the specific task group is updated.
Another way is Clone Stage, you mentioned they are the same environments, so you can config one stage, then clone stages from the existing one, just name the stages with the specific App names accordingly. But you still need to change the settings accordingly, for example the Artifact filters (see Release triggers) to determine the condition to trigger the specific stage. But in this way it will download all the artifacts first when you add multiple artifact sources, this will take long time to get sources...
I'm new to Aure DevOps. Trying to create build and release pipelines there's one thing I don't understand:
Commonly, every kind of build finally results in some output, called artifacts.
With Azure DevOps it seems like there is always a final copy or publish task necessary to copy the created artifact from A to B, so the release task may then access the compiled artifacts.
Why aren't these artifacts plain accessible to a release pipeline right from the location where they have been built? Why don't the build tasks automatically set a variable pointing to the right folder, so the release pipeline may access the files right from there?
Or is this already happening and I'm just missing something from the tutorials I watched?
There are so many reasons.
Two easy ones:
There is no guarantee that the agent's working folder still contains the files. Agents are reused from build to build and release to release, and a given build or release will always use the same working folder. The working folder is cleaned up between builds.
Releases may run on different agents. On different machines. In different domains. Or any combination. There's no guarantee that the agent where the build ran is accessible by the agent where the release is running. Publishing the artifact allows a guarantee: As long as the machine the release is running on has the ability to talk to Azure DevOps (which is a requirement for the agent to function in the first place), it can get the artifacts it needs.
Why aren't these artifacts plain accessible to a release pipeline
right from the location where they have been built?
Agree with Daniel.
The main reason for me is because we can't hold the hosted agent all the time. Since MS wants to protect resources efficiently, it is not occupied for a long time.
When we queue a build, MS will assign us a brand new clean agent to execute our task, and after the build is complete, the MS will reclaim the agent assigned to our build and restore the agent to its initial state in preparation for accepting the assignment of the next task.
So, we could not keep hold the hosted agent to use it in next release pipeline. We have to store the artifacts in the cloud/server, then we could download it in the release pipeline. Otherwise, we could not get the artifact we need from an agent that has been restored.
Besides, MS is randomly assigned to the agent, and we cannot guarantee that the same agent will be allocated and built during the release pipeline.
That is the main reason why we need to copy or publish the artifacts.
If you do not want to copy or publish the artifacts, you could setup your own private agent, and do not clean the agent before you execute the release pipeline.
Update:
why is the user, well, bothered to find a place for the artifacts
manually? I would have expected every build pipeline to come with a
personal space to store the latest build artifacts. A space where
Azure DevOps automatically copies the build artifacts to. To me it
looks like things have to be manually copied from A to B and then
later from B to C.
That because not all output is needed, for example, the test project, what we need is test result/Code coverage for the test project, not the output for the solution. In this case, we do not need to copy the output to the artifacts. On the other hand, we need to copy some special files to artfacts, then automatically copy the build artifacts will not meet your requirements.
That is also the reason why we provide the task to copy files to artifacts, so that we could customize our personality needs.
Of course, if you think that manual copying is superfluous, you can use the MSBuild parameter /p:output=$(build.artifactstagingdirectory) to set the output directly to artifacts.
If I need to copy things from A to B in the Build pipeline, then what
should keep me from copying it to C right away? Then a separate
Release pipeline would be, well, rather optional, if not redundant.
If you are in the build pipeline, there is another task Download build artifacts, which could download the build artifacts.
if you are in the release pipeline, you just need select the build artifacts as source, release pipeline will download that artifact automatically:
Check this document for some more details.
Hope this helps.
I have an ADO build pipeline that builds three artifacts. However, while I am developing, I only want one of the artifacts to be built since that is the artifact on which I am working and it takes longer for all of them to build and I don't want to wait. Can I disable the other artifacts from building?
As I know, you could not use UI to disable some of artifacts which build from one build pipeline. Because we does not provide this feature in VSTS now.
But as workaround, you can use Copy File task in your release to achieve this.
Specify the artifact what you want to work with in release pipeline, and copy it from File A(the source file where these build artifacts is) to File B(any file you want). And then, you can do other thing with this specific build.
Hope this could help -)
In my solution, I have two projects (a Asp.net MVC and a Windows Service). I want to create CI/CD pipeline to deploy web application and windows service on different VMs. But to achieve this my CI pipeline should be able to publish artifacts separately for both project and then I can feed these artifacts in CD pipeline for deployment. How artifacts for all projects can be published separately in a CI pipeline ?
PS: If I create two solutions each with one project only and create CI/CD pipeline separately, all works fine. But I want to achieve it with solution having multiple project as mentioned above.
You can use multiple, Publish tasks to create multiple artifacts in a single build definition.
For example lets say, you have below, as your current artifacts for a single project, comprising of _PublishedWebsites\MVS5WebApp (XCopy deployable website) and _PublishedWebsites\MVS5WebApp_Package (web deploy package).
If you want to separate these two, into two artifacts, you can use two Publish Artifact tasks as shown below, each one specifying exact path to publish (this path does not support wildcards, you just have to specify the folder you need to publish)
This will give you output as shown below.
In this example I just only used the Publish Artifacts task and created two artifacts using a single web site project. You can do same for your two project scenario. If you want to use wild card to filter more files before publish you can use "Copy File" task multiple times as required.
If you are using '.net core' task in the build pipeline then uncheck the checkbox 'Publish web projects' just after the command textbox.
Then it automatically creates publish artifacts separate for each project in the solution with the same name as each of the project files.
You have multiple ways to achieve that.
You can either create multiple build definitions targeting the project and not the solution in the build step with the proper arguments.
Or you can have one build definition with multiple build steps.
After that on the release side of things you can either leverage one release definition with multiple steps or multiple release definitions.