Is there a possibility to create a release pipeline, that consists of other pipelines?
Assets
In my case I have following pipelines:
- build job Job1 (to build some application artifacts)
- build job Job2 (to build some application artifacts)
- build job Job3 (to build other artifacts, e.g. tests)
- release job Release1 (to deploy application form artifacts from Job1 and Job2)
- release job Release2 (to prepare special DB for tests)
- release job Release3 (to run tests from Job3)
Each of above build consists of many tasks and task groups.
Goal
Is to prepare following pipeline, that will be scheduled every night (so just a regular nightly build-release):
[trigger: at specific time] -> (Job1, Job2, Job3) -> Release2 -> Release1 -> Release3
Restrictions
I can not merge any of mentioned jobs/releases, because we need to run some of them independently.
Ideas
One thing I conciser was to make task groups out of all jobs and releases (that already consists of task groups...) and to prepare new release pipeline, but this idea sounds like not-the-best option.
You may be able to accomplish this via the Trigger Build Task Azure DevOps extension. However, you should note the warning under the "Known Issues" section regarding using this extension with Release pipelines. Under this approach you'd create a single "mega" pipeline comprised of various calls to the extension which would trigger your existing build and release pipelines.
In my view, a better approach would be to adopt the use of YAML templates to create YAML based pipelines. You could create a template for each stage you've outlined in your question and then simply reference the template in your pipelines. This way you could have your standalone pipelines and your "mega" pipelines referencing the same templates to promote re-use and improve maintainability. There is even an easy way to export your current "classic" pipelines to YAML.
Related
I have an Azure DevOps classic release pipeline
There are 2 artifacts (type build) with continuous deployment trigger enabled, and 2 stages
Each stage has an "after release" trigger with an artifact filter, to the respective build artifact.
The builds are for different repos.
However, both stage are being triggered when every build artifact is created, even when I manually create a release...
Expected behavior is:
Build A completes, only stage A is triggered
Build B completes, only stage B is triggered
When I manually create a release, I would like to choose which stage will be triggered. If this is not possible, the trigger both stages
Is this a bug or am I missing something?
Azure DevOps Classic Release - Artifact Filter always triggering release
For this issue ,I am afraid that it is currently not supported in azure devops. If you want different artifacts to trigger the deployment of different rows of stages respectively, it is better to create two pipelines, set the artifact in each pipeline to trigger the deployment on the required stages.
And you could check this topic on developer community
Sorry I should have clarified a bit more. I don't think that triggers will solve all use cases, but having triggers on stages would at least allow you to have the following:
Stage A has trigger /src/appA
Stage B has trigger /src/appB
If you commited (script, code, etc.) to /src/appB, it should use previous artifacts and only build appB and further if requested.
If you think that two artifacts only trigger a part of the pipeline is necessary, you could submit a feature request in our Develop Community site. Our PM and product team will kindly review your suggestion.
I've been fighting with this for a while. I've been able to get it working with two separate release pipelines, but not with one. It seems that in one pipeline the PR trigger for releases fires for all environments, even with separate build artifacts.
Goal is to have the following in one release pipeline:
PR to dev and merge to dev deploys to dev environment
PR to main deploys to QA environment
Merge to main of said PR releases to prod (with manual approval step).
It would be great to get this one one release pipeline, as that would reduce overhead and management, etc.
I've tried all sorts of variations with artifact release triggers and the environment pre-deployment conditions, but to no avail.
I've even tried using 2 separate build pipelines (one for main, one for develop) and then loading both as artifacts and trying to separate that way, but even when only 1 build pipeline runs, it triggers the PR release condition for both environments.
You should be able to achieve this if you implement your release pipeline with the new YAML pipelines.
The simplest option (which is not quite what you are asking for but should get the same outcome) would be to create a template with your core deployment logic, then create separate pipelines that reuse this template. That should allow you to replicate the triggers that you already had working for two release pipelines without needing to duplicate the deployment code.
Alternatively, you could set up a single YAML pipeline with Stages for Dev, QA and Prod, and set conditions on the stages, based on the source branch and/or build reason.
Need your inputs as I am new to Azure DevOps. Is it possible to have multiple release pipeline & single build pipeline in Azure? As per the best practices, if within a single solution, there are multiple projects, it's best to have single build pipeline per project & then create corresponding release pipelines. Now information I have received from developers, their application is so integrated that it is very difficult to seperate out projects to their individual solutions. They are suggesting to create multiple release pipelines for windows services & web services for various projects.
is it still possible?
Is it possible to have multiple release pipeline & single build pipeline in Azure?
The answer is yes.
There is no one-to-one correspondence between the build pipeline and the release pipeline. We can have multiple release pipeline & single build pipeline, or have multiple build pipeline & single release pipeline. It all depends on your needs.
But when we are dealing with these special situations, we need to be very clear about the output of the build pipeline and the generation of artifacts or the input of the release pipeline and the deployment of artifacts, otherwise it is easy to cause confusion.
For your situation, I have create a simple sample to explain the process.
In the build pipeline, I have two projects in my build pipeline. Build the solution (or all projects) by Visual Studio build task, then we need create two artifacts for each projects:
Then publish the build artifacts with different artifact name:
So we get two artifact for one build pipeline:
Last but not least, we create multiple release pipeline (or we could create multiple stages) for windows services & web services. Release pipeline will download two artifacts from the build pipeline, We only need to deploy the artifacts of the corresponding project to the corresponding server instead of deploying all the artifacts to the server.
For example, if we only deploy the artifact from the project1, we just need to select the artifact, like:
The artifact source should be:
$(System.ArtifactsDirectory)\_TestBuild\drop1
We have been using a YAML file to do our CI in Azure DevOps for a few months with the idea that we would get our release configured using YAML in the future.
Well that time is now and I'm confused by how you would introduce a CD process. With the MyProject-CI.yml being a Build Pipeline and our Releases being Classic Pipelines I assumed that when the time came to get the CD process down as YAML we would create a MyProject-CD.yml. That would be triggered by the dropping of an Artifact within the MyProject-CI.yml CI.
However I think that was just a misunderstanding on my behalf and what we are supposed to do is convert the original MyProject-CI.yml into a multi-stage pipeline that has the following stages
Build and Run Unit Tests
Deploy to Development and run WebTests
Deploy to Production and run WebTests
Is the switch to a multi stage CI/CD in one file correct rtaher than Release and Build in separate files?
The short answer is yes, you got the idea. A single multi-stage pipeline yml is the way to do both build and deploy, and that is the base intention. Here is an exercise that parallels your case that might help.
As your pipelines get more complex, you will likely get into scenarios with multiple files, as you can template parts of your pipeline for reuse in multiple places, or to enforce conventions from a central location.
I have an Azure DevOps CI Build and Release pipeline in following setup:
CI Build runs with each new commit in develop branch and creates a Build Drop (Artifact)
Release pipeline runs with each new Artifact and deploys to INT and eventually to PROD (after manual approval)
I would like to add a 3rd stage (called eg. MONITOR) which would run after the PROD release every night using the same drop as the PROD stage used, with following schema:
[Build Drop] -> [INT] -> manual approval: [PROD] -> nightly scheduler: [MONITOR]
This seems to be impossible to me, do you know how to achieve this goal?
Following is crucial for me:
the MONITOR and PROD run always from exactly the same Artifact
MONITOR is executed only if the PROD was successful
if there is a newer PROD release, the old MONITOR is not executed any more and instead the newest one is executed using the newest Artifact which made it to PROD
I tried so far following:
merge develop to master when the commit made it to PROD. And then used scheduled nightly Build from master with MONITOR stage - it works, but MONITOR uses different Artifact than PROD, so not usable for me
used scheduled trigger for MONITOR after PROD - does not work, the MONITOR is executed only once at scheduled time and never again
created extra release pipeline based on specific Artifact version with a scheduled trigger - this works, but I have to maintain the specific Artifact version manually with each successful PROD release. Another caveat is that I have to use 2 separate pipelines which makes the overview not so nice. (but, so far the best solution I achieved)
do you have better ideas? many thanks
What I would do is have 2 separate Release Pipelines.
This allows you to schedule the release without producing a new artifact (scheduled build).
Then, I would do some of what #Soccerjoshj07 suggested in that I would invoke the REST api in a task on the MONITOR pipeline/stage.
I would make the REST api call to the Releases endpoint to get the top=1 releases for releasedefinitionid=x. Then use the Release Environment endpoint to get the PROD environment for that latest release id. With the environment in hand, check the status for succeeded. If not, fail the release.
Edit as per new requirement outlined in comment
Given PROD.1 is succeeded and PROD.2 is failed when MONITOR is triggered, then the artifact from PROD.1 should be used for MONITOR.
With this criteria I would change some things. Rather than have the MONITOR go digging for the latest PROD release and fail if the latest is failed, I would make the successful PROD stage tag its build artifact and employ artifact filters on the Monitor pipeline.
The tagging can occur via the REST api or using the Tag Build or Release Task from Colin's ALM Corner Build & Release Tools and might look like this:
Are you using a YAML template, and if so have you played with the cron schedules? https://learn.microsoft.com/en-us/azure/devops/pipelines/build/triggers?view=azure-devops&tabs=yaml#scheduled-triggers
If using classic Release UI, I think you can have the definition trigger be on a schedule but that would queue the entire definition. You might have to get creative with variables and maybe create 'isScheduled=true' and use that to determine if it should skip tasks.
Other ideas:
Create a logic app or function app that calls the REST API? Sample app and github link here: https://oshamrai.wordpress.com/2019/04/22/azure-devops-rest-api-19-queue-builds-and-download-build-results/
The Azure-Devops AZ CLI extension might be easier, though: https://learn.microsoft.com/en-us/cli/azure/ext/azure-devops/pipelines/build?view=azure-cli-latest#ext-azure-devops-az-pipelines-build-queue
Beside setting up two release pipelines, if you want to use scheduled trigger for only one Stage, I am afraid there is no such out of box way to achieve that, scheduled trigger is only for entire pipeline.
As a workaround, you can add a custom condition for the job of MONITOR stage.
For example in yaml:
- stage: MONITOR
jobs:
- job:
condition: and(always(), eq(variables['Release.Reason'], 'Schedule'))
steps:
In UI , you can set this in Run this job of agent job:
In this case, the stage only executed when the release triggered by scheduled trigger. If the release is triggered by other reasons, the MONITOR stage will be skipped .
The limitation of this workaround is that when your pipeline is triggered by a scheduled trigger, two other stages are also executed.
Or write a script with powershell task (in INT/PROD stages) to determine whether Release.Reason is Schedule. If yes , skip the current stage.
For how to obtain the latest artifact version of PROD and determine the deployment status of PROD, you can refer to the two answers above.