I have a multi-stage YAML pipeline:
Build Solution
|
▼
Terraform DEV \
| |
▼ |
Deploy Function App | DEV Environment (No Approval Required)
| |
▼ |
Provision API Mgmt /
|
▼
Terraform TEST \
| |
▼ |
Deploy Function App | TEST Environment (Approval Required)
| |
▼ |
Provision API Mgmt /
I have two environments configured (DEV and TEST) with an Approval configured on the TEST environment and the Terraform TEST stage has a deployment job configured to use the TEST environment. This means that when the pipeline reaches the Terraform TEST stage an email is sent to the approvers for the TEST environment and it waits.
When that stage is then approved the build continues.
The Deploy Function App stage also has a deployment job targetting the environment for that part of the pipeline. My issue is that when it reaches the Deploy Function App for the TEST environment it again asks for approval to deploy to the TEST environment.
My question is: Is this fixed behaviour? i.e. whenever a deployment is made to an environment with an approval is a new approval required? Or is there a way to change it so a pipeline only needs one approval to deploy (as many times as required) to a specific environment?
This is by design. One such scenario for this if rolling back changes to a previous pipeline run it would be best practice to have an approval before redeploying code to the environment. As for the scenario where you have 3 stages and each one requires an approval this is by design:
A stage can consist of many jobs, and each job can consume several resources. Before the execution of a stage can begin, all checks on all the resources used in that stage must be satisfied. Azure Pipelines pauses the execution of a pipeline prior to each stage, and waits for all pending checks to be completed. Checks are re-evaluation based on the retry interval specified in each check. If all checks are not successful till the timeout specified, then that stage is not executed. If any of the checks terminally fails (for example, if you reject an approval on one of the resources), then that stage is not executed.
In your given scenario may I suggest the Terraform, function app and APIM deployments be part of the same stage? Each one of these jobs could also be templatized so can reuse them in your additional environments. This would eliminate the possibility a user approves these incorrectly (Unless you have dependsOn outlined) or the possibility that the Terraform Apply is the only one that is release.
Related
We're using SonarQube for tests, and there's one token it uses, as long as one pipeline is running, it goes fine, but if I run different pipelines (all of them have E2E tests as final jobs), they all fail, because they keep calling a token that expires as soon as its used by one pipeline (job). Would it be possible to have -all- pipelines pause at job "x" if they detect some pipeline running job "x" already? The jobs have same names across all pipelines. Yes, I know this is solved by just running one pipeline at a time, but that's not what my devs wanna do.
The best way to make jobs run one by one is set demands for your agent job to run on a specific self-hosted agent. Just as below, set a user-defined capabilities for the self-hosted agent and then require run on the agent by setting demands in agent job.
In this way, agent jobs will only run on this agent. Build will run one by one until the previous one complete.
Besides, you could control if a job should run by defining approvals and checks. By using Invoke REST API check to make a call to a REST API such as Gets a list of builds, and define the success criteria as build count is zero, then, next build starts.
I have a question on Azure DevOps Release pipelines. My pipeline workflow is multi-stage where the build triggers the QA stage which then triggers the UAT stage which then triggers the PROD stage.
I use pipeline variables to manage each stage and require pre-approval on the UAT and PROD stages so that a change does not instantly get deployed to every stage sequentially.
My question is how to handle the case where I have multiple servers in an environment. I see that each environment should be treated as a stage but right now, I am treating each server in an environment as a stage where the tasks are run in parallel. This works for the first stage (QA), but becomes ugly for UAT since each server then requires pre-approval instead of the environment.
I have pipeline variables that specify paths for files to be dropped on servers as well. At a server per stage level, this works but not for multiple servers in a stage.
My pipeline currently looks like the picture below with UAT1 and UAT2 each requiring approval. How do I handle multiple servers for the QA and UAT stages, and later PROD?
My issue was that I was using a stage to represent a single server in an environment (i.e. QA, UAT, or PROD) instead of bundling the tasks performed per server into a task group and then use multiple task groups within the stage.
My pipeline now looks like the image below.
Within the stage, there is a task group per server.
The common tasks per server are contained in the task group
I'm having a hard time getting a grasp on this to be honest.
Right now my lab project is as follows:
PR to master -> Triggers Pre-Build Pipeline as condition to merge the code ->
On merge Infrastructure pipe runs only if any changes happen in my Infrastructure folder ->
On merge I want to run my deploy pipeline to deploy my web app to Azure.
The pipes in question do the things they ought to, i.e.
Pre build builds, publishes artifact, runs Unit tests, validates ARM templates.
Infra pipe deploys the necessary infra for my web app such as ResourceGroup, App plan, app service, key vault.
Deploy Pipe downloads the artifact produced in pre deploy and deploys to a stage slot and swaps it to production slot.
What I can't seem to get to work is the pipeline chaining through dependencies, if changes happen to both infra and web app code in master I want the infra pipe to run first and the deploy pipe only if it succeeds.
If I merge only app code I want only the deploy pipe to run regardless if the infra pipe ran or not.
If I merge only infra code I want only the infra pipe to run.
If I merge both app and infra code I want both infra and deploy pipe to run in specific order.
I feel this shouldn't be all that hard to accomplish, but I've spent way too much time trying to solve this to no avail, anyone able to help? :)
Edit:
Hey Sorry #HughLin-MSFT Been Trying to work around this a bit since we're trying to avoid running scripts left and right. :)
I saw you have Build Queuing planned in an upcoming release so for now I think we might have to wait for that.
If I were to merge my deploy and infra pipe, can I use:
trigger:
branches:
include:
- master
paths:
include:
- Infrastructure/*
At stage level and somehow skip a stage instead?
Seen multiple articles mention "Continue if skipped" but can't find any information on how to actually skip a stage.
For the first and second cases, you just need to set Path filters in Triggers, the pipeline only triggers when the file at the specified path is changed. Please refer to this.
For the third case, you can try to add two agent jobs in the infra pipe, add Trigger Azure DevOps Pipeline task to the second agent job to trigger the deploy pipe, and then set Only when all previous jobs have succeeded in Run this job drop-down box for job2. In addition, you need to add a powershell task before the Trigger Azure DevOps Pipeline task, and use a script to detect whether there is app code, run job2 if there is, and cancel job2 if not.
Update:
First you can create a new pipeline and create a variable:changedcode
Use Builds - Get rest api to get the commit , then get the changed code folder with Commits - Get Changes rest api.
Assign changed code folder name as value to changedcode variable.
Set custom conditions for the agent job. In the Infra job, if the changedcode variable value is Infra, run the Infra job. In the Infra job, use the Builds-Queue rest api or Trigger Azure DevOps Pipeline task to trigger the Infra pipeline. The same is true for Deploy job, the only difference is the custom condition expression.
Here is a sample structure in yaml:
jobs:
variables:
changedcode: ""
- job:
steps:
- powershell： |
#Get the changed code folder with rest api
- job: Infra
condition: containsValue($(changedcode), "Infra"))
- powershell： |
#queue Infra pipeline with rest api or Trigger Azure DevOps Pipeline task
- job: Deploy
condition: (containsValue($(changedcode), "deploy")),and ....
- powershell： |
#queue Deploy pipeline with rest api or Trigger Azure DevOps Pipeline task
I have 3 steps YAML pipeline Build (1), Deploy To Development (2) and Deploy to Test (3).
My needs are very basic:
a. When some code changes in master, pipelines triggers (OK)
b. With this changes, Build and Deploy To Development stages automatically runs (OK)
c. Deploy to Test stages waits until I approve (OK)
But I've facing a problem if I don't approve the Stage (3), the pipeline never ends and always shows an in-progress icon. So whenever I check the Pipelines page, I saw all pipelines are running but actually is not.
Worse part is the whole pipeline falls in Failed status after two weeks.
My question is:
Is there any way to mark Deploy to Test stage as optional? The pipeline should be completed without this stage but optionally I want to able to execute this step manually.
For your issue, I think it is currently not supported in multi-stage yaml.
If you want to run a stage manually in yaml, you can only through creating checks for your environments. There are only two options for review: Approve or Reject. So when you don't want to deploy to this stage, the stage will remain in the waiting state, and will automatically reject until the approval timeout. Once deployed to the stage is rejected, the pipeline will show as failed, even if the previous stages were successfully deployed.
So just as Shayki said in the comment, the multi-stage needs to be improved, it should give the stage a more reasonable manual trigger. Here I created a feature request for your issue in our developercommunity forum. You can vote and comment here to improve its priority.
In addition,as a workaround ,you can deploy with release pipeline. You can create Release pipeline in Releases Page. Then you can add test stage and set Manual only trigger for it. So that the pipeline can be completed without this stage .
Lets say that after every deployment I want to execute some system tests (putting a message and check if it reaches its destination).
What are my options?
I looked at post-deployment gates and the idea would be to invoke an azure function that would trigger the success, wait some time and then "assert". Is this the right way? What about timeouts since I'm going to wait (possibly for a minute or two).
In your release pipeline, you can create a first stage for deployment, and a second one for system tests.
In the system tests stage, you can choose the pre-deployment conditions to suit your requirements: e.g. after stage (and select the first deployment stage)
Optionally you can add pre-deployment approvals so somebody needs to manually approve this.