In Azure DevOps, I'm trying to make it so that a release job (not a YAML pipeline job) only runs if a specific prior job has failed. One of the pre-defined conditions for running a job is "Only when a previous job has failed", but this is not appropriate because it includes all prior jobs, rather than just the last job (or better yet, a job of the users choosing).
Please note that this question focuses on tasks within a job and is not the answer that I am looking for.
According to the documentation for conditions under "How can I trigger a job if a previous job succeeded with issues?", I can access the result of a previous job -
eq(dependencies.A.result,'SucceededWithIssues')
Looking at the logs for a previous release, the AGENT_JOBNAME for the job that I wish to check is Post Deployment Tests, so that would mean that my condition should look like this. However, Azure DevOps wont even let me save my release -
not(eq(dependencies.Post Deployment Tests.result, 'succeeded'))
Job condition for job "Swap Slots Back on Post Deployment Test Failure" in stage "Dev" is invalid: Unexpected symbol: 'Deployment'.
I've tried to wrap the job name in quotes, but I get similar errors -
not(eq(dependencies.'Post Deployment Tests'.result, 'succeeded'))
not(eq(dependencies."Post Deployment Tests".result, 'succeeded'))
I've also tried to reference my job using underscores, which does allow me to save the release but then results in an error at run time -
not(eq(dependencies.Post_Deployment_Tests.result, 'succeeded'))
Unrecognized value: 'dependencies'.
How can I achieve me goal of conditionally running a job only if a specific prior job has failed?
How can I achieve me goal of conditionally running a job only if a
specific prior job has failed?
1.You should know that Yaml pipeline and Classic pipeline use difference technologies.
See feature ability, they have different features. Also, they have quite different behavior for Conditions though they(Yaml,Classic Build,Classic Release) all support job conditions.
The eq(dependencies.'Post Deployment Tests'.result, 'succeeded') you're trying is not supported in Classic(UI) Release pipeline. It's for Yaml:
It's expected behavior to get error like Unrecognized value: 'dependencies' cause the job dependency is not supported in Release pipeline. See this old ticket.
Related
Currently I have a OneBranch DevOps pipeline that fails every now and then while restoring packages. Usually it fails because of some transient error like a socket exception or timeout. Re-trying the job usually fixes the issue.
Is there a way to configure a job or task to retry?
Azure Devops now supports the retryCountOnTaskFailure setting on a task to do just this.
See this page for further information:
https://learn.microsoft.com/en-us/azure/devops/release-notes/2021/pipelines/sprint-195-update
Update:
Automatic retries for a task was added and when you read this it should be available for usage.
It can be used as follow:
- task: <name of task>
retryCountOnTaskFailure: <max number of retries>
...
Here are a few things to note when using retries:
The failing task is retried immediately.
There is no assumption about the idempotency of the task. If the task has side-effects (for instance, if it created an external resource partially), then it may fail the second time it is run.
There is no information about the retry count made available to the task.
A warning is added to the task logs indicating that it has failed before it is retried.
All of the attempts to retry a task are shown in the UI as part of the same task node.
Original answer:
There is no way of doing that with native tasks. However, if you can script then you can put such logic inside.
You could do this for instance in this way:
n=0
until [ "$n" -ge 5 ]
do
command && break # substitute your command here
n=$((n+1))
sleep 15
done
However there is no native way of doing this for regular tasks.
Automatically retry a task in on roadmap so it could change in near future.
We have an Azure DevOps YAML multi-stage pipeline where code is built and then deployed to a sequence of environments. Deployment is achieved using Terraform.
i.e.
Builds -> Test -> Deploy to DEV -> Deploy to TEST - - ->
This pipeline is used for both CI/CD builds and also PR builds. For the PR build, the only part of the deployment stage is running terraform plan on the TF scripts.
For PRs, the pipeline is configured to cancel the ongoing build if changes are pushed to that PR.
The problem we're seeing is that when changes are pushed to the PR and the ongoing build is cancelled, sometimes that cancellation happens during the terraform plan step. This occasionally means that the blob lease taken by terraform plan is not released. From that point onwards, manual intervention is required (break the blob lease) in order for the deployment stages to run successfully.
I believe we can switch off the setting which causes the ongoing PR build to be cancelled if changes are pushed.
But I wondered if there was a way of marking a pipeline step as critical - i.e. it should run to completion if the build is cancelled?
There are other ways of cancelling a pipeline build and there must be other tasks/steps which should not be cancelled part-way through. Such a critical-task setting would cover these situations too.
Not sure if you solved this, but I had the exact same issue. Adding condition: always() to my task forced it to complete when DevOps cancelled the pipeline after additional changes were pushed.
See https://learn.microsoft.com/en-us/azure/devops/pipelines/process/conditions?view=azure-devops&tabs=yaml:
jobs:
- job: Foo
steps:
- script: echo Hello!
condition: always() # this step will always run, even if the pipeline is canceled
I'm afraid you won't be able to achieve this using just YAML. What you can do will require some additional effort (and in some cases quite big):
replace terraform script with bicep for isntance - for me syntax iq quite similar and what you get here is lack of terrafrom state
add in your very first step a check if your state is locked and break a lease if it is needed
I understood that you would like to hear something better than this but at the moment there is no way to mark a task as non-cancellable. However, this sound like a cool feature and candidate for feature request.
I have a Azure Devops pipeline (yml) with a stage that deploys an application to an environment and then runs a bunch of tests against it. This pipeline is triggered when a PR is created. We sometimes have multiple runs of the same pipelines happening at once resulting in two deployments to the same environment happening at the same time.
Is it possible to configure the pipeline in such a way that the deploy stage can only be executed one at a time?
Simple example of what I'm trying to do:
Pipeline (yml) with stages: 1) Build -> 2) Deploy/Test -> 3) Release
Run 1: Build: complete -> Deploy/Test In progress -> Release waiting for stage 2
Run 2: Build: complete -> Deploy/Test waiting for Run 1 stage 2
If you use YAML then make you release in deployment job where you use environment with enabled exclusive lock. However, this has some drawback:
On developer community you can find a feature request for better handling this. And there is one workaround which involved calling REST API. But, as someone already mentioned:
Workarounds involving polling end up with race conditions where two queued builds can both end up starting.
So there is no ideal solution, but if you can please upvote above mentioned request.
I want to rollback to previous successful deployment in case of any stage fails in a release/deployment. For that I am trying to use "Auto-redeploy trigger" under "Post-deployment conditions" in a release definition in Azure DevOps.
However, every time I have a failed deployment no redeployment is triggered. Am I missing any other/additional configuration ? Or how can i achieve this in any other simple/feasible way ?
Here is the release definition history. (I am sure that branch for all definitions is same.) All releases are trigger via CICD.
You can view the Deployments in Release pipeline to check if the deployment is triggered as expected, like this.
Which job do you use in your stage task? Agent job or Deployment group job? As my test, when I run the task in Agent job, the Auto-redeploy trigger isn’t triggered as expected, but when I run the task in Deployment group job, it works as expected. So it may be the reason for your problem, you can check it on your side.
Lets say that after every deployment I want to execute some system tests (putting a message and check if it reaches its destination).
What are my options?
I looked at post-deployment gates and the idea would be to invoke an azure function that would trigger the success, wait some time and then "assert". Is this the right way? What about timeouts since I'm going to wait (possibly for a minute or two).
In your release pipeline, you can create a first stage for deployment, and a second one for system tests.
In the system tests stage, you can choose the pre-deployment conditions to suit your requirements: e.g. after stage (and select the first deployment stage)
Optionally you can add pre-deployment approvals so somebody needs to manually approve this.