I have created a release pipeline in Azure DevOps, with several stages, deployments to each environment. On some of the environments (Test and Production), I have manual approval tasks (not set in YAML, but on the environment). If the approval task is not performed within a set time, I want the whole pipeline to cancel.
I have set a timeoutInMinutes on the stage itself, however, the timeout never starts, as the stage is waiting for the approval before it can start at all.
I haven't found a way to set a timeout on the approval/review activity, nor have I found a way to have a different stage/job independent of the others sit and wait for a timeout and cancel the job with a logging command ##vso[task.complete result=Canceled;]DONE
See the screenshot. The pipeline just sits and waits forever. Any ideas?
Timeout for a whole pipeline w/manual steps in Azure DevOps
Yes, you are right. I could reproduced this issue on my side.
As we know, the Timeout is used:
To avoid taking up resources when your job is hung or waiting too
long, it's a good idea to set a limit on how long your job is allowed
to run.
When we set the checks on the stage. Our job is in the pending state not Running, At this time, the timeout we set has not yet started working. It only starts timing after our job starts running. So, we need a timeout for the checks, just like the timeout for Pre-deployment approvals:
I could not find any solution/workaround after spending a long time, but I found this is a highly prioritized feature requirement that has been tracked by the Azure Devops teams after confirming with Azure devops teams.
Now, I could see the status of this feature timeout for the checks request is In Progress at sprint 158,
I believe this feature will meet us soon, please pay attention to the release notes of Azure devops. Thank you for helping us build a better Azure DevOps.
Hope this helps.
Related
On my Azure DevOps Server instance (2020 Update 1.1) I have easy multistage YAML pipeline with Build job (run against BuildPool) and release job (run against ReleasePool). Build job is executed successfully. In release pool there are many idle agents but job is in waiting state with message:
The agent request is not running because all potential agents are running other requests. Current position in queue: 1
No agents in pool ReleasePool are currently able to service this request.
Other pipelines on the server against ReleasePool are executed.
This pipeline was executed one month past also successfully, and since this execution the YAML definition stays unchanged.
Pipeline have no explicit demands, I'm trying to identify implicit demands (from used tasks - I have checked tasks.json task manifests for each used task) - but there isn't used no task with demands.
I have no idea what I could try next.
Is the way how to diagnostic how are agents assigned to the pipeline jobs? I have admin permissions and access to the DB, I'm ready to do very deep analysis.
Since other tasks are running fine, maybe a specific demands on selected agents are not met and this is preventing an agent to be found.
https://learn.microsoft.com/en-us/azure/devops/pipelines/process/demands?view=azure-devops-2020&tabs=yaml
Or else, it's a bug.
I found a similar issue here, and Microsoft responded July 2022 with:
"We have released a fix for this issue"
https://developercommunity.visualstudio.com/t/no-agent-found-in-pool-azure-pipelines/870000
It is however not clear to me if this only applied to Azure DevOps or also to Azure DevOps Server.
But since your on 2020 Update 1.1, updating couldn't harm you:
https://learn.microsoft.com/en-us/azure/devops/server/release-notes/azuredevops2020u1?view=azure-devops-2020
We're using SonarQube for tests, and there's one token it uses, as long as one pipeline is running, it goes fine, but if I run different pipelines (all of them have E2E tests as final jobs), they all fail, because they keep calling a token that expires as soon as its used by one pipeline (job). Would it be possible to have -all- pipelines pause at job "x" if they detect some pipeline running job "x" already? The jobs have same names across all pipelines. Yes, I know this is solved by just running one pipeline at a time, but that's not what my devs wanna do.
The best way to make jobs run one by one is set demands for your agent job to run on a specific self-hosted agent. Just as below, set a user-defined capabilities for the self-hosted agent and then require run on the agent by setting demands in agent job.
In this way, agent jobs will only run on this agent. Build will run one by one until the previous one complete.
Besides, you could control if a job should run by defining approvals and checks. By using Invoke REST API check to make a call to a REST API such as Gets a list of builds, and define the success criteria as build count is zero, then, next build starts.
Currently I have a OneBranch DevOps pipeline that fails every now and then while restoring packages. Usually it fails because of some transient error like a socket exception or timeout. Re-trying the job usually fixes the issue.
Is there a way to configure a job or task to retry?
Azure Devops now supports the retryCountOnTaskFailure setting on a task to do just this.
See this page for further information:
https://learn.microsoft.com/en-us/azure/devops/release-notes/2021/pipelines/sprint-195-update
Update:
Automatic retries for a task was added and when you read this it should be available for usage.
It can be used as follow:
- task: <name of task>
retryCountOnTaskFailure: <max number of retries>
...
Here are a few things to note when using retries:
The failing task is retried immediately.
There is no assumption about the idempotency of the task. If the task has side-effects (for instance, if it created an external resource partially), then it may fail the second time it is run.
There is no information about the retry count made available to the task.
A warning is added to the task logs indicating that it has failed before it is retried.
All of the attempts to retry a task are shown in the UI as part of the same task node.
Original answer:
There is no way of doing that with native tasks. However, if you can script then you can put such logic inside.
You could do this for instance in this way:
n=0
until [ "$n" -ge 5 ]
do
command && break # substitute your command here
n=$((n+1))
sleep 15
done
However there is no native way of doing this for regular tasks.
Automatically retry a task in on roadmap so it could change in near future.
I have 3 steps YAML pipeline Build (1), Deploy To Development (2) and Deploy to Test (3).
My needs are very basic:
a. When some code changes in master, pipelines triggers (OK)
b. With this changes, Build and Deploy To Development stages automatically runs (OK)
c. Deploy to Test stages waits until I approve (OK)
But I've facing a problem if I don't approve the Stage (3), the pipeline never ends and always shows an in-progress icon. So whenever I check the Pipelines page, I saw all pipelines are running but actually is not.
Worse part is the whole pipeline falls in Failed status after two weeks.
My question is:
Is there any way to mark Deploy to Test stage as optional? The pipeline should be completed without this stage but optionally I want to able to execute this step manually.
For your issue, I think it is currently not supported in multi-stage yaml.
If you want to run a stage manually in yaml, you can only through creating checks for your environments. There are only two options for review: Approve or Reject. So when you don't want to deploy to this stage, the stage will remain in the waiting state, and will automatically reject until the approval timeout. Once deployed to the stage is rejected, the pipeline will show as failed, even if the previous stages were successfully deployed.
So just as Shayki said in the comment, the multi-stage needs to be improved, it should give the stage a more reasonable manual trigger. Here I created a feature request for your issue in our developercommunity forum. You can vote and comment here to improve its priority.
In addition,as a workaround ,you can deploy with release pipeline. You can create Release pipeline in Releases Page. Then you can add test stage and set Manual only trigger for it. So that the pipeline can be completed without this stage .
Lets say that after every deployment I want to execute some system tests (putting a message and check if it reaches its destination).
What are my options?
I looked at post-deployment gates and the idea would be to invoke an azure function that would trigger the success, wait some time and then "assert". Is this the right way? What about timeouts since I'm going to wait (possibly for a minute or two).
In your release pipeline, you can create a first stage for deployment, and a second one for system tests.
In the system tests stage, you can choose the pre-deployment conditions to suit your requirements: e.g. after stage (and select the first deployment stage)
Optionally you can add pre-deployment approvals so somebody needs to manually approve this.