We have dozen of multistage YAML pipelines responsible for deploying microservices. Each of them has a stage A which targets EnvA environment with an Exclusive Lock enabled.
The Exclusive lock guarantees us two things:
the latest run only from each of the pipeline can be executed (batch behavior)
only one run across all of the pipelines can use the EnvA environment at the same time, other are waiting when the environment will be free
The environment has also an Azure Function check which verifies the time of the day and allows or forbids deployment. It means that for example, the Azure Function allows deployment between 8.00 - 10.00 and 14:00 - 16:00.
The only documentation about this feature is
The exclusive lock check allows only a single run from the pipeline to proceed. All stages in all runs of that pipeline which use the resource are paused. When the stage using the lock completes, then another stage can proceed to use the resource. Also, only one stage will be allowed to continue. Any other stages which tried to take the lock will be cancelled.
Exclusive lock
There is not any information on how the timeout works for the Exclusive lock:
Is it the max time the stage could occupy the EnvA environment and for example if the timeout is set for 10 minutes but the stage's job takes 20mins then after 10 minutes the lock is released and the next pipeline can use this environment even the previous one is still in progress?
When the timeout starts, when a particular pipeline's stage locked the environment? Or when the pipeline stage starts and is waiting in a queue for pipeline A, B, C, D to release the EnvA environment?
What are the states of the Exclusive lock, acquired, obtained? How to track the queue of the pipelines which waits for other pipelines to release the lock?
Is it the max time the stage could occupy the EnvA environment and for example if the timeout is set for 10 minutes but the stage's job takes 20mins then after 10 minutes the lock is released and the next pipeline can use this environment even the previous one is still in progress?
If you configure the Exclusive lock and set the timeout value to 10 minutes, it will not release the lock after 10 minutes, It will cause the next pipeline to fail after 10 minutes.
For example, we set the timeout value to 10 minutes, add the env to pipeline A and pipeline B, run the pipeline A which contain the power shell script Start-Sleep -Seconds 1000, then run pipeline B.
Result:
The pipeline B will fail after 10 minutes.
In addition, if we set the timeout value to 10 days. The pipeline B will continue check. When the pipeline A using the lock complete, the pipeline B will continue to run.
Related
I want to run a Veracode security scan on our big application. The problem is that the scan takes about 5 days and I do not want to keep the pipeline agent pinned down during all this time.
My current solution is this:
Have build 1 to upload the binaries and initiate the pre-scan.
Have build 2 run automatically when build 1 is over and poll the status of the upload. If the pre-scan is over - start the scan. If the scan is over - mark as success and clear the schedule. If the pre-scan or scan is still running - schedule the build 2 to run every 30 minutes from the current point in time by modifying the schedule in the build definition and mark the build as partially succeeded.
This scheme works, but I wonder if it can be simplified.
What if the build 2 was a Release tied to the build 1?
But is it possible to repeat the logic without scheduling a new Release every time and without pinning down the build agent for the entire duration of the scan (5 days) and by using YAML (i.e. unified pipeline instead of Build + Release) ?
After all, a single Release is able to deploy the same stage mutliple times - these are the different deployment attempts. The problem is that I do not know how to request an attempt in 30 minutes without just blocking the agent for the next 30 minutes, which is unacceptable.
So this brings up the question - can we use a YAML pipeline to start a build and then poll a third party service until it returns what we need (the payload must be parsed and examined, HTTP Status code is not enough) so that the next stage could run?
You're looking for the concept of "gates" or "approvals and checks" depending on whether you're using YAML or classic pipelines.
You can add an "invoke REST API" check that periodically queries a REST endpoint and only proceeds when it satisfies the necessary conditions. This is a "serverless" operation -- it does not tie up an agent.
According to documentation I can now use sequential deployments with usage of lockBehavior: sequential and exclusive locks on environment.
It works fine, stages that want to deploy to certain environment are queued and waiting for lock to be acquired but sometimes they are simply cancelled with information Did not obtain the lock like in the screen below. It happens randomly. When I rerun the pipeline again - sometimes it success sometimes fails with the same message. What could be the reason of cancelling these pipelines?
Infromation about sequential deployments:
https://learn.microsoft.com/en-us/azure/devops/release-notes/2021/sprint-190-update
My pipeline triggers on resources, schedule and merges. Sometimes these can happen almost at the same time and many pipeline runs can be created. I've noticed that the jobs that run don't always belong to the same run.
Example
one pipeline A includes 2 jobs j.1 and j.2
a resource triggers A.1 and starts j.1
another resource triggers A.2 also and queues j.1.
A.1 finishes a job and instead of starting j.2 it is A.2 j.1 that starts.
How do I lock the run so that A.1 j.1 and j.2 runs to completion before A.2 starts?
On the agent, the queue is for the job-level not pipeline-level. So, normally the agent will be allocate to the higher priority jobs in the pipelines regardless of whether the jobs are in the same pipeline run.
Currently, we have not method or settings to manager the sort of the queued jobs.
I want to know how to kill an Azure Pipeline task (or any level of execution - run, stage, job, etc.), so that I am not blocked waiting for an errant pipeline to finish executing or timeout.
For example, canceling the pipeline does not stop it immediately if a condition is configured incorrectly. If the condition resolves to true the task will execute even if the pipeline is cancelled.
This is really painful if your org/project only has 1 agent. :(
How can I kill (not cancel) an errant Azure Pipeline run, stage, job, or task?
For the hosted agent, we could not kill that azure pipeline directly, since we cannot directly access the running machine.
As workaround, we could reduce the time that the task continues to run after the job is cancelled by setting a shorter Build job cancel timeout in minutes:
For example, I create a pipeline with task, which will still run for 60 minutes after the job is cancelled. But if I set the value of Build job cancel timeout in minutes to 2 mins, the azure pipeline will be cancelled completely.
For the private agent, we could run services.msc, and look for "VSTS Agent (name of your agent)". Right-click the entry and then choose restart.
Is there a way to change the timeout for build pipeline, currently the pipeline time's out after 60 mintues. I want to reduce it to 30 minutes.
I looked at all the organization settings and project settings, but not able to find anything on the UI
Or else can it be set from YAML?
For a YAML pipeline the documentation says you can write
jobs:
- job: Test
timeoutInMinutes: 10 # how long to run the job before automatically cancelling
cancelTimeoutInMinutes: 2 # how much time to give 'run always even if cancelled tasks' before stopping them
timeoutInMinutes: 0 should also work for individual tasks, and 0 means max value (infinite for self-hosted agents).
azure devops build pipeline reduce the timeout to 30 minutes
Edit the pipeline you want to modify. On the Options tab, there is an option Build job timeout in minutes, which you can set the Build job timeout, the default value is 60 minutes.
This timeout are including all tasks in your build pipeline rather than a particular job, if one of your build step out of time. Then the whole build definition will be canceled by the server. Certainly, the whole build fails and all subsequent steps are aborted.
As per documentation ,
On the Options tab you can specify default values for all jobs in the
pipeline. If you specify a non-zero value for the job timeout, then it
overrides any value that is specified in the pipeline options. If you
specify a zero value, then the timeout value from the pipeline options
is used. If the pipeline value is also set to zero, then there is no
timeout.
more on,
https://learn.microsoft.com/en-us/azure/devops/pipelines/process/phases?view=azure-devops&tabs=classic&viewFallbackFrom=vsts#timeouts