Instead of manually redeploying a stage, I want to achieve an automatic way to redeploy(can be done manually).
My stage include some disk operations, which sometime fails on the first attempt but usually succeeds on the second attempt.
I am currently re-running the task group into another job in the same stage.
The second job basically executes only if the first one fails.
But this marks the stage as failed as out of two jobs, first one has failed.
But in my case both the jobs are same. Can't find a way to redeploy the same stage.
Your options are basically keep doing what you have or you can replace the steps that fail with custom PowerShell/Bash script that knows how to retry.
Edit: You could improve your existing solution little bit by putting your second attempt in the same job as your first attempt. That way you wouldn't get a failed stage. You can put conditions on steps and not just jobs.
https://learn.microsoft.com/en-us/azure/devops/pipelines/process/conditions?view=azure-devops&tabs=yaml
Related
My pipelines activities need the date of the run as a parameter. Now I get the current date in the pipeline from the utcnow() function. Ideally this would be something I could enter dynamically in the trigger so I could rerun a failed day and the parameter would be set right, now a rerun would lead to my pipeline being rerun but with the date of today not the failed run date.
I am used to airflow where such things are pretty easy to do, including scheduling reruns. Probably I think too much in terms of airflow but I can't wrap my head around a better solution.
In ADF,it is not supported directly to pass trigger date at which pipeline got failed to trigger.
You can get the trigger time using #pipeline().TriggerTime .
This system variable will give the time at which the trigger triggers the pipeline to run.
You can store this trigger value for every pipeline and use this as a parameter for the trigger which got failed and rerun the pipeline.
Reference: Microsoft document on System Variables on ADF
To resolve my problem I had to create a nested structure of pipelines, the top pipeline setting a variable for the date and then calling other pipelines passing that variable.
With this I still can't rerun the top pipeline but rerunning Execute Pipeline1/2/3 reruns them with the right variable set. It is still not perfect since the top pipeline run stays an error and it is difficult to keep track of what needs to be rerun, however it is a partial solution.
I have a configuration file for the Azure pipeline that is scheduled through the UI to run Mon to Fri. The file has different stages and each stage calls a different template. What I want to do is run different stages/templates in different days of the week.
I tried to save different schedules through the triggers UI, but they need to be applied to the entire file.
I was also reading this https://learn.microsoft.com/en-us/azure/devops/pipelines/process/scheduled-triggers?view=azure-devops&tabs=yaml but again, the schedule would be applied to the entire file.
Is there a way to apply a different schedule to each step?
No, there is an "out-of-box" way to do that. I think you may try to:
Divide your build into several and schedule them separately.
Or
Add a planning step that detects the day and sets an execution step variable like:
echo "##vso[task.setvariable variable=run.step1;]YES"
Set variables in scripts
Then use it in the conditions:
and(succeeded(), ne(variables['run.step1'], 'YES'))
Specify conditions
Basically I want two tasks, I want the second task (Task B) to look at the status of first task (Task A).
All the examples I see use yaml, within the Release section of setting up deployments, they all use a user interface.
If I use Agent.JobStatus in Step A or Step B, it shows the Job Status of what we are currently in. I figure I need to capture the value between Task A and Task B (not within either one), how does one capture that? I either can't find it, or not understanding something.
I have put it in the agent job variable expression of Task B....hoping it gathered what was the last job status, but it is null.
in(variables['Agent.JobStatus'], 'Failed', 'SucceededWithIssues')
I have several Azure DevOps release pipelines. I have a step which copies all my deployment scripts to a unique and one-time Azure Storage blob container, named for the release variable System.JobId. This is so that I can guarantee that whenever I run a release, I am always using up-to-date scripts and it shouldn't be possible to be referring to old copies, regardless of any unpredictable error conditions. The container is deleted at the end of a release.
The documentation for System.JobId states:
A unique identifier for a single attempt of a single job.
However, intermittently and at random, sometimes my pipeline fails on the blob copy step with:
2020-03-30T19:28:55.6860345Z [2020/03/30 19:28:55][ERROR] [path-to-my-file]: No input is received when user needed to make a choice among several given options.
I can't directly see that this is because the blob already exists, but when I navigate to the storage account, I can see that a container exists and it contains all the files I would expect it to. Most other times when the pipeline runs successfully, the container is gone at this point.
Why does it seem that sometimes the JobId is reused, when the documentation says it is unique? I am not redeploying the same release stage on multiple attempts to trigger this problem. Each occurrence is a new release.
Why does it seem that sometimes the JobId is reused, when the
documentation says it is unique?
Sorry for the mis-leading that our document guide.
The unique only represent relative to current pipeline, instead of the complete system.
Take a simple sample, there has 2 pipeline: p1, p2, and each pipeline has 2 jobs: job1, job2. In pipeline p1, the job id of job1 is unique and other jobs will not has same job id value.
BUT in meanwhile, the corresponding job id value in pipeline p2 is same with the one in pipeline p1. Also, it will has same value even after re-run pipeline p1.
To use your words, this job id value will re-use in a new pipeline execution, including one pipeline re-run.
I am running a pipeline where i am looping through all the tables in INFORMATION.SCHEMA.TABLES and copying it onto Azure Data lake store.My question is how do i run this pipeline for the failed tables only if any of the table fails to copy?
Best approach I’ve found is to code your process to:
0. Yes, root cause the failure and identify if it is something wrong with the pipeline or if it is a “feature” of your dependency you have to code around.
1. Be idempotent. If your process ensures a clean state as the very first step, similar to Command Design pattern’s undo (but more naive), then your process can re-execute.
* with #1, you can safely use “retry” in your pipeline activities, along with sufficient time between retries.
* this is an ADFv1 or v2 compatible approach
2. If ADFv2, then you have more options and can have more complex logic to handle errors:
* for the activity that is failing, wrap this in an until-success loop, and be sure to include a bound on execution.
* you can add more activities in the loop to handle failure and log, notify, or resolve known failure conditions due to externalities out of your control.
3. You can also use asynchronous communication to future process executions that save success to a central store. Then later executions “if” I already was successful then stop processing before the activity.
* this is powerful for more generalized pipelines, since you can choose where to begin
4. Last resort I know (and I would love to learn new ways to handle) is manual re-execution of failed activities.
Hope this helps,
J