Is there an option to set the custom condition that will test if the previous task has failed OR timed out?
Currently, I'm using the Only when a previous task has failed which works when the task fails. If the task times out, then it is not considered an error and it is skipped.
I need a custom condition then, something like or(failed(), timedout()). Is it possible?
Context
We have this intermittent problem with the npm install task that we can't find a reason for but it is resolved with next job run, so we were searching for a retry functionality. Partial solution was to duplicate npm install and use the Control Option but it wasnt working for all "failure" cases. Solution gave by #Levi Lu-MSFT seems to be working for all our needs (it does retry) but sadly it doesnt solve the problem, 2nd line repeated task also fails.
Sample errors:
20741 error stack: 'Error: EPERM: operation not permitted, unlink \'C:\\agent2\\_work\\4\\s\\node_modules\\.staging\\typescript-4440ace9\\lib\\tsc.js\'',
20741 error errno: -4048,
20741 error code: 'EPERM',
20741 error syscall: 'unlink',
20741 error path: 'C:\\agent2\\_work\\4\\s\\node_modules\\.staging\\typescript-4440ace9\\lib\\tsc.js',
20741 error parent: 's' }
20742 error The operation was rejected by your operating system.
20742 error It's possible that the file was already in use (by a text editor or antivirus),
20742 error or that you lack permissions to access it.
or
21518 verbose stack SyntaxError: Unexpected end of JSON input while parsing near '...ter/doc/TypeScript%20'
21518 verbose stack at JSON.parse (<anonymous>)
21518 verbose stack at parseJson (C:\agent2\_work\_tool\node\8.17.0\x64\node_modules\npm\node_modules\json-parse-better-errors\index.js:7:17)
21518 verbose stack at consumeBody.call.then.buffer (C:\agent2\_work\_tool\node\8.17.0\x64\node_modules\npm\node_modules\node-fetch-npm\src\body.js:96:50)
21518 verbose stack at <anonymous>
21518 verbose stack at process._tickCallback (internal/process/next_tick.js:189:7)
21519 verbose cwd C:\agent2\_work\7\s
21520 verbose Windows_NT 10.0.14393
21521 verbose argv "C:\\agent2\\_work\\_tool\\node\\8.17.0\\x64\\node.exe" "C:\\agent2\\_work\\_tool\\node\\8.17.0\\x64\\node_modules\\npm\\bin\\npm-cli.js" "install"
21522 verbose node v8.17.0
21523 verbose npm v6.13.4
21524 error Unexpected end of JSON input while parsing near '...ter/doc/TypeScript%20'
21525 verbose exit [ 1, true ]
Sometimes also time's out
It is possible to add a custom condition. If you want the task to be executed when previous task failed or skipped, you can use custom condition not(succeeded())
However there is a problem with above custom condition, it does not work in the multiple tasks scenario.
For example, there are three tasks A,B,C. The expected behavior is Task C gets executed only when Task B failed. But the actual behavior is Task C will also get executed when Task A failed even if Task B succeeded. Check below screenshot.
The workaround for above problem is to add a script task to call azure devops restful api to get the status of Task B and set it to a variable using this expression echo "##vso[task.setvariable variable=taskStatus]taskStatus".
For below example, Add a powershell task (You need to set conditon for this task to Even if a previous task has failed, even if the build was canceled to always run this powershell task) before Task C to run below inline scripts:
$url = "$(System.TeamFoundationCollectionUri)$(System.TeamProject)/_apis/build/builds/$(Build.BuildId)/timeline?api-version=5.1"
$result = Invoke-RestMethod -Uri $url -Headers #{authorization = "Bearer $env:SYSTEM_ACCESSTOKEN"} -ContentType "application/json" -Method get
#Get the task B's task result
$taskResult = $result.records | where {$_.name -eq "B"} | select result
#set the Task B's taskResult to variable taskStatus
echo "##vso[task.setvariable variable=taskStatus]$($taskResult.result)"
In order above scripts can access the access token, you also need to click the Agent job and check option Allow scripts to access the OAuth token. Refer to below screenshot.
At last you can use custom condition and(not(canceled()), ne(variables.taskStatus, 'succeeded')) for Task C. Task C should be executed only when Task B not succeeded.
Although I failed to find a built-in function to detect if a build step is timed out, you can try to emulate this with the help of variables.
Consider the following YAML piece of pipeline declaration:
steps:
- script: |
echo Hello from the first task!
sleep 90
echo "##vso[task.setvariable variable=timedOut]false"
timeoutInMinutes: 1
displayName: 'A'
continueOnError: true
- script: echo Previous task has failed or timed out!
displayName: 'B'
condition: or(failed(), ne(variables.timedOut, 'false'))
The first task (A) is set to time out after 1 minute, but the script inside emulates the long-running task (sleep 90) for 1.5 minutes. As a result, the task times out and the timedOut variable is NOT set to false. Hence, the condition of the task B evaluates to true and it executes. The same happens if you replace sleep 90 with exit 1 to emulate the task A failure.
On the other hand, if task A succeeds, neither of the condition parts of task B evaluates to true, and the whole task B is skipped.
This is a very simplified example, but it demonstrates the idea which you can tweak further to satisfy the needs of your pipeline.
Related
We are using the following command to deploy BizTalk assemblies via PowerShell:
BTSTask AddResource /ApplicationName:$App /Type:$BizTalkAssemblyType /Overwrite /Source:$Source /Options:GacOnAdd,GacOnInstall,GacOnImport
See: https://learn.microsoft.com/en-us/biztalk/core/addresource-command-biztalk-assembly
There are certain reasons this command can fail, e.g. an orchestration is not in the unenlisted state or one or more instances of the orchestration exists.
In this case the command does not raise an error so the script continues with an output like
Command failed with 1 errors, 0 warnings.
Because in this situation the assembly does not get deployed we would like to fail the PowerShell script e.g. by raising an error. How to achieve this?
You need to capture the output and check it for the failure, or rather, check for success and fail if it doesn't.
[array] $cmdOutput = BTSTask AddResource /ApplicationName:$App /Type:$BizTalkAssemblyType /Overwrite /Source:$Source /Options:"GacOnAdd,GacOnInstall,GacOnImport"
$line = $cmdOutput.Count-2
if ( $cmdOutput[$line] -eq "Command succeeded with 0 errors, 0 warnings.")
{
Write-Output "Deploy suceeded"
}
else
{
Throw "Deploy failed $cmdOutput"
}
Imagine I have a workflow with 5 steps.
Step 2 may or may not create a file as its output (which is then used as input to subsequent steps).
If the file is created, I want to run the subsequent steps.
If no file gets created in step 2, I want to mark the workflow as completed and not execute steps 3 through to 5.
I'm sure there must be a simple way to do this yet I'm failing to figure out how.
I tried by making step 2 return non-zero exit code when no file is created and then using
when: "{{steps.step2.outputs.exitCode}} == 0" on step 3, but that still executes step 4 and 5 (not to mention marks step 2 as "failed")
So I'm out of ideas, any suggestions are greatly appreciated.
By default, a step that exits with a non-zero exit code fails the workflow.
I would suggest writing an output parameter to determine whether the workflow should continue.
- name: yourstep
container:
command: [sh, -c]
args: ["yourcommand; if [ -f /tmp/yourfile ]; then echo continue > /tmp/continue; fi"]
outputs:
parameters:
- name: continue
valueFrom:
default: "stop"
path: /tmp/continue
Alternatively, you can override the fail-on-nonzero-exitcode behavior with continueOn.
continueOn:
failed: true
I'd caution against continueOn.failed: true. If your command throws a non-zero exit code for an unexpected reason, the workflow won't fail like it should, and the bug might go un-noticed.
It is kind of annoying we cannot change the logical order(AND/OR) of the Activity dependencies. however, I have got another issue. having said that I have activities for on failure to log the error messages in DB, since the logging activity succeeds, the entire pipeline succeeds too! is there any workaround to say if any activities failed the entire pipeline and the parent pipeline, if it is called from another pipeline, should be failed either?
In my screenshot, i have selected the on completion dependencies to log the successful or error.
I see that you defined "On Success" of the copy activity to run "usp_postexecution" . Please define a "On failure" of the copy activity and add any activity ( may be a set variable for testing ) and execute the pipeline . The pipeline will fail .
Just to give you more context what i tried .
I have a variable name "test" of the type boolean and I am failing it deliberately ( by assigning to a non-boolean value of true1 )
Pipeline will fail when I define both success and failure scenarios .
The pipeline will succeed when you have only "Failure" defined
The following code returns an error to rundeck.
#!/bin/bash
exit -1
And rundeck decides how to deal with it by running the next step or changing the execution "status" to "failed".
I would like to modify the status directly by inline script to support more than 2 states. I need "succeeded", "failed" and "nodata" to express that the data are missing.
Is there a way to express this?
There is none. Just like bash can return zero or non-zero
One possible alternative is raise an exception with message nodata and exit with non-zero code. Rundeck will mark this job as fail with NonZeroResultCode error. You should be able to get your error message nodata with ${result.message}
Under some conditions, I want to make a celery task fail from within that task. I tried the following:
from celery.task import task
from celery import states
#task()
def run_simulation():
if some_condition:
run_simulation.update_state(state=states.FAILURE)
return False
However, the task still reports to have succeeded:
Task sim.tasks.run_simulation[9235e3a7-c6d2-4219-bbc7-acf65c816e65]
succeeded in 1.17847704887s: False
It seems that the state can only be modified while the task is running and once it is completed - celery changes the state to whatever it deems is the outcome (refer to this question). Is there any way, without failing the task by raising an exception, to make celery return that the task has failed?
To mark a task as failed without raising an exception, update the task state to FAILURE and then raise an Ignore exception, because returning any value will record the task as successful, an example:
from celery import Celery, states
from celery.exceptions import Ignore
app = Celery('tasks', broker='amqp://guest#localhost//')
#app.task(bind=True)
def run_simulation(self):
if some_condition:
# manually update the task state
self.update_state(
state = states.FAILURE,
meta = 'REASON FOR FAILURE'
)
# ignore the task so no other state is recorded
raise Ignore()
But the best way is to raise an exception from your task, you can create a custom exception to track these failures:
class TaskFailure(Exception):
pass
And raise this exception from your task:
if some_condition:
raise TaskFailure('Failure reason')
I'd like to further expand on Pierre's answer as I've encountered some issues using the suggested solution.
To allow custom fields when updating a task's state to states.FAILURE, it is important to also mock some attributes that a FAILURE state would have (notice exc_type and exc_message)
While the solution will terminate the task, any attempt to query the state (For example - to fetch the 'REASON FOR FAILURE' value) will fail.
Below is a snippet for reference I took from:
https://www.distributedpython.com/2018/09/28/celery-task-states/
#app.task(bind=True)
def task(self):
try:
raise ValueError('Some error')
except Exception as ex:
self.update_state(
state=states.FAILURE,
meta={
'exc_type': type(ex).__name__,
'exc_message': traceback.format_exc().split('\n'),
'custom': '...'
})
raise Ignore()
I got an interesting reply on this question from Ask Solem, where he proposes an 'after_return' handler to solve the issue. This might be an interesting option for the future.
In the meantime I solved the issue by simply returning a string 'FAILURE' from the task when I want to make it fail and then checking for that as follows:
result = AsyncResult(task_id)
if result.state == 'FAILURE' or (result.state == 'SUCCESS' and result.get() == 'FAILURE'):
# Failure processing task