I have a group of celery tasks that I want to run in parallel, and then wait for them all to finish. I am currently using:
group(task_list).apply_async().get()
(see my other question for more detail: Wait for all tasks in a celery group to finish or error out)
When all of my tasks run without exceptions, this works perfectly. However, if any of the tasks throw an exception, it immediately returns.
I can add a try/except around every task and have it return a custom error object, but then it shows up in the flower dashboard as 'succeeded'.
Is it possible to wait for all errors AND exceptions to finish?
You should use a chord, not a group. From the celery docs:
A chord is a task that only executes after all of the tasks in a
group have finished executing.
result = chord(task_list)(handle_results)
The task id returned by chord() is the id of the callback, so you can wait for it to complete and get the final return value
result.get()
So what happens if one of the tasks raises an exception?
The chord callback result will transition to the failure state, and the error is set to the ChordError exception
print(result.traceback)
Related
What is Replay in Cadence/Temporal workflow? Is it the same as "retry"?
Why can't I simply use my own logger in workflow code due to replay?
“Retry” and "replay" are completely different.
Replay is for rebuilding the workflow thread states(process/thread stack).
Imagine this workflow code(in Java):
activityStub.doA()
LOG.info("first log")
activityStub.doB()
LOG.info("second log")
If LOG is not from Workflow.getLogger or not wrapped by Workflow.isReplay, the first log will be printed more than one times — just depend on how many times the code got replayed.
The timeline of causing duplicated logs:
After doA is completed, first log is printed.
And then doB is executed, let say doB will take 1 minute.
During the 1 minute, the worker crashes or got restarted.
And then the doB completed.
Then there will be a new workflow task to process the completion of doB.
The workflow task will then executed in a new worker host, which requires a replay to rebuild the Stack until the code of doB. During the replay, assuming LOG is your own logger without wrapping by workflow.isReplay(), the first log will be printed again.
And then doB will get completed and then the second log will be printed.
So at the end, you will see the logs:
first log
first log
second log
Is there any way to delay a Celery task from running based on a condition? Before it moves from scheduled to active I would like to perform a quick check to see if my machine can run the task based on the arguments provided and my machine's state at the time. If it's not, it halts the scheduled queue and waits until the condition is satisfied.
I've looked around at the following points but it didn't seem to cut it:
Celery's Signals: closest thing I could get to is task_prerun() but regardless of what I put in there, the task will get run and it doesn't halt the other scheduled tasks from running. There's also worker_ready() but that doesn't look at the upcoming task's arguments to do the check.
Database Lock (also here as well): I can have each of the tasks start running normally and then do the check at the beginning of the task's run but if I set a periodic interval to check if condition is met, I lose the order of the active queue as condition can be met at any point and one of the many active tasks will be able to continue. This is where the database lock comes in and is so far the most feasible solution. I can make a lock every time I do the check and if the condition's not met, it stays locked. When the condition's finally met, I release the lock for the next item in the queue, preserving the queue's original order.
I find it surprising that celery doesn't have this functionality to specify if/when the next item in the scheduled queue is ready to be run.
I have some tasks, which are further divided into runnables. Runnables execute as task instances. Runnables have dependencies within the tasks and also to other tasks's runnables. I have the information of deadlines and periods of tasks and the execution order of tasks and runnables i.e I can extract the data flow but the only point where I am stucking is that how can I get the information if the task instances are executing within the period i.e obeying the deadlines and if not executing withtin the deadline then that task instance will execute in the next cycle or next period.
Any ideas ? Suggestions ?
p.s I dont have timing information for the execution of runnables.
I have a big task,which i break down into smaller task and analyse them. I have a basic model.
Master,worker and listener .
Master creates the tasks,give them to worker actors. Once an worker actor completes,it asks for another task from the master. Once all task is completed ,they inform the listener. They usually take around less than 2 minutes to complete 1000 tasks.
Now,Some time the time taken for some tasks might be more than others. I want to set timer for each task,and if a task takes more time,then worker task should be aborted by the master and the task has to be resubmitted later as new one. How to implement this? I can calculate the time taken by a worker task,but how Master actor keeps tab on time taken by all worker actors in real time?
One way of handling this would be for each worker, on receipt of a task to start on, sets a timeout before changing state to process the task, eg:
context.setReceiveTimeout(5 minutes) // for the '5 minutes' notation - import scala.concurrent.duration._
If the timeout is received, the worker can abort the task (or whatever other action you deem appropriate - eg. kill itself, or pass a notification message back to the master). Don't forget to cancel the timeout (set duration = Duration.Undefined) if the task is completed or the like.
I'm new to celery and I would appreciate a little help with a design pattern(or example code) for a worker I have yet to write.
Below is a description of the desired characteristics of the worker.
The worker will run a task that collects data from an endless source, a generator.
The worker task will run forever feeding from the generator unless it is directed to stop.
The worker task should stop gracefully on the occurrence of any one of the following triggers.
It exceeds an execution time limit in seconds.
It exceeds a number of iterations of the endless generator loop.
The client sends a message instructing the worker task to finish immediately.
Below is some sudo code for how I believe I need to handle trigger scenarios 1 and 2.
What I don't know is how I send the 'finish immediately' signal from the client and how it is received and executed in the worker task.
Any advice or sample code would be appreciated.
from celery.task import task
from celery.exceptions import SoftTimeLimitExceeded
COUNTLIMIT = # some value sent to the worker task by the client
#task()
def getData():
try:
for count, data in enumerate(endlessGeneratorThing()):
# process data here
if count > COUNTLIMIT: # Handle trigger scenario 2
clean_up_task_nicely()
break
except SoftTimeLimitExceeded: # Handle trigger scenario 1
clean_up_task_nicely()
My understanding of revoke is that it only revokes a task prior to its execution. For (3), I think what you want to do is use an AbortableTask, which provides a cooperative way to end a task:
http://docs.celeryproject.org/en/latest/reference/celery.contrib.abortable.html
On the client end you are able to call task.abort(), on the task end, you are able to poll task.is_aborted()