In TaskResult model django celery stored some information about task. Here i have got task_id in string format. But I wanted to know the way of getting each task execution serial number in worker like 1, 2 ,3 ... n.
Related
When using the official Celery s3 results backend, how do I visualise logs (state, input and output of tasks) of my tasks conveniently?
Here are some examples actions I'd like to be easy:
filter over failed tasks that happened over the last day
check the input argument of a failed task to attempt to reproduce it.
calculate success rate over a period of time?
has a periodic task run?
I previously used django-celery-results and there is an example screenshot of the interface I get in the admin.
I have recently started using anylogic, and I am facing a problem with a shared ResourcePool in my process flow.
I want to create a process flow that consists of 10 tasks.
Each task is modeled by Seize-Delay-Release-Blocks and seizes either one operator and an additional (task-specific) tool or a machine. Each operator is capable of processing all tasks, so there is one shared ResourcePool Operator.
The problem I am facing is that although the capacity of the ResourcePool Operator is sufficiently high enough to be deployed on all tasks in parallel, the tasks are processed sequentially. For example, the model starts with processing all parts entering task 1; if the queue is empty, task 2 is processed, and so on. In case if new parts enter the queue of task 1, again, task1 is the only one being processed.
The Seize-Blocks lock the same as follows:
Picture Seize-Block.
It should seize one of the available Workers.
Using Task Priority in the Seize-Block with increasing priority per task (task 1 lowest -> task 10 highest) does not solve this problem as capacity remains unused because tasks are processed sequentially.
How do I have to set up the model that the shared ResourcePool Operator can be used for all tasks in parallel?
I have been expecting problems when celery has started 2 tasks with the same id in parallel.
We can prevent this by checking if the celery already has task with the specified id and do not send the task, however, the way is not a really beautiful one.
Are there any ideas on how to make it in a beauty manner?
It is explained in the Ensuring a task is only executed one at a time section of the Celery documentation.
1 ) Celery chain.
On the doc I read this:
Here’s a simple chain, the first task executes passing its return value to the next task in the chain, and so on.
>>> from celery import chain
>>> # 2 + 2 + 4 + 8
>>> res = chain(add.s(2, 2), add.s(4), add.s(8))()
>>> res.get()
16
But where exactly is chain item's result passed to next chain item? On the celery server side, or it passed to my app and then my app pass it to the next chain item?
It's important to me, because my results is quite big to pass them to app, and I want to do all this messaging right into celery server.
2 ) Celery group.
>>> g = group(add.s(i) for i in xrange(10))
>>> g(10).get()
[10, 11, 12, 13, 14, 15, 16, 17, 18, 19]
Can I be sure that these tasks will be executed as much as possible together. Will celery give priority certain group since the first task of the group start to be being executed?
For example I have 100 requests and each request run group of task, and I don't want to mix task from different groups between each other. First started request to be processed can be the last completed, while his the last task are waiting for free workers which are busy with tasks from others requests. It seems to be better if group of task will be executed as much as possible together.
I will really appreciate if you can help me.
1. Celery Chain
Results are passed on celery side using message passing broker such as rabbitmq. Result are stored using result backend(explicitly required for chord execution). You could verify this information by running your celery worker with loglevel 'INFO' and identify how tasks are invoked.
Celery maintains dependency graph once you invoke tasks, so it exactly knows how to chain your tasks.
Consider callbacks where you link two different tasks,
http://docs.celeryproject.org/en/latest/userguide/canvas.html#callbacks
2. Celery Group
When you call tasks in group celery executes(invokes) them in parallel. Celery worker will try to pick up them depending upon workload it can pick up. If you invoke large number of tasks than your worker can handle, it is certainly possible your first few tasks will get executed first then celery worker will pick rest gradually.
If you have very large no. of task to be invoked in parallel better to invoke then in chunks of certain pool size,
You can mention priority of tasks as mentioned in answer
Completion of tasks in group depends on how much time each task takes. Celery tries to do fair task scheduling as much as possible.
I'm new to celery and I would appreciate a little help with a design pattern(or example code) for a worker I have yet to write.
Below is a description of the desired characteristics of the worker.
The worker will run a task that collects data from an endless source, a generator.
The worker task will run forever feeding from the generator unless it is directed to stop.
The worker task should stop gracefully on the occurrence of any one of the following triggers.
It exceeds an execution time limit in seconds.
It exceeds a number of iterations of the endless generator loop.
The client sends a message instructing the worker task to finish immediately.
Below is some sudo code for how I believe I need to handle trigger scenarios 1 and 2.
What I don't know is how I send the 'finish immediately' signal from the client and how it is received and executed in the worker task.
Any advice or sample code would be appreciated.
from celery.task import task
from celery.exceptions import SoftTimeLimitExceeded
COUNTLIMIT = # some value sent to the worker task by the client
#task()
def getData():
try:
for count, data in enumerate(endlessGeneratorThing()):
# process data here
if count > COUNTLIMIT: # Handle trigger scenario 2
clean_up_task_nicely()
break
except SoftTimeLimitExceeded: # Handle trigger scenario 1
clean_up_task_nicely()
My understanding of revoke is that it only revokes a task prior to its execution. For (3), I think what you want to do is use an AbortableTask, which provides a cooperative way to end a task:
http://docs.celeryproject.org/en/latest/reference/celery.contrib.abortable.html
On the client end you are able to call task.abort(), on the task end, you are able to poll task.is_aborted()