How to revoke a task properly considering its state could happen to shift - mongodb

Question

I'm developing the service for customer's orders monitoring, using MongoDB as the standalone backend for tracking and storage of celery tasks' state. So far, it works well, refreshing and displaying the state of all the tasks submitted by a current customer, e.g. STARTED, SUCCESS, FAILURE.
The monitoring UI could be some format as following.
+----------+--------------+-------------+--------------------+
|task_id |created_at |status |operation |
|----------|--------------|-------------|--------------------|
|[uuid] |[timestamp] |[STARTED] |[DOWNLOAD] [DELETE] |
|[uuid] |[timestamp] |[SUCCESS] |[DOWNLOAD] [DELETE] |
|[uuid] |[timestamp] |[RECEIVED] |[DOWNLOAD] [DELETE] |
|[uuid] |[timestamp] |[FAILURE] |[DOWNLOAD] [DELETE] |
|... |... |... |... |
+----------+--------------+-------------+--------------------+
Now I want to implement this [DELETE] utility, which means the customer could revoke a task being executed, via a HTTP request. Considering state of a task could happed to switch into a SUCCESSor FAILURE or other state if there's a latency of request due to HTTP overhead, is it proper to use app.control.revoke(task_id, terminate=True) ?
UPDATED:
Now I configure worker_state_db='/var/run/celery/worker.state.db' in config file of celery for persistent revokes, and stick to app.control.revoke(..., terminate=True). Is it a right option ? I did realize how this revoke command works when I found related answers here.
Which is the best way to programatically terminate (cancel) a celery task
Revoke a task from celery
Celery Task Custom tracking method
Because the service couldn't know target task state when revoke command being broadcasted, it could be as following.
scenario1: target task state is SUCCESS or FAILURE:
worker node is executing another new task (just say task aaa) when being revoked, and will restart executing task aaa. So I have to synchronize REVOKED status of target task into MongoDB without usage of task_revoked signal
scenario2: target task state is RECEIVED or STARTED or other :
worker node is executing this target task, and task_revoked signal would be triggered. But I failed to use this task_revoked signal to synchronize task status into MongoDB. Thus I tried to manually update MongoDB in the same way of scenario1 given a reply received from app.control.revoke(..., terminate=True, reply=True). But I still got a problem as following.
[# ERROR/MainProcess] Task handler raised error: Terminated(15)
Traceback (most recent call last):
File "/usr/local/lib/python3.7/site-packages/billiard/pool.py", line 1774, in _set_terminated
raise Terminated(-(signum or 0))
billiard.exceptions.Terminated: 15
How to solve this problem ? And please correct me if I still got something wrong.

Answer 1

I have it implemented the way you describe. There are a couple of things to note:
If you send the revoke command and the task is already done (either in SUCCESS or FAILURE status), the revoke command won't do anything and simply reply that it doesn't know the task. (And more importantly it also won't terminate the child worker process.) See the code for yourself.
You have to be cautious when using terminate=True, though. Depending on how you configure Celery, the worker that's executing your task might have already started another task when the revoke command is sent. There's an explicit warning in the documentation.
On the other hand, once the Celery worker starts to process your task and you really need to cancel the processing, revoking and killing the worker process is the only option.