Where is a task running with multiple nodes having python Celery installed? - celery

Question

If I have multiple workers running on different nodes, how can I know a task is assigned to which worker then?
e.g. here are two workers 10.0.3.101 and 10.0.3.102; a Redis backend runs on 10.0.3.100; when a task is sent to the task queue to Redis backend, a worker gets and executes it. The worker is 10.0.3.101 or 10.0.3.102?
In addition, if a worker saying it is 10.0.3.101 running a task and suddenly halt, how can I know the failure? i.e. Is there any built-in fail over mechanism inside Celery?
Thanks.

Answer 1

I solved the problem by searching on Google.
The knowledge is mainly from Celery documentation.
We can get the hostname of a task-executing worker in task context or use command to get worker machine IP. The task defined as:
import time
import subprocess
from celery import current_task
#app.task
def report():
id = current_task.request.id
ip = subprocess.check_output(
"ip addr | grep eth0 | grep inet |" +
" cut -d t -f 2 | cut -d / -f 1", shell=True)
ip = ip.split('\n')[0].split(' ')[-1]
hostname = current_task.request.hostname
current_task.backend.store_result(
id, result={"ip": ip, "hostname": hostname}, status="READY")
time.sleep(100)
return {
"ip": ip,
"hostname": hostname
}
If start a worker on a machine or in a container:
celery worker -A node.tasks --hostname="visible_hostname_in_request.hostname"
Then we can use following lines to get worker's IP or hostname:
# python
>>> from node.tasks import report
>>> r = report.delay()
>>> r.result
As far as I know, there is no built-in fail-over mechanism in Celery, thus we need to implement it by self; also we can use 3rd party libs like dispy ...