Celery setup and teardown tasks - celery

Question

I am trying to use celery to parallelise the evaluation of a function with different parameters.
Here is a pseudo-code of why I am trying to achieve, which assumes that there is a function called evaluate decorated with #app.task
# 0. Setup cluster, celery or whatever parallelisation backend
pass
# 1. Prepare each node to simulate, this means sending some files
for node in mycluster:
#send files to node
pass
# 2. Evaluation phase
gen = Generator() # A Generator object creates parameter vectors that need to be evaluated
while not gen.finished():
par_list = gen.generate()
asyncs = []
for p in par_list:
asyncs.append(evaluate.delay(p))
results = [-1 for _ in par_list]
for i, pending in enumerate(asyncs):
if not pending.ready():
pending.wait()
if pending.successful():
results[i] = pending.get()
else:
pass # manage error
# send results to generator so that it generates a new set of parameters later
gen.tell(results)
# 3. Teardown phase
for node in mycluster:
#tell node to delete files
pass
The problems with this approach is that if my main application is running, and it has already passed the setup phase, then when new node connects, it certainly will not pass the setup phase. Similarly, the teardown phase will not be executed if a node disconnects.
A couple of solutions come to mind:
Instead of using a setup phase, chain two functions so that each node does setup | evaluate | teardown for each iteration of the "2. evaluation phase" loop. The problem here is that sending files through the message queue is something that I would like to avoid as much as possible.
Configure the workers to have a setup and teardown task so that they are automatically ready when they connect. I tried using bootsteps.StartStopStep , but I am not sure if this is the right way to go.
Setup a distributed file system so that there is no need to prepare and delete files before and after the evaluations
The concrete question here is, what's the recommended approach for these kind of tasks? I am sure that this is not a convoluted use-case and maybe one of you can provide some guidance on how should I approach this.

Answer 1

I'm not sure this is a worker issue - remember you may have a host of workers on a node. This sounds more like a node initialization issue. Why not have a job (a systemd task, an init script, whatever) that runs before the celery workers and which copies the files over. Similarly, in reverse, for tear down.