Celery task subprocess stdout to log - subprocess

Question

I have a celery task which calls other python script external to Django application with subprocess. This program have some print's in it, and I want to have these print's in my celery log file or in my database. When I set CELERY_ALWAYS_EAGER = True in Django settings.py file, everything works fine. If I don't set this setting, celery task log subprocess stdout only when it exit. It seems like p.stdout.readline() is blocking.
run-test.py is a long process, couple of minutes, but it print what it's doing. I want to capture this.
#shared_task
def run_tests(scenario_path, vu):
basedir = os.path.abspath(os.path.dirname(__file__))
config_path = '%s/../../scripts/config.ini' % basedir
cmd = ['python', '%s/../../scripts/aws/run-test.py' % basedir, '%s' % config_path, scenario_path, str(vu), str(2)]
p = subprocess.Popen(cmd, stdout=subprocess.PIPE)
while True:
line = p.stdout.readline()
if line != '':
logger.info(line)
else:
return

Answer 1

I found this to be very useful, using select for polling instead of blocking on readline.
https://gist.github.com/bgreenlee/1402841
child = subprocess.Popen(popenargs, stdout=subprocess.PIPE,
stderr=subprocess.PIPE, **kwargs)
log_level = {child.stdout: stdout_log_level,
child.stderr: stderr_log_level}
def check_io():
ready_to_read = select.select([child.stdout, child.stderr], [], [], 1000)[0]
for io in ready_to_read:
line = io.readline()
logger.log(log_level[io], line[:-1])
# keep checking stdout/stderr until the child exits
while child.poll() is None:
check_io()
check_io() # check again to catch anything after the process exits