How do you test a celery chord in a django app from inside of pytest? - pytest

Question

Using:
celery==5.2.7
django-celery-results==2.4.0
django==4.1
pytest==7.1.2
pytest-django==4.5.2
pytest-celery==0.0.0
I'm trying to test a task (start_task) that creates a chord (of N work_task tasks) with a callback task to summarize the work.
def test_function(db):
...
obj = make_obj
...
start_task.delay(obj)
I call start_task which creates a single work_task. The chord never
completes so that the summarize_task gets called. The work_task completes successfully (I can see that in the debugger). When I modify the test to:
def test_function(db, celery_app, celery_worker):
...
obj = make_obj
...
start_task.delay(obj)
The test dies on make_obj because the db connection is already closed.
E psycopg2.InterfaceError: connection already closed
My work around for the moment is to manually call tasks so that celery is not involved, but this does not test the chord mechanisms, only the logic that is invoked by the chord.
If someone has an example

Answer 1

It can be done using UnitTest style tests with pytest. I haven't solved this using native pytest yet. The secret sauce below is to use a TransactionTestCase vs a normal Django TestCase
from django.test import TransactionTestCase, override_settings
#pytest.mark.xdist_group(name="celery")
#override_settings(CELERY_TASK_ALWAYS_EAGER=False)
#override_settings(CELERY_TASK_EAGER_PROPAGATES=False)
class SyncTaskTestCase2(TransactionTestCase):
#classmethod
def setUpClass(cls):
super().setUpClass()
cls.celery_worker = start_worker(app, perform_ping_check=False)
cls.celery_worker.__enter__()
print(f"Celery Worker started {time.time()}")
#classmethod
def tearDownClass(cls):
print(f"Tearing down Superclass {time.time()}")
super().tearDownClass()
print(f"Tore down Superclass {time.time()}")
cls.celery_worker.__exit__(None, None, None)
print(f"Celery Worker torn down {time.time()}")
def test_success(self):
print(f"Starting test at {time.time()}")
self.task = do_average_in_chord.delay()
self.task.get()
print(f"Finished Averaging at {time.time()}")
assert self.task.successful()
cls.celery_work.__exit__(None, None, None) takes about 9 seconds to complete which is not particularly wonderful....