Celery backend cleanup failing with SQLAlchemy & MySQL - celery

Question

I'm facing following exception when celery is trying to cleanup back-end.
Most probably, this is happening due to MySQL disconnect issue and can be solved by using pool_recycle parameter and retrying the task later. But this is out of my hand - I guess celery needs to provide support for this?
Now my question is, what is backend cleanup task and how such a failed task may affect our system?
Log:
[2014-04-08 04:00:00,017: INFO/Beat] Scheduler: Sending due task celery.backend_cleanup (celery.backend_cleanup)
[2014-04-08 04:00:00,020: INFO/MainProcess] Received task: celery.backend_cleanup[b70acd50-e72d-43b1-a702-0bfa8e7e83a6] expires:[2014-04-08 16:00:00.018317+01:00]
[2014-04-08 04:00:00,036: ERROR/MainProcess] Task celery.backend_cleanup[b70acd50-e72d-43b1-a702-0bfa8e7e83a6] raised unexpected: OperationalError('(OperationalError) MySQL Connection not available.',)
Traceback (most recent call last):
File "/webapps/phoenix/lib/python3.3/site-packages/celery/app/trace.py", line 238, in trace_task
R = retval = fun(*args, **kwargs)
File "/webapps/phoenix/lib/python3.3/site-packages/celery/app/trace.py", line 416, in __protected_call__
return self.run(*args, **kwargs)
File "/webapps/phoenix/lib/python3.3/site-packages/celery/app/builtins.py", line 56, in backend_cleanup
app.backend.cleanup()
File "/webapps/phoenix/lib/python3.3/site-packages/celery/backends/database/__init__.py", line 180, in cleanup
Task.date_done < (now - expires)).delete()
File "/webapps/phoenix/lib/python3.3/site-packages/sqlalchemy/orm/query.py", line 2626, in delete
delete_op.exec_()
File "/webapps/phoenix/lib/python3.3/site-packages/sqlalchemy/orm/persistence.py", line 866, in exec_
self._do_exec()
File "/webapps/phoenix/lib/python3.3/site-packages/sqlalchemy/orm/persistence.py", line 991, in _do_exec
params=self.query._params)
File "/webapps/phoenix/lib/python3.3/site-packages/sqlalchemy/orm/session.py", line 978, in execute
clause, params or {})
File "/webapps/phoenix/lib/python3.3/site-packages/sqlalchemy/engine/base.py", line 664, in execute
return meth(self, multiparams, params)
File "/webapps/phoenix/lib/python3.3/site-packages/sqlalchemy/sql/elements.py", line 282, in _execute_on_connection
return connection._execute_clauseelement(self, multiparams, params)
File "/webapps/phoenix/lib/python3.3/site-packages/sqlalchemy/engine/base.py", line 761, in _execute_clauseelement
compiled_sql, distilled_params
File "/webapps/phoenix/lib/python3.3/site-packages/sqlalchemy/engine/base.py", line 828, in _execute_context
None, None)
File "/webapps/phoenix/lib/python3.3/site-packages/sqlalchemy/engine/base.py", line 1023, in _handle_dbapi_exception
exc_info
File "/webapps/phoenix/lib/python3.3/site-packages/sqlalchemy/util/compat.py", line 174, in raise_from_cause
reraise(type(exception), exception, tb=exc_tb, cause=exc_value)
File "/webapps/phoenix/lib/python3.3/site-packages/sqlalchemy/util/compat.py", line 167, in reraise
raise value.with_traceback(tb)
File "/webapps/phoenix/lib/python3.3/site-packages/sqlalchemy/engine/base.py", line 824, in _execute_context
context = constructor(dialect, self, conn, *args)
File "/webapps/phoenix/lib/python3.3/site-packages/sqlalchemy/engine/default.py", line 507, in _init_compiled
self.cursor = self.create_cursor()
File "/webapps/phoenix/lib/python3.3/site-packages/sqlalchemy/engine/default.py", line 671, in create_cursor
return self._dbapi_connection.cursor()
File "/webapps/phoenix/lib/python3.3/site-packages/sqlalchemy/pool.py", line 548, in cursor
return self.connection.cursor(*args, **kwargs)
File "/webapps/phoenix/lib/python3.3/site-packages/mysql/connector/connection.py", line 1231, in cursor
raise errors.OperationalError("MySQL Connection not available.")
sqlalchemy.exc.OperationalError: (OperationalError) MySQL Connection not available. 'DELETE FROM celery_taskmeta WHERE celery_taskmeta.date_done < %(date_done_1)s' [{}]
PS I've checked this SO question but seems like it's due to a different exception: Celery log shows cleanup failed

Answer 1

The closest Celery has to retrying is short lived sessions.
The task is cleaning out un-read task results. If it's failing, you may see those results start to build up, but should be OK otherwise.
You're right that there's very little documentation about it!