In Celery tasks, do retry_backoff and retry_backoff_max affect the manual self.retry() calls? - celery

Question

So according to the docs, default_retry_delayspecifies the time in seconds before a retry of the task should be executed. I just want to be sure that this only affects the manually called self.retry()calls and not the autoretries triggered by Celery when the task encounters predefined exceptions.
Likewise, I want to know if retry_backoff and retry_backoff_maxonly affect the autoretries and not the manual self.retry().
Finally, what happens when all of these are set?

Answer 1

That answer may change depending on the version of Celery that is used. However, as far as I have checked with the source code of versions v4.3.1, v4.4.7 and v5.0.0, retry_backoff and retry_backoff_max only affect the auto-retry. If autoretry_for is not set as True (or a True value such as positive integers), then the backoff parameters is not used.
As it can be seen from the implementation, when a True value is set to autoretry_for, Celery uses retry_backoff, retry_backoff_max, and retry_jitter arguments in order to compute the countdown of the next retry, before calling .retry method. Thus, these retry_xx arguments are not directly used by the .retry method; however, they affect the value of countdown. Thus, they are only used in the auto_retry.
When the countdown is explicity defined or computed via autoretry, then the value of default_retry_delay is ignored, as it can be seen in the implementation of v5.0.0. The same is valid when eta is set. Thus, to be able to use default_retry_delay, countdown, and eta should not be set. Additionally, autoretry_for should not be set or it should be set with a False value.
Another approach to use default_retry_delay and autoretry at the same time would be to define autoretry and retry variables; but for manual retry calls, set the eta and countdown to None explicitly. One example:
#app.task(bind=True, default_retry_delay=360, autoretry_for=[ExternalApiError], max_retries=5, retry_backoff=1800, retry_backoff_max=8 * 3600, retry_jitter=False)
def my_task(self, data):
try:
... # some code here that may raise ExternalApiError
except SomeOtherError:
try:
self.retry(countdown=None, eta=None)
except self.MaxRetriesExceededError:
... # when maximum number of retries has been exceeded.
If you want to use backoff parameters with manual retry calls, it can be implemented by using the method get_exponential_backoff_interval of Celery. One example could be this:
from celery.utils.time import get_exponential_backoff_interval
#app.task(bind=True, max_retries=5)
def my_task(self, data):
try:
... # some code here that may raise ExternalApiError
except SomeOtherError:
try:
retry_backoff = True
retry_backoff_max = 5
retry_jitter = True
countdown = get_exponential_backoff_interval(
factor=retry_backoff,
retries=self.request.retries,
maximum=retry_backoff_max,
full_jitter=retry_jitter)
)
self.retry(countdown=countdown)
except self.MaxRetriesExceededError:
... # when maximum number of retries has been exceeded.