exponential backoff to batch delete objects in python client - google-cloud-storage

Question

my request looks like Batch request with Google Cloud Storage python client
blobs_to_delete = [blob for blob in bucket.list_blobs(prefix="my/prefix/here")]
for c in _chunk(blobs, batch_size=100):
with storage_client.batch():
for blob in c:
blob.delete()
the error is:
[2018-09-12 21:28:41,726] {base_task_runner.py:98} INFO - Subtask: File "/usr/local/lib/python2.7/site-packages/google/cloud/storage/batch.py", line 243, in _finish_futures
[2018-09-12 21:28:41,731] {base_task_runner.py:98} INFO - Subtask: raise exceptions.from_http_response(exception_args)
[2018-09-12 21:28:41,731] {base_task_runner.py:98} INFO - Subtask: google.api_core.exceptions.InternalServerError: 500 BATCH contentid://None: Backend Error
How to add truncated exponential backoff to my code?

Answer 1

The batch code does not include retry itself, nor does it allow you to find out exactly which requests from inside it failed. This means (a) you will have to do the retrying yourself and (b) you will have to retry the whole batch.
(a) can be made easier by using the retrying package. (b) is not a problem, as deleting a blob is idempotent.
Put together, a solution might look like:
def retriable_exception(e):
return isinstance(e, GoogleAPICallError) and (e.code == 429 or e.code>=500)
#retry(retry_on_exception=retriable_exception,
stop_max_attempt_number=7,
wait_exponential_multiplier=1000,
wait_exponential_max=10000)
def delete_batch(c):
with storage_client.batch():
for blob in c:
blob.delete()
blobs_to_delete = [blob for blob in bucket.list_blobs(prefix="my/prefix/here")]
for c in _chunk(blobs, batch_size=100):
delete_batch(c)