I am experiencing slow deletion for a GCS Multi-Regional bucket.
I was wondering if this is normal performance to be expected since the bucket is Multi-Regional.
My bucket is being deleted programmatically by Terraform, and the delete step has taken 16 minutes:
google_storage_bucket.<REDACTED>: Still destroying... (ID: <REDACTED>, 16m30s elapsed)
When i go into the GCS console, when I check (click, select) the bucket, and click Delete, it takes a long time while a tooltip appears saying "Checking the bucket".
After that it asks me if I want to delete the X number of items.
After I choose yes, it tells me it failed.
When I first did the delete step in GCS console, it said 146 items.
I repeated it again later and it says 102 items, which probably means 40 items were deleted.
How can I delete this bucket properly?
Is this performance expected, since it is multi-regional?
UPDATE:
33 minutes and it is still deleting (as per Terraform)
UPDATE:
Deletion is complete
google_storage_bucket.vault: Destruction complete after 52m48s
So I don't need a fix, but it'll be nice to know if this is normal expected performance.
This is expected behavior, when a bucket is going to be deleted, all objects are recursively listed to know if they are all deleted. Usually this is pretty quick, but it can take a very long time if there are a lot of objects.
As a workaround you can delete the bucket faster running gsutil rm -m -r gs://bucket
to perform parallel (multi-threaded/multi-processing) removes.
Also there is an already filed feature request for this issue, you can click on the Me too! button to indicate that you are affected by this issue.
Related
I have a bucket containing many millions of blobs that I want to delete however I can't simply delete the bucket. This is the best method I have come up with to delete millions of blobs in the quickest time possible:
gsutil ls gs://bucket/path/to/dir/ | xargs gsutil -m rm -r
For what I want to do (which involves removing about 30million blobs) it still takes many hours to run, partly I guess because its at the mercy of the speed of my broadband connection.
Anyone know of a quicker way of achieving this? I had kinda hoped it'd be an instantaneous operation as in the backend the location could simply be marked as deleted - clearly not.
Google recommend using the console to do this
The Cloud Console can bulk delete up to several million objects and does so in the background. The Cloud Console can also be used to bulk delete only those objects that share a common prefix, which appear as part of a folder when using the Cloud Console.
https://cloud.google.com/storage/docs/best-practices#deleting
That said (personal opinion here) using the console might be quicker but you haven't a clue how far its got. At least with cli option you do know.
Another alternative is using lifecycle management to delete based on rules:
Delete objects in bulk
If you want to bulk delete a hundred thousand or more objects, avoid
using gsutil, as the process takes a long time to complete. Instead,
use the Google Cloud console, which can delete up to several million
objects, or Object Lifecycle Management, which can delete any number
of objects.
To bulk delete objects in your bucket using Object Lifecycle
Management, set a lifecycle configuration rule on your bucket where
the condition has Age set to 0 days, and the action is set to delete.
From: https://cloud.google.com/storage/docs/deleting-objects#delete-objects-in-bulk
However this won't work if you're in a rush:
After you have added or edited a rule, it may take up to 24 hours to take effect.
I need to backup a bucket in which every file is encrypted with a different key to a different bucket on Google Storage.
I want to create a daily snapshot of the data so in a case where the data has been deleted I could easily recover it.
My Research:
Using gsutil cp -r - because every file has a different key it does not work
Using Google Transfer | cloud - does not work on such buckets from the same reason
List all the files in the bucket and fetch all the keys from the database and copy each file - this will probably be very expensive to do because i have a lot of files and i want to do it daily
Object versioning - Does not cover a case where the bucket has been completely deleted
Are there any other solutions for that problem?
Unfortunately, as you mentioned the only option indeed, would be to follow your number 3 choice. As you said and as clarified in this official documentation here, download of encrypted data is a restricted feature, so you won't be able to download/snapshot the data, without fetching the keys and then copying the files.
Indeed, this will probably make a huge impact in your quota and pricing, since you will be performing multiple operations everyday, for multiple files, which will affect multiple aspects on the pricing. However, this seems to be the only available way right now. In addition to this, I would recommend you to raise a Feature Request in Google's Issue Tracker, so they can check about the possibility of implementing this in the future.
Let me know if the clarifed your doubts!
4 million JPG files, approximately 30TB in size. I deleted it via their web interface, and it currently states "Deleting 1 bucket", and has done for an hour.
Just after someone's experience for a rough estimation as to how long this operation will take - another hour? A day? A week?!
Region: europe-west1, if that makes a difference.
Thank you!
According to this documentation on the deletion request timeline, on step 2 it says that:
Once the deletion request is made, data is typically marked for deletion immediately and our goal is to perform this step within a maximum period of 24 hours.
A couple of points to be also considered are that:
This timeline will vary depending on the number of files, so your case might take longer that that.
If you files are organized in different folders, it would take longer to delete them since the system would have to enter each directory to delete.
One thing that you could do to speed up the deletion process is to use this command for parrallel deletion:
gsutil rm -m gs://bucket
NOTE: I don't think that the fact that your storage is a nearline storage has any effect on the timeline of deletion but I could not find any confirmation for that on the documentation.
I wanted to avoid using Aerospike clients (e.g. for Python) and delete records from a set using native asinfo command 'truncate' as it allows to do it quickly. But after I restarted Aerospike all deleted records were back. I saw this aerospike: delete all record in a set but it doesn't answer my question. Neither does this page from AS docs. It says, that a tombstone should be written after a durable delete, do I have to create it manually or are there some other ways?
UPD:
Thanks to #kporter who provided the accepted answer below I was able to look into differences between Community and Enterprise edition of Aerospike and found more information on the problem, some may find it helpful as well:
Persisted Delete [Community Edition]
This answer and the whole discussion from AS forum
And this thread
If I understood all of it correctly the best way to get your records deleted completely in CE is to ensure that they have right TTL and can expire naturally. And if for some reason you have a lot of old records without TTL as in my case, you can issue truncate command via asinfo and do not restart AS server until data on SSD is eventually overwritten. Or just truncate sets with old records on every restart.
Also I wonder if it is possible to wipe AS storage completely and then restore it from a backup of already truncated data as an emergency measure?
UPD1:
So, I was able to wipe SSD with Aerospike storage and restore only needed records from a backup. Here is how I did it:
Firstly, you need to remove old records from sets via asinfo and truncate command, links to docs are above
Then backup namespaces you want to save with asbackup
Stop your AS server, mine was in Docker container, so I just stopped said container
Zero out the disk that is used as AS storage, mine was /dev/sdb
Create necessary partitions on this disk
Start AS server
Restore data from the backup using arestore
Useful links: how to remove and clean up an aerospike server installation, AS docs on SSD setup
I am not sure if it is a good solution for large production setups but it worked as intended in my case with only one AS node and an opportunity to stop it for a while.
This way I was able to reduce the size of data in my AS from 160Gb to 11Gb and because of that my server now fully restarts only in half an hour instead of approximately eight hours as before.
You can find more information about truncating a set here:
https://www.aerospike.com/docs/operations/manage/sets/
As mentioned there, truncation is not durable in Aerospike Community.
In the Enterprise Edition, truncation is durable and preserves record deletions through a cold-restart. In the Community Edition, similar to record deletes, records in previously truncated sets are not durable and deletes can return through a cold-start.
I'm currently doing this, but it's VERY slow since I have several terabytes of data in the bucket:
gsutil du -sh gs://my-bucket-1/
And the same for a sub-folder:
gsutil du -sh gs://my-bucket-1/folder
Is it possible to somehow obtain the total size of a complete bucket (or a sub-folder) elsewhere or in some other fashion which is much faster?
The visibility for google storage here is pretty shitty
The fastest way is actually to pull the stackdriver metrics and look at the total size in bytes:
Unfortunately there is practically no filtering you can do in stackdriver. You can't wildcard the bucket name and the almost useless bucket resource labels are NOT aggregate-able in stack driver metrics
Also this is bucket level only- not prefixes
The SD metrics are updated daily so unless you can wait a day you cant use this to get the current size right now
UPDATE: Stack Driver metrics now support user metadata labels so you can label your GCS buckets and aggregate those metrics by custom labels you apply.
Edit
I want to add a word of warning if you are creating monitors off of this metric. There is a really crappy bug with this metric right now.
GCP occasionally has platform issues that cause this metric to stop getting written. And I think it's tenant specific (maybe?) so you also won't see it on their public health status pages. And it seems poorly documented for their internal support staff as well because every time we open a ticket to complain they seem to think we are lying and it takes some back and forth before they even acknowledge its broken.
I think this happens if you have many buckets and something crashes on their end and stops writing metrics to your projects. While it does not happen all the time we see it several times a year.
For example it just happened to us again. This is what I'm seeing in stack driver right now across all our projects:
Response from GCP support
Just adding the last response we got from GCP support during this most recent metric outage. I'll add all our buckets were accessible it was just this metric was not being written:
The product team concluded their investigation stating that this was indeed a widespread issue, not tied to your projects only. This internal issue caused unavailability for some GCS buckets, which was affecting the metering systems directly, thus the reason why the "GCS Bucket Total Bytes" metric was not available.
Unfortunately, no. If you need to know what size the bucket is right now, there's no faster way than what you're doing.
If you need to check on this regularly, you can enable bucket logging. Google Cloud Storage will generate a daily storage log that you can use to check the size of the bucket. If that would be useful, you can read more about it here: https://cloud.google.com/storage/docs/accesslogs#delivery
If the daily storage log you get from enabling bucket logging (per Brandon's suggestion) won't work for you, one thing you could do to speed things up is to shard the du request. For example, you could do something like:
gsutil du -s gs://my-bucket-1/a* > a.size &
gsutil du -s gs://my-bucket-1/b* > b.size &
...
gsutil du -s gs://my-bucket-1/z* > z.size &
wait
awk '{sum+=$1} END {print sum}' *.size
(assuming your subfolders are named starting with letters of the English alphabet; if not; you'd need to adjust how you ran the above commands).
Use the built in dashboard
Operations -> Monitoring -> Dashboards -> Cloud Storage
The graph at the bottom shows the bucket size for all buckets, or you can select an individual bucket to drill down.
Note that the metric is only updated once per day.
With python you can get the size of your bucket as follows:
from google.cloud import storage
storage_client = storage.Client()
blobs = storage_client.list_blobs(bucket_or_name='name_of_your_bucket')
blobs_total_size = 0
for blob in blobs:
blobs_total_size += blob.size # size in bytes
blobs_total_size / (1024 ** 3) # size in GB
Google Console
Platform -> Monitoring -> Dashboard -> Select the bucket
Scroll down can see the object size for that bucket
I found that that using the CLI it was frequently timing out. But that my be as I was reviewing a coldline storage.
For a GUI solution. Look at Cloudberry Explorer
GUI view of storage
For me following command helped:
gsutil ls -l gs://{bucket_name}
It then gives output like this after listing all files:
TOTAL: 6442 objects, 143992287936 bytes (134.1 GiB)