I have followed the directions provided by Google to delete some old buckets in the in my account ... it is very straight forward process listed however after confirming the deletion to occur the "Preparing to Delete" pops up on the bottom left, but the system never deletes the files and bucket ?
I have posted this several times but no one have suggested a solution or a reason why the process does not work.
If you have a lot of files in your bucket, it might simply take a long time to perform the operation.
As a workaround to the UI being unclear, you can use gsutil to remove all files in a bucket, followed by the bucket itself, using gsutil rm -r gs://bucket.
In my experience, when the bucket has lots of objects, using the web interface or gsutil alone is not the best way.
What I did, was that I added a life cycle rule, to have Google delete all the object in the bucket.
Then, coming back after a day, the bucket can easily be deleted.
Related
I want to ensure deleted files have a window of recovery. I'd like to use the primitives offered by google cloud storage such that I don't have to maintain the logic necessary to prevent files deleted in error from being irrecoverable.
I do not see a better way to achieve than doing:
create normal bucket for files that are displayed to the users
create trash bucket for files pending permanent deletion with lifecycle rule that deletes objects after N days of creation
upon file deletion request from the normal bucket, first copy files to the trash bucket , then deletion of file from normal bucket
What is the "idiomatic" way of implementing delayed permanent deletion GCP cloud storage?
NOTE: I am trying to avoid chron jobs or additional database interaction
NOTE: this is not a soft delete as the file is expected to eventually be permanently deleted without any trace/storage associated with it
You can keep all the files in the same bucket like this, assuming two things:
Each file is also referenced in a database that you're querying to build the UI.
You're able to write backend code to manage the bucket - peoeple are not dealing with files directly with a Cloud Storage SDK.
It uses Cloud Tasks to schedule the deletion:
User asks for file deletion.
File is marked as "deleted" in the database, not actually deleted from bucket.
Use Cloud Tasks to schedule the actual deletion 5 days from now.
On schedule, the task triggers a function, which deletes the file and its database record.
Your UI will have to query the database in order to differentiate between deleted and trashed files.
I am working on a project for a client and a couple of weeks ago most of the content "disappeared".
Images and videos are routed through FileStack (a file processing service) but actually stored on Google Cloud Storage in one bucket.
On the day in question everything was working, and then everything stopped working. When we investigated it turned out that the bucket FileStack was pointing to was non-existent, so we created a new bucket with the same name and everything magically worked itself out.
Now my question is, where did all the files from the disappeared bucket go? Is it possible to get them back? Is it possible to figure out what happened?
I have extensively reviewed the audit log in the Activity tab and it shows zero activity for the bucket in question. Is there anywhere else we can investigate?
Can you please send email to gs-team#google.com, noting the bucket name and an example object name from that bucket, along with the last time you were successfully able to access that bucket/object? Doing it that way will avoid exposing these names on the public forum. Please mention my name in the message, so I will get it and can investigate.
Thanks,
Mike Schwartz
GCS Team
When an object is deleted, it's deleted from the system and there isn't any option to recover it [1]. You can prevent this behavior by using object versioning [2]. And to get a better overview of the activity in Cloud Storage you can enable the "Data Access logs" [3].
About the reason why the objects has disappeared, as a first workaround you can review if there's an Object Lifecycle enabled [4].
https://cloud.google.com/storage/docs/deleting-objects
https://cloud.google.com/storage/docs/object-versioning
https://cloud.google.com/storage/docs/audit-logs
https://cloud.google.com/storage/docs/lifecycle
We store some of our sql files in storage, and load them to execute from time to time.
Today at some moment one of those files became 503 Service Unavailable.
The interesting part is that we have few of these files in one folder, and the rest were ok, except this one.
Is this google side issue, if so, what are guarantees that this won't happen again?
I can provide more detailed information to a google guy if needed, the project id, and the files, and the logs etc.
When objects are uploaded into a GCS bucket and shared publicly, the owner of the bucket is responsible for all of the costs of users downloading these objects. How do I change that so that the downloaders are billed instead of me?
This feature is called "Requester Pays." Its documentation is here: https://cloud.google.com/storage/docs/requester-pays
The idea is that you mark a bucket as being a "requester pays" bucket. Once you've done that, your project is only responsible for the price of storing the objects in the bucket (and, if it's a nearline or coldline bucket, any early deletion fees). Anyone that wants to download an object from this bucket (or upload a new object, copy an object, etc) must specify which of their projects GCS should bill for the operation.
This is a very useful configuration for situations where you wish to make objects publicly available but don't want to be responsible for the cost of distributing it to many end users.
To enable Requester Pays on a bucket, open the Cloud Storage browser, find your bucket, and click the "off" button in the "Requester Pays" column, and follow the prompts. You can also set this flag in other ways, see the docs: https://cloud.google.com/storage/docs/using-requester-pays#enable
Downloading objects from requester pays buckets requires a Google Cloud project with billing enabled. Once you have that, you can download the object from the cloud console or using gsutil:
$> gsutil -u [PROJECT_ID] cp gs://[BUCKET_NAME]/[OBJECT_NAME] [OBJECT_DESTINATION]
The trick to this command is the -u [PROJECT_ID] bit, which specifies which project should be billed for the download.
You can also download the object using our other APIs or with the cloud console. More in the docs: https://cloud.google.com/storage/docs/using-requester-pays#using
I use gcloud node v0.24 for interacting with Google Cloud Storage. I've encountered an issue when immediate list after upload doesn't return all the files that were uploaded.
So the question is
does Bucket#getFiles always list files right after Bucket#upload?
or
is there any delay between upload's callback and when file becomes available (e.g. can be listed, downloaded)?
Note: below answer is no longer up to date -- GCS object listing is strongly consistent.
Google Cloud Storage provides strong global consistency for all read-after-write, read-after-update, and read-after-delete operations, including both data and metadata. As soon as you get a success response from an upload message, you may immediately read the object.
However, object and bucket listing is only eventually consistent. Objects will show up in a list call after you upload them, but not necessarily immediately.
In other words, if you know the name of an object that you have just uploaded, you can immediately download it, but you cannot necessarily discover that object by listing the objects in a bucket immediately.
For more, see https://cloud.google.com/storage/docs/consistency.