I have been trying to see if we should setup a GCS bucket with region or multi-region. The needs today are regional only.
I am wondering if there is an option to change from region only to multi-region at a later point in time?
This also saves some $'s till we move to multi-region setting. Just having it enabled, doesn't help us much today.
There is no one-step solution for moving objects from being regional to multi-regional. If you have a bucket containing a lot of regional objects and you want them to be multi-regional, you'll need to copy them into a new multi-regional bucket. The Cloud Storage Transfer Service can manage this for you, or you can do it with gsutil.
You can change a bucket's configuration to cause new objects to have a different storage class, but it won't effect existing objects. Also, this will only help you switching to and from nearline and coldine storage classes, as multi-regional and regional storage classes do not share locations, and bucket locations cannot be changed.
Related
Currently we have buckets which are set to multi-region but this can be regional. Is there a way we can change it in Google Cloud Bucket settings (or) should I create a new regional bucket and move the data ? any links to Google documentation will be great.
I have been monitoring Cloud Storage billing daily and saw two unexpected, large spikes in "Download Worldwide Destinations (excluding Asia & Australia)" this month. The cost for this SKU is typically around US$2-4 daily; however, these two daily spikes have been $89 and $15.
I have enabled GCS Bucket Logging soon after the $89 spike, hoping to deduce what causes it the next time it happens, but when the $15 spike happened yesterday, I was unable to pinpoint which service or files downloaded have caused this spike.
There is a Log field named Location, but it appears to be linked to the region where a bucket is located, not the location of the downloader (that would contribute to the "Worldwide Destinations" egress).
As far as I know, my services are all in the southamerica-east1 region, but it's possible that there is either a legacy service or a misconfigured one that has been responsible for these spikes.
The bucket that did show up outside my region is in the U.S., but I concluded that it is not responsible for the spikes because the files there are under 30 kB and have only been downloaded 8 times according to the logs.
Is there any way to filter the logs so that it tells me as much information as possible to help me track down what is adding up the "Download Worldwide Destinations" cost? Specifically:
which files were downloaded
if it was one of my Google Cloud services, which one it was
Enable usage logs and export the log data to a new bucket.
Google Cloud Usage logs & storage logs
The logs will contain the IP address, you will need to use a geolocation service to map IP addresses to city/country.
Note:
Cloud Audit Logs do not track access to public objects.
Google Cloud Audit Logs restrictions
I'm trying to figure out if anyone can offer advice around bucket creation for an app that will have users with an album of photos. I was initially thinking of creating a single bucket and then prefixing the filename by user id, since google cloud storage doesn't recognize subdirectories, like so: /bucket-name/user-id1/file.png
Alternatively, I was considering creating a bucket and naming it by user id like so: /user-id1-which-is-also-bucket-name/file.png
I was wondering what I should consider in terms of cost and organization when setting up my google cloud storage. Thank you!
There is no difference in term of cost. In term of organization, it's different:
For the deletion, it's simpler to delete a bucket and not a folder in the unique bucket.
For performances, sharding is better is you have separate bucket (you have less chance to create an hotspot)
At billing perspective, you can add labels on the buckets, and get them in the billing exported to BigQuery. You can know the cost of the bucket per user, and maybe do a rebill to them
The biggest advantage of 1 bucket per user model is the security. You can grant a user (if the users have direct access to the bucket and don't use a backend service to access it) on a bucket, without the use of legacy (and almost deprecated) ACL on object. In addition, if you use ACL, you can't set ACL per folder, ACL are per object. So, everytime that you add an object in the unique bucket, you have to set the ACL on it. It's harder to achieve.
IMO, 1 bucket per user is the best model.
When objects are uploaded into a GCS bucket and shared publicly, the owner of the bucket is responsible for all of the costs of users downloading these objects. How do I change that so that the downloaders are billed instead of me?
This feature is called "Requester Pays." Its documentation is here: https://cloud.google.com/storage/docs/requester-pays
The idea is that you mark a bucket as being a "requester pays" bucket. Once you've done that, your project is only responsible for the price of storing the objects in the bucket (and, if it's a nearline or coldline bucket, any early deletion fees). Anyone that wants to download an object from this bucket (or upload a new object, copy an object, etc) must specify which of their projects GCS should bill for the operation.
This is a very useful configuration for situations where you wish to make objects publicly available but don't want to be responsible for the cost of distributing it to many end users.
To enable Requester Pays on a bucket, open the Cloud Storage browser, find your bucket, and click the "off" button in the "Requester Pays" column, and follow the prompts. You can also set this flag in other ways, see the docs: https://cloud.google.com/storage/docs/using-requester-pays#enable
Downloading objects from requester pays buckets requires a Google Cloud project with billing enabled. Once you have that, you can download the object from the cloud console or using gsutil:
$> gsutil -u [PROJECT_ID] cp gs://[BUCKET_NAME]/[OBJECT_NAME] [OBJECT_DESTINATION]
The trick to this command is the -u [PROJECT_ID] bit, which specifies which project should be billed for the download.
You can also download the object using our other APIs or with the cloud console. More in the docs: https://cloud.google.com/storage/docs/using-requester-pays#using
I am trying to figure out a proper solution for the following:
We have a client from which we want to receive data, for instance a binary that is 200Mbytes updated daily. We want them to deposit that data file(s) onto a local server near them (Europe).
We then want to do one of the following:
We want to retrieve the data, either from a local
server where we are (China/HK), or
We can log into their European
server where they have deposited the files and pull the files directly ourselves.
QUESTIONS:
Can Google's clould platform serve as a secure, easy way to provide a cloud drive for which to store and pull the data file?
Does Google's cloud platform distribute such that files pushed onto a server in Europe will be mirrored in a server over in East Asia? (that is, where and how would this distribution model work with regard to my example.)
For storing binary data, Google Cloud Storage is a fine solution. To answer your questions:
Secure: yes. Easy: yes, in that you don't need to write different code depending on your location, but there is a caveat on performance.
Google Cloud Storage replicates files for durability and availability, but it doesn't mirror files across all bucket locations. So for the best performance, you should store the data in a bucket located where you will access it the most frequently. For example, if you create the bucket and choose its location to be Europe, transfers to your European server will be fast but transfers to your HK server will be slow. See the Google Cloud Storage bucket locations documentation for details.
If you need frequent access from both locations, you could create one bucket in each location and keep them in sync with a tool like gsutil rsync