The costs to transfer data between regional standard bucket to regional nearline bucket (as backup) using the cloud storage transfer service is unclear because we don't know which operations the transfer job will execute in the background.
For example:
Regional standard buckets contains 100000 files in 1000 dirs and the total size is 1GB.
100 x class A storage.objects.list = 100/10000 x 0,05$ = 0,0005$
100000 x class B storage.*.get = 100000/10000 x 0,004$ = 0,04$
If target is empty, regional nearline bucket:
100000 x class A storage.*.insert = 100000/10000 x 0,1$= 1$
1GB x 0,013$/GB/month nearline storage = 0,013$
Will the count of operations be reduced if there is already some data in the target bucket?
thanks in advance,
noirabys
Per the Cloud Storage to Cloud Storage transfer pricing, Storage Transfer Service uses rewrite operations to move data between Cloud Storage buckets and it's a Class A operation. As it rewrites objects, it doesn't look like the number of operations is going to be reduced if some data is already in the target bucket.
As the Operations pricing mentions,
When changing the storage class of an object, either yourself or with
Object Lifecycle Management, the Class A rate associated with the
object's destination storage class applies. For example, changing
1,000 objects from Standard Storage to Coldline Storage counts as
1,000 Class A operations and is billed at the Class A operations rate
for Coldline Storage.
So if you transfer an object from a standard bucket to a nearline bucket, it's going to be calculated as a class A operation on a Nearline bucket.
Related
I'm using Firebase Realtime database, for my flutter project. I am curious about the storage which is limited to 1GB, I want to know how the used up storage appears, say for example if I store 1mb data it would show me 1mb storage occupied of GB storage, soo if I delete the 1MB data from the RTDB will it show 0MB used in 1Gb storage or it'll show 1MB because I already used it previously. I know this is a simple question, I just want to be clear.
Storage limit is what u store not what u read/write. So if u remove 1mb u have 1 GB to use.
For more info
Firebase bills for the data you store in your database and all outbound network traffic at the session layer (layer 5) of the OSI model. Storage is billed at $5 for each GB/month, evaluated daily. Outbound traffic includes connection and encryption overhead from all database operations and data downloaded through database reads. Both database reads and writes can lead to connection costs on your bill. All traffic to and from your database, including operations denied by security rules, leads to billable costs.
https://firebase.google.com/docs/database/usage/billing
I have the following questions with regards to GCS:
Is there a limit on a number of blobs a bucket can contain?
I am using java client to mass create blobs in a multi-threaded application. Is there a limit on a number of blobs created per time unit?
Does Google Cloud Storage API flag requests as a potential DDoS attack when a particular threshold is reached?
The Quotas and Limits page has some documentation about this. Specifically:
No, there is no limit on the number of objects in a bucket.
From the docs page:
There is no limit to writes across multiple objects. Buckets initially
support roughly 1000 writes per second and then scale as needed.
Google cloud storage documentation (https://cloud.google.com/storage/docs/storage-classes) suggests that there is "minimum storage duration" for Nearline (30days) and Coldline (90 days) storage but there is no description about regional and multi-regional storage.
Does it mean that there is absolutely no minimum storage duration? Is the unit by microsecond, second, minute, hour, or a day?
For example, (unrealistically) suppose that I created a google cloud storage bucket, and copied 10 petabyte of the data to the bucket, and removed each piece of data 1 minute after (or moved the data to another bucket I don't have to pay), then, is the cost of regional storage will be
$.02 (per GB per month) * 10,000,000 (GB) * 1/30/24/60 (months) = $4.62 ?
If the "unit" of GCS bucket usage time is an hour rather than a minute, then the cost will be $278. If it is 12 hours, the cost will be $3333, so there are huge differences in such an extreme case.
I want to create a "temporary bucket" that holds petabyte-scale data in a short period of time, and just wanted to know what the budget should be. The previous question (Minimum storage duration Google Cloud Storage buckets) did not help in answering to my question.
There is no minimum storage duration for regional or multi-regional buckets, but keep in mind that you will still have to pay operations costs to upload. At the time of this writing, that would be $0.05 per 10,000 files uploaded.
I believe the granularity of billing is seconds, so your initial calculation $4.62 would be correct. But note depending on the average size of your files, uploading a petabyte is likely to be more expensive than that; if you have a petabyte of files that are 100MB in size, the operations cost for the uploads would be ~$50.
I recently setup the Google Cloud Storage Access Logs & Storage Data and the logs are getting logged but I could see 4 logs at the same time.
For example:
usage_2017_02_14_07_00_00_00b564_v0
usage_2017_02_14_07_00_00_01b564_v0
usage_2017_02_14_07_00_00_02b564_v0
usage_2017_02_14_07_00_00_03b564_v0
So there are 4 usage logs logged for every hour, what's the different between them.
I connected all the logs to big query to query the table - and all 4 of them have different values.
Also analysing on storage logs - I could see storage_byte_hours to 43423002260.
How to calculate the cost from storage_byte_hours?
It is normal for GCS to sometimes produce more than one logfiles for the same hour. From Downloading logs (emphasis mine):
Note:
Any log processing of usage logs should take into account the possibility that they may be delivered later than 15 minutes after the
end of an hour.
Usually, hourly usage log object(s) contain records for all usage that occurred during that hour. Occasionally, an hourly usage log
object contains records for an earlier hour, but never for a later
hour.
Cloud Storage may write multiple log objects for the same hour.
Occasionally, a single record may appear twice in the usage logs. While we make our best effort to remove duplicate records, your log
processing should be able to remove them if it is critical to your log
analysis. You can use the s_request_id field to detect duplicates.
You calculate the bucket size from storage_byte_hours. From Access and storage log format:
Storage data fields:
Field Type Description
storage_byte_hours integer Average size in byte-hours over a 24 hour period of the bucket.
To get the total size of the bucket, divide byte-hours by 24.
In your case 43423002260 byte-hours / 24 hours = 1809291760 bytes
You can use the bucket size to estimate the cost for the storage itself:
1809291760 bytes = 1809291760 / 2^^30 GB ~= 1.685 GB
Assuming Multi Regional Storage (per GB per Month ) $0.026 your storage cost be:
1.685 GB x $0.026 = $0.04381 / month ~= $0.00146033333333 / day (w/ 30 days month)
But a pile of other data (network, ops, etc) is needed to compute additional related costs, see Google Cloud Storage Pricing.
I understand that local-ssd disk does not incur any charges as it is located with the compute engine. Assuming the disk to be standard persistent disk, does the ingress and egress charges between the compute engine and disks billable ?
No, there are no network charges associated with Persistent Disk. You are charged for the amount of provisioned space per disk (per GB, per month).
There are additional charges related to snapshots (for total snapshot size) and for network egress if you choose to restore a snapshot across regions.
Depends, as it is stated in the Google Cloud Platform website.
Egress in between the same zone is free of charge, but any egress from a zone will be billed, depending from where to where you move the data. This includes any data movement from persistent disks or buckets.
Even if you have a subnet in between two zones in the same region, you will be charged for the egress.