We are building our stack on google cloud builds and for building we are using custom docker base images which are stored in gcr.io/project-name/image-name
While using this method we are e getting charged on Download Worldwide Destinations (excluding Asia & Australia)
Is there any way that we can reduce the High download charges? if we will run cloud builds and pull docker images from same region i.e. running docker build on us-central1 and pulling docker image from us-central1-docker.dev.pkg/project-name/image-name will it reduce the download charges (No charge) ?
As we found one ref : https://cloud.google.com/storage/pricing#network-buckets
Or is there any other solution ?
Just to expand on #John Hanley's comment, according to this documentation on location considerations:
A good location balances latency, availability, and bandwidth costs for data consumers.
Choosing the closest and same region will help optimize latency and network bandwidths. It would also be convenient to choose the region where it contains the majority of your data users.
There is a Cloud Storage Always Free usage limits wherein 1GB Network Egress is free from North America to each GCP egress destination (excluding Australia and China) however starting October 1, 2022, it would be upgraded to 100GB You can check the full documentation on Changes to Always Free usage limits.
Related
Architecture which we are using currently is as below
Private Web App Services hosted in US Region and India Region.
Both the apps are behind the respective App Gateway, this app Gateway is behind the front door which helps us serve the request from the nearest app gateway. But Both apps uses the same postgres which is present in US region.
Now our issue is when we hit the api from US response time is less then 2sec whereas when we hit the api from India region it takes 70sec.
How can we reduce the latency ?
Actually, the problem is the APIs does write operation due to which we cannot a read replica.
There a few things you can do
1- Add a cache layer to both regions and rather than querying directly on DB, check if the data is available in the cache first, and if it's not, get it from DB and add to the cache layer.
2- Add a secondary database on India region which will be a read only.
PS: You may have stale data with both approaches so you should sync properly according to your requirements
I have been monitoring Cloud Storage billing daily and saw two unexpected, large spikes in "Download Worldwide Destinations (excluding Asia & Australia)" this month. The cost for this SKU is typically around US$2-4 daily; however, these two daily spikes have been $89 and $15.
I have enabled GCS Bucket Logging soon after the $89 spike, hoping to deduce what causes it the next time it happens, but when the $15 spike happened yesterday, I was unable to pinpoint which service or files downloaded have caused this spike.
There is a Log field named Location, but it appears to be linked to the region where a bucket is located, not the location of the downloader (that would contribute to the "Worldwide Destinations" egress).
As far as I know, my services are all in the southamerica-east1 region, but it's possible that there is either a legacy service or a misconfigured one that has been responsible for these spikes.
The bucket that did show up outside my region is in the U.S., but I concluded that it is not responsible for the spikes because the files there are under 30 kB and have only been downloaded 8 times according to the logs.
Is there any way to filter the logs so that it tells me as much information as possible to help me track down what is adding up the "Download Worldwide Destinations" cost? Specifically:
which files were downloaded
if it was one of my Google Cloud services, which one it was
Enable usage logs and export the log data to a new bucket.
Google Cloud Usage logs & storage logs
The logs will contain the IP address, you will need to use a geolocation service to map IP addresses to city/country.
Note:
Cloud Audit Logs do not track access to public objects.
Google Cloud Audit Logs restrictions
I have an application which require more than 30GB of memory and more than 4GB of disk space.
Can I run the app in any of cloud foundry environments (PCF or Bluemix - enterprise account)
Please help me on this query.
Bluemix default quota plan does not resolve your necessity, since the default plan allows only 8GB per instance (512GB max). You would need to open a ticket to change the quota plan of your organization.
Either way, to make sure about the quota plan being used by your organization, go to Manage > Account > Organization > Select Organization > Edit Org
In the quota section, look at the quota plan then login into cf tool and list the quota details:
cf login
cf quota QUOTA_PLAN
This link can give you a little more help.
This depends entirely on the Cloud Foundry provider that you're using and the limits that they put in place.
Behind the scenes, it also depends on the types of VMs being used for Diego Cells in the platform. The Cells are where your application code will run and there must be enough space on a Cell to run your app instance. As an example, if you have a total of 16G of RAM on your Diego Cells then it wouldn't be possible for a provider to support your use case of 30G for one application instance since there would be no Cells with that much free space. If you had Cells with 32G of RAM, that might work, depending on overhead and placement of other apps, but you'd probably need something even larger like 64G per Cell.
I mention all this because at the end of the day, if you're willing to run your own Cloud Foundry installation you can pretty much do whatever you want, so running with 30G or 100G isn't a problem as long as you configure and scale your platform accordingly.
Hope that helps!
I am trying to figure out a proper solution for the following:
We have a client from which we want to receive data, for instance a binary that is 200Mbytes updated daily. We want them to deposit that data file(s) onto a local server near them (Europe).
We then want to do one of the following:
We want to retrieve the data, either from a local
server where we are (China/HK), or
We can log into their European
server where they have deposited the files and pull the files directly ourselves.
QUESTIONS:
Can Google's clould platform serve as a secure, easy way to provide a cloud drive for which to store and pull the data file?
Does Google's cloud platform distribute such that files pushed onto a server in Europe will be mirrored in a server over in East Asia? (that is, where and how would this distribution model work with regard to my example.)
For storing binary data, Google Cloud Storage is a fine solution. To answer your questions:
Secure: yes. Easy: yes, in that you don't need to write different code depending on your location, but there is a caveat on performance.
Google Cloud Storage replicates files for durability and availability, but it doesn't mirror files across all bucket locations. So for the best performance, you should store the data in a bucket located where you will access it the most frequently. For example, if you create the bucket and choose its location to be Europe, transfers to your European server will be fast but transfers to your HK server will be slow. See the Google Cloud Storage bucket locations documentation for details.
If you need frequent access from both locations, you could create one bucket in each location and keep them in sync with a tool like gsutil rsync
I would like to better fit block size on some disks to the average file size, but it will be useless if it doesn't fit the stripe size.
I couldn't find a way to even view the stripe size about in the documentation.
Google Cloud Storage operates at a far higher level of abstraction than you appear to desire -- at that level, there's even no such thing as "block size", forget details such as "stripe size".
If you do want to work at very low levels of abstractions, therefore, you'll absolutely have to forget GCS and start thinking in terms of GCE instances (AKA VMs) with persistent disk, possibly of the SSD varieties (shareable or local). Even then, given the way Google's virtualization is architected, you may get frustrated with the still-too-high-for-you level of abstraction.
I'll admit I'm curious about the implied request for very-low-level access and I'd be glad to relay the details to my friends and colleagues in the Storage line -- or, you could of course open a detailed feature request in our public issue tracker, my group (cloud tech support) monitors that as well as Stackexchange sites and relevant Google Groups, and we're always glad to be a bridge between our customers and our Eng/PM colleagues.