NCBI (the National Center for Biotech Info) generously provided their data for 3rd parties to consume. The data is located in cloud buckets such as gs://sra-pub-run-1/. I would like to read this data without incurring additional costs, which I believe can be achieved by reading it from the same region as where the bucket is hosted. Unfortunately, I can't figure out in which region the bucket is hosted (NCBI mentions in their docs that's in the US, but not where in the US). So my questions are:
Is there a way to figure out in which region a bucket that I don't own, like gs://sra-pub-run-1/ is hosted?
Is my understanding correct that reading the data from instances in the same region is free of charge? What if the GCS bucket is multi-region?
Doing a simple gsutil ls -b -L either provides no information (when listing a specific directory within sra-pub-run-1 or I get a permission denied error if I try to list info on gs://sra-pub-run-1/ directly using:
gsutil -u metagraph ls -b gs://sra-pub-run-1/
You cannot specify a specific Compute Engine zone as a bucket location, but all Compute Engine VM instances in zones within a given region have similar performance when accessing buckets in that region.
Billing-wise, egressing data from Cloud Storage into a Compute Engine instance in the same location/region (for example, US-EAST1 to US-EAST1) is free, regardless of zone.
So, check the "Location constraint" of the GCS bucket (gsutil ls -Lb gs://bucketname ), and if it says "US-EAST1", and if your GCE instance is also in US-EAST1, downloading data from that GCS bucket will not incur an egress fee.
Related
I want to take Oracle database backup using RMAN directly into the Google Cloud Storage
I am unable to find the plugin to use to take the RMAN backups into Cloud Storage. We have a plugin for Amazon S3 and am looking for one such related to Google Cloud Storage.
I don't believe there's an official way of doing this. Although I did file a Feature Request for the Cloud Storage engineering team to look into that you can find here.
I recommend you to star the Feature Request, for easy visibility and access, allowing you to view its status updates. The Cloud Storage team might ask questions there too.
You can use gcsfuse to mount GCS bucket as file systems on your machine and use RMAN to create backups there.
You can find more information about gcsfuse on its github page. Here are the basic steps to mount a bucket and run RMAN:
Create a bucket oracle_bucket. Check that it doesn't have a retention policy defined on it (it looks like gcsfuse has some issues with retention policies).
Please have a look at mounting.md that describes credentials for GCS. For example, I created a service account with Storage Admin role and created a JSON key for it.
Next, set up credentials for gcsfuse on your machine. In my case, I set GOOGLE_APPLICATION_CREDENTIALS to the path to JSON key from step 1. Run:
sudo su - oracle
mkdir ./mnt_bucket
gcsfuse --dir-mode 755 --file-mode 777 --implicit-dirs --debug_fuse oracle_bucket ./mnt_bucket
From gcsfuse docs:
Important: You should run gcsfuse as the user who will be using the
file system, not as root. Do not use sudo.
Configure RMAN to create a backup in mnt_bucket. For example:
configure controlfile autobackup format for device type disk to '/home/oracle/mnt_bucket/%F';
configure channel device type disk format '/home/oracle/mnt_bucket/%U';
After you run backup database you'll see a backup files created in your GCS bucket.
I am currently using the Google Cloud Shell, and I wish to access the persistent disk of another user. (Not using local shell)
More info on topic of inquiry: https://medium.com/google-cloud/no-localhost-no-problem-using-google-cloud-shell-as-my-full-time-development-environment-22d5a1942439
Cloud Shell is a micro vm dedicated to you, free, and with a mounted personal disk.
EDITED: Thanks to #Johnhanley comment, you can access to the cloud shell file of someone else with this code provided. However, you need the credential of the target Cloud Shell env and it's not very secure and recommended.
However, you can mount a fuse directory. And the other user also. With fuse, you navigate in a bucket like in directory. But be carefull, Storage bucket is not a file system: performance and usage aren't the same. Moreover, Fuse don't guaranty the data integrity in case of simultaneous file use, especially writing concurrency. Use with precaution.
But you can have a common workspace if it's your requirement.
If you use Cloud Shell as dev environment, like a computer or a VM, the same best practice are to apply. The dev environment has to be considered as ephemeral (computer can have outage or be lost/stolen, People can leave a company and you no longer have access to their cloud shell), and thereby you have to save your sources frequently on safe space (Git repository, Cloud Storage with Fuse)
I have a wordpress site installed on a compute engine instance.
I would like to move it on a new instance with SSD disk and other region as now it runs on standard persistence disk and Google opened new datacenter in my country.
I have upgrated the Ubuntu server to get HTTP/2 support and installed ssl keys for the https support.
Is there a way to move the existing instance with all configuration to new without needing to do it all again?
I read about the disk snapshots but am not sure if it includes everything from the old instance or is it just partial (missing the server update, ssl keys etc)?
It would be great if you guys could help me out because I am a Google cloud noobie with limited technical knowledge.
Thank in advance!
If you're looking to move the compute engine instance to another zone within the same region you can simply just use the gcloud compute instances move command in the Google Cloud shell to move it to a different zone.
Quote from the documentation:
For example, running
gcloud compute instances move example-instance-1 \
--zone us-central1-b --destination-zone us-central1-f
Will move the instance called example-instance-1 with its all attached
persistent disks, currently running in us-central1-b, to
us-central1-f.
References:
https://cloud.google.com/compute/docs/regions-zones/
https://cloud.google.com/sdk/gcloud/reference/compute/instances/move
https://cloud.google.com/compute/docs/instances/moving-instance-across-zones
I would like to know if there is a way to mount google cloud storage bucket as a folder for the first time
and each time we read the file, cache it locally (so it won't use money/bandwidth).
GCSFUSE has two type of caching available, Stat caching and type caching. You can refer to this document which provide detailed information on these types of caching with there trade-offs.
When manually creating a new cloud storage bucket using the web-based storage console (https://console.developers.google.com/), is there a way to specify the DRA attribute? From the documentation, it appears that the only way to create buckets with that attribute is to either use Curl, gsutil or some other script, but not the console.
There is currently no way to do this.
At present, the storage console provides only a subset of the Cloud Storage API, so you'll need to use one of the tools you mentioned to create a DRA bucket.
For completeness, it's pretty easy to do this using gsutil (documentation at https://developers.google.com/storage/docs/gsutil/commands/mb):
gsutil mb -c DRA gs://some-bucket