I'm reading docs about how to use google cloud, particularly to store data on a bucket.
I can see the gcloud scp command to upload file to a VM in a secure way (highlighted in the doc).
To uload to a bucket, it's said to use gsutil cp
Is this command secure ? If I want to upload sensitive data, do I have to take more precautions (and how)
As per the documentation:
By default, gsutil accesses Cloud Storage through JSON API request endpoints. You can change this default to the XML API.
The JSON API request endpoint is HTTPS - so assuming the security provided by HTTPS is sufficient for your needs, it should be fine. That won't guard against attacks if your local machine has been compromised with a bogus version of gsutil, but at that point all bets are probably off.
Related
Using Google Cloud Launcher we've deployed a Mongodb replicaset.
We are know configuring backups being uploaded to buckets.
Under the VM Cloud API access scopes, with the machine stoped, we've given Full access to Storage
When we try to upload using gsutil cp, we get the following error:
Copying file://whateverfilewe try [Content-Type=application/octet-stream]...
AccessDeniedException: 403 Insufficient OAuth2 scope to perform this operation.
Acceptable scopes: https://www.googleapis.com/auth/cloud-platform
Reading documentation, that scope seems way too much.
How should we proceed in order to give access to to a bash script (using the machines assigned service account) within a Compute engine instance upload access to a bucket inside the same project? Is full access really necessary?
Seems gsutil was caching credentials. This comment gave me the solution:
gsutil copy returning "AccessDeniedException: 403 Insufficient Permission" from GCE
I'm using Google Cloud Storage to serve my static website to the public and wondering is there a way to enable Apache/NGINX like access logs for the bucket via the GCP web interface?
GCS does offer access logs, although they arrive as CSV files with a bunch of information and not as Apache logs.
Enabling them is fairly simple. Say you want access logs for bucket "mybucket".
First, create another bucket to hold the access logs. Let's call it "mylogsbucket".
Second, give GCS permission to write logs to that bucket with this gsutil command:
gsutil acl ch -g cloud-storage-analytics#google.com:W gs://mylogsbucket
Third, activate logging:
gsutil logging set on -b gs://mylogsbucket gs://mybucket
Usage logs for mybucket will now show up about once per hour in mylogsbucket, and storage logs recording how much data is being stored will show up once per day.
More documentation on this feature is here: https://cloud.google.com/storage/docs/access-logs
I've got a bucket in Google Cloud Storage, and a website. People can currently upload to the bucket through the website (using Google authentication).
However, I need to set it so that anyone can view the files that are uploaded (and can't modify them).
This can't be something that Google needs to authenticate, as some of our clients' IT departments have blocked Google (for whatever reason) and refuse to budge. It could be something where the request is made from my website, it could allow it (as I'll record the URL on the website's database).
Preferably, if this could be done without using gsutil that would be great.
You can set a default object ACL on the bucket that makes all objects uploaded to that bucket publicly readable. For example you could do it using gsutil:
gsutil defacl ch -u AllUsers:R gs://your-bucket
Note that the above command only affects newly written objects. If you already have objects in your bucket that need to be made public you could accomplish that with gsutil as well:
gsutil acl ch -u AllUsers:R gs://your-bucket/**
Regarding your point about making sure anyone can view the files but not modify them: You can accomplish this by making sure the bucket ACL only allows you (or your service account) to write objects, not all users.
I have a Google Compute VM (LAMP) webserver set up to copy files to a Google Storage Bucket, which then need to be accessed (read and write) by a program on a Google Compute VM (Windows 2008). I can't seem to find any documentation about how a Google Compute Engine Windows VM can access storage buckets.
Is there a way this is possible? Thanks.
I'm doing the same thing, but not with a windows VM, but I think the principle is the same.
First you need to allow Project Access for your VM from the Google Cloud Console https://console.developers.google.com/project, see the screenshot below:
Once you've done this you need to call the metadata server to get an access token from your program. You need to make a HTTP call to the metadata server, here is an example from the docs (https://cloud.google.com/compute/docs/authentication) using curl, bear in mind when programming this you need to also provide the header "Metadata-Flavor: Google":
$ curl "http://metadata/computeMetadata/v1/instance/service-accounts/default/token" \
-H "Metadata-Flavor: Google"
{
"access_token":"ya29.AHES6ZRN3-HlhAPya30GnW_bHSb_QtAS08i85nHq39HE3C2LTrCARA",
"expires_in":3599,
"token_type":"Bearer"
}
You obviously need to code this HTTP call and the parsing of the JSON data in whichever programming language you are using for your program and extract the "access_token", based on the "expires_in" field you might also need to implement a mechanism to fetch a new token once it expires. You can then use the Google supplied cloud storage client library (https://cloud.google.com/storage/docs/json_api/v1/libraries) for your programming language and use the access token above for authenticating calls to cloud storage. I use Java and the Cloud storage class in the API library has this method that can be used:
.setOauthToken("blah")
You can mount the drive with CloudBerry. I would like to find a better way to do it though using only Google Cloud. Please let me know if you find anything better.
I want to use WAL-E to backup my PostgreSQL data to Amazon S3. I am trying to determine if the data is encrypted in transit from Postgres to S3 using SSL/TLS but the documentation is not entirely clear on this point. I see WAL-E uses boto library and I believe by default it uses SSL/TLS but can anybody confirm or tell me how to configure WAL-E to ensure it does use SSL/TLS?
HTTPS is the default, but you can manually specify the S3 endpoint to force the protocol.
https://github.com/wal-e/wal-e#manually-specifying-the-s3-endpoint
The format is that of:
protocol+convention://hostname:port Where valid protocols are http and
https, and conventions are path, virtualhost, and subdomain.
Example:
Turns off encryption and specifies us-west-1 endpoint.
WALE_S3_ENDPOINT=http+path://s3-us-west-1.amazonaws.com:80
For radosgw.
WALE_S3_ENDPOINT=http+path://hostname
As seen when using Deis, which uses radosgw.
WALE_S3_ENDPOINT=http+path://deis-store-gateway:8888