I have a Google Compute VM (LAMP) webserver set up to copy files to a Google Storage Bucket, which then need to be accessed (read and write) by a program on a Google Compute VM (Windows 2008). I can't seem to find any documentation about how a Google Compute Engine Windows VM can access storage buckets.
Is there a way this is possible? Thanks.
I'm doing the same thing, but not with a windows VM, but I think the principle is the same.
First you need to allow Project Access for your VM from the Google Cloud Console https://console.developers.google.com/project, see the screenshot below:
Once you've done this you need to call the metadata server to get an access token from your program. You need to make a HTTP call to the metadata server, here is an example from the docs (https://cloud.google.com/compute/docs/authentication) using curl, bear in mind when programming this you need to also provide the header "Metadata-Flavor: Google":
$ curl "http://metadata/computeMetadata/v1/instance/service-accounts/default/token" \
-H "Metadata-Flavor: Google"
{
"access_token":"ya29.AHES6ZRN3-HlhAPya30GnW_bHSb_QtAS08i85nHq39HE3C2LTrCARA",
"expires_in":3599,
"token_type":"Bearer"
}
You obviously need to code this HTTP call and the parsing of the JSON data in whichever programming language you are using for your program and extract the "access_token", based on the "expires_in" field you might also need to implement a mechanism to fetch a new token once it expires. You can then use the Google supplied cloud storage client library (https://cloud.google.com/storage/docs/json_api/v1/libraries) for your programming language and use the access token above for authenticating calls to cloud storage. I use Java and the Cloud storage class in the API library has this method that can be used:
.setOauthToken("blah")
You can mount the drive with CloudBerry. I would like to find a better way to do it though using only Google Cloud. Please let me know if you find anything better.
Related
My Java backend server has to upload files to the Google Cloud Storage (GCS).
Right now I just run
public void store(MultipartFile multipartFile) throws IOException {
Storage storage = StorageOptions.getDefaultInstance().getService();
storage.create(
BlobInfo.newBuilder(
BUCKET_NAME,
Objects.requireNonNull(multipartFile.getOriginalFilename()))
.build(),
multipartFile.getBytes()
);
}
Having set GOOGLE_APPLICATION_CREDENTIALS=$PROJECT_DIR$/project-1234-abcdefg.json in my environment.
However, this makes things complicated for my deployment setup. I don't know how I would go about making this file available to my service.
Is there another way to get access to GCS for my service account?
Background
I am deploying my server to Heroku as a compiled jar file and I don't know how to make the credentials available to my server during deployment.
You need a Google Account to access to GCS, either personal or technical. Technical is a service account.
However, you have another solution, but not really easy to implement. I wrote an article for securing serverless product with Cloud Endpoint with and API Key. Here your serverless solution can be Cloud Storage. But that implies that you call GCS with REST API and not with the java library, not very fun. That also implies additional cost for the hosting and the processing time of Cloud Endpoint.
Note: you can improve the authorization from API Key to Firebase auth or something else if you prefer. Check the Cloud Endpoint authentication capabilities
Note2: Google is working on another authentication mechanism but I don't know at which stage are the developments, and if it's plan for 2020. In any case, your constraint is known and addressed by Google
Trying to make use of the Server to Server OAuth flow defined here:
https://developers.google.com/identity/protocols/OAuth2ServiceAccount
Since I'm running from a local dev environment, I've created a service account in GCP and downloaded the JSON file with the private key, but cannot find any Node.js code examples on how to:
1) load the json file
2) set delegated credentials (for G Suite domain-wide authorization)
Places I've looked (besides stackoverflow) are Google's git wiki for the node.js client library, which does talk about server to server auth, but seems to assume you're running from appengine or google cloud and don't need to load a key file:
https://github.com/googleapis/google-api-nodejs-client#service-to-service-authentication
The Admin SDK Activities Reports API has a Node example, but it's using the web-based flow assuming a user is present:
https://developers.google.com/admin-sdk/reports/v1/quickstart/nodejs
Buried deep in the Node.js samples is use of the Directory API, which does seem to take a keyfile as input, but when I try running locally it says getClient is not a constructor, and still this example doesn't show how to set the G Suite admin user for context (which is generally when a refresh token and access token are loaded into the app):
https://github.com/googleapis/google-api-nodejs-client/blob/master/samples/directory_v1/group-delete.js
So... does anybody have an example of this? I really don't want to switch to a Python runtime but Google seems to have left out important examples on this topic.
I am newbie at cloud servers and I've opened a google cloud storage to host image files. I've verified my domain and configured it, to view images via my domain. The problem is, same file is both accessible via my domain example.com/images/tiny.png and also via storage.googleapis.com/example.com/images/tiny.png Is there any solution to disable access via storage.googleapis.com and use only my domain?
Google Cloud Platform Support Version:
NOTE: This is the reply from Google Cloud Platform Support when contacted via email...
I understand that you have set up a domain name for one of your Cloud Storage buckets and you want to make sure only URLs starting with your domain name have access to this bucket.
I am afraid that this is not possible because of how Cloud Storage permission works.
Making a Cloud Storage bucket publicly readable also gives each of its files a public link. And currently this public link can’t be disabled.
A workaround would be implement a proxy program and running it on a Compute Engine virtual machine. This VM will need a static external IP so that you can map your domain to it. The proxy program will be in charged of returning the requested file from a predefined Cloud Storage bucket while the bucket keeps to be inaccessible to the public.
You may find these documents helpful if you are interested in this workaround:
1. Quick start to set up a Linux VM (1).
2. Python API for accessing Cloud Storage files (2).
3. How to download service account keys to grant a program access to a set of services (3).
4. Pricing calculator for getting a picture on how much a VM may cost (4).
(1) https://cloud.google.com/compute/docs/quickstart-linux
(2) https://pypi.org/project/google-cloud-storage/
(3) https://cloud.google.com/iam/docs/creating-managing-service-account-keys
(4) https://cloud.google.com/products/calculator/
My Version:
It seems the solution to this question is really a simple, just FUSE Google Cloud Storage with VM Instance.
After FUSE private files from GCS can be accessed through VM's IP address. It made Google Cloud Storage Bucket act like a directory.
The detailed documentation about how to setup FUSE in Google Cloud is here.
There is but it requires you to do more work.
Your current solution works because you've made access to the GCS bucket (example.com), public and then you're DNS aliasing from your domain.
An alternative approach would be for you to limit access to the GCS bucket to one (possibly several) accounts and then run a web-server that uses one of the accounts to access your image files. You could then also either permit access to your web-server to anyone or also limit access to it.
More work for you (and possibly cost) but more control.
Trying to get Google Cloud Storage working on my app. I successfully saved an image to a bucket, but when trying to retrieve the image, I receive this error:
GCS Storage (615.3ms) Generated URL for file at key: 9A95rZATRKNpGbMNDbu7RqJx ()
Completed 500 Internal Server Error in 618ms (ActiveRecord: 0.2ms)
Google::Cloud::Storage::SignedUrlUnavailable (Google::Cloud::Storage::SignedUrlUnavailable):
Any idea of what's going on? I can't find an explanation for this error in their documentation.
To provide some explanation here...
Google App Engine (as well as Google Compute Engine, Kubernetes Engine, and Cloud Run) provides "ambient" credentials associated with the VM or instance being run, but only in the form of OAuth tokens. For most API calls, this is sufficient and convenient.
However, there are a small number of exceptions, and Google Cloud Storage is one of them. Recent Storage clients (including the google-cloud-storage gem) may require a full service account key to support certain calls that involve signed URLs. This full key is not provided automatically by App Engine (or other hosting environments). You need to provide one yourself. So as a previous answer indicated, if you're using Cloud Storage, you may not be able to depend on the "ambient" credentials. Instead, you should create a service account, download a service account key, and make it available to your app (for example, via the ActiveStorage configs, or by setting the GOOGLE_APPLICATION_CREDENTIALS environment variable).
I was able to figure this out. I had been following Rail's guide on Active Storage with Google Storage Cloud, and was unclear on how to generate my credentials file.
google:
service: GCS
credentials: <%= Rails.root.join("path/to/keyfile.json") %>
project: ""
bucket: ""
Initially, I thought I didn't need a keyfile due to this sentence in Google's Cloud Storage authentication documentation:
If you're running your application on Google App Engine or Google
Compute Engine, the environment already provides a service account's
authentication information, so no further setup is required.
(I am using Google App Engine)
So I commented out the credentials line and started testing. Strangely, I was able to write to Google Cloud Storage without issue. However, when retrieving the image I would receive the 500 server error Google::Cloud::Storage::SignedUrlUnavailable.
I fixed this by generating my private key and adding it to my rails app.
Another possible solution as of google-cloud-storage gem version 1.27 in August 2020 is documented here. My Google::Auth.get_application_default as in the documentation returned an empty object, but using Google::Cloud::Storage::Credentials.default.client instead worked.
If you get Google::Apis::ClientError: badRequest: Request contains an invalid argument response when signing check that you have dash in the project name in the signing URL (i.e projects/-/serviceAccounts explicit project name in the path is deprecated and no longer valid) and that you have "issuer" string correct, as the full email address identifier of the service account not just the service account name.
If you get Google::Apis::ClientError: forbidden: The caller does not have permission verify the roles your Service Account have:
gcloud projects get-iam-policy <project-name>
--filter="bindings.members:<sa_name>"
--flatten="bindings[].members" --format='table(bindings.role)'
=> ROLE
roles/iam.serviceAccountTokenCreator
roles/storage.admin
serviceAccountTokenCreator is required to call the signBlob service, and you need storage.admin to have ownership of the thing you need to sign. I think these are project global rights, I couldn't get it to work with more fine grained permissions unfortunately (i.e one app is admin for a certain Storage bucket)
I have a django app running in my Google Compute Engine, and it needs to upload video files to my bucket in Google Cloud Storage. When searching for authentication methods, I found this doc. Under Setting the scope of service account access for instances section, it says I need to enable the Cloud Platform access in the settings when creating the VM. I wonder if it is a must and if there's any other way that I can access my cloud storage bucket from my apps in the compute engine. Because creating a new VM and set up the environment is very time-consuming. Any input would be greatly appreciated. Thanks in advance.
As documented on the page you linked to, to authenticate from Google Compute Engine to Google Cloud Storage, you have several options:
Use VM scopes: this must be set before creating the VM, because scopes are immutable once the VM is created. If you want read-only access, you need to add the scope devstorage.read_only (short form) or https://www.googleapis.com/auth/devstorage.read_only (full path). If you want read-write access, you should use the scope devstorage.read_write (short form) or https://www.googleapis.com/auth/devstorage.read_write (full path).
Note: there's also a feature gcloud beta compute instances set-scopes to update GCE VM scopes at runtime.
An alternative to using scopes is to use JSON authentication tokens, such as via Service accounts which can be used by Google API client libraries to connect to Google Cloud Storage.