I cannot figure out why I'm experiencing issues with buckets setup to serve content under custom domain. I have few that work just fine, few others that have been working before and do not now, and few I'm struggling to setup from scratch and cannot make them to work.
Here's what I do:
I setup an app on Google App Engine.
I setup a bucket named gcs.domain.com which prompts me to verify domain first with Google
Webmasters Tools and domain TXT record - after that I'm able to
create a bucket.
I setup CNAME to be able to access bucket under my
custom domain.
I add Service Account Name as an owner of the bucket, so my app can write and read from it.
When I run my app I get PHP Fatal error Invalid Google Cloud Storage path: gs://gcs.domain.com CloudStorageStreamWrapper.php on line 327
I cannot pin point why. This scenario works on some of the domains and apps and does not on others. I can setup a bucket with a regular name that works just fine. The problem seems to be only with custom domain buckets.
Related
message: '{"ResourceType":"storage.v1.bucket","ResourceErrorCode":"403","ResourceErrorMessage":{"code":403,"errors":[{"domain":"global","message":"The
bucket you tried to create requires domain ownership verification.","reason":"forbidden"}],"message":"The
bucket you tried to create requires domain ownership verification.","statusMessage":"Forbidden","requestPath":"https://www.googleapis.com/storage/v1/b","httpMethod":"POST","suggestion":"Consider
granting permissions to 1058303916595#cloudservices.gserviceaccount.com"}}'
Above is the message when I tried to create storagebuckets via Deployment Manager. I already tried adding 1058303916595#cloudservices.gserviceaccount.com to the Verified owners of that domain on the Google Search Console but still no luck
To create a Storage Bucket with domain name, your account should be a verified owner of the domain, not the cloudservices.gserviceaccount.com. Additional information can be found in Domain-Named Bucket Verification documentation.
You have to login to your domain name provider with the same account that you use to login to the Google Cloud Console. Then request a new domain name, go to Google Cloud Console App Engine page. Go to Settings and then click on Custom domains and then Add a custom domain. After adding the domain name there and verifying it, you will be able to create a bucket name using that domain as name. But make sure to do so with the same account that the domain was verified with.
I am newbie at cloud servers and I've opened a google cloud storage to host image files. I've verified my domain and configured it, to view images via my domain. The problem is, same file is both accessible via my domain example.com/images/tiny.png and also via storage.googleapis.com/example.com/images/tiny.png Is there any solution to disable access via storage.googleapis.com and use only my domain?
Google Cloud Platform Support Version:
NOTE: This is the reply from Google Cloud Platform Support when contacted via email...
I understand that you have set up a domain name for one of your Cloud Storage buckets and you want to make sure only URLs starting with your domain name have access to this bucket.
I am afraid that this is not possible because of how Cloud Storage permission works.
Making a Cloud Storage bucket publicly readable also gives each of its files a public link. And currently this public link can’t be disabled.
A workaround would be implement a proxy program and running it on a Compute Engine virtual machine. This VM will need a static external IP so that you can map your domain to it. The proxy program will be in charged of returning the requested file from a predefined Cloud Storage bucket while the bucket keeps to be inaccessible to the public.
You may find these documents helpful if you are interested in this workaround:
1. Quick start to set up a Linux VM (1).
2. Python API for accessing Cloud Storage files (2).
3. How to download service account keys to grant a program access to a set of services (3).
4. Pricing calculator for getting a picture on how much a VM may cost (4).
(1) https://cloud.google.com/compute/docs/quickstart-linux
(2) https://pypi.org/project/google-cloud-storage/
(3) https://cloud.google.com/iam/docs/creating-managing-service-account-keys
(4) https://cloud.google.com/products/calculator/
My Version:
It seems the solution to this question is really a simple, just FUSE Google Cloud Storage with VM Instance.
After FUSE private files from GCS can be accessed through VM's IP address. It made Google Cloud Storage Bucket act like a directory.
The detailed documentation about how to setup FUSE in Google Cloud is here.
There is but it requires you to do more work.
Your current solution works because you've made access to the GCS bucket (example.com), public and then you're DNS aliasing from your domain.
An alternative approach would be for you to limit access to the GCS bucket to one (possibly several) accounts and then run a web-server that uses one of the accounts to access your image files. You could then also either permit access to your web-server to anyone or also limit access to it.
More work for you (and possibly cost) but more control.
Trying to get Google Cloud Storage working on my app. I successfully saved an image to a bucket, but when trying to retrieve the image, I receive this error:
GCS Storage (615.3ms) Generated URL for file at key: 9A95rZATRKNpGbMNDbu7RqJx ()
Completed 500 Internal Server Error in 618ms (ActiveRecord: 0.2ms)
Google::Cloud::Storage::SignedUrlUnavailable (Google::Cloud::Storage::SignedUrlUnavailable):
Any idea of what's going on? I can't find an explanation for this error in their documentation.
To provide some explanation here...
Google App Engine (as well as Google Compute Engine, Kubernetes Engine, and Cloud Run) provides "ambient" credentials associated with the VM or instance being run, but only in the form of OAuth tokens. For most API calls, this is sufficient and convenient.
However, there are a small number of exceptions, and Google Cloud Storage is one of them. Recent Storage clients (including the google-cloud-storage gem) may require a full service account key to support certain calls that involve signed URLs. This full key is not provided automatically by App Engine (or other hosting environments). You need to provide one yourself. So as a previous answer indicated, if you're using Cloud Storage, you may not be able to depend on the "ambient" credentials. Instead, you should create a service account, download a service account key, and make it available to your app (for example, via the ActiveStorage configs, or by setting the GOOGLE_APPLICATION_CREDENTIALS environment variable).
I was able to figure this out. I had been following Rail's guide on Active Storage with Google Storage Cloud, and was unclear on how to generate my credentials file.
google:
service: GCS
credentials: <%= Rails.root.join("path/to/keyfile.json") %>
project: ""
bucket: ""
Initially, I thought I didn't need a keyfile due to this sentence in Google's Cloud Storage authentication documentation:
If you're running your application on Google App Engine or Google
Compute Engine, the environment already provides a service account's
authentication information, so no further setup is required.
(I am using Google App Engine)
So I commented out the credentials line and started testing. Strangely, I was able to write to Google Cloud Storage without issue. However, when retrieving the image I would receive the 500 server error Google::Cloud::Storage::SignedUrlUnavailable.
I fixed this by generating my private key and adding it to my rails app.
Another possible solution as of google-cloud-storage gem version 1.27 in August 2020 is documented here. My Google::Auth.get_application_default as in the documentation returned an empty object, but using Google::Cloud::Storage::Credentials.default.client instead worked.
If you get Google::Apis::ClientError: badRequest: Request contains an invalid argument response when signing check that you have dash in the project name in the signing URL (i.e projects/-/serviceAccounts explicit project name in the path is deprecated and no longer valid) and that you have "issuer" string correct, as the full email address identifier of the service account not just the service account name.
If you get Google::Apis::ClientError: forbidden: The caller does not have permission verify the roles your Service Account have:
gcloud projects get-iam-policy <project-name>
--filter="bindings.members:<sa_name>"
--flatten="bindings[].members" --format='table(bindings.role)'
=> ROLE
roles/iam.serviceAccountTokenCreator
roles/storage.admin
serviceAccountTokenCreator is required to call the signBlob service, and you need storage.admin to have ownership of the thing you need to sign. I think these are project global rights, I couldn't get it to work with more fine grained permissions unfortunately (i.e one app is admin for a certain Storage bucket)
So, I've spent about 5 days searching for an answer here and on Google Docs, including having one of their support people help me. My domain still doesn't resolve to the website.
For the record, the website works if I use the ugly url (http://storage.googleapis.com/7thgradeplay.org/index.html).
I have transferred the domain to google domains, days ago.
I have verified the domain with Google Search Console. Billing is enabled and accruing. Public_html is set on all files and folders.
I am using Google Domains name servers. I am not using Google Cloud DNS.
Per Google support:
Synthetic Records: 302 redirect #.7thegradeplay.org to www.7thgradeplay.org
Custom Resource Records: www CNAME 7thegradeplay.org
Does this matter? storage bucket name is 7thegradeplay.org.
I think that's about all the config I've done.
All of these changes were done on Friday (3 days ago), and I still get a 404 error when I try to go to the website. I have followed the instructions and tried to troubleshoot with these pages:
https://cloud.google.com/dns/troubleshooting
https://cloud.google.com/storage/docs/hosting-static-website
The only thing I varied was the name of the bucket in storage. I used a bucket name without the leading 'www.' Please don't tell me this is all it takes to break it.
All help is appreciated.
P.S. I added a bucket called www.7thegradeplay.org with all the same files. Waited 15 minutes. still 404 error.
P.P.S. I found an answer, but it didn't work: Connect Google domain to Google Cloud Bucket.
I will retry step #5 in the PPS above tomorrow, after the PS change has had time to 'stew'.
Again, any help is appreciated.
Your bucket name needs to match the URL exactly, so if you're visiting www.7thgradeplay.org, the bucket also needs to be named www.7thgradeplay.org.
Similarly, the DNS record for "www.7thgradeplay.org" must be a CNAME to "c.storage.googleapis.com.".
Checking DNS, I see a CNAME from "wwww.7thgradeplay.org" to "7thgradeplay.org". It needs to be "c.storage.googleapis.com." If you've already set that, you may need to wait a while for it to percolate. DNS can be slow to update.
Follow below steps to Host Static Website on Google Cloud Storage.
Creating a CNAME record in DNS :
Go to your respective Domain Service provider account and find DNS
settings and Create a CNAME record that points to
c.storage.googleapis.com.
NAME TYPE DATA
www.example.com CNAME c.storage.googleapis.com
After adding “CNAME” record it will take some time to propagate this
records.
Creating a Cloud Storage Bucket :
Go to Google cloud Console and select Storage from side menu, And
click on Create bucket.
Now create a bucket whose name must matches the CNAME record that
you have created for your domain in DNS settings.
For example, If you added a CNAME record pointing www.example.com
to c.storage.googleapis.com, then create a bucket with the name
www.example.com.
Uploading files to Cloud Storage Bucket :
Now In the list of buckets, click on the name of the bucket that you
have created.
Now create index.html file in your local system for your website
home page.
Now click on Upload files button and select index.html file that
you have created.
Browse the static website :
Now browse your website with your domain name in your web browser.
For example, If your domain name is www.example.com then browse
your website by going to http://www.example.com in your web
browser.
Now, You have successfully hosted your website on Google Cloud
Storage.
I've done an extensive research but I can't find a solution.
How can I enable Service Account Authentication for a project that is linked with Google's private owned Bucket for Double Click Manager data? (more info on the current setup of this project here https://support.google.com/dcm/partner/answer/2941575?hl=en&ref_topic=6107456&rd=1).
Separate user authentication works with gsutil(navigating to browser->get token->paste back in your cmd->issue commands) but when it comes to configuring a service account I keep getting
AccessDeniedException: 403 Forbidden
What am I missing? Since the Google documentation says that this specific bucket can't be listed under Cloud Storage for that project, then the project and the service account should be linked to that bucket by default so I can't see the issue here.
During set-up you should have created a Google Group to control access to your bucket. You should add the service account email address to that group, and it will then be able to access the bucket.