I am trying to create an environment in Google Cloud Composer. Link here
When creating the environment from scratch and selecting all the default fields, the following error appears:
CREATE operation on this environment failed 22 hours ago with the following error message:
CREATE operation failed. Composer Agent failed with: Cloud Storage Assertions Failed: Unable to write to GCS bucket.
GCS bucket write check failed.
I then created a google cloud storage bucket within the same project to see if that would help and the same error still appears.
Has anyone been able successfully create a Google Cloud Composer environment and if so please provide guidance on why this error message continues to appear?
Update: Need to update permissions to allow access it seems like. Here is a screenshot of my permissions page but not editable.
It seems like you haven't given the required IAM policies to the service account. I would advise you to read more about the IAM policies on Google Cloud here
When it comes to the permissions of the bucket, there are permissions like the Storage Object Admin that might fit your needs.
Related
I'm getting "internal error occurred while creating a Google Cloud Storage bucket" errors when I try to publish an App Engine App from Visual Studio 19. (It also fails with the exact same error when trying to publish from shell).
From what I can find here and elsewhere, theres a "quick fix" to this, using
gcloud config set app/use_deprecated_preparation True
but that returns
Section [app] has no property [use_deprecated_preparation]
I'm pretty confident this has to do with role settings.
I've set permissions in IAM; App Engine default service account has the roles Editor, Service Account Token Creator and Storage Object Creator. I personally have the role of Owner.
Can someone point me in the direction of what I'm missing in roles?
The installation of the NodeRED bluemix/IBM Cloud starter application fails with an IAM error message complaining about insufficient rights:
FAILED
Server error, status code: 502, error code: 10001, message: Service broker error: You do not have the required permission to create an instance. You must be assigned the IAM Editor role or higher. Contact the account owner to update your access.
Does anybody know how to fix this issue?
Looks like you don’t have proper IAM access permission. If you are the owner of the account, you can set the required permissions following the steps in this link .If you are not the owner ask the account owner for the permissions.
For best practices, refer this solution tutorial
The issue was actually related to the fact that the bluemix starter application tries to create a lite plan instance of cloudant. In my case, that was not possible because there already was such an instance and you are allowed only one per CF organization.
The solution was to patch the pipeline.yml to create a standard plan instance:
cf create-service cloudantNoSQLDB Standard "${CLOUDANT_NAME}"
Trying to get Google Cloud Storage working on my app. I successfully saved an image to a bucket, but when trying to retrieve the image, I receive this error:
GCS Storage (615.3ms) Generated URL for file at key: 9A95rZATRKNpGbMNDbu7RqJx ()
Completed 500 Internal Server Error in 618ms (ActiveRecord: 0.2ms)
Google::Cloud::Storage::SignedUrlUnavailable (Google::Cloud::Storage::SignedUrlUnavailable):
Any idea of what's going on? I can't find an explanation for this error in their documentation.
To provide some explanation here...
Google App Engine (as well as Google Compute Engine, Kubernetes Engine, and Cloud Run) provides "ambient" credentials associated with the VM or instance being run, but only in the form of OAuth tokens. For most API calls, this is sufficient and convenient.
However, there are a small number of exceptions, and Google Cloud Storage is one of them. Recent Storage clients (including the google-cloud-storage gem) may require a full service account key to support certain calls that involve signed URLs. This full key is not provided automatically by App Engine (or other hosting environments). You need to provide one yourself. So as a previous answer indicated, if you're using Cloud Storage, you may not be able to depend on the "ambient" credentials. Instead, you should create a service account, download a service account key, and make it available to your app (for example, via the ActiveStorage configs, or by setting the GOOGLE_APPLICATION_CREDENTIALS environment variable).
I was able to figure this out. I had been following Rail's guide on Active Storage with Google Storage Cloud, and was unclear on how to generate my credentials file.
google:
service: GCS
credentials: <%= Rails.root.join("path/to/keyfile.json") %>
project: ""
bucket: ""
Initially, I thought I didn't need a keyfile due to this sentence in Google's Cloud Storage authentication documentation:
If you're running your application on Google App Engine or Google
Compute Engine, the environment already provides a service account's
authentication information, so no further setup is required.
(I am using Google App Engine)
So I commented out the credentials line and started testing. Strangely, I was able to write to Google Cloud Storage without issue. However, when retrieving the image I would receive the 500 server error Google::Cloud::Storage::SignedUrlUnavailable.
I fixed this by generating my private key and adding it to my rails app.
Another possible solution as of google-cloud-storage gem version 1.27 in August 2020 is documented here. My Google::Auth.get_application_default as in the documentation returned an empty object, but using Google::Cloud::Storage::Credentials.default.client instead worked.
If you get Google::Apis::ClientError: badRequest: Request contains an invalid argument response when signing check that you have dash in the project name in the signing URL (i.e projects/-/serviceAccounts explicit project name in the path is deprecated and no longer valid) and that you have "issuer" string correct, as the full email address identifier of the service account not just the service account name.
If you get Google::Apis::ClientError: forbidden: The caller does not have permission verify the roles your Service Account have:
gcloud projects get-iam-policy <project-name>
--filter="bindings.members:<sa_name>"
--flatten="bindings[].members" --format='table(bindings.role)'
=> ROLE
roles/iam.serviceAccountTokenCreator
roles/storage.admin
serviceAccountTokenCreator is required to call the signBlob service, and you need storage.admin to have ownership of the thing you need to sign. I think these are project global rights, I couldn't get it to work with more fine grained permissions unfortunately (i.e one app is admin for a certain Storage bucket)
I have some data in Google Cloud SQL, which I am trying to transfer to Google Cloud Storage using Cloud SQL API(beta) (export) function. For this, I have installed jupyter notebook on google compute engine and I am running the python code (to connect to cloud sql API and export the date to cloud storage in a csv file) on that.
The python code does not throw any error and also there is no error body in API response. However, the API's response says "u'status': u'PENDING'" and I see the following error (in logs of Cloud SQL Dashboard) :
error uploading CSV file to GCS: gs://[BUCKET_NAME]/[FILE_NAME].csv: Access denied for account [SERVICE_ACCOUNT_NAME]#speckle-umbrella-11.iam.gserviceaccount.com (permission issue?)
How do I ensure I have all the relevant access for the account ([SERVICE_ACCOUNT_NAME]#speckle-umbrella-11.iam.gserviceaccount.com) ? I am unable to locate this account or give myself access to this .
By default Google Compute Engine instances have read only scope for Google Cloud Storage.
You should confirm that the Google Compute Engine instance scope is set to read/write.
You need to grant the Cloud SQL instance service account access to your bucket. Visit the export documentation and click the GCLOUD (2ND GEN) tab which will show you the commands you need to use.
The command to grant the service account access to the bucket is
gsutil acl ch -u [SERVICE_ACCOUNT_ADDRESS]:W gs://[BUCKET_NAME]
I've done an extensive research but I can't find a solution.
How can I enable Service Account Authentication for a project that is linked with Google's private owned Bucket for Double Click Manager data? (more info on the current setup of this project here https://support.google.com/dcm/partner/answer/2941575?hl=en&ref_topic=6107456&rd=1).
Separate user authentication works with gsutil(navigating to browser->get token->paste back in your cmd->issue commands) but when it comes to configuring a service account I keep getting
AccessDeniedException: 403 Forbidden
What am I missing? Since the Google documentation says that this specific bucket can't be listed under Cloud Storage for that project, then the project and the service account should be linked to that bucket by default so I can't see the issue here.
During set-up you should have created a Google Group to control access to your bucket. You should add the service account email address to that group, and it will then be able to access the bucket.