I am trying to perform a very basic command like:
gcloud compute machine-types list
And I get this error:
ERROR: (gcloud.compute.machine-types.list) There was a problem
refreshing your current auth tokens: invalid_grant: Bad Request Please
run:
It tells me to login using 'gcloud auth login' which opens up the browser.
Is it possible to use a ssh key to skip this authentication process or I have to do this always? ssh keys are for accessing compute instances only?
Just trying to understand what SSH keys are used for and how this web based authorization fits into the picture here.
Generally, you authenticate to gcloud (and GCP services) using credentials from a Google (often Gmail) account. Such accounts use 3-legged (O)Auth and this requires the browser prompt for the human to confirm the scopes etc.
If you haven't, you should confirm the prompt, copy the token provided and paste that back into gcloud so that auth will occur transparently.
This process is different than SSH'ing to Compute Engine instances.
When you run gcloud compute machine-types list, you're authenticating (and being authorized) by Google Cloud Platform to invoke (meta)services.
When you run gcloud compute ssh ..., the command uses ssh to connect you to the (Linux) instance.
NOTE gcloud auth login --no-launch-browser is available too (link). This requires you to separately launch a browser and complete the process but it doesn't launch the browser directly from the command.
If you are trying to automate some sort of service, that runs cloud commands on-demand, without operator/browser involved - your best bet would be to create a Service Account for that task, get the key for that account and activate it, using
gcloud auth activate-service-account --key-file=my-service-account-key-file.json
If this service runs on Google Cloud platform - you don't even need to deal with the key. Just associate the service account with an instance you are running.
https://cloud.google.com/compute/docs/access/create-enable-service-accounts-for-instances
I recently created a VM, but mistakenly gave the default service account Storage: Read Only permissions instead of the intended Read Write under "Identity & API access", so GCS write operations from the VM are now failing.
I realized my mistake, so following the advice in this answer, I stopped the VM, changed the scope to Read Write and started the VM. However, when I SSH in, I'm still getting 403 errors when trying to create buckets.
$ gsutil mb gs://some-random-bucket
Creating gs://some-random-bucket/...
AccessDeniedException: 403 Insufficient OAuth2 scope to perform this operation.
Acceptable scopes: https://www.googleapis.com/auth/cloud-platform
How can I fix this? I'm using the default service account, and don't have the IAM permissions to be able to create new ones.
$ gcloud auth list
Credentialed Accounts
ACTIVE ACCOUNT
* (projectnum)-compute#developer.gserviceaccount.com
I will suggest you to try add the scope "cloud-platform" to the instance by running the gcloud command below
gcloud alpha compute instances set-scopes INSTANCE_NAME [--zone=ZONE]
[--scopes=[SCOPE,…] [--service-account=SERVICE_ACCOUNT
As a scopes put "https://www.googleapis.com/auth/cloud-platform" since it give Full access to all Google Cloud Platform resources.
Here is gcloud documentation
Try creating the Google Cloud Storage bucket with your user account.
Type gcloud auth login and access the link you are provided, once there, copy the code and paste it into the command line.
Then do gsutil mb gs://bucket-name.
The security model has 2 things at play, API Scopes and IAM permissions. Access is determined by the AND of them. So you need an acceptable scope and enough IAM privileges in order to do whatever action.
API Scopes are bound to the credentials. They are represented by a URL like, https://www.googleapis.com/auth/cloud-platform.
IAM permissions are bound to the identity. These are setup in the Cloud Console's IAM & admin > IAM section.
This means you can have 2 VMs with the default service account but both have different levels of access.
For simplicity you generally want to just set the IAM permissions and use the cloud-platform API auth scope.
To check if you have this setup go to the VM in cloud console and you'll see something like:
Cloud API access scopes
Allow full access to all Cloud APIs
When you SSH into the VM by default gcloud will be logged in as the service account on the VM. I'd discourage logging in as yourself otherwise you more or less break gcloud's configuration to read the default service account.
Once you have this setup you should be able to use gsutil properly.
I am trying to create an internal app to upload files to google cloud. I don't want each individual user or this app to log in so I'm using a service account. I login into the service account and everything is ok, but when I try to upload it gives me this error:
ServiceException: 401 Anonymous caller does not have storage.objects.list access to bucket
As you can see I am logged in with a service account and my account and(neither service or personal) works
I had similar problem, and as always, it took me 2 hours but the solution was trivial, if only it was written somewhere... I needed to login (or authorize, what suits you) to the gsutil in addition to being authorized to the gcloud. I thought they are linked or whatever, but nah. After I ran gsutil config and authorized via the provided link (and code that I pasted back to the console), it started working for me.
Note that I was also logged in to gcloud via a service account linked to my project and having the service account .json key saved locally (see gcloud auth activate-service-account --help.
gcloud auth login solved my issue. You need both steps:
gcloud auth login
gcloud auth application-default login
It happened to me because I had an incomplete initialisation while running gcloud init.
I reinitialised the configuration using gcloud init command and it worked fine.
I can only think of a few things that might cause you to see this error:
Maybe you have an alias set up to a standalone installation of gsutil (which doesn't share credentials with gcloud)?Edit: it's also possible you're invoking the wrong gsutil entry point - make sure you're using <path-to-cloud-sdk>/google-cloud-sdk/bin/gsutil, and not <path-to-cloud-sdk>/google-cloud-sdk/platform/gsutil/gsutil. The platform path will not automatically know about your configured gcloud auth options.
Maybe your service account credentials have moved/are invalid now? If your boto file is referring to a keyfile path and the keyfile was moved, this might happen.
Maybe the gcloud boto file (that gcloud created to use with gsutil when you ran gcloud auth login) is gone. You can run gsutil version -l to see if it's shown in your config path. If gcloud's boto file is present, you should see a line similar to this:
config path(s):
/Users/Daniel/.config/gcloud/legacy_credentials/email#domain.tld/.boto
You can run gsutil version -l to get a bit more info and look into the possibilities above. In particular, these attributes from the output will probably be the most helpful: using cloud sdk, pass cloud sdk credentials to gsutil, config path(s), and gsutil path.
Use this command to resolve some issues
gsutil config
Follow the browser to get a code, then set it in your terminal.
I had the same issue, tried to do gsutil config then it recommended me gcloud auth login which opened google in the browser. After i logged in, i could download with gsutil cp -r gs://my_bucket/Directory local_save_path the entire bucket and save it locally.
I faced the same problem. It took me two days to get this thing working.
I am writing about the whole setup. please refer to step 2 for the answer to the question. FYI my OS is windows 10
Step 1:
Firstly, I faced problems installing gcloud and this is what i did.
The script(.\google-cloud-sdk\install.bat) which is supposed to add gcloud to the path was not working due to permission issues.
I had to add the path manually in two places
1) In the system variables, to the "PATH" variable i added the path to the gcloud bin which should look like - C:\Users\774610\google-cloud-sdk\bin - in my case
2) Additionally gcloud needs python so to the "PATHEXT" variable i appended ".PY" at the end.
After Performing these tasks gcloud started working.
Step 2:
Even though gcloud is working, maven is not able to connect to cloud storage and the error was "401 Anonymous caller does not have storage.objects.list access to bucket"
I was pretty sure i did login to my account and selected the correct project. I also tried adding environment variable as shown in this documentation "https://cloud.google.com/docs/authentication/getting-started"
Nothing seemed to be working even though all the credentials were perfectly setup.
while going through the gcloud documentation I came across this command - "gcloud auth application-default login" which was exactly what i needed.
Refer here for difference between gcloud auth login and gcloud auth application default login
In short what this command does is it obtains your credentials via a web flow and stores them in 'the well-known location for Application Default Credentials' and any code/SDK you run will be able to find the credentials automatically
After this, maven was successfully able to connect to google storage and do its stuff.
Hope this helps, thanks
Does your service account actually have the required permission? The role(s) that will give you this permission are roles/storage.objectViewer / roles/storage.objectAdmin / roles/storage.admin.
Please ensure the service account actually have the permissions in your Cloud Console and then it should work.
--- UPDATE ---
Since you have the correct permission in the account, there it's likely the correct account wasn't used in the gsutil command. This can happen if you have multiple installations of your gsutil tool, please ensure your gsutil has the correct path point to a .BOTO file. There's a similar issue reported on the github repo. You can see if the solution there works.
Ultimately, you can use a new machine / vm with a fresh install to test it out to see if it works. You can this easily by going to the Cloud Console and using the Cloud Shell. No real installation needed, should be very simple to test.
This should work and it will basically isolate your issue (to that of multiple installation) on your original machine. After that, you basically just have to do a clean install to fix it.
If you installed gsutil using python (without gcloud SDK), it may help to run gsutil config and complete steps of initialisation.
Thank you for all the replies.
I would like to share my own experience.
I had to login under the user which is defined when installing Gitlab Runner.
By default, the user indicated in the installation doc is : "gitlab-runner".
So, first, I added a password on this user:
passwd gitlab-runner
then :
su - gitlab-runner
gcloud auth login
gcloud auth application-default login
The issue is solved.
Maybe there is a better way, by directly putting the Google auth files under /home/gitlab-runner
I faced same issue. I used
gcloud auth login
and follow the link
If you are using a service account you need first to authorize it, otherwise gsutil won't have the permission to read/write
gcloud auth activate-service-account --key-file=service_account_file.json
Personally, I had an account with proper permissions registered but I got that error as well despite verifying that my account was running using "sudo gcloud init"
What solved it for me was navigating to the ~/.gutil directory and writing the following
sudo chown jovyan:jovyan *
which let my JupyterLab terminal run, not from root, but from default jovyan.
After that it used my account, not Anonymous caller
Here is another way to edit roles:
gsutil iam ch allUsers:objectViewer gs://tf-learn-objectdetection
Fore more documentation:
gsutil iam help
Use gcloud auth login
Goto mention link
Copy Verification code
Paste Verification code
In my case, even after using gsutils solutions discussed in other answers, I got the error. After checking other google search results, I found out that the reason was that I was authenticating with "my user" while running the gsutils as the root.
Thanks to the answer in the gsutils page in github: https://github.com/GoogleCloudPlatform/gsutil/issues/457
Let me expain what helped me step by step:
First my requirement is to enable CORS, but faced the asked issue, So I followed the below steps:
On Browser side:
Open google cloud console on your browser.
Open Cloud shell editor.
Type gcloud auth login.
Now it will show an command with an url.
Copy that command Don't close browser.
On PC GCloud software side:
Download GCloud Sdk Installer.exe
Open GCLoud in your pc It will ask you to sign In via browser
Signin with correct email id
Select your project from the shown list
Paste the previously copied command
Again it will ask you to signIn
Select the proper account to sign in
Now the GCloud cmd will show you another command with url as output
Copy the output Open your browser, then paste it.
Done! It will show like You are now logged in as xyz#gmail.com
Now I'm able to set CORS without any exception.
Hope these steps will be helpfull for someone who is new to the issue.
Looks like account information is not stored with gsutil
Step 1:
gsutil config
Step 2:
copy url in browser
Step3:
select account and grant permission
Step 4:
Copy key and share it in gsutil promt "step1 will be asking for this key to proceed"
Step 5:
Run command whose access was denied
Thank you Petr Krýže!!! you saved my day...
I had a gcloud user authenticated and was running gsutils fine from the command line (Windows 8.1). But I needed to access gsutils from a python application so I followed the instructions here:
https://cloud.google.com/storage/docs/xml-api/gspythonlibrary#credentials
I got as far as creating a .boto file, but now not only does the my python code fail (boto.exception.NoAuthHandlerFound: No handler was ready to authenticate.). But I can't run bsutils from the command line any more. I get this error:
C:\>gsutil ls
You are attempting to access protected data with no configured
credentials. Please visit https://cloud.google.com/console#/project
and sign up for an account, and then run the "gcloud auth login"
command to configure gsutil to use these credentials.
I have run gcloud auth and it appears to work, I can query my users:
C:\>gcloud auth list
Credentialed Accounts:
- XXXserviceuser#XXXXX.iam.gserviceaccount.com ACTIVE
- myname#company.name
To set the active account, run:
$ gcloud config set account `ACCOUNT`
I have tried both with the account associated with my email active, and the new serveruser account (created following instructions above). Same "protected data with no configured credentials." error. I tried removing the .boto file, and adding the secret CLIENT_ID and CLIENT_SECRET to my .boto file.
Anyone any ideas what the issue could be?
So I think the latest documentation/examples showing how to use (and authenticate) Google Cloud storage via python is in this repo:
https://github.com/GoogleCloudPlatform/python-docs-samples/tree/master/storage/api
That just works for me without messing around with keys and service users.
Would be nice if there was a comment somewhere in the old gspythonlibrary docs pointing this out.
I launched an instance with service level accounts enabled. For example it has storage-rw set. I verfied that the instance has those. Now whenever i run gsutil ls gs://my_bucket from within the instance I get the error: Failure: unauthorized_client.
gcloud auth list returns
Credentialed accounts:
- xxxx#developer.gserviceaccount.com (active)
I need to use gcloud sdk from an instance because i need more components other than the gcutil and gsutil.
So my question is how can I authorize gcloud to use the xxxx#developer.gserviceaccount.com account and thus the permissions only specified on the instance and not my personal user account which has full permissions to everything?
The gcloud CLI definitely handles Google Compute Engine service accounts. If you see it as "(active)" when you do $ gcloud auth list, that should be sufficient.
Two things can be going wrong here:
You are using the wrong gsutil.
When you install the Google Cloud SDK, it will create google-cloud-sdk/bin/gsutil, and THAT is the one you want to run. Do $ which gsutil to double check. If you're running google-cloud-sdk/platform/gsutil/gsutil, that's the wrong one, and it won't know about anything that gcloud can tell it.
The account doesn't have permissions to access the bucket you're trying to inspect. You'll have to ask the owner of the bucket to add it to the project that owns that bucket.
Source: Engineer for the Google Cloud SDK
See "Authenticating to Google Compute Engine" section in this doc: https://developers.google.com/compute/docs/gcutil/