GSUtil service account failure - google-cloud-storage

Question

I'm running debian on GCE and I've configured gsutil with gsutil config -e.
When I try to do gsutil -D ls I get the following error:
gsutil version 4.3
checksum f6a4b1f8400e2d1d145c9ec5e9ea8d90 (OK)
boto version 2.29.1
python version 2.7.3 (default, Mar 13 2014, 11:03:55)
[GCC 4.7.2]
config path: /etc/boto.cfg
gsutil path: /usr/local/share/google/gsutil/gsutil
compiled crcmod: False
installed via package manager: False
editable install: False
Command being run: /usr/local/bin/gsutil -D ls
config_file_list: ['/etc/boto.cfg', '/var/lib/postgresql/.boto']
config: [('debug', '0'), ('working_dir', '/mnt/pyami'), ('https_validate_certificates', 'True'), ('debug', '0'), ('working_dir', '/mnt/pyami'), ('default_project_id', 'e-vard'), ('default_api_version', '2'), ('content_language', 'en')]
Calling method storage.buckets.list with StorageBucketsListRequest: <StorageBucketsListRequest
maxResults: 100
project: 'e-vard'
projection: ProjectionValueValuesEnum(full, 0)>
Making http GET to https://www.googleapis.com/storage/v1/b?projection=full&prettyPrint=True&fields=nextPageToken%2Citems%2Fid&maxResults=100&project=e-vard&alt=json
Headers: {'accept': 'application/json',
'accept-encoding': 'gzip, deflate',
'content-length': '0',
'user-agent': 'apitools gsutil/4.3 (linux2)'}
Body: (none)
Attempting refresh to obtain initial access_token
{'iss': '1234567890#developer.gserviceaccount.com', 'scope': 'https://www.googleapis.com/auth/devstorage.full_control', 'aud': 'https://accounts.google.com/o/oauth2/token', 'exp': 1405436827L, 'iat': 1405433227L}
Refreshing access_token
connect: (accounts.google.com, 443)
send: 'POST /o/oauth2/token HTTP/1.1\r\nHost: accounts.google.com\r\nContent-Length: 726\r\ncontent-type: application/x-www-form-urlencoded\r\naccept-encoding: gzip, deflate\r\nuser-agent: Python-httplib2/0.7.7 (gzip)\r\n\r\ngrant_type=urn%3Aietf%3Aparams%3Aoauth%3Agrant-type%3Ajwt-bearer&assertion=ASSERTIONREMOVED'
reply: 'HTTP/1.1 400 Bad Request\r\n'
header: Content-Type: application/json
header: Cache-Control: no-cache, no-store, max-age=0, must-revalidate
header: Pragma: no-cache
header: Expires: Fri, 01 Jan 1990 00:00:00 GMT
header: Date: Tue, 15 Jul 2014 14:07:07 GMT
header: Content-Encoding: gzip
header: X-Content-Type-Options: nosniff
header: X-Frame-Options: SAMEORIGIN
header: X-XSS-Protection: 1; mode=block
header: Server: GSE
header: Alternate-Protocol: 443:quic
header: Transfer-Encoding: chunked
Failed to retrieve access token: {
"error" : "invalid_grant"
}
DEBUG: Exception stack trace:
Traceback (most recent call last):
File "/usr/local/share/google/gsutil/gslib/__main__.py", line 419, in _RunNamedCommandAndHandleExceptions
debug_level, parallel_operations)
File "/usr/local/share/google/gsutil/gslib/command_runner.py", line 194, in RunNamedCommand
return_code = command_inst.RunCommand()
File "/usr/local/share/google/gsutil/gslib/commands/ls.py", line 378, in RunCommand
bucket_fields=bucket_fields):
File "/usr/local/share/google/gsutil/gslib/wildcard_iterator.py", line 425, in IterBuckets
for blr in self._ExpandBucketWildcards(bucket_fields=bucket_fields):
File "/usr/local/share/google/gsutil/gslib/wildcard_iterator.py", line 339, in _ExpandBucketWildcards
provider=self.wildcard_url.scheme):
File "/usr/local/share/google/gsutil/gslib/gcs_json_api.py", line 380, in ListBuckets
global_params=global_params)
File "/usr/local/share/google/gsutil/gslib/third_party/storage_apitools/storage_v1_client.py", line 351, in List
config, request, global_params=global_params)
File "/usr/local/share/google/gsutil/gslib/third_party/storage_apitools/base_api.py", line 587, in _RunMethod
http, http_request, retries=self.__client.num_retries)
File "/usr/local/share/google/gsutil/gslib/third_party/storage_apitools/http_wrapper.py", line 152, in MakeRequest
redirections=redirections, connection_type=connection_type)
File "/usr/local/share/google/gsutil/third_party/google-api-python-client/oauth2client/util.py", line 132, in positional_wrapper
return wrapped(*args, **kwargs)
File "/usr/local/share/google/gsutil/third_party/google-api-python-client/oauth2client/client.py", line 475, in new_request
self._refresh(request_orig)
File "/usr/local/share/google/gsutil/third_party/google-api-python-client/oauth2client/client.py", line 663, in _refresh
self._do_refresh_request(http_request)
File "/usr/local/share/google/gsutil/third_party/google-api-python-client/oauth2client/client.py", line 710, in _do_refresh_request
raise AccessTokenRefreshError(error_msg)
AccessTokenRefreshError: invalid_grant
What have I done wrong?

Answer 1

Because gsutil makes HTTP calls to interact with Google Cloud Storage, it can't operate on a GCE instance with no external IP.

Answer 2

To use a service account with a GCE internal VM, you should check the Enable Compute Engine service account in the UI and give Storage a scope of your choosing. This will expose GCE service credentials in your instance. However, you can't run it with tools that operate over HTTP because you have no external IP to communicate from.
That means if you want to use Google Cloud Storage with a GCE VM, you need an external IP.