Unable to authenticate using google cloud service account key created by python API - google-authentication

Question

The sample below demonstrates failure to authenticate to google service account using the key created just the few lines above using python api.
I was not able to find any document on how these, programmatic keys, can be used.
The keys created by clicking thru console UI are working just fine.
However, for our use case, we need to create the keys using programmatic ways.
There is unanswered issue at github as well: https://github.com/googleapis/google-cloud-python/issues/7824
logger.info("Created new service account: {}".format(ret))
logger.info("Getting the new service account key")
request=iam.projects().serviceAccounts().keys().create(name=ret['name'],
body={'privateKeyType':'TYPE_GOOGLE_CREDENTIALS_FILE'})
key=request.execute()
>>>print json.dumps(key, indent=4) #just to verify what we got
{
"keyOrigin": "GOOGLE_PROVIDED",
"name": "goodandvalidname",
"validBeforeTime": "2029-06-28T15:09:59Z",
"privateKeyData": "datadata",
"privateKeyType": "TYPE_GOOGLE_CREDENTIALS_FILE",
"keyAlgorithm": "KEY_ALG_RSA_2048",
"validAfterTime": "2019-07-01T15:09:59Z"
}
>>> credentials = google.oauth2.service_account.Credentials.from_service_account_info(key)
Traceback (most recent call last):
File "/home/user/.p2/pool/plugins/org.python.pydev.core_7.2.1.201904261721/pysrc/_pydevd_bundle/pydevd_exec.py", line 3, in Exec
exec exp in global_vars, local_vars
File "<console>", line 1, in <module>
File "/home/user/.local/lib/python2.7/site-packages/google/oauth2/service_account.py", line 193, in from_service_account_info
info, require=['client_email', 'token_uri'])
File "/home/user/.local/lib/python2.7/site-packages/google/auth/_service_account_info.py", line 51, in from_dict
'fields {}.'.format(', '.join(missing)))
ValueError: Service account info was not in the expected format, missing fields token_uri, client_email.
Any help appreciated.

Answer 1

Answering my own issue and probably helping others...
The 'key' we get from python APIs is NOT the 'json key' as obtained from gcloud. The dict we get from iam.projects().serviceAccounts().keys().create() contains the field privateKeyData which itself contains ENTIRE 'json key' one needs to authenticate to google cloud.
The data in this field is base64 encoded and needs decoding, and subsequently dumping to json. Below is the snippet from functional code, demonstrating the credentials are loaded back from such key:
request=iam.projects().serviceAccounts().keys().create(name=ret['name'],
body={'privateKeyType':'TYPE_GOOGLE_CREDENTIALS_FILE'})
key=request.execute()
key=base64.decodestring(key['privateKeyData'])
key=json.loads(key)
credentials = google.oauth2.service_account.Credentials.from_service_account_info(key)
I figured this out by stepping thru gcloud service account key creating, line by line, using python debugger. Hope this helps others.