Google Cloud authorization keeps failing with Python 3 - Type is None, expected one of ('authorized_user', 'service_account') - google-cloud-storage

I am trying to download a file for the first time from Google Cloud Storage.
I set the path to the googstruct.json service account key file that I downloaded from https://cloud.google.com/storage/docs/reference/libraries#client-libraries-usage-python
Do need to set the authorization to Google Cloud outside the code somehow? Or is there a better "How to use Google Cloud Storage" then the one on the google site?
It seems like I am passing the wrong type to the storage_client = storage.Client()
the exception string is below.
Exception has occurred: google.auth.exceptions.DefaultCredentialsError
The file C:\Users\Cary\Documents\Programming\Python\QGIS\GoogleCloud\googstruct.json does not have a valid type.
Type is None, expected one of ('authorized_user', 'service_account').
MY PYTHON 3.7 CODE
from google.cloud import storage
import os
os.environ["GOOGLE_APPLICATION_CREDENTIALS"]="C:\\GoogleCloud\\googstruct.json"
# Instantiates a client
storage_client = storage.Client()
bucket_name = 'structure_ssi'
destination_file_name = "C:\\Users\\18809_PIPEM.shp"
source_blob_name = '18809_PIPEM.shp'
download_blob(bucket_name, source_blob_name, destination_file_name)
def download_blob(bucket_name, source_blob_name, destination_file_name):
"""Downloads a blob from the bucket."""
storage_client = storage.Client()
bucket = storage_client.get_bucket(bucket_name)
blob = bucket.blob(source_blob_name)
blob.download_to_filename(destination_file_name)
print('Blob {} downloaded to {}.'.format(
source_blob_name,
destination_file_name
)
)
I did look at this but I cannot tell if this is my issue. I have tried both.
('Unexpected credentials type', None, 'Expected', 'service_account') with oauth2client (Python)

This error means that the Json Service Account Credentials that you are trying to use C:\\GoogleCloud\\googstruct.json are corrupt or the wrong type.
The first (or second) line in the file googstruct.json should be "type": "service_account".
Another few items to improve your code:
You do not need to use \\, just use / to make your code easier
and cleaner to read.
Load your credentials directly and do not modify environment
variables:
storage_client = storage.Client.from_service_account_json('C:/GoogleCloud/googstruct.json')
Wrap API calls in try / except. Stack traces do not impress customers. It is better to have clear, simple, easy to read error messages.

Related

How to create a bucket using the python SDK?

I'm trying to create a bucket in cloud object storage using python. I have followed the instructions in the API docs.
This is the code I'm using
COS_ENDPOINT = "https://control.cloud-object-storage.cloud.ibm.com/v2/endpoints"
# Create client
cos = ibm_boto3.client("s3",
ibm_api_key_id=COS_API_KEY_ID,
ibm_service_instance_id=COS_INSTANCE_CRN,
config=Config(signature_version="oauth"),
endpoint_url=COS_ENDPOINT
)
s3 = ibm_boto3.resource('s3')
def create_bucket(bucket_name):
print("Creating new bucket: {0}".format(bucket_name))
s3.Bucket(bucket_name).create()
return
bucket_name = 'test_bucket_442332'
create_bucket(bucket_name)
I'm getting this error - I tried setting CreateBucketConfiguration={"LocationConstraint":"us-south"}, but it doesnt seem to work
"ClientError: An error occurred (IllegalLocationConstraintException) when calling the CreateBucket operation: The unspecified location constraint is incompatible for the region specific endpoint this request was sent to."
Resolved by going to https://cloud.ibm.com/docs/cloud-object-storage?topic=cloud-object-storage-endpoints#endpoints
And choosing the endpoint specific to the region I need. The "Endpoint" provided with the credentials, is not the actual endpoint.

No error / exception raised in Google Cloud Storage call via PHP client returns null

I'm trying to create a bucket in Google Cloud Storage using PHP client library but bucket is not being created (FYI: no gcs function working at all) and even its not returning an error code or exception so I can debug the issue.
I have gRPC and Protobuf installed
Using compute engine instance with full permissions granted
Enabled the billing for project and also Storage API is Enabled
If need anything else ask me in the comment.
Any hint / clue much appreciated.
Here's the code I'm using:
require 'vendor/autoload.php';
# Imports the Google Cloud client library
use Google\Cloud\Core\ServiceBuilder;
use Google\Cloud\Storage\StorageClient;
function create_bucket($bucketName, $options = [])
{
$config = [
'projectId' => 'abc-def-agen-1553542096432'
];
$storage = new StorageClient($config);
//var_dump($storage);
$bucket = $storage->createBucket($bucketName, $options);
//var_dump($bucket); //this returns nothing
printf('Bucket created: %s' . PHP_EOL, $bucket->name()); // this prints nothing
}
create_bucket("kjdsnuiew345asd");

Unable to authenticate using google cloud service account key created by python API

The sample below demonstrates failure to authenticate to google service account using the key created just the few lines above using python api.
I was not able to find any document on how these, programmatic keys, can be used.
The keys created by clicking thru console UI are working just fine.
However, for our use case, we need to create the keys using programmatic ways.
There is unanswered issue at github as well: https://github.com/googleapis/google-cloud-python/issues/7824
logger.info("Created new service account: {}".format(ret))
logger.info("Getting the new service account key")
request=iam.projects().serviceAccounts().keys().create(name=ret['name'],
body={'privateKeyType':'TYPE_GOOGLE_CREDENTIALS_FILE'})
key=request.execute()
>>>print json.dumps(key, indent=4) #just to verify what we got
{
"keyOrigin": "GOOGLE_PROVIDED",
"name": "goodandvalidname",
"validBeforeTime": "2029-06-28T15:09:59Z",
"privateKeyData": "datadata",
"privateKeyType": "TYPE_GOOGLE_CREDENTIALS_FILE",
"keyAlgorithm": "KEY_ALG_RSA_2048",
"validAfterTime": "2019-07-01T15:09:59Z"
}
>>> credentials = google.oauth2.service_account.Credentials.from_service_account_info(key)
Traceback (most recent call last):
File "/home/user/.p2/pool/plugins/org.python.pydev.core_7.2.1.201904261721/pysrc/_pydevd_bundle/pydevd_exec.py", line 3, in Exec
exec exp in global_vars, local_vars
File "<console>", line 1, in <module>
File "/home/user/.local/lib/python2.7/site-packages/google/oauth2/service_account.py", line 193, in from_service_account_info
info, require=['client_email', 'token_uri'])
File "/home/user/.local/lib/python2.7/site-packages/google/auth/_service_account_info.py", line 51, in from_dict
'fields {}.'.format(', '.join(missing)))
ValueError: Service account info was not in the expected format, missing fields token_uri, client_email.
Any help appreciated.
Answering my own issue and probably helping others...
The 'key' we get from python APIs is NOT the 'json key' as obtained from gcloud. The dict we get from iam.projects().serviceAccounts().keys().create() contains the field privateKeyData which itself contains ENTIRE 'json key' one needs to authenticate to google cloud.
The data in this field is base64 encoded and needs decoding, and subsequently dumping to json. Below is the snippet from functional code, demonstrating the credentials are loaded back from such key:
request=iam.projects().serviceAccounts().keys().create(name=ret['name'],
body={'privateKeyType':'TYPE_GOOGLE_CREDENTIALS_FILE'})
key=request.execute()
key=base64.decodestring(key['privateKeyData'])
key=json.loads(key)
credentials = google.oauth2.service_account.Credentials.from_service_account_info(key)
I figured this out by stepping thru gcloud service account key creating, line by line, using python debugger. Hope this helps others.

IBM - COS - SDK IAM token

I am trying to access my COS service using python.Referring IBM's Documentation was able to write the following code snippet
import ibm_boto3
from ibm_botocore.client import Config
api_key = 'key'
service_instance_id = 'resource-service-id'
auth_endpoint = 'http://iam.bluemix.net/'
service_endpoint = 'endpoint'
s3 = ibm_boto3.resource('s3',
ibm_api_key_id=api_key,
ibm_service_instance_id=service_instance_id,
ibm_auth_endpoint=auth_endpoint,
config=Config(signature_version='oauth'),
endpoint_url=service_endpoint)
s3.Bucket('bucket name').download_file('object name','location where the object must be saved')
Is this correct ? Also while trying to execute the above code the compiler is not able to retrieve the authentication token from auth_endpoint. Am i missing something?
Please to help
Thanks in advance!
I am including the output for your reference...
ibm_botocore.exceptions.CredentialRetrievalError: Error when retrieving credentials from https://iam.ng.bluemix.net/oidc/token: Retrieval of tokens from server failed
And I am using python 3.x
As instructed in README, the auth_endpoint should have /oidc/token at the end, for example, 'http://iam.bluemix.net/oidc/token'.
auth_endpoint = 'https://iam.bluemix.net/oidc/token'
The auth_endpoint should be https
See the example here
https://github.com/IBM/ibm-cos-sdk-python
To Connect with ibm cloud storage account we need api_key, service_instace_id,auth_endpoint and service_endpoint.
import ibm_boto3
from ibm_botocore.client import Config
api_key = '......' # u can find api_key in service credentials in ibm cloud account
service_instance_id = '.....' u can find service_instance_id in service credentials in ibm cloud account
auth_endpoint = 'https://iam.bluemix.net/oidc/token'
service_endpoint = 'https://s3-api.us-geo.objectstorage.softlayer.net'
cos = ibm_boto3.resource('s3',
ibm_api_key_id=api_key,
ibm_service_instance_id=service_instance_id,
ibm_auth_endpoint=auth_endpoint,
config=Config(signature_version='oauth'),
endpoint_url=service_endpoint)
to create a bucket
new_bucket = 'abcd1234'
def create_bucket():
cos.create_bucket(Bucket=new_bucket)
return "Bucket created sucessfully"
create_bucket()
to list Buckets in cloud
def get_buckets():
print("Retrieving list of buckets")
try:
buckets = cos.buckets.all()
for bucket in buckets:
print("Bucket Name: {0}".format(bucket.name))
except ClientError as be:
print("CLIENT ERROR: {0}\n".format(be))
except Exception as e:
print("Unable to retrieve list buckets: {0}".format(e))
get_buckets()

Flask response setting is not working in Bluemix Flask Buildpack

I am trying deploying my flask application to the Bluemix where it is configured by Bluemix Python Flask BuildPack and it is not a custom build back.
#app.route('/run', methods=['GET','POST'])
def stockModuleRunner():
stockCode = request.args.get("stock_code","")
# Store the reference, in case you want to show things again in standard output
old_stdout = sys.stdout
# This variable will store everything that is sent to the standard output
result = StringIO()
sys.stdout = result
# Here we can call anything
# Here we can call anything we like, like external modules, and everything that they will send to standard output will be stored on "result"
r = Resilience.Resilience()
r.main(stockCode)
# Redirect again the std output to screen
sys.stdout = old_stdout
# Then, get the stdout like a string and process it!
result_string = result.getvalue()
print(result_string)
#r = Resilience.Resilience()
#r.main(stockCode)
resp = Response("<html><pre>"+result_string+"</pre></html>")
resp.headers['content-type'] = 'text/html; charset=utf-8'
return
I set the content type of my response as a text/html. But it didn't work in the Bluemix, while it worked in my local environment. In the Bluemix, it changed to text/xml automatically. I confirmed it through the network tab of chrome developer tool. I got the error like the followings. As per the result of a google search, I found out that it is a sort of xml parsing error. Do you have any idea of this problem?
This page contains the following errors:
error on line 1 at column 1: Document is empty
Below is a rendering of the page up to the first error.