I would like to upload a number of photos to the Bluemix Object Storage service and then display them in a web app. Right now a GET request to the photo in the object storage container requires and auth token. Is there any way I can create a public URL to the object that would not require an auth token for a GET request?
I see there is an option of creating temporary URLs to objects but I don't want the URL to be temporary I want it to live forever. Is the only option to create a long lived temporary URL?
The correct way to do this is to modify the container ACL. You cannot do this via the Bluemix UI currently but you can using the Swift REST API. For example, to change the container ACL so anyone can read objects in the container you can issue the following PUT request.
curl -X PUT "https://dal.objectstorage.open.softlayer.com/v1/AUTH_123/mycontainer" \
-H "X-Auth-Token: token123" \
-H "X-Container-Read: .r:*"
I know this is a old post but with the help of Ryan Baxter and Object storage documentation in IBM I could resolve the Issue
Finally these too commands saved the day
First use swift and change access control of container
swift post container-name --read-acl ".r:*,.rlistings"
Next Using Curl Configure Container to a particular Url for accessing Files
curl -X GET " https://<access point>/<version>/AUTH_projectID/container-name" -H "X-Auth-Token:<auth token>" -H "X-Container-Read: .r:*,.rlistings"
And also very grateful for the help provided by Alex da Silva
Now BlueMix has an S3 endpoint capability. You can use curl or any other langage for exemple here is an boto3 that will upload an object, make it public and and some metadata :
( the function is using a json file on which you store the credentials and it uses 3 variables that are used in the global app : currentdirpath,ImagesToS3,ImageName )
def UploadImageDansBucket (currentdirpath,ImagesToS3,ImageName) :
currentdirpath = 'path/to/your/dir/current'
ImagesToS3 = ' /path/of/your/object/'
ImageName = 'Objectname'
with open("credentials.json", 'r') as f:
data = json.loads(f.read())
bucket_target = data["aws"]["targetBucket"]
print ('Open Connection to the bucket in the cloud..........')
s3ressource = boto3.resource(
service_name='s3',
endpoint_url= data["aws"]["hostEndPoint"],
aws_access_key_id= data["aws"]["idKey"],
aws_secret_access_key=data["aws"]["secretKey"],
use_ssl=True,
)
s3ressource.meta.client.meta.events.unregister('before-sign.s3', fix_s3_host)
s3ressource.Object(bucket_target, 'hello.txt').put(Body=b"I'm a test file")
s3ressource.Object(bucket_target, 'bin.txt').put(Body=b"0123456789abcdef"*10000)
fn = "%s%s" % (ImagesToS3,ImageName)
data = open(fn, 'rb')
#s3ressource.Bucket(bucket_target).put_object(Key=fn, Body=data)
now = datetime.datetime.now() # on recupere la date actuelle
timestamp = time.mktime(now.timetuple()) # on effectue la convertion
timestampstr = str (timestamp)
s3ressource.Bucket(bucket_target).upload_file(fn,ImageName, ExtraArgs={ "ACL": "public-read", "Metadata": {"METADATA1": "a" ,"METADATA2": "b","METADATA3": "c", "timestamp": timestampstr },},)
Related
I'm trying to create a bucket in cloud object storage using python. I have followed the instructions in the API docs.
This is the code I'm using
COS_ENDPOINT = "https://control.cloud-object-storage.cloud.ibm.com/v2/endpoints"
# Create client
cos = ibm_boto3.client("s3",
ibm_api_key_id=COS_API_KEY_ID,
ibm_service_instance_id=COS_INSTANCE_CRN,
config=Config(signature_version="oauth"),
endpoint_url=COS_ENDPOINT
)
s3 = ibm_boto3.resource('s3')
def create_bucket(bucket_name):
print("Creating new bucket: {0}".format(bucket_name))
s3.Bucket(bucket_name).create()
return
bucket_name = 'test_bucket_442332'
create_bucket(bucket_name)
I'm getting this error - I tried setting CreateBucketConfiguration={"LocationConstraint":"us-south"}, but it doesnt seem to work
"ClientError: An error occurred (IllegalLocationConstraintException) when calling the CreateBucket operation: The unspecified location constraint is incompatible for the region specific endpoint this request was sent to."
Resolved by going to https://cloud.ibm.com/docs/cloud-object-storage?topic=cloud-object-storage-endpoints#endpoints
And choosing the endpoint specific to the region I need. The "Endpoint" provided with the credentials, is not the actual endpoint.
I am trying to download a file for the first time from Google Cloud Storage.
I set the path to the googstruct.json service account key file that I downloaded from https://cloud.google.com/storage/docs/reference/libraries#client-libraries-usage-python
Do need to set the authorization to Google Cloud outside the code somehow? Or is there a better "How to use Google Cloud Storage" then the one on the google site?
It seems like I am passing the wrong type to the storage_client = storage.Client()
the exception string is below.
Exception has occurred: google.auth.exceptions.DefaultCredentialsError
The file C:\Users\Cary\Documents\Programming\Python\QGIS\GoogleCloud\googstruct.json does not have a valid type.
Type is None, expected one of ('authorized_user', 'service_account').
MY PYTHON 3.7 CODE
from google.cloud import storage
import os
os.environ["GOOGLE_APPLICATION_CREDENTIALS"]="C:\\GoogleCloud\\googstruct.json"
# Instantiates a client
storage_client = storage.Client()
bucket_name = 'structure_ssi'
destination_file_name = "C:\\Users\\18809_PIPEM.shp"
source_blob_name = '18809_PIPEM.shp'
download_blob(bucket_name, source_blob_name, destination_file_name)
def download_blob(bucket_name, source_blob_name, destination_file_name):
"""Downloads a blob from the bucket."""
storage_client = storage.Client()
bucket = storage_client.get_bucket(bucket_name)
blob = bucket.blob(source_blob_name)
blob.download_to_filename(destination_file_name)
print('Blob {} downloaded to {}.'.format(
source_blob_name,
destination_file_name
)
)
I did look at this but I cannot tell if this is my issue. I have tried both.
('Unexpected credentials type', None, 'Expected', 'service_account') with oauth2client (Python)
This error means that the Json Service Account Credentials that you are trying to use C:\\GoogleCloud\\googstruct.json are corrupt or the wrong type.
The first (or second) line in the file googstruct.json should be "type": "service_account".
Another few items to improve your code:
You do not need to use \\, just use / to make your code easier
and cleaner to read.
Load your credentials directly and do not modify environment
variables:
storage_client = storage.Client.from_service_account_json('C:/GoogleCloud/googstruct.json')
Wrap API calls in try / except. Stack traces do not impress customers. It is better to have clear, simple, easy to read error messages.
I am trying to access my COS service using python.Referring IBM's Documentation was able to write the following code snippet
import ibm_boto3
from ibm_botocore.client import Config
api_key = 'key'
service_instance_id = 'resource-service-id'
auth_endpoint = 'http://iam.bluemix.net/'
service_endpoint = 'endpoint'
s3 = ibm_boto3.resource('s3',
ibm_api_key_id=api_key,
ibm_service_instance_id=service_instance_id,
ibm_auth_endpoint=auth_endpoint,
config=Config(signature_version='oauth'),
endpoint_url=service_endpoint)
s3.Bucket('bucket name').download_file('object name','location where the object must be saved')
Is this correct ? Also while trying to execute the above code the compiler is not able to retrieve the authentication token from auth_endpoint. Am i missing something?
Please to help
Thanks in advance!
I am including the output for your reference...
ibm_botocore.exceptions.CredentialRetrievalError: Error when retrieving credentials from https://iam.ng.bluemix.net/oidc/token: Retrieval of tokens from server failed
And I am using python 3.x
As instructed in README, the auth_endpoint should have /oidc/token at the end, for example, 'http://iam.bluemix.net/oidc/token'.
auth_endpoint = 'https://iam.bluemix.net/oidc/token'
The auth_endpoint should be https
See the example here
https://github.com/IBM/ibm-cos-sdk-python
To Connect with ibm cloud storage account we need api_key, service_instace_id,auth_endpoint and service_endpoint.
import ibm_boto3
from ibm_botocore.client import Config
api_key = '......' # u can find api_key in service credentials in ibm cloud account
service_instance_id = '.....' u can find service_instance_id in service credentials in ibm cloud account
auth_endpoint = 'https://iam.bluemix.net/oidc/token'
service_endpoint = 'https://s3-api.us-geo.objectstorage.softlayer.net'
cos = ibm_boto3.resource('s3',
ibm_api_key_id=api_key,
ibm_service_instance_id=service_instance_id,
ibm_auth_endpoint=auth_endpoint,
config=Config(signature_version='oauth'),
endpoint_url=service_endpoint)
to create a bucket
new_bucket = 'abcd1234'
def create_bucket():
cos.create_bucket(Bucket=new_bucket)
return "Bucket created sucessfully"
create_bucket()
to list Buckets in cloud
def get_buckets():
print("Retrieving list of buckets")
try:
buckets = cos.buckets.all()
for bucket in buckets:
print("Bucket Name: {0}".format(bucket.name))
except ClientError as be:
print("CLIENT ERROR: {0}\n".format(be))
except Exception as e:
print("Unable to retrieve list buckets: {0}".format(e))
get_buckets()
I am trying deploying my flask application to the Bluemix where it is configured by Bluemix Python Flask BuildPack and it is not a custom build back.
#app.route('/run', methods=['GET','POST'])
def stockModuleRunner():
stockCode = request.args.get("stock_code","")
# Store the reference, in case you want to show things again in standard output
old_stdout = sys.stdout
# This variable will store everything that is sent to the standard output
result = StringIO()
sys.stdout = result
# Here we can call anything
# Here we can call anything we like, like external modules, and everything that they will send to standard output will be stored on "result"
r = Resilience.Resilience()
r.main(stockCode)
# Redirect again the std output to screen
sys.stdout = old_stdout
# Then, get the stdout like a string and process it!
result_string = result.getvalue()
print(result_string)
#r = Resilience.Resilience()
#r.main(stockCode)
resp = Response("<html><pre>"+result_string+"</pre></html>")
resp.headers['content-type'] = 'text/html; charset=utf-8'
return
I set the content type of my response as a text/html. But it didn't work in the Bluemix, while it worked in my local environment. In the Bluemix, it changed to text/xml automatically. I confirmed it through the network tab of chrome developer tool. I got the error like the followings. As per the result of a google search, I found out that it is a sort of xml parsing error. Do you have any idea of this problem?
This page contains the following errors:
error on line 1 at column 1: Document is empty
Below is a rendering of the page up to the first error.
Until now, I've been working with a context Broker Instance in stand-alone mode created in Fi-Lab/Cloud and using psb-orion-image. Now I would like to federate an instance of that CB with http//orion.lab.fi-ware.org. I use XML forms to create, update... and the name of the instance to federate is "UPCT:TEMPERATURE:SENSOR" wich sensor type is "UPCT:SENSOR".
So, connecting by SSH I send the next form:
(curl localhost:1026/NGSI10/subscribeContext -s -S --header 'Content-Type: application/xml' -d #- | xmllint --format -) <<EOF
<?xml version="1.0"?>
<subscribeContextRequest>
<entityIdList>
<entityId type="UPCT:SENSOR" isPattern="false">
<id>UPCT:TEMPERATURE:SENSOR</id>
</entityId>
</entityIdList>
<reference>http://orion.lab.fi-ware.eu:1026/ngsi10/notifyContext</reference>
<duration>P1M</duration>
<notifyConditions>
<notifyCondition>
<type>ONCHANGE</type>
<condValueList>
<condValue>temperature</condValue>
</condValueList>
</notifyCondition>
</notifyConditions>
<throttling>PT5S</throttling>
</subscribeContextRequest>
EOF
And I get a right reply, with and subscription ID. However, if I update de contextValue of my instance and try to send a query to http://orion.lab.fi.ware.eu:1026 I receive an error:
-:1: parser error : Start tag expected, '<' not found
Auth-token not found in request header
^
I think I should get the same value that I update to my Instance as indicate in
https://forge.fi-ware.org/plugins/mediawiki/wiki/fiware/index.php/Publish/Subscribe_Broker_-_Orion_Context_Broker_-_User_and_Programmers_Guide#Context_Broker_Federation
I need to know what's wrong and how could I do federete to the global CB.
Thank you
That error message is due to any request sent to Orion instance at orion.lab.fi-ware.org (including the notifications sent by other Orion instances) has to use autentication. At the present moment (i.e. version 0.14.1), Orion doesn't include the X-Auth-Token needed for authentication (see the quick start for programmers) in notifications.
However, the usual use case is federating Orion at orion.lab.fi-ware.org with private Orion user instances (i.e. orion.lab.fi-ware.org -> your Orion) not in the opposite way (i.e. your Orion -> orion.lab.fi-ware.org) as your are trying. This is given that typically what you want to do is to merge public information (e.g. Santander city sensors) with private information (e.g. the information produced by your sensors). Federating orion.lab.fi-ware.org -> your Orion in that way should work perfectly.
EDIT: the limitation in Orion 0.14.1 has been overcomed an current Orion version (2.0.0) propagates X-Auth-Token header in notifications (I don't remember in which exact version between 0.14.1 and 2.0.0, sorry...)