Using S3 Cloud Storage on IBM Bluemix - ibm-cloud

Question

I am planning to use S3 Cloudstorage in IBM Bluemix but then one strange thing I found is that there is no way to add the custom META-DATA to the objects which are stored in S3 bucket.
Is there a way I can add custom Meta-Data to the objects and if yes then can you please advise on how we can add it and access it.?

Answer 1

Thanks for pointing out a hole in the documentation!
Custom metadata is defined by passing a x-amz-meta-{key} header with a {value} value. As an example request:
PUT /{bucket-name}/{object-name} HTTP/1.1
Authorization: {authorization-string}
x-amz-meta-foo: bar
x-amz-date: 20160825T183001Z
x-amz-content-sha256:{hashed-body}
Content-Type: text/plain; charset=utf-8
Host: s3-api.us-geo.objectstorage.softlayer.net
Content-Length: 18
{
"foo": "bar"
}
A HEAD request to check the metadata would look like:
HEAD /{bucket-name}/{object-name} HTTP/1.1
Authorization: {authorization-string}
x-amz-date: 20160825T183244Z
Host: s3-api.us-geo.objectstorage.softlayer.net
And respond with:
HTTP/1.1 200 OK
Date: Thu, 25 Aug 2016 18:32:44 GMT
X-Clv-Request-Id: da214d69-1999-4461-a130-81ba33c484a6
Accept-Ranges: bytes
Server: Cleversafe/3.9.1.102
X-Clv-S3-Version: 2.5
ETag: {MD5-hash}
Content-Type: text/plain; charset=UTF-8
x-amz-meta-foo: bar
Last-Modified: Thu, 25 Aug 2016 17:49:06 GMT
Content-Length: 18
Using the CLI, the syntax would be:
$ aws --endpoint-url=https://{endpoint} s3 cp ~/new-file s3://bucket-1/ --metadata foo=bar
Hope that helps!

Answer 2

This is possible. I am using this every day . Adding meta data then sending the meta data in database by doing cron calls.
Here is a small example of python script to create/add/change metadata for a a list object :
import sys
import os
import boto3
import pprint
from boto3 import client
from botocore.utils import fix_s3_host
param_1= YOUR_ACCESS_KEY
param_2= YOUR_SECRETE_KEY
param_3= YOUR_END_POINT
param_4= YOUR_BUCKET
#Create the S3 client
s3ressource = client(
service_name='s3',
endpoint_url= param_3,
aws_access_key_id= param_1,
aws_secret_access_key=param_2,
use_ssl=True,
)
# Building a list of object per bucket
def BuildObjectListPerBucket (variablebucket):
global listofObjectstobeanalyzed
listofObjectstobeanalyzed = []
extensions = ['.jpg','.png']
for key in s3ressource.list_objects(Bucket=variablebucket)["Contents"]:
#print (key ['Key'])
onemoreObject=key['Key']
if onemoreObject.endswith(tuple(extensions)):
listofObjectstobeanalyzed.append(onemoreObject)
else :
s3ressource.delete_object(Bucket=variablebucket,Key=onemoreObject)
return listofObjectstobeanalyzed
# for a given existing object, create metadata
def createmetdata(bucketname,objectname):
s3ressource.upload_file(objectname, bucketname, objectname, ExtraArgs={"Metadata": {"metadata1":"ImageName","metadata2":"ImagePROPERTIES" ,"metadata3":"ImageCREATIONDATE"}})
# for a given existing object, add new metadata
def ADDmetadata(bucketname,objectname):
s3_object = s3ressource.get_object(Bucket=bucketname, Key=objectname)
k = s3ressource.head_object(Bucket = bucketname, Key = objectname)
m = k["Metadata"]
m["new_metadata"] = "ImageNEWMETADATA"
s3ressource.copy_object(Bucket = bucketname, Key = objectname, CopySource = bucketname + '/' + objectname, Metadata = m, MetadataDirective='REPLACE')
# for a given existing object, update a metadata with new value
def CHANGEmetadata(bucketname,objectname):
s3_object = s3ressource.get_object(Bucket=bucketname, Key=objectname)
k = s3ressource.head_object(Bucket = bucketname, Key = objectname)
m = k["Metadata"]
m.update({'watson_visual_rec_dic':'ImageCREATIONDATEEEEEEEEEEEEEEEEEEEEEEEEEE'})
s3ressource.copy_object(Bucket = bucketname, Key = objectname, CopySource = bucketname + '/' + objectname, Metadata = m, MetadataDirective='REPLACE')
def readmetadata (bucketname,objectname):
ALLDATAOFOBJECT = s3ressource.get_object(Bucket=bucketname, Key=objectname)
ALLDATAOFOBJECTMETADATA=ALLDATAOFOBJECT['Metadata']
print ALLDATAOFOBJECTMETADATA
# create the list of object on a per bucket basis
BuildObjectListPerBucket (variablebucket)
# Call functions to see the results
for objectitem in listofObjectstobeanalyzed:
readmetadata(param_4,objectitem)
createmetdata(param_4,objectitem)
readmetadata(param_4,objectitem)
ADDmetadata(param_4,objectitem)
readmetadata(param_4,objectitem)
CHANGEmetadata(param_4,objectitem)
readmetadata(param_4,objectitem)