I want to add metadata to Minio object while adding the file as object to Minio object storage using python. I am able to find accessing metadata of object stored on Minio. but there is no example of adding metadata while adding file to Minio storage.
Regards,
Ritu Ranjan
Well it there is a examples at python minio client test
content_type='application/octet-stream'
metadata = {'x-amz-meta-testing': 'value'}
client.put_object(bucket_name,
object_name+'-metadata',
MB_11_reader,
MB_11,
content_type,
metadata)
The trick is that metadata dict should have keys in format
'x-amz-meta-youkey'
You can use pyminio:
from pyminio import Pyminio
pyminio_client = Pyminio.from_credentials(
endpoint='<your-minio-endpoint>', # e.g. "localhost:9000/"
access_key='<your-minio-access-key>',
secret_key='<your-minio-secret-key>'
)
metadata = {'Pyminio-is': 'Awesome'}
pyminio_client.put_file(to_path='/foo/bar/baz', file_path='/mnt/some_file', metadata=metadata)
Its automaticly strips off the'x-amz-meta-' from the name of the variables so its more easy to use with pyminio_client.get('/foo/bar/baz')
Related
Got a use case where I have to do read and write operations to multiple object store which are S3 implementations(Ceph and AWS S3). I am using alpakka and added both config for ceph and s3 in app.config file like this:
s3{
aws{
credentials{
provider=static
access-key-id=<KEY>
secret-access-key=<KEY>
}
region{
provider=static
default-region=<AWS region>
}
}
endpoint-url=<S3 endpoint url>
}
#The following config is for Ceph
alpakka.s3{
aws{
credentials{
provider=static
access-key-id=<MINIO KEY>
secret-access-key=<MINIO KEY>
}
region{
provider=static
default-region=<AWS region>
}
}
endpoint-url=<MINIO endpoint url>
}
In the following code I am currently trying to access both the config to perform read/write:
private val s3ConfigAttribute1=S3Attributes.settings(system.settings.config.getConfig(S3Settings.ConfigPath)) #alpakka will by default load alpakka.s3 path
private val s3ConfigAttribute2=S3Attributes.settings(system.settings.config.getConfig(<Path to "s3" config>))
I am able to add one config for example:
S3.download.withAttributes(s3ConfigAttribute1).map.......
How to use composition of the config? How to use both config to read and write? Thanks in advance
I can't find some sort of list that would show accepted types for outputs in Tekton Tasks.
Is it somehow fixed or is it possible to use any file extension? I have been having troubles with .xml files in my case.
Thanks in advance
Actually, the tekton's output do not care what the file extension, all files output belong to type: storage.
validate types of outputs should be:
// PipelineResourceTypeGit indicates that this source is a GitHub repo.
PipelineResourceTypeGit PipelineResourceType = "git"
// PipelineResourceTypeStorage indicates that this source is a storage blob resource.
PipelineResourceTypeStorage PipelineResourceType = "storage"
// PipelineResourceTypeImage indicates that this source is a docker Image.
PipelineResourceTypeImage PipelineResourceType = "image"
// PipelineResourceTypeCluster indicates that this source is a k8s cluster Image.
PipelineResourceTypeCluster PipelineResourceType = "cluster"
// PipelineResourceTypePullRequest indicates that this source is a SCM Pull Request.
PipelineResourceTypePullRequest PipelineResourceType = "pullRequest"
// PipelineResourceTypeCloudEvent indicates that this source is a cloud event URI
PipelineResourceTypeCloudEvent PipelineResourceType = "cloudEvent"
// PipelineResourceTypeGCS is the subtype for the GCSResources, which is backed by a GCS blob/directory.
PipelineResourceTypeGCS PipelineResourceType = "gcs"
I'm trying to create a bucket in cloud object storage using python. I have followed the instructions in the API docs.
This is the code I'm using
COS_ENDPOINT = "https://control.cloud-object-storage.cloud.ibm.com/v2/endpoints"
# Create client
cos = ibm_boto3.client("s3",
ibm_api_key_id=COS_API_KEY_ID,
ibm_service_instance_id=COS_INSTANCE_CRN,
config=Config(signature_version="oauth"),
endpoint_url=COS_ENDPOINT
)
s3 = ibm_boto3.resource('s3')
def create_bucket(bucket_name):
print("Creating new bucket: {0}".format(bucket_name))
s3.Bucket(bucket_name).create()
return
bucket_name = 'test_bucket_442332'
create_bucket(bucket_name)
I'm getting this error - I tried setting CreateBucketConfiguration={"LocationConstraint":"us-south"}, but it doesnt seem to work
"ClientError: An error occurred (IllegalLocationConstraintException) when calling the CreateBucket operation: The unspecified location constraint is incompatible for the region specific endpoint this request was sent to."
Resolved by going to https://cloud.ibm.com/docs/cloud-object-storage?topic=cloud-object-storage-endpoints#endpoints
And choosing the endpoint specific to the region I need. The "Endpoint" provided with the credentials, is not the actual endpoint.
I am trying to download a file for the first time from Google Cloud Storage.
I set the path to the googstruct.json service account key file that I downloaded from https://cloud.google.com/storage/docs/reference/libraries#client-libraries-usage-python
Do need to set the authorization to Google Cloud outside the code somehow? Or is there a better "How to use Google Cloud Storage" then the one on the google site?
It seems like I am passing the wrong type to the storage_client = storage.Client()
the exception string is below.
Exception has occurred: google.auth.exceptions.DefaultCredentialsError
The file C:\Users\Cary\Documents\Programming\Python\QGIS\GoogleCloud\googstruct.json does not have a valid type.
Type is None, expected one of ('authorized_user', 'service_account').
MY PYTHON 3.7 CODE
from google.cloud import storage
import os
os.environ["GOOGLE_APPLICATION_CREDENTIALS"]="C:\\GoogleCloud\\googstruct.json"
# Instantiates a client
storage_client = storage.Client()
bucket_name = 'structure_ssi'
destination_file_name = "C:\\Users\\18809_PIPEM.shp"
source_blob_name = '18809_PIPEM.shp'
download_blob(bucket_name, source_blob_name, destination_file_name)
def download_blob(bucket_name, source_blob_name, destination_file_name):
"""Downloads a blob from the bucket."""
storage_client = storage.Client()
bucket = storage_client.get_bucket(bucket_name)
blob = bucket.blob(source_blob_name)
blob.download_to_filename(destination_file_name)
print('Blob {} downloaded to {}.'.format(
source_blob_name,
destination_file_name
)
)
I did look at this but I cannot tell if this is my issue. I have tried both.
('Unexpected credentials type', None, 'Expected', 'service_account') with oauth2client (Python)
This error means that the Json Service Account Credentials that you are trying to use C:\\GoogleCloud\\googstruct.json are corrupt or the wrong type.
The first (or second) line in the file googstruct.json should be "type": "service_account".
Another few items to improve your code:
You do not need to use \\, just use / to make your code easier
and cleaner to read.
Load your credentials directly and do not modify environment
variables:
storage_client = storage.Client.from_service_account_json('C:/GoogleCloud/googstruct.json')
Wrap API calls in try / except. Stack traces do not impress customers. It is better to have clear, simple, easy to read error messages.
I can see a sample of uploading a file to the Google Cloud storage. I, however, can't find a sample of deleting a file in the cloud storage. Does deleting a file API exist?
Here's the delete API documentation for the JSON API: https://cloud.google.com/storage/docs/json_api/v1/objects/delete
def delete_blob(bucket_name, destination_blob_name):
"""Deletes a blob from the bucket."""
try:
storage_client = storage.Client(project = project)
bucket = storage_client.get_bucket(bucket_name)
blob = bucket.blob(destination_blob_name)
blob.delete()
log.info('Blob {} deleted.'.format(destination_blob_name))
except Exception as e:
pass