I have several images in a container in Blob Storage that have metadata I want to store in CosmosDB. The images are Jpeg's, Would I have to pull the data into Data Factory or is there a simpler method?
I think the simplest way would be to just write a small console app to loop through all your blob containers and items and insert into Cosmos DB using it's SDK.
There are many ways to loop through images stored in Blob Storage and store them in Cosmos DB. Please refer to below links. A little tweak and required functionality can be achieved.
Using ADF
Link
Use Azure Logic App
Link 1 Link 2
Write a custom code and then populate the result into Cosmos DB
Link
Related
I have a setup which consists of devices sending data to Azure cloud IoT Hub using message routing (to storage endpoints) which land up as blobs in a container. The frequency of data push is high. On the other end, I want to be able to query my blob container to pull files based on specific dates.
I came across blob index tags which look like a promising solution to query and is supported by the Azure SDK for .net.
I was thinking to add tags to each blob ex: processedDate: <dd/mm/yyyy>, which would help me query on the same later.
I found out that while uploading the blobs manually it is possible to add the tags but not sure how to go about or where to configure the same in the message routing flow where blobs are created on the fly. So I am looking for a solution to add those tags in flight as they are being pushed on to the container.
Any help on this will be much appreciated.
Thanks much!
Presently, the Azure IoT Hub doesn't have a feature to populate a custom endpoint for instance headers, properties, tags, etc.
However, in your case such as a storage custom endpoint you can use an EventGridTrigger function to populate a blob based on your needs.
I want to create and maintain mongodb atlas datalake programmatically but seems there is no option available. I could find out one API which can be used to create/update/delete data lake but that it only allows to set some options. Here is the link I am following - https://docs.mongodb.com/datalake/reference/api/dataLakes-create-one-tenant/#request-body-parameters
Doesn anyone know how to setup other options like data store, storage etc which you can do from mongodb atlas UI?
I want to catalog data stored in csv files in the Azure Blob Storage. I tried to see if there is anyway to get metadata of Blob Storage and found Data Catalog is an option. Thing is, csv file is handled as a blob type and we can not profile it. I want, csv files in blob storage to act as tables.
Is this possible using Azure Data Catalog?
Yes you can use Data Catalog, For updated Data Catalog features, please use the new Azure Purview service, which offers unified data governance for your entire data estate. I would recommend to use : Azure Purview( Still you possible through Data Catalog)
Registering assets from a data source copies the assets’ metadata to Azure, but the data remains in the existing data-source location.
For updated Data Catalog features, please use the new Azure Purview service, which offers unified data governance for your entire data estate.
Introduction to Azure Purview (preview) - Azure Purview
This article provides an overview of Azure Purview, including its features and the problems it addresses. Azure Purview enables any user to register, discover, understand, and consume data sources.
This article outlines how to register an Azure Blob Storage account in Purview and set up a scan.
For more information on Blob index tags categorize data in your storage account using key-value tag attributes. These tags are automatically indexed and exposed as a searchable multi-dimensional index to easily find data. This article shows you how to set, get, and find data using blob index tags. Use blob index tags to manage and find data on Azure Blob Storage
I have a simple, single collection in Firestore, a museum visitor name and date with some other fields.
I successfully ran a gcloud export:
gcloud beta firestore export gs://climatemuseumexhibition2019gov.appspot.com --collection-ids=visitors
and the collection is now sitting in a bucket in Cloud Storage.
I tried downloading the data, which appears to be in several chunks, but it's in .dms format and I have no idea what that is.
Also, I've successfully linked BigQuery to my firestore project but don't see the collection, and I don't see the Cloud Storage object at all.
I gather the idea is create a Dataset from the Cloud Storage, then create a Table from the dataset. I'd appreciate specific details on how to do that.
I've read the manuals and they are opaque on this topic, so I'd appreciate some first hand experience. Thank you.
I have set up my Mongodb database have connected successfully.
However my project is to create an online cookbook.
My database currently is various recipes but with each document I want an image to be linked to it.
I know I can use gridFS but I would prefer to store the images in the same place as the recipes.
I have seen I can use base64 but that is no appearing for me
I am very new to using Mongo and some advise is greatly appreciated
Store the images in s3 bucket and storing the path of the s3 image in the DB.
Using s3 url you can access the image.