I have a simple, single collection in Firestore, a museum visitor name and date with some other fields.
I successfully ran a gcloud export:
gcloud beta firestore export gs://climatemuseumexhibition2019gov.appspot.com --collection-ids=visitors
and the collection is now sitting in a bucket in Cloud Storage.
I tried downloading the data, which appears to be in several chunks, but it's in .dms format and I have no idea what that is.
Also, I've successfully linked BigQuery to my firestore project but don't see the collection, and I don't see the Cloud Storage object at all.
I gather the idea is create a Dataset from the Cloud Storage, then create a Table from the dataset. I'd appreciate specific details on how to do that.
I've read the manuals and they are opaque on this topic, so I'd appreciate some first hand experience. Thank you.
Related
I'm using the Cloud Api product search, they ask me to store the images and the csv file in the google cloud storage but i'm already using firebase in my project and my images stored there so can i choose firebase than the google cloud ?
Yes, you can use it as an alternative. As Renaud Tarnec commented, Firebase Storage underneath is Google Cloud Storage. This means that any bucket in your Firebase console is actually a regular Google Cloud Storage bucket.
You can confirm this by creating a Storage Bucket in the Firebase console and uploading objects to it (images for your dataset and the CSV file). If you access the Cloud Storage page in the GCP console, the same bucket and objects will be there.
As for using the Vision Product Search API with Firebase Storage, I followed this quickstart guide, but edited the CSV file to point to images that were uploaded to the default Firebase Storage bucket (PROJECT-ID.appspot.com).
After letting the dataset images index, I was then able to send an image to be compared to the dataset as expected.
Keep in mind the service account you use to create a Vision API dataset must be able to read Cloud Storage Objects to access the images in your Firebase Storage bucket. Let me know if this was useful.
I am getting near-real time data from 100s of customers. I need to store this data in Google Cloud Storage buckets created for each customer i.e. /gcs/customer_id/yy/mm/day/hhhh/
My data is in Avro. I guess I can use Pub/Sub to Avro Files on Cloud Storage template.
However, I'm not sure if Google Pub/Sub can accept data from multiple customers.
Appreciate any help here, thanks!
The template is quite simple: it takes all the data of PubSub and store them in an avro file on GCS.
However, it's a good starting point and you can make evolutions on that base to add a split per customer, and the file path that you want.
You can find the template in Java format on GitHub
How to daily import Google workspace data automatically to Big Query database?
I'm new to Big Query and i can do it manually but i want to automate this process. Thanks.
With BigQuery you can create external tables, that enable you to query data that is stored in your Google Drive (CSV, Avro, JSON, or Google Sheets documents).
You can find a nice how-to here.
I have several images in a container in Blob Storage that have metadata I want to store in CosmosDB. The images are Jpeg's, Would I have to pull the data into Data Factory or is there a simpler method?
I think the simplest way would be to just write a small console app to loop through all your blob containers and items and insert into Cosmos DB using it's SDK.
There are many ways to loop through images stored in Blob Storage and store them in Cosmos DB. Please refer to below links. A little tweak and required functionality can be achieved.
Using ADF
Link
Use Azure Logic App
Link 1 Link 2
Write a custom code and then populate the result into Cosmos DB
Link
I will try to explain the context of the project on which I'm working and the problem which I currently try to overcome. I would like to collect mobile data for analytics purpose, e.g. using Firebase from Google. After collecting thoses data, I'd like to store them in a local database, such as PostgreSQL or MongoDB. But the thing is mobile data collecting platform such as Firebase mostly doesn't support connecting to a local database. I've found out that there is a possibility to export raw data from Firebase to import into a local database, but I have no detail information about this. I've searched through many documentations and I couldn't find anything.Has everyone ever had this kind of problem and can give a clear guidance about exporting data from Firebase and import into a local database? Thanks in advance.
Google Analytics for Firebase collects enormous amount of data and send it to the servers in form of batches or bundles. Three things here with your design:
SDK collects that data and does not expose it for you to parse in the app and store it in local DB.
Storing this much data locally, you are going to re-invent the wheel. In addition, this could cause performance issues.
If you need all the data collected by Firebase in raw form, just link Firebase to Big Query and all data would be available to you.