I have several separate Mongodb databases with identical collection structure. They have to be separate databases for legal reasons.
I would like to be able to create a query that pulls data across all separate databases, combine the data into a single dataset so I get one view across all of my separate databased.
Assume that my databases are named db1, db2, db3. Assume that the collections inside each are the same structure. col1.name, col1.address, col1.zip
What kind of query will pull names from each collection from all databases?
Related
I have two collections, that are store on Atlas.
I want to be able to search for a text, using the regex operator of Mongo Atlas Search on these two collections, but with only one aggregation.
These two collections have nothing in common, so I can't do a $lookup on it.
I need to do this search in one aggregation because the result of this aggregation must be paginated with $limit and $skip in the aggregation.
How to perform that ?
You can use Atlas Data Lake on top of these two collections to federate queries across both. To do this you would create a Data Lake, create a virtual collection in your Data Lake, and then have both of these collections listed as "datasources" for that virtual collection.
The Data Lake infrastructure will push down queries to the underlying cluster and union the results before returning them to the client application.
In this example there are multiple types of data sources, but you can just list multiple cluster collections.
https://docs.mongodb.com/datalake/reference/format/data-lake-configuration/#example-configuration-for-running-federated-queries
i have done with in the same database, Can someone help i need to join different database collection and push it to elasticsearch to visualize in kibana
We have a requirement in our project to write aggregation queries across multiple mongodb databases. I see that it is not possible with Mongo to query another database in an aggregate query. What would my options be in this case?
We would not mind to create the same collections manually across multiple databases as long as these derived collections sync automatically when the primary collection is updated. Does Mongo have any such solutions to keep collections across databases in sync?
"Almost." You can use the Change Stream feature to listen for changes on the source database but you must write a piece of code to insert/update the material into the target database.
We want to migrate data to mongodb using a CSV files dump created out of a teradata. Data needs to be refreshed in mongodb every night from a fresh teradata csv dump
Approach we are thinking of is:
Get the CSV files exported out of relational db. They are going to be very similar to table structure in relational db
Import the CSV files into mongodb staging collections subsequently, which will be mirroring relational db structure in terms of being normalised. This may be done using say mongoimport in overnight batches. This is going to result in many collections as we are thinking to import each 'type' of CSV into its own collection e.g. Customers.csv and Accounts.csv will result in two respective collections of same name.
Create de-normalised collections out of staging collections, ready to be exposed to UI. Run some schema migration script, which queries the staging collections and creates more denormalised and fewer collections ready for use in the application UI.Eg, Customers and Accounts collections, after running migration script should result in a third collection say AccountCustomers collection where Each account doc has embedded customers array (denormalised and no need of joins when UI needs data)
Question: Is there a better strategy , as all above steps are expected to complete nightly, every night?
Question: Is mongoimport OK to use for importing the CSV files in nightly batches.
Question: What is the best way to migrate (denormalise) collections within the same mongo instance?Eg we have stagingdb having collections customers and accounts and we want to reach a state where we have proddb having collection accountcustomers
Lets say i have two collections
The Family Collection
{"Name":"Steven", "Children":[{"Name":"Liv", "Children":[{"Name":"Milo"}] },{"Name":"Mia"},{"Name":"Chelsea"}]}
And the movie collection
{"Movie":"Rush Hour 3", "Actors":["Jackie","Mia"]}
{"Movie":"LOTR", "Actors":["Viggo","Liv"]}
Now i need a query to find the names of the family members that's not actors. The result should contain Steven, Milo and Chelsea.
You would have to do this "yourself" in your application logic. Mongo doesnt do joins. So essentially, your current collections are structured similar to relational independent tables -- in mongo, you would have to redesign your schema so that your query is possible (or , like I said, do it in the app)