How to query documents from an entire MongoDB database containing multiple collections at once (Pymongo) - mongodb

I have a DB containing thousands of collections, each collection containing documents.
How would I query all these documents from all the collections at once?
I have considered using the $lookup aggregation method but from my knowledge, it only combines an additional collection at once, whereas I have thousands of collections that I want to combine and query altogether.
Is there a way to achieve this?


MongoDB operations work on a single collection at a time. As you point out, $lookup can perform a cross-collection lookup, but it won't help you with a large number collections.
Re-design your data model.

Related

can mongodb handle queries with hundreds of filter conditions?

Supposed that a collection has billions of documents. Can mongodb handle queries that have hundreds of filter conditions on this collection by millions of users simultaneously? Assuming that this collection is indexed properly?

merge 2 collections of similar document structure together for aggregation query

I have two collections which have a similar document structure besides one has some additional fields. I have written an aggregation for data I need for one of these collections but I would rather prefer merging these collections into one before aggregating so I dont have to run that aggregation twice. Is there any way to do so?

MongoDB: Recommended schema for search in trees

Currently I have a tree with various depth that contains user's documents.
John\folder1\sub-folder2...\document
Peter\folder1...\document
But as I can see Mongo does not support indexes in nested documents.
I tried to de-normalize my DB to User\documents with children ids. But it seems search would search whole collection, not only documents for given app-user.
should I create collection for every app user?
What is the better solution to use built in Mongo aggregation methods?

You need to use only one collection and not create collection for every user.
Using aggregation first thing you would do is a match by userId which filters all the documents by that user and then do any aggregation operations.
Aggregation in mongo is pipeline. Documents move from one operation to another.
So if you do match on userId then only those documents would be chosen and the next aggregation operation will get only those documents which matches the userId. So your aggregation is still faster.

mongodb - keeping track of aggregated documents

I have a mongodb collection that stores raw information coming from an app. I wrote a multi-pipeline aggregation method to generate more meaningful data from the raw documents.
Using the $out operator in my aggregation function I store the aggregation results in another collection.
I would like to be able to either delete raw documents that were already aggregated, or somehow mark those documents so I know not to aggregate again.
I am worried that I cannot guaranty I won't miss out some documents that are created in between or create duplicate aggregated documents.
Is there a way to achieve this?

Why does MongoDB have collections

MongoDB being document-oriented, the structure of collections seems to be a special case of documents. By that I mean one can define a document to contain other documents. So a collection is just a document containing other documents.
So why do we need collections after all?

Logically yes, you could design a database system like that, but practically speaking no.
A collection has indexes on the documents in it.
A collection requires the documents in it to have unique ids.
A document is limited in size.

Object ids (_id top-level document attribute) must be unique within a collection. Multiple collections may have the same _id, just like in RDBMs where the key constraint is per-table, yet multiple tables may contain the same value for a key.

collections is a container for documents. so when you say a document that contain other documents that s kinda wrong because, already, a document can have inner documents.
Collection is the unit where you put together the documents. Be aware that due to schema free design, you can put anything in a collection but it s not a good design. so collection is kinda logical container for documents. same as tables in relational world.