What is the difference between Reference Mapping and Embedded Mapping in Doctrine MongoDB ODM?
I just need to implement one to many relationship.
Embedded documents are stored within the document itself. Referenced documents are stored elsewhere.
A simplified example of how Referenced docs are stored in the db:
//collection one
{
_id: "one_1"
many: [
"many_1",
"many_2",
"many_3"
]
}
//collection many
{
_id: "many_1",
name: "one"
},
{
_id: "many_2",
name: "two"
},
{
_id: "many_3",
name: "three"
}
and Embedded documents:
//collection one
{
_id: "one_1"
many: [
{ _id: "many_1", name: "one"},
{ _id: "many_2", name: "two"},
{ _id: "many_3", name: "three"}
]
}
The former is more flexible, the later is way quicker.
The rule of thumb is to go with references if you need to modify sub-documents individually, or if you need many-to-many refs, or if you can reach the 16MB size limit due to huge number of embedded documents in the foreseeable future.
Related
I'm new at Mongo and there might be a better way to do what I want. I'm dealing with a particular data structure that my application must process.
Suppose that I have a collection that contains two documents that contain information about universities and their student clubs to include the name of each club and the name of each student in each club along with their age:
{
_id: 1, // object ID
name: "Oxford University",
clubs: [{
name: "Soccer",
members: [{
name: "Alice",
age: 22
},
{
name: "Bob",
age: 23
}
]
}, {
name: "Gymnastics",
members: [{
name: "Charlie",
age: 20
},
{
name: "Dorothy",
age: 19
}
]
}]
}
{
_id: 2, // object ID
name: "Cambridge University",
clubs: [{
name: "Chess",
members: [{
name: "Ellen",
age: 30
},
{
name: "Frank",
age: 35
}
]
}, {
name: "Go",
members: [{
name: "Gilbert",
age: 25
},
{
name: "Hikari",
age: 40
}
]
}]
}
Suppose that I want to write a query on this collection that will find universities that have a club that has at least one member aged 40 or older.
How do I do that?
I sketched this example based off of the idea of taking some JSON documents and inserting them into a new collection. Maybe it would be a better idea to break this apart into multiple collections. I just had the idea to research if Mongo might be a good product to use in this situation given that a big part of my job here is to create something that can receive some JSON data, process it, and make it queryable.
MongoDB queries have a convenient feature to query documents that have a specific value or condition in embedded objects and arrays. In the query you can specify the "path" to the object or array using "dot notation" without having to specify the exact array index, etc.
Using your example, you can find the documents where a member of a club is aged 40 or older like this.
db.collection.find({
"clubs.members.age": {
"$gte": 40
}
})
This returns the second document in your example collection.
Try it on mongoplayground.net.
We're currently looking to migrate our legacy monolith User Groups implementation to our new Microservices platform. The current implementation is rather bloated and complicated, so we're looking to stream line things a bit and hopefully make it the system a bit easier to use and maintain. The User Groups also double up as our permissions table.
The Monolith is all MySQL and has the following tables.
groups - Defines the groups, contains the groups_hierarchy_fk if the Group is part of a Hierarchy
groups_hierarchy - Defines the Group Hierarchy
groups_hierarchy_flat - It's not really a flat table, but is basically a linking table for the parent group and child group. It's meant to speed up searching the hierarchies, not the best structure tbh.
groups_users A linking table between Groups and Users, which shows which Users are in
which Groups. Currently, the most amount of users we have in a single Group is over 1 million, while the most Groups a single user belongs to is in the mid 30s
As part of the move to Microservices, we evaluate if the current data source (MySQL) makes sense or not. A decision has already been made to move out Users table from MySQL to MongoDB, as the schema-less design is something we really need for our users, as each client can define their user fields. We also no longer have a need for the permisisons part of the Legacy Groups implementation, as that is handled with a different implementation in our Microservices environment.
I was thinking about moving our Groups implementation to MongoDB too, as I think it'll work much better for Tree Structures and searching them. I have two potential designs, but since no-one of the team is really an expert in MongoDB, I figured I'd crowdsource some opinions of the design to see if it's any good.
First Option
This option basically has the full Group Hierarchy defined in a single document, which makes it much easier to see the structure when looking directly at the data and possibly faster for searching too (I'm yet to actually do a POC).
Groups
Rather simple document structure, simple define Group names with an id.
{
_id: ObjectId("631f927fdcec0ea55c5a51f0"),
name: "Default Users"
}
{
_id: ObjectId("631f9290baeb487ed61cb00d"),
name: "BE"
}
{
_id: ObjectId("631f92ae9fa0b625cf9aaf20"),
name: "Product"
}
{
_id: ObjectId("631f92b7d7f7d457d32936bf"),
name: "Finance"
}
{
_id: ObjectId("631f962cb605dc6ff44cad35"),
name: "Architecture"
}
{
_id: ObjectId("631f96db145359142fa99a6c"),
name: "Honey Badger"
}
{
_id: ObjectId("631f96eb8a9e751d6893a5f5"),
name: "Hatchery"
}
{
_id: ObjectId("631f96ffd16b8cf34fb54771"),
name: "Framework"
}
{
_id: ObjectId("631f971bf25da6fcf5724e83"),
name: "Vitality"
}
{
_id: ObjectId("631fa2f8c600e66b37d07c78"),
name: "Social Soccer Team",
}
Users
So instead of having a linking table between Groups and Users, we would simple embed the list of Groups directly into the Users document.
{
_id: ObjectId("631f92f0670a5688a2ed3432"),
username: "first.last#domain.com",
first_name: "First",
last_name: "Last",
// Other fields...
groups: [
ObjectId("631f927fdcec0ea55c5a51f0"), // Default User Group
ObjectId("631f971bf25da6fcf5724e83"), // Vitality
ObjectId("631fa2f8c600e66b37d07c78") // Social Soccer Team
]
}
Group Hierarchy
In this document, we define the name of the Hierarchy, as well as include the actual Hierarchy structure it's self. While I like the idea of having the full hierarchy structure in a single document, I wonder if it'll actually be better, since indexing it might end up be harder and therefore, the searching would be slower.
{
_id: ObjectId("631f9c588d7aace74e8781e4")
name: "Default Organization Chart"
groups: [
{
group_id: ObjectId("631f9290baeb487ed61cb00d"), // BE
groups: [
{
group_id: ObjectId("631f962cb605dc6ff44cad35"), // Architecture
groups: [
{
group_id: ObjectId("631f96ffd16b8cf34fb54771") // Framework
}, {
group_id: ObjectId("631f971bf25da6fcf5724e83") // Vitality
}
]
},{
group_id: ObjectId("631f96db145359142fa99a6c") // Honey Badger
},{
group_id: ObjectId("631f96eb8a9e751d6893a5f5") // Hatchery
}
]
}, {
group_id: ObjectId("631f92ae9fa0b625cf9aaf20") // Product
}, {
group_id: ObjectId("631f92b7d7f7d457d32936bf") // Finance
}
]
}
Option 2
This option for the most part was preferred by the team, as it would be easier to index and would make searching possibly faster. It's also inline with the tree structures on the MongoDB website, which means we can also start using the MongoDB Graph Lookup functionality
Group Hierarchy
We start with this document structure first this time, as it's the most simple one, as we're simply defining the name of a Group Hierarchy with an id.
{
_id: ObjectId("631f9c588d7aace74e8781e4")
name: "Default Organization Chart"
}
Groups
In the Groups document, we define a field to contain the id of the Group Hierarchy it belongs to (if any), as well as what the parent Group is via the id. If it's the top most level of the Group, then the id will be null.
{
_id: ObjectId("631f927fdcec0ea55c5a51f0"),
name: "Default Users",
group_heirarchy: null,
parent_id: null
}
{
_id: ObjectId("631f9290baeb487ed61cb00d"),
name: "BE",
group_heirarchy: ObjectId("631f9c588d7aace74e8781e4"),
parent_id: null
}
{
_id: ObjectId("631f92ae9fa0b625cf9aaf20"),
name: "Product",
group_heirarchy: ObjectId("631f9c588d7aace74e8781e4"),
parent_id: null
}
{
_id: ObjectId("631f92b7d7f7d457d32936bf"),
name: "Finance",
group_heirarchy: ObjectId("631f9c588d7aace74e8781e4"),
parent_id: null
}
{
_id: ObjectId("631f962cb605dc6ff44cad35"),
name: "Architecture",
group_heirarchy: ObjectId("631f9c588d7aace74e8781e4"),
parent_id: ObjectId("631f9290baeb487ed61cb00d") // BE
}
{
_id: ObjectId("631f96db145359142fa99a6c"),
name: "Honey Badger",
group_heirarchy: ObjectId("631f9c588d7aace74e8781e4"),
parent_id: ObjectId("631f9290baeb487ed61cb00d") // BE
}
{
_id: ObjectId("631f96eb8a9e751d6893a5f5"),
name: "Hatchery",
group_heirarchy: ObjectId("631f9c588d7aace74e8781e4"),
parent_id: ObjectId("631f9290baeb487ed61cb00d") // BE
}
{
_id: ObjectId("631f96ffd16b8cf34fb54771"),
name: "Framework",
group_heirarchy: ObjectId("631f9c588d7aace74e8781e4"),
parent_id: ObjectId("631f962cb605dc6ff44cad35") // Architecture
}
{
_id: ObjectId("631f971bf25da6fcf5724e83"),
name: "Vitality",
group_heirarchy: ObjectId("631f9c588d7aace74e8781e4"),
parent_id: ObjectId("631f962cb605dc6ff44cad35") // Architecture
}
{
_id: ObjectId("631fa2f8c600e66b37d07c78"),
name: "Social Soccer Team",
group_heirarchy: null,
parent_id: null
}
Users
The document structure for users would be the same as the previous option.
Thoughts
I think overall Option 2 is more than likely the best solution of the two, but are there any other designs that might work better?
From mongodb collection I want to get only one property like if I had a array of object structure
[
{
_id: 'property1',
name: 'property1',
value: 1
},
{
_id: 'property2',
name: 'property2',
value: 2
}
]
So here I don't want to get whole collection, instead I only just need name from all the documents in the collection
I've two mongo collections - File and Folder.
Both have some common fields like name, createdAt etc. I've a resources API that returns a response having items from both collections, with a type property added. type can be file or folder
I want to support pagination and sorting in this list, for example sort by createdAt. Is it possible with aggregation, and how?
Moving them to a container collection is not a preferred option, as then I have to maintain the container collection on each create/update/delete on either of the collection.
I'm using mongoose too, if it has got any utility function for this, or a plugin.
In this case, you can use $unionWith. Something like:
Folder.aggregate([
{ $project: { name: 1, createdAt: 1 } },
{
$unionWith: {
coll: "files", pipeline: [ { $project: { name: 1, createdAt: 1 } } ]
}
},
... // your sorting go here
])
Whats the best way of pushing data into an array that's nested several levels deep in arrays...
Here's an example document ( made up ) that's representative of the data in the document, except the real document has more schools with more classes and more students. Each 'thing' has a unique id :-
{
_id: 4353467865,
school : [
{
_id: 3425353,
name: 'school of rock',
class: [
{
_id: 3242342
name: 'Room1',
students: [
{ _id: 345456562, name: 'Kevin' }
]
},
{
// more classes with more students nested underneath
}
]
},
{
// more schools, classes, students...
}
]
}
Now I know the id of the document, school and class, and now I want to do an update $push into the students array a {_id: 234554363, name: 'Barry'} for that class.
How do I construct an update query for that?
Do the below 2 links solve your problem?
http://docs.mongodb.org/manual/reference/operator/update/positional/#nested-arrays
Update multi nested array in Mongodb
I'll answer my own question since I've been using Mongo for a while now.
first answer...
When designing Mongo stuff, I was super paranoid to try and get things in one doc if I could since that's the whole benefit. Reality is you don't have to be too paranoid. School and Class could be separate collections. It does depend a bit on how you are using them, but for the most I would do separate collections these days.
second answer...
I have needed to do nested arrays, however any level of nesting array can be separated out into non nested arrays with keys which is then trivial to update using mongo
so
school : [
{
_id: 3425353,
name: 'school of rock',
}
],
classes: [
{
_id: 3242342
school_id: 3425353,
name: 'Room1',
}
],
students: [
{ _id: 345456562, class_id: 3242342, name: 'Kevin' }
{ _id: 345456563, class_id: 3242342, name: 'Gilbert' }
]