I have a relational SQL DB that's being changed to MongoDB. In SQL there are 3 tables that are relevant: Farm, Division, Wombat (names and purpose changed for this question). There's also a Farmer table which is the equivalent of a users table.
Using Mongoose I've come up with this new schema:
var mongoose = require('mongoose');
var farmSchema = new mongoose.Schema({
// reference to the farmer collection's _id key
farmerId: mongoose.Schema.ObjectId,
name: String, // name of farm
division: [{
divisionId: mongoose.Schema.ObjectId,
name: String,
wombats: [{
wombatId: mongoose.Schema.ObjectId,
name: String,
weight: Number
}]
}]
});
Each of the (now) nested collections has a unique field in it. This will allow me to use Ajax to send just the uniqueId and the weight (for example) to adjust that value instead of updating the entire document when only the weight changes.
This feels like an incorrect SQL adaptation for MongoDB. Is there a better way to do this?
In general, I believe that people tend to embed way too much when using MongoDB.
The most important argument is that having different writers to the same objects makes things a lot more complicated. Working with arrays and embedded objects can be tricky and some modifications are impossible, for instance because there's no positional operator matching in nested arrays.
For your particular scenario, take note that unique array keys might not behave as expected, and that behavior might change in future releases.
It's often desirable to opt for a simple SQL-like schema such as
Farm {
_id : ObjectId("...")
}
Division {
_id : ObjectId("..."),
FarmId : ObjectId("..."),
...
}
Wombat {
_id : ObjectId("..."),
DivisionId : ObjectId("..."),
...
}
Whether embedding is the right approach or not very much depends on usage patterns, data size, concurrent writes, etc. - a key difference to SQL is that there is no one right way to model 1:n or n:n relationships, so you'll have to carefully weigh the pros and cons for each scenario. In my experience, having a unique ID is a pretty strong indicator that the document should be a 'first-class citizen' and have its own collection.
Related
I am currently building an app with Meteor and MongoDB. I have a 3 level document structure with array in array:
{
_id: "shtZFiTeHrPKyJ8vR",
description: "Some title",
categories: [{
id: "shtZFiTeHrPKyJ8vR",
name: "Foo",
options: [{
id: "shtZFiTeHrPKyJ8vR",
name: "bar",
likes: ["abc", "bce"]
}]
}]
}
Now, the document could be manipulated at any level. Means:
description could be changed
categories can be added / removed / renamed
options can be added / removed / renamed
users can like options, so they must be added or removed
1 and 2 is quite easy. It is also relatively easy to add or remove a new option:
MyCollection.update({ _id: id, "categories.id": categoryId }, {
$push: {
"categories.$.options": {
id: Random.id
name: optionName
}
}
});
But manipulating the options hash requires to do that on javascript objects. That means I first need to find my document, iterate over the options and then write them back.
At least that's what I am doing right now. But I don't like that approach.
What I was thinking about is splitting the collection, at least to put the likes into it's own collection referencing the origin document.
Or is there another way? I don't really like both of my possible solutions.
For this kind of query one would normally use a the Mongo position operator. Although from the docs.
Nested Arrays
The positional $ operator cannot be used for queries
which traverse more than one array, such as queries that traverse
arrays nested within other arrays, because the replacement for the $
placeholder is a single value
Thus the only way to natively do what you want is by using specific indexes.
db.test.update({},{$pull:{"categories.0.options.0.likes":"abc"}})
Unfortunately Mongo does not allow to easily get the index of a match nested document.
I would normally say that once your queries become that difficult it's probably a good idea to revisit the way you store data. Also with that many arrays to which you will be pushing data, Mongo will probably be relocating a lot of documents. This is definitely something that you want to minimize.
So at this point you will need to separate your data out into different documents and even collections.
Your first documents would look like this:
{
_id: "shtZFiTeHrPKyJ8vR",
description: "Some title",
categories: [{
id: "shtZFiTeHrPKyJ8vR",
name: "Foo",
options: ["shtZFiTeHrPKyJ8vR"]
}]
}
This way you can easily add/remove options as you mentioned in your question. You would then need a second collection with documents that represent each option.
{
_id: "shtZFiTeHrPKyJ8vR",
name: "bar",
likes: ["abc", "bce"]
}
You can learn more about references here. This is similar to what you mentioned in your comment. The benefit of this is that you are already reducing the potential amount of relocation. Depending on how you use your data you may even be reducing network usage.
Now doing updates on the likes is easy.
MyCollection.update({ _id: id}, {
$push: {likes: "value"}
});
This does, however, require you to make two queries to the db. Although on the flip side you do a lot less on the client side and a lot less bandwidth is used.
Some other questions you need to ask yourself is if that depth of nesting is really needed. There might be an easier way to go about achieving your goal that doesn't require it to become so complicated.
I have two collections in my mongodb database :- Users , Clubs
User Schema :-
var UserSchema = new Schema({
Name: {
type: String ,
required: true
},
Clubs: [
{type: mongoose.Schema.Types.ObjectId, ref: 'Club'}
]});
Now when a user joins a club , i update the club array . But I also frequently need to fetch all the users for a particular club . Therefore I am creating the club schema as :-
var ClubSchema = new Schema({
clubName : {
type: String ,
unique: true ,
required: true
},
members : [
{type: mongoose.Schema.Types.ObjectId,
ref: 'User' ,
default: []
} ]});
My question is : Is this the right way to do so , or should I maintain this club information at the User Collection only ? Maybe I need to optimize the query related to fetching all the Users belonging to a Club.
It's quite hard to say what's "the right way" to be honest, as this is most likely case by case depending on your application queries and architecture.
I have seen some people do as you designed above; Solving many-to-many relationships by using reference in both collection.
This would work for your case queries above:
db.user.find({"clubs": ObjectId("000000")}); #Find all users belonging to certain club.
db.club.find({"users": ObjectId("111111")}); # Find all clubs where a user belong to.
With two indexes on:
db.user.ensureIndex({"clubs": 1});
db.club.ensureIndex({"users": 1});
Though this may reduce the overall consistency. i.e. when you delete a club, you need to update the other affected documents as well. While in the process of the mentioned update, your documents may not be up-to-date.
If you don't have mongodb on replicas, not accessing the db from distributed systems and you think this problem is not a big issue, then go for this design for the ease of query.
If you think the consistency mentioned above is a deal-breaker,
then go with inserting only clubs reference in users collection.
This is probably the most common design, and the one listed in
mongodb official site.
http://docs.mongodb.org/manual/tutorial/model-referenced-one-to-many-relationships-between-documents/
I would suggest optimising your query first, before choosing to add complexities in your update/insert (Because you have to update all related docs by having ref in Users and Clubs).
Hope this helps.
I have a question on best practises or ideal way how I should store the data in the database. As an example I have a Site that has a Country assigned.
Table Countries: id|name|alpha2
Table Sites: id|countryId|name
Each Site has a reference to the country ID.
I would like to create a new website using Meteor and its mongodb and was wondering how I should store the objects. Do I create a colleciton "countries" and "sites" and use the country _id to as a reference? Then resolve the references using transform?
Looking at SimpleSchema I came up with the following:
Schemas.Country = new SimpleSchema ({
name: {
type: String
},
alpha2: {
type: String,
max: 2
}
});
Schemas.Site = new SimpleSchema({
name: {
type: String,
label: "Site Name"
},
country: {
type: Schemas.Country
}
});
Countries = new Meteor.Collection("countries");
Countries.attachSchema(Schemas.Country);
Sites = new Meteor.Collection("sites");
Sites.attachSchema(Schemas.Site);
I was just wondering how this is then stored in the db. As I have 2 collections but inside the sites collection I do have defined country objects as well. What if a country changes its alpha2 code (very unlikely)?
Also this would continue where I have a collection called "conditions". Each condition will have a Site defined. I could now define the whole Site object into the condition object. What if the Sitename changes? Would I need to manually change it in all condition objects?
This confuses me a bit. I am very thankful for all your thoughts.
The challenge with Meteor is that its tightly bound to Mongo, which is not good to built OLTP app that require normalized DB design. Mongo is good for OLAP kind of apps which fall in WORM (Write Once Read Many) category. I would like to see Meteor supporting OrientDB as they do Mongo.
There can be two approaches:
Normalize the DB as we do in RDBMS and then retrieve data by hitting
data multiple times. Here is a good article explaining this approach - reactive joins in meteor.
Joins in
Meteor
are suggested in future. You can also try Meteor packages - publish
composite or
publish with
relations
Keep data de-normalized at least partially (for 1-N relation you can
embed things in document, for N-N relation you may having separate
collection). For instance, 'Student' can be embedded in 'Class' as
student will never be in more than 1 class, but to relate 'Student'
and 'Subject', they can be in different collections (N-N relation -
student will have more than one subject and each subject will be
taken by more than one student). For fetching N-N relation again you
can use the same approach that is mentioned point above.
I am not able to give you exact code example, but I hope it helps.
Background
I am storing table rows as MongoDb documents, with each column having a name. Let's say table has these columns of interest: Identifier, Person, Date, Count. The MongoDb document also has some extra fields separate from the table data, represented by timestamp. Columns are not fixed (which is why I use schema-free database to store them in the first place).
There will be need to do various complex, but so far unspecified queries. I am not very concerned about performance, though query performance may conceivably become a bottleneck. Once inserted, documents will not be modifed (a new document with same Identifier will be created instead), and insertions are not very frequent (let's say, 1000 new MongoDb documents per day). So amount of data will steadily grow over time.
Example
The straight-forward approach is having a collection of MongoDb documents like:
{
_id: XXXX,
insertDate: ISODate("2012-10-15T21:26:17Z"),
flag: true,
data: {
Identifier: "AB002",
Person: "John002",
Date: ISODate("2013-11-16T21:26:17Z"),
Count: 1
}
}
Now I have seen an alternative approach (for example in accepted answer of this question), using array with two fields per object:
{
_id: XXXX,
insertDate: ISODate("2012-10-15T21:26:17Z"),
flag: true,
data: [
{ field: "Identifier", value: "AB002" },
{ field: "Person", value: "John001" },
{ field: "Date", value: ISODate("2013-11-16T21:26:17Z") },
{ field: "Count", value: 1 }
]
}
Questions
Does the 2nd approach make any sense at all?
If yes, then how to choose which to use? Especially, are there some specific kinds of queries which are easy/cheap with one approach, hard/costly with another? Any "rules of thumb" on which way to go, or pro-con lists for both? Example real-life cases of one aproach being inconvenient would be especially valuable.
In your specific example the First version is a lot more appropriate and simple. You have to think in terms of how you would query your document.
It is a lot simpler to query your database like this: db.collection.find({"data.Identifier": "AB002"})
Although I'm not 100% sure why you even need the inner document. Why can't structure your document like:
{
_id: "AB002",
insertDate: ISODate("2012-10-15T21:26:17Z"),
flag: true,
Person: "John002",
Date: ISODate("2013-11-16T21:26:17Z"),
Count: 1
}
Pros of first example:
Simple to query
Enforces unique keys, but your data won't have two columns with the same name anyway
I would assume mongoDB would generate better query plans because the structure is a lot more simple (haven't tested)
Pros of second example:
Allows multiple entries with the same key/field, but I don't feel that is useful in your case
A single index on the array can be used for all of its entries regardless of their field name
I don't think that the situation in the other example here and yours are the same. In the other example, they're creating a list of items with one of two answers, which would be more appropriately in an array, and the goal is to return a list of subdocuments that match the criteria. In your example, you're really just describing an object since they all hold different types of information, and you won't need to retrieve searchable bits of the subdocuments.
I'm struggling with a large design choice for my applications' mongo collections and mongoose schemas.
My applications calls for two account types: Students and Teachers.
The only similarity between the two account types is that they both require the fields: firstName, lastName, email, and password. Other than that, they are different (teachers have "assignments", "tests", students have "homework", etc.)
I have pondered my options extensively, and considered the following design choices:
Use mongoose-schema-extend, and create an "abstract" schema for
all accounts. Then, extend this schema to create the Teacher and
Student schemas. This implies two collections, and therefore some
redundant fields. There are also issues with logging in and account creation (checking to see if the email used to log in is a student email or teacher email, etc.)
Create one collection "accounts", and add a type field to
indicate if the account is a "student" or a "teacher". This implies
that entries in the "accounts" collection will be dissimilar. This
also requires that I have two mongoose schemas for a single
collection.
Create an "accounts" collection, have a "type" field and an "accountId" field. In addition to a "student" collection and a "teacher" collection -- the "type" field will indicate which collection the student-specific or teacher-specific fields reside within, and the "accountId" field will indicate exactly which entry the account is matched with.
I appreciate all input, criticism or suggestions.
I've been down a similar road and I eventually landed on a mix of option 1 and 2.
mongoose-schema-extend simply modifies the prototype of Schema with an #extend() method which when invoked performs a deep copy of the passed schema. Most helpful. However, you can control which collection mongoose saves to in MongoDB by adding a collections property to the Schema:
var schema = new Schema({
foo: String,
bar: Boolean
}, { collection: "FooBarBaz" });
Remember: Mongoose understands the concept of a Schema but MongoDB does not. This means you can store dissimilar data and use your custom business logic to control the mess. With that said, you can create a base model called User, force mongoose to use the same collection by using the collection option and then extend off this base model to make your Teachers and Students models.
Make sure you add a type flag in the base model as you suggested in option 2. Not only is this convenient for quick lookups, but it will be critical when working commando with raw MongoDB data.
#jibsales has an excellent solution.
One more solution to consider is using Population with references http://mongoosejs.com/docs/populate.html from the Users collection to the Student and Teacher collections. Some benefits are:
Entries in each of the three collections (Users, Teachers, Students)
are similar in storage.
Allows you to obtain the fields for the "User" independently of
obtaining the fields for the referenced collection.
This would require that the schema is modified before an instance is created (and a model is created from the schema), where refType is the desired collection:
var userSchema = new Schema({
_id : Number,
name : String,
age : Number,
stories : [{ type: Schema.Types.ObjectId, ref: refType}]
});