MongoDB aggregate different collections - mongodb

i've been trying to do this aggregation since yesterday with no luck, hope you guys can give me some ideas. A little background, I have 2 collections, one for results and another for questions. Results is basically where people solve questions, so it can have 1 question or up to 99 if i'm not mistaken. This is the simplified schema:
Results Schema:
_id: ObjectId("6010664ac5c4f77f26f5d005")
questions: [
{
_id: ObjectId("5d2627c94bb703bfcc910763"),
correct: false
},
{
_id: ObjectId("5d2627c94bb703bfcc910764"),
correct: true
},
{
_id: ObjectId("5d2627c94bb703bfcc910765"),
correct: true
}
]
So, on this specific object, the user answered 3 questions and got 2 of them correct.
Questions Schema:
_id: ObjectId("5d2627c94bb703bfcc910763")
What i'm struggling to do is: for each element in all the questions schema, I have to check if the question was answered - i.e (check if there's an _id of questions array == _id on the Questions Schema, yes there can be multiple questions with the same _id as Questions Schema, but Questions Schema _id is unique). If that question was answered, I need to check if correct = true, if so, I add it to a correctAnswer variable, if not, wrongAnswer.
I've tried many things so far with conditions, group to get the $sum of correct and wrong answers, lookup to join both collections but so far, I can't even get to show just the aggregation result.
One of the things I tried (i was trying to do baby steps first) but as mentioned before, couldn't even get the result printed.
Result.aggregate([
$lookup: {
from: 'question',
localField: 'questions._id',
foreignField: '_id',
as: 'same'
},
But this gets me both collections combined, and 'same' comes as empty array, tried using match with also no luck.
I also did a $project to just get the information I wanted
$project: {
_id: 0,
questions: {
_id: 1,
correct: 1
}
},
Tried using $group:
$group: {
_id: "$_id",
$cond: {if: {"$correct": { $eq: true}}, then: {testField: {$sum: 1}}, else: {testField: 0}}
}
And as I said, i was just trying to do baby steps so it the testField was beeing manually set, also tried many other things from stackoverflow.
Would appreciate the help and sorry for the very long text, just wanted to put in some examples that I did and tried.
TLDR: Need to find a question from the Results Schema where _id matches an _id from the Questions Schema, if there is, check if correct: true or correct: false. Update Questions Schema accordingly with how many were correct and how many were wrong for each question from Questions Schema.
Example: newField: {correctAnswer: 4, wrongAnswer: 3} so in this case, there were 7 questions from the Result schema question array that matched an _id from Question Schema, 4 had correct: true and 3 had correct: false. Then it goes on like this for the rest of Question Schema

For a scenario where "$lookup" can't be used because the Question collection is in a different database, the Result collection may be used to generate output documents to update the Question collection.
Here's one way to do it.
db.Result.aggregate([
{
"$unwind": "$questions"
},
{
"$group": {
"_id": "$questions._id",
"correct": {
"$push": "$questions.correct"
}
}
},
{
"$project": {
"newField": {
"correctAnswers": {
"$size": {
"$filter": {
"input": "$correct",
"as": "bool",
"cond": "$$bool"
}
}
},
"wrongAnswers": {
"$size": {
"$filter": {
"input": "$correct",
"as": "bool",
"cond": { "$not": "$$bool" }
}
}
}
}
}
}
])
Try it on mongoplayground.net.

I don't know of a way to "$lookup" and update at the same time. There's probably a better way to do this, but the aggregation pipeline below creates the documents that could be used in a subsequent update. The pipeline correctly counts repeat questions by a single Result _id, in case someone keeps trying a question until they get it right. One possible issue is that if a question has no Result answers, then no "newField": { "correctAnswers": 0, "wrongAnswers": 0 } document is created.
db.Question.aggregate([
{
// lookup documents in Result that match _id
"$lookup": {
"from": "Result",
"localField": "_id",
"foreignField": "questions._id",
"as": "results"
}
},
{
// unwind everything
"$unwind": "$results"
},
{
// more of everything
"$unwind": "$results.questions"
},
{
// only keep answers that match question
"$match": {
"$expr": { "$eq": [ "$_id", "$results.questions._id" ] }
}
},
{
// reassemble and count correct/wrong answers
"$group": {
"_id": "$_id",
"correct": {
"$sum": {
"$cond": [ { "$eq": [ "$results.questions.correct", true ] }, 1, 0 ]
}
},
"wrong": {
"$sum": {
"$cond": [ { "$eq": [ "$results.questions.correct", false ] }, 1, 0 ]
}
}
}
},
{
// project what you want as output
"$project": {
newField: {
correctAnswers: "$correct",
wrongAnswers: "$wrong"
}
}
}
])
Try it on mongoplayground.net.

Related

How to create a mongo view providing data only for the latest as at date

In MSSQL the view can be easily created by running 2 queries:
The first one gets the latest date available
the second one extracts all records as at that date
How can I achieve the same results in Mongo? Mongo collection is Indexed by AsAtDate and Id.
Based on research, mongo only supports views created with a pipeline that doesn't support multiple queries.
Example:
db.getCollection('MyCollection').aggregate([
{ "$lookup":
{
"from": "CollectionAsAtDate",
"pipeline": [
{ "$sort": { "AsAtDate": -1 } },
{ "$limit": 1 },
{ "$project": { "_id": 0, "AsAtDate": 1 } }
],
"as": "latest"
}
},
{"$unwind":"$latest"},
{
"$match": {
$expr: {
{ $eq: [AsAtDate,"$latest.AsAtDate" ]}
}
}
}
])
This piece of code is the best solution, I have found so far but it is still not efficient.
It takes me 10+ seconds to get data from a collection by already indexed parameters!
The reason for that is just because here we are trying to find last date and attach it to all documents. After that, we are checking if AsAtDate == LastAsAtDate and remove if not. It works okay but it requires more resources rather than a simple search in an index!
How about moving the "sort-limit1" operation out from the sub-pipeline first? In this sense, you can ensure the index is leveraged and you are processing with only 1 document(i.e. the one with latest AsAtDate). You can avoid performing n-to-n $lookup with a 1-to-n $lookup.
db.collection.aggregate([
{
"$sort": {
"AsAtDate": -1
}
},
{
"$limit": 1
},
{
"$project": {
"_id": 0,
"AsAtDate": 1
}
},
{
"$lookup": {
"from": "collection",
"localField": "AsAtDate",
"foreignField": "AsAtDate",
"as": "latest"
}
},
{
"$unwind": "$latest"
},
{
"$replaceRoot": {
"newRoot": "$latest"
}
}
])
Here is the Mongo playground for your reference.

MongoDB aggregation to update more than one collection

was wondering if it's possible to create such aggregation that could modify three collection in one go?
For example imagine that we have these collections:
MasterCollection:
{
_id: ObjectId;
}
CollectionOne:
{
_id: ObjectId;
_masterCollectionId: ObjectId;
_parentId: ObjectId;
_title: String;
_displayTitle: String;
}
CollectionTwo:
{
_id: ObjectId;
_masterCollectionId: ObjectId;
_parentId: ObjectId;
_displayTitle: String;
}
CollectionThree:
{
_id: ObjectId;
_masterCollectionId: ObjectId
_parentId: ObjectId;
_displayTitle: String;
}
Aggregation to lookup:
[
{
$match: {
_id: new ObjectId("618552b66f82e69572e9bf10")
}
},
{
$lookup: {
from: "CollectionOne",
let: {
"collectionOneId": "$_id"
},
pipeline: [{
"$match": {
"$expr": {
"$eq": ["$_parentId", "$$collectionOneId"]
}
}
}, {
"$lookup": {
"from": "CollectionTwo",
"let": {
"collectionTwoId": "$_id"
},
"pipeline": [{
"$match": {
"$expr": {
"$eq": ["$_parentId", "$$collectionTwoId"]
}
}
}, {
"$lookup": {
"from": "CollectionThree",
"let": {
"collectionThreeId": "$_id"
},
"pipeline": [{
"$match": {
"$expr": {
"$eq": ["$_parentId", "$$collectionThreeId"]
}
}
}],
"as": "CollectionThree"
}
}],
"as": "CollectionTwo"
}
}],
as: "CollectionOne"
}
}]
With aggregation above I can get all children of MasterCollection record.
In similar way I could get CollectionOne record children.
But I am thinking if there could be a way to apply changes within pipeline for each record.
For example imagine situation that there is a need to change _masterCollectionId for CollectionOne record.
If we change that property we need to update all children records (CollectionTwo / CollectionThree).
Would need to update targeted CollectionOne record.
Would need to update CollectionTwo records that points to updated CollectionOne record.
Would need to update CollectionThree records that points to updated records of CollectionTwo.
Currently, programmatically I can do a loop and update everything, it kinda works. But I think there should be better solution, probably at database layer, using MongoDB aggregations.
Can MongoDB Aggregation achieve desired result ?
Any help would be appreciated.
Unfortunately I think what you're after is impossible using aggregation alone like you've explained.
The MongoDB documentation states that you can pass an aggregation pipeline into an updateMany() call where you can perform operations on the collection within the aggregation pipeline in MongoDB 4.2 or newer.
I haven't actually tried this technique personally, but it sounds like it might do the job. Based on the documentation, you'd need to run the updateMany() on each collection, which is still a huge improvement on what you've got already. Check it out here.
Something like this:
await CollectionOne.updateMany(
{ _masterClientId: <oldID> },
[
{ $set: { "_masterCollectionId": <newID> } }
]
);
// And repeat for CollectionTwo and CollectionThree

How to perform operations in pipeline in mongo $lookup [duplicate]

I have the following collections:
venue collection
{ "_id" : ObjectId("5acdb8f65ea63a27c1facf86"),
"name" : "ASA College - Manhattan Campus",
"addedBy" : ObjectId("5ac8ba3582c2345af70d4658"),
"reviews" : [
ObjectId("5acdb8f65ea63a27c1facf8b"),
ObjectId("5ad8288ccdd9241781dce698")
]
}
reviews collection
{ "_id" : ObjectId("5acdb8f65ea63a27c1facf8b"),
"createdAt" : ISODate("2018-04-07T12:31:49.503Z"),
"venue" : ObjectId("5acdb8f65ea63a27c1facf86"),
"author" : ObjectId("5ac8ba3582c2345af70d4658"),
"content" : "nice place",
"comments" : [
ObjectId("5ad87113882d445c5cbc92c8")
]
}
comment collection
{ "_id" : ObjectId("5ad87113882d445c5cbc92c8"),
"author" : ObjectId("5ac8ba3582c2345af70d4658"),
"comment" : "dcfdsfdcfdsfdcfdsfdcfdsfdcfdsfdcfdsfdcfdsfdcfdsf",
"review" : ObjectId("5acdb8f65ea63a27c1facf8b"),
"__v" : 0
}
author collection
{ "_id" : ObjectId("5ac8ba3582c2345af70d4658"),
"firstName" : "Bruce",
"lastName" : "Wayne",
"email" : "bruce#linkites.com",
"followers" : [ObjectId("5ac8b91482c2345af70d4650")]
}
Now the following populate query works fine
const venues = await Venue.findOne({ _id: id.id })
.populate({
path: 'reviews',
options: { sort: { createdAt: -1 } },
populate: [
{ path: 'author' },
{ path: 'comments', populate: [{ path: 'author' }] }
]
})
However, I want to achieve it with $lookup query, but it splits the venue when I am doing '$unwind' to the reviews... I want reviews in same array (like populate) and in same order...
I want to achieve following query with $lookup because author have followers field so I need to send field isFollow by doing $project which cannot be done using populate...
$project: {
isFollow: { $in: [mongoose.Types.ObjectId(req.user.id), '$followers'] }
}
There are a couple of approaches of course depending on your available MongoDB version. These vary from different usages of $lookup through to enabling object manipulation on the .populate() result via .lean().
I do ask that you read the sections carefully, and be aware that all may not be as it seems when considering your implementation solution.
MongoDB 3.6, "nested" $lookup
With MongoDB 3.6 the $lookup operator gets the additional ability to include a pipeline expression as opposed to simply joining a "local" to "foreign" key value, what this means is you can essentially do each $lookup as "nested" within these pipeline expressions
Venue.aggregate([
{ "$match": { "_id": mongoose.Types.ObjectId(id.id) } },
{ "$lookup": {
"from": Review.collection.name,
"let": { "reviews": "$reviews" },
"pipeline": [
{ "$match": { "$expr": { "$in": [ "$_id", "$$reviews" ] } } },
{ "$lookup": {
"from": Comment.collection.name,
"let": { "comments": "$comments" },
"pipeline": [
{ "$match": { "$expr": { "$in": [ "$_id", "$$comments" ] } } },
{ "$lookup": {
"from": Author.collection.name,
"let": { "author": "$author" },
"pipeline": [
{ "$match": { "$expr": { "$eq": [ "$_id", "$$author" ] } } },
{ "$addFields": {
"isFollower": {
"$in": [
mongoose.Types.ObjectId(req.user.id),
"$followers"
]
}
}}
],
"as": "author"
}},
{ "$addFields": {
"author": { "$arrayElemAt": [ "$author", 0 ] }
}}
],
"as": "comments"
}},
{ "$sort": { "createdAt": -1 } }
],
"as": "reviews"
}},
])
This can be really quite powerful, as you see from the perspective of the original pipeline, it really only knows about adding content to the "reviews" array and then each subsequent "nested" pipeline expression also only ever sees it's "inner" elements from the join.
It is powerful and in some respects it may be a bit clearer as all field paths are relative to the nesting level, but it does start that indentation creep in the BSON structure, and you do need to be aware of whether you are matching to arrays or singular values in traversing the structure.
Note we can also do things here like "flattening the author property" as seen within the "comments" array entries. All $lookup target output may be an "array", but within a "sub-pipeline" we can re-shape that single element array into just a single value.
Standard MongoDB $lookup
Still keeping the "join on the server" you can actually do it with $lookup, but it just takes intermediate processing. This is the long standing approach with deconstructing an array with $unwind and the using $group stages to rebuild arrays:
Venue.aggregate([
{ "$match": { "_id": mongoose.Types.ObjectId(id.id) } },
{ "$lookup": {
"from": Review.collection.name,
"localField": "reviews",
"foreignField": "_id",
"as": "reviews"
}},
{ "$unwind": "$reviews" },
{ "$lookup": {
"from": Comment.collection.name,
"localField": "reviews.comments",
"foreignField": "_id",
"as": "reviews.comments",
}},
{ "$unwind": "$reviews.comments" },
{ "$lookup": {
"from": Author.collection.name,
"localField": "reviews.comments.author",
"foreignField": "_id",
"as": "reviews.comments.author"
}},
{ "$unwind": "$reviews.comments.author" },
{ "$addFields": {
"reviews.comments.author.isFollower": {
"$in": [
mongoose.Types.ObjectId(req.user.id),
"$reviews.comments.author.followers"
]
}
}},
{ "$group": {
"_id": {
"_id": "$_id",
"reviewId": "$review._id"
},
"name": { "$first": "$name" },
"addedBy": { "$first": "$addedBy" },
"review": {
"$first": {
"_id": "$review._id",
"createdAt": "$review.createdAt",
"venue": "$review.venue",
"author": "$review.author",
"content": "$review.content"
}
},
"comments": { "$push": "$reviews.comments" }
}},
{ "$sort": { "_id._id": 1, "review.createdAt": -1 } },
{ "$group": {
"_id": "$_id._id",
"name": { "$first": "$name" },
"addedBy": { "$first": "$addedBy" },
"reviews": {
"$push": {
"_id": "$review._id",
"venue": "$review.venue",
"author": "$review.author",
"content": "$review.content",
"comments": "$comments"
}
}
}}
])
This really is not as daunting as you might think at first and follows a simple pattern of $lookup and $unwind as you progress through each array.
The "author" detail of course is singular, so once that is "unwound" you simply want to leave it that way, make the field addition and start the process of "rolling back" into the arrays.
There are only two levels to reconstruct back to the original Venue document, so the first detail level is by Review to rebuild the "comments" array. All you need to is to $push the path of "$reviews.comments" in order to collect these, and as long as the "$reviews._id" field is in the "grouping _id" the only other things you need to keep are all the other fields. You can put all of these into the _id as well, or you can use $first.
With that done there is only one more $group stage in order to get back to Venue itself. This time the grouping key is "$_id" of course, with all properties of the venue itself using $first and the remaining "$review" details going back into an array with $push. Of course the "$comments" output from the previous $group becomes the "review.comments" path.
Working on a single document and it's relations, this is not really so bad. The $unwind pipeline operator can generally be a performance issue, but in the context of this usage it should not really cause that much of an impact.
Since the data is still being "joined on the server" there is still far less traffic than the other remaining alternative.
JavaScript Manipulation
Of course the other case here is that instead of changing data on the server itself, you actually manipulate the result. In most cases I would be in favor of this approach since any "additions" to the data are probably best handled on the client.
The problem of course with using populate() is that whilst it may 'look like' a much more simplified process, it is in fact NOT A JOIN in any way. All populate() actually does is "hide" the underlying process of submitting multiple queries to the database, and then awaiting the results through async handling.
So the "appearance" of a join is actually the result of multiple requests to the server and then doing "client side manipulation" of the data to embed the details within arrays.
So aside from that clear warning that the performance characteristics are nowhere close to being on par with a server $lookup, the other caveat is of course that the "mongoose Documents" in the result are not actually plain JavaScript objects subject to further manipulation.
So in order to take this approach, you need to add the .lean() method to the query before execution, in order to instruct mongoose to return "plain JavaScript objects" instead of Document types which are cast with schema methods attached to the model. Noting of course that the resulting data no longer has access to any "instance methods" that would otherwise be associated with the related models themselves:
let venue = await Venue.findOne({ _id: id.id })
.populate({
path: 'reviews',
options: { sort: { createdAt: -1 } },
populate: [
{ path: 'comments', populate: [{ path: 'author' }] }
]
})
.lean();
Now venue is a plain object, we can simply process and adjust as needed:
venue.reviews = venue.reviews.map( r =>
({
...r,
comments: r.comments.map( c =>
({
...c,
author: {
...c.author,
isAuthor: c.author.followers.map( f => f.toString() ).indexOf(req.user.id) != -1
}
})
)
})
);
So it's really just a matter of cycling through each of the inner arrays down until the level where you can see the followers array within the author details. The comparison then can be made against the ObjectId values stored in that array after first using .map() to return the "string" values for comparison against the req.user.id which is also a string (if it is not, then also add .toString() on that ), since it is easier in general to compare these values in this way via JavaScript code.
Again though I need to stress that it "looks simple" but it is in fact the sort of thing you really want to avoid for system performance, as those additional queries and the transfer between the server and the client cost a lot in time of processing and even due to the request overhead this adds up to real costs in transport between hosting providers.
Summary
Those are basically your approaches you can take, short of "rolling your own" where you actually perform the "multiple queries" to the database yourself instead of using the helper that .populate() is.
Using the populate output, you can then simply manipulate the data in result just like any other data structure, as long as you apply .lean() to the query to convert or otherwise extract the plain object data from the mongoose documents returned.
Whilst the aggregate approaches look far more involved, there are "a lot" more advantages to doing this work on the server. Larger result sets can be sorted, calculations can be done for further filtering, and of course you get a "single response" to a "single request" made to the server, all with no additional overhead.
It is totally arguable that the pipelines themselves could simply be constructed based on attributes already stored on the schema. So writing your own method to perform this "construction" based on the attached schema should not be too difficult.
In the longer term of course $lookup is the better solution, but you'll probably need to put a little more work into the initial coding, if of course you don't just simply copy from what is listed here ;)

MongoDB lookup 3 collections and nesting the collections [duplicate]

I have the following collections:
venue collection
{ "_id" : ObjectId("5acdb8f65ea63a27c1facf86"),
"name" : "ASA College - Manhattan Campus",
"addedBy" : ObjectId("5ac8ba3582c2345af70d4658"),
"reviews" : [
ObjectId("5acdb8f65ea63a27c1facf8b"),
ObjectId("5ad8288ccdd9241781dce698")
]
}
reviews collection
{ "_id" : ObjectId("5acdb8f65ea63a27c1facf8b"),
"createdAt" : ISODate("2018-04-07T12:31:49.503Z"),
"venue" : ObjectId("5acdb8f65ea63a27c1facf86"),
"author" : ObjectId("5ac8ba3582c2345af70d4658"),
"content" : "nice place",
"comments" : [
ObjectId("5ad87113882d445c5cbc92c8")
]
}
comment collection
{ "_id" : ObjectId("5ad87113882d445c5cbc92c8"),
"author" : ObjectId("5ac8ba3582c2345af70d4658"),
"comment" : "dcfdsfdcfdsfdcfdsfdcfdsfdcfdsfdcfdsfdcfdsfdcfdsf",
"review" : ObjectId("5acdb8f65ea63a27c1facf8b"),
"__v" : 0
}
author collection
{ "_id" : ObjectId("5ac8ba3582c2345af70d4658"),
"firstName" : "Bruce",
"lastName" : "Wayne",
"email" : "bruce#linkites.com",
"followers" : [ObjectId("5ac8b91482c2345af70d4650")]
}
Now the following populate query works fine
const venues = await Venue.findOne({ _id: id.id })
.populate({
path: 'reviews',
options: { sort: { createdAt: -1 } },
populate: [
{ path: 'author' },
{ path: 'comments', populate: [{ path: 'author' }] }
]
})
However, I want to achieve it with $lookup query, but it splits the venue when I am doing '$unwind' to the reviews... I want reviews in same array (like populate) and in same order...
I want to achieve following query with $lookup because author have followers field so I need to send field isFollow by doing $project which cannot be done using populate...
$project: {
isFollow: { $in: [mongoose.Types.ObjectId(req.user.id), '$followers'] }
}
There are a couple of approaches of course depending on your available MongoDB version. These vary from different usages of $lookup through to enabling object manipulation on the .populate() result via .lean().
I do ask that you read the sections carefully, and be aware that all may not be as it seems when considering your implementation solution.
MongoDB 3.6, "nested" $lookup
With MongoDB 3.6 the $lookup operator gets the additional ability to include a pipeline expression as opposed to simply joining a "local" to "foreign" key value, what this means is you can essentially do each $lookup as "nested" within these pipeline expressions
Venue.aggregate([
{ "$match": { "_id": mongoose.Types.ObjectId(id.id) } },
{ "$lookup": {
"from": Review.collection.name,
"let": { "reviews": "$reviews" },
"pipeline": [
{ "$match": { "$expr": { "$in": [ "$_id", "$$reviews" ] } } },
{ "$lookup": {
"from": Comment.collection.name,
"let": { "comments": "$comments" },
"pipeline": [
{ "$match": { "$expr": { "$in": [ "$_id", "$$comments" ] } } },
{ "$lookup": {
"from": Author.collection.name,
"let": { "author": "$author" },
"pipeline": [
{ "$match": { "$expr": { "$eq": [ "$_id", "$$author" ] } } },
{ "$addFields": {
"isFollower": {
"$in": [
mongoose.Types.ObjectId(req.user.id),
"$followers"
]
}
}}
],
"as": "author"
}},
{ "$addFields": {
"author": { "$arrayElemAt": [ "$author", 0 ] }
}}
],
"as": "comments"
}},
{ "$sort": { "createdAt": -1 } }
],
"as": "reviews"
}},
])
This can be really quite powerful, as you see from the perspective of the original pipeline, it really only knows about adding content to the "reviews" array and then each subsequent "nested" pipeline expression also only ever sees it's "inner" elements from the join.
It is powerful and in some respects it may be a bit clearer as all field paths are relative to the nesting level, but it does start that indentation creep in the BSON structure, and you do need to be aware of whether you are matching to arrays or singular values in traversing the structure.
Note we can also do things here like "flattening the author property" as seen within the "comments" array entries. All $lookup target output may be an "array", but within a "sub-pipeline" we can re-shape that single element array into just a single value.
Standard MongoDB $lookup
Still keeping the "join on the server" you can actually do it with $lookup, but it just takes intermediate processing. This is the long standing approach with deconstructing an array with $unwind and the using $group stages to rebuild arrays:
Venue.aggregate([
{ "$match": { "_id": mongoose.Types.ObjectId(id.id) } },
{ "$lookup": {
"from": Review.collection.name,
"localField": "reviews",
"foreignField": "_id",
"as": "reviews"
}},
{ "$unwind": "$reviews" },
{ "$lookup": {
"from": Comment.collection.name,
"localField": "reviews.comments",
"foreignField": "_id",
"as": "reviews.comments",
}},
{ "$unwind": "$reviews.comments" },
{ "$lookup": {
"from": Author.collection.name,
"localField": "reviews.comments.author",
"foreignField": "_id",
"as": "reviews.comments.author"
}},
{ "$unwind": "$reviews.comments.author" },
{ "$addFields": {
"reviews.comments.author.isFollower": {
"$in": [
mongoose.Types.ObjectId(req.user.id),
"$reviews.comments.author.followers"
]
}
}},
{ "$group": {
"_id": {
"_id": "$_id",
"reviewId": "$review._id"
},
"name": { "$first": "$name" },
"addedBy": { "$first": "$addedBy" },
"review": {
"$first": {
"_id": "$review._id",
"createdAt": "$review.createdAt",
"venue": "$review.venue",
"author": "$review.author",
"content": "$review.content"
}
},
"comments": { "$push": "$reviews.comments" }
}},
{ "$sort": { "_id._id": 1, "review.createdAt": -1 } },
{ "$group": {
"_id": "$_id._id",
"name": { "$first": "$name" },
"addedBy": { "$first": "$addedBy" },
"reviews": {
"$push": {
"_id": "$review._id",
"venue": "$review.venue",
"author": "$review.author",
"content": "$review.content",
"comments": "$comments"
}
}
}}
])
This really is not as daunting as you might think at first and follows a simple pattern of $lookup and $unwind as you progress through each array.
The "author" detail of course is singular, so once that is "unwound" you simply want to leave it that way, make the field addition and start the process of "rolling back" into the arrays.
There are only two levels to reconstruct back to the original Venue document, so the first detail level is by Review to rebuild the "comments" array. All you need to is to $push the path of "$reviews.comments" in order to collect these, and as long as the "$reviews._id" field is in the "grouping _id" the only other things you need to keep are all the other fields. You can put all of these into the _id as well, or you can use $first.
With that done there is only one more $group stage in order to get back to Venue itself. This time the grouping key is "$_id" of course, with all properties of the venue itself using $first and the remaining "$review" details going back into an array with $push. Of course the "$comments" output from the previous $group becomes the "review.comments" path.
Working on a single document and it's relations, this is not really so bad. The $unwind pipeline operator can generally be a performance issue, but in the context of this usage it should not really cause that much of an impact.
Since the data is still being "joined on the server" there is still far less traffic than the other remaining alternative.
JavaScript Manipulation
Of course the other case here is that instead of changing data on the server itself, you actually manipulate the result. In most cases I would be in favor of this approach since any "additions" to the data are probably best handled on the client.
The problem of course with using populate() is that whilst it may 'look like' a much more simplified process, it is in fact NOT A JOIN in any way. All populate() actually does is "hide" the underlying process of submitting multiple queries to the database, and then awaiting the results through async handling.
So the "appearance" of a join is actually the result of multiple requests to the server and then doing "client side manipulation" of the data to embed the details within arrays.
So aside from that clear warning that the performance characteristics are nowhere close to being on par with a server $lookup, the other caveat is of course that the "mongoose Documents" in the result are not actually plain JavaScript objects subject to further manipulation.
So in order to take this approach, you need to add the .lean() method to the query before execution, in order to instruct mongoose to return "plain JavaScript objects" instead of Document types which are cast with schema methods attached to the model. Noting of course that the resulting data no longer has access to any "instance methods" that would otherwise be associated with the related models themselves:
let venue = await Venue.findOne({ _id: id.id })
.populate({
path: 'reviews',
options: { sort: { createdAt: -1 } },
populate: [
{ path: 'comments', populate: [{ path: 'author' }] }
]
})
.lean();
Now venue is a plain object, we can simply process and adjust as needed:
venue.reviews = venue.reviews.map( r =>
({
...r,
comments: r.comments.map( c =>
({
...c,
author: {
...c.author,
isAuthor: c.author.followers.map( f => f.toString() ).indexOf(req.user.id) != -1
}
})
)
})
);
So it's really just a matter of cycling through each of the inner arrays down until the level where you can see the followers array within the author details. The comparison then can be made against the ObjectId values stored in that array after first using .map() to return the "string" values for comparison against the req.user.id which is also a string (if it is not, then also add .toString() on that ), since it is easier in general to compare these values in this way via JavaScript code.
Again though I need to stress that it "looks simple" but it is in fact the sort of thing you really want to avoid for system performance, as those additional queries and the transfer between the server and the client cost a lot in time of processing and even due to the request overhead this adds up to real costs in transport between hosting providers.
Summary
Those are basically your approaches you can take, short of "rolling your own" where you actually perform the "multiple queries" to the database yourself instead of using the helper that .populate() is.
Using the populate output, you can then simply manipulate the data in result just like any other data structure, as long as you apply .lean() to the query to convert or otherwise extract the plain object data from the mongoose documents returned.
Whilst the aggregate approaches look far more involved, there are "a lot" more advantages to doing this work on the server. Larger result sets can be sorted, calculations can be done for further filtering, and of course you get a "single response" to a "single request" made to the server, all with no additional overhead.
It is totally arguable that the pipelines themselves could simply be constructed based on attributes already stored on the schema. So writing your own method to perform this "construction" based on the attached schema should not be too difficult.
In the longer term of course $lookup is the better solution, but you'll probably need to put a little more work into the initial coding, if of course you don't just simply copy from what is listed here ;)

Inner Join on two Fields

I have the following schemas
var User = mongoose.Schema({
email:{type: String, trim: true, index: true, unique: true, sparse: true},
password: String,
name:{type: String, trim: true, index: true, unique: true, sparse: true},
gender: String,
});
var Song = Schema({
track: { type: Schema.Types.ObjectId, ref: 'track' },//Track can be deleted
author: { type: Schema.Types.ObjectId, ref: 'user' },
url: String,
title: String,
photo: String,
publishDate: Date,
views: [{ type: Schema.Types.ObjectId, ref: 'user' }],
likes: [{ type: Schema.Types.ObjectId, ref: 'user' }],
collaborators: [{ type: Schema.Types.ObjectId, ref: 'user' }],
});
I want to select all users (without the password value) , but I want each user will have all the songs where he is the author or one of the collaborators and the was published in the last 2 weeks.
What is the best strategy perform this action (binding between the user.id and song .collaborators) ? Can it be done in one select?
It's very possible in one request, and the basic tool for this with MongoDB is $lookup.
I would think this actually makes more sense to query from the Song collection instead, since your criteria is that they must be listed in one of two properties on that collection.
Optimal INNER Join - Reversed
Presuming the actual "model" names are what is listed above:
var today = new Date.now(),
oneDay = 1000 * 60 * 60 * 24,
twoWeeksAgo = new Date(today - ( oneDay * 14 ));
var userIds; // Should be assigned as an 'Array`, even if only one
Song.aggregate([
{ "$match": {
"$or": [
{ "author": { "$in": userIds } },
{ "collaborators": { "$in": userIds } }
],
"publishedDate": { "$gt": twoWeeksAgo }
}},
{ "$addFields": {
"users": {
"$setIntersection": [
userIds,
{ "$setUnion": [ ["$author"], "$collaborators" ] }
]
}
}},
{ "$lookup": {
"from": User.collection.name,
"localField": "users",
"foreignField": "_id",
"as": "users"
}},
{ "$unwind": "$users" },
{ "$group": {
"_id": "$users._id",
"email": { "$first": "$users.email" },
"name": { "$first": "$users.name" },
"gender": { "$first": "$users.gender" },
"songs": {
"$push": {
"_id": "$_id",
"track": "$track",
"author": "$author",
"url": "$url",
"title": "$title",
"photo": "$photo",
"publishedDate": "$publishedDate",
"views": "$views",
"likes": "$likes",
"collaborators": "$collaborators"
}
}
}}
])
That to me is the most logical course as long as it's an "INNER JOIN" you want from the results, meaning that "all users MUST have a mention on at least one song" in the two properties involved.
The $setUnion takes the "unique list" ( ObjectId is unique anyway ) of combining those two. So if an "author" is also a "collaborator" then they are only listed once for that song.
The $setIntersection "filters" the list from that combined list to only those that were specified in the query condition. This removes any other "collaborator" entries that would not have been in the selection.
The $lookup does the "join" on that combined data to get the users, and the $unwind is done because you want the User to be the main detail. So we basically reverse the "array of users" into "array of songs" in the result.
Also, since the main criteria is from Song, then it makes sense to query from that collection as the direction.
Optional LEFT Join
Doing this the other way around is where the "LEFT JOIN" is wanted, being "ALL Users" regardless if there are any associated songs or not:
User.aggregate([
{ "$lookup": {
"from": Song.collection.name,
"localField": "_id",
"foreignField": "author",
"as": "authors"
}},
{ "$lookup": {
"from": Song.collection.name,
"localField": "_id",
"foreignField": "collaborators",
"as": "collaborators"
}},
{ "$project": {
"email": 1,
"name": 1,
"gender": 1,
"songs": { "$setUnion": [ "$authors", "$collaborators" ] }
}}
])
So the listing of the statement "looks" shorter, but it is forcing "two" $lookup stages in order to obtain results for possible "authors" and "collaborators" rather than one. So the actual "join" operations can be costly in execution time.
The rest is pretty straightforward in applying the same $setUnion but this time the the "result arrays" rather than the original source of the data.
If you wanted similar "query" conditions to above on the "filter" for the "songs" and not the actual User documents returned, then for LEFT Join you actually $filter the array content "post" $lookup:
User.aggregate([
{ "$lookup": {
"from": Song.collection.name,
"localField": "_id",
"foreignField": "author",
"as": "authors"
}},
{ "$lookup": {
"from": Song.collection.name,
"localField": "_id",
"foreignField": "collaborators",
"as": "collaborators"
}},
{ "$project": {
"email": 1,
"name": 1,
"gender": 1,
"songs": {
"$filter": {
"input": { "$setUnion": [ "$authors", "$collaborators" ] },
"as": "s",
"cond": {
"$and": [
{ "$setIsSubset": [
userIds
{ "$setUnion": [ ["$$s.author"], "$$s.collaborators" ] }
]},
{ "$gte": [ "$$s.publishedDate", oneWeekAgo ] }
]
}
}
}
}}
])
Which would mean that by LEFT JOIN Conditions, ALL User documents are returned but the only ones which will contain any "songs" will be those that met the "filter" conditions as being part of the supplied userIds. And even those users which were contained in the list will only show those "songs" within the required range for publishedDate.
The main addition within the $filter is the $setIsSubset operator, which is a short way of comparing the supplied list in userIds to the "combined" list from the two fields present in the document. Noting here the the "current user" already had to be "related" due to the earlier conditions of each $lookup.
MongoDB 3.6 Preview
A new "sub-pipeline" syntax available for $lookup from the MongoDB 3.6 release means that rather than "two" $lookup stages as shown for the LEFT Join variant, you can in fact structure this as a "sub-pipeline", which also optimally filters content before returning results:
User.aggregate([
{ "$lookup": {
"from": Song.collection.name,
"let": {
"user": "$_id"
},
"pipeline": [
{ "$match": {
"$or": [
{ "author": { "$in": userIds } },
{ "collaborators": { "$in": userIds } }
],
"publishedDate": { "$gt": twoWeeksAgo },
"$expr": {
"$or": [
{ "$eq": [ "$$user", "$author" ] },
{ "$setIsSubset": [ ["$$user"], "$collaborators" ]
]
}
}}
],
"as": "songs"
}}
])
And that is all there is to it in that case, since $expr allows usage of the $$user variable declared in "let" to be compared with each entry in the song collection to select only those that are matching in addition to the other query criteria. The result being only those matching songs per user or an empty array. Thus making the whole "sub-pipeline" simply a $match expression, which is pretty much the same as additional logic as opposed to fixed local and foreign keys.
So you could even add a stage to the pipeline following $lookup to filter out any "empty" array results, making the overall result an INNER Join.
So personally I would go for the first approach when you can and only use the second approach where you need to.
NOTE: There are a couple of options here that don't really apply as well. The first being a special case of $lookup + $unwind + $match coalescence in which whilst the basic case applies to the initial INNER Join example it cannot be applied with the LEFT Join Case.
This is because in order for a LEFT Join to be obtained, the usage of $unwind must be implemented with preserveNullAndEmptyArrays: true, and this breaks the rule of application in that the unwinding and matching cannot be "rolled up" within the $lookup and applied to the foreign collection "before" returning results.
Hence why it is not applied in the sample and we use $filter on the returned array instead, since there is no optimal action that can be applied to the foreign collection "before" the results are returned, and nothing stopping all results for songs matching on simply the foreign key from returning. INNER Joins are of course different.
The other case is .populate() with mongoose. The most important distinction being that .populate() is not a single request, but just a programming "shorthand" for actually issuing multiple queries. So at any rate, there would actually be multiple queries issued and always requiring ALL results in order to apply any filtering.
Which leads to the limitation on where the filtering is actually applied, and generally means that you cannot really implement "paging" concepts when you utilize "client side joins" that require conditions to be applied on the foreign collection.
There are some more details on this on Querying after populate in Mongoose, and an actual demonstration of how the basic functionality can be wired in as a custom method in mongoose schema's anyway, but actually using the $lookup pipeline processing underneath.