Mongodb many to many relations among sub-documents - mongodb

Question

TL;DR: Imagine first $match stage gave you several documents, but you want to refine them inside, like $redact does. But the problem is your sub-documents have relations, and you want to make $where like checks among them. How can one accomplish that? I cannot $unwind, because it is causing performance problems, (1.5 mb of document with 5 times 1000 array length, a single unwind is causing 1000x~1mb documents).
My schema looks like:
{
userName: "user44",
userID: "44",
posts : [
...
{
title : "post1",
id : "123"
...
},
{
title : "post2",
id : "124"
...
},
...
],
comments: [
...
{
id: "1910",
postId : "123",
commentTitle : "comment1",
comment : "some comment",
user: "user13"
},
{
id: "1911",
postId : "124",
title : "comment2",
commentTitle : "some comment",
user: "user22"
},
{
id: "1912",
postId : "124",
title : "comment2",
commentTitle : "some comment",
user: "user22"
},
...
],
commentUpvotes: [
...
{
id : 12,
commentId : "1910",
upvotedBy: "user91",
upvoteDate: 1000,
},
{
id: 13,
commentId : "1910",
upvotedBy: "user92",
upvoteDate: 2000
},
{
id: 14,
commentId : "1911",
upvotedBy: "user92",
upvoteDate: 2100
},
...
]
}
Although this has nothing to do with my database, original schema is exactly as above. So, this example above is a user collection, where I store posts of the user; comments that made to the posts by other users, commentUpvotes to store information about who upvoted. Don't think about the logic of its design & contents; I made them up, and please don't suggest any other schema.
Question: I am looking a way to find posts and comments which has upvoted after a specific date, such
db.users.find("commentUpvotes.upvoteDate" : {$gte:0})
and result:
{
"_id" : ObjectId("539065d3cd0f2aac5f55778e"),
"posts" : [
{
title : "post1",
id : "123"
...
},
{
title : "post2",
id : "124"
...
},
],
"comments" : [
{
id: 1910,
postId : "123",
title : "comment1",
comment : "some comment",
user: "user13"
},
{
id: 1911,
postId : "124",
title : "comment2",
comment : "some comment",
user: "user22"
},
],
"commentUpVotes" : [
{
id : 12,
commentId : "1910",
upvotedBy: "user91",
upvoteDate: 1000,
},
{
id: 13,
commentId : "1910",
upvotedBy: "user92",
upvoteDate: 2000
},
{
id: 14,
commentId : "1911",
upvotedBy: "user92",
upvoteDate: 2100
}
]
}
NOTE: It is a post-question, and former one can be found here. I wanted to extend it a bit in this one.

Answer 1

I let this sit for a while as I did comment to you on the last question what the basic process would be to do this. I also commented that $redact is not the animal for this type of operation, and for reasons that are two involved to explain in addition to the answer here. Suffice to say that you know the filtered values and not just filter them.
So much as was given before, you still need some usage of $unwind but rather than the traditional usage that can blow out the number of documents to process in the pipeline, it is only being used after the array contents have been filtered. The only real differences here are that we are mindful that the "filtered array" actually is going to contain more than one element, so you handle it appropriately:
db.users.aggregate([
{ "$match": {
"commentUpvotes.upvoteDate": { "$gte": 0 }
}},
{ "$project": {
"posts": 1,
"comments": 1,
"commentUpVotes": {
"$setDifference": [
{
"$map": {
"input": "$commentUpvotes",
"as": "el",
"in": {
"$cond": [
{ "$gte": [ "$$el.upvoteDate", 0 ] },
"$$el",
false
]
}
}
},
[false]
]
}
}},
{ "$project": {
"posts": 1,
"comments": 1,
"kcommentUpVotes": "$commentUpVotes",
"commentUpVotes": 1
}},
{ "$unwind": "$commentUpVotes" },
{ "$project": {
"posts": 1,
"comments": {
"$setDifference": [
{
"$map": {
"input": "$comments",
"as": "el",
"in": {
"$cond": [
{
"$eq": [
{ "$substr": [ "$$el.id", 0, 4 ] },
"$commentUpVotes.commentId"
]
},
"$$el",
false
]
}
}
},
[false]
]
},
"commentUpVotes": "$kcommentUpVotes"
}},
{ "$unwind": "$comments" },
{ "$group": {
"_id": "$_id",
"posts": { "$first": "$posts" },
"comments": { "$addToSet": "$comments" },
"kcomments": { "$addToSet": "$comments" },
"commentUpVotes": { "$first": "$commentUpVotes" }
}},
{ "$unwind": "$comments" },
{ "$project": {
"posts": {
"$setDifference": [
{
"$map": {
"input": "$posts",
"as": "el",
"in": {
"$cond": [
{
"$eq": [
"$$el.id",
"$comments.postId"
]
},
"$$el",
false
]
}
}
},
[false]
]
},
"comments": "$kcomments",
"commentUpVotes": 1
}},
{ "$unwind": "$posts" },
{ "$group": {
"_id": "$_id",
"posts": { "$addToSet": "$posts" },
"comments": { "$first": "$comments" },
"commentUpVotes": { "$first": "$commentUpVotes" }
}}
])
So there is a point here to understand exactly what each stage ( or the repeated process ) is doing and why the $unwind operations here are important.
If you take the first $project here into consideration, the result returned is always going to be an array. This is how the "filtering" with $map works, and makes perfect sense as you are expecting the possibility of several ( in this example, all ) matches.
The important part takes place before you try to match those values against another array in your document as when you look at the anatomy of a $map the point is to compare the element against a singular value. This is why you need to $unwind in order to get at those "singular" values to compare.
So aside from keeping a copy of the "filtered" array to make things a bit cleaner, let's skip down to the part after matching against the "comments" array. Since the "commentUpvotes" array was "unwound" there is a copy for each document now with it's own filtered version of the array there. Noting that each result array can only contain a single element.
As these are indeed arrays, in order to combine them between documents you need to unwind these "single element" arrays and then group them back together. Keep in mind here that while there were "three" matches for "commentUpvotes" there are only "two" comments that would match but of the "three" matches "two" of them share the same id. This is where grouping back using $addToSet becomes important as you do not want to duplicate that matching post.
Once all the matched elements are in the array it's time again to $unwind and repeat.
So the general premise remains the same as the previous example and question. Indeed the approach here can be considered a "version 2.0" of the previous listing as it will cater for singular and "many" matches in all cases.
The one "caveat" to mention here is the basic principle that these items are indeed related and there is no "orphaned" detail in any of the arrays. The obvious reason for this that anything that was tested to match from one array to the other that did not match would result in an empty array. There may be other matches, but if one of those tests came up empty then you would have to handle the empty array that was produced.
The concept of that final note is easy by simply testing the $size of the result and otherwise entering a singe value of false and filtering that out at a later stage. But for the purpose of the exercise I am considering that your "relationships" are indeed intact, and leaving any additional handling up to your own implementation.
The end result is of course you get the desired result without resorting to the same level of "blowout" by simply unwinding the unfiltered arrays against each other and trying to match equality against those records.

Answer 2

I found a way to make it work without using $unwind and with $redact + $$ROOT. As you know $redact scans the document from parent to child, therefore to make comparison among subdocuments I needed to use $$ROOT.
Since it is only processed inside of the document, I believe this is the most efficient way. I will still be glad if some propose better ways. There are not many resources on $redact, and I believe the code below still can be improved:
// first query match
{
"$match": {
"commentUpvotes.upvoteDate": {
"$gte": 0
}
}
},
// exclude commentUpvotes
{
$redact: {
$cond: {
if: {
$or: [
{
$gte: [
"$upvoteDate",
0
]
},
{
$not: "$upvoteDate"
}
]
},
then: "$$DESCEND",
else: "$$PRUNE"
}
}
},
// exclude comments
{
$redact: {
$cond: {
if: {
$or: [
{
$not: "$postId"
},
{
$anyElementTrue: { $map: {
input: "$$ROOT.commentUpvotes",
as: "el",
in: { $cond: { if: { $eq: [ "$$el.commentId", "$id" ] },
then: true, else: false
}
}
}
}
}
]
},
then: "$$DESCEND",
else: "$$PRUNE"
}
}
},
// exclude posts
{
$redact: {
$cond: {
if: {
$or: [
{
$not: "$title"
},
{
$anyElementTrue: {
$map: {
input: "$$ROOT.comments",
as: "el",
in: {
$cond: {
if: {
$eq: [
"$$el.postId",
"$id"
]
},
then: true,
else: false
}
}
}
}
}
]
},
then: "$$DESCEND",
else: "$$PRUNE"
}
}
}