Complex MongoDB Aggregation - mongodb

Question

I have a situation where I need to perform a group by operation based on an array value that sums up occurrences of a field value. The counts are then filtered on and the results are prepared so that they can be displayed according to the condition. Essentially, the documents are transformed back to how they would be presented if you simply used the find function. I am running into an issue of the temporary documents being too large due to the number of items collected in the matchedDocuments array. Any suggestions on how to improve this would be helpful.
db.collection1.aggregate([
{
'$unwind': '$arrayOfValues'
}, {
'$group': {
'_id': '$arrayOfValues',
'x_count': {
$sum: {
$cond: [{
$eq: ['$field.value', 'x']
},
1, 0
]
}
},
'y_count': {
$sum: {
$cond: [{
$eq: ['$field.value', 'y']
},
1, 0
]
}
},
'matchedDocuments': {
'$push': '$$CURRENT'
}
}
},
{'$match': {'$or': [{'x_count': {'$gte': 2}}, {'y_count': { '$gte': 1}}]}},
{'$unwind': '$matchedDocuments'},
{
'$group': {
'_id': '$matchedDocuments.key',
'document': {
'$last': '$$CURRENT.matchedDocuments'
}
}
}
], {
allowDiskUse: true
})
Below are some sample documents and the expected result based on the criteria above:
// Sample documents
{ "_id" : ObjectId("5407c76b7b1c276c74f90524"), "field" : "x", "arrayOfValues" : [ "a", "b", "c" ] }
{ "_id" : ObjectId("5407c76b7b1c276c74f90525"), "field" : "x", "arrayOfValues" : [ "b", "c" ] }
{ "_id" : ObjectId("5407c76b7b1c276c74f90526"), "field" : "z", "arrayOfValues" : [ "a" ] }
{ "_id" : ObjectId("5407c76b7b1c276c74f90527"), "field" : "x", "arrayOfValues" : [ "a", "c" ] }
{ "_id" : ObjectId("5407c76b7b1c276c74f90528"), "field" : "z", "arrayOfValues" : [ "b" ] }
{ "_id" : ObjectId("5407c76b7b1c276c74f90529"), "field" : "y", "arrayOfValues" : [ "k" ] }
// Expected Result
[
{ "_id" : ObjectId("5407c76b7b1c276c74f90524"), "field" : "x", "arrayOfValues" : [ "a", "b", "c" ] }
{ "_id" : ObjectId("5407c76b7b1c276c74f90525"), "field" : "x", "arrayOfValues" : [ "b", "c" ] }
{ "_id" : ObjectId("5407c76b7b1c276c74f90527"), "field" : "x", "arrayOfValues" : [ "a", "c" ] }
{ "_id" : ObjectId("5407c76b7b1c276c74f90529"), "field" : "y", "arrayOfValues" : [ "k" ] }
]

Answer 1

I think ultimately you are asking a little too much from a single query, as clearly the biggest problem here is trying to store all of the original documents from whence the array element came whilst trying to aggregate a total.
For me, I would just try to identify which conditions on the document would result in a match and then issue a separate query to get the actual documents back. You could adapt the aggregation below to try to return the document, but I think it's very likely to fail when doing so as it would be the reverse of what you should be using arrays for.
The process is also generally much more efficient in the way it goes about the matching alowing you to firstly "Select the elements you are interested in with a match condition" and secondly, "Use the natural grouping conditions rather than rely on conditional sums".
var cursor = db.collection.aggregate([
{ "$match": { "field": { "$in": ["x", "y"] } } },
{ "$unwind": "$arrayOfValues" },
{ "$group": {
"_id": {
"elem": "$arrayOfValues",
"field": "$field"
},
"count": { "$sum": 1 }
}},
{ "$match": {
"$or": [
{ "_id.field": "x", "count": { "$gte": 2 } },
{ "_id.field": "y", "count": { "$gte": 1 } }
]
}},
{ "$group": {
"_id": "$_id.field",
"values": { "$push": "$_id.elem" }
}}
])
var query = { "$or": [] };
cursor.forEach(function(doc) {
query["$or"].push({
"field": doc._id,
"arrayOfValues": { "$in": doc.values }
});
});
db.collection.find(query)
For the record the query should come out like this, given the supplied data:
{
"$or" : [
{
"field" : "x",
"arrayOfValues" : {
"$in" : [
"c",
"b",
"a"
]
}
},
{
"field" : "y",
"arrayOfValues" : {
"$in" : [
"k"
]
}
}
]
}
The basic logic is met by just looking for the values of "field" that you are interested in, so at least eliminating all others from the possible results. Then you basically want to tally up the counts for each array element under each of those "field" values and test where the required occurrences were met.
This may or may not work best the other way around, but the sample here shows the greatest variation by the "arrayOfValues" so that makes sense as the second level of grouping.
As stated earlier, I think it is too much to ask to basically "stuff" all of the parent document information into an array for each "arrayOfValues" element as this works beyond the basic principles of a sensible schema, where that sort of relation would naturally be stored as separate documents. So the end principle here is just find the "conditions" that match those documents which is what the end result comes out as.
The transformed query is then issued against the collection, where that will return all documents that meet the conditions determined from the previous analysis. At the end of the day, moving the responsibility of "fetching" the matching documents off to another query rather than trying to store the documents that match in arrays.
This seems the most logical and scale-able approach, but if you mostly tend to use your data in this type of result you should be looking at re-designing your schema to suit this better. But there really is not enough specific information here to comment on that further.