I have an aggregation pipeline query (I've removed unnecesary stuff) that work when using $expr but doesn't work without. However, I want to avoid using the $expr for better performance, so the indices will be used. Logically there is a many to many relation here between rule and resource. I want to summarize the cost of the resources per rule. The problem here is to match the resources inside the grouped array without using an expression.
with $expr:
db.collection.aggregate([
{'$group': {'_id': {'rule_id': '$rule_id'}, 'rule_id': {'$first': '$rule_id'}, 'resources_ids': {'$push': '$resource_id'}}},
{'$lookup':
{'from': 'other_collection',
'let': {'resources_ids': '$resources_ids'},
'pipeline': [
{'$match':
{'$expr":
{'$and': [
{'$in':['$resource_id', '$$resources_ids']}
]}
}
},
{'$group': {'_id': {}, 'total_cost': {'$sum': '$cost'}}}], 'as': 'results'}}])
without $expr:
db.collection.aggregate([
{'$group': {'_id': {'rule_id': '$rule_id'}, 'rule_id': {'$first': '$rule_id'}, 'resources_ids': {'$push': '$resource_id'}}},
{'$lookup':
{'from': 'cost_data',
'let': {'resources_ids': '$resources_ids'},
'pipeline': [
{'$match':
{'$and': [
{'resource_id': {'$in': '$$resources_ids'}},
]}
},
{'$group': {'_id': {}, 'total_cost': {'$sum': '$cost'}}}], 'as': 'results'}}])
I think #rickhg12hs' comment gives the right answer. There shouldn't be any need for $expr here. The localField/foreignField syntax will work correctly when the localField is an array without needing to $unwind (or use $expr) as documented here. Therefore the matching component of your $lookup can effectively look like this:
$lookup: {
from: "foreign",
localField: "resources_ids",
foreignField: "resource_id",
as: "result"
}
You can compare the outputs of this syntax above with the more verbose pipeline/$expr version to see that they are the same.
A few other thoughts come to mind. The first is that you can combine the localField/foreignField syntax with the pipeline syntax so that the second $group can still be nested inside of the $lookup. This would make the final version of the $lookup stage structured as follows:
{
$lookup: {
from: "foreign",
localField: "resources_ids",
foreignField: "resource_id",
pipeline: [
{
"$group": {
"_id": {},
"total_cost": {
"$sum": "$cost"
}
}
}
],
as: "result"
}
}
Playground demonstration of that component is here.
The second thing is that using an index to perform the $lookups is important and will likely improve performance, but it may not make this operation "fast". As written, this aggregation will perform a full collection scan to process all of the documents in the source collection. (You will still see that COLLSCAN in the explain output from this source collection even if the index in the other collection is used for the $lookup).
Finally, the index on the other collection should probably be { resource_id: 1, cost: 1 }. This should allow the database to cover the query when doing the $lookups and avoid fetching those documents altogether.
Edit to address this comment:
$expr is not required inside the match. I've done this already. However, here because of the array I am building through the pipeline it doesn't let me use it in the match without an $expr.
This is not correct. Specifically the source of the array isn't relevant here. Whether the array is a field in the source document directly or generated in an earlier pipeline stage doesn't matter to the $lookup stage at all. In fact, it won't even know where that array comes from, just that it is a field in the generated document that is passed to it.
Rather, what you are describing is the behavior of $match itself. From the documentation:
$match takes a document that specifies the query conditions. The query syntax is identical to the read operation query syntax; i.e. $match does not accept raw aggregation expressions. Instead, use a $expr query expression to include aggregation expression in $match.
Said another way, you presently cannot reference any fields from the document (regardless of where they come from or what type and value they have) without $expr.
But that fact should mostly be irrelevant for your use case. You can use the localField/foreignField syntax for this array matching. If you need to match on additional filters then you can also leverage the let/pipeline syntax in the same $lookup. Here is an arbitrary demonstration of that (note the _id: 4 document doesn't match due to the mismatched otherVal).
It is also worth noting that $expr itself does not preclude the usage of indexes in general. One current exception, unfortunately, seems to be with $in (reference). Again though, that shouldn't matter for you if you place that part of the $lookup matching into the localField/foreignField parameters.
I'm trying to look up all records that match a certain condition, in this case _id being certain values, and then return only the top 2 results, sorted by the name field.
This is what I have
db.getCollection('col1').aggregate([
{$match: {fk: {$in: [1, 2]}}},
{$sort: {fk: 1, name: -1}},
{$group: {_id: "$fk", items: {$push: "$$ROOT"} }},
{$project: {items: {$slice: ["$items", 2]} }}
])
and it works, BUT, it's not guaranteed. According to this Mongo thread $group does not guarantee document order.
This would also mean that all of the suggested solutions here and elsewhere, which recommend using $unwind, followed by $sort, and then $group, would also not work, for the same reason.
What is the best way to accomplish this with Mongo (any version)? I've seen suggestions that this could be accomplished in the $project phase, but I'm not quite sure how.
You are correct in saying that the result of $group is never sorted.
$group does not order its output documents.
Hence doing a;
{$sort: {fk: 1}}
then grouping with
{$group: {_id: "$fk", ... }},
will be a wasted effort.
But there is a silver lining with sorting before $group stage with name: -1. Since you are using $push (not an $addToSet), inserted objects will retain the order they've had in the newly created items array in the $group result. You can see this behaviour here (copy of your pipeline)
The items array will always have;
"items": [
{
..
"name": "Michael"
},
{
..
"name": "George"
}
]
in same order, therefore your nested array sort is a non-issue! Though I am unable to find an exact quote in documentation to confirm this behaviour, you can check;
this,
or this where it is confirmed.
Also, accumulator operator list for $group, where $addToSet has "Order of the array elements is undefined." in its description, whereas the similar operator $push does not, which might be an indirect evidence? :)
Just a simple modification of your pipeline where you move the fk: 1 sort from pre-$group stage to post-$group stage;
db.getCollection('col1').aggregate([
{$match: {fk: {$in: [1, 2]}}},
{$sort: {name: -1}},
{$group: {_id: "$fk", items: {$push: "$$ROOT"} }},
{$sort: {_id: 1}},
{$project: {items: {$slice: ["$items", 2]} }}
])
should be sufficient to have the main result array order fixed as well. Check it on mongoplayground
$group doesn't guarantee the document order but it would keep the grouped documents in the sorted order for each bucket. So in your case even though the documents after $group stage are not sorted by fk but each group (items) would be sorted by name descending. If you would like to keep the documents sorted by fk you could just add the {$sort:{fk:1}} after $group stage
You could also sort by order of values passed in your match query should you need by adding a extra field for each document. Something like
db.getCollection('col1').aggregate([
{$match: {fk: {$in: [1, 2]}}},
{$addField:{ifk:{$indexOfArray:[[1, 2],"$fk"]}}},
{$sort: {ifk: 1, name: -1}},
{$group: {_id: "$ifk", items: {$push: "$$ROOT"}}},
{$sort: {_id : 1}},
{$project: {items: {$slice: ["$items", 2]}}}
])
Update to allow array sort without group operator : I've found the jira which is going to allow sort array.
You could try below $project stage to sort the array.There maybe various way to do it. This should sort names descending.Working but a slower solution.
{"$project":{"items":{"$reduce":{
"input":"$items",
"initialValue":[],
"in":{"$let":{
"vars":{"othis":"$$this","ovalue":"$$value"},
"in":{"$let":{
"vars":{
//return index as 0 when comparing the first value with initial value (empty) or else return the index of value from the accumlator array which is closest and less than the current value.
"index":{"$cond":{
"if":{"$eq":["$$ovalue",[]]},
"then":0,
"else":{"$reduce":{
"input":"$$ovalue",
"initialValue":0,
"in":{"$cond":{
"if":{"$lt":["$$othis.name","$$this.name"]},
"then":{"$add":["$$value",1]},
"else":"$$value"}}}}
}}
},
//insert the current value at the found index
"in":{"$concatArrays":[
{"$slice":["$$ovalue","$$index"]},
["$$othis"],
{"$slice":["$$ovalue",{"$subtract":["$$index",{"$size":"$$ovalue"}]}]}]}
}}}}
}}}}
Simple example with demonstration how each iteration works
db.b.insert({"items":[2,5,4,7,6,3]});
othis ovalue index concat arrays (parts with counts) return value
2 [] 0 [],0 [2] [],0 [2]
5 [2] 0 [],0 [5] [2],-1 [5,2]
4 [5,2] 1 [5],1 [4] [2],-1 [5,4,2]
7 [5,4,2] 0 [],0 [7] [5,4,2],-3 [7,5,4,2]
6 [7,5,4,2] 1 [7],1 [6] [5,4,2],-3 [7,6,5,4,2]
3 [7,6,5,4,2] 4 [7,6,5,4],4 [3] [2],-1 [7,6,5,4,3,2]
Reference - Sorting Array with JavaScript reduce function
There is a bit of a red herring in the question as $group does guarantee that it will be processing incoming documents in order (and that's why you have to sort of them before $group to get an ordered arrays) but there is an issue with the way you propose doing it, since pushing all the documents into a single grouping is (a) inefficient and (b) could potentially exceed maximum document size.
Since you only want top two, for each of the unique fk values, the most efficient way to accomplish it is via a "subquery" using $lookup like this:
db.coll.aggregate([
{$match: {fk: {$in: [1, 2]}}},
{$group:{_id:"$fk"}},
{$sort: {_id: 1}},
{$lookup:{
from:"coll",
as:"items",
let:{fk:"$_id"},
pipeline:[
{$match:{$expr:{$eq:["$fk","$$fk"]}}},
{$sort:{name:-1}},
{$limit:2},
{$project:{_id:0, fk:1, name:1}}
]
}}
])
Assuming you have an index on {fk:1, name:-1} as you must to get efficient sort in your proposed code, the first two stages here will use that index via DISTINCT_SCAN plan which is very efficient, and for each of them, $lookup will use that same index to filter by single value of fk and return results already sorted and limited to first two. This will be the most efficient way to do this at least until https://jira.mongodb.org/browse/SERVER-9377 is implemented by the server.
I am using this solution for indexing messages with many varying fields. Specifically, I am using Solution#2.
The example of range syntax
db.generic2.find({"props": { $elemMatch: {$gte: {"prop1": 6}, $lt: {"prop1": 99999999 } }}})
I have never seen this syntax in MongoDB docs, rather I see everywhere syntax like
db.generic2.find({"props": { $elemMatch: {"prop1": {$gte: 6, $lt: 99999999 }}}})
What is the difference? Funny using the first one I get fast query using indexing, using the second I get a slow query with collection scan. Both results are correct, however different.
I'm using mongo 2.6.8 and have the following problem:
Collection users has indexes _id_1 and b_1. When I perform query
db.users.find({"$and": [
{"b": {"$gt": ISODate("somedate")}},
{"b": {"$lt": ISODate("anotherdate")}},
{"_id": {"$gt": "somevalue"}},
{"_id": {"$lt": "anothervalue"}},
]})
I expect that mongo will perform index intersection and will use intersected index, but it chooses only b_1 index. When executing explain on this query allPlans section even doesn't contain intersected index, only _id_1 and b_1.
Why does mongo not perform index intersection?
I think this could result from the fact, that you have two restrictions in your query on the same indexed key ($gt and $lt on b (and same for _id)). What happens to your explain if you change your query to the following. If It's using intersection I would be right:
db.users.find({"$and": [
{"b": {"$gt": ISODate("somedate")}},
{"_id": {"$gt": "somevalue"}},
]})
In this case using both restrictions on one index could be faster than using only one restriction of both indexes and use the intersection.