Optimizing MongoDB aggregate query on large Index objects - mongodb

Question

I have 20 Million objects in my MongoDb collection. Currently running on M30 MongoDb instance with 7.5Gb ram and 40Gb disk.
Data is stored in collection like this -
{
_id:xxxxx,
id : 1 (int),
from : xxxxxxxx (int),
to : xxxxxx (int),
status : xx (int)
.
.
.
.
},
{
_id:xxxxx,
id : 2 (int),
from : xxxxxxxx (int),
to : xxxxxx (int),
status : xx (int)
.
.
.
.
}
.
.
.
. and so on..
id is unique Index & from is a Index in this collection.
I am running a query to group 'to' and return me the max id and sort by max id with a given condition i.e 'from'
$collection->aggregate([
['$project' => ['id'=>1,'to'=>1,'from'=>1],
[ '$match'=> [
'$and'=>
[
[ 'from'=> xxxxxxxxxx],
[ 'status'=> xx ],
]
]
],
['$group' => [
'_id' =>
'$to',
'max_revision'=>['$max' => '$id'],
]
],
['$sort' => ['max_revision' => -1]],
['$limit' => 20],
]);
Above query runs just fine (~2 sec) on small data set on Index from like for 50-100k of same 'from' value in collection. But for conditions like, for example if 2M objects are having same 'from' value, then it is taking over >10 sec to execute and giving the result.
A quick example,
case 1- same query runs under 2 sec if it is executed with from as 12345, As 12345 is present 50k times in the collection.
case 2- query takes over 10 sec if it executed with from as 98765, As 98765 is present 2M times in the collection.
Edit : Explained query below -
{
"command": {
"aggregate": "mycollection",
"pipeline": [
{
"$project": {
"id": 1,
"to": 1,
"from": 1
}
},
{
"$match": {
"$and": [
{
"from": {
"$numberLong": "12345"
}
},
{
"status": 22
}
]
}
},
{
"$group": {
"_id": "$to",
"max_revision": {
"$max": "$id"
}
}
},
{
"$sort": {
"max_revision": -1
}
},
{
"$limit": 20
}
],
"allowDiskUse": false,
"cursor": {},
"$db": "mongo_jc",
"lsid": {
"id": {
"$binary": "8LktsSkpTjOzF3GIC+m1DA==",
"$type": "03"
}
},
"$clusterTime": {
"clusterTime": {
"$timestamp": {
"t": 1597230985,
"i": 1
}
},
"signature": {
"hash": {
"$binary": "PHh4eHh4eD4=",
"$type": "00"
},
"keyId": {
"$numberLong": "6859724943999893507"
}
}
}
},
"planSummary": [
{
"IXSCAN": {
"from": 1
}
}
],
"keysExamined": 1246529,
"docsExamined": 1246529,
"hasSortStage": 1,
"cursorExhausted": 1,
"numYields": 9747,
"nreturned": 0,
"queryHash": "29DAFB9E",
"planCacheKey": "F5EBA6AE",
"reslen": 231,
"locks": {
"ReplicationStateTransition": {
"acquireCount": {
"w": 9847
}
},
"Global": {
"acquireCount": {
"r": 9847
}
},
"Database": {
"acquireCount": {
"r": 9847
}
},
"Collection": {
"acquireCount": {
"r": 9847
}
},
"Mutex": {
"acquireCount": {
"r": 100
}
}
},
"storage": {
"data": {
"bytesRead": {
"$numberLong": "6011370213"
},
"timeReadingMicros": 4350129
},
"timeWaitingMicros": {
"cache": 2203
}
},
"protocol": "op_msg",
"millis": 8548
}

Answer 1

For this specific case the mongod query executor can use an index for the initial match, but not for the sort.
If you were to reorder and modify the stages a bit, it could use an index on {from:1, status:1, id:1} for both matching and sorting:
$collection->aggregate([
[ '$match'=> [
'$and'=>
[
[ 'from'=> xxxxxxxxxx],
[ 'status'=> xx ],
]
]
],
['$sort' => ['id' => -1]],
['$project' => ['id'=>1,'to'=>1,'from'=>1],
['$group' => [
'_id' => '$to',
'max_revision'=>['$first' => '$id'],
]
],
['$limit' => 20],
]);
This way the it should be able to combine the $match and $sort stages into a single index scan.