Question about how mongodb chooses indexes or $or queries - mongodb

Question

Background
I have a collection with the following schema:
{ appId, createdAt, lastSeen, lastHeard, deleted, ...otherFields }
and the following indexes:
{ appId: 1, deleted: 1, lastSeen: 1 }
{ appId: 1, deleted: 1, lastHeard: 1 }
{ appId: 1, createdAt: 1, deleted: 1 }
{ appId: 1, deleted: 1, createdAt: 1, lastSeen: 1, lastHeard: 1 }
In my application I have an aggregation:
db.getCollection('client_users').aggregate([
{
$match: {
deleted: false,
appId: 'appid',
$or: [
{ createdAt: { $gt: new Date('2020-10-19T17:00:00.000Z') } },
{ lastSeen: { $gt: new Date('2020-10-19T17:00:00.000Z') } },
{ lastHeard: { $gt: new Date('2020-10-19T17:00:00.000Z') } },
]
}
},
{
$group: {
_id: '$geoLocation.city',
count: {
$sum: 1
}
}
},
{
$sort: {
count: -1
}
}
]);
My intention was to use the first three indexes from the above for this aggregation, as I understand that the $or query is parsed into 3 separate queries. However, from the explain output, the winning plan uses the forth index ({ appId: 1, deleted: 1, createdAt: 1, lastSeen: 1, lastHeard: 1 } ) for the 2 last clauses:
"winningPlan" : {
"stage" : "FETCH",
"inputStage" : {
"stage" : "OR",
"inputStages" : [
{
"stage" : "IXSCAN",
"keyPattern" : {
"appId" : 1,
"createdAt" : 1,
"deleted" : 1
},
"indexName" : "appId_1_createdAt_1_deleted_1",
"isMultiKey" : false,
"multiKeyPaths" : {
"appId" : [],
"createdAt" : [],
"deleted" : []
},
"isUnique" : false,
"isSparse" : false,
"isPartial" : false,
"indexVersion" : 2,
"direction" : "forward",
"indexBounds" : {
"appId" : [
"[\"appid\", \"appid\"]"
],
"createdAt" : [
"(new Date(1603126800000), new Date(9223372036854775807)]"
],
"deleted" : [
"[false, false]"
]
}
},
{
"stage" : "IXSCAN",
"keyPattern" : {
"appId" : 1,
"deleted" : 1,
"createdAt" : 1,
"lastSeen" : 1,
"lastHeard" : 1
},
"indexName" : "appId_1_deleted_1_createdAt_1_lastSeen_1_lastHeard_1",
"isMultiKey" : false,
"multiKeyPaths" : {
"appId" : [],
"deleted" : [],
"createdAt" : [],
"lastSeen" : [],
"lastHeard" : []
},
"isUnique" : false,
"isSparse" : false,
"isPartial" : false,
"indexVersion" : 2,
"direction" : "forward",
"indexBounds" : {
"appId" : [
"[\"appid\", \"appid\"]"
],
"deleted" : [
"[false, false]"
],
"createdAt" : [
"[MinKey, MaxKey]"
],
"lastSeen" : [
"[MinKey, MaxKey]"
],
"lastHeard" : [
"(new Date(1603126800000), new Date(9223372036854775807)]"
]
}
},
{
"stage" : "IXSCAN",
"keyPattern" : {
"appId" : 1,
"deleted" : 1,
"createdAt" : 1,
"lastSeen" : 1,
"lastHeard" : 1
},
"indexName" : "appId_1_deleted_1_createdAt_1_lastSeen_1_lastHeard_1",
"isMultiKey" : false,
"multiKeyPaths" : {
"appId" : [],
"deleted" : [],
"createdAt" : [],
"lastSeen" : [],
"lastHeard" : []
},
"isUnique" : false,
"isSparse" : false,
"isPartial" : false,
"indexVersion" : 2,
"direction" : "forward",
"indexBounds" : {
"appId" : [
"[\"appid\", \"appid\"]"
],
"deleted" : [
"[false, false]"
],
"createdAt" : [
"[MinKey, MaxKey]"
],
"lastSeen" : [
"(new Date(1603126800000), new Date(9223372036854775807)]"
],
"lastHeard" : [
"[MinKey, MaxKey]"
]
}
}
]
}
},
Which is not what I want. What's stranger is that when I try it with only 1 of the clauses, as in this $match stage:
$match: {
deleted: false,
appId: 'appid',
lastSeen: {$gt: new Date('2020-10-19T17:00:00.000Z') },
}
It uses the correct index ({ appId: 1, deleted: 1, lastSeen: 1 }). I know this from the explain output and from timing the actual aggregation. Specifically, running it with no hint or with hint: appId_1_deleted_1_lastSeen_1 takes three times shorter than with hint: appId_1_deleted_1_createdAt_1_lastSeen_1_lastHeard_1. This makes me very confused about how mongodb chooses the index.
Can someone explain to me what could have been the reason for this behavior? Is there a way for me to force mongodb to use the indexes I want in this case? Thanks.

Answer 1

I figured it out. It was precisely because of the $or query. Mongodb chooses the query plan by letting them do a small race against each other. The more inefficient plan luckily won because the first $or clause took care of everything (remember it was $or so only 1 clause is enough). I fixed this by dropping the forth index.