Why will Mongodb only use my index if I use hint - mongodb

Question

I have a database with a ISODate() type field with an index (I also tried this experiment with string fields - same result). I am using the open source version of MongoDB (4.x) and when I do a query / sort to find the max _finish_time, the index is not used unless I specify a hint.
My query is:
db.getCollection("test").find({}, { _finish_time: 1}).sort({_finish_time: -1}).limit(1)
which explains as:
{
"queryPlanner" : {
"plannerVersion" : 1.0,
"namespace" : "vdm-service-ts-staging.test",
"indexFilterSet" : false,
"parsedQuery" : {
},
"winningPlan" : {
"stage" : "PROJECTION",
"transformBy" : {
"_finish_time" : 1.0
},
"inputStage" : {
"stage" : "SORT",
"sortPattern" : {
"_finish_time" : -1.0
},
"limitAmount" : 1.0,
"inputStage" : {
"stage" : "SORT_KEY_GENERATOR",
"inputStage" : {
"stage" : "COLLSCAN",
"direction" : "forward"
}
}
}
},
"rejectedPlans" : [
]
},
"serverInfo" : {
"host" : "ip-10-82-245-45.us-west-2.compute.internal",
"port" : 27017.0,
"version" : "4.0.1",
"gitVersion" : "54f1582fc6eb01de4d4c42f26fc133e623f065fb"
},
"ok" : 1.0,
"operationTime" : Timestamp(1573220526, 1),
"$clusterTime" : {
"clusterTime" : Timestamp(1573220526, 1),
"signature" : {
"hash" : BinData(0, "blIkiGcam87SDdbKeZKex/9JXBU="),
"keyId" : NumberLong(6715502669504446467)
}
}
}
Which scans the entire collection. When I specify a hint for my available index, as in:
db.getCollection("test").find({}, { _finish_time: 1}).sort({_finish_time: -1}).limit(1).hint("_finish_time")
I get the query plan:
{
"queryPlanner" : {
"plannerVersion" : 1.0,
"namespace" : "vdm-service-ts-staging.test",
"indexFilterSet" : false,
"parsedQuery" : {
},
"winningPlan" : {
"stage" : "LIMIT",
"limitAmount" : 1.0,
"inputStage" : {
"stage" : "PROJECTION",
"transformBy" : {
"_finish_time" : 1.0
},
"inputStage" : {
"stage" : "FETCH",
"inputStage" : {
"stage" : "IXSCAN",
"keyPattern" : {
"_finish_time" : -1.0
},
"indexName" : "_finish_time",
"isMultiKey" : false,
"multiKeyPaths" : {
"_finish_time" : [
]
},
"isUnique" : false,
"isSparse" : true,
"isPartial" : false,
"indexVersion" : 2.0,
"direction" : "forward",
"indexBounds" : {
"_finish_time" : [
"[MaxKey, MinKey]"
]
}
}
}
}
},
"rejectedPlans" : [
]
},
"serverInfo" : {
"host" : "ip-10-82-245-45.us-west-2.compute.internal",
"port" : 27017.0,
"version" : "4.0.1",
"gitVersion" : "54f1582fc6eb01de4d4c42f26fc133e623f065fb"
},
"ok" : 1.0,
"operationTime" : Timestamp(1573220603, 3),
"$clusterTime" : {
"clusterTime" : Timestamp(1573220603, 3),
"signature" : {
"hash" : BinData(0, "qsGhD1DpI306XbqtNZDYVINPid8="),
"keyId" : NumberLong(6715502669504446467)
}
}
}
Which uses the index. I would prefer not to have to add hint() to my queries and I am perplexed why it refuses to use the index.
My index is sparse and not unique.
I have tried other indexes and searching around, but I cannot find any reference to this problem in Stack overflow or elsewhere.

Answer 1

According to documentation of MongoDB
If a sparse index would result in an incomplete result set for queries
and sort operations, MongoDB will not use that index unless a hint()
explicitly specifies the index.
To use the sparse index, explicitly specify the index with hint()
As hint() forces query optimizer to use index while performing execution of query