Compound mongo index optimization for query with null, false, and true - mongodb

Question

I'm trying to figure out the best way to create an index mongo can use to make this query faster:
"query" : {
"deleted_at" : null,
"placed_at" : {
"$exists" : true
},
"exported_at" : null,
"failed_export" : false
}
Currently, it's taking 2-3 min to go through the table even when there's no results. Explain shows that it's looking through hundreds of thousands of records and not using an index.
I tried running this:
db.some_table.createIndex({deleted_at: -1, placed_at: 1, exported_at: -1, failed_export: -1}, {background: true})
When I run the query afterward:
db.some_table.find({deleted_at: null, placed_at: {$exists: true}, exported_at: null, failed_export: false}).explain("executionStats")
I see it's using the correct index, but it's very slow. It's examining all 330494 rows. Here are the execution stats:
"executionStats" : {
"executionSuccess" : true,
"nReturned" : 0,
"executionTimeMillis" : 1585381,
"totalKeysExamined" : 330494,
"totalDocsExamined" : 330494,
"executionStages" : {
"stage" : "FETCH",
"filter" : {
"$and" : [
{
"placed_at" : {
"$exists" : true
}
},
{
"deleted_at" : {
"$eq" : null
}
},
{
"exported_at" : {
"$eq" : null
}
},
{
"failed_export" : {
"$eq" : false
}
}
]
},
The winning plan was:
"winningPlan" : {
"stage" : "FETCH",
"filter" : {
"$and" : [
{
"placed_at" : {
"$exists" : true
}
},
{
"deleted_at" : {
"$eq" : null
}
},
{
"exported_at" : {
"$eq" : null
}
},
{
"failed_export" : {
"$eq" : false
}
}
]
},
"inputStage" : {
"stage" : "IXSCAN",
"keyPattern" : {
"placed_at" : 1
},
"indexName" : "placed_at_1",
"isMultiKey" : false,
"direction" : "forward",
"indexBounds" : {
"placed_at" : [
"[MinKey, MaxKey]"
]
}
}
},
And it did list the index I created in one of the rejected plans.
Any ideas on why it would be going through every record in the database? This is hurting our performance.
I've tried hinting at the right record and that didn't seem to help much.

Answer 1

Instead of querying for deleted_at: null it would be better to create a new status field or an isDeleted field and configure your app servers to populate that field. Then, you can create a more effective index on this field to find all of your soft-deleted documents.
From the Performance Best Practices for MongoDB white paper:
Avoid negation in queries. Like most database systems, MongoDB does not index the absence of values and negation conditions may require scanning all documents. If negation is the only condition and it is not selective (for example, querying an orders table where 99% of the orders are complete to identify those that have not been fulfilled), all records will need to be scanned.