Why isn't MongoDB using my Text index? - mongodb

Question

I've got a Mongo collection with 1691721 items within it, containing essentially location information. I'm attempting to do a regex search into this, and it's pretty slow - but I don't understand why, as I thought I had appropriate indexes in place.
A typical document
{
"_id" : ObjectId("58c08029ef4468c8157455fa"),
"ng" : [
394235,
806529
],
"postcode" : "AB101AB"
}
Indexes
I've created a text index on the postcode field, which you can see here in the full list of indexes:
db.locations.getIndexes()
[
{
"v" : 2,
"key" : {
"_id" : 1
},
"name" : "_id_",
"ns" : "Traders.locations"
},
{
"v" : 2,
"key" : {
"_fts" : "text",
"_ftsx" : 1
},
"name" : "postcode_text",
"ns" : "Traders.locations",
"weights" : {
"postcode" : 1
},
"default_language" : "english",
"language_override" : "language",
"textIndexVersion" : 3
}
]
Query
At this point in time, all I care about is the postcode field. So I've tried writing a query to obtain the last value:
db.locations.find({ postcode: { $regex: /^ZE29XN$/ } }, { postcode: 1, _id: 0 })
Now this takes a while to run, roughly 700ms to be precise which is a lot longer than I was expecting. As far as I was concerned this is a covered query, I've got a text index on the single field I care about. However if I explain the above query it suggests it's using a COLLSCAN, but I don't understand why:
db.locations.find({ postcode: { $regex: /^ZE29XN$/ } }, { postcode: 1, _id: 0 }).explain("allPlansExecution")
{
"queryPlanner" : {
"plannerVersion" : 1,
"namespace" : "Traders.locations",
"indexFilterSet" : false,
"parsedQuery" : {
"postcode" : {
"$regex" : "^ZE29XN$"
}
},
"winningPlan" : {
"stage" : "PROJECTION",
"transformBy" : {
"postcode" : 1,
"_id" : 0
},
"inputStage" : {
"stage" : "COLLSCAN",
"filter" : {
"postcode" : {
"$regex" : "^ZE29XN$"
}
},
"direction" : "forward"
}
},
"rejectedPlans" : [ ]
},
"executionStats" : {
"executionSuccess" : true,
"nReturned" : 1,
"executionTimeMillis" : 732,
"totalKeysExamined" : 0,
"totalDocsExamined" : 1691721,
"executionStages" : {
"stage" : "PROJECTION",
"nReturned" : 1,
"executionTimeMillisEstimate" : 697,
"works" : 1691723,
"advanced" : 1,
"needTime" : 1691721,
"needYield" : 0,
"saveState" : 13223,
"restoreState" : 13223,
"isEOF" : 1,
"invalidates" : 0,
"transformBy" : {
"postcode" : 1,
"_id" : 0
},
"inputStage" : {
"stage" : "COLLSCAN",
"filter" : {
"postcode" : {
"$regex" : "^ZE29XN$"
}
},
"nReturned" : 1,
"executionTimeMillisEstimate" : 676,
"works" : 1691723,
"advanced" : 1,
"needTime" : 1691721,
"needYield" : 0,
"saveState" : 13223,
"restoreState" : 13223,
"isEOF" : 1,
"invalidates" : 0,
"direction" : "forward",
"docsExamined" : 1691721
}
},
"allPlansExecution" : [ ]
},
"serverInfo" : {
"host" : "DESKTOP",
"port" : 27017,
"version" : "3.4.2",
"gitVersion" : "3f76e40c105fc223b3e5aac3e20dcd026b83b38b"
},
"ok" : 1
}
My Question
Why is the text index I've created not being used, and ultimately how can I make my query faster?
I should note that I'm open to alternative to using the $regex however I do need to be able to allow for a "starts with" - so ZE.* or ZE2.* or ZE29XN should all be searchable at speed.
Something potentially useful to note, I wondered if when I eventually get my index working, that marking it as unique: true might help speed things up. However running it produced a duplicate key error (despite the fact I can't find one when I run an aggregation - can dig into this if need be, but I'm not sure it's related).

Answer 1

A quick summary of text search operators in MongoDB:
$regex: provides regular expression capabilities for pattern matching strings in queries. The $regex operator does support partial matches but it will only be index covered if the search string is anchored (i.e. uses a leading ^).
$text: performs a text search on the content of the fields indexed with a text index (use of the $text operator is a necessary precondition for MongoDB to use a text index). These searches are typically 'fast' (subjective term but when you have one working you'll see what that means) but they do not support partial matches so you won't be able to 'text search' for partial postcodes.
With that in mind it looks like you are attempting to use partial matching (via $regex) against a text index. This will not work because a text index is only engaged for the $text operator.
Your stated requirements are:
You want partial string matches
You want index coverage
You can meet these requirements by (1) using $regex and (2) indexing (a normal index not a text index) the postcode field. This is subject to one (important!) caveat: your search strings must be anchored. So this requirement: "need to be able to allow for a "starts with" - so ZE.* or ZE2.* or ZE29XN" should be fine. But a search such as: .*29XN will not be index covered.