MongoDB - How does it avoid full collection scan? - mongodb

Question

I have this users collection with 1000000 rows.
The structure of each document is shown below by a call to findOne.
The indexes are shown too through a call to getIndexes. So I have
two compound indexes on it, only the order of their keys is different.
All the username values are unique in this collection,
they are of the form "user" + k, for k=0,1,2,...,999999.
Also, I don't have empty ages or usernames.
[test] 2014-03-08 20:08:10.135 >>> db.users.aggregate({'$match':{ 'username':{'$exists':false} }}) ;
{ "result" : [ ], "ok" : 1 }
[test] 2014-03-08 20:08:27.760 >>> db.users.aggregate({'$match':{ 'age':{'$exists':false} }}) ;
{ "result" : [ ], "ok" : 1 }
[test] 2014-03-08 20:08:41.198 >>> db.users.find({username : null}).count();
0
[test] 2014-03-08 20:12:01.456 >>> db.users.find({age : null}).count();
0
[test] 2014-03-08 20:12:06.790 >>>
What I don't understand in this explain I am running is the following:
How is MongoDB able to scan only 996291 document and to avoid scanning
the remaining 3709 documents. How is MongoDB sure he is not missing
any documents (from these 3709 ones) which match the query criterion?
I don't see how that is possible if we assume MongoDB is only using
the username_1_age_1 index.
C:\>C:\Programs\MongoDB\bin\mongo.exe
MongoDB shell version: 2.4.8
connecting to: test
Welcome to the MongoDB shell!
[test] 2014-03-08 19:31:41.683 >>> db.users.count();
1000000
[test] 2014-03-08 19:31:45.68 >>> db.users.findOne();
{
"_id" : ObjectId("5318fac5e22bd6bc482baf88"),
"i" : 0,
"username" : "user0",
"age" : 10,
"created" : ISODate("2014-03-06T22:46:29.225Z")
}
[test] 2014-03-08 19:32:06.352 >>> db.users.getIndexes();
[
{
"v" : 1,
"key" : {
"_id" : 1
},
"ns" : "test.users",
"name" : "_id_"
},
{
"v" : 1,
"key" : {
"age" : 1,
"username" : 1
},
"ns" : "test.users",
"name" : "age_1_username_1"
},
{
"v" : 1,
"key" : {
"username" : 1,
"age" : 1
},
"ns" : "test.users",
"name" : "username_1_age_1"
}
]
[test] 2014-03-08 19:31:49.941 >>> db.users.find({"age" : {"$gte" : 21, "$lte" : 30}}).sort({"username" : 1}).hint({"username" : 1, "age" : 1}).explain();
{
"cursor" : "BtreeCursor username_1_age_1",
"isMultiKey" : false,
"n" : 167006,
"nscannedObjects" : 167006,
"nscanned" : 996291,
"nscannedObjectsAllPlans" : 167006,
"nscannedAllPlans" : 996291,
"scanAndOrder" : false,
"indexOnly" : false,
"nYields" : 3,
"nChunkSkips" : 0,
"millis" : 3177,
"indexBounds" : {
"username" : [
[
{
"$minElement" : 1
},
{
"$maxElement" : 1
}
]
],
"age" : [
[
21,
30
]
]
},
"server" : "mongo020:27017"
}
[test] 2014-03-08 19:32:06.352 >>>
UPDATE - Here is an exact description how to reproduce:
C:\>mongo
C:\>C:\Programs\MongoDB\bin\mongo.exe
MongoDB shell version: 2.4.8
connecting to: test
Welcome to the MongoDB shell!
[test] 2014-03-11 05:13:00.941 >>> function populate(){
...
... for (i=0; i<1000000; i++) {
... db.users.insert({
... "i" : i,
... "username" : "user"+i,
... "age" : Math.floor(Math.random()*60),
... "created" : new Date()
... }
... );
... }
... }
[test] 2014-03-11 05:13:33.139 >>>
[test] 2014-03-11 05:15:46.689 >>> populate();
[test] 2014-03-11 05:16:46.366 >>> db.users.ensureIndex({username:1, age:1});
[test] 2014-03-11 05:17:05.476 >>>
[test] 2014-03-11 05:17:05.476 >>> db.users.count();
1000000
[test] 2014-03-11 05:18:35.297 >>> db.users.getIndexes();
[
{
"v" : 1,
"key" : {
"_id" : 1
},
"ns" : "test.users",
"name" : "_id_"
},
{
"v" : 1,
"key" : {
"username" : 1,
"age" : 1
},
"ns" : "test.users",
"name" : "username_1_age_1"
}
]
[test] 2014-03-11 05:19:54.657 >>>
[test] 2014-03-11 05:19:54.657 >>> db.users.find({"age" : {"$gte" : 21, "$lte" : 30}}).sort({"username" : 1}).hint({"username" : 1, "age" : 1}).explain();
{
"cursor" : "BtreeCursor username_1_age_1",
"isMultiKey" : false,
"n" : 166799,
"nscannedObjects" : 166799,
"nscanned" : 996234,
"nscannedObjectsAllPlans" : 166799,
"nscannedAllPlans" : 996234,
"scanAndOrder" : false,
"indexOnly" : false,
"nYields" : 2,
"nChunkSkips" : 0,
"millis" : 2730,
"indexBounds" : {
"username" : [
[
{
"$minElement" : 1
},
{
"$maxElement" : 1
}
]
],
"age" : [
[
21,
30
]
]
},
"server" : "mongo020:27017"
}
[test] 2014-03-11 05:20:44.15 >>>

Answer 1

I'm pretty sure this is a 2.4 bug caused by this bit of code:
// If nscanned is increased by more than 20 before a matching key is found, abort
// skipping through the btree to find a matching key. This iteration cutoff
// prevents unbounded internal iteration within BtreeCursor::init() and
// BtreeCursor::advance() (the callers of skipAndCheck()). See SERVER-3448.
if ( _nscanned > startNscanned + 20 ) {
skipUnusedKeys();
// If iteration is aborted before a key matching _bounds is identified, the
// cursor may be left pointing at a key that is not within bounds
// (_bounds->matchesKey( currKey() ) may be false). Set _boundsMustMatch to
// false accordingly.
_boundsMustMatch = false;
return;
}
and more imporantly here:
//don't include unused keys in nscanned
//++_nscanned;
As you scan the index, you'll lose an increment of nscanned every time you have 20 consecutive misses.
You can reproduce with a very simple example:
> db.version()
2.4.8
>
> for (var i = 1; i<=100; i++){db.foodle.save({_id:i, name:'a'+i, age:1})}
> db.foodle.ensureIndex({name:1, age:1})
> db.foodle.find({ age:{ $gte:10, $lte:20 }}).hint({name:1, age:1}).explain()
{
"cursor" : "BtreeCursor name_1_age_1",
"isMultiKey" : false,
"n" : 0,
"nscannedObjects" : 0,
"nscanned" : 96,
"nscannedObjectsAllPlans" : 0,
"nscannedAllPlans" : 96,
"scanAndOrder" : false,
"indexOnly" : false,
"nYields" : 0,
"nChunkSkips" : 0,
"millis" : 1,
"indexBounds" : {
"name" : [
[
{
"$minElement" : 1
},
{
"$maxElement" : 1
}
]
],
"age" : [
[
10,
20
]
]
},
"server" : "Jeffs-MacBook-Air.local:27017"
}
If you change the ages so you don't get 20 misses, the value of nscanned is what you would expect:
for (var i = 1; i<=100; i++){
var theAge = 1;
if (i%10 == 0){ theAge = 15;}
db.foodle.save({ _id:i, name:'a'+i, age: theAge });
}
{
"cursor" : "BtreeCursor name_1_age_1",
"isMultiKey" : false,
"n" : 10,
"nscannedObjects" : 10,
"nscanned" : 100,
"nscannedObjectsAllPlans" : 10,
"nscannedAllPlans" : 100,
"scanAndOrder" : false,
"indexOnly" : false,
"nYields" : 0,
"nChunkSkips" : 0,
"millis" : 0,
"indexBounds" : {
"name" : [
[
{
"$minElement" : 1
},
{
"$maxElement" : 1
}
]
],
"age" : [
[
10,
20
]
]
},
"server" : "Jeffs-MacBook-Air.local:27017"
}
I'm not sure why the increment is commented out, but this code has all been changed in 2.6 and should return the nscanned that you expect.

Answer 2

The correct "solution" is not to force the query optimizer to use an index that doesn't match its idea of a "qualifying" index, but instead include the leading field as well as the field you are constraining. This has the advantage of using the index in 2.6 without the (hacky) "hint" (which might hurt your performance if you later add another index, on {age:1,name:1}.
Query:
db.names.find({ name:{$lt:MaxKey ,$gt:MinKey}, age: {$gte: 21, $lte: 30}},
{_id:0, age:1, name:1}).explain()
2.6 explain:
{
"cursor" : "BtreeCursor name_1_age_1",
"isMultiKey" : false,
"n" : 6010,
"nscannedObjects" : 0,
"nscanned" : 6012,
"nscannedObjectsAllPlans" : 0,
"nscannedAllPlans" : 6012,
"scanAndOrder" : false,
"indexOnly" : true,
"nYields" : 46,
"nChunkSkips" : 0,
"millis" : 8,
"indexBounds" : {
"name" : [
[
{
"$minElement" : 1
},
{
"$maxElement" : 1
}
]
],
"age" : [
[
21,
30
]
]
},
"server" : "Asyas-MacBook-Pro.local:27017",
"filterSet" : false
}
2.4 explain (you have to add either hint({name:1,age:1}) or .sort({name:1,age:1}) to force use of the index:
{
"cursor" : "BtreeCursor name_1_age_1",
"isMultiKey" : false,
"n" : 6095,
"nscannedObjects" : 0,
"nscanned" : 6096,
"nscannedObjectsAllPlans" : 103,
"nscannedAllPlans" : 6199,
"scanAndOrder" : false,
"indexOnly" : true,
"nYields" : 0,
"nChunkSkips" : 0,
"millis" : 10,
"indexBounds" : {
"name" : [
[
{
"$minElement" : 1
},
{
"$maxElement" : 1
}
]
],
"age" : [
[
21,
30
]
]
},
"server" : "Asyas-MacBook-Pro.local:24800"
}
I added projection to show "indexOnly" is true in both cases, if you remove projection, the plans is identical but nscannedObjects becomes the same as n rather than 0.

Answer 3

This is really call about mongo "giving up" after it realizes that the matches that are possible have been exhausted, and there will be no more items to match. The index is helping here by providing some bounds.
Actually this is the part that explains it:
"indexBounds" : {
"age" : [
[
21,
30
]
]
Since that is a field in the selected index, mongo has set bounds on where to start and where to end. So it only needs to read the documents that fall in between those bounds. The list of those documents is a part of the index.
Here is some code to easily reproduce:
people = [
"Marc", "Bill", "George", "Eliot", "Matt", "Trey", "Tracy",
"Greg", "Steve", "Kristina", "Katie", "Jeff"];
for (var i=0; i<200000; i++){
name = people[Math.floor(Math.random()*people.length)];
age = Math.floor(Math.random() * ( 50 - 18 + 1)) + 18;
boolean = [true,false][Math.floor(Math.random()*2)];
db.names.insert({
name: name,
age: age,
boolean: boolean,
added: new Date()
});
}
Adding the index:
db.names.ensureIndex( { name: 1, age: 1 });
And running the query:
db.names.find({
age: {$gte: 21, $lte: 30}
}).hint( { name: 1, age: 1 } ).explain()
Will get you results something like:
{
"cursor" : "BtreeCursor name_1_age_1",
"isMultiKey" : false,
"n" : 60226,
"nscannedObjects" : 60226,
"nscanned" : 60250,
"nscannedObjectsAllPlans" : 60226,
"nscannedAllPlans" : 60250,
"scanAndOrder" : false,
"indexOnly" : false,
"nYields" : 0,
"nChunkSkips" : 0,
"millis" : 227,
"indexBounds" : {
"name" : [
[
{
"$minElement" : 1
},
{
"$maxElement" : 1
}
]
],
"age" : [
[
21,
30
]
]
},
"server" : "ubuntu:27017"
}
So you can see that nscanned is higher than n yet less than the total documents. Which goes to show that the "bounds" were taken into consideration, and when outside of those bounds the match will return no more.
What is happening here? Why are less documents returned than are in the collection? Basically the essence of the question.
So consider this. You know that your compound index does not specify the field that is being matched first. But do not think of a compound index as a joined statement (more later) think of it as a list of elements. So it does have discrete values of the age field in there somewhere.
Next, we have a large number of documents to go through. So the optimizer is going to naturally hate to scan. But since we didn't give a condition to match or range on the first element of the compound index it's going to have to start doing so. So we begin to chug along. Now for a more visual demonstration.
miss, miss, miss, hit, hit, "lots of hits", miss, miss, "more misses", STOP.
Why the STOP. This is an optimize condition. Since we had the discrete values of age, and determined a bounds exists within the chosen index the question gets asked.
"Wait just one moment. I should be scanning these in order, but I just got a load of misses. I think I missed my bus stop".
Colloquially speaking, that is exactly what the optimizer does. And realizing it just when past the point where it will find any more matches it "jumps off the bus" and walks back home with the result. So the matches have been "exhausted" past the point where it can reasonably determine that there will be any further matches.
Of course if the index order of fields was flipped, so that age was first or the only consideration, then nscanned and n would match as there was a distinctly clear start and end point.
The purpose of explain is that it can explain what is happening when the query statement is analysed. In this has it has "told" you that since your query conditions asked for a range in your query and that range can be matched in an index then it will use that information in scanning the results.
So what happened here was, that given the bounds on the index that was being used to search, the optimiser had an "idea" of where to start and then where to end. And given the factors, once matches "No longer seem" to be found the matching is exhausted and the process "gives up" considering it was not going to find anything else that resides out of those bounds.
Any other conditions such as where you were wondering if you had documents without a username would be irrelevant and would only apply if the index was "sparse", and then they would not be in the index at all. And this is not a sparse index nor are there nulls. But that was never the important part of understanding why the query did not go through all the documents.
What you may be struggling with is that this is a compound index. But that is not like an index on "concatenated" terms, so the index has to scan the username + the age. Instead both the fields can be considered, as long as they can be considered in "order". Which is why the explain output shows that this has matched those bounds.
The documentation is not stellar on this.but does define what indexBounds means.
EDIT
The final statement is that that is the confirmed and intended behavior, and the claimed "Bug" is actually not a bug, but rather one that was introduced in the 2.6 release, which includes a major re-factor of the Index interface code. See SERVER-13197 which was reported by me.
So the same results as shown can be achieved in 2.6 by altering the query like so:
db.names.find({
"name": { "$gt": MinKey, "$lt": MaxKey },
"age": {$gte: 21, $lte: 30}
}).sort( { "name": 1, "age": 1 } ).explain()
{
"cursor" : "BtreeCursor name_1_age_1",
"isMultiKey" : false,
"n" : 60770,
"nscannedObjects" : 60770,
"nscanned" : 60794,
"nscannedObjectsAllPlans" : 60770,
"nscannedAllPlans" : 60794,
"scanAndOrder" : false,
"indexOnly" : false,
"nYields" : 474,
"nChunkSkips" : 0,
"millis" : 133,
"indexBounds" : {
"name" : [
[
{
"$minElement" : 1
},
{
"$maxElement" : 1
}
]
],
"age" : [
[
21,
30
]
]
},
"server" : "ubuntu:27017",
"filterSet" : false
}
This shows that by including the MinKey and MaxKey values over the first index element, then the optimizer correctly detects that the bounds on the second element can be used in the way that has been already described.
Of course, this is not required in earlier versions as the use of the sort is enough to both specify this index and for the optimizer to detect the bounds correctly without the explicit modification to the query.
As noted on the issue, the fix for this is intended for release in a future version.