I have a large set of documents like this:
{
"_id" : ObjectId("572e33baf082e29c46cadb7b"),
"nodes" : [
[
39.2035598754883,
51.6601028442383
],
[
39.2038421630859,
51.6602439880371
],
[
39.2038688659668,
51.6602249145508
]
]
}
And I want to search documents by coordinates. My query is:
db.points.find({nodes: {$elemMatch: {$geoWithin:
{$box: [[39.1981, 51.660], [39.206, 51.664]]}}}})
I also added an index
db.points.createIndex( { "nodes": "2d" } )
but it has no effect. Collection stats:
{
"ns" : "base.points",
"count" : 215583,
"size" : 61338720,
"avgObjSize" : 284,
"storageSize" : 86310912,
"numExtents" : 10,
"nindexes" : 3,
"lastExtentSize" : 27869184,
"paddingFactor" : 1.0,
"systemFlags" : 0,
"userFlags" : 1,
"totalIndexSize" : 39759888,
"indexSizes" : {
"_id_" : 7006832,
"nodes_2d" : 26719168
},
"ok" : 1.0
}
And explain:
{
"cursor" : "BasicCursor",
"isMultiKey" : false,
"n" : 61,
"nscannedObjects" : 215583,
"nscanned" : 215583,
"nscannedObjectsAllPlans" : 215583,
"nscannedAllPlans" : 215583,
"scanAndOrder" : false,
"indexOnly" : false,
"nYields" : 1684,
"nChunkSkips" : 0,
"millis" : 1466,
"server" : "DEVELOP-PC:27017",
"filterSet" : false
}
The solution was very simple:
db.points.find({nodes: {$geoWithin: {$box: [[39.1981, 51.660],
[39.206, 51.664]]}}})
should be excluded from the query $elemMatch and the index will work correctly
Related
I'm running a mongoDB query and it's taking too long. I'm querying the collection "play_sessions" for the data of 9 users as seen in (1). My documents contain data for a gameplay session for a user as seen (2). I have an index on "user_id" and this index is being used as seen in the .explain() output in (3). My indexes in the .stats() output are shown in (4).
The mongoDB version is 2.6.1. There are approximately 4 million entires in "play_sessions" and 43,000 distinct users.
This example query takes around 2 min and the actual query of 800 users takes a lot longer. I'd like to know why this query is slow and what I can do to speed it up.
(1) The query:
db.play_sessions.find({user_id: {$in: users}}, {play_data:-1}
(2) Example document:
{
"_id" : 1903200,
"score" : 1,
"user_id" : 60538,
"time" : ISODate("2014-02-12T03:49:59.919Z"),
"level" : 1,
"user_attempt_no" : 2,
"game_id" : 181,
"play_data" : [
**Some JSON in here**
],
"time_sec" : 7.989
}
(3) .explain() output
{
"cursor" : "BtreeCursor user_id_1",
"isMultiKey" : false,
"n" : 13724,
"nscannedObjects" : 13724,
"nscanned" : 13732,
"nscannedObjectsAllPlans" : 14128,
"nscannedAllPlans" : 14140,
"scanAndOrder" : false,
"indexOnly" : false,
"nYields" : 4463,
"nChunkSkips" : 0,
"millis" : 123631,
"indexBounds" : {
"user_id" : [
[
41930,
41930
],
...,
[
67112,
67112
]
]
},
"server" : "...",
"filterSet" : false
}
(4) .stats() output for the collection:
{
"ns" : "XXX.play_sessions",
"count" : 3957328,
"size" : 318453446112,
"avgObjSize" : 80471,
"storageSize" : 319917328096,
"numExtents" : 169,
"nindexes" : 10,
"lastExtentSize" : 2146426864,
"paddingFactor" : 1,
"systemFlags" : 1,
"userFlags" : 1,
"totalIndexSize" : 1962280880,
"indexSizes" : {
"_id_" : 184205280,
"game_id_1" : 167681584,
"user_id_1" : 113997968,
"user_id_1_game_id_1_level_1_time_1" : 288972544,
"game_id_1_level_1" : 141027824,
"game_id_1_level_1_user_id_1_time_1" : 301645344,
"user_id_1_game_id_1_level_1" : 228674544,
"game_id_1_level_1_user_id_1" : 245549808,
"user_id_1_user_attempt_no_1" : 135958704,
"user_id_1_time_1" : 154567280
},
"ok" : 1
}
I use MongoDB 2.6.4.
My indexes looks like this:
{
"v" : 1,
"key" : {
"isFolder" : 1
},
"name" : "isFolder_1",
"ns" : "Tenant_51.files",
"background" : true
},
{
"v" : 1,
"key" : {
"isForeign" : 1
},
"name" : "isForeign_1",
"ns" : "Tenant_51.files",
"background" : true
},
My query looks like this:
db.files.find({ isFolder: true, isForeign: false }).explain(true)
For some reason, it chooses to use only 1 index (VERY SLOW: 680 seconds!!)
It looks like it does calculate the Complex Plan, however, decides not to use it, and I don't understand why.
Here is the execution plan:
{
"cursor" : "BtreeCursor isFolder_1",
"isMultiKey" : false,
"n" : 107441,
"nscannedObjects" : 110580,
"nscanned" : 110580,
"nscannedObjectsAllPlans" : 110689,
"nscannedAllPlans" : 110801,
"scanAndOrder" : false,
"indexOnly" : false,
"nYields" : 21056,
"nChunkSkips" : 0,
"millis" : 679121,
"indexBounds" : {
"isFolder" : [
[
true,
true
]
]
},
"allPlans" : [
{
"cursor" : "BtreeCursor isFolder_1",
"isMultiKey" : false,
"n" : 107441,
"nscannedObjects" : 110580,
"nscanned" : 110580,
"scanAndOrder" : false,
"indexOnly" : false,
"nChunkSkips" : 0,
"indexBounds" : {
"isFolder" : [
[
true,
true
]
]
}
},
{
"cursor" : "BtreeCursor isForeign_1",
"isMultiKey" : false,
"n" : 68,
"nscannedObjects" : 109,
"nscanned" : 110,
"scanAndOrder" : false,
"indexOnly" : false,
"nChunkSkips" : 0,
"indexBounds" : {
"isForeign" : [
[
false,
false
]
]
}
},
{
"cursor" : "Complex Plan",
"n" : 42,
"nscannedObjects" : 0,
"nscanned" : 111,
"nChunkSkips" : 0
}
],
"server" : "XXX",
"filterSet" : false,
"stats" : {
"type" : "KEEP_MUTATIONS",
"works" : 128743,
"yields" : 21056,
"unyields" : 21056,
"invalidates" : 13834,
"advanced" : 107441,
"needTime" : 3140,
"needFetch" : 18161,
"isEOF" : 1,
"children" : [
{
"type" : "FETCH",
"works" : 128743,
"yields" : 21056,
"unyields" : 21056,
"invalidates" : 13834,
"advanced" : 107441,
"needTime" : 3140,
"needFetch" : 18161,
"isEOF" : 1,
"alreadyHasObj" : 0,
"forcedFetches" : 0,
"matchTested" : 107441,
"children" : [
{
"type" : "IXSCAN",
"works" : 110581,
"yields" : 21056,
"unyields" : 21056,
"invalidates" : 13834,
"advanced" : 110580,
"needTime" : 1,
"needFetch" : 0,
"isEOF" : 1,
"keyPattern" : "{ isFolder: 1 }",
"isMultiKey" : 0,
"boundsVerbose" : "field #0['isFolder']: [true, true]",
"yieldMovedCursor" : 0,
"dupsTested" : 0,
"dupsDropped" : 0,
"seenInvalidated" : 0,
"matchTested" : 0,
"keysExamined" : 110580,
"children" : [ ]
}
]
}
]
}
}
From the MongoDB docs about indexing:
MongoDB can only use one index to support any given operation.
The solution, however, is as easy as the explanation: Use a compound index.
db.files.ensureIndex({isFolder:1,isForeign:1})
Hey I want to evaluate the performance of index intersection but I'm not able to get an intersection between two indices.
I've inserted some dummy records into my DB along this manual.
http://docs.mongodb.org/manual/core/index-intersection/
Insert code:
for(var i=0;i<1000;i++){
for(var j=0;j<100;j++){
db.t.insert({item:"abc"+i,qty:j})
}
}
Indices:
[
{
"v" : 1,
"key" : {
"_id" : 1
},
"name" : "_id_",
"ns" : "db.t"
},
{
"v" : 1,
"key" : {
"qty" : 1
},
"name" : "qty_1",
"ns" : "db.t"
},
{
"v" : 1,
"key" : {
"item" : 1
},
"name" : "item_1",
"ns" : "db.t"
}
]
Query:
db.t.find({item:"abc123",qty:{$gt:15}}).explain()
Result of explain:
{
"cursor" : "BtreeCursor item_1",
"isMultiKey" : false,
"n" : 84,
"nscannedObjects" : 100,
"nscanned" : 100,
"nscannedObjectsAllPlans" : 201,
"nscannedAllPlans" : 305,
"scanAndOrder" : false,
"indexOnly" : false,
"nYields" : 2,
"nChunkSkips" : 0,
"millis" : 1,
"indexBounds" : {
"item" : [
[
"abc123",
"abc123"
]
]
},
"server" : "brews18:27017",
"filterSet" : false
}
My question is why mongo is only using item as an index an does not use an intersection.
Thanks in advance
Well it actually does even though it does not in this case. To really see what is happening you need to look at the "verbose" form of explain, by adding true:
db.t.find({item:"abc123",qty:{$gt:15}}).explain(true)
{
"cursor" : "BtreeCursor item_1",
"isMultiKey" : false,
"n" : 84,
"nscannedObjects" : 100,
"nscanned" : 100,
"nscannedObjectsAllPlans" : 201,
"nscannedAllPlans" : 304,
"scanAndOrder" : false,
"indexOnly" : false,
"nYields" : 2,
"nChunkSkips" : 0,
"millis" : 2,
"indexBounds" : {
"item" : [
[
"abc123",
"abc123"
]
]
},
"allPlans" : [
{
"cursor" : "BtreeCursor item_1",
"isMultiKey" : false,
"n" : 84,
"nscannedObjects" : 100,
"nscanned" : 100,
"scanAndOrder" : false,
"indexOnly" : false,
"nChunkSkips" : 0,
"indexBounds" : {
"item" : [
[
"abc123",
"abc123"
]
]
}
},
{
"cursor" : "BtreeCursor qty_1",
"isMultiKey" : false,
"n" : 0,
"nscannedObjects" : 101,
"nscanned" : 102,
"scanAndOrder" : false,
"indexOnly" : false,
"nChunkSkips" : 0,
"indexBounds" : {
"qty" : [
[
15,
Infinity
]
]
}
},
{
"cursor" : "Complex Plan",
"n" : 0,
"nscannedObjects" : 0,
"nscanned" : 102,
"nChunkSkips" : 0
}
],
Cut short, but the last part is what you are looking for. As explained in the manual, the appearance of "Complex Plan" means an intersection is being used.
{
"cursor" : "Complex Plan",
"n" : 0,
"nscannedObjects" : 0,
"nscanned" : 102,
"nChunkSkips" : 0
}
The only case here is that while it is being "looked at" it is not being chosen by the optimizer in this case as the most "optimal" query. So the optimizer is saying that in fact the plan using just the one selected index, is the one that will complete in the most responsive fashion.
So while the "intersection" was considered, it was not the "best fit" and the single index was chosen.
I have a mongodb collection named rooms, and it has a 2d index for field location. I've queried like this:
db.rooms.find( { "location" : { "$near" : { "latitude" : 37.3356135, "longitude" : 127.12383030000001 } }, "status": "open", "updated" : { "$gt" : ISODate("2014-06-03T15:34:22.213Z") }}).explain()
The result:
{
"cursor" : "GeoSearchCursor",
"isMultiKey" : false,
"n" : 7,
"nscannedObjects" : 143247,
"nscanned" : 143247,
"nscannedObjectsAllPlans" : 143247,
"nscannedAllPlans" : 143247,
"scanAndOrder" : false,
"indexOnly" : false,
"nYields" : 0,
"nChunkSkips" : 0,
"millis" : 1457,
"indexBounds" : {
},
"server" : "ip-10-162-39-56:27017",
"filterSet" : false
}
Sometimes it takes more than 2000ms. But if I remove the $gt condition for updated field, the query is fast, about 5~30ms.
> db.rooms.find( { "location" : { "$near" : { "latitude" : 37.3356135, "longitude" : 127.12383030000001 } }, "status": "open"}).explain()
{
"cursor" : "GeoSearchCursor",
"isMultiKey" : false,
"n" : 100,
"nscannedObjects" : 1635,
"nscanned" : 2400,
"nscannedObjectsAllPlans" : 1635,
"nscannedAllPlans" : 2400,
"scanAndOrder" : false,
"indexOnly" : false,
"nYields" : 0,
"nChunkSkips" : 0,
"millis" : 22,
"indexBounds" : {
},
"server" : "ip-10-162-39-56:27017",
"filterSet" : false
}
I've tried compound index for {location:"2d", updated: -1} but it didn't work. How can I make this query faster?
I store a very long string to my document in mongodb server, my document looks like
{
"_id": ObjectId("5280efdbe4b062c93b582118"),
"viewID": 1,
"content": "a very long string"
}
and there are about hundreds of such documents in the same collection. I have indexed the collection.
The problem is that I just execute a very simple query
db.blog.find({viewID:1});
and then get a very long time (over several hours) to wait for the response, and no any error message displayed, server status is OK. But I just add db.blog.find({viewID:1}).limit(1); mongodb returns the result at once.
How can I do to solve the problem or improve the performance??
Here is my explain:
db.blog.find({viewID:1}).explain();
{
"cursor" : "BtreeCursor viewID_1",
"isMultiKey" : false,
"n" : 377,
"nscannedObjects" : 377,
"nscanned" : 377,
"nscannedObjectsAllPlans" : 377,
"nscannedAllPlans" : 377,
"scanAndOrder" : false,
"indexOnly" : false,
"nYields" : 0,
"nChunkSkips" : 0,
"millis" : 45,
"indexBounds" : {
"viewID" : [
[
1,
1
]
]
},
"server" : "Server:27017"
}
Here is colStat:
db.blog.stats();
{
"ns" : "***.blog",
"count" : 98582,
"size" : 2330759632,
"avgObjSize" : 23642.851960804204,
"storageSize" : 2958364672,
"numExtents" : 18,
"nindexes" : 2,
"lastExtentSize" : 778276864,
"paddingFactor" : 1,
"systemFlags" : 1,
"userFlags" : 0,
"totalIndexSize" : 6540800,
"indexSizes" : {
"_id_" : 4055296,
"viewID_1" : 2485504
},
"ok" : 1
}