Compound index not reducing docsExamined - mongodb

My mongodb database has documents for each minute for each device. I have a query that basically searches documents matching with month field.
db.data.aggregate([
{
"$match":{ dId:14,month:6}
},
{
"$project" :{
dayOfMonth_agg :{$dayOfMonth : "$ts"},
month : "$month",
year_agg : {$year : "$ts"},
hour_agg : {$hour : "$ts"},
a0 :1,
a1:1,
d2:1,
f0:1,
dId:1
}
},
{
"$match":{year_agg:2017}
},
{
$group :{
_id : "$dayOfMonth_agg",
totalEnergy : { $sum: { $multiply: [{$subtract:["$a1","$a0"]},"$f0"] } },
avg_d0:{$avg:"$a0"},
avg_d1:{$avg:"$a1"},
avg_d2:{$avg:"$d2"},
}
},
{
$sort:{"_id":1}
}
])
I wanted to optimise query performance so I added compound index as:
db.data.createIndex({dId:1,month:1})
Now, in execution stats, docsExamined are equal to that of stats when single index on dId was created.
Exec stats with single Index on dId:
"keysExamined" : 103251,
"docsExamined" : 103251,
"hasSortStage" : true,
"cursorExhausted" : true,
"numYield" : 814,
"locks" : {
"Global" : {
"acquireCount" : {
"r" : NumberLong(1654)
}
},
"Database" : {
"acquireCount" : {
"r" : NumberLong(827)
}
},
"Collection" : {
"acquireCount" : {
"r" : NumberLong(826)
}
}
},
"nreturned" : 8,
"responseLength" : 770,
"protocol" : "op_command",
"millis" : 656,
"planSummary" : "IXSCAN { dId: 1 }",
"ts" : ISODate("2017-11-15T08:41:50.782Z"),
Exec stats with compound index (dId and month)
"keysExamined" : 103251,
"docsExamined" : 103251,
"hasSortStage" : true,
"cursorExhausted" : true,
"numYield" : 810,
"locks" : {
"Global" : {
"acquireCount" : {
"r" : NumberLong(1646)
}
},
"Database" : {
"acquireCount" : {
"r" : NumberLong(823)
}
},
"Collection" : {
"acquireCount" : {
"r" : NumberLong(822)
}
}
},
"nreturned" : 8,
"responseLength" : 770,
"protocol" : "op_command",
"millis" : 678,
"planSummary" : "IXSCAN { dId: 1, month: 1 }",
"ts" : ISODate("2017-11-15T08:45:38.536Z"),
"client" : "127.0.0.1",
"appName" : "MongoDB Shell",
"allUsers" : [ ],
"user" : ""
As per my knowledge, the docsExamined should have reduced in comparison with single index.
Why is it so?
Edit: Please refer to first comment for the answer

Related

MongoDB geospatial index on $center

Collection Schema
{
"_id" : ObjectId("5d3562bf1b48d90ea4b06a74"),
"name" : "19",
"location" : {
"type" : "Point",
"coordinates" : [
50.0480208,
30.5239127
]
}
}
Indexes
> db.places.getIndexes()
[
{
"v" : 2,
"key" : {
"_id" : 1
},
"name" : "_id_",
"ns" : "test.places"
},
{
"v" : 2,
"key" : {
"location" : "2dsphere"
},
"name" : "location_2dsphere",
"ns" : "test.places",
"2dsphereIndexVersion" : 3
}
There is 2 milion documents is stored in collection.
First I ran query like this.
db.places.find({ location: {$geoWithin: { $center: [[60.0478308, 40.5237227], 10] } }})
But it takes 2 seconds. So I examine query via explain().
> db.places.find({ location: {$geoWithin: { $center: [[60.0478308, 40.5237227], 10] } }}).explain('executionStats')
{
"queryPlanner" : {
"plannerVersion" : 1,
"namespace" : "test.places",
"indexFilterSet" : false,
"parsedQuery" : {
"location" : {
"$geoWithin" : {
"$center" : [
[
60.0478308,
40.5237227
],
10
]
}
}
},
"winningPlan" : {
"stage" : "COLLSCAN",
"filter" : {
"location" : {
"$geoWithin" : {
"$center" : [
[
60.0478308,
40.5237227
],
10
]
}
}
},
"direction" : "forward"
},
"rejectedPlans" : [ ]
},
"executionStats" : {
"executionSuccess" : true,
"nReturned" : 1414213,
"executionTimeMillis" : 2093,
"totalKeysExamined" : 0,
"totalDocsExamined" : 2000000,
"executionStages" : {
"stage" : "COLLSCAN",
"filter" : {
"location" : {
"$geoWithin" : {
"$center" : [
[
60.0478308,
40.5237227
],
10
]
}
}
},
"nReturned" : 1414213,
"executionTimeMillisEstimate" : 1893,
"works" : 2000002,
"advanced" : 1414213,
"needTime" : 585788,
"needYield" : 0,
"saveState" : 15681,
"restoreState" : 15681,
"isEOF" : 1,
"invalidates" : 0,
"direction" : "forward",
"docsExamined" : 2000000
}
},
"serverInfo" : {
"host" : "Johnui-iMac",
"port" : 27017,
"version" : "4.0.3",
"gitVersion" : "7ea530946fa7880364d88c8d8b6026bbc9ffa48c"
},
"ok" : 1
}
You know that query stage is COLLSCAN.
I wonder that, I already created index for location fields, but it seems doesnt' work.
So I create more indexes.
"v" : 2,
"key" : {
"location.coordinates" : 1
},
"name" : "location.coordinates_1",
"ns" : "test.places"
},
{
"v" : 2,
"key" : {
"location" : 1
},
"name" : "location_1",
"ns" : "test.places"
}
But it doesn't work too.
Is there any issue on my index configuration?
You seem to have created a 2dsphere Index on your location, but the MongoDB docs on $centre specify that:
Only the 2d geospatial index supports $center.
Therefore, I suggest you create a 2d index on the location field and the scan will be performed using this index

MongoDB very slow update on non-indexed field

I am trying to debug to find a root cause on why single field update taking 60+ seconds, but not able to figure out. Will be great if you help me to get some direction on how to proceed.
Scenario: updating a non-indexed date field on a collection.
MongoDB Version : 3.0.12. Storage engine: WiredTiger
The query is using the proper index. scanned and modified are the same number of docs.
system.pofile document :
{
"_id" : ObjectId("5aa2e63b27001947449f1eed"),
"op" : "update",
"ns" : "abc.xyz",
"query" : {
"aId" : "5aa298dce4b0ef9feffe70e1",
},
"updateobj" : {
"$set" : {
"psdt" : ISODate("2018-03-09T17:36:31.277Z")
}
},
"nscanned" : 20,
"nscannedObjects" : 20,
"nMatched" : 20,
"nModified" : 20,
"keyUpdates" : 0,
"writeConflicts" : 0,
"numYield" : 5,
"locks" : {
"Global" : {
"acquireCount" : {
"r" : NumberLong(26),
"w" : NumberLong(26)
}
},
"Database" : {
"acquireCount" : {
"w" : NumberLong(26)
}
},
"Collection" : {
"acquireCount" : {
"w" : NumberLong(6)
}
},
"oplog" : {
"acquireCount" : {
"w" : NumberLong(20)
}
}
},
"millis" : 69765
}

mongodb find query getmore operation hangs

We are running MongoDB 3.0.2 on Linux.
The "getmore" operation for a find query periodically hangs.
The operation is available currentOp(), and one item I cannot explain is that all the acquireCount values in lockStats continue to grow while the query hangs (for Global, MMAPV1Journal, Database, and Collection).
This is the operation in question, at this point running for more than 1000 secs, returned by db.currentOp() :
{
"desc" : "conn60",
"threadId" : "0x2a99ee0",
"connectionId" : 60,
"opid" : 67792,
"active" : true,
"secs_running" : 1098,
"microsecs_running" : NumberLong(1098289543),
"op" : "getmore",
"ns" : "dbName.collectionName",
"query" : {
"d" : {
"$gt" : ISODate("2016-03-13T18:00:00.261Z"),
"$lt" : ISODate("2016-03-14T22:45:17.718Z")
},
"cc" : "US",
"dc" : {
"$in" : [
"26",
"31",
"17",
"29",
"35"
]
},
"pr" : {
"$gte" : 4
}
},
"client" : "10.0.0.111:33670",
"numYields" : 317557,
"locks" : {
},
"waitingForLock" : false,
"lockStats" : {
"Global" : {
"acquireCount" : {
"r" : NumberLong(635114)
}
},
"MMAPV1Journal" : {
"acquireCount" : {
"r" : NumberLong(317557)
}
},
"Database" : {
"acquireCount" : {
"r" : NumberLong(317557)
}
},
"Collection" : {
"acquireCount" : {
"R" : NumberLong(317557)
}
}
}
}
Any insights would be much appreciated ! Thank you in advance.

execStats is always empty in MongoDB "aggregate" commands profiling results

I am trying to profile the performance of an aggregation pipeline, specifically checking whether indices are used, how many objects are scanned, etc.
I'm setting the DB to full profiling:
db.setProfilingLevel(2)
But then in the db's 'system.profile' collection, in the result record for the aggregation command, the execStats is always empty.
Here is the full result for the command:
{
"op" : "command",
"ns" : "mydb.$cmd",
"command" : {
"aggregate" : "mycolection",
"pipeline" : [{
"$match" : {
"date" : {
"$gte" : "2013-11-26"
}
}
}, {
"$sort" : {
"user_id" : 1
}
}, {
"$project" : {
"user_id" : 1,
"_id" : 0
}
}, {
"$group" : {
"_id" : "$user_id",
"agg_val" : {
"$sum" : 1
}
}
}],
"allowDiskUse" : true
},
"keyUpdates" : 0,
"numYield" : 16,
"lockStats" : {
"timeLockedMicros" : {
"r" : NumberLong(3143653),
"w" : NumberLong(0)
},
"timeAcquiringMicros" : {
"r" : NumberLong(140),
"w" : NumberLong(3)
}
},
"responseLength" : 4990,
"millis" : 3237,
"execStats" : { },
"ts" : ISODate("2014-11-26T16:20:59.576Z"),
"client" : "127.0.0.1",
"allUsers" : [],
"user" : ""
}
Support execStats for aggregation command was added in mongo 3.4.

Aggregation framework performance on a 10M collection

I have a collection of 10M documents, that is a pre-aggregation of daily events.
A simple $group took more than 8s, is this performance normal ?
Some date from the profiler :
{
"op" : "command",
"ns" : "analytics.$cmd",
"command" : {
"aggregate" : "aggregation",
"pipeline" : [
{
"$group" : {
"_id" : "",
"hits" : {
"$sum" : "$hits"
}
}
}
]
},
"ntoreturn" : 1,
"keyUpdates" : 0,
"numYield" : 15,
"lockStats" : {
"timeLockedMicros" : {
"r" : NumberLong(17169805),
"w" : NumberLong(0)
},
"timeAcquiringMicros" : {
"r" : NumberLong(8582619),
"w" : NumberLong(294)
}
},
"responseLength" : 78,
"millis" : 8594,
"ts" : ISODate("2013-12-04T15:57:38.217Z"),
"client" : "127.0.0.1",
"allUsers" : [ ],
"user" : ""
}
Here is one single document
{
"_id" : ObjectId("529e21ee67e807418500daeb"),
"date" : ISODate("2012-09-19T00:00:00Z"),
"hits" : 1,
"infos" : {
"sourceValue" : NumberLong(1),
"eventType" : "createUser",
"sourceType" : "user",
"instance" : "xxx",
"targetType" : "user",
"targetValue" : NumberLong(15)
}
}