MongoDB find slow on subarray query - mongodb

Question

I have a mondoDB collection as follows which contains almost a million entries:
{
_id: 'object id',
link: 'a url',
channels: [ array of ids ]
pubDate: Date
}
I have the following query that I perform pretty often:
db.articles.find({ $and: [ { pubDate: { $gte: new Date(<some date>) } }, { channels: ObjectId(<some object id>) } ] })
The query is extremely slow even though I have certain indexes in place. Recently, I ran an explain on it and here is the result:
{
"cursor" : "BtreeCursor pubDate_-1_channels_1",
"isMultiKey" : true,
"n" : 2926,
"nscannedObjects" : 4245,
"nscanned" : 52611,
"nscannedObjectsAllPlans" : 8125,
"nscannedAllPlans" : 56491,
"scanAndOrder" : false,
"indexOnly" : false,
"nYields" : 5,
"nChunkSkips" : 0,
"millis" : 5378,
"indexBounds" : {
"pubDate" : [
[
ISODate("0NaN-NaN-NaNTNaN:NaN:NaNZ"),
ISODate("2016-03-04T21:00:00Z")
]
],
"channels" : [
[
ObjectId("54239b9477456cf777dd0d31"),
ObjectId("54239b9477456cf777dd0d31")
]
]
}
}
Looks like it is using the correct index but still taking more than 5 seconds to run.
Am I missing something here? Is something wrong with my index?
Here are the indexes on the collection btw:
[
{
"v" : 1,
"name" : "_id_",
"key" : {
"_id" : 1
},
"ns" : "dbname.articles"
},
{
"v" : 1,
"name" : "pubDate_-1_channels_1",
"key" : {
"pubDate" : -1,
"channels" : 1
},
"ns" : "dbname.articles",
"background" : true
},
{
"v" : 1,
"name" : "pubDate_-1",
"key" : {
"pubDate" : -1
},
"ns" : "dbname.articles",
"background" : true
},
{
"v" : 1,
"name" : "link_1",
"key" : {
"link" : 1
},
"ns" : "dbname.articles",
"background" : true
}
]
Here is what I see when I run stats on the collection:
{
"ns" : "dbname.articles",
"count" : 2402741,
"size" : 2838416144,
"avgObjSize" : 1181.3242226274076,
"storageSize" : 3311443968,
"numExtents" : 21,
"nindexes" : 4,
"lastExtentSize" : 862072832,
"paddingFactor" : 1.000000000020535,
"systemFlags" : 0,
"userFlags" : 0,
"totalIndexSize" : 775150208,
"indexSizes" : {
"_id_" : 100834608,
"pubDate_-1_channels_1" : 180812240,
"pubDate_-1" : 96378688,
"link_1" : 397124672
},
"ok" : 1
}

Answer 1

So, according to db.my_collection.stats(), indexed fields takes 0.77gb ("totalIndexSize" : 775150208 bytes), and your collection takes 3.31gb ("storageSize" : 3311443968 bytes). You mentioned that your instance uses 1.5gb of RAM.
MongoDB can therefore keep all indexes in memory, but does not have enough memory to hold all the documents. So, when it needs to do a query on documents that are not loaded in memory, it is slower. I bet that if you run the same query twice, it would take much less time since the necessary documents would already be loaded in memory.
I would recommend trying with 5gb of RAM. Do a couple of queries so that all the documents are loaded in memory, and then compare the speeds.

Answer 2

Your index { pubDate : -1, channel : 1 } looks good to me.
I would try { channel : 1 , pubDate : -1 } though.
The reason for this suggestion is the following:
Notice that indexes have order.
The index you are using is pubDate_-1_channels_1, which will be different from the index channels_1_pubDate_-1 (in reversed order).
Depending on the number of channels you have, I would expect that one index should be more efficient than another for your query.
See the manual on prefixes for details.