Mongodb query excessively slow - mongodb

Question

I have a collection of tweets, with indexes on userid and tweeted_at (date). I want to find the dates of the oldest and newest tweets in the collection for a user, but the query runs very slowly.
I used explain, and here's what I got. I tried reading the documentation for explain, but I don't understand what is going on here. Is the explain just on the sort? If so, why does it take so long when it's using the index?
> db.tweets.find({userid:50263}).sort({tweeted_at:-1}).limit(1).explain(1)
{
"cursor" : "BtreeCursor tweeted_at_1 reverse",
"isMultiKey" : false,
"n" : 0,
"nscannedObjects" : 12705,
"nscanned" : 12705,
"nscannedObjectsAllPlans" : 12705,
"nscannedAllPlans" : 12705,
"scanAndOrder" : false,
"indexOnly" : false,
"nYields" : 188,
"nChunkSkips" : 0,
"millis" : 7720,
"indexBounds" : {
"tweeted_at" : [
[
{
"$maxElement" : 1
},
{
"$minElement" : 1
}
]
]
},
"allPlans" : [
{
"cursor" : "BtreeCursor tweeted_at_1 reverse",
"isMultiKey" : false,
"n" : 0,
"nscannedObjects" : 12705,
"nscanned" : 12705,
"scanAndOrder" : false,
"indexOnly" : false,
"nChunkSkips" : 0,
"indexBounds" : {
"tweeted_at" : [
[
{
"$maxElement" : 1
},
{
"$minElement" : 1
}
]
]
}
}
],
"server" : "adams-server:27017",
"filterSet" : false,
"stats" : {
"type" : "LIMIT",
"works" : 12807,
"yields" : 188,
"unyields" : 188,
"invalidates" : 0,
"advanced" : 0,
"needTime" : 12705,
"needFetch" : 101,
"isEOF" : 1,
"children" : [
{
"type" : "FETCH",
"works" : 12807,
"yields" : 188,
"unyields" : 188,
"invalidates" : 0,
"advanced" : 0,
"needTime" : 12705,
"needFetch" : 101,
"isEOF" : 1,
"alreadyHasObj" : 0,
"forcedFetches" : 0,
"matchTested" : 0,
"children" : [
{
"type" : "IXSCAN",
"works" : 12705,
"yields" : 188,
"unyields" : 188,
"invalidates" : 0,
"advanced" : 12705,
"needTime" : 0,
"needFetch" : 0,
"isEOF" : 1,
"keyPattern" : "{ tweeted_at: 1.
0 }",
"boundsVerbose" : "field #0['twe
eted_at']: [MaxKey, MinKey]",
"isMultiKey" : 0,
"yieldMovedCursor" : 0,
"dupsTested" : 0,
"dupsDropped" : 0,
"seenInvalidated" : 0,
"matchTested" : 0,
"keysExamined" : 12705,
"children" : [ ]
}
]
}
]
}
}
>
> db.tweets.getIndexes()
[
{
"v" : 1,
"key" : {
"_id" : 1
},
"name" : "_id_",
"ns" : "honeypot.tweets"
},
{
"v" : 1,
"unique" : true,
"key" : {
"tweet_id" : 1
},
"name" : "tweet_id_1",
"ns" : "honeypot.tweets",
"dropDups" : true
},
{
"v" : 1,
"key" : {
"tweeted_at" : 1
},
"name" : "tweeted_at_1",
"ns" : "honeypot.tweets"
},
{
"v" : 1,
"key" : {
"keywords" : 1
},
"name" : "keywords_1",
"ns" : "honeypot.tweets"
},
{
"v" : 1,
"key" : {
"user_id" : 1
},
"name" : "user_id_1",
"ns" : "honeypot.tweets"
}
]
>

Answer 1

By looking at the cursor field you can see which index was used:
"cursor" : "BtreeCursor tweeted_at_1 reverse",
BtreeCursor indicates that the query used an index and tweeted_at_1 reverse is the name of the index that was used.
You should check the documentation for each field in the explain to see a detailed description for each field.
Your query lasted 7720 ms (milis) and 12705 documents were scanned(nscanned).
The query is slow because MongoDB scanned all documents that matched your criteria. This happened because MongoDB didn't use your index for querying, but for sorting the data.
To create an index that will be used for querying and sorting, you should create a compound index. Compound index is a single index structure that references multiple fields. You can create a compound index with up to 31 field. You can create a compound index like this (order or fields is important):
db.tweets.ensureIndex({userid: 1, tweeted_at: -1});
This index will be used for searching on userid field and to sort by tweeted_at field.
You can read and see more examples about adding indexes for sorting here.
Edit
If you have other indexes MongoDB is maybe using them. When you're testing query performance you can use hint to use a specific index.
When testing performance of your queries you should always do multiple tests and take an approx. of the results.
Also, if your queries are slow, even when using indexes, then I would check if you have enough memory on the server. Loading the data from disk is order of magnitude slower then loading from the memory. You should always ensure that you have enough RAM, so that all of your data and indexes fit in memory.

Answer 2

Looks like you need to create an index on tweeted_at and userid.
db.tweets.ensureIndex({'tweeted_at':1, 'userid':1})
That should make the query very quick indeed (but at a cost of storage and insert time)