Reducing latency of MongoDB querying for large datasets - mongodb

Question

I have the following code for querying ideas in my web app. I'm trying to figure out how I can optimize this code to run faster. Currently, for 100 entries it takes around 350ms, for 1000 entries it takes around 1000ms, and for 10,000 entries it takes around 6000ms for it to send a response. I've looked into indexing, but it doesn't seem to make it any faster. I used MongoLabs to create the index on tool, username, and rating_level for now. I was thinking about using Redis, but not sure how to implement it.
app.get('/api/ideas', function(req, res, next) {
var query;
if (req.query.tool) {
query = Idea.find({ tool : req.query.tool });
}
else if (req.query.username) {
query = Idea.find({ username : req.query.username });
}
else {
query = Idea.find();
}
if(req.query.sortRank) {
query = query.sort({rating_level: req.query.sortRank});
}
else if(req.query.sortDate) {
query = query.sort({datetime: req.query.sortDate});
}
else {
query = query.sort({rating_level: -1});
}
query.exec(function(err, ideas) {
if (err) return next(err);
res.send(ideas);
});

Answer 1

You are generating several different query patterns and sort combinations in your code:
tool, rating_level
tool, datetime
username, rating_level
username, datetime
(none), rating_level
(none), datetime
Performant queries will match the keys and sort order (asc/desc) defined for a compound index, or a proper subset of the keys from left to right.
If you only have a single index on "tool, username, rating_level", this will not efficiently support any of your queries as listed above.
The index you've suggested would be useful for queries on:
tool
tool, username
tool, username, rating_level
To return results without having to perform an in-memory sort, the sort order in the compound index should match the sort order in the queries.
For example:
db.collection.find({tool:1}).sort({rating_level: -1})
...would ideally want an index of
{tool:1, rating_level:-1}
Given it looks like your sort order could include both ascending & descending variations, there are potentially a lot of indexes required to cover all your query variations efficiently.
You'll have to consider the tradeoffs for query performance vs index maintenance and storage overhead.
Some approaches to consider include:
Add all the required indexes. With only 10,000 documents it's probably reasonable to do so.
Optimise index covered for your most common queries. Use the explain() command to get a better understanding of how queries are using indexes.
Adjust your data model to simplify indexes. For example, you could consider using a multikey array so that tool and username are key/value pairs that can be included in the same index.
Adjust your application user interface to simplify the number of query and sort permutations.
Some helpful references:
Use Indexes to Sort Query Results
Optimizing MongoDB Compound Indexes
Indexing Schemaless Documents in Mongo