MongoDB index effectiveness evaluation best practices - mongodb

Question

We would like to evaluate the effectiveness of our indexes in a MongoDB-based REST service setup. The idea is to populate a collection with a synthetic dataset (e.g. 10,000,000 documents) then run a load injector process doing random REST operations (each one involving a query at MongoDB layer) to evaluate which indexes are being used and statistical information about them (e.g. per index hit rate).
We have considered using explain() command or indexStats. However, regarding explain(), it has two problems: 1) it allows only evaluate the effectiveness of a single query, 2) it is difficult to use in a “black box” environment in which our load injector process interacts with the REST service on top on MongoDB but not MonoDB itself. Regarding indexStats, as far as I understand, it shows information about the index structure “on disk” but not about index usage.
Thus, which is the best way of doing that kind of test? Any procedure description or URL to information about the topic is highly welcomed.

Answer 1

You should read about performance profiling.
You can turn on the profiling on with:
db.setProfilingLevel(2);
Or if you don't want to much noise and you want to check only queries that took more than e.g. 20ms:
db.setProfilingLevel(1,20);
You can query the system.profile to get the information about slow queries, e.g. find all operations slower than 30 ms:
db.system.profile.find( { millis : { $gt : 30 } } ).pretty()
You can than manually profile each slow query with explain().
For real-time monitoring you can use mongotop and mongostat. You should also consider setting up MMS (free monitoring service). In MMS you can check btree hits/misses and compare them to other relevant statistics.
Edit
You can get the relevant indexCounters data by using serverStatus command:
db.serverStatus()