MongoDB high cpu usage/long read time - mongodb

Question

I am a new user of MongoDB, and I am hoping to get pointed in the right direction. I will provide any further needed information I have missed as this question develops.
I am using a Perl program to upload and annotate/modify documents to a/in a MongoDB database via the MongoDB cpan module. Indexes are being used (I believe) for this program, but the problem I have is that reading from MongoDB takes increasingly long. Based on mongotop, it takes ~500 ms to read and only 10-15 ms to write. After allowing the program to run for a considerable amount of time, the read time increases significantly, taking more then 3000+ ms after many hours of running.
Monitoring the program while its running using top, Perl starts out at around 10-20% CPU usage and MongoDB starts at 70-90% CPU usage. While running, within a few minutes Perl drops below 5% and mongoDB is 90-95%. After running for a much longer period of time (12+ hours), MongoDB is ~98% CPU usage while Perl is around 0.3%, but only pops up every 5-10 seconds in top.
Based on this trend, an indexing issue seems very likely but I am not sure how to check this, and all I know is that the appropriate indexes are at least made, but not necessarily being used.
Additional information:
$ ulimit -a
core file size (blocks, -c) 0
data seg size (kbytes, -d) unlimited
scheduling priority (-e) 0
file size (blocks, -f) unlimited
pending signals (-i) 19209
max locked memory (kbytes, -l) 64
max memory size (kbytes, -m) unlimited
open files (-n) 1024
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority (-r) 0
stack size (kbytes, -s) 8192
cpu time (seconds, -t) unlimited
max user processes (-u) 19209
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited
As the program runs, I see that the indexSize and dataSize change (via db.stats() in the Mongo shell), which makes me think that they are at least being used to some degree
Is this something that could be affected by the power of my computer? I am under the impression that indexing should make a lot of this process very manageable for the computer

Answer 1

That sounds a lot like it could be doing a collection scan rather than using the index. I.e. as your collection grows, the reads are getting slower.
If you're using the find method, you can run explain on the resulting cursor to get information on how the query would execute.
Here's a trivial example:
use MongoDB;
use JSON::MaybeXS;
my $coll = MongoDB->connect->ns("test.foo");
$coll->drop();
$coll->insert_one({x => $_}) for 1 .. 1000;
my $cursor = $coll->find({x => 42});
my $diag = $cursor->explain;
my $json = JSON::MaybeXS->new(
allow_blessed => 1, convert_blessed => 1, pretty => 1, canonical => 1
);
print $json->encode($diag->{queryPlanner}{winningPlan});
Looking at just the 'winningPlan' part of the output you can see 'COLLSCAN':
{
"direction" : "forward",
"filter" : {
"x" : {
"$eq" : 42
}
},
"stage" : "COLLSCAN"
}
Now I'll do it again, but first creating an index on 'x' before the insertions with $coll->indexes->create_one([x => 1]). You can see in the output that the query plan is now using the index (IXSCAN).
{
"inputStage" : {
"direction" : "forward",
"indexBounds" : {
"x" : [
"[42, 42]"
]
},
"indexName" : "x_1",
"indexVersion" : 2,
"isMultiKey" : false,
"isPartial" : false,
"isSparse" : false,
"isUnique" : false,
"keyPattern" : {
"x" : 1
},
"multiKeyPaths" : {
"x" : []
},
"stage" : "IXSCAN"
},
"stage" : "FETCH"
}
There's a lot more you can discover from the full 'explain' output. You can watch a great video from MongoDB World 2016 to learn more about it: Deciphering Explain Output.