mongodb insert really slow - mongodb

i use mongodb to manage device log datas. Right now, it has over one million documents. the document contains more than 30 fields which combine with embed fields. Now, it's really slow when i insert new documents. The insert cost more than 1000ms. From the slow query ops, i get the logs like this:
{
"op" : "insert",
"ns" : "xxx.LogDeviceReport",
"query" : {
"_id" : ObjectId("xxxx"),
"deviceId" : ObjectId("xxxx"),
"en" : "xxxxxx",
"status" : 1,
'other fields, more than 30 fields...'
...
...
},
"ninserted" : 1,
"keyUpdates" : 0,
"writeConflicts" : 0,
"numYield" : 0,
"locks" : {
"Global" : {
"acquireCount" : {
"w" : NumberLong(2)
}
},
"MMAPV1Journal" : {
"acquireCount" : {
"w" : NumberLong(3)
}
},
"Database" : {
"acquireCount" : {
"w" : NumberLong(2)
}
},
"Collection" : {
"acquireCount" : {
"W" : NumberLong(1)
},
"acquireWaitCount" : {
"W" : NumberLong(1)
},
"timeAcquiringMicros" : {
"W" : NumberLong(1477481)
}
},
"oplog" : {
"acquireCount" : {
"w" : NumberLong(1)
}
}
},
"millis" : 977,
"execStats" : {
},
"ts" : ISODate("2016-08-02T22:01:01.270Z"),
"client" : "xxx.xxx.xxxx",
"allUsers" : [
{
"user" : "xxx",
"db" : "xxx"
}
],
"user" : "xx#xx"
}
I checked the index, like this:
[
{
"v" : 1,
"key" : {
"_id" : 1
},
"name" : "_id_",
"ns" : "xxx.LogDeviceReport"
},
{
"v" : 1,
"key" : {
"time" : 1
},
"name" : "time_1",
"ns" : "xxx.LogDeviceReport",
"expireAfterSeconds" : 604800,
"background" : true
}
]
Only an _id index and a ttl index by time, no any other indexes.
I guess the 'query' slow the operate. In mongodb doc, it tells that only the _id will be checked the unique, but in the logs, all fields in the 'query', does it matter?
if not this reason, what makes it so slow? Can any one help me ?

If you are using mongodb 3+ you can consider using WiredTiger as storage engine than MMAPV1 which is being used in your case.
I have personally saw a 4x improvement when I have inserted up to 156000 documents in a single go.
MMAPV1 took around 40 min and when I switched to WiredTiger same task was completed in 10 min.
Please check this link from MongoDB blog for more information
Note :: This is only from MongoDB 3.0 +

Related

MongoDB stuck creating index with "Index Build: draining writes received during build" message

I have 4 rows in the test collection:
{ "_id" : ObjectId("5f4ce50e19b13337216dd477"), "test" : 1 }
{ "_id" : ObjectId("5f4ce50e19b13337216dd478"), "test" : 2 }
{ "_id" : ObjectId("5f4ce50e19b13337216dd479"), "test" : 3 }
{ "_id" : ObjectId("5f4ce50e19b13337216dd47a"), "test" : 4 }
After running db.test.createIndex({test:1},{background:1}); to create an index, it just hangs. It was hanging for at least a few hours. Here is what I found in the db.currentOp() about this operation:
{
"type" : "op",
"host" : "HOSTNAME:27017",
"desc" : "IndexBuildsCoordinatorMongod-13",
"active" : true,
"currentOpTime" : "2020-08-31T12:11:13.159+00:00",
"opid" : 8721867,
"secs_running" : NumberLong(20),
"microsecs_running" : NumberLong(20888590),
"op" : "command",
"ns" : "test.test",
"command" : {
"createIndexes" : "test",
"indexes" : [
{
"v" : 2,
"key" : {
"test" : 1
},
"name" : "test_1",
"background" : 1
}
],
"lsid" : {
"id" : UUID("07b43083-8ab9-4bcb-8768-919a3f27655f")
},
"$clusterTime" : {
"clusterTime" : Timestamp(1598875647, 409),
"signature" : {
"hash" : BinData(0,"+/YcdPyQriT8RL1LtFUhxe2BtCE="),
"keyId" : NumberLong("6861636045532823556")
}
},
"$db" : "test"
},
"msg" : "Index Build: draining writes received during build",
"numYields" : 0,
"locks" : {
},
"waitingForLock" : false,
"lockStats" : {
"ReplicationStateTransition" : {
"acquireCount" : {
"w" : NumberLong(6)
}
},
"Global" : {
"acquireCount" : {
"r" : NumberLong(1),
"w" : NumberLong(4)
}
},
"Database" : {
"acquireCount" : {
"r" : NumberLong(1),
"w" : NumberLong(4)
}
},
"Collection" : {
"acquireCount" : {
"r" : NumberLong(1),
"w" : NumberLong(3),
"W" : NumberLong(1)
}
},
"Mutex" : {
"acquireCount" : {
"r" : NumberLong(4)
}
}
},
"waitingForFlowControl" : false,
"flowControlStats" : {
"acquireCount" : NumberLong(3),
"timeAcquiringMicros" : NumberLong(1)
}
}
This Index Build: draining writes received during build makes no sense since there was no read/writes to the test collection during index creation.
Also index creation hangs only in non-empty collection. Index creates successfully in empty collection.
What might be an issue in this case? I'm out of ideas.
Finally figured it out with the help of MongoDB team.
The node can't communicate with itself so it will hang trying to commit the index build. This was the reason. Adding keyfile fixed the issue:
rm -f mongo.keyfile
openssl rand -base64 756 > mongo.keyfile
chmod 400 mongo.keyfile
bin/mongod --config mongo.conf --keyFile mongo.keyfile
Here are the links to the MongoDB's Jira issues which cover this subject:
https://jira.mongodb.org/browse/SERVER-50665 and
https://jira.mongodb.org/browse/SERVER-48516

MongoDB very slow update on non-indexed field

I am trying to debug to find a root cause on why single field update taking 60+ seconds, but not able to figure out. Will be great if you help me to get some direction on how to proceed.
Scenario: updating a non-indexed date field on a collection.
MongoDB Version : 3.0.12. Storage engine: WiredTiger
The query is using the proper index. scanned and modified are the same number of docs.
system.pofile document :
{
"_id" : ObjectId("5aa2e63b27001947449f1eed"),
"op" : "update",
"ns" : "abc.xyz",
"query" : {
"aId" : "5aa298dce4b0ef9feffe70e1",
},
"updateobj" : {
"$set" : {
"psdt" : ISODate("2018-03-09T17:36:31.277Z")
}
},
"nscanned" : 20,
"nscannedObjects" : 20,
"nMatched" : 20,
"nModified" : 20,
"keyUpdates" : 0,
"writeConflicts" : 0,
"numYield" : 5,
"locks" : {
"Global" : {
"acquireCount" : {
"r" : NumberLong(26),
"w" : NumberLong(26)
}
},
"Database" : {
"acquireCount" : {
"w" : NumberLong(26)
}
},
"Collection" : {
"acquireCount" : {
"w" : NumberLong(6)
}
},
"oplog" : {
"acquireCount" : {
"w" : NumberLong(20)
}
}
},
"millis" : 69765
}

mongodb - indexes disappear after few minutes

Have tested this on MongoDB 3.4 and 3.6:
Create one or more indexes in a collection
rs1:PRIMARY> db.coll.createIndex({checkinDate:1}, {background:1})
{
"createdCollectionAutomatically" : false,
"numIndexesBefore" : 1,
"numIndexesAfter" : 2,
"ok" : 1,
"operationTime" : Timestamp(1518162276, 2),
"$clusterTime" : {
"clusterTime" : Timestamp(1518162276, 2),
"signature" : {
"hash" : BinData(0,"AAAAAAAAAAAAAAAAAAAAAAAAAAA="),
"keyId" : NumberLong(0)
}
}
}
Now list the indexes.
rs1:PRIMARY> db.coll.getIndexes()
[
{
"v" : 2,
"key" : {
"_id" : 1
},
"name" : "_id_",
"ns" : "cico.coll"
},
{
"v" : 2,
"key" : {
"checkinDate" : 1
},
"name" : "checkinDate_1",
"ns" : "cico.coll",
"background" : 1
}
]
Wait for some time (few mins)
List the indexes again:
rs1:PRIMARY> db.coll.getIndexes()
[
{
"v" : 2,
"key" : {
"_id" : 1
},
"name" : "_id_",
"ns" : "cico.coll"
}
]
I have no clue why these indexes created are getting deleted? Any help appreciated.
Index option background is of type boolean, try:
db.coll.createIndex (
{ checkinDate:1 }, { background: true }
)
this is actually a non-issue. there was a program, running in the background, which was dropping all the indexes periodically (for a completely different reason).

What is mongo $mergeCursors?

We see a lot of slow queries in mongo logs like below (with pipeline op mergeCursors). We have a shaded mongo with 2 shards with only primaries. What is mergeCursors command? Please let me know if any other information is required.
{
"_id" : ObjectId("5571b739f65f7e64bb806362"),
"op" : "command",
"ns" : "mongrel.$cmd",
"command" : {
"aggregate" : "collection1",
"pipeline" : [
{
"$mergeCursors" : [
{
"host" : "endpoint:27005",
"id" : NumberLong(82775337156)
}
]
}
]
},
"keyUpdates" : 0,
"numYield" : 0,
"lockStats" : {
"timeLockedMicros" : {
"r" : NumberLong(12),
"w" : NumberLong(0)
},
"timeAcquiringMicros" : {
"r" : NumberLong(2),
"w" : NumberLong(2680)
}
},
"responseLength" : 12312,
"millis" : 6142,
"execStats" : {},
"ts" : ISODate("2015-06-05T12:35:40.801Z"),
"client" : "10.167.212.83",
"allUsers" : [],
"user" : ""
}
I was recently reading this post (http://dbattish.tumblr.com/post/108652372056/joins-in-mongodb) which seems to say that it is an internal aggregate command to merge queries across shards.

execStats is always empty in MongoDB "aggregate" commands profiling results

I am trying to profile the performance of an aggregation pipeline, specifically checking whether indices are used, how many objects are scanned, etc.
I'm setting the DB to full profiling:
db.setProfilingLevel(2)
But then in the db's 'system.profile' collection, in the result record for the aggregation command, the execStats is always empty.
Here is the full result for the command:
{
"op" : "command",
"ns" : "mydb.$cmd",
"command" : {
"aggregate" : "mycolection",
"pipeline" : [{
"$match" : {
"date" : {
"$gte" : "2013-11-26"
}
}
}, {
"$sort" : {
"user_id" : 1
}
}, {
"$project" : {
"user_id" : 1,
"_id" : 0
}
}, {
"$group" : {
"_id" : "$user_id",
"agg_val" : {
"$sum" : 1
}
}
}],
"allowDiskUse" : true
},
"keyUpdates" : 0,
"numYield" : 16,
"lockStats" : {
"timeLockedMicros" : {
"r" : NumberLong(3143653),
"w" : NumberLong(0)
},
"timeAcquiringMicros" : {
"r" : NumberLong(140),
"w" : NumberLong(3)
}
},
"responseLength" : 4990,
"millis" : 3237,
"execStats" : { },
"ts" : ISODate("2014-11-26T16:20:59.576Z"),
"client" : "127.0.0.1",
"allUsers" : [],
"user" : ""
}
Support execStats for aggregation command was added in mongo 3.4.