looking at different mongodb operations i found out that mongodb 3.0 , WT engine, is generating many different locks , sometimes even upgrading to global lock like the following:
{
"desc" : "conn4981286",
"threadId" : "0x113f133400",
"connectionId" : 4981286,
"opid" : -371204669,
"active" : true,
"secs_running" : 0,
"microsecs_running" : NumberLong(377),
"op" : "update",
"ns" : "xxx",
"query" : {
"_id" : ObjectId("56c82738ccb2d079eec4867c")
},
"client" : "xxx",
"numYields" : 0,
"locks" : {
"Global" : "w",
"local" : "w",
"Database" : "w",
"Collection" : "w"
},
"waitingForLock" : false,
"lockStats" : {
"Global" : {
"acquireCount" : {
"r" : NumberLong(2),
"w" : NumberLong(2)
}
},
"Database" : {
"acquireCount" : {
"w" : NumberLong(2)
}
},
"Collection" : {
"acquireCount" : {
"w" : NumberLong(1)
}
},
"oplog" : {
"acquireCount" : {
"w" : NumberLong(1)
}
}
}
},
why would an update generate all of these locks?
thanks!
Related
I have 4 rows in the test collection:
{ "_id" : ObjectId("5f4ce50e19b13337216dd477"), "test" : 1 }
{ "_id" : ObjectId("5f4ce50e19b13337216dd478"), "test" : 2 }
{ "_id" : ObjectId("5f4ce50e19b13337216dd479"), "test" : 3 }
{ "_id" : ObjectId("5f4ce50e19b13337216dd47a"), "test" : 4 }
After running db.test.createIndex({test:1},{background:1}); to create an index, it just hangs. It was hanging for at least a few hours. Here is what I found in the db.currentOp() about this operation:
{
"type" : "op",
"host" : "HOSTNAME:27017",
"desc" : "IndexBuildsCoordinatorMongod-13",
"active" : true,
"currentOpTime" : "2020-08-31T12:11:13.159+00:00",
"opid" : 8721867,
"secs_running" : NumberLong(20),
"microsecs_running" : NumberLong(20888590),
"op" : "command",
"ns" : "test.test",
"command" : {
"createIndexes" : "test",
"indexes" : [
{
"v" : 2,
"key" : {
"test" : 1
},
"name" : "test_1",
"background" : 1
}
],
"lsid" : {
"id" : UUID("07b43083-8ab9-4bcb-8768-919a3f27655f")
},
"$clusterTime" : {
"clusterTime" : Timestamp(1598875647, 409),
"signature" : {
"hash" : BinData(0,"+/YcdPyQriT8RL1LtFUhxe2BtCE="),
"keyId" : NumberLong("6861636045532823556")
}
},
"$db" : "test"
},
"msg" : "Index Build: draining writes received during build",
"numYields" : 0,
"locks" : {
},
"waitingForLock" : false,
"lockStats" : {
"ReplicationStateTransition" : {
"acquireCount" : {
"w" : NumberLong(6)
}
},
"Global" : {
"acquireCount" : {
"r" : NumberLong(1),
"w" : NumberLong(4)
}
},
"Database" : {
"acquireCount" : {
"r" : NumberLong(1),
"w" : NumberLong(4)
}
},
"Collection" : {
"acquireCount" : {
"r" : NumberLong(1),
"w" : NumberLong(3),
"W" : NumberLong(1)
}
},
"Mutex" : {
"acquireCount" : {
"r" : NumberLong(4)
}
}
},
"waitingForFlowControl" : false,
"flowControlStats" : {
"acquireCount" : NumberLong(3),
"timeAcquiringMicros" : NumberLong(1)
}
}
This Index Build: draining writes received during build makes no sense since there was no read/writes to the test collection during index creation.
Also index creation hangs only in non-empty collection. Index creates successfully in empty collection.
What might be an issue in this case? I'm out of ideas.
Finally figured it out with the help of MongoDB team.
The node can't communicate with itself so it will hang trying to commit the index build. This was the reason. Adding keyfile fixed the issue:
rm -f mongo.keyfile
openssl rand -base64 756 > mongo.keyfile
chmod 400 mongo.keyfile
bin/mongod --config mongo.conf --keyFile mongo.keyfile
Here are the links to the MongoDB's Jira issues which cover this subject:
https://jira.mongodb.org/browse/SERVER-50665 and
https://jira.mongodb.org/browse/SERVER-48516
I have a Pageable request which query for about 1.6 millions documents and each request limit the response to 1000 documents. The query use an index... Well at least it should... I'm using db.currentOp() to look for the request that takes a while to process. When I launched the query (yesterday at 01:00PM) it used the index for each portion of 1000. But it seems that at some point he decided not to use it anymore... And I can't figure out why...
I also have a command operation for the same query and it is using the index.
{
"desc" : "conn7758",
"threadId" : "1012",
"connectionId" : 7758,
"client" : "10.10.10.3:64670",
"clientMetadata" : {
"driver" : {
"name" : "mongo-java-driver",
"version" : "unknown"
},
"os" : {
"type" : "Windows",
"name" : "Windows Server 2008 R2",
"architecture" : "x86",
"version" : "6.1"
},
"platform" : "Java/Oracle Corporation/1.8.0_73-b02"
},
"active" : true,
"opid" : 119588895,
"secs_running" : 1299,
"microsecs_running" : NumberLong(1299666014),
"op" : "command",
"ns" : "thingsplay.data",
"query" : {
"count" : "data",
"query" : {
"device" : "S000110",
"topic" : "INPUTS/COIL1000",
"date" : {
"$gt" : ISODate("2015-08-08T06:00:00Z"),
"$lt" : ISODate("2018-06-14T11:28:08.369Z")
}
}
},
"planSummary" : "IXSCAN { device: 1, topic: 1, vhost: 1}",
"numYields" : 81449,
"locks" : {
"Global" : "r",
"Database" : "r",
"Collection" : "r"
},
"waitingForLock" : false,
"lockStats" : {
"Global" : {
"acquireCount" : {
"r" : NumberLong(162900)
}
},
"Database" : {
"acquireCount" : {
"r" : NumberLong(81450)
}
},
"Collection" : {
"acquireCount" : {
"r" : NumberLong(81450)
}
}
}
}
The request is still running... and I think it will be for a couple of hours.
Here is the last operations that is running :
{
"desc" : "conn7758",
"threadId" : "1012",
"connectionId" : 7758,
"client" : "10.10.10.3:64670",
"clientMetadata" : {
"driver" : {
"name" : "mongo-java-driver",
"version" : "unknown"
},
"os" : {
"type" : "Windows",
"name" : "Windows Server 2008 R2",
"architecture" : "x86",
"version" : "6.1"
},
"platform" : "Java/Oracle Corporation/1.8.0_73-b02"
},
"active" : true,
"opid" : 120813021,
"secs_running" : 134,
"microsecs_running" : NumberLong(134586683),
"op" : "query",
"ns" : "thingsplay.data",
"query" : {
"find" : "data",
"filter" : {
"device" : "S000110",
"topic" : "INPUTS/COIL1000",
"date" : {
"$gt" : ISODate("2015-08-08T06:00:00Z"),
"$lt" : ISODate("2018-06-14T11:28:08.369Z")
}
},
"projection" : {
},
"skip" : 1124000,
"limit" : 1000
},
"numYields" : 8445,
"locks" : {
"Global" : "r",
"Database" : "r",
"Collection" : "r"
},
"waitingForLock" : false,
"lockStats" : {
"Global" : {
"acquireCount" : {
"r" : NumberLong(16892)
}
},
"Database" : {
"acquireCount" : {
"r" : NumberLong(8446)
}
},
"Collection" : {
"acquireCount" : {
"r" : NumberLong(8446)
}
}
}
}
Unfortunately I don't have the operation of the first operations that went well but you can see here that it doesn't use the index.
[
{
"v" : 2,
"key" : {
"_id" : 1
},
"name" : "_id_",
"ns" : "thingsplay.data"
},
{
"v" : 2,
"key" : {
"device" : 1,
"topic" : 1,
"date" : 1
},
"name" : "device_1_topic_1_date_1",
"ns" : "thingsplay.data",
"background" : true
},
{
"v" : 2,
"key" : {
"device" : 1,
"topic" : 1,
"vhost" : 1
},
"name" : "device_1_topic_1_vhost_1",
"ns" : "thingsplay.data",
"background" : true
}
]
I'm using MongoDB to store data with the following structure :
{ "_id" : ObjectId("5a497a1f451bf00880331913"), "device" : "deviceId", "topic" : "topicId", "vhost" : "vhostId", "date" : ISODate("2018-01-01T00:00:31.220Z"), "value" : 452 }
with the following indexes :
[
{
"v" : 2,
"key" : {
"_id" : 1
},
"name" : "_id_",
"ns" : "thingsplay.data"
},
{
"v" : 2,
"key" : {
"device" : 1,
"topic" : 1,
"date" : 1
},
"name" : "device_1_topic_1_date_1",
"ns" : "thingsplay.data",
"background" : true
},
{
"v" : 2,
"key" : {
"device" : 1,
"topic" : 1,
"vhost" : 1
},
"name" : "device_1_topic_1_vhost_1",
"ns" : "thingsplay.data",
"background" : true
}
]
But when I'm making a query with device, topic and date fields it uses device_1_topic_1_vhost_1 index...
For another request with the explain option I just had the right winning plan (device_1_topic_1_date_1) but when I execute it... I'm still waiting for it to complete and with db.currentOp() I don't see it using any index...
{
"desc" : "conn2172",
"threadId" : "2836",
"connectionId" : 2172,
"client" : "x.x.x.x:50177",
"appName" : "MongoDB Shell",
"clientMetadata" : {
"application" : {
"name" : "MongoDB Shell"
},
"driver" : {
"name" : "MongoDB Internal Client",
"version" : "3.6.2"
},
"os" : {
"type" : "Windows",
"name" : "Microsoft Windows Server 2008 R2",
"architecture" : "x86_64",
"version" : "6.1 SP1 (build 7601)"
}
},
"active" : true,
"opid" : 22137990,
"secs_running" : 322,
"microsecs_running" : NumberLong(322246758),
"op" : "command",
"ns" : "company.data",
"query" : {
"explain" : {
"find" : "data",
"filter" : {
"device" : "deviceId",
"topic" : "topicId",
"date" : {
"$gt" : ISODate("2018-01-01T00:00:00Z"),
"$lt" : ISODate("2018-06-12T16:18:00Z")
}
}
},
"verbosity" : "executionStats"
},
"numYields" : 19245,
"locks" : {
"Global" : "r",
"Database" : "r",
"Collection" : "r"
},
"waitingForLock" : false,
"lockStats" : {
"Global" : {
"acquireCount" : {
"r" : NumberLong(38492)
}
},
"Database" : {
"acquireCount" : {
"r" : NumberLong(19246)
}
},
"Collection" : {
"acquireCount" : {
"r" : NumberLong(19246)
}
}
}
}
I don't understand why the index is not being used...
I face a problem which connected with MongoDb performance. Queries are tens of seconds and by means of command db.currentOp() I can see, that all my queries are waiting database lock. For example:
just simple query
"desc" : "conn36316",
"threadId" : "7420",
"connectionId" : 36316,
"client" : "127.0.0.1:34538",
"active" : true,
"opid" : 197242270,
"secs_running" : 190,
"microsecs_running" : NumberLong(190574655),
"op" : "query",
"ns" : "db.items",
"query" : {
"find" : "items",
"filter" : {
"$msg" : "query not recording (too large)"
}
},
"numYields" : 92,
"locks" : {
"Global" : "r",
"Database" : "r"
},
"waitingForLock" : true,
"lockStats" : {
"Global" : {
"acquireCount" : {
"r" : NumberLong(186)
},
"acquireWaitCount" : {
"r" : NumberLong(1)
},
"timeAcquiringMicros" : {
"r" : NumberLong(24392)
}
},
"Database" : {
"acquireCount" : {
"r" : NumberLong(93)
},
"acquireWaitCount" : {
"r" : NumberLong(53)
},
"timeAcquiringMicros" : {
"r" : NumberLong(60099891)
}
},
"Collection" : {
"acquireCount" : {
"r" : NumberLong(92)
}
}
}
build index query
"desc" : "TTLMonitor",
"threadId" : "12592",
"active" : true,
"opid" : 958,
"op" : "none",
"ns" : "",
"query" : {
},
"numYields" : 0,
"locks" : {
"Global" : "r",
"Database" : "r"
},
"waitingForLock" : true,
"lockStats" : {
"Global" : {
"acquireCount" : {
"r" : NumberLong(4)
}
},
"Database" : {
"acquireCount" : {
"r" : NumberLong(2)
},
"acquireWaitCount" : {
"r" : NumberLong(1)
},
"timeAcquiringMicros" : {
"r" : NumberLong("3571165481")
}
},
"Collection" : {
"acquireCount" : {
"r" : NumberLong(3)
}
}
}
As you can see waiting for acquiring lock takes enormous time. For common query it equals 60 sec., and for index it equals 3571 sec.!
My work computer can create similar index much faster that it can be create on server.
My server is virtual machine with 10 cores, 40GB of memory and disk is raid1 array.
I'm using MongoDb version 3.2.6 (WiredTiger).
What could be the problem? How can I diagnose the problem?
My problem was the low speed disk subsystem, the replacement of the ssd according to the official documentation solved the problem
We are running MongoDB 3.0.2 on Linux.
The "getmore" operation for a find query periodically hangs.
The operation is available currentOp(), and one item I cannot explain is that all the acquireCount values in lockStats continue to grow while the query hangs (for Global, MMAPV1Journal, Database, and Collection).
This is the operation in question, at this point running for more than 1000 secs, returned by db.currentOp() :
{
"desc" : "conn60",
"threadId" : "0x2a99ee0",
"connectionId" : 60,
"opid" : 67792,
"active" : true,
"secs_running" : 1098,
"microsecs_running" : NumberLong(1098289543),
"op" : "getmore",
"ns" : "dbName.collectionName",
"query" : {
"d" : {
"$gt" : ISODate("2016-03-13T18:00:00.261Z"),
"$lt" : ISODate("2016-03-14T22:45:17.718Z")
},
"cc" : "US",
"dc" : {
"$in" : [
"26",
"31",
"17",
"29",
"35"
]
},
"pr" : {
"$gte" : 4
}
},
"client" : "10.0.0.111:33670",
"numYields" : 317557,
"locks" : {
},
"waitingForLock" : false,
"lockStats" : {
"Global" : {
"acquireCount" : {
"r" : NumberLong(635114)
}
},
"MMAPV1Journal" : {
"acquireCount" : {
"r" : NumberLong(317557)
}
},
"Database" : {
"acquireCount" : {
"r" : NumberLong(317557)
}
},
"Collection" : {
"acquireCount" : {
"R" : NumberLong(317557)
}
}
}
}
Any insights would be much appreciated ! Thank you in advance.