I'm using MongoDb-CE 4.4
I have a 2.4 GB mongod.log file.
I executed
db.setProfilingLevel(0);
I obtained
{
"was" : 0.0,
"slowms" : 100.0,
"sampleRate" : 1.0,
"ok" : 1.0
}
But still it is logging slow query... why?
setting profiling level to zero turns off recording data to the collection system.profile, but will continue to log to the log file any operations slower than slowms. You cannot stop logging slow operations, but you can set the slowms to a high value - db.setProfilingLevel(0, 5000000) which may have the same effect. In this example it will log if it takes more than 5,000,000 ms (i.e., 5,000 seconds).
Related
I have a Mongo db version 4.4.1 and I am profiling for slow queries. With profiling level 1 and verbosity level 0 nothing gets logged, but with profiling level 1 and verbosity level 1 all queries get logged as slow even though durationMillis is lower than slowms. All I found was the official information in the documentation:
https://www.mongodb.com/docs/manual/tutorial/manage-the-database-profiler/#profiling-levels
https://www.mongodb.com/docs/manual/reference/method/db.setLogLevel/#db.setloglevel--
https://www.mongodb.com/docs/manual/reference/configuration-options/#mongodb-setting-operationProfiling.slowOpThresholdMs
At higher logLevel settings, all operations appear in the diagnostic log regardless of their latency with the following exception: the logging of slow oplog entry messages by the secondaries. The secondaries log only the slow oplog entries; increasing the logLevel does not log all oplog entries.
It is unclear if verbosity level 1 and higher automatically logs all queries as slow.
What is the behavour of the different verbosity levels and why are queries with duratonMillis lower than slowms logged as slow
I think this question is conflating a few related topics, for understandable reasons as we'll see below.
The first pair of concepts are the profiler and slow query logging. The database profiler is a capped collection that collects runtime information about CRUD operations (among other things). Details about CRUD operations can separately be captured in a log file directly. You adjust both of these using the a single db.setProfilingLevel() method.
Now the aforementioned log file is the same diagnostic log file that is used by the mongod process for its other operational needs. So that single file is serving double duty as it will also capture details about many other components such as networking, replication, and access control. To compound the problem further one of the components that can be configured and logged is indeed the QUERY one. Adjusting the verbosity for components is done via the db.setLogLevel() method. The system defaults to a verbosity level of 0 for all components, which includes "Informational" (I) details.
Now "slow query" log lines are one of the types of messages generated by and are classified as QUERY components at the "Informational" level. The result of this fact is that you can increase the number of slow query entries that get logged by either:
Decreasing the slowms threshold via the db.setProfilingLevel() method
Increasing the verbosity for the QUERY component via the db.setLogLevel() method.
Finally to return back to the beginning, slow query logging and profiling also overlap with their usage of the slowms (and other related settings). Specifically if you set the (profiling) level to 1, then slowms will now be used to control both what entries go into the log file as well as what entries get captured by the profiler. A setting of 0 or 2 makes profiling ignore slowms (either not capturing anything in the profiler or capturing everything, respectively).
So the section of the documentation that you quoted, including the sentence before it, is what attempts to tie all of this together (when it comes to behavior on the PRIMARY):
When logLevel is set to 0, MongoDB records slow operations to the diagnostic log at a rate determined by slowOpSampleRate. At higher logLevel settings, all operations appear in the diagnostic log regardless of their latency
Noting that:
"slow operations" here are defined as those that exceed the slowms setting
logLevel here refers to that verbosity setting of db.setLogLevel() as opposed to the level parameter of db.setProfilingLevel().
I can find no details on why the component strings for the db.setLogLevel() method are shown in lowercase yet they appear to be uppercase when logged.
Returning back to the original questions with this additional context, the described behavior is aligned with the documented behavior if we tighten up the wording a bit. I assume that the statement about "profiling for slow queries" is only about capturing slow queries in the log file. If so, then:
With profiling level 1 and verbosity level 0 nothing gets logged
That is correct if all of the queries that are executed are also below the slowms threshold (which defaults to 100ms).
with profiling level 1 and verbosity level 1 all queries get logged as slow even though durationMillis is lower than slowms
It is unclear if verbosity level 1 and higher automatically logs all queries as slow.
That is also correct per the documentation quoted earlier, or the Logging Slow Operations section here which succinctly states that "Client operations (such as queries) appear in the log if their duration exceeds the slow operation threshold or when the log verbosity level is 1 or higher."
I am importing a lot of data (18GB, 3 million documents) over time, almost all the data are indexed, so there are lots of indexing going on. The system consist of a single client (single process on a separate machine) establishing a single connection (using pymongo) and doing insertMany in batch of 1000 docs.
MongoDB setup:
single instance,
journaling enabled,
WiredTiger with default cache,
RHEL 7,
version 4.2.1
192GB RAM, 16 CPUs
1.5 TB SSD,
cloud machine.
When I start the server (after full reboot) and insert the collection, it takes 1.5 hours. If the server run for a while inserting some other data (from a single client), it finishes to insert the data, I delete the collection and run the same data to insert - it takes 6 hours to insert it (there is still sufficient disk more than 60%, nothing else making connections to the db). It feels like the server performance degrades over time, may be OS specific. Any similar experience, ideas?
I had faced similar issue, the problem was RAM.
After full restart the server had all RAM free, but after insertions the RAM was full. Deletion of collection and insertion same data again might take time as some RAM was still utilised and less was free for mongo.
Try freeing up RAM and cache after you drop the collection, and check if same behaviour persists.
As you haven't provided any specific details, I would recommend you enable profiling; this will allow you to examine performance bottlenecks. At the mongo shell run:
db.setProfilingLevel(2)
Then run:
db.system.profile.find( { "millis": { "$gt": 10 } }, { "millis": 1, "command": 1 }) // find operations over 10 milliseconds
Once done set reset the profiling mode:
db.setProfilingLevel(0)
I have a server which consistently insert and update the data in MongoDb collections.
I have 3 collections : handTab, video, handHistory.
When the no of documents in those collection reaches 40000, 40000, 80000 respectively, then the insert and update and findAndModify command's performance degrades.
In the mongo Compass I have seen that these query are the slowest operations.
Also, the CPU utilization on production server goes upto 100%.
This leads to the server connection timeout of my game-server API.
I have database on a machine and game-server running on other machine, Why the CPU utilization of database machine reaches 100% after the data size become large in collections, i have same number of mongo operations previously as well as now?
I have a mongodb collection with custom _id and 500M+ documents. Size of the _id's index is ≈25Gb and whole collection is ≈125 Gb. Server has 96 Gb RAM. Read activity is only range queries by _id. Explain() shows that queries use the index. Mongo works rather fast some time after load tests start and slows down after a while. I can see in a log a lot of entries like this:
[conn116] getmore csdb.archive query: { _id: { $gt: 2812719756651008, $lt: 2812720361451008 } } cursorid:444942282445272280 ntoreturn:0 keyUpdates:0 numYields: 748 locks(micros) r:7885031 nreturned:40302 reslen:1047872 10329ms
A piece of db.currentOp():
"waitingForLock" : false,
"numYields" : 193,
"lockStats" : {
"timeLockedMicros" : {
"r" : NumberLong(869051),
"w" : NumberLong(0)
},
"timeAcquiringMicros" : {
"r" : NumberLong(1369404),
"w" : NumberLong(0)
}
}
What is locks(micros) r? What can I do to cut it down?
What is locks(micros) r?
The amount of time that read locks was held (in microseconds).
R - Global read lock
W - Global write lock
r - Database specific read lock
w - Database specific write lock
What can I do to cut it down?
How does sharding affect concurrency?
Sharding improves concurrency by distributing collections over multiple mongod instances, allowing shard servers (i.e. mongos processes) to perform any number of operations concurrently to the various downstream mongod instances.
Diagnosing Performance Issues (Locks)
MongoDB uses a locking system to ensure data set consistency. However, if certain operations are long-running, or a queue forms, performance will slow as requests and operations wait for the lock. Lock-related slowdowns can be intermittent. To see if the lock has been affecting your performance, look to the data in the globalLock section of the serverStatus output. If globalLock.currentQueue.total is consistently high, then there is a chance that a large number of requests are waiting for a lock. This indicates a possible concurrency issue that may be affecting performance.
If globalLock.totalTime is high relative to uptime, the database has existed in a lock state for a significant amount of time. If globalLock.ratio is also high, MongoDB has likely been processing a large number of long running queries. Long queries are often the result of a number of factors: ineffective use of indexes, non-optimal schema design, poor query structure, system architecture issues, or insufficient RAM resulting in page faults and disk reads.
How We Scale MongoDB (Vertically)
Sadly, MongoDB itself will usually become a bottleneck before the capacity of a server is exhausted. Write lock is almost always the biggest problem (though there are practical limits to how much IO capacity a single MongoDB process can take advantage of).
I am using 10 threads to do find-and-modify on large documents (one document at a time on each thread). The documents are about 600kB each and contain each a 10,000 elements array. The client and the MongoDB server are on the same LAN in a datacenter and are connected via a very fast connection. The client runs on small machine (CPU is fairly slow, 768 MB of RAM).
The problem is that every find-and-modify operation is extremely slow.
For example here is the timeline of an operation:
11:56:04: Calling FindAndModify on the client.
12:05:59: The operation is logged on the server ("responseLength" : 682598, "millis" : 7), so supposedly the server returns almost immediately.
12:38:39: FindAndModify returns on the client.
The CPU usage on the machine is only 10-20%, but the memory seems to be running low. The available bytes performance counter is around 40 MB, and the Private Bytes of the process that is calling MongoDB is 800 MB (which is more than the physical RAM on the machine). Also the Page/sec performance counter is hovering around 4,000.
My theory is that it took 33 minutes for the driver to deserialize the document, and that could be because the OS is swapping due to high memory usage caused by the deserialization of the large documents into CLR objects. Still, this does not really explain why it took 10 minutes between the call to FindAndModify and the server side execution.
How likely is it that this is the case? Do you have another theory? How can I verify this is the case?