With a 11 GB working set (db.records.totalSize()), I ran the touch command in order to get Mongo to use as much memory as possible on my 16-GB RAM box. Before running touch, the serverStatus command showed that Mongo's mem.resident equaled 5800 (roughly 6 GB RAM).
db.runCommand({ touch: "records", data: true, index: true })
{ "ok" : 1 }
But, after running touch, Mongo's using roughly the same amount of RAM.
"mem" : {
"bits" : 64,
"resident" : 5821, /* only a 21 MB increase */
"virtual" : 29010,
"supported" : true,
"mapped" : 14362,
"mappedWithJournal" : 28724
},
Why did the touch command hardly increase how much RAM Mongo uses (mem.resident)?
The way that MongoDB db.serverStatus() command reports resident memory is by counting how many pages in physical RAM were actually accessed by mongod process.
This means that while your collection and indexes were read into RAM they won't show up in "res" value until you start actually querying against it.
You can verify that the data was read into RAM (if it was definitely cold before) just by seeing how much RAM mongod process has (not virtual memory).
Related
I'm using Mongo 2.6.9 with a cluster of 2 shards, each shard has 3 replicas, one of which is hidden.
It's a 5 machines deployment running on RedHat where 4 machines contain a single replica of 1 shard and the 5th machine contains the hidden replicas of both shards.
There is a load running of around 250 inserts per second and 50 updates per second. These are simple inserts and updates of pretty small documents.
In addition there is a small load of small files inserted to GridFS (around 1 file / second). The average file size is less than 1 MB.
There are 14 indexes defined for the involved collections. Those will be required when I will be adding the application that will be reading from the DB.
From the logs of the primary replicas I see during the whole run a huge amount of simple inserts and updates or even GetLastError requests that take hundreds of ms or even sometimes seconds (the default logging level only shows queries that took more than 100ms). For example this simple update uses an index for the query and doesn't update any index:
2015-10-12T06:12:17.258+0000 [conn166086] update chatlogging.SESSIONS query: { _id: "743_12101506113018605820fe43610c0a81eb9_IM" } update: { $set: { EndTime: new Date(1444630335126) } } nscanned:1 nscannedObjects:1 nMatched:1 nModified:1 keyUpdates:0 numYields:0 locks(micros) w:430 2131ms
2015-10-12T06:12:17.259+0000 [conn166086] command chatlogging.$cmd command: update { update: "SESSIONS", updates: [ { q: { _id: "743_12101506113018605820fe43610c0a81eb9_IM" }, u: { $set: { EndTime: new Date(1444630335126) } }, multi: false, upsert: false } ], writeConcern: { w: 1 }, ordered: true, metadata: { shardName: "S1R", shardVersion: [ Timestamp 17000|3, ObjectId('56017697ca848545f5f47bf5') ], session: 0 } } ntoreturn:1 keyUpdates:0 numYields:0 reslen:155 2132ms
All inserts and updates are made with w:1, j:1.
The machines have plenty of available CPU and memory. The disk I/O is significant, but not coming anywhere near 100% when these occur.
I really need to figure out what's causing this unexpectedly slow responsiveness of the DB. It's very possible that I need to change something in the way the DB is set up. Mongo runs with default configuration including the logging level.
An update -
I've continued looking into this issue and here are additional details that I'm hoping will allow to pinpoint the root cause of the problem or at least point me to the right direction:
The total DB size for a single shard is more than 200GB at the moment. The indexes being almost 50GB. Here is the relevant part from db.stats() and the mem part from db.ServerStatus() from the primary replica of one of the shards:
"collections" : 7,
"objects" : 73497326,
"avgObjSize" : 1859.9700916465995,
"dataSize" : 136702828176,
"storageSize" : 151309253648,
"numExtents" : 150,
"indexes" : 14,
"indexSize" : 46951096976,
"fileSize" : 223163187200,
"nsSizeMB" : 16,
"mem" : {
"bits" : 64,
"resident" : 5155,
"virtual" : 526027,
"supported" : true,
"mapped" : 262129,
"mappedWithJournal" : 524258
},
The servers have 8GB of RAM, out of which the mongod process use around 5GB. So the majority of the data and probably more important the indexes is not kept in memory. Can this be the our root cause? When I previously wrote that the system has plenty of free memory, I was refering to the fact that the mongod process isn't using as much as it could and also that most of the RAM is used for cached memory that can be released if required:
free -m output
Here is the output of mongostat from the same mongod:
mongostat output
I do see few faults in these, but these numbers look too low to me to indicate a real problem. Am I wrong?
Also I don't know whether the numbers seen in "locked db" are considered reasonable, or do those indicate that we have locks contention?
During the same timeframe when these stats were taken, many simple update operations that find a document based on an index and don't update an index, like the following one took hundreds of ms:
2015-10-19T09:44:09.220+0000 [conn210844] update chatlogging.SESSIONS query: { _id: "838_19101509420840010420fe43620c0a81eb9_IM" } update: { $set: { EndTime: new Date(1445247849092) } } nscanned:1 nscannedObjects:1 nMatched:1 nModified:1 keyUpdates:0 numYields:0 locks(micros) w:214 126ms
Many other types of insert or update operations take hundreds of ms too. So the issue looks to be system wide and not related to a specific type of query. Using mtools I'm not able to find operations that scan lots of documents.
I'm hoping that here I'll be able to get help with regards to finding the root cause of the problem. I can provide whatever additional info or statistics from the system.
Thank you in advance,
Leonid
1) First you need to increase the logging level
2) Use mtools to figure out what queries are slow
3) Tune that queries to figure out your bottleneck
Taking a look at the serverStatus command, I see the following data.
>db.runCommand( { serverStatus: 1} )
...
"mem" : {
"bits" : 64,
"resident" : 2138, // Mongo uses 2 GB RAM
"virtual" : 33272,
"supported" : true,
"mapped" : 16489, // equals db.coll.totalSize()
"mappedWithJournal" : 32978
},
Mongo recommends that the working set size fit in RAM.
If I understand correctly, then 16.4 GB of Mongo documents/indexes are memory mapped. Since Mongo is only using 2 GB of RAM, whenever Mongo needs to access an address outside of that 2 GB, then Mongo will need to fetch the contents of the address on disk and then load them into memory?
Is this my explanation the main reason that working set must fit into RAM?
Well, I am new to mongo and today morning I had a (bad) idea. I was playing around with indexes from the shell and decided to create a large collection with many documents (100 million). So I executed the following command:
for (i = 1; i <= 100; i++) {
for (j = 100; j > 0; j--) {
for (k = 1; k <= 100; k++) {
for (l = 100; l > 0; l--) {
db.testIndexes.insert({a:i, b:j, c:k, d:l})
}
}
}
}
However, the things didn't go as I expected:
It took 45 minutes complete the request.
It created 16 GB data on my hard disk.
It used 80% of my RAM (8GB total) and it won't release them till I restarted my PC.
As you can see in the photo below, as the number of documents inside the collection was growing, the time of the insertion of documents was growing as well. I suggest that by the last modification time of the data files:
Is this an expected behavior? I don't think that 100 million simple documents are too much.
P.S. I am now really afraid to run an ensureIndex command.
Edit:
I executed the following command:
> db.testIndexes.stats()
{
"ns" : "test.testIndexes",
"count" : 100000000,
"size" : 7200000056,
"avgObjSize" : 72.00000056,
"storageSize" : 10830266336,
"numExtents" : 28,
"nindexes" : 1,
"lastExtentSize" : 2146426864,
"paddingFactor" : 1,
"systemFlags" : 1,
"userFlags" : 0,
"totalIndexSize" : 3248014112,
"indexSizes" : {
"_id_" : 3248014112
},
"ok" : 1
}
So, the default index on _id has more than 3GB size.
It took 45 minutes complete the request.
Not surprised.
It created 16 GB data on my hard disk.
As #Abhishek states everything seems fine, MongoDB does use a fair amount of space without compression currently (that's coming later hopefully).
It seems that the data size is about 7.2GB while the average object size is 72 bytes, it seems this is working perfectly (since 72 bytes fits into 7.2GB) with the 3GB overhead of the _id index it seems that the storage size of 10GB is fitting quite well.
Though I am concerned that it has used 6GB more than the statistics say it needs to, that might need more looking into. I am guessing it is because of how MongoDB wrote to the data files, it might even be because you was not using a non fire and forget write concern (w>0), all in all; hmmm.
It used 80% of my RAM (8GB total) and it won't release them till I restarted my PC.
MongoDB will try and take as much RAM as the OS will let it. If the OS lets it take 80% then 80% it will take. This is actually a good sign, it shows that MongoDB has the right configuration values to store your working set efficiently.
When running ensureIndex mongod will never free up RAM. It simply has no hooks for that, instead the OS will shrink its allocated block to make room for more (or should rather).
This is an expected behavior, mongo db files starts with filesize 16MB ( test.0 ), and grow till 2GB and then 2GB is constant.
100 million ( 16 GB ) documents in nothing.
You can run ensureIndex, it shouldn't take much time.
You need not to restart your pc, the moment other process needed RAM, mongod will free RAM.
FYI : test.12 is completely empty.
I am guessing you are not worried about 16GB size just for 100 million documents ?
I have Mongo 2.2.2 running on Windows 7 x64 on i7 eight-core CPU. Our production servers are running under Red Hat Enterprise on 256-core machines with same version of Mongo.
In my tests of following call on my Windows machine
db.users_v2_prod.aggregate( { $group : {_id : "$email", total : { $sum : 1 } } }, { $match : { total : { $gte : 3 } } }, { $sort : {total : -1} }, {$limit : 5} )
I noticed that mongo underutilizes available resources. During the query total load on CPU is ~10%. According to Process Explorer computation occurs only in one thread. mongod seems to be using only 3 cores out of 8 I have and even they're used partially.
Could Mongo's engineers please explain their rationale to this implementation ? I'm curious why not use more resources if they are available. Why not parallel the load across all cores since you have index for a field I'm grouping at.
Given query was executed on collection with 6.5M documents (mongobackup produces 5GB file). So it's nothing crazy.
PS. And bonus question: have you thought about using GPU ? I have 1024-cores GPU on my laptop :)
In all likelihood, CPU is not the bounding factor here - that is true most of the time with typical use cases for MongoDB. Your query does not look computationally intensive, so it's more likely to be hitting a limit in terms of paging data off disk or running out of RAM.
It's hard to say without seeing actual stats for the run (for that I would recommend having the host in MMS with munin-node installed), but I have rarely seen the CPU be the bottleneck on a MongoDB instance.
Having said all that, the parallelization can probably be improved, but it may not be the quickest thing to get implemented. If none of the above is happening ore relevant, then I would see if you can run multiple jobs in parallel, or perhaps split up the work more on the client side to see if you can improve matters that way. You should also probably watch/vote/comment on these issues:
https://jira.mongodb.org/browse/SERVER-5091 (parallelize aggregating operations)
https://jira.mongodb.org/browse/SERVER-5088 (parallel query)
https://jira.mongodb.org/browse/SERVER-4504 (adding explain for aggregation framework) (added in 2.6)
I read that MongoDB documents are limited to 4 MB in size. I also read that when you insert a document, MongoDB puts some padding in so that if you add something to the document, the entire document doesn't have to be moved and reindexed.
So I was wondering, does it store documents in 4MB chunks on disk?
Thanks
As of 1.8, individual documents are now limited to 16MB in size (was previously 4MB). This is an arbitary limitation imposed as when you read a document off disk, the whole document is read into RAM. So I think the intention is that this limitation is there to try and safeguard memory / make you think about your schema design.
Data is then stored across multiple data files on disk - I forget the initial file size, but every time the database grows, a new file is created to expand into, where each new file is created bigger than the previous file until a single file size of 2GB is reached. From this point on, if the database continues to grow, subsequent 2GB data files are created for documents to be inserted into.
"chunks" has a meaning in the sharding aspect of MongoDB. Whereby documents are stored in "chunks" of a configurable size and when balancing needs to be done, it's these chunks of data (n documents) that are moved around.
The simple answer is "no." The actual space a document takes up in Mongo's files is variable, but it isn't the maximum document size. The DB engine watches to see how much your documents tend to change after insertion and calculates the padding factor based on that. So it changes all the time.
If you're curious, you can see the actual padding factor and storage space of your data using the .stats() function on a collection in the mongo shell. Here's a real-world example (with some names changed to protect the innocent clients):
{14:42} ~/my_directory ➭ mongo
MongoDB shell version: 1.8.0
connecting to: test
> show collections
schedule_drilldown
schedule_report
system.indexes
> db.schedule_report.stats()
{
"ns" : "test.schedule_report",
"count" : 16749,
"size" : 60743292,
"avgObjSize" : 3626.681712341035,
"storageSize" : 86614016,
"numExtents" : 10,
"nindexes" : 3,
"lastExtentSize" : 23101696,
"paddingFactor" : 1.4599999999953628,
"flags" : 1,
"totalIndexSize" : 2899968,
"indexSizes" : {
"_id_" : 835584,
"WeekEnd_-1_Salon_1" : 925696,
"WeekEnd_-1_AreaCode_1" : 1138688
},
"ok" : 1
}
So my test collection has about 16,749 records in it, with an average size of about 3.6 KB ("avgObjSize") and a total data size of about 60 MB ("size"). However, it turns out they actually take up about 86 MB on disk ("storageSize") because of the padding factor. That padding factor has varied over time as the collection's documents have been updated, but if I inserted a new document right now, it'd allocate 1.46 times as much space as the document needs ("paddingFactor") to avoid having to move things around if I change it later. To me that's a fair size/speed tradeoff.