I have a memcached instance set up with 10GB of space on an ec2 instance with plenty of memory (15 gb I believe). I'm confused to see that the stat "total_items" is larger than the curr_items stat considering that when items are written to memcache, the timeout is set to 0 (never expire). Could someone explain these stats for me and why it does not seem to have every item in the cache still?
STAT cmd_set 174978
...
STAT delete_misses 0
STAT delete_hits 0
...
STAT bytes_read 45749254
STAT bytes_written 26472821
STAT limit_maxbytes 10737418240
...
STAT bytes 25386981
STAT curr_items 99198
STAT total_items 174978
STAT evictions 0
STAT reclaimed 0
I don't think total_items means what you think it means. It is a simple counter which increases whenever something is stored. A simple test verifies this. Store the same object twice. curr_items will be one, total_items will be two. Since your evictions and reclaims are both zero, everything that was ever stored is still there, as nothing has been evicted and no expired keys have been reclaimed. The data may have changed over time, but everything is still there.
Related
I am running PostgreSQL server and limited the shared_buffers to 4GB.
When I insert a large number of records in the database, the checkpointer process start consuming the RAM. This process neither ends nor decreases the RAM consumption even after a day.
Any idea why this is happening?
That is perfectly fine. The memory reported as allocated to the process is in fact the shared memory of shared_buffers that remains allocated for the entire life of the PostgreSQL server process.
Since it is the job of the checkpointer to read dirty pages from shared buffers and write them to disk, it is to be expected that the process reads a lot if that memory.
If you want to reduce the amount of disk I/O the checkpointer has to perform during bulk inserts, increase max_wal_size.
To see how much RAM is still free on your machine, consult the “available” field of the free output.
The checkpointer is a permanent process. It can nap for long periods of time when there is nothing to do, but it doesn't go away while the server is still running. Most of the memory it uses is the same memory being shared by everyone else. The top command (which I think is what you are showing there) includes all the shared memory a process has ever touched, and doesn't prorate it for the number of other processes it is being shared with.
Indeed if shared_buffers is 4GB, then the numbers you report here are suspiciously low, not high.
First off, here is a pastebin with all my important numbers (mongostat, db.stats, db.serverStatus).
The machine it is running on has plenty of memory free:
total used free shared buffers cached
Mem: 15965 2277 13688 0 89 1630
-/+ buffers/cache: 557 15407
Swap: 5119 0 5119
The database files themselves are on a block device (we previously had issues due to NFS drives) and should be working just fine. To avoid overfilling the logs, I even set --slowms 1000, but still a lot of the queries are slow and it seems like mongodb is just not interested in using more memory to service faster updates.
The use case is that it is 95% updates, with occasionally queries thrown in when we pull out data and compile it. Previously it has worked just fine with this setup, so I am trying to track down any reasons it might not be working.
I am hoping someone more familiar with mongodb can look at the numbers and point me in the right direction.
I'm trying to understand a MongoDB memory use pattern that I see in our MMS logs.
Normally, resident memory sits around 3GB, virtual memory is steady at 84GB, and mapped memory is about 41GB. Then in a series of peaks and troughs, usually for just a few minutes, mapped memory disappears completely, virtual memory drops to around 41GB, and resident memory 41GB or spikes to 84GB. In one recent episode, however, the peaks and troughs lasted 3.5 hours.
MongoDB appears to be running normally and other metrics such as opcounters and network are normal, but graphs suddenly changing dramatically when there was unlikely to be a significant load change makes me ... curious.
This is a standalone instance running MongoDB 1.8.3.
Typical memory usage, not during an episode (I only found the longer episode as it was ending):
$ free -m
total used free shared buffers cached
Mem: 32176 31931 245 0 628 29449
-/+ buffers/cache: 1854 30322
Swap: 1983 0 1983
What is causing this?
MMS gets memory statistics from the operating system by reading /proc/$PID/stat. The fluctuations in virtual and resident memory are reporting errors, and can safely be ignored.
(if you hover over the spikes, you'll notice that they occur at times when 1 or 2 of the 3 stats- virtual memory, mapped memory, or resident memory - is missing...)
I am on Ubuntu Linux 11 and Postgresql 9.1. I use CREATE TABLE .. SELECT over a dblink, and with a table of around 2 million rows I get
ERROR: out of memory
DETAIL: Failed on request of size 432.
So I am taking contents of an entire table from one database, and inserting (or creating them) inside another database (on the same machine). I am using default values of Postgresql, however I experimented with values from pgtune as well to no avail. During the insert I do see memory usage going up, however the error occurs before my machine's limit is reached. ulimit -a says
core file size (blocks, -c) 0
data seg size (kbytes, -d) unlimited
scheduling priority (-e) 0
file size (blocks, -f) unlimited
pending signals (-i) 30865
max locked memory (kbytes, -l) 64
max memory size (kbytes, -m) unlimited
open files (-n) 1024
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority (-r) 0
stack size (kbytes, -s) 8192
cpu time (seconds, -t) unlimited
max user processes (-u) 30865
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited
If I do create table as ... select inside the same database, then it works without problems. Any ideas?
Edit: I tried adjusting the various memory settings in the postgresql.conf and it didn't help. What am I missing?
My guess from this is that intermediate sets are being allocated to memory only and cannot be materialized per se. Your best options are to find a workaround or work with the dblink people to correct this problem. Some potential workarounds are:
Creating a csv file with COPY and insert that into your db.
Chunking the query to say, 100k rows at a time.
To be clear, my guess would be that dblink handles things by allocating a result set, allocating memory required, and handing the data on to Postgresql. It is possible this might be done in a way that lets requests be proxied over quickly (and transferred over the network connection) when they might not be allocated entirely in memory in the dblink module itself.
However for INSERT ... SELECT it may be allocating the entire result set in memory first, and then trying to process it and insert it into the table at once.
However this is a gut feeling without a detailed review of the code (I did open dblink.c and scanned it quickly). You have to remember here that PostgreSQL is acting simultaneously as the db client to the other server and as a db server itself, so the memory gotchas both of libpq and in the backend will come together.
Edit: after a little more review it looks like this is mostly right. dblink uses cursors internally. My guess is everything is being fetched from the cursor before insert so it can have a go at it at once.
I have a KDB/Q databse which has around ~2M records per day consuming about ~2G of memory. At end of day it runs some reporting stuff doing joins between the tables and outputting result into files on disk. During the computation the memory usage grows to ~15G. My problem is that once this operation finishes the memory is never released back and until the DB is restarted it consumes all the 15G of memory.
I would like to tell KDB to unload some tables from memory (not drop them though) but I don't want to restart the DB since some other apps are still connecting to it.
Is there a way to tell KDB to unload something from memory?
EDIT:
If anyone finds it interesting I suggest to have a look on .Q.gc[] for KDB 2.5+, looks promising.
Here is the sum up of my research:
KDB prior to ver. 2.5 allocates 64MB memory chunks on need and never release them. It is able to reuse them though.
recent KDB versions allow .Q.gc[] call which is on-request call to garbage collector (KDB uses ref. counting btw.)
this is especially useful when you invoke some memory intensive computation which allocates lot of memory (in my case it was ~20gB) and you want to release the memory after the computation finishes.
you can always consider putting the memory intensive script into a separate Q process so the memory will be released once the script finishes
This may be obvious, but in addition to checking the garbage collection modes for your version of q, do make sure that you've actually gotten rid of the in-memory data that's using the memory. If you're ok with getting rid of the whole table (e.g. this is a temp table involved in the computation), just delete it from the root namespace
delete table from`.
if not, you can delete all its rows
delete from`table
For anyone trying this in future the easiest way would have been to:
Start a new KDB process.
From that process query to select out the smallest limited subsets of data needed.
Perform any joins / calculations / writing to file from that process.
(allowing the original to carry on processing requests)
Close the process, freeing up all memory.
As mentioned by the above posters newer versions of KDB free up memory better but not perfect.
There's a good article on our company website that details KDB+ Memory Management:
http://timestored.com/kdbGuides/memoryManagement
http://code.kx.com/q4m3/12_Workspace_Organization/#125-expunging-from-a-context
I used a few different commands. As long as your table is stored on disk before you expunge it you should be OK.
This is the session before creating the table.
q).Q.w[]
used| 290192
heap| 67108864
peak| 67108864
wmax| 0
mmap| 0
mphy| 8589934592
syms| 629
symw| 20704
This command creates the table and then saves it to disk.
q)t:([]10000?"ab"; 10000?5)
q)save `t
`:t
The table is still in memory
q).Q.w[]
used| 437808
heap| 67108864
peak| 67108864
wmax| 0
mmap| 0
mphy| 8589934592
syms| 629
symw| 20704
Let's expunge the variable from memory and garbage collect.
q)delete t from `.
`.
q).Q.gc[]
0
Now the memory `used has been reduced to an amount similar to the start of the session.
q).Q.w[]
used| 290208
heap| 67108864
peak| 67108864
wmax| 0
mmap| 0
mphy| 8589934592
syms| 630
symw| 20730
q)\v
`symbol$()