What is the optimum setup for memcached ? for example, how many MB memory we should assign ?
And also how long should we kept before it expires ? I tried setting memcached to expire in 1 hour. The first day it's running okay. But the next day, the server running very slow like it's running out of memory. So I delete all cache and retry with expiration down to 10 minutes.
Is there a way to tell memcached to flush if the memory used is full ? So not too make the whole website becoming very slow ?
Memcached uses an LRU algorithm to remove keys when it runs out of space. You can specify how much memory memcached uses with the -m parameter. By default memcached will only use 64MB of memory.
It's possible that things are running very slowly because you may not be using the -m parameter and that would mean your cache is only 64MB meaning you aren't actually caching much.
Related
I am migrating a database to heroku. I am using pg:diagnose to try and ensure that the database will be running smoothly.
Yesterday I noted that my "overall cache hit rate" was around 94%, which is lower than the recommended 99%. My database was running on the "Premium 3" tier which has 15 GB of RAM. So I decided to upgrade to a plan with more RAM, hoping this would lead to a higher cache hit rate. I switched to "Standard 4", which has more than double the RAM. The cache hit rate was low at first, but that was because it was cold.
But now it's the next day, the cache is warm, and my "overall cache hit rate" is back to 94%, right where it started! I must have missed something - I doubled the RAM but I'm not getting any more cache hits?
I would consider upgrading to a yet higher plan, but upgrading plans doesn't seem to help. My data size is 38.9 GB, and my current plan has 30.5 GB of RAM.
Thanks in advance to anyone who can help me understand what's going on here!
The cache-hit rate you are looking at from pg:diagnose seems to be measured about the same way that PostgreSQL itself would derive it--it considers everything found in shared_buffers to be a hit, and every thing else to be a miss. But for the misses, many of them could also be found in memory, it would just be the kernels filecache memory, not PostgreSQL's shared_buffers. From a performance perspective, this should also be hits, but there is no mechanism to count them as such.
I don't know how heroku manages shared_buffers. If shared_buffers stayed the same when you increased the instance size, then you would expect the reported hit rate to also stay the same, even if the true hit rate increased (i.e. more of the buffer misses are being served out of filecache rather truly being read from disk).
I have a dedicated OLTP server with SQL Server 2008 R2, 24 CPU Cores and 32 GB RAM. Earlier the SQL Server max memory setting had the default value of 0 - 2147483647 MB. And the ETL(mainly stored procedures) had good performance. But last week, somehow we inadvertently changed the SQL Server Max Memory setting to 0 - 16 GB. And the overall performance of ETL degraded and now it is taking twice the time as earlier. I tried to change it back by manually setting it back to 2147483647, also tried running the below query:
EXEC sp_configure'Show Advanced Options',1;
GO
RECONFIGURE;
GO
EXEC sp_configure'max server memory (MB)',2147483647;
GO
RECONFIGURE;
GO
But I cannot see the improvement in the performance. I even restarted the server after the changes but no luck. I also tried to reset the settings via Tools-->Import and Export Settings --> Reset all settings, but still no luck. Earlier through task manager, it was showing that SQL server is utilizing 95% of the total memory all the time.Now the memory utilization is very low. I need the earlier setting back.
Can anyone help me, how I can restore the default settings (I cannot reinstall the SQL Server as its already in production and have large amount of data)
Memory setting changes are dynamic and SQL Server will acquire memory as it needs to IF the OS has free memory available. This could take a while but no need to reboot.
When you rebooted, you did at least a couple of things to "reset" performance. For starters, you flushed the data cache so everything SQL Server needs to process your stored procs have to be retrieved from disk. This resolves itself relatively quickly without any additional action. As the SPs are run again, cache gets warmed up.
Second, you also flushed all query plans. This is a bit trickier. Depending on statistics freshness and how your queries were written, you may have had some very efficient plans earlier but the current/new plans are bad (there are a number of reasons for it).
Check to ensure sp_configure shows the correct "run_value". Run through the usual performance and workload monitoring steps. If things get back to "normal", then come back here with specific perf tuning questions.
I am using the using the flush all command to delete all the key/value pair on my Memcache server.
While the values get deleted, there are two things I can't understand when looking at the Memcache server through phpMemcachedAdmin
The memory usage is not reset to 0 after flushing it all. I still have 77% used and 22% wasted (just an example, but you get the spirit). How is that possible?
All the previous slab with the previous items are still there. For example, looking at a specific slab would show all the previous key/value pairs, despite the flush all command. How is that possible?
Thanks
This happens because memcache flushes on read, not on write. When you flush_all, that operations is designed for performance: it just means anything read beyond that time will be instantly expired, even though it is still in the cache. It just updates a single number and then checks that on each fetch.
This optimization significantly improves memcache performance. Imagine if several other people are searching or inserting at the same time as a flush.
Mongodb Background Flushing blocks all the requests:
Server: Windows server 2008 R2
CPU Usage: 10 %
Memory: 64G, Used 7%, 250MB for Mongod
Disk % Read/Write Time: less than 5% (According to Perfmon)
Mongodb Version: 2.4.6
Mongostat Normally:
insert:509 query:608 update:331 delete:*0 command:852|0 flushes:0 mapped:63.1g vsize:127g faults:6449 locked db:Radius:12.0%
Mongostat Before(maybe while) Flushing:
insert:1 query:4 update:3 delete:*0 command:7|0 flushes:0 mapped:63.1g vsize:127g faults:313 locked db:local:0.0%
And Mongostat After Flushing:
insert:1572 query:1849 update:1028 delete:*0 command:2673|0 flushes:1 mapped:63.1g vsize:127g faults:21065 locked db:.:99.0%
As you see when flushes happening lock is 99% just at this point mongod stops responding any read/write operation (mongotop and mongostat also stop). The flushing takes about 7 to 8 seconds to complete which does not increase disk load more than 10%.
Is there any suggestions?
Under Windows server 2008 R2 (and other versions of Windows I would suspect, although I don't know for sure), MongoDB's (2.4 and older) background flush process imposes a global lock, doing substantial blocking of reads and writes, and the length of the flush time tends to be proportional to the amount of memory MongoDB is using (both resident and system cache for memory-mapped files), even if very little actual write activity is going on. This is a phenomenon we ran into at our shop.
In one replica set where we were using MongoDB version 2.2.2, on a host with some 128 GBs of RAM, when most of the RAM was in use either as resident memory or as standby system cache, the flush time was reliably between 10 and 15 seconds under almost no load and could go as high as 30 to 40 seconds under load. This could cause Mongo to go into long pauses of unresponsiveness every minute. Our storage did not show signs of being stressed.
The basic problem, it seems, is that Windows handles flushing to memory-mapped files differently than Linux. Apparently, the process is synchronous under Windows and this has a number of side effects, although I don't understand the technical details well enough to comment.
MongoDb, Inc., is aware of this issue and is working on optimizations to address it. The problem is documented in a couple of tickets:
https://jira.mongodb.org/browse/SERVER-13444
https://jira.mongodb.org/browse/SERVER-12401
What to do?
The phenomenon is tied, to some degree, to the minimum latency of the disk subsystem as measured under low stress, so you might try experimenting with faster disks, if you can. Some improvements have been reported with this approach.
A strategy that worked for us in some limited degree is avoiding provisioning too much RAM. It happened that we really didn't need 128 GBs of RAM, so by dialing back on the RAM, we were able to reduce the flush time. Naturally, that wouldn't work for everyone.
The latest versions of MongoDB (2.6.0 and later) seem to handle the
situation better in that writes are still blocked during the long
flush but reads are able to proceed.
If you are working with a sharded cluster, you could try dividing the RAM by putting multiple shards on the same host. We didn't try this ourselves, but it seems like it might have worked. On the other hand, careful design and testing would be highly recommended in any such scenario to avoid compromising performance and/or high availability
We tried playing with syncdelay. Reducing it didn't help (the long flush times just happened more frequently). Increasing it helped a little (there was more time between flushes to get work done), but increasing it too much can exacerbate the problem severely. We boosted the syncdelay to five minutes (300 seconds), at one point, and were rewarded with a background flush of 20 minutes.
Some optimizations are in the works at MongoDB, Inc. These may be available soon.
In our case, to relieve the pressure on the primary host, we periodically rebooted one of the secondaries (clearing all memory) and then failed over to it. Naturally, there is some performance hit due to re-caching, and I think this only worked for us because our workload is write-heavy. Moreover, this technique not in any sense a solution. But if high flush times are causing serious disruption, this may be one way to "reduce the fever" so to speak.
Consider running on Linux... :-)
Background flush by default does not block read/write. mongod does flush every 60s, unless otherwise specified with -syncDelay parameter. syncDelay uses fsync() operation, which can set to block write while in-memory pages flush to disk. A blocked write could have potential to block reads as well. Read more: http://docs.mongodb.org/manual/reference/command/fsync/
However, normally a flush should not take more than 1000ms (1 second). If it does, it is likely the amount of data flushing to disk is too large for your disk to handle.
Solution: upgrade to a faster disk like SSD, or decrease flush interval (try 30s, rather than the default 60s).
All,
I am running CentOS 6.0 with Postgresql 8.4 and can't seem to figure out how to prevent so much disc swap from occurring. I have 12 gigs of RAM and 4 processors and I am doing some simple updates (1 table at a time). I thought for a minute that the inserts happening in parallel from a script I wrong was causing the large memory usage but when I saw the simple update causing it too I basically threw in the towel and decided to ask for help.
I pasted the conf file here. http://pastebin.com/e0jdBu0J
You can see that I set the buffers relatively low and the connection amounts high. The DB service will not start if I set the shared buffers any higher than 64 megs. Anyone have an idea what may be causing this for me?
Thanks,
Adam
If you're going into swap, increasing shared_buffers will make the problem worse; you'll be taking RAM away from the part that's running out and swapping, instead dedicating memory to the database caching. It's worth fixing SHMMAX etc. just on general principle and for later tuning work, but that's not going to help with this problem.
Guessing at the identify of your memory gobbling source is a crapshoot. Far better to look at data from "top -c" and ps to find which processes are using a lot of it. It's possible for a really bad query to consume way more memory than it should. If you see memory use spike up for a PostgreSQL process running something, check the process ID against the information in pg_stat_tables to see what it's doing.
There are a couple of things that can cause this sort of issue that often surprise people. If you are doing a large number of row updates in a single transaction, and there are foreign key checks or triggers involved, that can run out of memory. The queue of things to check in each of those cases is kept in RAM, and can be surprisingly big.
There are two problems with your PostgreSQL settings that might be related. Databases don't actually work very well if you have a lot more active connections than cores in the server; best performance is normally 2 to 3 active clients per core. And all sorts of things go wrong once you've got more than a few hundred connection. There is some connections^2 behavior that gets ugly there performance wise, and there are some memory issues too. If you really need 1250 connections, you should be using a connection pooler such as pgBouncer or pgpool-II.
And effective_io_concurrency = 1000 is way too high for any hardware on the planet. Useful values for that in a small multiple of how many disks you have in the server. I have no idea what happens as far as memory usage goes when you set it that high, but it's not been tested very well at that range. Normal settings more like 1 to 25. The parameters outlined at Tuning Your PostgreSQL Server are much more important than it is; the concurrency value only impacts one particular type of table scan.
Centos 6 seems to have a very conservative shmmax as a default
Set your shared buffers to that recommended by postgres tuning resources
see for explanation and how to set.
To experiment you can (as root) use sysctl -w kernel.shmmax = n
where n is the value that the startup error message that postgres is trying to allocate on startup. When you identify the value you wish to use permanently then set that in /etc/sysctl.conf