i have an installation on memcache which i want to use in my production environment but when i have ran a couple of tests it seems that memcache doesn't free up memory even after it has used up all of it allocated memory, Also i logged in and ran a flush_all command but the objects are still in the cache.
Here are outputs from some tests
memcached-tool
memcache-top v0.6 (default port: 11211, color: on, refresh: 3 seconds)
INSTANCE USAGE HIT % CONN TIME EVICT/s READ/s WRITE/s
127.0.0.1:11211 427.1% 0.0% 18 1.4ms 0.0 244 261.0K
AVERAGE: 427.1% 0.0% 18 1.4ms 0.0 244 261.0K
TOTAL: 4.3MB/ 1.0MB 18 1.4ms 0.0 244 261.0K
memcached-tool 127.0.0.1:11211 display
No Item_Size Max_age Pages Count Full? Evicted Evict_Time OOM
1 560B 4s 1 1872 yes 0 0 15488
2 704B 32s 1 559 no 0 0 0
3 880B 4s 1 1191 yes 0 0 1335
4 1.1K 9s 1 116 no 0 0 0
5 1.4K 21s 1 14 no 0 0 0
6 1.7K 4s 1 17 no 0 0 0
7 2.1K 84s 1 24 no 0 0 0
8 2.7K 130s 1 60 no 0 0 0
9 3.3K 25s 1 290 no 0 0 0
10 4.2K 9s 1 194 no 0 0 0
11 5.2K 9s 1 116 no 0 0 0
15 12.7K 816s 1 1 no 0 0 0
16 15.9K 769s 1 5 no 0 0 0
18 24.8K 786s 1 1 no 0 0 0
21 48.5K 816s 1 1 no 0 0 0
memcached-tool 127.0.0.1:11211 stats
127.0.0.1:11211 Field Value
accepting_conns 1
auth_cmds 0
auth_errors 0
bytes 4478060
bytes_read 23964596
bytes_written 546642860
cas_badval 0
cas_hits 0
cas_misses 0
cmd_flush 0
cmd_get 240894
cmd_set 4504
conn_yields 0
connection_structures 21
curr_connections 18
curr_items 4461
decr_hits 0
decr_misses 0
delete_hits 0
delete_misses 0
evictions 0
get_hits 43756
get_misses 197138
incr_hits 0
incr_misses 0
limit_maxbytes 1048576
listen_disabled_num 0
pid 8731
pointer_size 64
reclaimed 0
rusage_system 5.047232
rusage_user 4.311344
threads 4
time 1306247929
total_connections 3092
total_items 4504
uptime 1240
version 1.4.5
-m tells memcached how much RAM to use for item storage (in megabytes). Note
carefully that this isn't a global
memory limit, so memcached will use a
few % more memory than you tell it to.
Set this to safe values. Setting it to
less than 48 megabytes does not work
properly in 1.4.x and earlier. It will
still use the memory.
Source: https://github.com/memcached/memcached/wiki/ConfiguringServer#commandline-arguments
Related
When I run the following command to parse haproxy logs, the output doesn't contain any headers, and I'm not able to understand the meanings of the numbers in each of the columns.
Command halog -pct < haproxy.log > percentiles.txt
the output that I see is:
0.1 3493 18 0 0 0
0.2 6986 25 0 0 0
0.3 10479 30 0 0 0
0.4 13972 33 0 0 0
0.5 17465 37 0 0 0
0.6 20958 40 0 0 0
0.7 24451 43 0 0 0
0.8 27944 46 0 0 0
0.9 31438 48 0 0 0
1.0 34931 49 0 0 0
1.1 38424 50 0 0 0
1.2 41917 51 0 0 0
1.3 45410 52 0 0 0
1.4 48903 53 0 0 0
1.5 52396 55 0 0 0
1.6 55889 56 0 0 0
1.7 59383 57 0 0 0
1.8 62876 58 0 0 0
1.9 66369 60 0 0 0
2.0 69862 61 0 0 0
3.0 104793 74 0 0 0
4.0 139724 80 0 1 0
5.0 174656 89 0 1 0
6.0 209587 94 0 1 0
7.0 244518 100 0 1 0
8.0 279449 106 0 1 0
9.0 314380 112 0 1 0
10.0 349312 118 0 1 0
15.0 523968 144 0 1 0
20.0 698624 168 0 1 0
25.0 873280 180 0 2 0
30.0 1047936 190 0 2 0
35.0 1222592 200 0 3 0
40.0 1397248 210 0 3 0
45.0 1571904 220 0 4 0
50.0 1746560 230 0 6 0
55.0 1921216 241 0 7 0
60.0 2095872 258 0 9 0
65.0 2270528 279 0 10 0
70.0 2445184 309 0 16 0
75.0 2619840 354 1 18 0
80.0 2794496 425 1 20 0
85.0 2969152 545 1 22 0
90.0 3143808 761 1 39 1
91.0 3178740 821 1 80 1
92.0 3213671 921 1 217 1
93.0 3248602 1026 1 457 1
94.0 3283533 1190 1 683 1
95.0 3318464 1408 1 889 1
96.0 3353396 1721 1 1107 1
97.0 3388327 2181 1 1328 1
98.0 3423258 2902 1 1555 1
98.1 3426751 3000 1 1580 1
98.2 3430244 3094 1 1607 1
98.3 3433737 3196 1 1635 1
98.4 3437231 3301 1 1666 1
98.5 3440724 3420 1 1697 1
98.6 3444217 3550 1 1731 1
98.7 3447710 3690 1 1770 1
98.8 3451203 3848 1 1815 1
98.9 3454696 4030 1 1864 1
99.0 3458189 4249 1 1923 2
99.1 3461682 4490 1 1993 2
99.2 3465176 4766 2 2089 2
99.3 3468669 5085 2 2195 2
99.4 3472162 5441 3 2317 97
99.5 3475655 5899 5 2440 365
99.6 3479148 6517 11 2567 817
99.7 3482641 7403 14 2719 1555
99.8 3486134 8785 16 2992 2779
99.9 3489627 11650 997 3421 4931
100.0 3493121 85004 4008 20914 71716
The first column looks to be the percentile, (like P50, P90, P99, etc) but the what are the values in the 2nd, 3rd, 4th, 5th and 6th columns? Also, are they total values (halog reports total times when provided with other options), or average values or maximum values?
<percentile> <request count> <Request Time*> <Connect Time**> <Response Time***> <Data Time****>
* Referred to as TR in the documentation.
** Referred to as Tc in the documentation.
*** Referred to as Tr in the documentation.
**** Referred to as Td in the documentation.
The source provides some good pointers.
My WinDBG version is 10.0.10240.9 AMD64 and while casually debugging some native memory dump I realized that my !heap command behaves different than described and I am unable to figure out why.
There are plenty of resources mentioning !heap -s:
https://msdn.microsoft.com/en-us/library/windows/hardware/ff563189%28v=vs.85%29.aspx?f=255&MSPPError=-2147217396
http://windbg.info/doc/1-common-cmds.html
When I execute !heap -s
I get this truncated list:
0:000> !heap -s
************************************************************************************************************************
NT HEAP STATS BELOW
************************************************************************************************************************
LFH Key : 0x000000c42ceaf6ca
Termination on corruption : ENABLED
Heap Flags Reserv Commit Virt Free List UCR Virt Lock Fast
(k) (k) (k) (k) length blocks cont. heap
-------------------------------------------------------------------------------------
Virtual block: 0000000003d40000 - 0000000003d40000 (size 0000000000000000)
... many more virtual blocks
0000000000b90000 00000002 3237576 3220948 3237576 20007 1749 204 359 0 LFH
0000000000010000 00008000 64 8 64 5 1 1 0 0
... more heaps
-------------------------------------------------------------------------------------
Ok fine, b90000 looks big but contrary to those docs above and !heap -s -? I cannot get information for this heap, each of those commands produce the exact same output as seen above (as if I would not specify anything after -s):
!heap -s b90000
!heap -s -h b90000
!heap -s 1
I get a load of virtual blocks and a dump of all heaps instead of the single specified one.
Anyone having the same issue?
My "Windows Debugger Version 10.0.10586.567 AMD64" behaved like yours, but
“Microsoft (R) Windows Debugger Version 6.3.9600.16384 AMD64” I have in in:
C:\Program Files\Windows Kits\8.1\Debuggers\x64
0:000> !heap -s -h 0000000000220000
Walking the heap 0000000000220000 ..................Virtual block: 0000000015f20000 - 0000000015f20000 (size 0000000000000000)
Virtual block: 000000001b2e0000 - 000000001b2e0000 (size 0000000000000000)
Virtual block: 000000001f1e0000 - 000000001f1e0000 (size 0000000000000000)
Virtual block: 0000000023c10000 - 0000000023c10000 (size 0000000000000000)
Virtual block: 000000001c060000 - 000000001c060000 (size 0000000000000000)
Virtual block: 000000001ddc0000 - 000000001ddc0000 (size 0000000000000000)
0: Heap 0000000000220000
Flags 00000002 - HEAP_GROWABLE
Reserved memory in segments 226880 (k)
Commited memory in segments 218204 (k)
Virtual bytes (correction for large UCR) 218740 (k)
Free space 12633 (k) (268 blocks)
External fragmentation 5% (268 free blocks)
Virtual address fragmentation 0% (30 uncommited ranges)
Virtual blocks 6 - total 0 KBytes
Lock contention 0
Segments 1
Low fragmentation heap 00000000002291e0
Lock contention 0
Metadata usage 90112 bytes
Statistics:
Segments created 993977
Segments deleted 992639
Segments reused 0
Block cache:
3: 1024 bytes ( 17, 0)
4: 2048 bytes ( 42, 0)
5: 4096 bytes ( 114, 0)
6: 8192 bytes ( 231, 2)
7: 16384 bytes ( 129, 9)
8: 32768 bytes ( 128, 11)
9: 65536 bytes ( 265, 58)
10: 131072 bytes ( 357, 8)
11: 262144 bytes ( 192, 49)
Buckets info:
Size Blocks Seg Empty Aff Distribution
------------------------------------------------
------------------------------------------------
Default heap Front heap Unused bytes
Range (bytes) Busy Free Busy Free Total Average
------------------------------------------------------------------
0 - 1024 577 140 1035286 11608 10563036 10
1024 - 2048 173 3 586 374 27779 36
2048 - 3072 17 19 47 224 1605 25
3072 - 4096 20 12 1 126 348 16
4096 - 5120 35 3 1 30 677 18
5120 - 6144 2 8 0 0 33 16
6144 - 7168 5 9 0 0 56 11
7168 - 8192 0 11 0 0 0 0
8192 - 9216 14 0 0 15 236 16
9216 - 10240 1 0 0 0 8 8
12288 - 13312 1 0 0 0 17 17
14336 - 15360 1 0 0 18 1 1
15360 - 16384 1 0 0 0 32 32
16384 - 17408 10 0 0 0 160 16
22528 - 23552 1 0 0 0 9 9
23552 - 24576 2 0 0 0 32 16
27648 - 28672 1 0 0 0 8 8
30720 - 31744 0 1 0 0 0 0
32768 - 33792 18 0 0 0 250 13
33792 - 34816 0 1 0 0 0 0
39936 - 40960 0 2 0 0 0 0
40960 - 41984 0 1 0 0 0 0
43008 - 44032 0 2 0 0 0 0
44032 - 45056 0 5 0 0 0 0
45056 - 46080 0 1 0 0 0 0
46080 - 47104 0 2 0 0 0 0
47104 - 48128 0 1 0 0 0 0
49152 - 50176 0 3 0 0 0 0
50176 - 51200 1 0 0 0 16 16
51200 - 52224 0 4 0 0 0 0
57344 - 58368 0 1 0 0 0 0
58368 - 59392 0 1 0 0 0 0
62464 - 63488 0 1 0 0 0 0
63488 - 64512 200 1 0 0 3200 16
64512 - 65536 0 1 0 0 0 0
65536 - 66560 1029 2 0 0 10624 10
79872 - 80896 100 0 0 0 900 9
131072 - 132096 9 0 0 0 144 16
193536 - 194560 1 0 0 0 9 9
224256 - 225280 1 0 0 0 16 16
262144 - 263168 49 27 0 0 784 16
327680 - 328704 1 0 0 0 17 17
384000 - 385024 0 1 0 0 0 0
523264 - 524288 1 5 0 0 23 23
------------------------------------------------------------------
Total 2271 268 1035921 12395 10610020 10
This might be a walkaround,
can’t answer why the win 10 version don’t work :-(
I use MongoDB to store price events for stocks. Depending on what you want to screen, the number of event can rapidly grow to 1Go-2Go.
I run MongoDB on a single machine and it is taking longer and longer to load the data. I am not able to find a clear answer on the web if "sharding on a single server" is a benefit to read speed.
Is it the right path to increase the read speed?
insert query update delete getmore command flushes mapped vsize res faults locked db
0 1 395 0 1 395 0 63.9g 128g 2.53g 7484 prices:70.5%
0 0 14726 0 5 14728 0 63.9g 128g 2.48g 31555 prices:7.4%
0 0 0 0 0 1 0 63.9g 128g 2.48g 436 prices:0.0%
0 0 0 0 1 1 0 63.9g 128g 2.48g 0 prices:0.0%
0 0 0 0 0 1 1 63.9g 128g 2.49g 3877 .:83.9%
0 0 0 0 0 1 0 63.9g 128g 2.49g 0 prices:0.0%
In Perl, I generate a huge read-only data-structure once, then fork().
This is to take advantage of COW on RSS pages when forking. It works really well, but when a child process exits, it allocates all the RAM from itelf just prior dying.
Is there a way to avoid this useless allocation ?
Here is sample Perl code that shows the issue.
#! /usr/bin/perl
my $a = [];
# Allocate 100 MiB
for my $i (1 .. 100000) {
push #$a, "x" x 1024;
}
# Fork 10 other process
for my $j (1 .. 10) {
last unless fork();
}
# Sleep for a while to be able to see the RSS
sleep(5);
In the sample vmstat output, we can see that it first allocates only 100MiB, then after the 1rst sleep it allocates the whole for a short while, and then releases all of it.
procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu------
r b swpd free buff cache si so bi bo in cs us sy id wa st
0 0 0 1329660 80596 86936 0 0 21 18 160 25 0 0 100 0 0
1 0 0 1328048 80596 86936 0 0 0 0 1013 44 0 0 100 0 0
0 0 0 1223888 80596 86936 0 0 0 0 1028 76 11 5 84 0 0
0 0 0 1223888 80596 86936 0 0 0 0 1010 40 0 0 100 0 0
0 0 0 1223888 80596 86936 0 0 0 0 1026 54 0 0 100 0 0
0 0 0 1223888 80596 86936 0 0 0 0 1006 39 0 0 100 0 0
13 0 0 741156 80596 86936 0 0 0 0 1012 66 13 58 28 0 0
0 0 0 1329288 80596 86936 0 0 0 0 1032 60 0 0 100 0 0
Note: it seems it isn't a Perl version specific issue. As I tested 5.8.8, 5.10.1 & 5.14.2 and they all do exhibit this behavior.
Update:
As #choroba asked in comments, I also tried to undef the data-structure, but it seems that it triggers the memory-touching as the RAM is then allocated.
You can add the following snippet at the end of the first script.
# Unallocate $a
undef $a;
# Sleep for a while to be able to see the RSS
sleep(5);
Actually, as I found out myself, this behavior is a feature, and the answer lies in the Perl doc:
The exit() function does not always exit immediately.
Likewise any object destructors that need to be called
are called before the real exit.
If this is a problem, you can
call POSIX::_exit($status) to avoid END and destructor processing.
And indeed, adding it at the end of the original code sample does avoid the behavior.
# XXX - To be added just before ending the process
# Use POSIX::_exit($status) to end without allocating copy-on-write RAM
use POSIX;
POSIX::_exit(0);
Note: for this to work, the child has to exit also before the data-structure goes out of scope.
I'm running a MongoDB instance using a Replica Set, when there are a lot of insert, I can see very weird statistics on faults and locked %.
How come locked % can be more than 100 ?!
Where does the faults happen, I have no logs mentioning any fault, does someone have any clue about what it means ?
insert query update delete getmore command flushes mapped vsize res faults locked % idx miss % qr|qw ar|aw netIn netOut conn set repl time
9 0 0 0 1 4 0 70.3g 141g 4.77g 20 124 0 0|0 0|1 1m 2m 10 socialdb M 18:49:49
18 0 0 0 3 1 0 70.3g 141g 4.77g 17 73.8 0 0|0 0|1 1m 2m 10 socialdb M 18:49:50
21 0 0 0 1 5 0 70.3g 141g 4.77g 18 104 0 0|0 0|1 1m 1m 10 socialdb M 18:49:51
20 0 0 0 3 1 0 70.3g 141g 4.78g 18 98.8 0 0|0 0|1 1m 3m 10 socialdb M 18:49:52
172 0 0 0 5 4 0 70.3g 141g 4.79g 133 72.8 0 0|0 0|0 7m 12m 10 socialdb M 18:49:53
76 0 0 0 3 1 0 70.3g 141g 4.8g 114 65.1 0 0|0 0|1 6m 10m 10 socialdb M 18:49:54
54 0 0 0 4 4 1 70.3g 141g 4.81g 45 90.6 0 0|0 0|1 2m 8m 10 socialdb M 18:49:55
85 0 0 0 4 2 0 70.3g 141g 4.84g 101 98.1 0 0|0 0|1 6m 11m 10 socialdb M 18:49:56
77 0 0 0 3 4 0 70.3g 141g 4.82g 78 74.5 0 0|0 0|1 4m 9m 10 socialdb M 18:49:57
72 0 0 0 3 1 0 70.3g 141g 4.84g 111 95.7 0 0|0 0|1 6m 10m 10 socialdb M 18:49:58
Is there a better (standard) monitoring tool, free ?
Not sure about the other two but this could be the answer to your first question, if you are using v2.2:
http://docs.mongodb.org/manual/reference/mongostat/The above page mentions:
locked:
The percent of time in a global write lock.
(Changed in version 2.2: The locked db field replaces the locked % field to more appropriate data regarding the database specific locks in version 2.2)
locked db:
New in version 2.2.
The percent of time in the per-database context-specific lock. mongostat will report the database that has spent the most time since the last mongostat call with a write lock.
This value represents the amount of time the database had a database specific lock and the time that the mongod spent in the global lock. Because of this, and the sampling method, you may see some values greater than 100%.