I have an Akka program with threads that are spending most of their time in the futex_ state.
This is from an Akka program that isn't doing much. I am not familiar with the internals of Akka, so I'm not sure if this is normal or not.
F S UID PID PPID LWP C PRI NI ADDR SZ WCHAN TTY TIME CMD
4 S 1 19 0 19 0 80 0 - 1149556 futex_ ? 0:00 /usr/bin/java
5 S 1 19 0 25 0 80 0 - 1149556 futex_ ? 0:04 /usr/bin/java
1 S 1 19 0 26 0 80 0 - 1149556 futex_ ? 0:00 /usr/bin/java
1 S 1 19 0 27 0 80 0 - 1149556 futex_ ? 0:00 /usr/bin/java
1 S 1 19 0 28 0 80 0 - 1149556 futex_ ? 0:00 /usr/bin/java
1 S 1 19 0 29 0 80 0 - 1149556 futex_ ? 0:00 /usr/bin/java
1 S 1 19 0 30 1 80 0 - 1149556 futex_ ? 0:12 /usr/bin/java
1 S 1 19 0 31 0 80 0 - 1149556 futex_ ? 0:04 /usr/bin/java
1 S 1 19 0 32 0 80 0 - 1149556 futex_ ? 0:00 /usr/bin/java
1 S 1 19 0 33 0 80 0 - 1149556 futex_ ? 0:00 /usr/bin/java
1 S 1 19 0 43 0 80 0 - 1149556 futex_ ? 0:02 /usr/bin/java
1 S 1 19 0 44 0 80 0 - 1149556 futex_ ? 0:00 /usr/bin/java
1 S 1 19 0 45 0 80 0 - 1149556 futex_ ? 0:00 /usr/bin/java
1 S 1 19 0 46 0 80 0 - 1149556 futex_ ? 0:01 /usr/bin/java
1 S 1 19 0 47 0 80 0 - 1149556 futex_ ? 0:02 /usr/bin/java
1 S 1 19 0 50 0 80 0 - 1149556 futex_ ? 0:00 /usr/bin/java
1 S 1 19 0 53 0 80 0 - 1149556 futex_ ? 0:01 /usr/bin/java
1 S 1 19 0 60 0 80 0 - 1149556 futex_ ? 0:03 /usr/bin/java
1 S 1 19 0 121 0 80 0 - 1149556 futex_ ? 0:00 /usr/bin/java
1 S 1 19 0 122 0 80 0 - 1149556 futex_ ? 0:01 /usr/bin/java
1 S 1 19 0 123 0 80 0 - 1149556 ep_pol ? 0:00 /usr/bin/java
1 S 1 19 0 282 0 80 0 - 1149556 futex_ ? 0:00 /usr/bin/java
1 S 1 19 0 283 0 80 0 - 1149556 futex_ ? 0:00 /usr/bin/java
1 S 1 19 0 284 0 80 0 - 1149556 futex_ ? 0:00 /usr/bin/java
1 S 1 19 0 285 0 80 0 - 1149556 futex_ ? 0:00 /usr/bin/java
1 S 1 19 0 295 0 80 0 - 1149556 futex_ ? 0:01 /usr/bin/java
1 S 1 19 0 383 0 80 0 - 1149556 futex_ ? 0:00 /usr/bin/java
1 S 1 19 0 433 0 80 0 - 1149556 futex_ ? 0:01 /usr/bin/java
1 S 1 19 0 1109 0 80 0 - 1149556 futex_ ? 0:00 /usr/bin/java
Checking the syscalls, there are a ton of timed out futex calls constantly happening. Here's a snippet from just a fraction of a second:
[pid 43] futex(0x7f36266dfc28, FUTEX_WAKE_PRIVATE, 1) = 0
[pid 43] futex(0x7f36266dfc54, FUTEX_WAIT_BITSET_PRIVATE, 1, {tv_sec=14532, tv_nsec=950443844}, 0xffffffff <unfinished ...>
[pid 47] <... futex resumed> ) = -1 ETIMEDOUT (Connection timed out)
[pid 47] futex(0x7f3626f8ab28, FUTEX_WAKE_PRIVATE, 1) = 0
[pid 47] futex(0x7f3626f8ab54, FUTEX_WAIT_BITSET_PRIVATE, 1, {tv_sec=14532, tv_nsec=954449564}, 0xffffffff <unfinished ...>
[pid 33] <... futex resumed> ) = -1 ETIMEDOUT (Connection timed out)
[pid 33] futex(0x7f3624a11128, FUTEX_WAKE_PRIVATE, 1) = 0
[pid 33] futex(0x7f3624a11154, FUTEX_WAIT_BITSET_PRIVATE, 1, {tv_sec=14532, tv_nsec=998037087}, 0xffffffff <unfinished ...>
[pid 43] <... futex resumed> ) = -1 ETIMEDOUT (Connection timed out)
[pid 43] futex(0x7f36266dfc28, FUTEX_WAKE_PRIVATE, 1) = 0
[pid 43] futex(0x7f36266dfc54, FUTEX_WAIT_BITSET_PRIVATE, 1, {tv_sec=14532, tv_nsec=960733704}, 0xffffffff <unfinished ...>
[pid 47] <... futex resumed> ) = -1 ETIMEDOUT (Connection timed out)
[pid 47] futex(0x7f3626f8ab28, FUTEX_WAKE_PRIVATE, 1) = 0
[pid 47] futex(0x7f3626f8ab54, FUTEX_WAIT_BITSET_PRIVATE, 1, {tv_sec=14532, tv_nsec=964665830}, 0xffffffff <unfinished ...>
[pid 43] <... futex resumed> ) = -1 ETIMEDOUT (Connection timed out)
[pid 43] futex(0x7f36266dfc28, FUTEX_WAKE_PRIVATE, 1) = 0
[pid 43] futex(0x7f36266dfc54, FUTEX_WAIT_BITSET_PRIVATE, 1, {tv_sec=14532, tv_nsec=971001771}, 0xffffffff <unfinished ...>
[pid 47] <... futex resumed> ) = -1 ETIMEDOUT (Connection timed out)
[pid 47] futex(0x7f3626f8ab28, FUTEX_WAKE_PRIVATE, 1) = 0
[pid 47] futex(0x7f3626f8ab54, FUTEX_WAIT_BITSET_PRIVATE, 1, {tv_sec=14532, tv_nsec=973981797}, 0xffffffff <unfinished ...>
[pid 43] <... futex resumed> ) = -1 ETIMEDOUT (Connection timed out)
[pid 43] futex(0x7f36266dfc28, FUTEX_WAKE_PRIVATE, 1) = 0
[pid 43] futex(0x7f36266dfc54, FUTEX_WAIT_BITSET_PRIVATE, 1, {tv_sec=14532, tv_nsec=980553240}, 0xffffffff <unfinished ...>
[pid 47] <... futex resumed> ) = -1 ETIMEDOUT (Connection timed out)
My theory is that this may be the Akka actors rapidly checking to see if there's any new work for them to do, but I am seeking confirmation if that is a valid theory. Thank you!
Related
We are testing our application for performance, which is using Postgresql 13 as a database. It is very insert and update heavy and we cannot get more than 65k TPS on the database. But none of the most common bottlenecks do apply (CPU/network/disk IO).
We have also run multiple combinations with pgbench but also those cannot go past 65k.
E.g.
pgbench -i -s 50 -U postgres -h <DB_HOST> -d <DB_NAME>
pgbench -c 64 -j 32 -t 100000 -h <DB_HOST> -U postgres <DB_NAME>
starting vacuum...end.
transaction type: <builtin: TPC-B (sort of)>
scaling factor: 50
query mode: simple
number of clients: 64
number of threads: 32
number of transactions per client: 100000
number of transactions actually processed: 6400000/6400000
latency average = 0.976 ms
tps = 65584.664360 (including connections establishing)
tps = 65594.330678 (excluding connections establishing)
(Using more clients resulted in fewer TPS)
The database server has the following specs:
64 Cores (Intel Xeon Gold 6130 (Skylake, 2.10GHz, 2 CPUs/node, 16 cores/CPU))
192 GiB RAM (12 * 16GiB DIMM DDR4 Synchronous Registered (Buffered) 2666 MHz (0.4 ns))
2 * SSD SATA Samsung MZ7KM240HMHQ0D3 (one is used for the WAL and the other for the data)
10 Gbps network link
OS: Debian 11
And we are using the following configuration:
shared_buffers=65551953kB
effective_cache_size=147491895kB
huge_pages=on
min_wal_size=4GB
max_wal_size=16GB
wal_buffers=1GB
work_mem=2GB
maintenance_work_mem=4GB
checkpoint_completion_target=0.9
checkpoint_timeout = 15min
random_page_cost=1.1
bgwriter_flush_after = 2MB
effective_io_concurrency = 200
# Disabled just for performance experiments
fsync = off
synchronous_commit = off
full_page_writes = on
max_worker_processes=64
max_parallel_workers=64
max_parallel_workers_per_gather=10
max_parallel_maintenance_workers=12
When we reach the 65k TPS we have the following resource usage on the server:
Our application:
CPU: 25%
I/O Utilization: 30% resp. 60% (WAL)
Network: in 200Mbps out 125Mbps
pgbench:
CPU: 75%
I/O Utilization: 2% resp. 5% (WAL)
Network: in/out ~500Mbps
We have also temporarily placed the WAL and data into RAM to verify that the disks are not the issue, but this had no impact either.
Does anyone have a guess where our bottleneck might be?
(We have thought about sharding already but would like to use the single server as much as possible)
Edit:
vmstat 1 -S M while running pgbench:
procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu-----
r b swpd free buff cache si so bi bo in cs us sy id wa st
1 0 0 92009 25 18316 0 0 3 318 405 22 26 11 63 0 0
0 0 0 92009 25 18316 0 0 0 0 261 320 0 0 100 0 0
0 0 0 92009 25 18316 0 0 0 46616 216 257 0 0 100 0 0
45 0 0 92260 25 17996 0 0 0 0 509124 781029 35 13 52 0 0
44 1 0 92252 25 17998 0 0 0 82192 743920 1140285 51 20 30 0 0
50 0 0 92247 25 18003 0 0 0 65564 740366 1152212 52 18 30 0 0
41 0 0 92243 25 18005 0 0 0 0 736052 1138409 50 20 30 0 0
35 0 0 92240 25 18009 0 0 0 0 734265 1138483 50 20 30 0 0
36 0 0 92238 25 18012 0 0 0 264 741315 1152409 52 19 29 0 0
43 0 0 92236 25 18016 0 0 0 36 740374 1147759 51 20 29 0 0
46 0 0 92231 25 18020 0 0 0 163856 737491 1166498 53 19 28 0 0
47 0 0 92228 25 18023 0 0 0 0 741582 1171814 52 19 28 0 0
43 0 0 92223 25 18027 0 0 0 61584 741537 1168119 53 19 28 0 0
46 0 0 92220 25 18030 0 0 0 36 739695 1167456 53 19 28 0 0
43 0 0 92216 25 18034 0 0 0 228 741992 1150333 52 20 29 0 0
38 0 0 92214 25 18036 0 0 0 147464 740589 1166289 52 19 28 0 0
41 0 0 92209 25 18042 0 0 0 0 737148 1162946 52 19 28 0 0
44 0 0 92207 25 18044 0 0 0 2480 741757 1173128 53 19 28 0 0
39 0 0 92205 25 18049 0 0 0 48 740404 1170644 53 19 28 0 0
52 0 0 92201 25 18051 0 0 0 292 739032 1159037 52 19 28 0 0
42 1 0 92198 25 18054 0 0 0 20072 740101 1165594 52 20 28 0 0
51 0 0 92194 25 18059 0 0 0 45464 738055 1165382 53 19 28 0 0
41 0 0 92190 25 18062 0 0 0 0 742838 1172377 53 19 28 0 0
45 0 0 92185 25 18067 0 0 0 36 740704 1174534 53 19 28 0 0
50 0 0 92182 25 18069 0 0 0 92 741691 1150716 52 19 28 0 0
42 0 0 92177 25 18073 0 0 0 28 740220 1168488 53 18 28 0 0
44 0 0 92174 25 18077 0 0 0 0 738818 1164769 53 19 28 0 0
46 0 0 92172 25 18080 0 0 0 0 740720 1169902 53 19 28 0 0
46 0 0 92166 25 18083 0 0 0 90404 592524 945810 42 15 43 0 0
45 0 0 92161 25 18087 0 0 0 3884 746310 1159898 52 19 28 0 0
49 0 0 92157 25 18090 0 0 0 20 747242 1177750 53 19 28 0 0
36 0 0 92152 25 18094 0 0 0 0 744477 1173832 53 19 28 0 0
46 0 0 92149 25 18098 0 0 0 0 746194 1172700 53 19 28 0 0
39 0 0 92147 26 18101 0 0 0 49768 745651 1177462 54 18 28 0 0
43 0 0 92143 26 18105 0 0 0 212 744968 1161110 53 19 28 0 0
43 0 0 92138 26 18109 0 0 0 0 743223 1176960 54 19 28 0 0
43 0 0 92135 26 18112 0 0 0 81920 745168 1173574 53 19 28 0 0
48 0 0 92132 26 18116 0 0 0 0 743174 1169255 53 19 28 0 0
48 0 0 92129 26 18120 0 0 0 68 592295 933445 42 15 43 0 0
41 0 0 92124 26 18123 0 0 0 76 740354 1162221 52 19 28 0 0
49 0 0 92120 26 18125 0 0 0 0 738456 1158291 53 19 28 0 0
39 0 0 92117 26 18129 0 0 0 147536 740735 1162479 52 20 28 0 0
49 0 0 92113 26 18133 0 0 0 0 737209 1165532 53 20 28 0 0
49 0 0 92111 26 18137 0 0 0 40 741185 1168133 54 19 28 0 0
45 0 0 92110 26 18140 0 0 0 4000 740693 1141945 52 20 28 0 0
42 0 0 92105 26 18144 0 0 0 0 741857 1168830 53 19 28 0 0
43 0 0 92102 26 18147 0 0 0 8 742546 1168867 54 18 28 0 0
43 0 0 92101 26 18150 0 0 0 147456 741941 1166646 53 19 28 0 0
41 1 0 92097 26 18154 0 0 0 64192 740052 1169040 53 19 28 0 0
48 0 0 92094 26 18158 0 0 0 27484 737224 1139511 52 20 28 0 0
47 0 0 92087 26 18162 0 0 0 0 740821 1165037 53 19 28 0 0
54 0 0 92059 26 18164 0 0 0 4 737109 1155098 53 19 27 0 0
38 0 0 92051 26 18170 0 0 0 147456 701847 1075069 55 20 25 0 0
35 0 0 92064 26 18174 0 0 0 44 723153 1125736 54 19 27 0 0
48 0 0 92056 26 18179 0 0 0 53008 734590 1134838 52 19 29 0 0
46 0 0 92053 26 18183 0 0 0 0 741595 1166891 53 19 28 0 0
46 0 0 92049 26 18186 0 0 0 0 740196 1170838 54 19 27 0 0
31 1 0 92045 26 18191 0 0 0 98304 741800 1170076 54 18 28 0 0
44 0 0 92043 26 18194 0 0 0 49188 733352 1173652 53 20 27 0 0
43 0 0 92028 26 18198 0 0 0 116 733522 1151497 53 20 27 0 0
44 0 0 92037 26 18201 0 0 0 0 730364 1137665 53 19 27 0 0
37 0 0 92035 26 18204 0 0 0 0 742348 1164945 53 19 28 0 0
44 0 0 92031 26 18208 0 0 0 0 739273 1165044 53 19 28 0 0
52 0 0 92028 26 18211 0 0 0 147888 739274 1164496 53 19 28 0 0
50 0 0 92024 26 18215 0 0 0 144 739684 1145210 53 19 28 0 0
50 0 0 92020 26 18219 0 0 0 0 742847 1167779 54 18 28 0 0
38 0 0 92016 26 18223 0 0 0 0 738079 1166580 53 19 28 0 0
36 0 0 92013 26 18226 0 0 0 0 742687 1171101 54 18 27 0 0
48 0 0 92009 26 18229 0 0 0 147500 741536 1166846 53 19 28 0 0
40 0 0 92006 26 18233 0 0 0 94600 740746 1147102 52 20 28 0 0
45 0 0 92001 26 18238 0 0 0 0 741119 1163851 53 19 28 0 0
48 0 0 91999 26 18241 0 0 0 0 740995 1167197 53 19 28 0 0
35 0 0 91996 26 18244 0 0 0 0 742235 1165666 53 19 28 0 0
44 1 0 91993 26 18248 0 0 0 49192 741392 1164506 53 19 28 0 0
43 1 0 91990 26 18251 0 0 0 124876 743695 1144639 52 19 29 0 0
48 0 0 91987 26 18255 0 0 0 24864 737759 1159383 52 20 28 0 0
44 0 0 91983 26 18258 0 0 0 0 740224 1164983 53 19 28 0 0
43 0 0 91980 26 18262 0 0 0 0 741742 1168140 54 19 27 0 0
18 0 0 91976 26 18267 0 0 0 36 737449 1162293 53 19 28 0 0
49 0 0 91973 26 18269 0 0 0 147576 741462 1148048 52 20 28 0 0
43 0 0 91969 26 18274 0 0 0 0 742408 1168332 54 19 27 0 0
43 0 0 91966 26 18277 0 0 0 0 738803 1164992 53 19 28 0 0
39 0 0 91963 26 18280 0 0 0 4 737891 1159372 52 19 28 0 0
43 0 0 91962 26 18283 0 0 0 40 741888 1166835 53 19 28 0 0
48 0 0 91958 26 18287 0 0 0 164144 738677 1145900 52 20 28 0 0
46 0 0 91955 26 18291 0 0 0 0 740956 1165789 53 19 28 0 0
44 0 0 91952 26 18295 0 0 0 0 741055 1166460 53 19 28 0 0
44 0 0 91948 26 18299 0 0 0 8 739414 1165698 53 19 28 0 0
46 0 0 91945 26 18301 0 0 0 48 743218 1165277 53 19 28 0 0
36 0 0 91941 26 18305 0 0 0 208 736320 1134425 51 20 29 0 0
47 0 0 91936 26 18309 0 0 0 239096 739799 1159730 52 19 28 0 0
45 0 0 91932 26 18312 0 0 0 0 742477 1167618 53 20 28 0 0
45 0 0 91928 26 18316 0 0 0 0 736442 1159690 52 20 28 0 0
47 0 0 91926 26 18319 0 0 0 76 737145 1157620 52 20 28 0 0
48 0 0 91921 26 18323 0 0 0 64 739999 1146323 52 19 29 0 0
50 0 0 91918 26 18326 0 0 0 197176 739590 1159797 52 19 28 0 0
50 0 0 91915 26 18330 0 0 0 0 740533 1166111 53 19 28 0 0
52 0 0 91911 26 18334 0 0 0 0 739776 1161328 52 20 28 0 0
42 0 0 91907 26 18338 0 0 0 60 590783 929545 41 16 43 0 0
39 0 0 91904 26 18341 0 0 0 4248 744434 1161062 52 19 29 0 0
41 1 0 91900 26 18345 0 0 0 114688 741817 1163511 53 19 28 0 0
14 0 0 91928 26 18349 0 0 0 32768 598242 996868 43 15 42 0 0
0 0 0 91951 26 18349 0 0 0 0 41914 84357 3 1 96 0 0
0 0 0 91952 26 18349 0 0 0 36 174 204 0 0 100 0 0
0 0 0 91954 26 18349 0 0 0 276 7897 13403 0 0 99 0 0
0 0 0 91954 26 18349 0 0 0 0 1911 3678 0 0 100 0 0
0 0 0 91954 26 18349 0 0 0 147456 330 351 0 0 100 0 0
Edit:
The same pgbench benchmark as above gave about 68k when it was run on the same system as the database and select only 540k (507k from a remote system).
Is CPU:75% user or user + system?
Anyway, 75% of OS CPU with hyper-threading means all cores 100% busy.
You use simple protocol tpc-b which is stressing network and context switches.
See https://franckpachot.medium.com/do-you-know-what-you-are-measuring-with-pgbench-d8692a33e3d6
I want to parse these columns to psql table but how can I handle empty fields ?
with this command : " awk '{print $3;}' text.txt " it ignores spaces and writes next column number..
StartTime LastTime SrcAddr DstAddr Sport Dport SrcPkts DstPkts SrcBytes DstBytes
13:14:52.088291 13:14:52.088291 192.168.0.23 192.168.0.255 57621 57621 1 0 86 0
13:14:54.936682 13:14:54.936682 192.168.0.23 192.168.0.255 17500 17500 1 0 243 0
13:14:54.936479 13:14:54.936479 192.168.0.23 17500 17500 1 0 243 0
13:14:56.056179 13:14:56.056179 192.168.0.163 17500 17500 1 0 208 0
13:14:56.056370 13:14:56.056370 192.168.0.163 192.168.0.255 17500 17500 1 0 208 0
13:15:00.027462 13:15:00.027462 192.168.0.170 192.168.0.255 17500 17500 1 0 146 0
13:15:00.652690 13:15:00.652874 192.168.0.166 443 57201 1 1 121 66
13:15:04.636177 13:15:04.636177 192.168.0.163 192.168.0.255 57621 57621 1 0 86 0
13:15:09.622847 13:15:09.622847 192.168.0.166 38029 53 1 0 76 0
13:15:13.138197 13:15:13.138197 192.168.0.158 192.168.0.255 57621 57621 1 0 86 0
13:15:15.429763 13:15:15.429763 192.168.0.151 192.168.0.255 57621 57621 1 0 86 0
13:15:19.793651 13:15:19.793651 192.168.0.163 192.168.0.255 138 138 1 0 280 0
13:15:19.795815 13:15:19.795815 192.168.0.90 192.168.0.255 137 137 1 0 92 0
13:15:20.342669 13:15:20.342669 192.168.0.23 192.168.0.255 137 137 1 0 92 0
13:14:40.721537 13:14:42.016881 192.168.0.136 192.168.0.166 22 59301 11 14 1334 1404
13:15:24.981466 13:15:24.981466 192.168.0.23 255.255.255.255 17500 17500 1 0 243 0
13:15:24.981666 13:15:24.981666 192.168.0.23 192.168.0.255 17500 17500 1 0 243 0
13:14:40.996353 13:14:41.996328 192.168.0.22 2 0 120 0
13:14:41.203309 13:15:41.171881 66 66 31 0 1984 0
13:15:26.116537 13:15:26.116537 192.168.0.163 17500 17500 1 0 208 0
13:15:26.116728 13:15:26.116728 192.168.0.163 192.168.0.255 17500 17500 1 0 208 0
13:14:41.863153 13:15:41.852795 0 0 31 0 1860 0
13:15:01.195960 13:15:03.192229 192.168.0.234 3 0 180 0
13:15:04.636774 13:15:04.636774 192.168.0.91 192.168.0.163 1 0 60 0
13:15:10.398423 13:15:10.398423 192.168.0.110 192.168.0.1 1 0 60 0
13:15:40.800831 13:15:40.913802 192.168.0.152 192.168.0.1 49556 53 2 2 148 689
You can specify by position where to get data, trim it and print it with awk
awk '{a=substr($0,40,17);gsub(/ */,"",a);print a}' file
SrcAddr
192.168.0.23
192.168.0.23
192.168.0.23
192.168.0.163
192.168.0.170
192.168.0.163
192.168.0.166
192.168.0.158
192.168.0.151
192.168.0.163
192.168.0.90
192.168.0.23
192.168.0.136
192.168.0.23
192.168.0.23
192.168.0.163
192.168.0.163
192.168.0.91
192.168.0.110
192.168.0.152
It LOOKS like you have fixed-width fields so using GNU awk:
$ cat file
abcd ef ghi klmno
1234 12 123
$ awk -v FIELDWIDTHS="4 1 2 1 3 1 5" '{print "<" $3 ">"}' file
<ef>
<12>
$ awk -v FIELDWIDTHS="4 1 2 1 3 1 5" '{print "<" $5 ">"}' file
<ghi>
< >
and if you want to strip leading and trailing white space from each field for output just add a gsub():
$ awk -v FIELDWIDTHS="4 1 2 1 3 1 5" '{gsub(/^\s+|\s+$/,"",$5); print "<" $5 ">"}' file
<ghi>
<>
$ awk -v FIELDWIDTHS="4 1 2 1 3 1 5" '{print "<" $7 ">"}' file
<klmno>
< 123 >
$ awk -v FIELDWIDTHS="4 1 2 1 3 1 5" '{gsub(/^\s+|\s+$/,"",$7); print "<" $7 ">"}' file
<klmno>
<123>
I am using mongodb for our Application .
I ran the db.setProfilingLevel(1,25) on the mongo shell to identify the slow queries from system.profile collection .
I observed that the read operations are fast , but the update operations are very slow
This is a sample of my mongostat
insert/s query/s update/s delete/s getmore/s command/s flushes/s mapped vsize res faults/s locked % idx miss % q t|r|w conn time
0 950 469 0 0 471 0 10396 12295 3207 27 34.9 0 0|0|0 152 07:18:49
0 838 418 0 0 422 0 10396 12295 3209 21 34.6 0 0|0|0 152 07:18:50
0 1005 502 0 0 504 0 10396 12295 3211 21 35.5 0 0|0|0 152 07:18:51
0 837 410 0 0 418 0 10396 12295 3212 20 35.7 0 0|0|0 152 07:18:52
0 754 377 0 0 379 0 10396 12295 3214 19 36.7 0 0|0|0 152 07:18:53
0 841 420 0 0 422 0 10396 12295 3216 24 35.9 0 0|0|0 152 07:18:54
0 877 438 0 0 442 0 10396 12295 3217 23 37.2 0 0|0|0 152 07:18:55
0 799 393 0 0 395 0 10396 12295 3219 21 37 0 0|0|0 152 07:18:56
0 947 471 0 0 479 0 10396 12295 3221 26 39 0 0|0|0 152 07:18:57
0 855 427 0 0 429 0 10396 12295 3222 24 38.4 0 0|0|0 152 07:18:58
0 1007 504 0 0 506 0 10396 12295 3224 31 36 0 0|0|0 152 07:18:59
0 841 413 0 0 417 0 10396 12295 3226 23 37.2 0 0|0|0 152 07:19:00
The stats are from dev environment , cant assume really from prod environment .
As per the architecture i cannot reduce the index size on that collection , but i saw that usePowerOf2Sizes can help me in this case in improving the write/update response time in mongodb .
I have heard lot of usePowerOf2Sizes , which says that
As usePowerOf2Sizes can reduce fragmentation .
all data will be set in mmemory and performance will be great .
With this option MongoDB will be able to more effectively reuse space.
usePowerOf2Sizes is useful for collections where you will be inserting and deleting large numbers of documents to ensure that MongoDB will effectively use space on disk.
I want to know if there will be any negative impact in using the option usePowerOf2Sizes ?? I have got 17 collections in my mongodb and want to use usePowerOf2Sizes for only one collection .
Please let me know , thanks in advance .
Is there any way to load read-only variables into memory before forking and keep them there without using up virt memory x number of children?
It seems to be a common opinion that read-only memory is shared by default and is copied on write. I ran a test and found this to be untrue:
#!/usr/bin/perl
my $data;
$$data = 'a'x 1_000_000; #keep it in a ref just in case that matters
foreach (0..10){
last unless my $pid = fork();
}
<STDIN>;
While the process is sitting on STDIN, I check top:
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
15982 wdev 20 0 121m 2540 188 S 0.0 0.1 0:00.00 foo.t
15983 wdev 20 0 121m 2540 188 S 0.0 0.1 0:00.00 foo.t
15984 wdev 20 0 121m 2540 188 S 0.0 0.1 0:00.00 foo.t
15985 wdev 20 0 121m 2540 188 S 0.0 0.1 0:00.00 foo.t
15986 wdev 20 0 121m 2540 188 S 0.0 0.1 0:00.00 foo.t
15987 wdev 20 0 121m 2540 188 S 0.0 0.1 0:00.00 foo.t
15988 wdev 20 0 121m 2540 188 S 0.0 0.1 0:00.00 foo.t
15989 wdev 20 0 121m 2540 188 S 0.0 0.1 0:00.00 foo.t
15990 wdev 20 0 121m 2540 188 S 0.0 0.1 0:00.00 foo.t
15991 wdev 20 0 121m 2540 188 S 0.0 0.1 0:00.00 foo.t
15992 wdev 20 0 121m 2540 188 S 0.0 0.1 0:00.00 foo.t
Sure enough, practically all the memory is in individual child processes.
Is there something I am missing, or does forking a perl process really copy over the entire set of data structures for each child?
Either this is operating-system dependent or you're not seeing what you think you're seeing. I modified your test script to:
#!/usr/bin/env perl
print `free`;
my $data = [ ('a') x 1000000 ];
for (1 .. 10) {
print `free`;
last unless fork;
}
sleep 10;
Running it with ./forktest | grep Mem, I get output such as:
Mem: 3979908 2866552 1113356 0 667388 1258560
Mem: 3979908 2917888 1062020 0 667388 1258560
Mem: 3979908 2918284 1061624 0 667388 1258560
Mem: 3979908 2918532 1061376 0 667388 1258560
Mem: 3979908 2918936 1060972 0 667388 1258560
Mem: 3979908 2919404 1060504 0 667388 1258560
Mem: 3979908 2919900 1060008 0 667388 1258560
Mem: 3979908 2919900 1060008 0 667388 1258560
Mem: 3979908 2920148 1059760 0 667388 1258560
Mem: 3979908 2920148 1059760 0 667388 1258560
Mem: 3979908 2920496 1059412 0 667388 1258560
in which the second column of numbers (third column overall) is the total system RAM usage. Note that it increases from 2866552 to 2917888 when $data is assigned at the start of the program, then stays fairly steady from there on as the forks are done.
I suspect that what you're seeing in top is that it uses "shared memory" in the IPC-specific sense (i.e., chunks of memory which have been explicitly requested and allocated as "shared") and pages which are currently available to multiple processes on a copy-on-write basis do not meet that definition.
The perl docs on fork say "File descriptors (and sometimes locks on those descriptors) are shared, while everything else is copied."
hello i need to perform quantization to dct cofficents for an image, for a block
size of 8*8 pixles in matlab. can you help me with the syntax, thank you.
There is a built-in function in MATLAB for DCT.
You need the signal processing tool box. Type 'ver' (without quotes) in the MATLAB command to see if you have it.
The code:
image = image; % define your image
[m,n] = size(image); % get size of your image
imvector = reshape(image, m*n, 1); % reshape your image to a vector to compute DCT
imdct = dct(imvector); % compute DCT
imagedct = reshape(imdct,m,n); \ reshape result back to original form of your image
There is an example in the help file as well which is very nice:
I = imread('cameraman.tif');
I = im2double(I);
T = dctmtx(8);
dct = #(block_struct) T * block_struct.data * T';
B = blockproc(I,[8 8],dct);
mask = [1 1 1 1 0 0 0 0
1 1 1 0 0 0 0 0
1 1 0 0 0 0 0 0
1 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0];
B2 = blockproc(B,[8 8],#(block_struct) mask .* block_struct.data);
invdct = #(block_struct) T' * block_struct.data * T;
I2 = blockproc(B2,[8 8],invdct);
imshow(I), figure, imshow(I2)
To quantize DCT coefficients, you simply divide each coefficient by a quantization term and round to integers. The quantization terms are often unique for each coefficient, and are stored in a quantization matrix.
Wikipedia has a nice example. Here is how to implement that example in Matlab.
coef = [
-415 -33 -58 35 58 -51 -15 -12;
5 -34 49 18 27 1 -5 3;
-46 14 80 -35 -50 19 7 -18;
-53 21 34 -20 2 34 36 12;
9 -2 9 -5 -32 -15 45 37;
-8 15 -16 7 -8 11 4 7;
19 -28 -2 -26 -2 7 -44 -21;
18 25 -12 -44 35 48 -37 -3
];
quant = [
16 11 10 16 24 40 51 61;
12 12 14 19 26 58 60 55;
14 13 16 24 40 57 69 56;
14 17 22 29 51 87 80 62;
18 22 37 56 68 109 103 77;
24 35 55 64 81 104 113 92;
49 64 78 87 103 121 120 101;
72 92 95 98 112 100 103 99
];
quantCoef = round(coef ./ quant)
quantCoef =
-26 -3 -6 2 2 -1 0 0
0 -3 4 1 1 0 0 0
-3 1 5 -1 -1 0 0 0
-4 1 2 -1 0 0 0 0
1 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0
clc
clear all
close all
image=imread('Water lilies.jpg');
Q=[8 36 36 36 39 45 52 65;
36 36 36 37 41 47 56 68;
36 36 38 42 47 54 64 78;
36 37 42 50 59 69 81 98;
39 41 47 54 73 89 108 130;
45 47 54 69 89 115 144 178;
53 56 64 81 108 144 190 243;
65 68 78 98 130 178 243 255];
GrayImage=rgb2gray(image);
NewImage=uint8(zeros(size(GrayImage)));
Q=uint32(Q);
for i=1:size(GrayImage,1)/8
for j=1:size(GrayImage,2)/8
block=GrayImage(((i-1)*8)+1:((i-1)*8)+8,((j-1)*8)+1:((j-1)*8)+8);
dct=dct2(block);
dct=uint32(dct);
a=dct./Q;
z=a.*Q;
idct=idct2(z);
NewImage(((i-1)*8)+1:((i-1)*8)+8,((j-1)*8)+1:((j-1)*8)+8)=uint8(idct);
end
end
imwrite(NewImage,'NewImage.jpg');
GrayImage=double(GrayImage);
NewImage=double(NewImage);
MSE=0;
for i=1:size(GrayImage,1)
for j=1:size(GrayImage,2)
d=(GrayImage(i,j)-NewImage(i,j))^2;
MSE=d+MSE;
end
end