MongoDB slow - memory usage very high - mongodb

I have a 5 note replicaset mongoDB - 1 primary, 3 secondaries and 1 arbiter.
I am using mong version 4.2.3
Sizes:
“dataSize” : 688.4161271536723,
“indexes” : 177,
“indexSize” : 108.41889953613281
My Primary is very slow - each command from the shell takes a long time to return.
Memory usage seems very high, and looks like mongodb is consuming more than 50% of the RAM:
# free -lh
total used free shared buff/cache available
Mem: 188Gi 187Gi 473Mi 56Mi 740Mi 868Mi
Low: 188Gi 188Gi 473Mi
High: 0B 0B 0B
Swap: 191Gi 117Gi 74Gi
------------------------------------------------------------------
Top Memory Consuming Process Using ps command
------------------------------------------------------------------
PID PPID %MEM %CPU CMD
311 49145 97.8 498 mongod --config /etc/mongod.conf
23818 23801 0.0 3.8 /bin/prometheus --config.file=/etc/prometheus/prometheus.yml
23162 23145 0.0 8.4 /usr/bin/cadvisor -logtostderr
25796 25793 0.0 0.4 postgres: checkpointer
23501 23484 0.0 1.0 /postgres_exporter
24490 24473 0.0 0.1 grafana-server --homepath=/usr/share/grafana --config=/etc/grafana/grafana.ini --packaging=docker cfg:default.log.mode=console
top:
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
311 systemd+ 20 0 313.9g 184.6g 2432 S 151.7 97.9 26229:09 mongod
23818 nobody 20 0 11.3g 150084 17988 S 20.7 0.1 8523:47 prometheus
23162 root 20 0 12.7g 93948 5964 S 65.5 0.0 18702:22 cadvisor
serverStatus memeory shows this:
octopusrs0:PRIMARY> db.serverStatus().mem
{
"bits" : 64,
"resident" : 189097,
"virtual" : 321404,
"supported" : true
}
octopusrs0:PRIMARY> db.serverStatus().tcmalloc.tcmalloc.formattedString
------------------------------------------------
MALLOC: 218206510816 (208097.9 MiB) Bytes in use by application
MALLOC: + 96926863360 (92436.7 MiB) Bytes in page heap freelist
MALLOC: + 3944588576 ( 3761.9 MiB) Bytes in central cache freelist
MALLOC: + 134144 ( 0.1 MiB) Bytes in transfer cache freelist
MALLOC: + 713330688 ( 680.3 MiB) Bytes in thread cache freelists
MALLOC: + 1200750592 ( 1145.1 MiB) Bytes in malloc metadata
MALLOC: ------------
MALLOC: = 320992178176 (306122.0 MiB) Actual memory used (physical + swap)
MALLOC: + 13979086848 (13331.5 MiB) Bytes released to OS (aka unmapped)
MALLOC: ------------
MALLOC: = 334971265024 (319453.5 MiB) Virtual address space used
MALLOC:
MALLOC: 9420092 Spans in use
MALLOC: 234 Thread heaps in use
MALLOC: 4096 Tcmalloc page size
------------------------------------------------
Call ReleaseFreeMemory() to release freelist memory to the OS (via madvise()).
Bytes released to the OS take up virtual address space but no physical memory.
How can I detrmine what is causing this high memory consumption, and what can I do to return to nornal memory consumption?
Thanks,
Tamar

Related

what are using large mem_reserve in a .net application?

We have an asp.net web service application that is having OutOfMemory issue with IIS w3wp.exe. We got mini memory dump of w3wp when OOM happened, and we see it has large mem_reserve but small managed heap size. Question is how can we find out what is using mem_reserve (virtual address space). I expect to see some mem_reserve but not 2.9G. Since mem_reserve doesn't add up to eeheap, would it be from unmanaged heap? then anyway to confirm that?
0:000> !address -summary
Mapping file section regions...
Mapping module regions...
Mapping PEB regions...
Mapping TEB and stack regions...
Mapping heap regions...
Mapping page heap regions...
Mapping other regions...
Mapping stack trace database regions...
Mapping activation context regions...
--- Usage Summary ---------------- RgnCount ----------- Total Size -------- %ofBusy %ofTotal
<unknown> 1664 d4c1d000 ( 3.324 GB) 92.34% 83.11%
Free 348 19951000 ( 409.316 MB) 9.99%
Image 1184 ad58000 ( 173.344 MB) 4.70% 4.23%
Heap 75 5ab4000 ( 90.703 MB) 2.46% 2.21%
Stack 193 1200000 ( 18.000 MB) 0.49% 0.44%
TEB 64 40000 ( 256.000 kB) 0.01% 0.01%
Other 10 35000 ( 212.000 kB) 0.01% 0.01%
PEB 1 1000 ( 4.000 kB) 0.00% 0.00%
--- Type Summary (for busy) ------ RgnCount ----------- Total Size -------- %ofBusy %ofTotal
MEM_PRIVATE 1323 d6f04000 ( 3.358 GB) 93.28% 83.96%
MEM_IMAGE 1828 dd75000 ( 221.457 MB) 6.01% 5.41%
MEM_MAPPED 40 1a26000 ( 26.148 MB) 0.71% 0.64%
--- State Summary ---------------- RgnCount ----------- Total Size -------- %ofBusy %ofTotal
MEM_RESERVE 746 b8b4e000 ( 2.886 GB) 80.16% 72.15%
MEM_COMMIT 2445 2db51000 ( 731.316 MB) 19.84% 17.85%
MEM_FREE 348 19951000 ( 409.316 MB) 9.99%
--- Protect Summary (for commit) - RgnCount ----------- Total Size -------- %ofBusy %ofTotal
PAGE_READWRITE 1024 1f311000 ( 499.066 MB) 13.54% 12.18%
PAGE_EXECUTE_READ 248 b2e6000 ( 178.898 MB) 4.85% 4.37%
PAGE_READONLY 613 1c1d000 ( 28.113 MB) 0.76% 0.69%
PAGE_WRITECOPY 219 103e000 ( 16.242 MB) 0.44% 0.40%
PAGE_EXECUTE_READWRITE 144 59c000 ( 5.609 MB) 0.15% 0.14%
PAGE_EXECUTE_WRITECOPY 69 21f000 ( 2.121 MB) 0.06% 0.05%
PAGE_READWRITE|PAGE_GUARD 128 144000 ( 1.266 MB) 0.03% 0.03%
--- Largest Region by Usage ----------- Base Address -------- Region Size ----------
<unknown> c2010000 4032000 ( 64.195 MB)
Free 86010000 4000000 ( 64.000 MB)
Image 72f48000 f5d000 ( 15.363 MB)
Heap 64801000 fcf000 ( 15.809 MB)
Stack 1ed0000 7a000 ( 488.000 kB)
TEB ffe8a000 1000 ( 4.000 kB)
Other fffb0000 23000 ( 140.000 kB)
PEB fffde000 1000 ( 4.000 kB)
0:000> !ao
---------Heap 1 ---------
Managed OOM occured after GC #2924 (Requested to allocate 0 bytes)
Reason: Didn't have enough memory to allocate an LOH segment
Detail: LOH: Failed to reserve memory (117440512 bytes)
---------Heap 7 ---------
Managed OOM occured after GC #2308 (Requested to allocate 0 bytes)
Reason: Could not do a full GC
0:000> !vmstat
TYPE MINIMUM MAXIMUM AVERAGE BLK COUNT TOTAL
~~~~ ~~~~~~~ ~~~~~~~ ~~~~~~~ ~~~~~~~~~ ~~~~~
Free:
Small 4K 64K 34K 219 7,475K
Medium 72K 832K 273K 80 21,903K
Large 1,080K 65,536K 7,954K 49 389,759K
Summary 4K 65,536K 1,204K 348 419,139K
Reserve:
Small 4K 64K 14K 436 6,519K
Medium 88K 1,020K 303K 169 51,303K
Large 1,088K 65,528K 21,052K 141 2,968,407K
Summary 4K 65,528K 4,056K 746 3,026,231K
Commit:
Small 4K 64K 16K 2,019 33,386K
Medium 68K 1,024K 287K 275 79,043K
Large 1,028K 65,736K 7,315K 87 636,435K
Summary 4K 65,736K 314K 2,381 748,866K
Private:
Small 4K 64K 28K 851 23,911K
Medium 88K 1,024K 313K 222 69,687K
Large 1,028K 65,736K 18,429K 186 3,427,951K
Summary 4K 65,736K 2,797K 1,259 3,521,550K
Mapped:
Small 4K 64K 18K 23 431K
Medium 68K 1,004K 385K 11 4,239K
Large 1,520K 6,640K 3,684K 6 22,104K
Summary 4K 6,640K 669K 40 26,775K
Image:
Small 4K 64K 9K 1,581 15,562K
Medium 68K 1,000K 267K 211 56,419K
Large 1,028K 15,732K 4,299K 36 154,787K
Summary 4K 15,732K 124K 1,828 226,771K
0:000> !eeheap -gc
Number of GC Heaps: 8
------------------------------
Heap 0 (01cd1720)
generation 0 starts at 0x44470de4
generation 1 starts at 0x43f51000
generation 2 starts at 0x02161000
ephemeral segment allocation context: none
segment begin allocated size
02160000 02161000 02dee4a8 0xc8d4a8(13161640)
43f50000 43f51000 444b9364 0x568364(5669732)
Large object heap starts at 0x12161000
segment begin allocated size
12160000 12161000 121bd590 0x5c590(378256)
c2010000 c2011000 c6041020 0x4030020(67305504)
Heap Size: Size: 0x5281dbc (86515132) bytes.
------------------------------
Heap 1 (01cd65a8)
generation 0 starts at 0x8c4b38e0
generation 1 starts at 0x8c011000
generation 2 starts at 0x04161000
ephemeral segment allocation context: none
segment begin allocated size
04160000 04161000 054114f0 0x12b04f0(19596528)
25c40000 25c41000 25fc7328 0x386328(3695400)
8c010000 8c011000 8c4c4778 0x4b3778(4929400)
Large object heap starts at 0x13161000
segment begin allocated size
13160000 13161000 13161010 0x10(16)
Heap Size: Size: 0x1ae9fa0 (28221344) bytes.
------------------------------
Heap 2 (01cdb5c0)
generation 0 starts at 0x4a71a420
generation 1 starts at 0x49f51000
generation 2 starts at 0x06161000
ephemeral segment allocation context: none
segment begin allocated size
06160000 06161000 06e89b18 0xd28b18(13798168)
27c40000 27c41000 27f8f6dc 0x34e6dc(3466972)
9a010000 9a011000 9a5900b8 0x57f0b8(5763256)
49f50000 49f51000 4a72cc2c 0x7dbc2c(8240172)
Large object heap starts at 0x14161000
segment begin allocated size
14160000 14161000 14161010 0x10(16)
Heap Size: Size: 0x1dd1ee8 (31268584) bytes.
------------------------------
Heap 3 (01ce05d8)
generation 0 starts at 0x34fbe540
generation 1 starts at 0x34d11000
generation 2 starts at 0x08161000
ephemeral segment allocation context: none
segment begin allocated size
08160000 08161000 08c26248 0xac5248(11293256)
2bc40000 2bc41000 2bfc9294 0x388294(3703444)
34d10000 34d11000 34fcdbb0 0x2bcbb0(2870192)
Large object heap starts at 0x15161000
segment begin allocated size
15160000 15161000 151c3b10 0x62b10(404240)
Heap Size: Size: 0x116cb9c (18271132) bytes.
------------------------------
Heap 4 (01ce55f0)
generation 0 starts at 0x934a7cec
generation 1 starts at 0x93011000
generation 2 starts at 0x0a161000
ephemeral segment allocation context: none
segment begin allocated size
0a160000 0a161000 0b4bf898 0x135e898(20310168)
45f50000 45f51000 45f70df4 0x1fdf4(130548)
93010000 93011000 934b83ec 0x4a73ec(4879340)
Large object heap starts at 0x16161000
segment begin allocated size
16160000 16161000 161ab050 0x4a050(303184)
Heap Size: Size: 0x186fac8 (25623240) bytes.
------------------------------
Heap 5 (01cec608)
generation 0 starts at 0x31f1d4d0
generation 1 starts at 0x31c41000
generation 2 starts at 0x0c161000
ephemeral segment allocation context: none
segment begin allocated size
0c160000 0c161000 0d2120b0 0x10b10b0(17502384)
7aea0000 7aea1000 7af5c074 0xbb074(766068)
31c40000 31c41000 31f2caac 0x2ebaac(3062444)
Large object heap starts at 0x17161000
segment begin allocated size
17160000 17161000 17179ad0 0x18ad0(101072)
Heap Size: Size: 0x14706a0 (21431968) bytes.
------------------------------
Heap 6 (01cf5a20)
generation 0 starts at 0x5fbb5598
generation 1 starts at 0x5f821000
generation 2 starts at 0x0e161000
ephemeral segment allocation context: none
segment begin allocated size
0e160000 0e161000 0eb22284 0x9c1284(10228356)
5f820000 5f821000 5fbc9838 0x3a8838(3835960)
Large object heap starts at 0x18161000
segment begin allocated size
18160000 18161000 18161010 0x10(16)
Heap Size: Size: 0xd69acc (14064332) bytes.
------------------------------
Heap 7 (01cfaa38)
generation 0 starts at 0xad530e00
generation 1 starts at 0xad011000
generation 2 starts at 0x10161000
ephemeral segment allocation context: none
segment begin allocated size
10160000 10161000 109bd218 0x85c218(8765976)
a4010000 a4011000 a42f418c 0x2e318c(3027340)
41f50000 41f51000 42a99090 0xb48090(11829392)
ad010000 ad011000 ad5399a0 0x5289a0(5409184)
Large object heap starts at 0x19161000
segment begin allocated size
19160000 19161000 1980ddb0 0x6acdb0(6999472)
Heap Size: Size: 0x225cb84 (36031364) bytes.
------------------------------
GC Heap Size: Size: 0xf950f98 (261427096) bytes.
0:000> !heap -s
SEGMENT HEAP ERROR: failed to initialize the extention
LFH Key : 0x3a251113
Termination on corruption : ENABLED
Heap Flags Reserv Commit Virt Free List UCR Virt Lock Fast
(k) (k) (k) (k) length blocks cont. heap
-----------------------------------------------------------------------------
Virtual block: 1a830000 - 1a830000 (size 00000000)
Virtual block: 1a9d0000 - 1a9d0000 (size 00000000)
007b0000 00000002 16384 15520 16384 198 418 5 2 3 LFH
00280000 00001002 3136 1608 3136 7 23 3 0 0 LFH
00a40000 00001002 1088 828 1088 79 25 2 0 0 LFH
00c30000 00001002 1088 308 1088 13 5 2 0 0 LFH
00100000 00041002 256 4 256 2 1 1 0 0
01560000 00001002 256 24 256 3 4 1 0 0
01510000 00001002 64 4 64 2 1 1 0 0
01670000 00001002 64 4 64 2 1 1 0 0
00290000 00011002 256 4 256 1 2 1 0 0
01870000 00001002 256 8 256 3 3 1 0 0
004e0000 00001002 256 4 256 1 2 1 0 0
019e0000 00001002 256 4 256 2 1 1 0 0
01af0000 00001002 256 4 256 2 1 1 0 0
02080000 00001002 1280 396 1280 1 34 2 0 0 LFH
01fc0000 00041002 256 4 256 2 1 1 0 0
1b700000 00041002 256 256 256 6 4 1 0 0 LFH
21840000 00001002 64192 1880 64192 1639 13 12 0 0 LFH
External fragmentation 87 % (13 free blocks)
-----------------------------------------------------------------------------
If we recycle w3wp, then all memory will go back to system. So there seems no memory leak. I understand that mem_reserve is mapped to virtual address, and are allocated space but not committed. Question is: is there a way to figure out what is using up all these virtual address.
I have memory dump, but I don't have access the the server to get performance counter or something.
Thanks folks!!

What is the difference bewteen SWAP and Virtual mmeory in Linux

Why I am not seeing any SWAP usage in below scenario, even though the virtual memory is showing 20gig ?
Tasks: 186 total, 1 running, 185 sleeping, 0 stopped, 0 zombie
Cpu(s): 0.2%us, 0.2%sy, 0.0%ni, 99.7%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Mem: 32880880k total, 17555744k used, 15325136k free, 300268k buffers
Swap: 16383996k total, 0k used, 16383996k free, 2287700k cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
10314 abcuser 20 0 20.3g 6.8g 25m S 0.7 21.8 23:47.38 java
25119 abcuser 20 0 20.3g 6.7g 25m S 0.3 21.4 23:38.25 java
total used free shared buffers cached
Mem: 31 16 14 0 0 2
-/+ buffers/cache: 14 17
Swap: 15 0 15
Total: 46 16 30
Regards
Sanjeev

Postgres Out of Memory

I am setting up a new install of postgres 10.4 and during load testing I keep receiving an out of memory error.
Configuration:
$ cat /etc/security/limits.conf
postgres hard memlock 508559360
postgres soft memlock 508559360
$ cat /etc/sysctl.conf
vm.nr_hugepages = 248320
vm.hugetlb_shm_group = 118
vm.overcommit_memory=2
vm.swappiness=1
vm.vfs_cache_pressure=50
kernel.sem = 250 32000 32 128
System Specs:
$ cat /proc/cpuinfo | grep "core id" | wc -l
32
Each CPU is an Intel(R) Xeon(R) CPU E5-2695 v4 # 2.10GHz. 16 physical cores, 32 logical as shown above.
$ free -m
free -m
total used free shared buff/cache available
Mem: 510969 498048 2434 3098 10485 8343
Swap: 1071 1071 0
Note: we have the memlock set at approximately 485 GB dedicated directly to postgres which shows in the high "used" column.
$ df -h /dev/shm
Filesystem Size Used Avail Use% Mounted on
tmpfs 250G 64K 250G 1% /dev/shm
$ cat /proc/meminfo
MemTotal: 523232496 kB
MemFree: 1216648 kB
MemAvailable: 8348808 kB
Buffers: 65220 kB
Cached: 10211780 kB
SwapCached: 7620 kB
Active: 5556092 kB
Inactive: 5432428 kB
Active(anon): 2208064 kB
Inactive(anon): 1593084 kB
Active(file): 3348028 kB
Inactive(file): 3839344 kB
Unevictable: 24128 kB
Mlocked: 24128 kB
SwapTotal: 1097724 kB
SwapFree: 0 kB
Dirty: 52 kB
Writeback: 0 kB
AnonPages: 728044 kB
Mapped: 62904 kB
Shmem: 3085592 kB
Slab: 1454276 kB
SReclaimable: 1345008 kB
SUnreclaim: 109268 kB
KernelStack: 11984 kB
PageTables: 22856 kB
NFS_Unstable: 0 kB
Bounce: 0 kB
WritebackTmp: 0 kB
CommitLimit: 15770860 kB
Committed_AS: 7920136 kB
VmallocTotal: 34359738367 kB
VmallocUsed: 0 kB
VmallocChunk: 0 kB
HardwareCorrupted: 0 kB
AnonHugePages: 71680 kB
CmaTotal: 0 kB
CmaFree: 0 kB
HugePages_Total: 248320
HugePages_Free: 245883
HugePages_Rsvd: 1800
HugePages_Surp: 0
Hugepagesize: 2048 kB
DirectMap4k: 329600 kB
DirectMap2M: 534444032 kB
Postgres Info:
SELECT version();
PostgreSQL 10.4 on x86_64-pc-linux-gnu, compiled by gcc (Ubuntu 5.4.0-6ubuntu1~16.04.9) 5.4.0 20160609, 64-bit
max_connections=600
shared_buffers= '8192 MB'
work_mem = '20 MB'
maintenance_work_mem = 2GB
max_parallel_workers = 8
wal_buffers = 16MB
max_wal_size = 20GB
min_wal_size = 1GB
checkpoint_completion_target = 0.9
effective_cache_size = '364 GB'
default_statistics_target = 1000
log_timezone = 'US/Eastern'
track_activities = on
track_counts = on
track_io_timing = on
stats_temp_directory = 'pg_stat_tmp'
datestyle = 'iso, mdy'
timezone = 'US/Eastern'
default_text_search_config = 'pg_catalog.english'
transform_null_equals = on
shared_preload_libraries = 'pg_stat_statements'
track_activity_query_size = 16384
track_functions = all
track_io_timing = true
pg_stat_statements.track = all
session_preload_libraries = 'auto_explain'
auto_explain.log_min_duration = '3s'
auto_explain.log_nested_statements='on'
auto_explain.log_analyze=true
Note: I've started w/ work memory set as high as 250 MB and have slowly brought it down to 20 MB and still receive the errors. I've also verified that connections don't go higher than 120 connections. We use PGBouncer in front of the instance in session mode.
The errors are too large for stack overflow but here they are linked:
https://codepad.co/snippet/derPU4E8
The standout errors are:
2018-06-18 15:02:22 EDT,28197,mydb,ERROR: could not resize shared memory segment "/PostgreSQL.1552129380" to 192088 bytes: No space left on device
I don't understand what device its talking about. I don't have any that appear to be even close to OOM let alone the tiny 192088 bytes its talking about.
2018-06-18 15:02:22 EDT,19708,mydb,ERROR: out of memory
2018-06-18 15:02:22 EDT,19708,mydb,DETAIL: Failed on request of size 7232.
2018-06-18 15:02:22 EDT,16688,,LOG: could not fork worker process: Cannot allocate memory
2018-06-18 15:02:22 EDT,4555,,ERROR: out of memory
2018-06-18 15:02:22 EDT,4555,,DETAIL: Failed on request of size 78336.
2018-06-18 15:02:22 EDT,4552,,LOG: could not open directory "/usr/lib/postgresql/10/share/timezone": Cannot allocate memory
2018-06-18 15:02:22 EDT,19935,mydb,ERROR: could not load library "/usr/lib/postgresql/10/lib/auto_explain.so": /usr/lib/postgresql/10/lib/auto_explain.so: failed to map segment from shared object
2018-06-18 15:02:22 EDT,28193,mydb,ERROR: out of memory
2018-06-18 15:02:22 EDT,28193,mydb,DETAIL: Failed on request of size 8192.
2018-06-18 15:02:22 EDT,26927,mydb,ERROR: could not resize shared memory segment "/PostgreSQL.1101931262" to 192088 bytes: No space left on device
Question: How do I debug this issue and more importantly how can I resolve it?
I followed Laurenz advice and no longer run OOM.
vm.overcommit_ratio = 100.
Additionally, I removed the memlock and huge pages definition in sysctl/limits.
Check to see if your queries are so complicated that they need to consume a large amount of temporary tablespace.

How to find out what caused PTE exhaustion?

I have my Windows 10 bluescreen several times and I have memory dump and running !vm produces output below showing 0 available PTEs. How do I find out postmortem who leaked those or monitor on live system which process/driver responsible for leak?
0: kd> !vm
Page File: \??\C:\pagefile.sys
Current: 9961472 Kb Free Space: 9961464 Kb
Minimum: 9961472 Kb Maximum: 62351824 Kb
Page File: \??\C:\swapfile.sys
Current: 16384 Kb Free Space: 16376 Kb
Minimum: 16384 Kb Maximum: 49881460 Kb
No Name for Paging File
Current: 129378408 Kb Free Space: 129362880 Kb
Minimum: 129378408 Kb Maximum: 129378408 Kb
Physical Memory: 16756646 ( 67026584 Kb)
Available Pages: 11876350 ( 47505400 Kb)
ResAvail Pages: 15917045 ( 63668180 Kb)
Locked IO Pages: 0 ( 0 Kb)
Free System PTEs: 0 ( 0 Kb)
********** Running out of system PTEs **************
Modified Pages: 249289 ( 997156 Kb)
Modified PF Pages: 249261 ( 997044 Kb)
Modified No Write Pages: 25 ( 100 Kb)
NonPagedPool Usage: 3042 ( 12168 Kb)
NonPagedPoolNx Usage: 103383 ( 413532 Kb)
NonPagedPool Max: 4294967296 (17179869184 Kb)
PagedPool 0: 144048 ( 576192 Kb)
PagedPool 1: 31595 ( 126380 Kb)
PagedPool 2: 31923 ( 127692 Kb)
PagedPool 3: 31631 ( 126524 Kb)
PagedPool 4: 31714 ( 126856 Kb)
PagedPool Usage: 270911 ( 1083644 Kb)
PagedPool Maximum: 4294967296 (17179869184 Kb)
Processor Commit: 1348 ( 5392 Kb)
Session Commit: 17782 ( 71128 Kb)
Shared Commit: 658461 ( 2633844 Kb)
Special Pool: 0 ( 0 Kb)
Kernel Stacks: 26919 ( 107676 Kb)
Pages For MDLs: 395401 ( 1581604 Kb)
Pages For AWE: 0 ( 0 Kb)
NonPagedPool Commit: 97838 ( 391352 Kb)
PagedPool Commit: 270911 ( 1083644 Kb)
Driver Commit: 19721 ( 78884 Kb)
Boot Commit: 2732 ( 10928 Kb)
PFN Array Commit: 196913 ( 787652 Kb)
System PageTables: 3267 ( 13068 Kb)
ProcessLockedFilePages: 306 ( 1224 Kb)
Pagefile Hash Pages: 0 ( 0 Kb)
Sum System Commit: 1691599 ( 6766396 Kb)
Total Private: 4330147 ( 17320588 Kb)
Misc/Transient Commit: 9281 ( 37124 Kb)
Committed pages: 6031027 ( 24124108 Kb)
Commit limit: 19247014 ( 76988056 Kb)
Pid ImageName Commit SharedCommit Debt
598c vmmem 3677256 Kb 0 Kb 0 Kb
4d4 RemoteDesktopManager.exe 1367276 Kb 473008 Kb 0 Kb
6300 vmmem 1050684 Kb 0 Kb 0 Kb
207c vmmem 1050684 Kb 0 Kb 0 Kb
1d18 vmmem 1050684 Kb 0 Kb 0 Kb
276c powershell.exe 713052 Kb 4660 Kb 0 Kb
483c chrome.exe 635112 Kb 130192 Kb 0 Kb
3ad4 chrome.exe 525988 Kb 20672 Kb 0 Kb
From Russinovich, Mark; Solomon, David; Ionescu, Alex. Windows Internals, Part 2 (6th Edition) (Developer Reference)
you can enable system PTE tracking by creating a new DWORD value in
the HKLM\ SYSTEM\ CurrentControlSet\ Control\ Session Manager\ Memory
Management key called TrackPtes and setting its value to 1. You can
then use !sysptes 4 to show a list of allocators
So you could try to
set the registry key and reboot
use livekdon the running system and execute !sysptes 4 to get al list of allocators
or execute !sysptes 4 on a dump you collect

Can't run sbt on ec2 due to lack of memory (not being able to allocate new memory)

For some reason, I can't run sbt at amazon ec2 free tier
ubuntu#ip-xx-xx-xx-xx:~$ sbt seed
Loading /usr/share/sbt/bin/sbt-launch-lib.bash
Java HotSpot(TM) 64-Bit Server VM warning: INFO: os::commit_memory(0x00000000c5550000, 715849728, 0) failed; error='Cannot allocate memory' (errno=12)
#
# There is insufficient memory for the Java Runtime Environment to continue.
# Native memory allocation (malloc) failed to allocate 715849728 bytes for committing reserved memory.
# An error report file with more information is saved as:
# /home/ubuntu/hs_err_pid10392.log
Here is top:
ubuntu#ip-xx-xx-xx-xx:~$ top > 1.txt
ubuntu#ip-xx-xx-xx-xx:~$ cat 1.txt
top - 03:33:21 up 2 days, 2:41, 1 user, load average: 0.00, 0.01, 0.05
Tasks: 66 total, 1 running, 65 sleeping, 0 stopped, 0 zombie
%Cpu(s): 0.0 us, 0.0 sy, 0.0 ni,100.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
KiB Mem: 603108 total, 438676 used, 164432 free, 34976 buffers
KiB Swap: 0 total, 0 used, 0 free, 337716 cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
1 root 20 0 26828 2172 924 S 0.0 0.4 0:01.05 init
2 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kthreadd
3 root 20 0 0 0 0 S 0.0 0.0 0:00.01 ksoftirqd/0
4 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kworker/0:0
5 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 kworker/0:0H
6 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kworker/u2:0
//....................
Note sbt seed does work at my local machine ("seed" is my custom task)
ubuntu#ip-xx-xx-xx-xx:~$ free
total used free shared buffers cached
Mem: 603108 438668 164440 0 35012 337720
-/+ buffers/cache: 65936 537172
Swap: 0 0 0
Micro instances come with 613 MB of physical memory total. You can enable swap if you need more, but keep in mind, that will be much slower.