We are running a postgres cluster with 32 CPU and 84 GB RAM in production with db size close to ~800GB.
It has replica setup as well with streaming wall replication (another host) kept as hot-standby.
Linux 2.6.32-642.6.2.el6.x86_64 (CentOS 6.4, upgrade pending)
Postgres Version - 11.11
We see some weird behaviour since last month with our instance having memory usage going every 80 hours even without having any cronjobs running at the specified moment (usually cron setup works specified-time/hourly/daily/weekly/monthly).
At the same time we see huge(~70gb) of temp files , e.g pgsql_tmp90128.0.sharedfileset/i231326of1048576.p2.0) being created and not getting deleted even after the postgres started working fine after 10 minutes.
My related settings for postgres memory utilization are following
work_mem 8 MB
temp_buffers 8 MB
shared_buffers 22 GB
max_connections 2500
During the window we see system cpu shoots up and came down again once memory refreshes.
$ sar -f /var/log/sa/sa17 -s 08:36:00 -e 08:47:00
>
08:36:21 AM CPU %user %nice %system %iowait %steal %idle
08:37:21 AM all 18.66 0.01 26.05 9.54 0.00 45.74
08:38:21 AM all 5.21 0.01 81.71 2.11 0.00 10.96
08:39:21 AM all 2.71 0.01 96.90 0.04 0.00 0.34
08:40:08 AM all 1.55 0.00 98.44 0.00 0.00 0.00
08:41:08 AM all 9.96 0.02 38.49 2.20 0.00 49.34
08:42:08 AM all 0.12 0.00 1.50 1.63 0.00 96.74
08:43:08 AM all 0.10 0.01 1.32 1.79 0.00 96.79
08:44:08 AM all 0.20 0.01 2.25 0.76 0.00 96.79
08:45:08 AM all 1.74 0.01 1.83 5.89 0.00 90.53
08:46:08 AM all 10.09 0.01 7.37 18.90 0.00 63.63
Average: all 5.11 0.01 34.18 4.36 0.00 56.34
RAM usage pattern
$ sar -r -f /var/log/sa/sa17 -s 08:36:00 -e 08:47:00
>
08:36:21 AM kbmemfree kbmemused %memused kbbuffers kbcached kbcommit %commit
08:37:21 AM 430888 88277480 99.51 12176 23869496 79753764 87.83
08:38:21 AM 420612 88287756 99.53 7212 23425316 80174100 88.29
08:39:21 AM 424276 88284092 99.52 3552 23246368 81653952 89.92
08:40:08 AM 417172 88291196 99.53 4316 22819904 82023344 90.33
08:41:08 AM 84692800 4015568 4.53 978760 1124852 25077588 27.62
08:42:08 AM 78749436 9958932 11.23 2098912 1139416 25067128 27.61
08:43:08 AM 73598564 15109804 17.03 3065228 1140756 25067516 27.61
08:44:08 AM 70213124 18495244 20.85 3175984 1141168 25067016 27.61
08:45:08 AM 56661972 32046396 36.13 3179640 13233168 27063188 29.80
08:46:08 AM 30585252 58123116 65.52 3207036 30364932 32130264 35.38
Average: 39619410 49088958 55.34 1573282 14150538 48307786 53.20
I have the following questions in this respect:
What can be the probable cause of this behaviour of memory usage going dropped at regular interval ?
What these temp_files are and is it safer to delete manually ?
Is there a bug in Postgres, why it's not getting automatically
cleaned, related-thread
I see several run of CHECKPOINT and even then the temp files are not getting cleared.
I'm trying to rescue a 1TB disk which has read errors. Because I didn't have a free 1TB drive, I created a raid 0 of two 500GB drives.
I used the command line from Wikipedia for the first run:
sudo ddrescue -f -n /dev/sdk /dev/md/md_test /home/user/rescue.map
ddrescue already completed this run after approximately 20 hours and more than 7000 read errors.
Now I'm trying to do a second run
sudo ddrescue -d -f -v -r3 /dev/sdk /dev/md/md_test /home/user/rescue.map
and read the non tried blocks but ddrescue gives me this:
GNU ddrescue 1.23
About to copy 1000 GBytes from '/dev/sdk' to '/dev/md/md_test'
Starting positions: infile = 0 B, outfile = 0 B
Copy block size: 128 sectors Initial skip size: 19584 sectors
Sector size: 512 Bytes
Press Ctrl-C to interrupt
Initial status (read from mapfile)
rescued: 635060 MB, tried: 0 B, bad-sector: 0 B, bad areas: 0
Current status
ipos: 1000 GB, non-trimmed: 0 B, current rate: 0 B/s
opos: 1000 GB, non-scraped: 0 B, average rate: 0 B/s
non-tried: 365109 MB, bad-sector: 0 B, error rate: 0 B/s
rescued: 635060 MB, bad areas: 0, run time: 0s
pct rescued: 63.49%, read errors: 0, remaining time: n/a
time since last successful read: n/a
Copying non-tried blocks... Pass 1 (forwards)
ddrescue: Write error: Invalid argument
I can't figure out what this write errors means, already searched the manual for answers.
Any help is appreciated! Thx!
After a while I found the cause for the write error, the capacity of the corrupt drive is 931,5G but the total capacity of the raid 0 was just 931,3G.
Realized it, while I took a closer look to the output of lsblk command.
So I rebuild the raid 0 array with 3 500G drives and ddrescue now works as expected.
I'm trying to get a Techwell TW6869 driver to work. This PCIe-chip is able to capture analog video signals. Therefore I'm using a driver which can be found here: GitHub
The chip is connected to a Freescale imx.6 processor which is running Angström distribution. The driver already worked on the target but I didn't use it for some time and somehow it doesn't do it's job anymore.
So what I did, was implementing kernel messages in the beginning of each function so I know what does happen exactly. Finally I found out, that no PCIe Interrupt is generated anymore. Though the interrupt is registered which I found out here:
root#freescaleimx6:~# cat /proc/interrupts | grep tw6869
155: 0 0 0 0 GIC tw6869
Running something on the videodevice unfortunately does not generate an interrupt.
root#freescaleimx6:~# gst-launch-1.0 v4l2src device=/dev/video2 ! imxipuvideosink
Setting pipeline to PAUSED ...
Pipeline is live and does not need PREROLL ...
Setting pipeline to PLAYING ...
New clock: GstSystemClock
^Chandling interrupt.
Interrupt: Stopping pipeline ...
Execution ended after 0:00:08.992961001
Setting pipeline to PAUSED ...
Setting pipeline to READY ...
Setting pipeline to NULL ...
Freeing pipeline ...
root#freescaleimx6:~#
This could also possibly help (does it?)
root#freescaleimx6:~# cat /proc/bus/pci/devices
0000 16c3abcd 180 1000000 0 0 0 0 0 1200000 100000 0 0 0 0 0 10000 pcieport
0100 17976869 9b 1100008 0 0 0 0 0 0 1000 0 0 0 0 0 0 tw6869
Does anyone have an idea?
I'm trying to do a !heap -s in Windbg to get heap information. When I attempt it I get the following output:
Heap Flags Reserv Commit Virt Free List UCR Virt Lock Fast
(k) (k) (k) (k) length blocks cont. heap
-----------------------------------------------------------------------------
00000000005d0000 08000002 512 28 512 10 3 1 0 0
Error: Heap 0000000000000000 has an invalid signature eeffeeff
Front-end heap type info is not available
Front-end heap type info is not available
Virtual block: 0000000000000000 - 0000000000000000 (size 0000000000000000)
HEAP 0000000000000000 (Seg 0000000000000000) At 0000000000000000 Error: Unable to read virtual block
0000000000000000 00000000 0 0 0 0 0 0 1 0
-----------------------------------------------------------------------------
I can't find any reference as to what the unusual error/not available lines mean.
Can someone please give me a summary as to why I'm not getting an expected list of heaps?
The only thing I execute prior to !heap -s is !wow64exts.sw because the process dumps are from a 32 bit process but created by a 64 bit Task Manager.
After testing with the 32 and 64 bit Task Managers it appears that process dumps of 32 bit processes created by the 64 bit Task Manager can only be debugged successfully in some areas using !wow64exts.sw in Windbg to use 32 bit debugging.
That extension allows call stacks to be reviewed correctly, but !heap -s does not appear to work correctly under it. Instead you end up with the errors in the question.
For example, the output from a process dump of the 32 bit process using the 32 bit Task Manager:
0:000> !heap -s
NtGlobalFlag enables following debugging aids for new heaps:
stack back traces
LFH Key : 0x06b058a2
Termination on corruption : DISABLED
Heap Flags Reserv Commit Virt Free List UCR Virt Lock Fast
(k) (k) (k) (k) length blocks cont. heap
-----------------------------------------------------------------------------
031b0000 08000002 1024 236 1024 2 13 1 0 0 LFH
001d0000 08001002 1088 188 1088 18 9 2 0 0 LFH
01e30000 08001002 1088 160 1088 4 3 2 0 0 LFH
03930000 08001002 256 4 256 2 1 1 0 0
038a0000 08001002 64 16 64 13 1 1 0 0
-----------------------------------------------------------------------------
The output from a process dump of the 32 bit process using the 64 bit Task Manager without !wow64exts.sw:
0:000> !heap -s
NtGlobalFlag enables following debugging aids for new heaps:
stack back traces
LFH Key : 0x000000b406b058a2
Termination on corruption : ENABLED
Heap Flags Reserv Commit Virt Free List UCR Virt Lock Fast
(k) (k) (k) (k) length blocks cont. heap
-------------------------------------------------------------------------------------
0000000001f70000 08000002 512 28 512 10 3 1 0 0
0000000000020000 08008000 64 4 64 1 1 1 0 0
-------------------------------------------------------------------------------------
The output from a process dump of the 32 bit process using the 64 bit Task Manager with !wow64exts.sw:
0:000> !wow64exts.sw
Switched to 32bit mode
0:000:x86> !heap -s
NtGlobalFlag enables following debugging aids for new heaps:
stack back traces
LFH Key : 0x000000b406b058a2
Termination on corruption : ENABLED
Heap Flags Reserv Commit Virt Free List UCR Virt Lock Fast
(k) (k) (k) (k) length blocks cont. heap
-----------------------------------------------------------------------------
0000000001f70000 08000002 512 28 512 10 3 1 0 0
Error: Heap 0000000000000000 has an invalid signature eeffeeff
Front-end heap type info is not available
Front-end heap type info is not available
Virtual block: 0000000000000000 - 0000000000000000 (size 0000000000000000)
HEAP 0000000000000000 (Seg 0000000000000000) At 0000000000000000 Error: Unable to read virtual block
0000000000000000 00000000 0 0 0 0 0 0 1 0
-----------------------------------------------------------------------------
Those were all taken from the same process.
Closed. This question does not meet Stack Overflow guidelines. It is not currently accepting answers.
Closed 7 years ago.
We don’t allow questions seeking recommendations for books, tools, software libraries, and more. You can edit the question so it can be answered with facts and citations.
This question does not appear to be about a specific programming problem, a software algorithm, or software tools primarily used by programmers. If you believe the question would be on-topic on another Stack Exchange site, you can leave a comment to explain where the question may be able to be answered.
Improve this question
On Linux, the "top" command shows a detailed but high level overview of your memory usage, showing:
Total Memory, Used Memory, Free Memory, Buffer Usage, Cache Usage, Swap size and Swap Usage.
My question is, what commands are available to show these memory usage figures in a clear and simple way? Bonus points if they're present in the "Core" install of Solaris. 'sar' doesn't count :)
Here are the basics. I'm not sure that any of these count as "clear and simple" though.
ps(1)
For process-level view:
$ ps -opid,vsz,rss,osz,args
PID VSZ RSS SZ COMMAND
1831 1776 1008 222 ps -opid,vsz,rss,osz,args
1782 3464 2504 433 -bash
$
vsz/VSZ: total virtual process size (kb)
rss/RSS: resident set size (kb, may be inaccurate(!), see man)
osz/SZ: total size in memory (pages)
To compute byte size from pages:
$ sz_pages=$(ps -o osz -p $pid | grep -v SZ )
$ sz_bytes=$(( $sz_pages * $(pagesize) ))
$ sz_mbytes=$(( $sz_bytes / ( 1024 * 1024 ) ))
$ echo "$pid OSZ=$sz_mbytes MB"
vmstat(1M)
$ vmstat 5 5
kthr memory page disk faults cpu
r b w swap free re mf pi po fr de sr rm s3 -- -- in sy cs us sy id
0 0 0 535832 219880 1 2 0 0 0 0 0 -0 0 0 0 402 19 97 0 1 99
0 0 0 514376 203648 1 4 0 0 0 0 0 0 0 0 0 402 19 96 0 1 99
^C
prstat(1M)
PID USERNAME SIZE RSS STATE PRI NICE TIME CPU PROCESS/NLWP
1852 martin 4840K 3600K cpu0 59 0 0:00:00 0.3% prstat/1
1780 martin 9384K 2920K sleep 59 0 0:00:00 0.0% sshd/1
...
swap(1)
"Long listing" and "summary" modes:
$ swap -l
swapfile dev swaplo blocks free
/dev/zvol/dsk/rpool/swap 256,1 16 1048560 1048560
$ swap -s
total: 42352k bytes allocated + 20192k reserved = 62544k used, 607672k available
$
top(1)
An older version (3.51) is available on the Solaris companion CD from Sun, with the disclaimer that this is "Community (not Sun) supported".
More recent binary packages available from sunfreeware.com or blastwave.org.
load averages: 0.02, 0.00, 0.00; up 2+12:31:38 08:53:58
31 processes: 30 sleeping, 1 on cpu
CPU states: 98.0% idle, 0.0% user, 2.0% kernel, 0.0% iowait, 0.0% swap
Memory: 1024M phys mem, 197M free mem, 512M total swap, 512M free swap
PID USERNAME LWP PRI NICE SIZE RES STATE TIME CPU COMMAND
1898 martin 1 54 0 3336K 1808K cpu 0:00 0.96% top
7 root 11 59 0 10M 7912K sleep 0:09 0.02% svc.startd
sar(1M)
And just what's wrong with sar? :)
# echo ::memstat | mdb -k
Page Summary Pages MB %Tot
------------ ---------------- ---------------- ----
Kernel 7308 57 23%
Anon 9055 70 29%
Exec and libs 1968 15 6%
Page cache 2224 17 7%
Free (cachelist) 6470 50 20%
Free (freelist) 4641 36 15%
Total 31666 247
Physical 31256 244
"top" is usually available on Solaris.
If not then revert to "vmstat" which is available on most UNIX system.
It should look something like this (from an AIX box)
vmstat
System configuration: lcpu=4 mem=12288MB ent=2.00
kthr memory page faults cpu
----- ----------- ------------------------ ------------ -----------------------
r b avm fre re pi po fr sr cy in sy cs us sy id wa pc ec
2 1 1614644 585722 0 0 1 22 104 0 808 29047 2767 12 8 77 3 0.45 22.3
the colums "avm" and "fre" tell you the total memory and free memery.
a "man vmstat" should get you the gory details.
Top can be compiled from sources or downloaded from sunfreeware.com. As previously posted, vmstat is available (I believe it's in the core install?).
The command free is nice. Takes a short while to understand the "+/- buffers/cache", but the idea is that cache and buffers doesn't really count when evaluating "free", as it can be dumped right away. Therefore, to see how much free (and used) memory you have, you need to remove the cache/buffer usage - which is conveniently done for you.