I recently started using a tool called pg_top that shows statistics for Postgres, however since I am not verify versed with the internals of Postgres I needed a bit of clarification on the output.
last pid: 6152; load avg: 19.1, 18.6, 20.4; up 119+20:31:38 13:09:41
41 processes: 5 running, 36 sleeping
CPU states: 52.1% user, 0.0% nice, 0.8% system, 47.1% idle, 0.0% iowait
Memory: 47G used, 16G free, 2524M buffers, 20G cached
DB activity: 151 tps, 0 rollbs/s, 253403 buffer r/s, 86 hit%, 1550639 row r/s,
21 row w/s
DB I/O: 0 reads/s, 0 KB/s, 35 writes/s, 2538 KB/s
DB disk: 233.6 GB total, 195.1 GB free (16% used)
Swap:
My question is under the DB Activity, the 1.5 million row r/s, is that a lot? If so what can be done to improve it? I am running puppetdb 2.3.8, with 6.8 million resources, 2500 nodes, and Postgres 9.1. All of this runs on a single 24 core box with 64GB of memory.
i'm new to the iPhone application development, when i'm running my application in device application is crashing and giving following log in organizer
here is the log report :
Incident Identifier: 23B91310-BCEC-497B-821A-8CF8E709ACF7
CrashReporter Key: 3202fd13cd0c5e4ab0706615fabf36ebc9396206
Hardware Model: iPod4,1
OS Version: iPhone OS 4.3.5 (8L1)
Kernel Version: Darwin Kernel Version 11.0.0: Sat Jul 9 00:59:43 PDT 2011; root:xnu- 1735.47~1/RELEASE_ARM_S5L8930X
Date: 2012-08-22 10:45:15 +0530
Time since snapshot: 54 ms
Free pages: 588
Wired pages: 16800
Purgeable pages: 38
Largest process: BJS
Processes
Name UUID Count resident pages
BJS <312d3dae46e433f9982c26738e40060b> 36928 (jettisoned) (active)
debugserver <919dbac91c0e3133a517d6c7a99e667e> 184
SCHelper <f8cf7ee034ac3991a79d8c78e435c1e7> 131
Preferences <cbdecd3d02e031e48e9675235a51306c> 1045 (jettisoned)
springboardservi <5ab19f16a3973514b6d7d62201d6abde> 309
syslog_relay <344c7c41bec5360aae33f4fd412ea95f> 94
notification_pro <698dca6c4cba390a8017315bd25f18f8> 112
syslog_relay <344c7c41bec5360aae33f4fd412ea95f> 97
notification_pro <698dca6c4cba390a8017315bd25f18f8> 108
ptpd <6072e173aed83310b9b7589a70a24b0b> 580
talkmeim <eb62d6230d5f304f9819c6aebc2d2298> 1141 (jettisoned)
lsd <3fafa485b73836acb973d50feabd963a> 248
notifyd <9966082842de313a8e05a001c783faf4> 126
BTServer <01550e9527353eecae41ebee0f889603> 308
CommCenter <7d9446365b4836968ae361626ef8f939> 271
misd <8f94228bddf8342994baf5ca9af1154d> 154
SpringBoard <5c55c6fba0843b0e924e116413b8c9d4> 4143 (active)
accessoryd <d30e340e36df356bbde3347a6ed1ef87> 148
apsd <47ffc9ce9f84371588bd3f937aaa20bb> 285
configd <a6d457fca42732d9ba809d03a2b3e3ae> 401
fairplayd.N81 <144f0ff89c123fa5a1cfa40da72fb024> 165
imagent <9e0b26bad4a538a5b0e5e5ee7eeeb7be> 234
locationd <9088e845dcbe37d890c8758655bf34c6> 685
mDNSResponder <caf94711b8093dc5bc5736306f8ae818> 198
mediaremoted <21af791e80823c9f90f0be2b77a3d885> 199
mediaserverd <c731263114c33a07aef7bccdcf667271> 643
lockdownd <1c7f2b41744c35dc92f679e90a73e240> 268
syslogd <d81669e7bdb93f9b9012020beac826f4> 100
usbethernetshari <25130d2f9a0334e3ae28780250343144> 105
launchd <e2d41e07a0743a089eadbae765709c82> 84
**End**
can any one pls help how to track the log report.
This isn't an application crash, so much as application termination: iOS has a mechanism called Jetsam (or, officially, memory status) which "jettisons" (i.e. throws out) the largest process (in terms of memory) when physical memory is low. This is a necessary measure because iOS has no swap. In your log, which is generated by jetsam, you can see a list of all the processes which were active, and BJS - which has been thrown out. The format is :
Incident Identifier: Automatically generated GUID for Apple
CrashReporter Key: Hash, probably of GUID
Hardware Model: i-Device stepping (4,1 is 4th gen iPod in your case)
OS Version: Version of iOS
Kernel Version: Kernel version, as per uname -a
Date: ... figure this one out
Time since snapshot: How much time has passed - the snapshot may be "dated"
Free pages: How many free pages of RAM remain. Page = 4k
Wired pages: How many resident pages
Purgeable pages: How many pages MAY be freed, if all else fails
Largest process: The culprit
Processes
Name UUID Count resident pages
BJS <312d3dae46e433f9982c26738e40060b> 36928 (jettisoned) (active)
Table, as per column header, where (jettisoned) implies jetsam killed you. (active) implies you were the foreground app.
Hope this helps, (if it doesn't - comment or ask me directly)
MongoDB throwing following error in syslogs:-
LogFile::synchronousAppend failed with 16384 bytes unwritten out of 16384 bytes; b=0x7fed27ffe000 errno:28 No space left on device Tue Jun 26 00:13:42 Got signal: 6 (Aborted)
It is a 64 bit Cent OS machine. My Disk Usage is 65% only.
One that happens, mongodb crashes and needs to be started again.
What is the problem and solution?
Customers are reporting problems almost every day on about the same hours. This app is running on 2 nodes. It is Metastorm BPM platform and it's calling our code.
In some dumps I noticed very long running threads (~50 minutes) but not in all of them. Administrators are also telling me that just before users report problems memory usage goes up. Then everything slows down to the point they can't work and admins have to restart platforms on both nodes. My first thought was deadlocks (long running threads) but didn't manage to confirm that. !syncblk isn't returning anything. Then I looked at memory usage. I noticed a lot of dynamic assemblies so thought maybe assemblies leak. But it looks it's not that. I have received dump from day where everything was working fine and number of dynamic assemblies is similar. So maybe memory leak I thought. But also cannot confirm that. !dumpheap -stat shows memory usage grows but I haven't found anything interesting with !gcroot. But there is one thing I don't know what it is. Threadpool Completion Port. There's a lot of them. So maybe sth is waiting on sth? Here is data I can give You so far that will fit in this post. Could You suggest anything that could help diagnose this situation?
Users not reporting problems:
Node1 Node2
Size of dump: 638MB 646MB
DynamicAssemblies 259 265
GC Heaps: 37MB 35MB
Loader Heaps: 11MB 11MB
Node1:
Number of Timers: 12
CPU utilization 2%
Worker Thread: Total: 5 Running: 0 Idle: 5 MaxLimit: 2000 MinLimit: 200
Completion Port Thread:Total: 2 Free: 2 MaxFree: 16 CurrentLimit: 4 MaxLimit: 1000 MinLimit: 8
!dumpheap -stat (biggest)
0x793041d0 32,664 2,563,292 System.Object[]
0x79332b9c 23,072 3,485,624 System.Int32[]
0x79330a00 46,823 3,530,664 System.String
0x79333470 22,549 4,049,536 System.Byte[]
Node2:
Number of Timers: 12
CPU utilization 0%
Worker Thread: Total: 7 Running: 0 Idle: 7 MaxLimit: 2000 MinLimit: 200
Completion Port Thread:Total: 3 Free: 1 MaxFree: 16 CurrentLimit: 5 MaxLimit: 1000 MinLimit: 8
!dumpheap -stat
0x793041d0 30,678 2,537,272 System.Object[]
0x79332b9c 21,589 3,298,488 System.Int32[]
0x79333470 21,825 3,680,000 System.Byte[]
0x79330a00 46,938 5,446,576 System.String
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Users start to report problems:
Node1 Node2
Size of dump: 662MB 655MB
DynamicAssemblies 236 235
GC Heaps: 159MB 113MB
Loader Heaps: 10MB 10MB
Node1:
Work Request in Queue: 0
Number of Timers: 14
CPU utilization 20%
Worker Thread: Total: 7 Running: 0 Idle: 7 MaxLimit: 2000 MinLimit: 200
Completion Port Thread:Total: 48 Free: 1 MaxFree: 16 CurrentLimit: 49 MaxLimit: 1000 MinLimit: 8
!dumpheap -stat
0x7932a208 88,974 3,914,856 System.Threading.ReaderWriterLock
0x79333054 71,397 3,998,232 System.Collections.Hashtable
0x24f70350 319,053 5,104,848 Our.Class
0x79332b9c 53,190 6,821,588 System.Int32[]
0x79333470 52,693 6,883,120 System.Byte[]
0x79333150 72,900 11,081,328 System.Collections.Hashtable+bucket[]
0x793041d0 247,011 26,229,980 System.Object[]
0x79330a00 644,807 34,144,396 System.String
Node2:
Work Request in Queue: 1
Number of Timers: 17
CPU utilization 17%
Worker Thread: Total: 6 Running: 0 Idle: 6 MaxLimit: 2000 MinLimit: 200
Completion Port Thread:Total: 48 Free: 2 MaxFree: 16 CurrentLimit: 49 MaxLimit: 1000 MinLimit: 8
!dumpheap -stat
0x7932a208 76,425 3,362,700 System.Threading.ReaderWriterLock
0x79332b9c 42,417 5,695,492 System.Int32[]
0x79333150 41,172 6,451,368 System.Collections.Hashtable+bucket[]
0x79333470 44,052 6,792,004 System.Byte[]
0x793041d0 175,973 18,573,780 System.Object[]
0x79330a00 397,361 21,489,204 System.String
Edit:
I downloaded debugdiag and let it analyze my dumps. Here is part of output:
The following threads in process_name name_of_dump.dmp are making a COM call to thread 193 within the same process which in turn is waiting on data to be returned from another server via WinSock.
The call to WinSock originated from 0x0107b03b and is destined for port xxxx at IP address xxx.xxx.xxx.xxx
( 18 76 172 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 210 211 212 213 214 215 216 217 218 224 225 226 227 228 229 231 232 233 236 239 )
14,79% of threads blocked
And the recommendation is:
Several threads making calls to the same STA thread can cause a performance bottleneck due to serialization. Server side COM servers are recommended to be thread aware and follow MTA guidelines when multiple threads are sharing the same object instance.
I checked using windbg what thread 193 does. It is calling our code. Our code is calling some Metastorm engine code and it hangs on some remoting call. But !runaway shows it is hanging for 8 seconds. So not that long. So I checked what are those waiting threads. All except thread 18 are:
System.Threading._IOCompletionCallback.PerformIOCompletionCallback(UInt32, UInt32, System.Threading.NativeOverlapped*) I could understand one, but why so many of them. Is it specific to business process modeling engine we're using or is it something typical? I guess it's taking threads that could be used by other clients and that's why the slowdown reported by users. Are those threads Completion Port Threads I asked about before? Can I do anything more to diagnose or did I found our code to be the cause?
From the looks of the output most of the memory is not on the .net heaps (only 35 MB out of ~650) so if you are looking at the .net heaps I think you are looking in the wrong place. The memory is probably either in assemblies or in native memory if you are using some native component for file transfers or similar. You would want to use Debug Diag to monitor that.
It is hard to say if you are leaking dynamic assemblies without looking at the pattern of growth so I would suggest for that that you look at perfmon and #current assemblies to see if it keeps growing over time, if it does then you would have to investigate that further by looking at what the dynamic assemblies are with !dda
I encountered a crash in my game, here's the crash log (it's a stress test):
PID RPRVT RSHRD RSIZE Command
1 340K 224K 436K launchd
14 124K 160K 216K update
15 568K 164K 620K syslogd
16 792K 612K 1.16M lockdownd
17 2.22M 664K 3.04M mediaserverd
18 296K 160K 440K mDNSResponder
20 540K 568K 1.25M iapd
21 248K 236K 456K fairplayd
22 640K 168K 1012K configd
23 6.42M 6.73M 9.14M SpringBoard
26 660K 212K 1.01M CommCenter
27 308K 164K 620K BTServer
28 19.1M 692K 19.9M TQServer
29 232K 188K 284K notifyd
1830 368K 596K 672K ptpd
1833 140K 164K 280K afcd
1835 148K 164K 288K afcd
1837 140K 160K 260K notification_pro
1848 3.83M 4.89M 4.76M MobileMusicPlaye
1855 19.7M 7.65M 23.6M MyGame ****
1856 616K 5.25M 2.27M MobilePhone
1857 292K 240K 1.25M ReportCrash
The RSIZE of MyGame is only 23.6MB, but the "Memory status: 11" indicate that the program recieves a shutdown command from OS due to memory problem.
So if the memory problem is not caused by my program, is there any way to ignore the shutdown command post from OS?
And from the iPhone develop guide, the OS will terminate a bg process which enconters a memory problem. But the SpringBoard is not terminated. So I think there should be a method to turn the memory management off in program.
You can't ignore the memory warning. The best you can do is try to free up some memory and hope your app doesn't get killed.
See this discussion for some insight.