JVM crash :- Problematic frame: # V [libjvm.so+0x546720] - jboss
Production server has gone down and the preliminary analysis find that it has encountered a JVM crash. The jboss server log and GC log seems to be ok. And the jvm error log mention the “Problematic frame: # V [libjvm.so+0x546720]”.
Configuration details:-
The server is 64 bit machine having 32 bit RHEL-5.2.
JAVA : jdk1.6.0_11
In memory (RAM) : 2GB
Heap size of Java virtual machine is -Xmx1024m.
`An unexpected error has been detected by Java Runtime Environment:
SIGSEGV (0xb) at pc=0x06546720, pid=5127, tid=1822063504
Java VM: Java HotSpot(TM) Server VM (11.0-b16 mixed mode linux-x86)
Problematic frame:
V [libjvm.so+0x546720]
If you would like to submit a bug report, please visit:
http://java.sun.com/webapps/bugreport/crash.jsp
--------------- T H R E A D ---------------
Current thread (0x09a05400): GCTaskThread [stack:
0x6c927000,0x6c9a8000] [id=5130]
siginfo:si_signo=SIGSEGV: si_errno=0, si_code=1 (SEGV_MAPERR),
si_addr=0x00000008
Registers: EAX=0x00000008, EBX=0xa60239c0, ECX=0x0000000c,
EDX=0x0000000c ESP=0x6c9a6ea0, EBP=0x6c9a6f18, ESI=0x00003ece,
EDI=0xe2c30913 EIP=0x06546720, CR2=0x00000008, EFLAGS=0x00010202
Top of Stack: (sp=0x6c9a6ea0) 0x6c9a6ea0: 004ac889 00000000 004b7ff4
6cb57c56 0x6c9a6eb0: 00000000 09a06300 00000400 1c00dc04 0x6c9a6ec0:
00a039c8 00000008 00000000 01505e98 0x6c9a6ed0: 09a06328 09a06328
ffffffff ffffffff 0x6c9a6ee0: b3d06048 b3d0604c 6c9a6f18 0000000d
0x6c9a6ef0: b3d28450 00001000 00000001 00000001 0x6c9a6f00:
a6026fc1 b3d05db8 b3d06048 b3d2861c 0x6c9a6f10: 00003ece 09a5fc40
6c9a6f58 065462c3
Instructions: (pc=0x06546720) 0x06546710: e0 01 84 c0 0f 84 16 05 00
00 8b 45 ac 83 e0 fd 0x06546720: 8b 00 c1 e8 03 83 e0 0f 3b 05 cc 28
6b 06 0f 8d
Stack: [0x6c927000,0x6c9a8000], sp=0x6c9a6ea0, free space=511k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code,
C=native code) V [libjvm.so+0x546720] V [libjvm.so+0x5462c3] V
[libjvm.so+0x255c38] V [libjvm.so+0x5493ba] V [libjvm.so+0x32826b] V
[libjvm.so+0x506709] C [libpthread.so.0+0x545b]
--------------- P R O C E S S ---------------
Java Threads: ( => current thread ) 0x0a745000 JavaThread
"Keep-Alive-Timer" daemon [_thread_blocked, id=9822,
stack(0x68fe4000,0x69005000)] 0x6ae59000 JavaThread "Java2D
Disposer" daemon [_thread_blocked, id=5496,
stack(0x6c2a7000,0x6c2c8000)] 0x6b245800 JavaThread "TP-Monitor"
daemon [_thread_blocked, id=5493, stack(0x69ca6000,0x69cc7000)]
0x6b244000 JavaThread "TP-Processor4" daemon [_thread_in_native,
id=5492, stack(0x69cc7000,0x69ce8000)] 0x6b276c00 JavaThread
"TP-Processor3" daemon [_thread_blocked, id=5491,
stack(0x69ce8000,0x69d09000)] 0x6b276800 JavaThread "TP-Processor2"
daemon [_thread_blocked, id=5490, stack(0x69d09000,0x69d2a000)]
0x6b276000 JavaThread "TP-Processor1" daemon [_thread_blocked,
id=5489, stack(0x69d2a000,0x69d4b000)] 0x0ad39800 JavaThread
"http-0.0.0.0-8080-Monitor" [_thread_blocked, id=5488,
stack(0x69d4b000,0x69d6c000)] 0x0ad38400 JavaThread
"http-0.0.0.0-8080-Processor25" daemon [_thread_blocked, id=5487,
stack(0x69d6c000,0x69d8d000)] 0x0bd06c00 JavaThread
"http-0.0.0.0-8080-Processor24" daemon [_thread_blocked, id=5486,
stack(0x69d8d000,0x69dae000)] 0x0bd05400 JavaThread
"http-0.0.0.0-8080-Processor23" daemon [_thread_blocked, id=5485,
stack(0x69dae000,0x69dcf000)] 0x0bd03c00 JavaThread
"http-0.0.0.0-8080-Processor22" daemon [_thread_blocked, id=5484,
stack(0x69dcf000,0x69df0000)] 0x0bd02400 JavaThread
"http-0.0.0.0-8080-Processor21" daemon [_thread_blocked, id=5483,
stack(0x69df0000,0x69e11000)] 0x0bd00c00 JavaThread
"http-0.0.0.0-8080-Processor20" daemon [_thread_in_native, id=5482,
stack(0x69e11000,0x69e32000)] 0x0bcff400 JavaThread
"http-0.0.0.0-8080-Processor19" daemon [_thread_in_native, id=5481,
stack(0x69e32000,0x69e53000)] 0x0bcfa800 JavaThread
"http-0.0.0.0-8080-Processor18" daemon [_thread_blocked, id=5480,
stack(0x69e53000,0x69e74000)] 0x0bcf9000 JavaThread
"http-0.0.0.0-8080-Processor17" daemon [_thread_blocked, id=5479,
stack(0x69e74000,0x69e95000)] 0x0aaa8400 JavaThread
"http-0.0.0.0-8080-Processor16" daemon [_thread_blocked, id=5478,
stack(0x69e95000,0x69eb6000)] 0x0aaa6c00 JavaThread
"http-0.0.0.0-8080-Processor15" daemon [_thread_blocked, id=5477,
stack(0x69eb6000,0x69ed7000)] 0x0aaa2400 JavaThread
"http-0.0.0.0-8080-Processor14" daemon [_thread_blocked, id=5476,
stack(0x69ed7000,0x69ef8000)] 0x0a371400 JavaThread
"http-0.0.0.0-8080-Processor13" daemon [_thread_blocked, id=5475,
stack(0x69ef8000,0x69f19000)] 0x0a370000 JavaThread
"http-0.0.0.0-8080-Processor12" daemon [_thread_blocked, id=5474,
stack(0x69f19000,0x69f3a000)] 0x0bd26c00 JavaThread
"http-0.0.0.0-8080-Processor11" daemon [_thread_blocked, id=5473,
stack(0x69f3a000,0x69f5b000)] 0x0a93bc00 JavaThread
"http-0.0.0.0-8080-Processor10" daemon [_thread_blocked, id=5472,
stack(0x69f5b000,0x69f7c000)] 0x0a93a400 JavaThread
"http-0.0.0.0-8080-Processor9" daemon [_thread_blocked, id=5471,
stack(0x69f7c000,0x69f9d000)] 0x0a9e1c00 JavaThread
"http-0.0.0.0-8080-Processor8" daemon [_thread_blocked, id=5470,
stack(0x69f9d000,0x69fbe000)] 0x0aa7bc00 JavaThread
"http-0.0.0.0-8080-Processor7" daemon [_thread_blocked, id=5469,
stack(0x69fbe000,0x69fdf000)] 0x0a213c00 JavaThread
"http-0.0.0.0-8080-Processor6" daemon [_thread_blocked, id=5468,
stack(0x69fdf000,0x6a000000)] 0x0aa0a400 JavaThread
"http-0.0.0.0-8080-Processor5" daemon [_thread_blocked, id=5467,
stack(0x6a519000,0x6a53a000)] 0x0aa09400 JavaThread
"http-0.0.0.0-8080-Processor4" daemon [_thread_blocked, id=5466,
stack(0x6a53a000,0x6a55b000)] 0x0a5d2000 JavaThread
"http-0.0.0.0-8080-Processor3" daemon [_thread_blocked, id=5465,
stack(0x6a55b000,0x6a57c000)] 0x0a10bc00 JavaThread
"http-0.0.0.0-8080-Processor2" daemon [_thread_blocked, id=5464,
stack(0x6a57c000,0x6a59d000)] 0x09d72800 JavaThread
"http-0.0.0.0-8080-Processor1" daemon [_thread_blocked, id=5463,
stack(0x6a59d000,0x6a5be000)] 0x0a006c00 JavaThread
"JBossLifeThread" [_thread_blocked, id=5462,
stack(0x6a5be000,0x6a5df000)] 0x6b652400 JavaThread "Timer-4" daemon
[_thread_blocked, id=5461, stack(0x6a5df000,0x6a600000)] 0x0a45f400
JavaThread "TestScheduler_QuartzSchedulerThread" [_thread_blocked,
id=5460, stack(0x6a832000,0x6a853000)] 0x0a542c00 JavaThread
"TestScheduler_Worker-11" [_thread_blocked, id=5459,
stack(0x6a853000,0x6a874000)] 0x09fe9000 JavaThread
"TestScheduler_Worker-10" [_thread_blocked, id=5458,
stack(0x6a874000,0x6a895000)] 0x0a3c8000 JavaThread
"TestScheduler_Worker-9" [_thread_blocked, id=5457,
stack(0x6a895000,0x6a8b6000)] 0x0b27d800 JavaThread
"TestScheduler_Worker-8" [_thread_blocked, id=5456,
stack(0x6a8b6000,0x6a8d7000)] 0x0bc9e800 JavaThread
"TestScheduler_Worker-7" [_thread_blocked, id=5455,
stack(0x6a8d7000,0x6a8f8000)] 0x0a595000 JavaThread
"TestScheduler_Worker-6" [_thread_blocked, id=5454,
stack(0x6a8f8000,0x6a919000)] 0x0abe7c00 JavaThread
"TestScheduler_Worker-5" [_thread_blocked, id=5453,
stack(0x6a919000,0x6a93a000)] 0x0a1be400 JavaThread
"TestScheduler_Worker-4" [_thread_blocked, id=5452,
stack(0x6a93a000,0x6a95b000)] 0x0a996c00 JavaThread
"TestScheduler_Worker-3" [_thread_blocked, id=5451,
stack(0x6a95b000,0x6a97c000)] 0x0abea400 JavaThread
"TestScheduler_Worker-2" [_thread_blocked, id=5450,
stack(0x6a97c000,0x6a99d000)] 0x0a5f7800 JavaThread
"TestScheduler_Worker-1" [_thread_blocked, id=5449,
stack(0x6a99d000,0x6a9be000)] 0x0a5f7400 JavaThread
"TestScheduler_Worker-0" [_thread_blocked, id=5448,
stack(0x6a9be000,0x6a9df000)] 0x0a2c1c00 JavaThread "Thread-12"
daemon [_thread_blocked, id=5447, stack(0x6a9df000,0x6aa00000)]
0x0a85cc00 JavaThread "Thread-11" daemon [_thread_in_native, id=5446,
stack(0x6ab19000,0x6ab3a000)] 0x0a7cb800 JavaThread
"ElementEventQueue.QProcessor-1" daemon [_thread_blocked, id=5445,
stack(0x6ab5b000,0x6ab7c000)] 0x0a504800 JavaThread "Thread-9"
[_thread_blocked, id=5442, stack(0x6ab3a000,0x6ab5b000)] 0x0ad5c400
JavaThread "Timer-3" daemon [_thread_blocked, id=5437,
stack(0x6ab7c000,0x6ab9d000)] 0x0b284000 JavaThread "Timer-2" daemon
[_thread_blocked, id=5436, stack(0x6ab9d000,0x6abbe000)] 0x0aef7c00
JavaThread "Thread-5" [_thread_blocked, id=5435,
stack(0x6abbe000,0x6abdf000)] 0x0a1dd400 JavaThread "Thread-4"
[_thread_blocked, id=5434, stack(0x6b06d000,0x6b08e000)] 0x0a7df400
JavaThread "Connection Consumer for dest
Subscription[subId=-2147483648connection=ConnectionToken:ID:2/b744868be8f2f16c9ebefdb987f81c71
destination=TOPIC.MastersSubscriberMDB messageSelector=null Local
Create] id=2" [_thread_blocked, id=5433, stack(0x6b08e000,0x6b0af000)]
0x0b26fc00 JavaThread "Connection Consumer for dest
Subscription[subId=-2147483648connection=ConnectionToken:ID:1/fcba95a2a4e911261ee0d2731536dba2
destination=TOPIC.EPrescriptionMDB messageSelector=null Local Create]
id=1" [_thread_blocked, id=5432, stack(0x6b0af000,0x6b0d0000)]
0x6b62f000 JavaThread "UILServerILService Accept Thread"
[_thread_in_native, id=5431, stack(0x6b35b000,0x6b37c000)]
0x6b1e8400 JavaThread "JCA PoolFiller" [_thread_blocked, id=5430,
stack(0x6b37c000,0x6b39d000)] 0x6bbf4400 JavaThread "IdleRemover"
[_thread_blocked, id=5429, stack(0x6b39d000,0x6b3be000)] 0x6b6aec00
JavaThread "TimeoutFactory" daemon [_thread_blocked, id=5428,
stack(0x6b3be000,0x6b3df000)] 0x6bc8f400 JavaThread "Timer-1" daemon
[_thread_blocked, id=5427, stack(0x6abdf000,0x6ac00000)] 0x6bccd400
JavaThread "JBossMQ Cache Reference Softner" daemon [_thread_blocked,
id=5426, stack(0x6a800000,0x6a821000)] 0x0a514400 JavaThread "HSQLDB
Timer #1ab5dae" daemon [_thread_blocked, id=5424,
stack(0x6b3df000,0x6b400000)] 0x0a411400 JavaThread
"ContainerBackgroundProcessor[StandardEngine[jboss.web]]" daemon
[_thread_blocked, id=5422, stack(0x6b788000,0x6b7a9000)] 0x09f58400
JavaThread "PooledInvokerAcceptor#0-4445" [_thread_in_native, id=5421,
stack(0x6b7f8000,0x6b819000)] 0x09f59000 JavaThread "RMI TCP
Accept-4444" daemon [_thread_in_native, id=5420,
stack(0x6b819000,0x6b83a000)] 0x09f46800 JavaThread
"ClassLoadingPool(2)-1" daemon [_thread_in_native, id=5419,
stack(0x6b83a000,0x6b85b000)] 0x09f3b000 JavaThread "JBoss System
Threads(1)-1" daemon [_thread_in_native, id=5418,
stack(0x6b85b000,0x6b87c000)] 0x09f87400 JavaThread "GC Daemon"
daemon [_thread_blocked, id=5417, stack(0x6b87c000,0x6b89d000)]
0x09f88c00 JavaThread "RMI Reaper" [_thread_blocked, id=5416,
stack(0x6b89d000,0x6b8be000)] 0x09f88400 JavaThread "RMI TCP
Accept-1098" daemon [_thread_in_native, id=5415,
stack(0x6b8be000,0x6b8df000)] 0x09f78c00 JavaThread "ScannerThread"
daemon [_thread_blocked, id=5414, stack(0x6b8df000,0x6b900000)]
0x09f3e800 JavaThread "Timer-0" daemon [_thread_blocked, id=5413,
stack(0x6ba39000,0x6ba5a000)] 0x099fcc00 JavaThread "DestroyJavaVM"
[_thread_blocked, id=5128, stack(0xb7f24000,0xb7f45000)] 0x09a9f000
JavaThread "Low Memory Detector" daemon [_thread_blocked, id=5137,
stack(0x6c349000,0x6c36a000)] 0x09a9d400 JavaThread
"CompilerThread1" daemon [_thread_blocked, id=5136,
stack(0x6c36a000,0x6c3eb000)] 0x09a9a400 JavaThread
"CompilerThread0" daemon [_thread_blocked, id=5135,
stack(0x6c3eb000,0x6c46c000)] 0x09a98c00 JavaThread "Signal
Dispatcher" daemon [_thread_blocked, id=5134,
stack(0x6c46c000,0x6c48d000)] 0x09a85c00 JavaThread "Finalizer"
daemon [_thread_blocked, id=5133, stack(0x6c68d000,0x6c6ae000)]
0x09a84400 JavaThread "Reference Handler" daemon [_thread_blocked,
id=5132, stack(0x6c6ae000,0x6c6cf000)]
Other Threads:
0x09a80c00 VMThread [stack: 0x6c6cf000,0x6c750000] [id=5131]
0x09aa1000 WatcherThread [stack: 0x6c2c8000,0x6c349000] [id=5138]
=>0x09a05400 (exited) GCTaskThread [stack: 0x6c927000,0x6c9a8000] [id=5130]
VM state:at safepoint (normal execution)
VM Mutex/Monitor currently owned by a thread: ([mutex/lock_event])
[0x099faff8] UNKNOWN - owner thread: 0x09a80c00 [0x099fb408] UNKNOWN -
owner thread: 0x0bd03c00
Heap PSYoungGen total 328704K, used 313529K [0x9f8b0000,
0xb4e00000, 0xb4e00000) eden space 307904K, 100% used
[0x9f8b0000,0xb2560000,0xb2560000) from space 20800K, 27% used
[0xb2560000,0xb2ade6b0,0xb39b0000) to space 20352K, 25% used
[0xb3a20000,0xb3f2e800,0xb4e00000) PSOldGen total 699072K,
used 489011K [0x74e00000, 0x9f8b0000, 0x9f8b0000) object space
699072K, 69% used [0x74e00000,0x92b8cca8,0x9f8b0000) PSPermGen
total 131072K, used 102017K [0x6ce00000, 0x74e00000, 0x74e00000)
object space 131072K, 77% used [0x6ce00000,0x731a0488,0x74e00000)
Dynamic libraries:
00312000-0032c000 r-xp 00000000 03:03 20776307 /lib/ld-2.5.so
0032c000-0032d000 r-xp 00019000 03:03 20776307 /lib/ld-2.5.so
0032d000-0032e000 rwxp 0001a000 03:03 20776307 /lib/ld-2.5.so
00330000-0046d000 r-xp 00000000 03:03 20776308 /lib/libc-2.5.so
0046d000-0046f000 r-xp 0013d000 03:03 20776308 /lib/libc-2.5.so
0046f000-00470000 rwxp 0013f000 03:03 20776308 /lib/libc-2.5.so
00470000-00473000 rwxp 00470000 00:00 0 00475000-0049a000 r-xp
00000000 03:03 20776309 /lib/libm-2.5.so 0049a000-0049b000 r-xp
00024000 03:03 20776309 /lib/libm-2.5.so 0049b000-0049c000 rwxp
00025000 03:03 20776309 /lib/libm-2.5.so 0049e000-004a0000 r-xp
00000000 03:03 20776314 /lib/libdl-2.5.so 004a0000-004a1000 r-xp
00001000 03:03 20776314 /lib/libdl-2.5.so 004a1000-004a2000 rwxp
00002000 03:03 20776314 /lib/libdl-2.5.so 004a4000-004b7000 r-xp
00000000 03:03 20776311 /lib/libpthread-2.5.so 004b7000-004b8000
r-xp 00012000 03:03 20776311 /lib/libpthread-2.5.so
004b8000-004b9000 rwxp 00013000 03:03 20776311
/lib/libpthread-2.5.so 004b9000-004bb000 rwxp 004b9000 00:00 0
008da000-008e1000 r-xp 00000000 03:03 20776312 /lib/librt-2.5.so
008e1000-008e2000 r-xp 00006000 03:03 20776312 /lib/librt-2.5.so
008e2000-008e3000 rwxp 00007000 03:03 20776312 /lib/librt-2.5.so
008eb000-008fa000 r-xp 00000000 03:03 20776327 /lib/libresolv-2.5.so
008fa000-008fb000 r-xp 0000e000 03:03 20776327 /lib/libresolv-2.5.so
008fb000-008fc000 rwxp 0000f000 03:03 20776327 /lib/libresolv-2.5.so
008fc000-008fe000 rwxp 008fc000 00:00 0 00c74000-00c87000 r-xp
00000000 03:03 20776313 /lib/libnsl-2.5.so 00c87000-00c88000 r-xp
00012000 03:03 20776313 /lib/libnsl-2.5.so 00c88000-00c89000 rwxp
00013000 03:03 20776313 /lib/libnsl-2.5.so 00c89000-00c8b000 rwxp
00c89000 00:00 0 06000000-06676000 r-xp 00000000 03:03 4620339
/usr/local/java/jdk1.6.0_11/jre/lib/i386/server/libjvm.so
06676000-066bb000 rwxp 00675000 03:03 4620339
/usr/local/java/jdk1.6.0_11/jre/lib/i386/server/libjvm.so
066bb000-06add000 rwxp 066bb000 00:00 0 08048000-08052000 r-xp
00000000 03:03 4685976 /usr/local/java/jdk1.6.0_11/bin/java
08052000-08053000 rwxp 00009000 03:03 4685976
/usr/local/java/jdk1.6.0_11/bin/java 099f7000-0bd51000 rwxp 099f7000
00:00 0 68c00000-68cd8000 rwxp 68c00000 00:00 0 68cd8000-68d00000
--xp 68cd8000 00:00 0 68e00000-68efc000 rwxp 68e00000 00:00 0 68efc000-68f00000 --xp 68efc000 00:00 0 68fe4000-68fe7000 --xp
68fe4000 00:00 0 68fe7000-69005000 rwxp 68fe7000 00:00 0
69005000-69008000 rwxp 69005000 00:00 0 69008000-69026000 rwxp
69008000 00:00 0 69026000-69045000 r-xs 00000000 03:03 4166654
/usr/share/X11/fonts/Type1/courb.pfa 69045000-69059000 r-xs 00000000
03:03 4166663 /usr/share/X11/fonts/Type1/l047016t.pfa
............... b7f59000-b7f5a000 rwxp b7f59000 00:00 0
b7f5a000-b7f5b000 --xp b7f5a000 00:00 0 b7f5b000-b7f5c000 rwxp
b7f5b000 00:00 0 b7f5c000-b7f5d000 r-xp b7f5c000 00:00 0
[vdso] bfb27000-bfb3d000 rwxp bfb27000 00:00 0 [stack]
VM Arguments:
jvm_args: -Dprogram.name=run.sh -Xmx1024m -Xms1024m -Xss128k
-Dsalmon.props.path=/usr/local/jboss/jboss-3.2.7/server/default/deploy/jbossweb-tomcat50.sar/salmonprops
-Djava.awt.headless=true -verbose:gc -XX:NewRatio=2 -XX:+PrintGCTimeStamps -XX:+PrintGCDetails -XX:PermSize=128m -XX:MaxPermSize=128m -Dsun.rmi.dgc.client.gcInterval=3600000 -Dsun.rmi.dgc.server.gcInterval=3600000 -Djava.io.tmpdir=/tmp -Xloggc:/usr/local/jboss/jboss-3.2.7/server/default/log/gclog.log -XX:+DisableExplicitGC -Djava.endorsed.dirs=/usr/local/jboss/jboss-3.2.7/lib/endorsed java_command: org.jboss.Main Launcher Type: SUN_STANDARD
Environment Variables:
PATH=/usr/kerberos/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/jboss/bin
LD_LIBRARY_PATH=/usr/local/java/jdk1.6.0_11/jre/lib/i386/server:/usr/local/java/jdk1.6.0_11/jre/lib/i386:/usr/local/java/jdk1.6.0_11/jre/../lib/i386
SHELL=/bin/bash
Signal Handlers:
SIGSEGV: [libjvm.so+0x606fd0], sa_mask[0]=0x7ffbfeff,
sa_flags=0x10000004 SIGBUS: [libjvm.so+0x606fd0],
sa_mask[0]=0x7ffbfeff, sa_flags=0x10000004 SIGFPE:
[libjvm.so+0x504690], sa_mask[0]=0x7ffbfeff, sa_flags=0x10000004
SIGPIPE: [libjvm.so+0x504690], sa_mask[0]=0x7ffbfeff,
sa_flags=0x10000004 SIGXFSZ: [libjvm.so+0x504690],
sa_mask[0]=0x7ffbfeff, sa_flags=0x10000004 SIGILL:
[libjvm.so+0x504690], sa_mask[0]=0x7ffbfeff, sa_flags=0x10000004
SIGUSR1: SIG_DFL, sa_mask[0]=0x00000000, sa_flags=0x00000000 SIGUSR2:
[libjvm.so+0x506b60], sa_mask[0]=0x00000000, sa_flags=0x10000004
SIGHUP: [libjvm.so+0x506900], sa_mask[0]=0x7ffbfeff,
sa_flags=0x10000004 SIGINT: SIG_IGN, sa_mask[0]=0x00000000,
sa_flags=0x00000000 SIGTERM: [libjvm.so+0x506900],
sa_mask[0]=0x7ffbfeff, sa_flags=0x10000004 SIGQUIT:
[libjvm.so+0x506900], sa_mask[0]=0x7ffbfeff, sa_flags=0x10000004
--------------- S Y S T E M ---------------
OS:Red Hat Enterprise Linux Server release 5.2 (Tikanga)
uname:Linux 2.6.18-92.el5 #1 SMP Tue Apr 29 13:16:12 EDT 2008 i686
libc:glibc 2.5 NPTL 2.5
rlimit: STACK 10240k, CORE 0k, NPROC 32601, NOFILE 1024, AS infinity
load average:0.10 0.06 0.02
CPU:total 2 (2 cores per cpu, 1 threads per core) family 6 model 15 stepping 13, cmov, cx8, fxsr, mmx, sse, sse2, sse3, ssse3
Memory: 4k page, physical 2065048k(434208k free), swap 4192956k(4192956k free)
vm_info: Java HotSpot(TM) Server VM (11.0-b16) for linux-x86 JRE (1.6.0_11-b03), built on Nov 10 2008 01:21:35 by "java_re" with gcc 3.2.1-7a (J2SE release)
time: Tue Jun 14 15:50:47 2011
elapsed time: 17909 seconds`
Well, you're running you're JVM with a 2GB heap on a machine with about 2GB ram, and the heap isn't the only thing in the JVM (or system for that matter) using memory. Might it be possible that a heap access is causing a page fault (forcing it to go into swap)? I wouldn't expect the JVM to puke in that scenario, but I suppose it's possible.
EDIT: Sorry, misread that. 1GB heap, 2GB ram. I guess I have no idea, then.
Try to check the hardware in the system. Maybe Memory has gone old.
First benchmark the hardware (use sysbench incase of linux).
Just change the memory, it will work fine.
Related
How to display user-mode stack of current kernel-mode context?
Suppose I'm broken into Kernel Debugger, during a system call or an IOCTL that started from user mode. I want to see the full stack - starting from user mode and switching to kernel mode. I know some of the pieces: I can list all the processes with !process 0 0, and see the threads inside them. Or, drowse using !dml_proc. If I know the current usermode thread, I can switch to it and see the whole start: !thread <usermode_thread> 15. But how do I tie them together? Specifically: How do I know the current usermode thread? Can I retrieve it programatically, so I can have a one-liner !thread <some_magic> 15?
WinDbg keeps track of the current thread in both user mode as well as kernel mode in the pseudo register #$thread, and current process in the pseudo register #$proc. To see current thread stack: !thread #$thread 0x1f. This is the full stack, starting from _RtlUserThreadStart up to the last api which usually would be SwapContext() if the thread ceded execution. To see current process with all its threads' stacks: !process #$proc 0x1f here is a complete current threads stack spat out by both commands !proc #$proc 1f as well as !thread #$thread 1f the first will print the callstack of all threads of a specific process the second will print the call stack of a specific currently running thread you can notice the wait status as RUNNING on the first line of Threads output :\>head -2 sample.txt kd> !process #$proc 0x17 PROCESS 84efc748 SessionId: 0 Cid: 0470 Peb: 7ffdc000 ParentCid: 01e8 :\>wc -l sample.txt 517 sample.txt :\>grep THREAD sample.txt THREAD 84efcb60 Cid 0470.0474 Teb: 7ffdf000 Win32Thread: ff9f1c00 WAIT: (UserRequest) UserMode Non-Alertable THREAD 84f166b8 Cid 0470.0478 Teb: 7ffde000 Win32Thread: 00000000 WAIT: (UserRequest) UserMode Alertable THREAD 85303820 Cid 0470.0490 Teb: 7ffd9000 Win32Thread: 00000000 WAIT: (UserRequest) UserMode Non-Alertable THREAD 85307bb0 Cid 0470.0494 Teb: 7ffd8000 Win32Thread: 00000000 WAIT: (UserRequest) UserMode Non-Alertable THREAD 853064e8 Cid 0470.0498 Teb: 7ffd7000 Win32Thread: 00000000 WAIT: (UserRequest) UserMode Non-Alertable THREAD 84e71d48 Cid 0470.049c Teb: 7ffd6000 Win32Thread: 00000000 WAIT: (UserRequest) UserMode Non-Alertable THREAD 8530eb80 Cid 0470.04a8 Teb: 7ffda000 Win32Thread: 00000000 WAIT: (UserRequest) UserMode Non-Alertable THREAD 85371030 Cid 0470.0538 Teb: 7ffd4000 Win32Thread: 00000000 WAIT: (WrQueue) UserMode Alertable THREAD 8538ea08 Cid 0470.0588 Teb: 7ffad000 Win32Thread: 00000000 WAIT: (UserRequest) UserMode Non-Alertable THREAD 853d6310 Cid 0470.05a4 Teb: 7ffaa000 Win32Thread: 00000000 WAIT: (WrQueue) UserMode Alertable THREAD 8540fa78 Cid 0470.06bc Teb: 7ffaf000 Win32Thread: 00000000 WAIT: (UserRequest) UserMode Non-Alertable THREAD 84173030 Cid 0470.0740 Teb: 7ffa6000 Win32Thread: 00000000 WAIT: (WrQueue) UserMode Non-Alertable THREAD 84f77990 Cid 0470.0174 Teb: 7ffdd000 Win32Thread: 00000000 RUNNING on processor 0 THREAD 84f77990 Cid 0470.0174 Teb: 7ffdd000 Win32Thread: 00000000 RUNNING on processor 0 :\>grep !thread sample.txt kd> !thread #$thread 1f :\>grep -A 2 !thread sample.txt kd> !thread #$thread 1f THREAD 84f77990 Cid 0470.0174 Teb: 7ffdd000 Win32Thread: 00000000 RUNNING on processor 0 Not impersonating :\> the full call stack of a currently running thread broken on NtDeviceIoControlFile kd> !thread #$thread 1f THREAD 84f77990 Cid 0470.0174 Teb: 7ffdd000 Win32Thread: 00000000 RUNNING on processor 0 Not impersonating DeviceMap 919b0008 Owning Process 84efc748 Image: svchost.exe Attached Process N/A Image: N/A Wait Start TickCount 49461 Ticks: 1 (0:00:00:00.010) Context Switch Count 659 IdealProcessor: 0 UserTime 00:00:00.040 KernelTime 00:00:03.314 Win32 Start Address ntdll!TppWorkerThread (0x770403e7) Stack Init 8c0e3fd0 Current 8c0e3bf8 Base 8c0e4000 Limit 8c0e1000 Call 00000000 Priority 8 BasePriority 8 PriorityDecrement 0 IoPriority 2 PagePriority 5 ChildEBP RetAddr 8c0e3d04 8283787a nt!NtDeviceIoControlFile 8c0e3d04 770570b4 nt!KiFastCallEntry+0x12a (FPO: [0,3] TrapFrame # 8c0e3d34) 0292f518 77055864 ntdll!KiFastSystemCallRet (FPO: [0,0,0]) 0292f51c 76dd14c9 ntdll!ZwDeviceIoControlFile+0xc (FPO: [10,0,0]) 0292f560 76dd15f9 NSI!NsiIoctl+0x5d (FPO: [Non-Fpo]) 0292f584 76dd15c7 NSI!NsiEnumerateObjectsAllParametersEx+0x23 (FPO: [Non-Fpo]) 0292f5d0 718718e2 NSI!NsiEnumerateObjectsAllParameters+0x79 (FPO: [Non-Fpo]) 0292f620 71871858 ncsi!CNcsiNsiTable::AllocateAndGetTable+0x51 (FPO: [Non-Fpo]) 0292f644 718717c6 ncsi!UpdateInterfaceStatsByFamily+0xb0 (FPO: [Non-Fpo]) 0292f664 71871e49 ncsi!UpdateInterfaceStats+0x26 (FPO: [Non-Fpo]) 0292f694 71871d4e ncsi!PassiveProbe+0x6f (FPO: [Non-Fpo]) 0292f6f4 718715d5 ncsi!NcsiUpdateConnectivityStatusList+0x4a1 (FPO: [Non-Fpo]) 0292f6f8 7704112c ncsi!NcsiUpdateConnectivityStatusListTimer+0x5 (FPO: [3,0,0]) 0292f71c 77040842 ntdll!TppTimerpExecuteCallback+0x10f (FPO: [Non-Fpo]) 0292f87c 756f3c45 ntdll!TppWorkerThread+0x572 (FPO: [Non-Fpo]) 0292f888 770737f5 kernel32!BaseThreadInitThunk+0xe (FPO: [Non-Fpo]) 0292f8c8 770737c8 ntdll!__RtlUserThreadStart+0x70 (FPO: [Non-Fpo]) 0292f8e0 00000000 ntdll!_RtlUserThreadStart+0x1b (FPO: [Non-Fpo])
Stuck with crash dump analysis - user32!NtUserSetFocus freezes app?
We have an old VB6 application that keeps freezing and/or crashing. We setup an easy shortcut for users to create a full dump when the program freezes.
We now have a week's worth of dumps, and most of them show a cryptic stack from the hung thread, which doesn't even include any of our functions, and the last call is to user32!NtUserSetFocus.
We're really stuck here... can anyone help?
0:000> kb
ChildEBP RetAddr Args to Child
0018d788 755eee3e 002609ac 00000001 755eed34 user32!NtUserSetFocus+0x15
0018d7a0 755962fa 00260f22 00000110 002609ac user32!MB_DlgProc+0x10a
0018d7cc 755bf9df 755eed34 00260f22 00000110 user32!InternalCallWinProc+0x23
0018d848 755bf784 00000000 755eed34 00260f22 user32!UserCallDlgProcCheckWow+0xd7
0018d898 755bf889 0160cf70 00000000 00000110 user32!DefDlgProcWorker+0xb7
0018d8b8 755962fa 00260f22 00000110 002609ac user32!DefDlgProcW+0x29
0018d8e4 75596d3a 755bf860 00260f22 00000110 user32!InternalCallWinProc+0x23
0018d95c 7559965e 00000000 76ee3d54 00260f22 user32!UserCallWinProcCheckWow+0x109
0018d9a0 755c206f 0160cf70 00000000 76ee3d54 user32!SendMessageWorker+0x581
0018da74 755bcf4b 75580000 00000008 00000000 user32!InternalCreateDialog+0xb9f
0018daac 755ef73c 75580000 1736f6a8 0010193a user32!InternalDialogBox+0xc1
0018db60 755efa18 00000030 ffffffff ffff0000 user32!SoftModalMessageBox+0x757
0018dcb8 755efc65 0018dcd0 00000000 755efbd1 user32!MessageBoxWorker+0x269
0018dd38 729af829 0018dd50 0010193a 0010193a user32!MessageBoxIndirectA+0x94
0018dd78 729af6a5 0018ddbc 00000000 00000000 msvbvm60!VBMessageBox2+0x92
0018dda0 729af9a0 729af7ce 0018ddbc 0018ddbc msvbvm60!MessageBoxPVoid+0x4b
0018ddd0 729a3d68 00000000 12aa8ef8 0018ddf8 msvbvm60!DlgEnableModeless+0x5e
0018de34 729a3db6 032807d4 002308e4 00000030 msvbvm60!_Scanint+0x13
0018de54 72a0c411 174be0e4 00000000 00000030 msvbvm60!RefMemberIDFromHxmod+0x39
0018de70 72a0c6f3 174be0e4 00000000 00000030 msvbvm60!EbShowError+0x3
0018de94 72a2497c 010831b8 00000000 00000000 msvbvm60!GetErrMsg+0x90
0018deb0 770fb6ad 0018df9c 00000000 0018dfec msvbvm60!SehUpdateStack+0x29
0018ded4 770fb67f 0018df9c 0018f6f8 0018dfec ntdll!ExecuteHandler2+0x26
0018def8 770fb620 0018df9c 0018f6f8 0018dfec ntdll!ExecuteHandler+0x24
0018df84 770b0163 0018df9c 0018dfec 0018df9c ntdll!RtlDispatchException+0x127
0018df84 74f3c42d 0018df9c 0018dfec 0018df9c ntdll!KiUserExceptionDispatcher+0xf
0018e4a0 72a10dcf c000008f 00000001 00000002 KERNELBASE!RaiseException+0x58
0018e4c0 72a0e228 010831b8 800a0061 0018e584 msvbvm60!CEnumConPnts::QueryInterface+0x34
0018e4d4 72a0e28c 010831b8 00000000 00000000 msvbvm60!BasicExcepDeferredFillIn+0x65
0018e4e4 72a0be99 00000061 11035a85 00000000 msvbvm60!BasicExcepDeferredFillIn+0xd2
00000000 00000000 00000000 00000000 00000000 msvbvm60!ValidateArray+0xb4
!runaway proves that this is indeed the hung thread.
0:000> !runaway
User Mode Time
Thread Time
0:ca8 0 days 0:01:00.325
8:13a4 0 days 0:00:00.171
6:10b4 0 days 0:00:00.062
10:1554 0 days 0:00:00.031
19:1598 0 days 0:00:00.000
18:a88 0 days 0:00:00.000
17:7b0 0 days 0:00:00.000
16:ba0 0 days 0:00:00.000
15:770 0 days 0:00:00.000
14:103c 0 days 0:00:00.000
13:16bc 0 days 0:00:00.000
12:17e4 0 days 0:00:00.000
11:1160 0 days 0:00:00.000
9:1194 0 days 0:00:00.000
7:dc8 0 days 0:00:00.000
5:1510 0 days 0:00:00.000
4:af0 0 days 0:00:00.000
3:e5c 0 days 0:00:00.000
2:10f4 0 days 0:00:00.000
1:ff8 0 days 0:00:00.000
!analyze -hang -v produces the following:
0:000> !analyze -hang -v
*******************************************************************************
* *
* Exception Analysis *
* *
*******************************************************************************
*** WARNING: Unable to verify checksum for mscorlib.ni.dll
GetUrlPageData2 (WinHttp) failed: 12152.
FAULTING_IP:
+0
00000000 ?? ???
EXCEPTION_RECORD: ffffffff -- (.exr 0xffffffffffffffff)
ExceptionAddress: 00000000
ExceptionCode: 80000003 (Break instruction exception)
ExceptionFlags: 00000000
NumberParameters: 0
CONTEXT: 00000000 -- (.cxr 0x0;r)
eax=00000001 ebx=00000000 ecx=00000000 edx=00000000 esi=0018dcd0 edi=00260f22
eip=755a218a esp=0018d788 ebp=0018d7a0 iopl=0 nv up ei pl zr na pe nc
cs=0023 ss=002b ds=002b es=002b fs=0053 gs=002b efl=00200246
user32!NtUserSetFocus+0x15:
755a218a 83c404 add esp,4
FAULTING_THREAD: 00000000
BUGCHECK_STR: HANG
DEFAULT_BUCKET_ID: APPLICATION_HANG
PROCESS_NAME: OurProcess.exe
ERROR_CODE: (NTSTATUS) 0xcfffffff - <Unable to get error code text>
EXCEPTION_CODE: (NTSTATUS) 0xcfffffff - <Unable to get error code text>
NTGLOBALFLAG: 0
APPLICATION_VERIFIER_FLAGS: 0
APP: OurProcess.exe
ANALYSIS_VERSION: 6.3.9600.16384 (debuggers(dbg).130821-1623) x86fre
MANAGED_STACK: !dumpstack -EE
OS Thread Id: 0xca8 (0)
Current frame:
ChildEBP RetAddr Caller, Callee
DERIVED_WAIT_CHAIN:
Dl Eid Cid WaitType
-- --- ------- --------------------------
0 758.ca8 Unknown
WAIT_CHAIN_COMMAND: ~0s;k;;
BLOCKING_THREAD: 00000ca8
PRIMARY_PROBLEM_CLASS: APPLICATION_HANG
LAST_CONTROL_TRANSFER: from 755eee3e to 755a218a
STACK_TEXT:
0018d788 755eee3e 002609ac 00000001 755eed34 user32!NtUserSetFocus+0x15
0018d7a0 755962fa 00260f22 00000110 002609ac user32!MB_DlgProc+0x10a
0018d7cc 755bf9df 755eed34 00260f22 00000110 user32!InternalCallWinProc+0x23
0018d848 755bf784 00000000 755eed34 00260f22 user32!UserCallDlgProcCheckWow+0xd7
0018d898 755bf889 0160cf70 00000000 00000110 user32!DefDlgProcWorker+0xb7
0018d8b8 755962fa 00260f22 00000110 002609ac user32!DefDlgProcW+0x29
0018d8e4 75596d3a 755bf860 00260f22 00000110 user32!InternalCallWinProc+0x23
0018d95c 7559965e 00000000 76ee3d54 00260f22 user32!UserCallWinProcCheckWow+0x109
0018d9a0 755c206f 0160cf70 00000000 76ee3d54 user32!SendMessageWorker+0x581
0018da74 755bcf4b 75580000 00000008 00000000 user32!InternalCreateDialog+0xb9f
0018daac 755ef73c 75580000 1736f6a8 0010193a user32!InternalDialogBox+0xc1
0018db60 755efa18 00000030 ffffffff ffff0000 user32!SoftModalMessageBox+0x757
0018dcb8 755efc65 0018dcd0 00000000 755efbd1 user32!MessageBoxWorker+0x269
0018dd38 729af829 0018dd50 0010193a 0010193a user32!MessageBoxIndirectA+0x94
0018dd78 729af6a5 0018ddbc 00000000 00000000 msvbvm60!VBMessageBox2+0x92
0018dda0 729af9a0 729af7ce 0018ddbc 0018ddbc msvbvm60!MessageBoxPVoid+0x4b
0018ddd0 729a3d68 00000000 12aa8ef8 0018ddf8 msvbvm60!DlgEnableModeless+0x5e
0018de34 729a3db6 032807d4 002308e4 00000030 msvbvm60!_Scanint+0x13
0018de54 72a0c411 174be0e4 00000000 00000030 msvbvm60!RefMemberIDFromHxmod+0x39
0018de70 72a0c6f3 174be0e4 00000000 00000030 msvbvm60!EbShowError+0x3
0018de94 72a2497c 010831b8 00000000 00000000 msvbvm60!GetErrMsg+0x90
0018deb0 770fb6ad 0018df9c 00000000 0018dfec msvbvm60!SehUpdateStack+0x29
0018ded4 770fb67f 0018df9c 0018f6f8 0018dfec ntdll!ExecuteHandler2+0x26
0018def8 770fb620 0018df9c 0018f6f8 0018dfec ntdll!ExecuteHandler+0x24
0018df84 770b0163 0018df9c 0018dfec 0018df9c ntdll!RtlDispatchException+0x127
0018df84 74f3c42d 0018df9c 0018dfec 0018df9c ntdll!KiUserExceptionDispatcher+0xf
0018e4a0 72a10dcf c000008f 00000001 00000002 KERNELBASE!RaiseException+0x58
0018e4c0 72a0e228 010831b8 800a0061 0018e584 msvbvm60!CEnumConPnts::QueryInterface+0x34
0018e4d4 72a0e28c 010831b8 00000000 00000000 msvbvm60!BasicExcepDeferredFillIn+0x65
0018e4e4 72a0be99 00000061 11035a85 00000000 msvbvm60!BasicExcepDeferredFillIn+0xd2
00000000 00000000 00000000 00000000 00000000 msvbvm60!ValidateArray+0xb4
FOLLOWUP_IP:
msvbvm60!VBMessageBox2+92
729af829 8bd8 mov ebx,eax
SYMBOL_STACK_INDEX: e
SYMBOL_NAME: msvbvm60!VBMessageBox2+92
FOLLOWUP_NAME: MachineOwner
MODULE_NAME: msvbvm60
IMAGE_NAME: msvbvm60.dll
DEBUG_FLR_IMAGE_TIMESTAMP: 4a5bda6c
STACK_COMMAND: ~0s ; kb
BUCKET_ID: HANG_msvbvm60!VBMessageBox2+92
FAILURE_BUCKET_ID: APPLICATION_HANG_cfffffff_msvbvm60.dll!VBMessageBox2
ANALYSIS_SOURCE: UM
FAILURE_ID_HASH_STRING: um:application_hang_cfffffff_msvbvm60.dll!vbmessagebox2
FAILURE_ID_HASH: {e6dc63dc-251f-a6d5-c66e-f5e07e418955}
Followup: MachineOwner
---------
Firstly I'd find out what it's doing. When it hangs get a couple of crash dumps and compare the stacks to find out where exactly it's spinning. Also some of the symbols have large offsets (user32!InternalCreateDialog+0xb9f), are you using a symbol server?
Mixed mode crash from workerthread pool, but no managed thread
It’s a large 32 bits mixed mode MFC 7.0 app on XP, the user tells that he was using a feature which is implemented in managed code. The crach is in a thread that has acquired the LoaderLock, and seems to orgin from .NET workerthread pool. 0:016> !cs -o -l ----------------------------------------- DebugInfo = 0x7c97e1a0 Critical section = 0x7c97e174 (ntdll!LdrpLoaderLock+0x0) LOCKED LockCount = 0x4 OwningThread = 0x00000260 RecursionCount = 0x1 LockSemaphore = 0x7BC SpinCount = 0x00000000 OwningThread DbgId = ~16s OwningThread Stack = ChildEBP RetAddr Args to Child 0f66e400 7c90df4a 7c8648a2 00000002 0f66e57c ntdll!KiFastSystemCallRet (FPO: [0,0,0]) 0f66e404 7c8648a2 00000002 0f66e57c 00000001 ntdll!ZwWaitForMultipleObjects+0xc (FPO: [5,0,0]) 0f66e74c 7c83ab50 0f66e774 7c839b39 0f66e77c kernel32!UnhandledExceptionFilter+0x8b9 (FPO: [Non-Fpo]) 0f66e754 7c839b39 0f66e77c 00000000 0f66e77c kernel32!BaseThreadStart+0x4d (FPO: [Non-Fpo]) 0f66e77c 7c9032a8 0f66e868 0f66ffdc 0f66e884 kernel32!_except_handler3+0x61 (FPO: [Uses EBP] [3,0,7]) 0f66e7a0 7c90327a 0f66e868 0f66ffdc 0f66e884 ntdll!ExecuteHandler2+0x26 0f66e850 7c90e48a 00000000 0f66e884 0f66e868 ntdll!ExecuteHandler+0x24 0f66e850 79247eb4 00000000 0f66e884 0f66e868 ntdll!KiUserExceptionDispatcher+0xe (FPO: [2,0,0]) (CONTEXT # 0f66e884) 0f66eb4c 7929a46e 0e715d80 792483ef 0e715d80 mscorwks!Thread::UnhijackThread+0xb (FPO: [0,0,0]) 0f66eb54 792483ef 0e715d80 00000000 00000000 mscorwks!Thread::RareEnablePreemptiveGC+0x36 (FPO: [0,0,0]) 0f66eb64 792a6ff9 06ee0000 00000000 00000000 mscorwks!Thread::RareDisablePreemptiveGC+0x5f (FPO: [0,0,0]) 0f66ec10 79247e14 06ee0000 00000003 00000000 mscorwks!SystemDomain::RunDllMain+0x7d (FPO: [Non-Fpo]) 0f66ee98 603d6a2c 00000001 00000003 00000000 mscorwks!ExecuteDLL+0x3c0 (FPO: [Non-Fpo]) 0f66eed8 603d70a3 06ee0000 0f66eebc 00000000 mscoreei!CorDllMainWorker+0x153 (FPO: [Non-Fpo]) 0f66ef14 79015012 00000000 00000003 00000000 mscoreei!_CorDllMain+0x111 (FPO: [Non-Fpo]) 0f66ef30 7c90118a 06ee0000 00000003 00000000 mscoree!ShellShim__CorDllMain+0xad (FPO: [Non-Fpo]) 0f66ef50 7c91397b 06ef841e 06ee0000 00000003 ntdll!LdrpCallInitRoutine+0x14 0f66efc8 7c80c136 00000000 793fa180 7c80934a ntdll!LdrShutdownThread+0xd7 (FPO: [Non-Fpo]) 0f66f000 792ee8ad 00000000 00000000 792ee78a kernel32!ExitThread+0x3e (FPO: [Non-Fpo]) 0f66f020 792edfcb 00000000 00000000 00000000 mscorwks!ThreadpoolMgr::WorkerThreadStart+0x123 (FPO: [Non-Fpo]) Some interesting vales on the stack might be the 06ee0000 and 0f66eebc. The first is the base address for myMixedModeDll, and the second: 0:016> ln 06ef841e (06ef841e) myMixedModeDll!CorDllMain | (06ef8424) myMixedModeDll!CDialog::CDialog Exact matches: The actual exception should be here: 0:000> .cxr 0f66e884;kb eax=000000df ebx=00000000 ecx=0e715d80 edx=000003a4 esi=0e715d80 edi=00010000 eip=79247eb4 esp=0f66eb50 ebp=0f66ec10 iopl=0 nv up ei ng nz na po nc cs=001b ss=0023 ds=0023 es=0023 fs=003b gs=0000 efl=00010282 mscorwks!Thread::UnhijackThread+0xb: 79247eb4 8910 mov dword ptr [eax],edx ds:0023:000000df=???????? *** Stack trace for last set context - .thread/.cxr resets it ChildEBP RetAddr Args to Child 0f66eb4c 7929a46e 0e715d80 792483ef 0e715d80 mscorwks!Thread::UnhijackThread+0xb 0f66eb54 792483ef 0e715d80 00000000 00000000 mscorwks!Thread::RareEnablePreemptiveGC+0x36 And yes the eax is not good: 0:000> u 79247eae mscorwks!Thread::UnhijackThread+0x5: 79247eae 8b5178 mov edx,dword ptr [ecx+78h] 79247eb1 8b417c mov eax,dword ptr [ecx+7Ch] 79247eb4 8910 mov dword ptr [eax],edx Yes, ECX has been restored properly 0:016> dd #ecx+0x78 L1 0e715df8 000003a4 0:016> dd #ecx+0x7c L1 0e715dfc 000000df 0:016> dd #ecx L0x20 0e715d80 0e6f4798 00000000 ffffffff 00000000 0e715d90 00000000 00000020 00000000 0e715da0 0e715da0 0e715da0 0e715da0 00000000 00000000 0e715db0 00000000 000000df 00000000 00000000 0e715dc0 00000000 00000000 00000000 00000000 0e715dd0 00000000 00000000 00000000 00000000 0e715de0 00000000 00000000 00000000 00000000 0e715df0 0e7093e8 00002733 000003a4 000000df The last error value 0:016> !gle LastErrorValue: (Win32) 0 (0) - The operation completed successfully. LastStatusValue: (NTSTATUS) 0xc0000034 - Object Name not found. This .NET is version 1.1.4322 , and the sos! Claims that thread #16 is not a managed thread. 0:016> !t ThreadCount: 10 UnstartedThread: 0 BackgroundThread: 10 PendingThread: 0 DeadThread: 0 PreEmptive GC Alloc Lock ID ThreadOBJ State GC Context Domain Count APT Exception 0 0xc8c 0x001ae598 0x4220 Enabled 0x1b7df804:0x1b7df8d8 0x001fda98 0 STA 5 0xcd4 0x001caea0 0xb220 Enabled 0x00000000:0x00000000 0x001fda98 0 MTA (Finalizer) 8 0xe28 0x0c56ac40 0x220 Enabled 0x00000000:0x00000000 0x001fda98 0 Ukn 10 0x8a8 0x0e5f4b48 0x800220 Enabled 0x1b822518:0x1b824458 0x001fda98 0 MTA (Threadpool Completion Port) 11 0xc18 0x0e6d6a60 0x800220 Enabled 0x1b8651cc:0x1b867008 0x001fda98 0 MTA (Threadpool Completion Port) 12 0xa54 0x00190c28 0x220 Enabled 0x1b5247f0:0x1b52650c 0x001fda98 0 Ukn 13 0xe9c 0x0e6627f8 0x220 Enabled 0x1b5307f0:0x1b53250c 0x001fda98 0 Ukn 14 0xe58 0x0e6b11a0 0x1800220 Enabled 0x00000000:0x00000000 0x001fda98 0 MTA (Threadpool Worker) 15 0x8dc 0x0e6d68a8 0x220 Enabled 0x00000000:0x00000000 0x001fda98 0 Ukn 17 0xbcc 0x0e709378 0x220 Enabled 0x1b52c7f0:0x1b52e50c 0x001fda98 0 Ukn 0:016> !ClrStack Thread 16 Not a managed thread. How can I find out more to reveal the cause to this crash ?
hang analysis - lock on xls OleDbConnection?
Admins told us that there is some prolem in production. They noticed big jump in memory usage and in requests waiting. I received one crash dump. I need help analysing it. Using Debug Diagnostic Tool I found this: Detected possible blocking or leaked critical section at 0x1e5bd320 owned by thread 141 in dllhst3g.exe__Metastorm Process Engine__PID__7444__Date__10_25_2011__Time_01_19_15PM__686__Manual Dump.dmp Impact of this lock 11,59% of threads blocked (Threads 97 137 142 143 144 147 207 208 211 212 213 214 215 216 217 218 219 221 222 223 224 225 226 227 228 229 230) The following functions are trying to enter this critical section ACECORE+20eb The following module(s) are involved with this critical section C:\Program Files\Common Files\Microsoft Shared\OFFICE12\ACECORE.DLL from Microsoft Corporation From the listed threads only thread 142 is waiting for thread 141. All others are waiting for thread 142. ~141 141 Id: 1d14.1b38 Suspend: 0 Teb: 7fee4000 Unfrozen Start: msvcrt!_endthreadex+0x2f (77bcb4bc) Priority: 0 Priority class: 32 Affinity: ff ~141s : Edit: new stacktrace after fixing symbols kb 2a2efdcc 7c827b89 77e6202c 00000003 2a2efe1c ntdll!KiFastSystemCallRet 2a2efdd0 77e6202c 00000003 2a2efe1c 00000001 ntdll!NtWaitForMultipleObjects+0xc 2a2efe78 7739bbd1 00000003 2a2efea0 00000000 kernel32!WaitForMultipleObjectsEx+0x11a 2a2efed4 7739ce36 00000002 2a2eff74 ffffffff user32!RealMsgWaitForMultipleObjectsEx+0x141 2a2efef0 4a77cb28 00000002 2a2eff74 00000000 user32!MsgWaitForMultipleObjects+0x1f 2a2eff84 77bcb530 33c23fe8 00000000 00000000 comsvcs!CSTAThread::WorkerLoop+0x1f9 2a2effb8 77e6482f 37e3b7e8 00000000 00000000 msvcrt!_endthreadex+0xa3 2a2effec 00000000 77bcb4bc 37e3b7e8 00000000 kernel32!BaseThreadStart+0x34 ~142 142 Id: 1d14.1128 Suspend: 0 Teb: 7feb9000 Unfrozen Start: msvcrt!_endthreadex+0x2f (77bcb4bc) Priority: 0 Priority class: 32 Affinity: ff ~142s : Edit: new stacktrace after fixing symbols kb 31b5aa88 7c827b99 7c83d09c 00006c44 00000000 ntdll!KiFastSystemCallRet 31b5aa8c 7c83d09c 00006c44 00000000 00000000 ntdll!ZwWaitForSingleObject+0xc 31b5aac8 7c83d0e7 00006c44 00000004 00000000 ntdll!RtlpWaitOnCriticalSection+0x1a3 *** ERROR: Module load completed but symbols could not be loaded for ACECORE.DLL 31b5aae8 3c9e20eb 1e5bd320 31b5ab3c 2ae13a61 ntdll!RtlEnterCriticalSection+0xa8 WARNING: Stack unwind information not available. Following frames may be wrong. 00000000 00000000 00000000 00000000 00000000 ACECORE+0x20eb !clrstack Shows call to System.Data.OleDb.OleDbConnection.Open() I found it connects to xls Provider=Microsoft.ACE.OLEDB.12.0;Data Source=c:\temp\somefile.xls;Extended Properties="Excel 8.0;HDR=Yes;" ~97 97 Id: 1d14.730 Suspend: 0 Teb: 7fee5000 Unfrozen Start: mscorwks!CorExitProcess+0x21ef9 (79f756cf) Priority: 0 Priority class: 32 Affinity: ff Edit: new stacktrace after fixing symbols ~97s kb 27e1ca10 7c827b99 77e61d1e 000018e4 00000000 ntdll!KiFastSystemCallRet 27e1ca14 77e61d1e 000018e4 00000000 00000000 ntdll!ZwWaitForSingleObject+0xc 27e1ca84 77e61c8d 000018e4 ffffffff 00000000 kernel32!WaitForSingleObjectEx+0xac 27e1ca98 7769c7ad 000018e4 ffffffff 2386d3a8 kernel32!WaitForSingleObject+0x12 27e1cab4 7778b5cb 24a1c758 2386d3a8 00000000 ole32!GetToSTA+0x7c 27e1cad4 7778c38b 27e1cb9c 27e1cc9c 22d3e674 ole32!CRpcChannelBuffer::SwitchAptAndDispatchCall+0xcb 27e1cbb4 776c0585 22d3e674 27e1ccac 27e1cc9c ole32!CRpcChannelBuffer::SendReceive2+0xd3 27e1cc20 776c051a 22d3e674 27e1ccac 27e1cc9c ole32!CAptRpcChnl::SendReceive+0xab 27e1cc74 77ce347f 22d3e674 27e1ccac 27e1cc9c ole32!CCtxComChnl::SendReceive+0x1a9 27e1cc90 77ce352f 45e02be4 27e1ccd8 0600016e rpcrt4!NdrProxySendReceive+0x43 27e1d080 77ce35a6 776762b8 776794ba 27e1d0b8 rpcrt4!NdrClientCall2+0x206 27e1d0a0 77c65037 00000014 00000004 27e1d0d0 rpcrt4!ObjectStublessClient+0x8b 27e1d0b0 776ad951 45e02be4 00000000 27e1d89c rpcrt4!ObjectStubless+0xf 27e1d0d0 776acb4b 77794960 00000001 00000000 ole32!CProcessActivator::CCICallback+0x6d 27e1d0f0 776acafc 77794960 27e1d6f8 00000000 ole32!CProcessActivator::AttemptActivation+0x2c 27e1d12c 776ada3b 77794960 27e1d6f8 00000000 ole32!CProcessActivator::ActivateByContext+0x4f 27e1d154 776aaf9e 77794960 00000000 27e1d89c ole32!CProcessActivator::CreateInstance+0x49 27e1d194 4a777108 27e1d89c 00000000 27e1d5d8 ole32!ActivationPropertiesIn::DelegateCreateInstance+0xf7 27e1d210 776aaf9e 443dd8f8 00000000 27e1d89c comsvcs!CSTAPoolActivator::CreateInstance+0x5a9 27e1d250 4a766303 27e1d89c 00000000 27e1d5d8 ole32!ActivationPropertiesIn::DelegateCreateInstance+0xf7 27e1d2a0 776aaf9e 341d3168 00000000 27e1d89c comsvcs!CStdContextActivator::CreateInstance+0x221 27e1d2e0 77727f8a 27e1d89c 00000000 27e1d5d8 ole32!ActivationPropertiesIn::DelegateCreateInstance+0xf7 27e1d310 776f5c55 0009a0f8 00000000 27e1d89c ole32!CSurrogateProcessActivator::CreateInstance+0xf7 27e1d344 776aaf9e 7779487c 00000000 27e1d89c ole32!CClientContextActivator::CreateInstance+0xc9 27e1d384 776ab12f 27e1d89c 00000000 27e1d5d8 ole32!ActivationPropertiesIn::DelegateCreateInstance+0xf7 27e1d68c 776a67ba 361f8c54 00000000 00000015 ole32!ICoCreateInstanceEx+0x3f8 27e1d6c0 7769b9b3 361f8c54 00000000 00000000 ole32!CComActivator::DoCreateInstance+0x6a 27e1ddc8 4a7516d8 45bfeb58 361f8c54 00000000 ole32!CComActivator::StandardCreateInstance+0x7c 27e1de3c 4a751fc6 00000000 361f8c40 00000000 comsvcs!CClassFactoryWrapper::ActivateOnMachine+0xaf 27e1de74 7a078d9b 361f8c40 00000000 79edda70 comsvcs!CClassFactoryWrapper::CreateInstance+0x80 27e1e118 7a07a1e6 361f8c40 00000000 00000000 mscorwks!ComClassFactory::CreateInstanceFromClassFactory+0x102 27e1e174 7a0bf10a 1e772994 00000000 27e1e1b4 mscorwks!ComClassFactory::CreateInstance+0x91 27e1e184 7a0c870b 1e772994 1e772994 79faa672 mscorwks!AllocateComObject_ForManaged+0x1e 27e1e1b4 79e9c82b 1e772994 234b20e8 79faa737 mscorwks!AllocateObject+0x38 27e1e1c0 79faa737 05ff2627 00000001 072336fc mscorwks!MethodTable::Allocate+0x35 27e1e260 792c25c3 1e772994 27e1e278 792c257c mscorwks!CRemotingServices::AllocateUninitializedObject+0xdf 27e1e278 792c1951 0f5b1b50 27e1e2d0 03100120 mscorlib_ni+0x2025c3 27e1e290 792c143e 44194464 00000000 00000016 mscorlib_ni+0x201951 27e1e2ac 79e71e04 00000001 27e1e334 79faa569 mscorlib_ni+0x20143e 27e1e2b8 79faa569 792c1400 072336fc 00000000 mscorwks!CTPMethodTable__CallTargetHelper3+0xf 27e1e334 79faa5d8 072336fc 00000000 00000001 mscorwks!CTPMethodTable::CallTarget+0xdd 27e1e348 79faa614 790fd65c 072336fc 00000000 mscorwks!CTPMethodTable::CallTarget+0x14 27e1e368 7a04b578 1e772994 00000000 00000001 mscorwks!CRemotingServices::CreateProxyOrObject+0x38 27e1e414 1e923c99 0f5b1a58 0f5b1b1c 27e1e464 mscorwks!JIT_NewCrossContextHelper+0xa9 WARNING: Frame IP not in any known module. Following frames may be wrong. 27e1e430 776e2fae 00000000 00000064 00000001 0x1e923c99 27e1e464 79e71b4c 27e1ef70 00000064 00000001 ole32!CoGetContextToken+0x29 27e1e494 79e821b9 27e1ee70 00000008 27e1ee20 mscorwks!CallDescrWorker+0x33 27e1e514 7a0f851b 27e1ee70 00000008 27e1ee20 mscorwks!CallDescrWorkerWithHandler+0xa3 27e1e5e4 79270454 00000001 00000000 00000000 mscorwks!CMessage::Dispatch+0x162 27e1e63c 7977c16e 00000001 00000000 075d788c mscorlib_ni+0x1b0454 27e1e658 6744d146 0752309c 0f5b19d8 0752304c mscorlib_ni+0x6bc16e 27e1e690 197cb7c7 0d0e3640 79e7a6b8 27e1ea80 System_EnterpriseServices_ni+0x5d146 27e1e70c 197f3d04 27e1eae4 0071f13b 36c6b460 System_EnterpriseServices_Wrapper!__dyn_tls_init_callback (System_EnterpriseServices_Wrapper+0x2b7c7) 27e1e74c 77720df0 27e1eae4 35ccbed0 00000000 System_EnterpriseServices_Wrapper_197f0000!System::EnterpriseServices::Thunk::FilteringCallbackFunction+0x44 27e1e798 7772189c 00000000 35ccbed0 197f3cc0 ole32!EnterForCallback+0xc4 27e1e8f8 776f0418 27e1e7d0 197f3cc0 27e1eae4 ole32!SwitchForCallback+0x1a3 27e1e924 7769c194 35ccbed0 197f3cc0 27e1eae4 ole32!PerformCallback+0x54 27e1e9bc 776e316c 36c6b460 197f3cc0 27e1eae4 ole32!CObjectContext::InternalContextCallback+0x159 27e1ea0c 79e71d8b 36c6b470 197f3cc0 27e1eae4 ole32!CObjectContext::ContextCallback+0x85 27e1ea68 197cbc82 776e30e7 0f5b1a28 072323f0 mscorwks!PInvokeCalliReturnFromCall 27e1ea88 197cbc82 0010c8f0 00000000 00000008 System_EnterpriseServices_Wrapper!__dyn_tls_init_callback (System_EnterpriseServices_Wrapper+0x2bc82) 27e1eab4 197cbb51 07231a74 27e1eb2c 79282f85 System_EnterpriseServices_Wrapper!__dyn_tls_init_callback (System_EnterpriseServices_Wrapper+0x2bc82) 27e1eac0 79282f85 27e1eb44 00000000 00000000 System_EnterpriseServices_Wrapper!__dyn_tls_init_callback (System_EnterpriseServices_Wrapper+0x2bb51) 27e1eb2c 797f5a0d 03100788 072323f0 075d7830 mscorlib_ni+0x1c2f85 00000000 00000000 00000000 00000000 00000000 mscorlib_ni+0x735a0d And all other threads waiting for thread 142 have same stack trace. I don't know what thread 141 is doing. Thread 142 cannot open connection to xls file. And not sure why other threads wait for thread 142. Maybe you have any ideas? EDIT: analyze -v -hang GetPageUrlData failed, server returned HTTP status 404 URL requested: http://watson.microsoft.com/StageOne/dllhst3g_exe/5_2_3790_3959/unknown/0_0_0_0/00000000.htm?Retriage=1 FAULTING_IP: +1e02faf0157df58 00000000 ?? ??? EXCEPTION_RECORD: ffffffff -- (.exr 0xffffffffffffffff) ExceptionAddress: 00000000 ExceptionCode: 80000003 (Break instruction exception) ExceptionFlags: 00000000 NumberParameters: 0 FAULTING_THREAD: 0000008d BUGCHECK_STR: HANG PROCESS_NAME: dllhst3g.exe ERROR_CODE: (NTSTATUS) 0xcfffffff - EXCEPTION_CODE: (NTSTATUS) 0xcfffffff - MOD_LIST: NTGLOBALFLAG: 0 APPLICATION_VERIFIER_FLAGS: 0 MANAGED_STACK: !dumpstack -EE OS Thread Id: 0xc2c (17) Current frame: ChildEBP RetAddr Caller,Callee 1974f450 655f1110 (MethodDesc 0x651b7efc +0x30 System.Data.ProviderBase.WrappedIUnknown.ReleaseHandle()) 1974f93c 792e5e4f (MethodDesc 0x79107064 +0xf System.Runtime.InteropServices.SafeHandle.Dispose(Boolean)) 1974f944 792e5d6b (MethodDesc 0x79107030 +0x1b System.Runtime.InteropServices.SafeHandle.Finalize()) DERIVED_WAIT_CHAIN: Dl Eid Cid WaitType 0 1d14.71c Speculated (Triage) --> 17 1d14.c2c Critical Section --> 141 1d14.1b38 Event WAIT_CHAIN_COMMAND: ~0s;k;;~17s;k;;~141s;k;; BLOCKING_THREAD: 00001b38 DEFAULT_BUCKET_ID: APPLICATION_HANG_WRONG_SYMBOLS PRIMARY_PROBLEM_CLASS: APPLICATION_HANG_WRONG_SYMBOLS LAST_CONTROL_TRANSFER: from 7c827b89 to 7c82847c STACK_TEXT: 2a2efdcc 7c827b89 77e6202c 00000003 2a2efe1c ntdll!KiFastSystemCallRet 2a2efdd0 77e6202c 00000003 2a2efe1c 00000001 ntdll!NtWaitForMultipleObjects+0xc 2a2efe78 7739bbd1 00000003 2a2efea0 00000000 kernel32!WaitForMultipleObjectsEx+0x11a 2a2efed4 7739ce36 00000002 2a2eff74 ffffffff user32!RealMsgWaitForMultipleObjectsEx+0x141 2a2efef0 4a77cb28 00000002 2a2eff74 00000000 user32!MsgWaitForMultipleObjects+0x1f 2a2eff84 77bcb530 33c23fe8 00000000 00000000 comsvcs!CSTAThread::WorkerLoop+0x1f9 2a2effb8 77e6482f 37e3b7e8 00000000 00000000 msvcrt!_endthreadex+0xa3 2a2effec 00000000 77bcb4bc 37e3b7e8 00000000 kernel32!BaseThreadStart+0x34 FOLLOWUP_IP: ntdll!KiFastSystemCallRet+0 7c82847c c3 ret SYMBOL_STACK_INDEX: 0 SYMBOL_NAME: ntdll!KiFastSystemCallRet+0 FOLLOWUP_NAME: MachineOwner MODULE_NAME: ntdll IMAGE_NAME: ntdll.dll DEBUG_FLR_IMAGE_TIMESTAMP: 4cc18322 STACK_COMMAND: ~141s ; kb BUCKET_ID: HANG_ntdll!KiFastSystemCallRet+0 FAILURE_BUCKET_ID: APPLICATION_HANG_WRONG_SYMBOLS_cfffffff_ntdll.dll!KiFastSystemCallRet WATSON_STAGEONE_URL: http://watson.microsoft.com/StageOne/dllhst3g_exe/5_2_3790_3959/45d69678/unknown/0_0_0_0/bbbbbbb4/cfffffff/00000000.htm?Retriage=1 Followup: MachineOwner Not sure I understand it. But thread 17 is finalizer thread. So looks like it is blocked? And I thinkg it as again some kind of access to xls? ~17s kb ChildEBP RetAddr Args to Child 1974f08c 7c827b99 7c83d09c 00003a98 00000000 ntdll!KiFastSystemCallRet 1974f090 7c83d09c 00003a98 00000000 00000000 ntdll!ZwWaitForSingleObject+0xc 1974f0cc 7c83d0e7 00003a98 00000004 00000001 ntdll!RtlpWaitOnCriticalSection+0x1a3 1974f0ec 2ae1fa94 29202124 32787727 00000000 ntdll!RtlEnterCriticalSection+0xa8 WARNING: Stack unwind information not available. Following frames may be wrong. 1974f160 4c856487 2263eff4 00000000 00000005 ACEOLEDB!DllGetClassObject+0xdc2d 1974f174 4c862d06 2263eff4 4c8633c4 2263efe8 oledb32!CACMDynamic::CmFinalRelease+0x50 1974f17c 4c8633c4 2263efe8 2263efe8 1974f1e4 oledb32!CDCM::FinalRelease+0x1b 1974f198 4c8633f6 00000000 2263efe8 1974f1b8 oledb32!ATL::CComPolyObject::~CComPolyObject+0x2a 1974f1a8 4c88d5ad 2263efe8 2370a39c 1974f1f4 oledb32!ATL::CComPolyObject::Release+0x25 1974f1b8 4a757681 1a271fa0 2263efe8 2370a39c oledb32!CDCMCreator::DestroyResource+0xf 1974f1f4 4a75793e 2370a39c 2263efe8 00000000 comsvcs!CHolder::SafeDispenserDriver::DestroyResource+0x20 1974f20c 4a758a35 23151cd0 00000000 2fa44808 comsvcs!CHolder::ProcessDestroyList+0x2e 1974f238 4c88d30e 23151cd0 2263efe8 2263efe8 comsvcs!CHolder::FreeResource+0x7f 1974f268 4c878a3a 2263efe8 00000000 271aa824 oledb32!CDCMCreator::ReleaseResource+0x31 1974f288 4c8545e7 271aa824 00000000 4c85456a oledb32!CDPO::ReturnDCMToPool+0x89 1974f294 4c85456a 271aa818 271aa818 1974f320 oledb32!CDPO::FinalRelease+0xb 1974f2b0 4c88e78a 271aa818 1974f2d0 4c85218c oledb32!ATL::CComPolyObject::~CComPolyObject+0x2a 1974f2bc 4c85218c 00000001 00000000 000eaaf8 oledb32!ATL::CComPolyObject::`scalar deleting destructor'+0xd 1974f2d0 4c8521aa 271aa818 1974f330 79f63ff9 oledb32!ATL::CComPolyObject::Release+0x27 1974f2dc 79f63ff9 271aa824 3b6a3777 000eaaf8 oledb32!ATL::CComContainedObject::Release+0x11 1974f330 79f640ac 271aa828 3b6a3733 000eaaf8 mscorwks!ReleaseTransitionHelper+0x5f 1974f374 79f64110 271aa828 00000000 3b6a37ef mscorwks!SafeReleaseHelper+0x8c 1974f3a8 79f53b5d 271aa828 00000000 3b6a3017 mscorwks!SafeRelease+0x2f 1974f450 655f1110 000eaaf8 00000000 1974f470 mscorwks!MarshalNative::Release+0xb0 1974f460 79e71b4c 1974f4dc 000c5958 1974f4f0 System_Data_ni!_bidW103 (System_Data_ni+0x4a1110) 1974f470 79e821b9 1974f540 00000000 1974f510 mscorwks!CallDescrWorker+0x33 1974f4f0 79e96531 1974f540 00000000 1974f510 mscorwks!CallDescrWorkerWithHandler+0xa3 1974f634 79e96564 655f10e0 1974f6bc 1974f6a4 mscorwks!MethodDesc::CallDescr+0x19c 1974f650 79e96c4c 655f10e0 1974f6bc 1974f6a4 mscorwks!MethodDesc::CallTargetWorker+0x1f 1974f670 79eccd2a 1974f6a4 0b402780 00000004 mscorwks!MethodDescCallSite::Call_RetObjPtr+0x1c 1974f770 79eccd5e 0b402780 000eaaf8 000eaaf8 mscorwks!SafeHandle::RunReleaseMethod+0x89 1974f870 79eccbf4 00000001 0d0e3640 79e7a1c8 mscorwks!SafeHandle::Release+0x11b 1974f89c 79f83999 3b6a3d7b 00000000 000eaaf8 mscorwks!SafeHandle::Dispose+0x23 1974f93c 792e5e4f 1974f970 792e5d6b 0b3ac100 mscorwks!SafeHandle::Finalize+0xab 1974f944 792e5d6b 0b3ac100 ffffffff 00000000 mscorlib_ni+0x225e4f 1974f970 79f7169a 1974f9c4 79ef465c 1974fc1c mscorlib_ni+0x225d6b 1974f9d4 79f7175b 0b402780 792e5d50 08000000 mscorwks!MethodTable::GetObjCreateDelegate+0xaf 1974f9f4 79f71609 0b402780 0b402780 00000000 mscorwks!MethodTable::CallFinalizer+0xa0 1974fa08 79fd46a6 0b402780 3b6a3e1f 00000000 mscorwks!SVR::CallFinalizer+0xa6 1974fa58 79fd45d7 000eaaf8 00000415 1974fadf mscorwks!SVR::GCHeap::TraceGCSegments+0x1b0 1974fae0 79f5832f 0740eeb4 00000000 1974feb0 mscorwks!SVR::GCHeap::TraceGCSegments+0x2f6 1974faf4 79e9848f 1974fd04 000eaaf8 00000000 mscorwks!SVR::ProfScanRootsHelper+0x69 1974fb08 79e9842b 1974feb0 1974fb90 79fa6a6b mscorwks!Thread::DoADCallBack+0x32a 1974fb9c 79e98351 1974feb0 3b6a3f9f 000eaaf8 mscorwks!Thread::ShouldChangeAbortToUnload+0xe3 1974fbd8 79ec4322 1974feb0 00000000 1974fc98 mscorwks!Thread::ShouldChangeAbortToUnload+0x30a 1974fbe8 79f581f3 1974feb0 1974fc8c 79fa6a6b mscorwks!Thread::RaiseCrossContextException+0x434 1974fc98 79f58279 1a9cb678 79ec430e 1974feb0 mscorwks!Thread::DoADCallBack+0xcd 1974fcb8 79f58265 1974feb0 1974fd20 79f582e4 mscorwks!Thread::DoADCallBack+0x322 1974fcc4 79f582e4 1a9cb678 79f58316 1974fd04 mscorwks!ManagedThreadBase::FinalizerAppDomain+0x25 1974fd20 79fd45d7 000eaaf8 00000000 1974fda7 mscorwks!SVR::GCHeap::TraceGCSegments+0x251 1974fda8 79fd48a8 00000000 00000000 1974feb0 mscorwks!SVR::GCHeap::TraceGCSegments+0x2f6 1974fdc0 79e9848f 1974feb0 00000000 00000000 mscorwks!SVR::GCHeap::FinalizerThreadWorker+0xb7 1974fdd4 79e9842b 1974feb0 1974fe5c 79fa6a6b mscorwks!Thread::DoADCallBack+0x32a 1974fe68 79e98351 1974feb0 3b6a3ae3 00000000 mscorwks!Thread::ShouldChangeAbortToUnload+0xe3 1974fea4 79f074d4 1974feb0 00000000 1a9cb678 mscorwks!Thread::ShouldChangeAbortToUnload+0x30a 1974fecc 79f074e5 79fd4809 00000008 1974ff14 mscorwks!ManagedThreadBase_NoADTransition+0x32 1974fedc 79f090b3 79fd4809 3b6a3b53 00000000 mscorwks!ManagedThreadBase::FinalizerBase+0xd 1974ff14 79f75715 00000000 00000007 ffffffff mscorwks!SVR::GCHeap::FinalizerThreadStart+0xbb 1974ffb8 77e6482f 000eb528 00000000 00000000 mscorwks!Thread::intermediateThreadProc+0x49 1974ffec 00000000 79f756cf 000eb528 00000000 kernel32!BaseThreadStart+0x34
You need to find who owns the following critical section that your thread is waiting on: 1974f0ec 2ae1fa94 29202124 32787727 00000000 ntdll!RtlEnterCriticalSection+0xa8 You can do an automated critical section analysis with call stacks: !locks -v This will dump all critical section locks that are in a locked state and the call stacks of the threads, you then need to scan each call stack for each lock to see if say Thread A is waiting on lock 1 which is owned by Thread B, Thread B is waiting on lock 2 which is owned by Thread A. Hope this helps
Emulator won't load 2.2 but will load 2.3
On a Windows XP system, using Eclipse 3.5, JDK 1.6.24, and the Android R10 SDK. Like a number of others, I'm unable to get the emulator to load the home page, but this is happening only for the 2.2 image. I am able to get the emulator fully loaded and running with 2.3. I've tried waiting for a long period of time (I think 2 hours is more than enough), killing and restarting the adb server, etc. The emulator is stuck in a reboot loop caused by the serverthread throwing a SIGSTKFLT signal. This is a signal that's not actually used by current versions of Linux but is sometimes thrown by app's to signal an application error. Unfortunately, there's no specific error message (at least that I can find) to indicate why the signal is being thrown. The pertinent part of the logcat is: I/dalvikvm( 54): "android.server.ServerThread" prio=5 tid=7 RUNNABLE I/dalvikvm( 54): | group="main" sCount=0 dsCount=0 s=N obj=0x43e31da0 self=0x11c9a8 I/dalvikvm( 54): | sysTid=69 nice=-2 sched=0/0 cgrp=default handle=1139800 I/dalvikvm( 54): | schedstat=( 62983168741 24276655402 1105 ) I/dalvikvm( 54): at java.lang.String.<init>(String.java:~468) I/dalvikvm( 54): at java.util.jar.InitManifest.readValue(InitManifest.java:205) I/dalvikvm( 54): at java.util.jar.InitManifest.readHeader(InitManifest.java:117) I/dalvikvm( 54): at java.util.jar.InitManifest.initEntries(InitManifest.java:75) I/dalvikvm( 54): at java.util.jar.JarVerifier.verifyCertificate(JarVerifier.java:329) I/dalvikvm( 54): at java.util.jar.JarVerifier.readCertificates(JarVerifier.java:272) I/dalvikvm( 54): at java.util.jar.JarFile.getInputStream(JarFile.java:392) I/dalvikvm( 54): at android.content.pm.PackageParser.loadCertificates(PackageParser.java:337) I/dalvikvm( 54): at android.content.pm.PackageParser.collectCertificates(PackageParser.java:480) I/dalvikvm( 54): at com.android.server.PackageManagerService.collectCertificatesLI(PackageManagerService.java:2535) I/dalvikvm( 54): at com.android.server.PackageManagerService.scanPackageLI(PackageManagerService.java:2621) I/dalvikvm( 54): at com.android.server.PackageManagerService.scanDirLI(PackageManagerService.java:2479) I/dalvikvm( 54): at com.android.server.PackageManagerService.<init>(PackageManagerService.java:920) I/dalvikvm( 54): at com.android.server.PackageManagerService.main(PackageManagerService.java:681) I/dalvikvm( 54): at com.android.server.ServerThread.run(SystemServer.java:122) I/dalvikvm( 54): I/dalvikvm( 54): "Binder Thread #2" prio=5 tid=6 NATIVE I/dalvikvm( 54): | group="main" sCount=1 dsCount=0 s=N obj=0x43e31a50 self=0x12fc60 I/dalvikvm( 54): | sysTid=60 nice=0 sched=0/0 cgrp=default handle=1294704 I/dalvikvm( 54): | schedstat=( 1000041725 990947204 173 ) I/dalvikvm( 54): at dalvik.system.NativeStart.run(Native Method) I/dalvikvm( 54): I/dalvikvm( 54): "Binder Thread #1" prio=5 tid=5 NATIVE I/dalvikvm( 54): | group="main" sCount=1 dsCount=0 s=N obj=0x43e2f3b8 self=0x11ce48 I/dalvikvm( 54): | sysTid=59 nice=0 sched=0/0 cgrp=default handle=1165432 I/dalvikvm( 54): | schedstat=( 991350956 356395337 175 ) I/dalvikvm( 54): at dalvik.system.NativeStart.run(Native Method) I/dalvikvm( 54): I/dalvikvm( 54): "JDWP" daemon prio=5 tid=4 VMWAIT I/dalvikvm( 54): | group="system" sCount=1 dsCount=0 s=N obj=0x43e2e2a0 self=0x11c5e0 I/dalvikvm( 54): | sysTid=58 nice=0 sched=0/0 cgrp=default handle=1146120 I/dalvikvm( 54): | schedstat=( 9625246 195868748 7 ) I/dalvikvm( 54): at dalvik.system.NativeStart.run(Native Method) I/dalvikvm( 54): I/dalvikvm( 54): "Signal Catcher" daemon prio=5 tid=3 VMWAIT I/dalvikvm( 54): | group="system" sCount=1 dsCount=0 s=N obj=0x43e2e1e8 self=0x117b18 I/dalvikvm( 54): | sysTid=57 nice=0 sched=0/0 cgrp=default handle=1145544 I/dalvikvm( 54): | schedstat=( 4465371 1466668 2 ) I/dalvikvm( 54): at dalvik.system.NativeStart.run(Native Method) I/dalvikvm( 54): I/dalvikvm( 54): "HeapWorker" daemon prio=5 tid=2 VMWAIT I/dalvikvm( 54): | group="system" sCount=1 dsCount=0 s=N obj=0x4306e1e8 self=0x1178d8 I/dalvikvm( 54): | sysTid=55 nice=0 sched=0/0 cgrp=default handle=1232848 I/dalvikvm( 54): | schedstat=( 3618171309 20811992594 393 ) I/dalvikvm( 54): at com.android.internal.os.BinderInternal$GcWatcher.finalize(BinderInternal.java:~48) I/dalvikvm( 54): at dalvik.system.NativeStart.run(Native Method) I/dalvikvm( 54): D/dalvikvm( 54): threadid=7: sending two SIGSTKFLTs to threadid=2 (tid=55) to cause debuggerd dump W/SharedBufferStack( 68): waitForCondition(DequeueCondition) timed out (identity=0, status=0). CPU may be pegged. trying again. I/DEBUG ( 31): *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** D/dalvikvm( 54): Sent, pausing to let debuggerd run I/DEBUG ( 31): Build fingerprint: 'generic/sdk/generic/:2.2/FRF91/43546:eng/test-keys' I/DEBUG ( 31): pid: 54, tid: 55 >>> system_server <<< I/DEBUG ( 31): signal 16 (SIGSTKFLT), fault addr 00000036 I/DEBUG ( 31): r0 fffffe00 r1 00000080 r2 00000002 r3 00000000 I/DEBUG ( 31): r4 80888fe8 r5 00000002 r6 00000000 r7 000000f0 I/DEBUG ( 31): r8 00000001 r9 400091e8 10 80888c90 fp 001361b8 I/DEBUG ( 31): ip 80888098 sp 100ffcd8 lr afd10530 pc afd0eb08 cpsr 80000010 I/DEBUG ( 31): #00 pc 0000eb08 /system/lib/libc.so I/DEBUG ( 31): #01 pc 0001052c /system/lib/libc.so I/DEBUG ( 31): #02 pc 0001b77c /system/lib/libdvm.so I/DEBUG ( 31): #03 pc 0001bc20 /system/lib/libdvm.so I/DEBUG ( 31): #04 pc 00048ece /system/lib/libdvm.so I/DEBUG ( 31): #05 pc 000143ac /system/lib/libdvm.so I/DEBUG ( 31): #06 pc 0001e8c4 /system/lib/libdvm.so I/DEBUG ( 31): #07 pc 0001d790 /system/lib/libdvm.so I/DEBUG ( 31): #08 pc 00053eec /system/lib/libdvm.so I/DEBUG ( 31): #09 pc 00054102 /system/lib/libdvm.so I/DEBUG ( 31): #10 pc 0004a550 /system/lib/libdvm.so I/DEBUG ( 31): #11 pc 0004a5ea /system/lib/libdvm.so I/DEBUG ( 31): #12 pc 0004aada /system/lib/libdvm.so I/DEBUG ( 31): #13 pc 00048174 /system/lib/libdvm.so I/DEBUG ( 31): #14 pc 0001103c /system/lib/libc.so I/DEBUG ( 31): #15 pc 00010b20 /system/lib/libc.so I/DEBUG ( 31): I/DEBUG ( 31): code around pc: I/DEBUG ( 31): afd0eae8 0afffff7 e8bd4010 e12fff1e e1010090 I/DEBUG ( 31): afd0eaf8 e12fff1e e92d0090 e3a070f0 ef000000 I/DEBUG ( 31): afd0eb08 e8bd0090 e12fff1e e92d0090 e1a03002 I/DEBUG ( 31): afd0eb18 e1a02001 e3a01000 e3a070f0 ef000000 I/DEBUG ( 31): afd0eb28 e8bd0090 e12fff1e e92d0090 e1a02001 I/DEBUG ( 31): I/DEBUG ( 31): code around lr: I/DEBUG ( 31): afd10510 e3500000 0a00000e e3560000 03a07080 I/DEBUG ( 31): afd10520 13a07000 e3865002 ea000000 ebfff986 I/DEBUG ( 31): afd10530 e1a01004 e1a00005 ebfff96d e1560000 I/DEBUG ( 31): afd10540 e1a01007 e1a00004 e1a02005 e3a03000 I/DEBUG ( 31): afd10550 1afffff5 e3a00000 e8bd87f0 e3a00016 I/DEBUG ( 31): I/DEBUG ( 31): stack: I/DEBUG ( 31): 100ffc98 100ffcf8 I/DEBUG ( 31): 100ffc9c 00000003 I/DEBUG ( 31): 100ffca0 423bbb71 /data/dalvik-cache/system#framework#framework.jar#classes.dex I/DEBUG ( 31): 100ffca4 4103afa8 I/DEBUG ( 31): 100ffca8 100ffcd8 I/DEBUG ( 31): 100ffcac 4103afa0 I/DEBUG ( 31): 100ffcb0 4103af8c I/DEBUG ( 31): 100ffcb4 afd34300 /system/lib/libc.so I/DEBUG ( 31): 100ffcb8 100ffcc0 I/DEBUG ( 31): 100ffcbc a811ad81 /system/lib/libutils.so I/DEBUG ( 31): 100ffcc0 000497bc [heap] I/DEBUG ( 31): 100ffcc4 00000000 I/DEBUG ( 31): 100ffcc8 100ffcf8 I/DEBUG ( 31): 100ffccc ad33e4ef /system/lib/libandroid_runtime.so I/DEBUG ( 31): 100ffcd0 df002777 I/DEBUG ( 31): 100ffcd4 e3a070ad I/DEBUG ( 31): #00 100ffcd8 80888fe8 /system/lib/libdvm.so I/DEBUG ( 31): 100ffcdc 00000080 I/DEBUG ( 31): #01 100ffce0 80888fe8 /system/lib/libdvm.so I/DEBUG ( 31): 100ffce4 00000001 I/DEBUG ( 31): 100ffce8 001178d8 [heap] I/DEBUG ( 31): 100ffcec 00000022 I/DEBUG ( 31): 100ffcf0 00000018 I/DEBUG ( 31): 100ffcf4 400091e8 /dev/ashmem/mspace/dalvik-heap/zygote/0 (deleted) I/DEBUG ( 31): 100ffcf8 80888c90 /system/lib/libdvm.so I/DEBUG ( 31): 100ffcfc 8081b780 /system/lib/libdvm.so D/dalvikvm( 54): Continuing E/dalvikvm( 54): VM aborting I/ServiceManager( 28): service 'batteryinfo' died I/ServiceManager( 28): service 'entropy' died I/ServiceManager( 28): service 'SurfaceFlinger' died I/ServiceManager( 28): service 'usagestats' died I/ServiceManager( 28): service 'power' died I/ServiceManager( 28): service 'telephony.registry' died D/BootAnimation( 68): SurfaceFlinger died, exiting... I/DEBUG ( 31): debuggerd committing suicide to free the zombie! D/Zygote ( 33): Process 54 terminated by signal (11) I/Zygote ( 33): Exit zygote because system server (54) has terminated E/installd( 35): eof E/installd( 35): failed to read size I/installd( 35): closing connection I/ServiceManager( 28): service 'media.audio_flinger' died I/ServiceManager( 28): service 'media.player' died I/ServiceManager( 28): service 'media.camera' died I/ServiceManager( 28): service 'media.audio_policy' died I/ ( 80): ServiceManager: 0xacd0 D/AudioHardwareInterface( 80): setMode(NORMAL) I/DEBUG ( 81): debuggerd: Jun 30 2010 13:59:20 D/AndroidRuntime( 82): D/AndroidRuntime( 82): >>>>>>>>>>>>>> AndroidRuntime START <<<<<<<<<<<<<< D/AndroidRuntime( 82): CheckJNI is ON D/AndroidRuntime( 82): --- registering native functions --- And then the whole thing runs through in an infinite loop of start/fault/dump/restart. I don't think this is a showstopper for me as I can target in 2.2 and run 2.3 in the emulator till it's time to load on a real 2.2 device. But it's something of an annoyance. UPDATE I was able to partially load froyo outside of eclipse using the command line. Starting it with "emulator #froyo -no-boot-anim -logcat '*:v'" it loaded as far as showing the top bar with the battery indicator. Omitting the logcat switch it started as far as showing the background before crashing. This looks more and more like a timing issue. This is also now just happening on a laptop. When installed on one of my desktops Froyo comes up and is working. Both have the same speed CPU, but the laptop is a P4 2.2ghz with 1G memory and the desktop is a dual Xeon 2.2 with 2Gb memory. Both have similar application and service loads.