JBoss / HotSpot JVM crashing - jboss

We have a web app built on Jboss that has been in production for several years; for the last 18 months it has never gone down. However, in the last two days, the jvm has crashed 4 times.
When the JVM crashes, we get a error report that I am struggling to decipher. I have looked at several other crash reports (on this site and others), but none are exactly like ours; and because I'm a novice at reading them, I can't tell what I'm missing.
[Edited crash reports to add more information]
I had four crash reports...I will post 2 of them (with the libraries excepted for brevity) in case that helps diagnose the issue. None of the 4 look exactly the same. I am only posting 2 because there is a 30000 char limit on the body.
CRASH REPORT #1:
#
# An unexpected error has been detected by HotSpot Virtual Machine:
#
# SIGSEGV (0xb) at pc=0x0000002a, pid=7674, tid=2159016848
#
# Java VM: Java HotSpot(TM) Server VM (1.5.0_20-b02 mixed mode)
# Problematic frame:
# C 0x0000002a
#
--------------- T H R E A D ---------------
Current thread (0x80bbff08): VMThread [id=7679]
siginfo:si_signo=11, si_errno=0, si_code=1, si_addr=0x0000002a
Registers:
EAX=0x0c0c0c0c, EBX=0xb7b35810, ECX=0x8130b9c8, EDX=0x9df81dd0
ESP=0x80afebbc, EBP=0x80afebd8, ESI=0xb7b26780, EDI=0xb7b44c2c
EIP=0x0000002a, CR2=0x0000002a, EFLAGS=0x00010206
Top of Stack: (sp=0x80afebbc)
0x80afebbc: b796e7e2 8130b9c8 9df81dd0 b7b33d00
0x80afebcc: b796e7aa b7b35810 b7b44c2c 80afec68
0x80afebdc: b79e0b6e b7b44c2c 00000000 80b2f980
0x80afebec: 80afec08 006d9b7b 007c4120 00028098
0x80afebfc: 80afec10 00028098 80b2f980 80afec38
0x80afec0c: b79a167a 00000000 00000001 00000001
0x80afec1c: 00000000 00000000 000aa873 000f4240
0x80afec2c: 00000000 80b2f980 09a07ea0 80afec68
Instructions: (pc=0x0000002a)
0x0000001a:
[error occurred during error reporting, step 100, id 0xb]
Stack: [0x80a7f000,0x80b00000), sp=0x80afebbc, free space=510k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
C 0x0000002a
V [libjvm.so+0x474b6e]
V [libjvm.so+0x473e9f]
V [libjvm.so+0x473b07]
V [libjvm.so+0x51fd4a]
V [libjvm.so+0x51f236]
V [libjvm.so+0x51e847]
V [libjvm.so+0x51eab5]
V [libjvm.so+0x51e620]
V [libjvm.so+0x43bc78]
C [libpthread.so.0+0x550b]
VM_Operation (0x7a47dbe0): parallel gc system gc, mode: safepoint, requested by thread 0x7ab234d8
**[processes excerpted for brevity]**
Other Threads:
=>0x80bbff08 VMThread [id=7679]
0x80bd18a0 WatcherThread [id=7687]
VM state:at safepoint (normal execution)
VM Mutex/Monitor currently owned by a thread: ([mutex/lock_event])
[0x09a0a6a0/0x09a0a6c8] Threads_lock - owner thread: 0x80bbff08
[0x09a0aaf0/0x09a0ab08] Heap_lock - owner thread: 0x7ab234d8
Heap
PSYoungGen total 54464K, used 398K [0xad860000, 0xb1060000, 0xb1140000)
eden space 54016K, 0% used [0xad860000,0xad860000,0xb0d20000)
from space 448K, 88% used [0xb0d20000,0xb0d83a40,0xb0d90000)
to space 1664K, 0% used [0xb0ec0000,0xb0ec0000,0xb1060000)
PSOldGen total 466048K, used 245327K [0x91140000, 0xad860000, 0xad860000)
object space 466048K, 52% used [0x91140000,0xa00d3d20,0xad860000)
PSPermGen total 107776K, used 67420K [0x81140000, 0x87a80000, 0x91140000)
object space 107776K, 62% used [0x81140000,0x853173f8,0x87a80000)
**[dynamic libraries exceprted]**
VM Arguments:
jvm_args: -Dprogram.name=run.sh -Xms128m -Xmx512m -XX:MaxPermSize=256m -Dorg.jboss.resolver.warning=true -Dsun.rmi.dgc.client.gcInterval=3600000 -Dsun.rmi.dgc.server.gcInterval=3600000 -Djava.net.preferIPv4Stack=true -Djava.endorsed.dirs=/opt/jboss/jboss-5.1.0.GA/lib/endorsed
java_command: org.jboss.Main -c livingdiary
Launcher Type: SUN_STANDARD
Environment Variables:
JAVA_HOME=/usr/java/jdk
JRE_HOME=/usr/java/jdk/jre
PATH=/usr/kerberos/bin:/usr/bin:/bin:/usr/sbin:/sbin:/home/jboss/bin:/opt/jboss/jboss-5.1.0.GA/bin
LD_LIBRARY_PATH=/usr/java/jdk1.5.0_20/jre/lib/i386/server:/usr/java/jdk1.5.0_20/jre/lib/i386:/usr/java/jdk1.5.0_20/jre/../lib/i386
SHELL=/bin/bash
Signal Handlers:
SIGSEGV: [libjvm.so+0x517270], sa_mask[0]=0x7ffbfeff, sa_flags=0x10000004
SIGBUS: [libjvm.so+0x517270], sa_mask[0]=0x7ffbfeff, sa_flags=0x10000004
SIGFPE: [libjvm.so+0x43a0a0], sa_mask[0]=0x7ffbfeff, sa_flags=0x10000004
SIGPIPE: [libjvm.so+0x43a0a0], sa_mask[0]=0x7ffbfeff, sa_flags=0x10000004
SIGILL: [libjvm.so+0x43a0a0], sa_mask[0]=0x7ffbfeff, sa_flags=0x10000004
SIGUSR1: SIG_DFL, sa_mask[0]=0x00000000, sa_flags=0x00000000
SIGUSR2: [libjvm.so+0x43c4f0], sa_mask[0]=0x00000004, sa_flags=0x10000004
SIGHUP: SIG_IGN, sa_mask[0]=0x00000000, sa_flags=0x00000000
SIGINT: SIG_IGN, sa_mask[0]=0x00000000, sa_flags=0x00000000
SIGQUIT: [libjvm.so+0x43bf20], sa_mask[0]=0x7ffbfeff, sa_flags=0x10000004
SIGTERM: [libjvm.so+0x43bf20], sa_mask[0]=0x7ffbfeff, sa_flags=0x10000004
--------------- S Y S T E M ---------------
OS:Fedora release 8 (Werewolf)
uname:Linux 2.6.26.8-57.fc8 #1 SMP Thu Dec 18 19:19:45 EST 2008 i686
libc:glibc 2.7 NPTL 2.7
rlimit: STACK 10240k, CORE 0k, NPROC 65536, NOFILE 1024, AS infinity
load average:0.06 0.05 0.00
CPU:total 4 (cores per cpu 4, threads per core 1) family 6 model 7 stepping 7, cmov, cx8, fxsr, mmx, sse, sse2
Memory: 4k page, physical 3358636k(327528k free), swap 511k(511k free)
vm_info: Java HotSpot(TM) Server VM (1.5.0_20-b02) for linux-x86, built on Jun 18 2009 12:08:29 by java_re with gcc 3.2.1-7a (J2SE release)
time: Fri May 13 01:49:23 2011
elapsed time: 5968 seconds
CRASH REPORT #2
#
# An unexpected error has been detected by HotSpot Virtual Machine:
#
# SIGSEGV (0xb) at pc=0x006d7426, pid=9790, tid=1949301648
#
# Java VM: Java HotSpot(TM) Server VM (1.5.0_20-b02 mixed mode)
# Problematic frame:
# C [libc.so.6+0x69426]
#
--------------- T H R E A D ---------------
Current thread (0x7859f560): JavaThread "HDScanner" [_thread_in_native, id=9828]
siginfo:si_signo=11, si_errno=0, si_code=1, si_addr=0x0000000c
Registers:
EAX=0x98ec2e90, EBX=0x007c2ff4, ECX=0x00000001, EDX=0x00000000
ESP=0x742fe718, EBP=0x742fe780, ESI=0x09ba30f0, EDI=0x09ba3150
EIP=0x006d7426, CR2=0x0000000c, EFLAGS=0x00210246
Top of Stack: (sp=0x742fe718)
0x742fe718: 00000050 00000000 0000000d 742fe650
0x742fe728: 742fe790 b77fbd91 09ba30f8 00000060
0x742fe738: 02400010 0079893c 7859f560 00000001
0x742fe748: 00000000 b77e027a 7859f560 007c4120
0x742fe758: 0000004c 742fe778 006d9b7b 007c4120
0x742fe768: 0000004d b754a058 7859f620 007c2ff4
0x742fe778: 007c4120 09ba30f8 742fe7b8 006db0f0
0x742fe788: 007c4120 09ba30f8 09ba30f8 b754a058
Instructions: (pc=0x006d7426)
0x006d7416: 44 17 04 01 0f 85 05 03 00 00 8b 57 08 8b 47 0c
0x006d7426: 39 7a 0c 0f 84 80 05 00 00 8b 83 d0 00 00 00 89
Stack: [0x7427f000,0x74300000), sp=0x742fe718, free space=509k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
C [libc.so.6+0x69426]
C [libc.so.6+0x6d0f0] cfree+0x90
C [libjava.so+0x1162f] JNU_ReleaseStringPlatformChars+0x1f
C [libjava.so+0x15ae8] Java_java_io_UnixFileSystem_getBooleanAttributes0+0xc8
Java frames: (J=compiled Java code, j=interpreted, Vv=VM code)
J java.io.UnixFileSystem.getBooleanAttributes0(Ljava/io/File;)I
J org.jboss.virtual.plugins.context.file.FileHandler.exists()Z
J org.jboss.virtual.plugins.context.file.FileHandler.createChildHandler(Ljava/lang/String;)Lorg/jboss/virtual/spi/VirtualFileHandler;
J org.jboss.virtual.plugins.context.AbstractVirtualFileHandler.structuredFindChild(Ljava/lang/String;)Lorg/jboss/virtual/spi/VirtualFileHandler;
J org.jboss.virtual.plugins.context.file.FileHandler.getChild(Ljava/lang/String;)Lorg/jboss/virtual/spi/VirtualFileHandler;
J org.jboss.virtual.VirtualFile.getChild(Ljava/lang/String;)Lorg/jboss/virtual/VirtualFile;
J org.jboss.deployers.vfs.spi.structure.modified.UpdateDeleteVisitor.doVisit(Lorg/jboss/virtual/VirtualFile;)V
J org.jboss.deployers.vfs.spi.structure.modified.SynchVisitor.visit(Lorg/jboss/virtual/VirtualFile;)V
J org.jboss.virtual.plugins.vfs.helpers.WrappingVirtualFileHandlerVisitor.visit(Lorg/jboss/virtual/spi/VirtualFileHandler;)V
J org.jboss.virtual.plugins.context.AbstractVFSContext.visit(Lorg/jboss/virtual/spi/VirtualFileHandler;Lorg/jboss/virtual/spi/VirtualFileHandlerVisitor;ZZZZLorg/jboss/virtual/VirtualFileFilter;)V
J org.jboss.virtual.plugins.context.AbstractVFSContext.visit(Lorg/jboss/virtual/spi/VirtualFileHandler;Lorg/jboss/virtual/spi/VirtualFileHandlerVisitor;ZZZZLorg/jboss/virtual/VirtualFileFilter;)V
J org.jboss.virtual.plugins.context.AbstractVFSContext.visit(Lorg/jboss/virtual/spi/VirtualFileHandler;Lorg/jboss/virtual/spi/VirtualFileHandlerVisitor;)V
J org.jboss.virtual.VFS.visit(Lorg/jboss/virtual/VirtualFile;Lorg/jboss/virtual/VirtualFileVisitor;)V
v ~I2CAdapter
j org.jboss.virtual.VirtualFile.visit(Lorg/jboss/virtual/VirtualFileVisitor;)V+13
j org.jboss.deployers.vfs.spi.structure.modified.SynchWrapperModificationChecker.hasStructureBeenModifed(Lorg/jboss/virtual/VirtualFile;Lorg/jboss/deployers/vfs/spi/structure/VFSDeploymentContext;)Z+58
j org.jboss.deployers.vfs.spi.structure.modified.AbstractStructureModificationChecker.hasStructureBeenModified(Lorg/jboss/deployers/vfs/spi/structure/VFSDeploymentContext;Z)Z+90
v ~C2IAdapter
J org.jboss.deployers.vfs.spi.structure.modified.AbstractStructureModificationChecker.hasStructureBeenModified(Lorg/jboss/virtual/VirtualFile;)Z
J org.jboss.system.server.profileservice.repository.HotDeploymentRepository.getModifiedDeployments()Ljava/util/Collection;
v ~I2CAdapter
j org.jboss.system.server.profile.repository.AbstractProfile.getModifiedDeployments()Ljava/util/Collection;+15
v ~C2IAdapter
J org.jboss.system.server.profileservice.hotdeploy.HDScanner.scan()V
v ~I2CAdapter
j org.jboss.system.server.profileservice.hotdeploy.HDScanner.run()V+1
j java.util.concurrent.Executors$RunnableAdapter.call()Ljava/lang/Object;+4
j java.util.concurrent.FutureTask$Sync.innerRunAndReset()Z+22
j java.util.concurrent.FutureTask.runAndReset()Z+4
j java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(Ljava/util/concurrent/ScheduledThreadPoolExecutor$ScheduledFutureTask;)Z+1
j java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic()V+1
j java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run()V+8
j java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Ljava/lang/Runnable;)V+43
v ~C2IAdapter
J java.util.concurrent.ThreadPoolExecutor$Worker.run()V
v ~OSRAdapter
j java.lang.Thread.run()V+11
v ~StubRoutines::call_stub
-------------- P R O C E S S ---------------
Java Threads: ( => current thread )
0x77192cd8 JavaThread "http-127.0.0.1-28080-20" daemon [_thread_blocked, id=28750]
0x7219d8b8 JavaThread "WorkManager(2)-17" daemon [_thread_blocked, id=25933]
0x0c90bc98 JavaThread "WorkManager(2)-16" daemon [_thread_blocked, id=25932]
0x73156008 JavaThread "WorkManager(2)-15" daemon [_thread_blocked, id=25895]
0x0b16ff58 JavaThread "WorkManager(2)-14" daemon [_thread_blocked, id=25894]
0x730058a0 JavaThread "WorkManager(2)-13" daemon [_thread_blocked, id=25609]
0x09415a90 JavaThread "WorkManager(2)-12" daemon [_thread_blocked, id=25608]
0x74064110 JavaThread "http-127.0.0.1-28080-19" daemon [_thread_blocked, id=25000]
0x72d62ee0 JavaThread "http-127.0.0.1-28080-18" daemon [_thread_blocked, id=24949]
0x73dfcc28 JavaThread "http-127.0.0.1-28080-17" daemon [_thread_blocked, id=24948]
0x73077f18 JavaThread "http-127.0.0.1-28080-16" daemon [_thread_blocked, id=23122]
0x74aac3a8 JavaThread "http-127.0.0.1-28080-15" daemon [_thread_blocked, id=17793]
0x71bdede8 JavaThread "WorkManager(2)-11" daemon [_thread_blocked, id=9050]
0x0a961a60 JavaThread "WorkManager(2)-10" daemon [_thread_blocked, id=9048]
0x72b5fef8 JavaThread "http-127.0.0.1-28080-14" daemon [_thread_blocked, id=7880]
0x74aadf58 JavaThread "WorkManager(2)-9" daemon [_thread_blocked, id=4977]
0x74a72b20 JavaThread "WorkManager(2)-8" daemon [_thread_in_native, id=4960]
0x0da25ec8 JavaThread "WorkManager(2)-7" daemon [_thread_blocked, id=4958]
0x11c024a0 JavaThread "ConnectionValidator" daemon [_thread_blocked, id=24055]
0x7f7b0198 JavaThread "JBoss System Threads(1)-10" daemon [_thread_blocked, id=23943]
0x77094fb8 JavaThread "WorkManager(2)-6" daemon [_thread_blocked, id=23942]
0x115b5fd8 JavaThread "WorkManager(2)-5" daemon [_thread_blocked, id=23941]
0x6edfa668 JavaThread "JBoss System Threads(1)-9" daemon [_thread_blocked, id=23929]
0x7a94be80 JavaThread "JBoss System Threads(1)-8" daemon [_thread_blocked, id=20996]
0x7215f9c0 JavaThread "http-127.0.0.1-28080-13" daemon [_thread_blocked, id=20954]
0x79dfde68 JavaThread "JBoss System Threads(1)-7" daemon [_thread_blocked, id=19144]
0x7d240cd8 JavaThread "JBoss System Threads(1)-6" daemon [_thread_blocked, id=13260]
0x70e85de0 JavaThread "JBoss System Threads(1)-5" daemon [_thread_blocked, id=12405]
0x772fd448 JavaThread "http-127.0.0.1-28080-12" daemon [_thread_blocked, id=12404]
0x730785b8 JavaThread "http-127.0.0.1-28080-11" daemon [_thread_blocked, id=11301]
0x731502d0 JavaThread "http-127.0.0.1-28080-10" daemon [_thread_blocked, id=11138]
0x7d27ac88 JavaThread "JBoss System Threads(1)-4" daemon [_thread_blocked, id=10273]
0x715a7ba8 JavaThread "http-127.0.0.1-28080-9" daemon [_thread_blocked, id=10270]
0x1327a9f0 JavaThread "Timer-3" daemon [_thread_blocked, id=9943]
0x6dee6250 JavaThread "JBoss System Threads(1)-3" daemon [_thread_blocked, id=9939]
0x73fac440 JavaThread "http-127.0.0.1-28080-8" daemon [_thread_blocked, id=9938]
0x740905b8 JavaThread "http-127.0.0.1-28080-7" daemon [_thread_blocked, id=9937]
0x73007570 JavaThread "http-127.0.0.1-28080-6" daemon [_thread_blocked, id=9936]
0x71b2cd30 JavaThread "http-127.0.0.1-28080-5" daemon [_thread_blocked, id=9935]
0x740f7d68 JavaThread "http-127.0.0.1-28080-4" daemon [_thread_blocked, id=9933]
0x11cfa830 JavaThread "http-127.0.0.1-28080-3" daemon [_thread_blocked, id=9932]
0x0d009408 JavaThread "Java2D Disposer" daemon [_thread_blocked, id=9913]
0x0aad7a20 JavaThread "File Reaper" daemon [_thread_blocked, id=9912]
0x11bff2b8 JavaThread "http-127.0.0.1-28080-2" daemon [_thread_blocked, id=9911]
0x0b856a48 JavaThread "http-127.0.0.1-28080-1" daemon [_thread_blocked, id=9909]
0x7dcdf8d0 JavaThread "ajp-127.0.0.1-28009-Acceptor-0" daemon [_thread_in_native, id=9883]
0x78c99c80 JavaThread "http-127.0.0.1-28080-Acceptor-0" daemon [_thread_in_native, id=9882]
0x78693c70 JavaThread "JBossLifeThread" [_thread_blocked, id=9881]
0x792f6f78 JavaThread "timerFactory" [_thread_blocked, id=9880]
0x78738ce0 JavaThread "timerFactory" [_thread_blocked, id=9879]
0x78694b90 JavaThread "Connection Consumer for dest JBossQueue[chatHistoryArchiver] id=3" daemon [_thread_blocked, id=9876]
0x777948b0 JavaThread "Connection Consumer for dest JBossQueue[mailMessageInternalDelivery] id=1" daemon [_thread_blocked, id=9874]
0x771c1d80 JavaThread "Connection Consumer for dest JBossQueue[mailMessageSMTPDelivery] id=2" daemon [_thread_blocked, id=9873]
0x08ac5f78 JavaThread "Timer-2" daemon [_thread_blocked, id=9867]
0x725eba80 JavaThread "WorkManager(2)-4" daemon [_thread_blocked, id=9866]
0x731227e0 JavaThread "WorkManager(2)-3" daemon [_thread_blocked, id=9865]
0x0f39b070 JavaThread "WorkManager(2)-2" daemon [_thread_blocked, id=9864]
0x09a31fc0 JavaThread "MySQL Statement Cancellation Timer" daemon [_thread_blocked, id=9862]
0x0a8ee368 JavaThread "IdleRemover" daemon [_thread_blocked, id=9861]
0x0d6130a0 JavaThread "Thread-16" daemon [_thread_blocked, id=9856]
0x0d36c178 JavaThread "JCA PoolFiller" [_thread_blocked, id=9855]
0x72dadad0 JavaThread "HSQLDB Timer #858c07" daemon [_thread_blocked, id=9854]
0x72fd1920 JavaThread "DefaultQuartzScheduler_QuartzSchedulerThread" [_thread_blocked, id=9853]
0x7336ee60 JavaThread "DefaultQuartzScheduler_Worker-9" [_thread_blocked, id=9852]
0x7375e560 JavaThread "DefaultQuartzScheduler_Worker-8" [_thread_blocked, id=9851]
0x77ffae50 JavaThread "DefaultQuartzScheduler_Worker-7" [_thread_blocked, id=9850]
0x77ffacc8 JavaThread "DefaultQuartzScheduler_Worker-6" [_thread_blocked, id=9849]
0x735b5df8 JavaThread "DefaultQuartzScheduler_Worker-5" [_thread_blocked, id=9848]
0x0bdb85b0 JavaThread "DefaultQuartzScheduler_Worker-4" [_thread_blocked, id=9847]
0x0a4d4bf0 JavaThread "DefaultQuartzScheduler_Worker-3" [_thread_blocked, id=9846]
0x090d3208 JavaThread "DefaultQuartzScheduler_Worker-2" [_thread_blocked, id=9845]
0x0c1b9d08 JavaThread "DefaultQuartzScheduler_Worker-1" [_thread_blocked, id=9844]
0x09089938 JavaThread "DefaultQuartzScheduler_Worker-0" [_thread_blocked, id=9843]
0x0a8b1228 JavaThread "WorkManager(2)-1" daemon [_thread_blocked, id=9842]
0x0cfb05a0 JavaThread "AOPListner" daemon [_thread_blocked, id=9841]
0x0da31c10 JavaThread "ContainerBackgroundProcessor[StandardEngine[jboss.web]]" daemon [_thread_blocked, id=9840]
0x7376f478 JavaThread "PooledInvokerAcceptor#0-4445" [_thread_in_native, id=9839]
0x738fec80 JavaThread "RMI TCP Accept-4444" daemon [_thread_in_native, id=9838]
0x7386f278 JavaThread "Thread-13" [_thread_blocked, id=9837]
0x739fdd38 JavaThread "Listener:4712" daemon [_thread_in_native, id=9836]
0x7dcdeb50 JavaThread "Thread-12" daemon [_thread_blocked, id=9834]
0x758f8aa8 JavaThread "Thread-11" daemon [_thread_blocked, id=9833]
0x78b61ba8 JavaThread "Thread-10" daemon [_thread_blocked, id=9832]
0x74548198 JavaThread "Listener:4713" daemon [_thread_in_native, id=9831]
0x78ff8b08 JavaThread "AcceptorThread[ServerSocket[addr=/127.0.0.1,port=0,localport=4446]]" [_thread_in_native, id=9830]
0x7759edb8 JavaThread "ServerSocketRefresh" daemon [_thread_blocked, id=9829]
=>0x7859f560 JavaThread "HDScanner" [_thread_in_native, id=9828]
0x78190340 JavaThread "AcceptorThread[ServerSocket[addr=/127.0.0.1,port=0,localport=3873]]" [_thread_in_native, id=9827]
0x78694df8 JavaThread "ServerSocketRefresh" daemon [_thread_blocked, id=9826]
0x788f7910 JavaThread "secondaryServerSocketThread[0]" daemon [_thread_in_native, id=9825]
0x78341100 JavaThread "AcceptorThread[ServerSocket[addr=/127.0.0.1,port=0,localport=4457]]" [_thread_in_native, id=9824]
0x78bf81e0 JavaThread "ServerSocketRefresh" daemon [_thread_blocked, id=9823]
0x09787878 JavaThread "RMI LeaseChecker" daemon [_thread_blocked, id=9820]
0x768ee010 JavaThread "RMI RenewClean-[127.0.0.1:59098]" daemon [_thread_blocked, id=9818]
0x7972ff40 JavaThread "RMI TCP Accept-0" daemon [_thread_in_native, id=9816]
0x790f9558 JavaThread "RMI TCP Accept-1090" daemon [_thread_in_native, id=9815]
0x7f3301b8 JavaThread "SubscriptionWatchDog" [_thread_blocked, id=9814]
0x09550498 JavaThread "JBoss System Threads(1)-2" daemon [_thread_in_native, id=9813]
0x0b58ef60 JavaThread "GC Daemon" daemon [_thread_blocked, id=9812]
0x09dfebf0 JavaThread "RMI Reaper" [_thread_blocked, id=9811]
0x0b59e7e8 JavaThread "Timer-1" daemon [_thread_blocked, id=9810]
0x08947368 JavaThread "RMI TCP Accept-21098" daemon [_thread_in_native, id=9809]
0x0b87ecf0 JavaThread "JBoss System Threads(1)-1" daemon [_thread_in_native, id=9808]
0x08947e18 JavaThread "Timer-Log4jService" daemon [_thread_blocked, id=9807]
0x7f78a070 JavaThread "Timer-0" daemon [_thread_blocked, id=9805]
0x80bf78b0 JavaThread "DestroyJavaVM" [_thread_blocked, id=9790]
0x80bcfe10 JavaThread "Low Memory Detector" daemon [_thread_blocked, id=9802]
0x80bcea30 JavaThread "CompilerThread1" daemon [_thread_blocked, id=9801]
0x80bcdab0 JavaThread "CompilerThread0" daemon [_thread_blocked, id=9800]
0x80bccaf0 JavaThread "AdapterThread" daemon [_thread_blocked, id=9799]
0x80bcbcc8 JavaThread "Signal Dispatcher" daemon [_thread_blocked, id=9798]
0x80bc2848 JavaThread "Finalizer" daemon [_thread_blocked, id=9797]
0x80bc2318 JavaThread "Reference Handler" daemon [_thread_blocked, id=9796]
Other Threads:
0x80bbff08 VMThread [id=9795]
0x80bd19b8 WatcherThread [id=9803]
VM state:not at safepoint (normal execution)
VM Mutex/Monitor currently owned by a thread: None
Heap
PSYoungGen total 41472K, used 4100K [0xad860000, 0xb0740000, 0xb1140000)
eden space 35328K, 1% used [0xad860000,0xad90cb38,0xafae0000)
from space 6144K, 55% used [0xb0140000,0xb0494728,0xb0740000)
to space 6336K, 0% used [0xafae0000,0xafae0000,0xb0110000)
PSOldGen total 466048K, used 214068K [0x91140000, 0xad860000, 0xad860000)
object space 466048K, 45% used [0x91140000,0x9e24d278,0xad860000)
PSPermGen total 71168K, used 70964K [0x81140000, 0x856c0000, 0x91140000)
object space 71168K, 99% used [0x81140000,0x8568d1f0,0x856c0000)
**[dynamic libraries excerpted]**
VM Arguments:
jvm_args: -Dprogram.name=run.sh -Xms128m -Xmx512m -XX:MaxPermSize=256m -Dorg.jboss.resolver.warning=true -Dsun.rmi.dgc.client.gcInterval=3600000 -Dsun.rmi.dgc.server.gcInterval=3600000 -Djava.net.preferIPv4Stack=true - Djava.endorsed.dirs=/opt/jboss/jboss-5.1.0.GA/lib/endorsed
java_command: org.jboss.Main -c livingdiary
Launcher Type: SUN_STANDARD
Environment Variables:
JAVA_HOME=/usr/java/jdk
JRE_HOME=/usr/java/jdk/jre
PATH=/usr/kerberos/bin:/usr/bin:/bin:/usr/sbin:/sbin:/home/jboss/bin:/opt/jboss/jboss-5.1.0.GA/bin
LD_LIBRARY_PATH=/usr/java/jdk1.5.0_20/jre/lib/i386/server:/usr/java/jdk1.5.0_20/jre/lib/i386:/usr/java/jdk1.5.0_20/jre/../lib/i386
SHELL=/bin/bash
Signal Handlers:
SIGSEGV: [libjvm.so+0x517270], sa_mask[0]=0x7ffbfeff, sa_flags=0x10000004
SIGBUS: [libjvm.so+0x517270], sa_mask[0]=0x7ffbfeff, sa_flags=0x10000004
SIGFPE: [libjvm.so+0x43a0a0], sa_mask[0]=0x7ffbfeff, sa_flags=0x10000004
SIGPIPE: [libjvm.so+0x43a0a0], sa_mask[0]=0x7ffbfeff, sa_flags=0x10000004
SIGILL: [libjvm.so+0x43a0a0], sa_mask[0]=0x7ffbfeff, sa_flags=0x10000004
SIGUSR1: SIG_DFL, sa_mask[0]=0x00000000, sa_flags=0x00000000
SIGUSR2: [libjvm.so+0x43c4f0], sa_mask[0]=0x00000000, sa_flags=0x10000004
SIGHUP: SIG_IGN, sa_mask[0]=0x00000000, sa_flags=0x00000000
SIGINT: SIG_IGN, sa_mask[0]=0x00000000, sa_flags=0x00000000
SIGQUIT: [libjvm.so+0x43bf20], sa_mask[0]=0x7ffbfeff, sa_flags=0x10000004
SIGTERM: [libjvm.so+0x43bf20], sa_mask[0]=0x7ffbfeff, sa_flags=0x10000004
--------------- S Y S T E M ---------------
OS:Fedora release 8 (Werewolf)
uname:Linux 2.6.26.8-57.fc8 #1 SMP Thu Dec 18 19:19:45 EST 2008 i686
libc:glibc 2.7 NPTL 2.7
rlimit: STACK 10240k, CORE 0k, NPROC 65536, NOFILE 1024, AS infinity
load average:0.01 0.02 0.00
CPU:total 4 (cores per cpu 4, threads per core 1) family 6 model 7 stepping 7, cmov, cx8, fxsr, mmx, sse, sse2
Memory: 4k page, physical 3358636k(383020k free), swap 511k(511k free)
vm_info: Java HotSpot(TM) Server VM (1.5.0_20-b02) for linux-x86, built on Jun 18 2009 12:08:29 by java_re with gcc 3.2.1-7a (J2SE release)
time: Wed May 11 07:00:36 2011
elapsed time: 853441 seconds
Other things that I have done:
1. I have added -Xcheck:jni to the JVM options to see if there is some sort of problem with the native library.
2. Turned up debugging on Jboss to see if I can capture more info before the JVM dies.
Thanks very much if you have some advice!
UPDATE SINCE LAST NIGHT
So, the server was up for a couple of days...and then crashed again for the fifth time. I added a new question with the new report here: https://stackoverflow.com/questions/6005976/jboss-hotspot-jvm-crashing-again
UPDATE AFTER RUNNING H/W DIAGNOSTICS
Just received the following reply from GoDaddy after having them run diagnostics:
Thank you for contacting Server
Support regarding your xxxxx server.
We have performed a hardware check
against your server and did find that
the chassis has failed and needs to be
replaced. We have attempted to replace
the chassis with one of the same model
but it is not available. Therefore we
will need to swap the chassis with a
different model, but this will require
the server to be re-provisioned since
the operating system will not boot
with a different hardware
configuration. A re-provision will
erase all data on the server and
reinstall the operating system along
with Simple Control Panel. For us to
continue the chassis swap we do need
your explicit permission to
re-provision your server on your
behalf.


Are the crash reports always the same? Do you have the stack traces? i.e. the bit in the crash dump headed by
--------------- P R O C E S S ---------------
It's a v vague dump tbh. One possibility is dodgy RAM as it seems to have died while at a safepoint reached in order to collect garbage. Have you done any hardware diagnostics on the box?


I agree with Matt, it does sound like hardware problems. I've seen your three crash reports and they are all failing in different parts. One, mostly when executing VM code. The second, in native code. And the third (in the question you should have not opened, but added here), in standard Java code.
Of course, if you have several production machines and the issue is happening in different ones, then I would seriously consider to buy subscriptions and get real support.

Related

Scala/Akka/Dropwizard Metrics app hanging for 10s before shutting down

I have a Scala 2.11 app that uses Akka. The GitHub repo for it is here. To reproduce what I'm seeing, just clone it, build it via ./gradlew fullBuild and then run it via java -jar build/libs/akka-scala-troubleshooting.jar (its a self-contained executable JAR). But essentially, I'm noticing that when I kill the running JVM process via Ctrl+C that it takes exactly 10 seconds for the JAR to shut down!
IIRC, Ctrl+C should issue a SIGKILL to the JVM process, so how it lives for another 10 seconds thereafter is blowing my mind! Even if I'm recalling that incorrectly, I'd still like to know why my app isn't shutting down immediately (it's causing problems by not doing so).
I'm using Dropwizard (formerly Coda Hale) Metrics via the Scala Metrics library. From what I can tell there might be a scheduled reporter daemon still alive that isn't shutting down right away, even when the JVM receives the SIGKILL (or whatever Ctrl+C is sending). Here's the main Driver (app entry point):
object Driver extends App {
println("Starting upp the app...")
//SLF4JBridgeHandler.install()
lazy val metricRegistry = new MetricRegistry()
ConsoleReporter
.forRegistry(metricRegistry)
.convertRatesTo(TimeUnit.SECONDS)
.convertDurationsTo(TimeUnit.MILLISECONDS)
.build()
.start(15, TimeUnit.SECONDS)
lazy val cortex = ActorSystem("cortex")
cortex.registerOnTermination {
System.exit(0)
}
val master = cortex.actorOf(Props[Master], name = "Master")
println("About to fire a StartUp message at Master...")
master ! StartUp
println("Fired! Actor system is spinning up...")
}
When you run the JAR, you'll see this output to the logs:
java -jar build/libs/akka-scala-troubleshooting.jar
Starting upp the app...
About to fire a StartUp message at Master...
Fired! Actor system is spinning up...
Master has received a command to start up the actor system!
Child will make it happen!
At this point it will just idle and do nothing -- that's fine and intended! But when I hit Ctrl+C on the terminal, I immediately get this log INFO:
^C[INFO] [08/03/2017 06:15:57.236] [Thread-0] [CoordinatedShutdown(akka://cortex)] Starting coordinated shutdown from JVM shutdown hook
Then 10 seconds goes by (exactly 10s every time), then finally a log WARNing:
[WARN] [08/03/2017 06:16:07.248] [Thread-0] [CoordinatedShutdown(akka://cortex)] CoordinatedShutdown from JVM shutdown failed: Futures timed out after [10000 milliseconds]
Any idea what could be going on here?
Update:
I did a thread dump:
2017-08-03 14:08:58
Full thread dump OpenJDK 64-Bit Server VM (25.131-b11 mixed mode):
"cortex-akka.actor.default-dispatcher-5" #19 prio=5 os_prio=0 tid=0x00007fcba8009000 nid=0x547d waiting on condition [0x00007fcc16e6f000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x000000076ff34e28> (a akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
at akka.dispatch.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075)
at akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
at akka.dispatch.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
Locked ownable synchronizers:
- None
"Thread-0" #15 prio=5 os_prio=0 tid=0x00007fcba0001000 nid=0x547c waiting on condition [0x00007fcc17170000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x0000000770b33920> (a scala.concurrent.impl.Promise$CompletionLatch)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedNanos(AbstractQueuedSynchronizer.java:1037)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(AbstractQueuedSynchronizer.java:1328)
at scala.concurrent.impl.Promise$DefaultPromise.tryAwait(Promise.scala:212)
at scala.concurrent.impl.Promise$DefaultPromise.ready(Promise.scala:222)
at scala.concurrent.impl.Promise$DefaultPromise.ready(Promise.scala:157)
at scala.concurrent.Await$$anonfun$ready$1.apply(package.scala:169)
at scala.concurrent.Await$$anonfun$ready$1.apply(package.scala:169)
at scala.concurrent.BlockContext$DefaultBlockContext$.blockOn(BlockContext.scala:53)
at scala.concurrent.Await$.ready(package.scala:169)
at akka.actor.CoordinatedShutdown$$anonfun$initJvmHook$1.apply(CoordinatedShutdown.scala:161)
at akka.actor.CoordinatedShutdown$$anon$2.run(CoordinatedShutdown.scala:446)
Locked ownable synchronizers:
- None
"SIGTERM handler" #18 daemon prio=9 os_prio=0 tid=0x00007fcbdc002800 nid=0x547b in Object.wait() [0x00007fcc17271000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x00000007707bb150> (a akka.actor.CoordinatedShutdown$$anon$2)
at java.lang.Thread.join(Thread.java:1252)
- locked <0x00000007707bb150> (a akka.actor.CoordinatedShutdown$$anon$2)
at java.lang.Thread.join(Thread.java:1326)
at java.lang.ApplicationShutdownHooks.runHooks(ApplicationShutdownHooks.java:106)
at java.lang.ApplicationShutdownHooks$1.run(ApplicationShutdownHooks.java:46)
at java.lang.Shutdown.runHooks(Shutdown.java:123)
at java.lang.Shutdown.sequence(Shutdown.java:167)
at java.lang.Shutdown.exit(Shutdown.java:212)
- locked <0x00000007707bb5d0> (a java.lang.Class for java.lang.Shutdown)
at java.lang.Terminator$1.handle(Terminator.java:52)
at sun.misc.Signal$1.run(Signal.java:212)
at java.lang.Thread.run(Thread.java:748)
Locked ownable synchronizers:
- None
"Attach Listener" #17 daemon prio=9 os_prio=0 tid=0x00007fcbdc001000 nid=0x545f waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
Locked ownable synchronizers:
- None
"DestroyJavaVM" #16 prio=5 os_prio=0 tid=0x00007fcc3800c000 nid=0x5427 waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
Locked ownable synchronizers:
- None
"cortex-akka.actor.default-dispatcher-4" #14 prio=5 os_prio=0 tid=0x00007fcbb0010000 nid=0x5445 waiting for monitor entry [0x00007fcc17473000]
java.lang.Thread.State: BLOCKED (on object monitor)
at java.lang.Shutdown.exit(Shutdown.java:212)
- waiting to lock <0x00000007707bb5d0> (a java.lang.Class for java.lang.Shutdown)
at java.lang.Runtime.exit(Runtime.java:109)
at java.lang.System.exit(System.java:971)
at hotmeatballsoup.Driver$$anonfun$1.apply$mcV$sp(Driver.scala:24)
at hotmeatballsoup.Driver$$anonfun$1.apply(Driver.scala:24)
at hotmeatballsoup.Driver$$anonfun$1.apply(Driver.scala:24)
at akka.actor.ActorSystemImpl$$anon$3.run(ActorSystem.scala:810)
at akka.actor.ActorSystemImpl$TerminationCallbacks$$anonfun$addRec$1$1.applyOrElse(ActorSystem.scala:987)
at akka.actor.ActorSystemImpl$TerminationCallbacks$$anonfun$addRec$1$1.applyOrElse(ActorSystem.scala:987)
at scala.concurrent.Future$$anonfun$andThen$1.apply(Future.scala:436)
at scala.concurrent.Future$$anonfun$andThen$1.apply(Future.scala:435)
at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:36)
at akka.dispatch.BatchingExecutor$AbstractBatch.processBatch(BatchingExecutor.scala:55)
at akka.dispatch.BatchingExecutor$BlockableBatch$$anonfun$run$1.apply$mcV$sp(BatchingExecutor.scala:91)
at akka.dispatch.BatchingExecutor$BlockableBatch$$anonfun$run$1.apply(BatchingExecutor.scala:91)
at akka.dispatch.BatchingExecutor$BlockableBatch$$anonfun$run$1.apply(BatchingExecutor.scala:91)
at scala.concurrent.BlockContext$.withBlockContext(BlockContext.scala:72)
at akka.dispatch.BatchingExecutor$BlockableBatch.run(BatchingExecutor.scala:90)
at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:38)
at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(ForkJoinExecutorConfigurator.scala:43)
at akka.dispatch.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
at akka.dispatch.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
at akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
at akka.dispatch.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
Locked ownable synchronizers:
- None
"cortex-akka.actor.default-dispatcher-3" #13 prio=5 os_prio=0 tid=0x00007fcc386e8800 nid=0x5444 waiting on condition [0x00007fcc17574000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x000000076ff34e28> (a akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
at akka.dispatch.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075)
at akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
at akka.dispatch.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
Locked ownable synchronizers:
- None
"cortex-akka.actor.default-dispatcher-2" #12 prio=5 os_prio=0 tid=0x00007fcc386db000 nid=0x5443 waiting on condition [0x00007fcc17675000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x000000076ff34e28> (a akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
at akka.dispatch.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075)
at akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
at akka.dispatch.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
Locked ownable synchronizers:
- None
"cortex-scheduler-1" #11 prio=5 os_prio=0 tid=0x00007fcc38645800 nid=0x5442 waiting on condition [0x00007fcc17976000]
java.lang.Thread.State: TIMED_WAITING (sleeping)
at java.lang.Thread.sleep(Native Method)
at akka.actor.LightArrayRevolverScheduler.waitNanos(LightArrayRevolverScheduler.scala:85)
at akka.actor.LightArrayRevolverScheduler$$anon$4.nextTick(LightArrayRevolverScheduler.scala:265)
at akka.actor.LightArrayRevolverScheduler$$anon$4.run(LightArrayRevolverScheduler.scala:235)
at java.lang.Thread.run(Thread.java:748)
Locked ownable synchronizers:
- None
"metrics-console-reporter-1-thread-1" #10 daemon prio=5 os_prio=0 tid=0x00007fcc384ee800 nid=0x5441 waiting on condition [0x00007fcc17c77000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x000000076e328370> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Locked ownable synchronizers:
- None
Note that 2 threads are trying to call Shutdown.exit...

The 10 seconds is defined here in the configuration:
actor-system-terminate {
timeout = 10 s
depends-on = [before-actor-system-terminate]
}
When system.terminate() is called, CoordinatedShutdown by default is initiated. Notice that this method returns a Future, meaning that termination will happen asynchronously. By default, there is a 10-second timeout for the actor system to shut down.
In your driver, you call registerOnTermination. The Scaladoc for this method states:
Note that ActorSystem will not terminate until all the registered callbacks are finished.
Your registered callback invokes System.exit, and you're getting the timeout warning because the actor system is unable to properly shut down within the 10-second window. The System.exit call is probably interfering with the actor system's coordinated shutdown. Removing the System.exit call in registerOnTermination, or removing the call to registerOnTermination altogether, will remove the warning.

Jboss EAP Server Hanged with many waiting threads

Can someone explain me based on the following image. How to find which objects are in waiting state?
We see the system hangs.. Our system has a External authentication to each page of the portal.. So our system is getting slow and hanged
Threaddump showed the following image..
"http-serverIP-8080-107" daemon prio=10 tid=0x0000000052017000 nid=0x4fa3 in Object.wait() [0x00002ae69d5cf000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x00000007145a6148> (a org.apache.tomcat.util.net.JIoEndpoint$Worker)
at java.lang.Object.wait(Object.java:485)
at org.apache.tomcat.util.net.JIoEndpoint$Worker.await(JIoEndpoint.java:416)
- locked <0x00000007145a6148> (a org.apache.tomcat.util.net.JIoEndpoint$Worker)
at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:442)
at java.lang.Thread.run(Thread.java:662)

Eclipse hung on Mac OS X when clicking on TabItems and then switch to a list

I'm developing an eclipse plugin on Mac OS X 10.7.3. The UI consists of a TabFolder with 4 TabItems, and some other lists. When I switch between the tabs several times and then clicking another list, eclipse will hung. from the Activity Monitor I could see the CPU usage rushed to 100% and Real Mem also keep increasing. The JVM dump is as below:
2012-12-07 10:14:14
Full thread dump Java HotSpot(TM) 64-Bit Server VM (20.4-b02-402 mixed mode):
"Attach Listener" daemon prio=9 tid=7fa8a4906800 nid=0x116fe7000 waiting on condition [00000000]
java.lang.Thread.State: RUNNABLE
"[ThreadPool Manager] - Idle Thread" daemon prio=5 tid=7fa8a396a000 nid=0x116ae1000 in Object.wait() [116ae0000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <7d0133cb8> (a org.eclipse.equinox.internal.util.impl.tpt.threadpool.Executor)
at java.lang.Object.wait(Object.java:485)
at org.eclipse.equinox.internal.util.impl.tpt.threadpool.Executor.run(Executor.java:106)
- locked <7d0133cb8> (a org.eclipse.equinox.internal.util.impl.tpt.threadpool.Executor)
"Worker-3" prio=5 tid=7fa89fe4f000 nid=0x114752000 in Object.wait() [114751000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <7d43e4648> (a org.eclipse.core.internal.jobs.WorkerPool)
at org.eclipse.core.internal.jobs.WorkerPool.sleep(WorkerPool.java:188)
- locked <7d43e4648> (a org.eclipse.core.internal.jobs.WorkerPool)
at org.eclipse.core.internal.jobs.WorkerPool.startJob(WorkerPool.java:220)
at org.eclipse.core.internal.jobs.Worker.run(Worker.java:50)
"org.eclipse.jface.text.reconciler.MonoReconciler" daemon prio=1 tid=7fa8a3a7c000 nid=0x11730c000 in Object.wait() [11730b000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <7d5c6c590> (a org.eclipse.jface.text.reconciler.DirtyRegionQueue)
at org.eclipse.jface.text.reconciler.AbstractReconciler$BackgroundThread.run(AbstractReconciler.java:179)
- locked <7d5c6c590> (a org.eclipse.jface.text.reconciler.DirtyRegionQueue)
"JavaScript indexing" daemon prio=4 tid=7fa8a2aca800 nid=0x11653f000 in Object.wait() [11653e000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <7d576a0c8> (a org.eclipse.wst.jsdt.internal.core.search.indexing.IndexManager)
at java.lang.Object.wait(Object.java:485)
at org.eclipse.wst.jsdt.internal.core.search.processing.JobManager.run(JobManager.java:370)
- locked <7d576a0c8> (a org.eclipse.wst.jsdt.internal.core.search.indexing.IndexManager)
at java.lang.Thread.run(Thread.java:680)
"Java indexing" daemon prio=4 tid=7fa8a1c47800 nid=0x1169de000 in Object.wait() [1169dd000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <7d50c39d8> (a org.eclipse.jdt.internal.core.search.indexing.IndexManager)
at java.lang.Object.wait(Object.java:485)
at org.eclipse.jdt.internal.core.search.processing.JobManager.run(JobManager.java:382)
- locked <7d50c39d8> (a org.eclipse.jdt.internal.core.search.indexing.IndexManager)
at java.lang.Thread.run(Thread.java:680)
"Worker-2" prio=5 tid=7fa89f49a000 nid=0x11641b000 in Object.wait() [11641a000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <7d43e4648> (a org.eclipse.core.internal.jobs.WorkerPool)
at org.eclipse.core.internal.jobs.WorkerPool.sleep(WorkerPool.java:188)
- locked <7d43e4648> (a org.eclipse.core.internal.jobs.WorkerPool)
at org.eclipse.core.internal.jobs.WorkerPool.startJob(WorkerPool.java:220)
at org.eclipse.core.internal.jobs.Worker.run(Worker.java:50)
"Worker-1" prio=5 tid=7fa89f3cc000 nid=0x114dcf000 in Object.wait() [114dce000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <7d43e4648> (a org.eclipse.core.internal.jobs.WorkerPool)
at org.eclipse.core.internal.jobs.WorkerPool.sleep(WorkerPool.java:188)
- locked <7d43e4648> (a org.eclipse.core.internal.jobs.WorkerPool)
at org.eclipse.core.internal.jobs.WorkerPool.startJob(WorkerPool.java:220)
at org.eclipse.core.internal.jobs.Worker.run(Worker.java:50)
"Worker-0" prio=5 tid=7fa89d0b0000 nid=0x116169000 in Object.wait() [116168000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <7d43e4648> (a org.eclipse.core.internal.jobs.WorkerPool)
at org.eclipse.core.internal.jobs.WorkerPool.sleep(WorkerPool.java:188)
- locked <7d43e4648> (a org.eclipse.core.internal.jobs.WorkerPool)
at org.eclipse.core.internal.jobs.WorkerPool.startJob(WorkerPool.java:220)
at org.eclipse.core.internal.jobs.Worker.run(Worker.java:50)
"Worker-JM" prio=5 tid=7fa89e886000 nid=0x113f33000 in Object.wait() [113f32000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <7d4277db8> (a java.util.ArrayList)
at org.eclipse.core.internal.jobs.InternalWorker.run(InternalWorker.java:58)
- locked <7d4277db8> (a java.util.ArrayList)
"Bundle File Closer" daemon prio=5 tid=7fa89b33f000 nid=0x114baf000 in Object.wait() [114bae000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <7d447b468> (a org.eclipse.osgi.framework.eventmgr.EventManager$EventThread)
at java.lang.Object.wait(Object.java:485)
at org.eclipse.osgi.framework.eventmgr.EventManager$EventThread.getNextEvent(EventManager.java:400)
- locked <7d447b468> (a org.eclipse.osgi.framework.eventmgr.EventManager$EventThread)
at org.eclipse.osgi.framework.eventmgr.EventManager$EventThread.run(EventManager.java:336)
"[Timer] - Main Queue Handler" daemon prio=5 tid=7fa89ec29800 nid=0x114ef4000 in Object.wait() [114ef3000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <7d42e9a50> (a java.lang.Object)
at org.eclipse.equinox.internal.util.impl.tpt.timer.TimerImpl.run(TimerImpl.java:141)
- locked <7d42e9a50> (a java.lang.Object)
at java.lang.Thread.run(Thread.java:680)
"Framework Event Dispatcher" daemon prio=5 tid=7fa89b2f9800 nid=0x1141c6000 in Object.wait() [1141c5000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <7d43147f0> (a org.eclipse.osgi.framework.eventmgr.EventManager$EventThread)
at java.lang.Object.wait(Object.java:485)
at org.eclipse.osgi.framework.eventmgr.EventManager$EventThread.getNextEvent(EventManager.java:400)
- locked <7d43147f0> (a org.eclipse.osgi.framework.eventmgr.EventManager$EventThread)
at org.eclipse.osgi.framework.eventmgr.EventManager$EventThread.run(EventManager.java:336)
"Start Level Event Dispatcher" daemon prio=5 tid=7fa89e90b800 nid=0x11464f000 in Object.wait() [11464e000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <7d4314870> (a org.eclipse.osgi.framework.eventmgr.EventManager$EventThread)
at java.lang.Object.wait(Object.java:485)
at org.eclipse.osgi.framework.eventmgr.EventManager$EventThread.getNextEvent(EventManager.java:400)
- locked <7d4314870> (a org.eclipse.osgi.framework.eventmgr.EventManager$EventThread)
at org.eclipse.osgi.framework.eventmgr.EventManager$EventThread.run(EventManager.java:336)
"State Data Manager" daemon prio=5 tid=7fa89b15c000 nid=0x113ce4000 waiting on condition [113ce3000]
java.lang.Thread.State: TIMED_WAITING (sleeping)
at java.lang.Thread.sleep(Native Method)
at org.eclipse.osgi.internal.baseadaptor.StateManager.run(StateManager.java:297)
at java.lang.Thread.run(Thread.java:680)
"Framework Active Thread" prio=5 tid=7fa89b157800 nid=0x113be1000 in Object.wait() [113be0000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <7d4021de0> (a org.eclipse.osgi.framework.internal.core.Framework)
at org.eclipse.osgi.framework.internal.core.Framework.run(Framework.java:1862)
- locked <7d4021de0> (a org.eclipse.osgi.framework.internal.core.Framework)
at java.lang.Thread.run(Thread.java:680)
"Poller SunPKCS11-Darwin" daemon prio=1 tid=7fa89c10c800 nid=0x112448000 waiting on condition [112447000]
java.lang.Thread.State: TIMED_WAITING (sleeping)
at java.lang.Thread.sleep(Native Method)
at sun.security.pkcs11.SunPKCS11$TokenPoller.run(SunPKCS11.java:692)
at java.lang.Thread.run(Thread.java:680)
"Thread-1" daemon prio=6 tid=7fa89c002800 nid=0x7fff734ca960 runnable [7fff68d24000]
java.lang.Thread.State: RUNNABLE
at org.eclipse.swt.internal.cocoa.OS.objc_msgSendSuper(Native Method)
at org.eclipse.swt.widgets.Widget.callSuper(Widget.java:221)
at org.eclipse.swt.widgets.Widget.mouseDownSuper(Widget.java:1093)
at org.eclipse.swt.widgets.List.mouseDownSuper(List.java:765)
at org.eclipse.swt.widgets.Widget.mouseDown(Widget.java:1085)
at org.eclipse.swt.widgets.Control.mouseDown(Control.java:2538)
at org.eclipse.swt.widgets.Display.windowProc(Display.java:5493)
at org.eclipse.swt.internal.cocoa.OS.objc_msgSendSuper(Native Method)
at org.eclipse.swt.widgets.Widget.callSuper(Widget.java:221)
at org.eclipse.swt.widgets.Widget.windowSendEvent(Widget.java:2102)
at org.eclipse.swt.widgets.Shell.windowSendEvent(Shell.java:2299)
at org.eclipse.swt.widgets.Display.windowProc(Display.java:5557)
at org.eclipse.swt.internal.cocoa.OS.objc_msgSendSuper(Native Method)
at org.eclipse.swt.widgets.Display.applicationSendEvent(Display.java:5002)
at org.eclipse.swt.widgets.Display.applicationProc(Display.java:5151)
at org.eclipse.swt.internal.cocoa.OS.objc_msgSend(Native Method)
at org.eclipse.swt.internal.cocoa.NSApplication.sendEvent(NSApplication.java:128)
at org.eclipse.swt.widgets.Display.readAndDispatch(Display.java:3616)
at org.eclipse.e4.ui.internal.workbench.swt.PartRenderingEngine$9.run(PartRenderingEngine.java:1029)
at org.eclipse.core.databinding.observable.Realm.runWithDefault(Realm.java:332)
at org.eclipse.e4.ui.internal.workbench.swt.PartRenderingEngine.run(PartRenderingEngine.java:923)
at org.eclipse.e4.ui.internal.workbench.E4Workbench.createAndRunUI(E4Workbench.java:86)
at org.eclipse.ui.internal.Workbench$5.run(Workbench.java:588)
at org.eclipse.core.databinding.observable.Realm.runWithDefault(Realm.java:332)
at org.eclipse.ui.internal.Workbench.createAndRunWorkbench(Workbench.java:543)
at org.eclipse.ui.PlatformUI.createAndRunWorkbench(PlatformUI.java:149)
at org.eclipse.ui.internal.ide.application.IDEApplication.start(IDEApplication.java:124)
at org.eclipse.equinox.internal.app.EclipseAppHandle.run(EclipseAppHandle.java:196)
at org.eclipse.core.runtime.internal.adaptor.EclipseAppLauncher.runApplication(EclipseAppLauncher.java:110)
at org.eclipse.core.runtime.internal.adaptor.EclipseAppLauncher.start(EclipseAppLauncher.java:79)
at org.eclipse.core.runtime.adaptor.EclipseStarter.run(EclipseStarter.java:353)
at org.eclipse.core.runtime.adaptor.EclipseStarter.run(EclipseStarter.java:180)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.eclipse.equinox.launcher.Main.invokeFramework(Main.java:629)
at org.eclipse.equinox.launcher.Main.basicRun(Main.java:584)
at org.eclipse.equinox.launcher.Main.run(Main.java:1438)
at org.eclipse.equinox.launcher.Main.main(Main.java:1414)
"Low Memory Detector" daemon prio=5 tid=7fa89e80b000 nid=0x111ed4000 runnable [00000000]
java.lang.Thread.State: RUNNABLE
"C2 CompilerThread1" daemon prio=9 tid=7fa89e80a800 nid=0x111dd1000 waiting on condition [00000000]
java.lang.Thread.State: RUNNABLE
"C2 CompilerThread0" daemon prio=9 tid=7fa89e809800 nid=0x111cce000 waiting on condition [00000000]
java.lang.Thread.State: RUNNABLE
"JDWP Command Reader" daemon prio=5 tid=7fa89b0c9000 nid=0x111bcb000 runnable [00000000]
java.lang.Thread.State: RUNNABLE
"JDWP Event Helper Thread" daemon prio=5 tid=7fa89b0c7000 nid=0x111ac8000 runnable [00000000]
java.lang.Thread.State: RUNNABLE
"JDWP Transport Listener: dt_socket" daemon prio=5 tid=7fa89b0c8000 nid=0x1119c5000 runnable [00000000]
java.lang.Thread.State: RUNNABLE
"Signal Dispatcher" daemon prio=9 tid=7fa89b0c5800 nid=0x1118bc000 runnable [00000000]
java.lang.Thread.State: RUNNABLE
"Surrogate Locker Thread (Concurrent GC)" daemon prio=5 tid=7fa89b0c4800 nid=0x1117b9000 waiting on condition [00000000]
java.lang.Thread.State: RUNNABLE
"Finalizer" daemon prio=8 tid=7fa89d041800 nid=0x1114f5000 in Object.wait() [1114f4000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <7d4024710> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:118)
- locked <7d4024710> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:134)
at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:159)
"Reference Handler" daemon prio=10 tid=7fa89d041000 nid=0x1113f2000 in Object.wait() [1113f1000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <7d4020878> (a java.lang.ref.Reference$Lock)
at java.lang.Object.wait(Object.java:485)
at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:116)
- locked <7d4020878> (a java.lang.ref.Reference$Lock)
"main" prio=5 tid=7fa89b001000 nid=0x1099f9000 runnable [00000000]
java.lang.Thread.State: RUNNABLE
"VM Thread" prio=9 tid=7fa89d03c000 nid=0x1112ef000 runnable
"Gang worker#0 (Parallel GC Threads)" prio=9 tid=7fa89b002800 nid=0x10cd72000 runnable
"Gang worker#1 (Parallel GC Threads)" prio=9 tid=7fa89b003000 nid=0x10ce75000 runnable
"Gang worker#2 (Parallel GC Threads)" prio=9 tid=7fa89b003800 nid=0x10cf78000 runnable
"Gang worker#3 (Parallel GC Threads)" prio=9 tid=7fa89b004000 nid=0x10d07b000 runnable
"Concurrent Mark-Sweep GC Thread" prio=9 tid=7fa89b07f800 nid=0x110f90000 runnable
"VM Periodic Task Thread" prio=10 tid=7fa89e81d000 nid=0x111fd7000 waiting on condition
"Exception Catcher Thread" prio=10 tid=7fa89b001800 nid=0x109b99000 runnable
JNI global references: 11239
Does any one have idea on what is happening?

After investigation i finally found the root cause. It's not caused by the Tab but the List. When clicking on the empty area on a list (the list has more rows than entries), the list.getSelection() will still be the current selected item on Windows, but will be empty on Mac. And when retrieving the value by list.getSelection()[0], eclipse will hang instead of throw ArrayIndexOutOfBoundsException(). I've worked around this issue by remembering the current selection with local variable and set the selection when user clicked on empty areas.

JBoss, thousand of threads in WAITING state, is this normal?

I am running JBoss 5.0.0 on Solaris 10 SPARC with java build 1.6.0_27-b07.
When I check with ps -eLf | grep -c 'Xss256k' (I use java command line arg Xss256k to minimize thread size) I am seeing 8000 threads (and counting)..
Thread dump also reveals a lof of process in WAITING state, is this normal?
Here is an excerpt of the thread dump (I have thousand of thread like this),
Thread: pool-159-thread-10 : priority:5, demon:false, threadId:2231, threadState:WAITING
- waiting on <0x169635f> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
java.lang.Thread.run(Thread.java:662)

Scala REPL ~20secs to startup -- is this normal?

I'm using Scala 2.9.0.final on Ubuntu 11.04. Starting scala from the command line takes about 20 seconds before a prompt appears.
I realize I should expect some delay because it has to lead compiler JAR files, but I have a hard time believing that it should be 20 seconds -- I worry I may have something configured wrong.
EDIT
So I tried jstack and it gives a long output that is above my head. I see one thread waiting on a mutex. But looking at the output from time:
21.96s user 0.66s system 135% cpu 16.711 total
It would seem that it spent a decent amount of cycles on both cores... so maybe it's not just blocking on a mutex for that time?
2011-07-28 11:45:02
Full thread dump OpenJDK 64-Bit Server VM (20.0-b11 mixed mode):
"Attach Listener" daemon prio=10 tid=0x00000000023ea000 nid=0x19a9 waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"Thread-0" prio=10 tid=0x00007f7768304800 nid=0x19a8 runnable [0x00007f776cd92000]
java.lang.Thread.State: RUNNABLE
at scala.collection.generic.Growable$$anonfun$$plus$plus$eq$1.<init>(Growable.scala:48)
at scala.collection.generic.Growable$class.$plus$plus$eq(Growable.scala:48)
at scala.collection.mutable.ListBuffer.$plus$plus$eq(ListBuffer.scala:128)
at scala.collection.TraversableOnce$class.toList(TraversableOnce.scala:242)
at scala.collection.mutable.WrappedArray.toList(WrappedArray.scala:33)
at scala.collection.immutable.List$.apply(List.scala:434)
at scala.Option.toList(Option.scala:231)
at scala.tools.nsc.util.DirectoryClassPath.asURLs(ClassPath.scala:321)
at scala.tools.nsc.util.ClassPath.sortString(ClassPath.scala:284)
at scala.tools.nsc.util.ClassPath.hashCode(ClassPath.scala:289)
at scala.collection.mutable.FlatHashTable$HashUtils$class.elemHashCode(FlatHashTable.scala:349)
at scala.collection.mutable.HashSet.elemHashCode(HashSet.scala:39)
at scala.collection.mutable.FlatHashTable$class.containsEntry(FlatHashTable.scala:109)
at scala.collection.mutable.HashSet.containsEntry(HashSet.scala:39)
at scala.collection.mutable.HashSet.contains(HashSet.scala:55)
at scala.collection.GenSetLike$class.apply(GenSetLike.scala:44)
at scala.collection.mutable.HashSet.apply(HashSet.scala:39)
at scala.collection.SeqLike$$anonfun$distinct$1.apply(SeqLike.scala:505)
at scala.collection.LinearSeqOptimized$class.foreach(LinearSeqOptimized.scala:59)
at scala.collection.immutable.List.foreach(List.scala:45)
at scala.collection.SeqLike$class.distinct(SeqLike.scala:504)
at scala.collection.immutable.List.distinct(List.scala:45)
at scala.tools.util.PathResolver$Calculated$.containers(PathResolver.scala:206)
- locked <0x00000000faba8098> (a scala.tools.util.PathResolver$Calculated$)
at scala.tools.util.PathResolver.containers(PathResolver.scala:228)
at scala.tools.util.PathResolver.result(PathResolver.scala:231)
- locked <0x00000000faba8050> (a scala.tools.util.PathResolver)
at scala.tools.nsc.backend.JavaPlatform$class.classPath(JavaPlatform.scala:18)
at scala.tools.nsc.Global$$anon$2.classPath(Global.scala:54)
- locked <0x00000000faba7ff0> (a scala.tools.nsc.Global$$anon$2)
at scala.tools.nsc.backend.JavaPlatform$class.rootLoader(JavaPlatform.scala:19)
at scala.tools.nsc.Global$$anon$2.rootLoader(Global.scala:54)
at scala.tools.nsc.Global$$anon$2.rootLoader(Global.scala:54)
at scala.tools.nsc.Global.rootLoader(Global.scala:57)
at scala.tools.nsc.symtab.Definitions$definitions$.RootClass(Definitions.scala:138)
- locked <0x00000000faba5620> (a scala.tools.nsc.symtab.Definitions$definitions$)
at scala.tools.nsc.symtab.Definitions$definitions$.EmptyPackage(Definitions.scala:141)
- locked <0x00000000faba5620> (a scala.tools.nsc.symtab.Definitions$definitions$)
at scala.tools.nsc.symtab.Definitions$definitions$.EmptyPackageClass(Definitions.scala:142)
- locked <0x00000000faba5620> (a scala.tools.nsc.symtab.Definitions$definitions$)
at scala.tools.nsc.symtab.Definitions$definitions$.init(Definitions.scala:804)
at scala.tools.nsc.Global$Run.<init>(Global.scala:693)
at scala.tools.nsc.interpreter.IMain.scala$tools$nsc$interpreter$IMain$$_initialize(IMain.scala:120)
at scala.tools.nsc.interpreter.IMain$$anonfun$initialize$1.apply$mcZ$sp(IMain.scala:154)
at scala.tools.nsc.interpreter.IMain$$anonfun$initialize$2.apply(IMain.scala:154)
at scala.tools.nsc.interpreter.IMain$$anonfun$initialize$2.apply(IMain.scala:154)
at scala.concurrent.ThreadRunner$$anon$2$$anonfun$run$2.apply(ThreadRunner.scala:45)
at scala.concurrent.ThreadRunner.scala$concurrent$ThreadRunner$$tryCatch(ThreadRunner.scala:31)
at scala.concurrent.ThreadRunner$$anon$2.run(ThreadRunner.scala:45)
at java.lang.Thread.run(Thread.java:679)
"Low Memory Detector" daemon prio=10 tid=0x00007f77680b9000 nid=0x1996 runnable [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"C2 CompilerThread1" daemon prio=10 tid=0x00007f77680b6800 nid=0x1995 runnable [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"C2 CompilerThread0" daemon prio=10 tid=0x00007f77680b3800 nid=0x1994 runnable [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"Signal Dispatcher" daemon prio=10 tid=0x00007f77680b2000 nid=0x1993 runnable [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"Finalizer" daemon prio=10 tid=0x00007f7768092800 nid=0x1992 in Object.wait() [0x00007f776e27c000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x00000000fb2d82d0> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:133)
- locked <0x00000000fb2d82d0> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:149)
at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:177)
"Reference Handler" daemon prio=10 tid=0x00007f7768091000 nid=0x1991 in Object.wait() [0x00007f776e37d000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x00000000fb2d8370> (a java.lang.ref.Reference$Lock)
at java.lang.Object.wait(Object.java:502)
at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:133)
- locked <0x00000000fb2d8370> (a java.lang.ref.Reference$Lock)
"main" prio=10 tid=0x0000000001db4800 nid=0x198d in Object.wait() [0x00007f7775067000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x00000000fab8d110> (a scala.concurrent.SyncVar)
at java.lang.Object.wait(Object.java:502)
at scala.concurrent.SyncVar.get(SyncVar.scala:22)
- locked <0x00000000fab8d110> (a scala.concurrent.SyncVar)
at scala.concurrent.ThreadRunner$$anonfun$submit$1.apply(ThreadRunner.scala:48)
at scala.Function0$class.apply$mcZ$sp(Function0.scala:34)
at scala.runtime.AbstractFunction0.apply$mcZ$sp(AbstractFunction0.scala:12)
at scala.tools.nsc.interpreter.IMain.global(IMain.scala:162)
- locked <0x00000000fb2d8580> (a scala.tools.nsc.interpreter.ILoop$ILoopInterpreter)
at scala.tools.nsc.interpreter.JLineCompletion.<init>(JLineCompletion.scala:17)
at scala.tools.nsc.interpreter.ILoop.chooseReader(ILoop.scala:791)
at scala.tools.nsc.interpreter.ILoop.process(ILoop.scala:807)
at scala.tools.nsc.MainGenericRunner.runTarget$1(MainGenericRunner.scala:67)
at scala.tools.nsc.MainGenericRunner.process(MainGenericRunner.scala:80)
at scala.tools.nsc.MainGenericRunner$.main(MainGenericRunner.scala:89)
at scala.tools.nsc.MainGenericRunner.main(MainGenericRunner.scala)
"VM Thread" prio=10 tid=0x00007f776808a000 nid=0x1990 runnable
"GC task thread#0 (ParallelGC)" prio=10 tid=0x0000000001dc0000 nid=0x198e runnable
"GC task thread#1 (ParallelGC)" prio=10 tid=0x0000000001dc1800 nid=0x198f runnable
"VM Periodic Task Thread" prio=10 tid=0x00007f77680bb800 nid=0x1997 waiting on condition
JNI global references: 1010

The startup time will be reduced in the upcoming 2.9.1 release. If yu want to you can already check it out by downloading the RC here :
http://www.scala-lang.org/downloads