jstack understanding - jboss5.x

I'm using JBoss 5.1.0 in a production environment and the shutdown time is quite long at times(>10 minutes). I've used jstack during one of the shutdown operations but I can't really understand its output. I'm attaching a snippet if there is some one who can help me.
Thank you.
"Thread-7326 (group:HornetQ-remoting-threads318818410-409317605)" prio=10 tid=0x000000004097b000 nid=0xbae waiting on condition [0x00007fc43f829000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000000c9d55b28> (a java.util.concurrent.SynchronousQueue$TransferStack)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:662)
"JBoss Shutdown Hook" daemon prio=10 tid=0x0000000041b59800 nid=0xba9 runnable [0x00007fc4530ca000]
java.lang.Thread.State: RUNNABLE
at java.lang.Throwable.fillInStackTrace(Native Method)
- locked <0x00000000f1e29d48> (a java.lang.NoSuchMethodException)
at java.lang.Throwable.<init>(Throwable.java:196)
at java.lang.Exception.<init>(Exception.java:41)
at java.lang.NoSuchMethodException.<init>(NoSuchMethodException.java:32)
at java.lang.Class.getDeclaredMethod(Class.java:1937)
at org.jboss.ejb3.interceptors.lang.SecurityActions$2.run(SecurityActions.java:58)
at org.jboss.ejb3.interceptors.lang.SecurityActions$2.run(SecurityActions.java:55)
at java.security.AccessController.doPrivileged(Native Method)
at org.jboss.ejb3.interceptors.lang.SecurityActions.getDeclaredMethod(SecurityActions.java:53)
at org.jboss.ejb3.interceptors.lang.ClassHelper.getDeclaredMethod(ClassHelper.java:165)
at org.jboss.ejb3.interceptors.lang.ClassHelper.getDeclaredMethod(ClassHelper.java:176)
at org.jboss.ejb3.interceptors.lang.ClassHelper.getDeclaredMethod(ClassHelper.java:176)
at org.jboss.ejb3.interceptors.lang.ClassHelper.getMethod(ClassHelper.java:138)
at org.jboss.ejb3.interceptors.lang.ClassHelper.isOverridden(ClassHelper.java:229)
at org.jboss.ejb3.interceptors.aop.LifecycleCallbacks.createLifecycleCallbackInterceptors(LifecycleCallbacks.java:99)
at org.jboss.ejb3.EJBContainer.invokeCallback(EJBContainer.java:1112)
at org.jboss.ejb3.EJBContainer.invokePreDestroy(EJBContainer.java:1149)
at org.jboss.ejb3.pool.AbstractPool.remove(AbstractPool.java:112)
at org.jboss.ejb3.InfinitePool.destroy(InfinitePool.java:44)
at org.jboss.ejb3.pool.ThreadlocalPool.destroy(ThreadlocalPool.java:71)
at org.jboss.ejb3.EJBContainer.lockedStop(EJBContainer.java:934)
at org.jboss.ejb3.session.SessionContainer.lockedStop(SessionContainer.java:276)
at org.jboss.ejb3.session.SessionSpecContainer.lockedStop(SessionSpecContainer.java:588)
at org.jboss.ejb3.stateless.StatelessContainer.lockedStop(StatelessContainer.java:221)
at org.jboss.ejb3.EJBContainer.stop(EJBContainer.java:923)
at sun.reflect.GeneratedMethodAccessor5513.invoke(Unknown Source)

Related

Failed to return from Redis call from Scala/spark, Shows some Deadlock in thread dump

I am new to scala and spark world, some where in scala code I am seeing invocation to Redis call via Redisson 3.9.1 to get keys data which is very few number of records and this leads me some deadlock as seen below trace. Could someone please acknowledge what could be issue that I can take a hit on.
Full thread dump OpenJDK 64-Bit Server VM (25.282-b08 mixed mode):
"Attach Listener" #124 daemon prio=9 os_prio=0 tid=0x00007fbde8002800 nid=0x494e1 waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"Keep-Alive-Timer" #123 daemon prio=8 os_prio=0 tid=0x00007fbd68021000 nid=0x493d1 waiting on condition [0x00007fbb2fffe000]
java.lang.Thread.State: TIMED_WAITING (sleeping)
at java.lang.Thread.sleep(Native Method)
at sun.net.www.http.KeepAliveCache.run(KeepAliveCache.java:172)
at java.lang.Thread.run(Thread.java:748)
"ForkJoinPool-1-worker-5" #117 daemon prio=5 os_prio=0 tid=0x00007fbc9c87b000 nid=0x475ba waiting on condition [0x00007fbb1dbfc000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00007fc68d00ed50> (a java.util.concurrent.CountDownLatch$Sync)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304)
at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:231)
at org.redisson.command.CommandAsyncService.get(CommandAsyncService.java:182)
at org.redisson.RedissonKeys$2.iterator(RedissonKeys.java:127)
at org.redisson.RedissonKeys$2.iterator(RedissonKeys.java:123)
at org.redisson.BaseIterator.hasNext(BaseIterator.java:54)
at scala.collection.convert.Wrappers$JIteratorWrapper.hasNext(Wrappers.scala:42)
at scala.collection.Iterator$class.foreach(Iterator.scala:893)
at scala.collection.AbstractIterator.foreach(Iterator.scala:1336)
at scala.collection.IterableLike$class.foreach(IterableLike.scala:72)
at scala.collection.AbstractIterable.foreach(Iterable.scala:54)
at scala.collection.TraversableLike$class.filterImpl(TraversableLike.scala:247)
at scala.collection.TraversableLike$class.filterNot(TraversableLike.scala:267)
at scala.collection.AbstractTraversable.filterNot(Traversable.scala:104)
at com.mycomosi.eaa.common.infrastructure.topology.store.TopologyStoreEntityService$$anonfun$getTopologyInstanceIdsExcludingVersion$1$$anonfun$apply$7.apply(TopologyStoreEntityService.scala:73)
at com.mycomosi.eaa.common.infrastructure.topology.store.TopologyStoreEntityService$$anonfun$getTopologyInstanceIdsExcludingVersion$1$$anonfun$apply$7.apply(TopologyStoreEntityService.scala:70)
at scala.collection.immutable.Set$Set1.foreach(Set.scala:94)
at com.mycomosi.eaa.common.infrastructure.topology.store.TopologyStoreEntityService$$anonfun$getTopologyInstanceIdsExcludingVersion$1.apply(TopologyStoreEntityService.scala:70)
at com.mycomosi.eaa.common.infrastructure.topology.store.TopologyStoreEntityService$$anonfun$getTopologyInstanceIdsExcludingVersion$1.apply(TopologyStoreEntityService.scala:69)
at scala.collection.Iterator$class.foreach(Iterator.scala:893)
at scala.collection.AbstractIterator.foreach(Iterator.scala:1336)

Actual cause of this was slow response from keyByPattern query from redisson client. When there is huge volume of data, default keyByPattern query search with count of 10 and it build iterator for complete volume of keys. If instead of default provide some higher count value it can perform much better.
I also have referenced this issue at github plesae refer
https://github.com/redisson/redisson/issues/4635

Scala/Akka/Dropwizard Metrics app hanging for 10s before shutting down

I have a Scala 2.11 app that uses Akka. The GitHub repo for it is here. To reproduce what I'm seeing, just clone it, build it via ./gradlew fullBuild and then run it via java -jar build/libs/akka-scala-troubleshooting.jar (its a self-contained executable JAR). But essentially, I'm noticing that when I kill the running JVM process via Ctrl+C that it takes exactly 10 seconds for the JAR to shut down!
IIRC, Ctrl+C should issue a SIGKILL to the JVM process, so how it lives for another 10 seconds thereafter is blowing my mind! Even if I'm recalling that incorrectly, I'd still like to know why my app isn't shutting down immediately (it's causing problems by not doing so).
I'm using Dropwizard (formerly Coda Hale) Metrics via the Scala Metrics library. From what I can tell there might be a scheduled reporter daemon still alive that isn't shutting down right away, even when the JVM receives the SIGKILL (or whatever Ctrl+C is sending). Here's the main Driver (app entry point):
object Driver extends App {
println("Starting upp the app...")
//SLF4JBridgeHandler.install()
lazy val metricRegistry = new MetricRegistry()
ConsoleReporter
.forRegistry(metricRegistry)
.convertRatesTo(TimeUnit.SECONDS)
.convertDurationsTo(TimeUnit.MILLISECONDS)
.build()
.start(15, TimeUnit.SECONDS)
lazy val cortex = ActorSystem("cortex")
cortex.registerOnTermination {
System.exit(0)
}
val master = cortex.actorOf(Props[Master], name = "Master")
println("About to fire a StartUp message at Master...")
master ! StartUp
println("Fired! Actor system is spinning up...")
}
When you run the JAR, you'll see this output to the logs:
java -jar build/libs/akka-scala-troubleshooting.jar
Starting upp the app...
About to fire a StartUp message at Master...
Fired! Actor system is spinning up...
Master has received a command to start up the actor system!
Child will make it happen!
At this point it will just idle and do nothing -- that's fine and intended! But when I hit Ctrl+C on the terminal, I immediately get this log INFO:
^C[INFO] [08/03/2017 06:15:57.236] [Thread-0] [CoordinatedShutdown(akka://cortex)] Starting coordinated shutdown from JVM shutdown hook
Then 10 seconds goes by (exactly 10s every time), then finally a log WARNing:
[WARN] [08/03/2017 06:16:07.248] [Thread-0] [CoordinatedShutdown(akka://cortex)] CoordinatedShutdown from JVM shutdown failed: Futures timed out after [10000 milliseconds]
Any idea what could be going on here?
Update:
I did a thread dump:
2017-08-03 14:08:58
Full thread dump OpenJDK 64-Bit Server VM (25.131-b11 mixed mode):
"cortex-akka.actor.default-dispatcher-5" #19 prio=5 os_prio=0 tid=0x00007fcba8009000 nid=0x547d waiting on condition [0x00007fcc16e6f000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x000000076ff34e28> (a akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
at akka.dispatch.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075)
at akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
at akka.dispatch.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
Locked ownable synchronizers:
- None
"Thread-0" #15 prio=5 os_prio=0 tid=0x00007fcba0001000 nid=0x547c waiting on condition [0x00007fcc17170000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x0000000770b33920> (a scala.concurrent.impl.Promise$CompletionLatch)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedNanos(AbstractQueuedSynchronizer.java:1037)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(AbstractQueuedSynchronizer.java:1328)
at scala.concurrent.impl.Promise$DefaultPromise.tryAwait(Promise.scala:212)
at scala.concurrent.impl.Promise$DefaultPromise.ready(Promise.scala:222)
at scala.concurrent.impl.Promise$DefaultPromise.ready(Promise.scala:157)
at scala.concurrent.Await$$anonfun$ready$1.apply(package.scala:169)
at scala.concurrent.Await$$anonfun$ready$1.apply(package.scala:169)
at scala.concurrent.BlockContext$DefaultBlockContext$.blockOn(BlockContext.scala:53)
at scala.concurrent.Await$.ready(package.scala:169)
at akka.actor.CoordinatedShutdown$$anonfun$initJvmHook$1.apply(CoordinatedShutdown.scala:161)
at akka.actor.CoordinatedShutdown$$anon$2.run(CoordinatedShutdown.scala:446)
Locked ownable synchronizers:
- None
"SIGTERM handler" #18 daemon prio=9 os_prio=0 tid=0x00007fcbdc002800 nid=0x547b in Object.wait() [0x00007fcc17271000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x00000007707bb150> (a akka.actor.CoordinatedShutdown$$anon$2)
at java.lang.Thread.join(Thread.java:1252)
- locked <0x00000007707bb150> (a akka.actor.CoordinatedShutdown$$anon$2)
at java.lang.Thread.join(Thread.java:1326)
at java.lang.ApplicationShutdownHooks.runHooks(ApplicationShutdownHooks.java:106)
at java.lang.ApplicationShutdownHooks$1.run(ApplicationShutdownHooks.java:46)
at java.lang.Shutdown.runHooks(Shutdown.java:123)
at java.lang.Shutdown.sequence(Shutdown.java:167)
at java.lang.Shutdown.exit(Shutdown.java:212)
- locked <0x00000007707bb5d0> (a java.lang.Class for java.lang.Shutdown)
at java.lang.Terminator$1.handle(Terminator.java:52)
at sun.misc.Signal$1.run(Signal.java:212)
at java.lang.Thread.run(Thread.java:748)
Locked ownable synchronizers:
- None
"Attach Listener" #17 daemon prio=9 os_prio=0 tid=0x00007fcbdc001000 nid=0x545f waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
Locked ownable synchronizers:
- None
"DestroyJavaVM" #16 prio=5 os_prio=0 tid=0x00007fcc3800c000 nid=0x5427 waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
Locked ownable synchronizers:
- None
"cortex-akka.actor.default-dispatcher-4" #14 prio=5 os_prio=0 tid=0x00007fcbb0010000 nid=0x5445 waiting for monitor entry [0x00007fcc17473000]
java.lang.Thread.State: BLOCKED (on object monitor)
at java.lang.Shutdown.exit(Shutdown.java:212)
- waiting to lock <0x00000007707bb5d0> (a java.lang.Class for java.lang.Shutdown)
at java.lang.Runtime.exit(Runtime.java:109)
at java.lang.System.exit(System.java:971)
at hotmeatballsoup.Driver$$anonfun$1.apply$mcV$sp(Driver.scala:24)
at hotmeatballsoup.Driver$$anonfun$1.apply(Driver.scala:24)
at hotmeatballsoup.Driver$$anonfun$1.apply(Driver.scala:24)
at akka.actor.ActorSystemImpl$$anon$3.run(ActorSystem.scala:810)
at akka.actor.ActorSystemImpl$TerminationCallbacks$$anonfun$addRec$1$1.applyOrElse(ActorSystem.scala:987)
at akka.actor.ActorSystemImpl$TerminationCallbacks$$anonfun$addRec$1$1.applyOrElse(ActorSystem.scala:987)
at scala.concurrent.Future$$anonfun$andThen$1.apply(Future.scala:436)
at scala.concurrent.Future$$anonfun$andThen$1.apply(Future.scala:435)
at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:36)
at akka.dispatch.BatchingExecutor$AbstractBatch.processBatch(BatchingExecutor.scala:55)
at akka.dispatch.BatchingExecutor$BlockableBatch$$anonfun$run$1.apply$mcV$sp(BatchingExecutor.scala:91)
at akka.dispatch.BatchingExecutor$BlockableBatch$$anonfun$run$1.apply(BatchingExecutor.scala:91)
at akka.dispatch.BatchingExecutor$BlockableBatch$$anonfun$run$1.apply(BatchingExecutor.scala:91)
at scala.concurrent.BlockContext$.withBlockContext(BlockContext.scala:72)
at akka.dispatch.BatchingExecutor$BlockableBatch.run(BatchingExecutor.scala:90)
at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:38)
at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(ForkJoinExecutorConfigurator.scala:43)
at akka.dispatch.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
at akka.dispatch.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
at akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
at akka.dispatch.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
Locked ownable synchronizers:
- None
"cortex-akka.actor.default-dispatcher-3" #13 prio=5 os_prio=0 tid=0x00007fcc386e8800 nid=0x5444 waiting on condition [0x00007fcc17574000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x000000076ff34e28> (a akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
at akka.dispatch.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075)
at akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
at akka.dispatch.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
Locked ownable synchronizers:
- None
"cortex-akka.actor.default-dispatcher-2" #12 prio=5 os_prio=0 tid=0x00007fcc386db000 nid=0x5443 waiting on condition [0x00007fcc17675000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x000000076ff34e28> (a akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
at akka.dispatch.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075)
at akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
at akka.dispatch.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
Locked ownable synchronizers:
- None
"cortex-scheduler-1" #11 prio=5 os_prio=0 tid=0x00007fcc38645800 nid=0x5442 waiting on condition [0x00007fcc17976000]
java.lang.Thread.State: TIMED_WAITING (sleeping)
at java.lang.Thread.sleep(Native Method)
at akka.actor.LightArrayRevolverScheduler.waitNanos(LightArrayRevolverScheduler.scala:85)
at akka.actor.LightArrayRevolverScheduler$$anon$4.nextTick(LightArrayRevolverScheduler.scala:265)
at akka.actor.LightArrayRevolverScheduler$$anon$4.run(LightArrayRevolverScheduler.scala:235)
at java.lang.Thread.run(Thread.java:748)
Locked ownable synchronizers:
- None
"metrics-console-reporter-1-thread-1" #10 daemon prio=5 os_prio=0 tid=0x00007fcc384ee800 nid=0x5441 waiting on condition [0x00007fcc17c77000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x000000076e328370> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Locked ownable synchronizers:
- None
Note that 2 threads are trying to call Shutdown.exit...

The 10 seconds is defined here in the configuration:
actor-system-terminate {
timeout = 10 s
depends-on = [before-actor-system-terminate]
}
When system.terminate() is called, CoordinatedShutdown by default is initiated. Notice that this method returns a Future, meaning that termination will happen asynchronously. By default, there is a 10-second timeout for the actor system to shut down.
In your driver, you call registerOnTermination. The Scaladoc for this method states:
Note that ActorSystem will not terminate until all the registered callbacks are finished.
Your registered callback invokes System.exit, and you're getting the timeout warning because the actor system is unable to properly shut down within the 10-second window. The System.exit call is probably interfering with the actor system's coordinated shutdown. Removing the System.exit call in registerOnTermination, or removing the call to registerOnTermination altogether, will remove the warning.

Output of jStack

We are using Adobe AEM on a windows machine.
Our Author instances get very slow and to investigate the same we try to take thread dumps by using the command
jstack.exe pid > output file
The problem is that the output of this command is not same always.
Below are the two different output obtained.
This one hardly gives any info as to which thread is blocked
Deadlock Detection:
No deadlocks found.
Thread 636: (state = BLOCKED)
- java.lang.Object.wait(long) #bci=0 (Compiled frame; information may be imprecise)
- java.lang.Object.wait() #bci=2, line=503 (Interpreted frame)
- org.apache.sling.event.impl.jobs.queues.TopicRoundRobinJobQueue.take() #bci=32, line=98 (Interpreted frame)
- org.apache.sling.event.impl.jobs.queues.AbstractJobQueue.runJobQueue() #bci=14, line=454 (Compiled frame)
- org.apache.sling.event.impl.jobs.queues.AbstractJobQueue.access$000(org.apache.sling.event.impl.jobs.queues.AbstractJobQueue) #bci=1, line=50 (Interpreted frame)
- org.apache.sling.event.impl.jobs.queues.AbstractJobQueue$1.run() #bci=56, line=137 (Interpreted frame)
- java.lang.Thread.run() #bci=11, line=722 (Interpreted frame)
Thread 635: (state = BLOCKED)
- sun.misc.Unsafe.park(boolean, long) #bci=0 (Compiled frame; information may be imprecise)
- java.util.concurrent.locks.LockSupport.park(java.lang.Object) #bci=14, line=186 (Compiled frame)
- java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await() #bci=42, line=2043 (Compiled frame)
- java.util.concurrent.LinkedBlockingQueue.take() #bci=29, line=442 (Compiled frame)
- java.util.concurrent.ThreadPoolExecutor.getTask() #bci=156, line=1068 (Compiled frame)
- java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker) #bci=26, line=1130 (Interpreted frame)
- java.util.concurrent.ThreadPoolExecutor$Worker.run() #bci=5, line=615 (Interpreted frame)
- java.lang.Thread.run() #bci=11, line=722 (Interpreted frame)
Thread 634: (state = BLOCKED)
- sun.misc.Unsafe.park(boolean, long) #bci=0 (Compiled frame; information may be imprecise)
- java.util.concurrent.locks.LockSupport.park(java.lang.Object) #bci=14, line=186 (Compiled frame)
- java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await() #bci=42, line=2043 (Compiled frame)
- java.util.concurrent.LinkedBlockingQueue.take() #bci=29, line=442 (Compiled frame)
- java.util.concurrent.ThreadPoolExecutor.getTask() #bci=156, line=1068 (Compiled frame)
- java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker) #bci=26, line=1130 (Interpreted frame)
- java.util.concurrent.ThreadPoolExecutor$Worker.run() #bci=5, line=615 (Interpreted frame)
- java.lang.Thread.run() #bci=11, line=722 (Interpreted frame)
Thread 633: (state = BLOCKED)
- sun.misc.Unsafe.park(boolean, long) #bci=0 (Compiled frame; information may be imprecise)
- java.util.concurrent.locks.LockSupport.park(java.lang.Object) #bci=14, line=186 (Compiled frame)
- java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await() #bci=42, line=2043 (Compiled frame)
- java.util.concurrent.LinkedBlockingQueue.take() #bci=29, line=442 (Compiled frame)
- java.util.concurrent.ThreadPoolExecutor.getTask() #bci=156, line=1068 (Compiled frame)
- java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker) #bci=26, line=1130 (Interpreted frame)
- java.util.concurrent.ThreadPoolExecutor$Worker.run() #bci=5, line=615 (Interpreted frame)
- java.lang.Thread.run() #bci=11, line=722 (Interpreted frame)
And below is the second output
2015-03-12 11:53:07
Full thread dump Java HotSpot(TM) 64-Bit Server VM (23.7-b01 mixed mode):
"Thread-49205" daemon prio=6 tid=0x000000002cc69000 nid=0x1c48 in Object.wait() [0x000000004925f000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at EDU.oswego.cs.dl.util.concurrent.LinkedQueue.poll(Unknown Source)
- locked <0x000000061148ca10> (a java.lang.Object)
at EDU.oswego.cs.dl.util.concurrent.PooledExecutor.getTask(Unknown Source)
at EDU.oswego.cs.dl.util.concurrent.PooledExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Thread.java:722)
"Thread-49203" daemon prio=6 tid=0x000000002cc69800 nid=0x1f90 in Object.wait() [0x0000000048e5f000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at EDU.oswego.cs.dl.util.concurrent.LinkedQueue.poll(Unknown Source)
- locked <0x000000061148ca10> (a java.lang.Object)
at EDU.oswego.cs.dl.util.concurrent.PooledExecutor.getTask(Unknown Source)
at EDU.oswego.cs.dl.util.concurrent.PooledExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Thread.java:722)
"Thread-49202" daemon prio=6 tid=0x000000002cc6c000 nid=0x9c0 waiting for monitor entry [0x000000004835f000]
java.lang.Thread.State: BLOCKED (on object monitor)
at com.day.cq.audit.impl.DefaultAuditLog.internalAdd(DefaultAuditLog.java:208)
- waiting to lock <0x0000000625c66710> (a com.day.cq.audit.impl.DefaultAuditLog)
at com.day.cq.audit.impl.DefaultAuditLog.add(DefaultAuditLog.java:203)
at com.day.cq.wcm.core.impl.event.PageEventAuditListener.handleEvent(PageEventAuditListener.java:91)
at org.apache.felix.eventadmin.impl.handler.EventHandlerProxy.sendEvent(EventHandlerProxy.java:315)
at org.apache.felix.eventadmin.impl.tasks.SyncDeliverTasks.execute(SyncDeliverTasks.java:117)
at org.apache.felix.eventadmin.impl.tasks.AsyncDeliverTasks$TaskExecuter.run(AsyncDeliverTasks.java:109)
at EDU.oswego.cs.dl.util.concurrent.PooledExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Thread.java:722)
"Thread-49200" daemon prio=6 tid=0x000000002cc6d800 nid=0x1a34 in Object.wait() [0x0000000048d5f000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at EDU.oswego.cs.dl.util.concurrent.LinkedQueue.poll(Unknown Source)
- locked <0x000000061148ca10> (a java.lang.Object)
at EDU.oswego.cs.dl.util.concurrent.PooledExecutor.getTask(Unknown Source)
at EDU.oswego.cs.dl.util.concurrent.PooledExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Thread.java:722)
"Thread-49168" daemon prio=6 tid=0x000000002cc67800 nid=0x1ca0 in Object.wait() [0x000000004ac5f000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at EDU.oswego.cs.dl.util.concurrent.LinkedQueue.poll(Unknown Source)
- locked <0x000000061148ca10> (a java.lang.Object)
at EDU.oswego.cs.dl.util.concurrent.PooledExecutor.getTask(Unknown Source)
at EDU.oswego.cs.dl.util.concurrent.PooledExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Thread.java:722)
"Thread-49166" daemon prio=6 tid=0x000000002cc6c800 nid=0x1ed0 waiting for monitor entry [0x000000004965f000]
java.lang.Thread.State: BLOCKED (on object monitor)
at com.day.cq.audit.impl.DefaultAuditLog.internalAdd(DefaultAuditLog.java:208)
- waiting to lock <0x0000000625c66710> (a com.day.cq.audit.impl.DefaultAuditLog)
at com.day.cq.audit.impl.DefaultAuditLog.add(DefaultAuditLog.java:203)
at com.day.cq.wcm.core.impl.event.PageEventAuditListener.handleEvent(PageEventAuditListener.java:91)
at org.apache.felix.eventadmin.impl.handler.EventHandlerProxy.sendEvent(EventHandlerProxy.java:315)
at org.apache.felix.eventadmin.impl.tasks.SyncDeliverTasks.execute(SyncDeliverTasks.java:117)
at org.apache.felix.eventadmin.impl.tasks.AsyncDeliverTasks$TaskExecuter.run(AsyncDeliverTasks.java:109)
at EDU.oswego.cs.dl.util.concurrent.PooledExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Thread.java:722)
"Thread-49121" daemon prio=6 tid=0x000000002cc6f000 nid=0x19f8 waiting for monitor entry [0x000000004935f000]
java.lang.Thread.State: BLOCKED (on object monitor)
at com.day.cq.audit.impl.DefaultAuditLog.internalAdd(DefaultAuditLog.java:208)
- waiting to lock <0x0000000625c66710> (a com.day.cq.audit.impl.DefaultAuditLog)
at com.day.cq.audit.impl.DefaultAuditLog.add(DefaultAuditLog.java:203)
at com.day.cq.wcm.core.impl.event.PageEventAuditListener.handleEvent(PageEventAuditListener.java:91)
at org.apache.felix.eventadmin.impl.handler.EventHandlerProxy.sendEvent(EventHandlerProxy.java:315)
at org.apache.felix.eventadmin.impl.tasks.SyncDeliverTasks.execute(SyncDeliverTasks.java:117)
at org.apache.felix.eventadmin.impl.tasks.AsyncDeliverTasks$TaskExecuter.run(AsyncDeliverTasks.java:109)
at EDU.oswego.cs.dl.util.concurrent.PooledExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Thread.java:722)
"Thread-49111" daemon prio=6 tid=0x000000002cc6b000 nid=0x21d8 waiting for monitor entry [0x0000000048f5f000]
java.lang.Thread.State: BLOCKED (on object monitor)
at com.day.cq.audit.impl.DefaultAuditLog.internalAdd(DefaultAuditLog.java:208)
- waiting to lock <0x0000000625c66710> (a com.day.cq.audit.impl.DefaultAuditLog)
at com.day.cq.audit.impl.DefaultAuditLog.add(DefaultAuditLog.java:203)
at com.day.cq.wcm.core.impl.event.PageEventAuditListener.handleEvent(PageEventAuditListener.java:91)
at org.apache.felix.eventadmin.impl.handler.EventHandlerProxy.sendEvent(EventHandlerProxy.java:315)
at org.apache.felix.eventadmin.impl.tasks.SyncDeliverTasks.execute(SyncDeliverTasks.java:117)
at org.apache.felix.eventadmin.impl.tasks.AsyncDeliverTasks$TaskExecuter.run(AsyncDeliverTasks.java:109)
at EDU.oswego.cs.dl.util.concurrent.PooledExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Thread.java:722)
"Thread-49109" daemon prio=6 tid=0x000000002cc6a800 nid=0x18ac waiting for monitor entry [0x000000004875f000]
java.lang.Thread.State: BLOCKED (on object monitor)
at com.day.cq.audit.impl.DefaultAuditLog.internalAdd(DefaultAuditLog.java:208)
- waiting to lock <0x0000000625c66710> (a com.day.cq.audit.impl.DefaultAuditLog)
at com.day.cq.audit.impl.DefaultAuditLog.add(DefaultAuditLog.java:203)
at com.day.cq.wcm.core.impl.event.PageEventAuditListener.handleEvent(PageEventAuditListener.java:91)
at org.apache.felix.eventadmin.impl.handler.EventHandlerProxy.sendEvent(EventHandlerProxy.java:315)
at org.apache.felix.eventadmin.impl.tasks.SyncDeliverTasks.execute(SyncDeliverTasks.java:117)
at org.apache.felix.eventadmin.impl.tasks.AsyncDeliverTasks$TaskExecuter.run(AsyncDeliverTasks.java:109)
at EDU.oswego.cs.dl.util.concurrent.PooledExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Thread.java:722)
"Thread-48974" daemon prio=6 tid=0x000000002cc68000 nid=0x1f84 in Object.wait() [0x000000004945e000]
java.lang.Thread.State: BLOCKED (on object monitor)
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:503)
at EDU.oswego.cs.dl.util.concurrent.WriterPreferenceReadWriteLock$WriterLock.acquire(Unknown Source)
- locked <0x0000000613135e20> (a EDU.oswego.cs.dl.util.concurrent.WriterPreferenceReadWriteLock$WriterLock)
at org.apache.jackrabbit.core.journal.AbstractJournal.internalLockAndSync(AbstractJournal.java:295)
at org.apache.jackrabbit.core.journal.AbstractJournal.lockAndSync(AbstractJournal.java:284)
at org.apache.jackrabbit.core.journal.DefaultRecordProducer.append(DefaultRecordProducer.java:51)
at org.apache.jackrabbit.core.cluster.ClusterNode$WorkspaceUpdateChannel.updateCreated(ClusterNode.java:599)
at org.apache.jackrabbit.core.state.SharedItemStateManager$Update.begin(SharedItemStateManager.java:565)
at org.apache.jackrabbit.core.state.SharedItemStateManager.beginUpdate(SharedItemStateManager.java:1462)
at org.apache.jackrabbit.core.state.SharedItemStateManager.update(SharedItemStateManager.java:1492)
at org.apache.jackrabbit.core.state.LocalItemStateManager.update(LocalItemStateManager.java:400)
at org.apache.jackrabbit.core.state.XAItemStateManager.update(XAItemStateManager.java:354)
at org.apache.jackrabbit.core.state.LocalItemStateManager.update(LocalItemStateManager.java:375)
at org.apache.jackrabbit.core.state.SessionItemStateManager.update(SessionItemStateManager.java:275)
at org.apache.jackrabbit.core.ItemSaveOperation.perform(ItemSaveOperation.java:258)
at org.apache.jackrabbit.core.session.SessionState.perform(SessionState.java:216)
at org.apache.jackrabbit.core.ItemImpl.perform(ItemImpl.java:91)
at org.apache.jackrabbit.core.ItemImpl.save(ItemImpl.java:329)
at org.apache.jackrabbit.core.session.SessionSaveOperation.perform(SessionSaveOperation.java:64)
at org.apache.jackrabbit.core.session.SessionState.perform(SessionState.java:216)
at org.apache.jackrabbit.core.SessionImpl.perform(SessionImpl.java:361)
at org.apache.jackrabbit.core.SessionImpl.save(SessionImpl.java:812)
at com.day.crx.core.CRXSessionImpl.save(CRXSessionImpl.java:131)
at com.day.cq.audit.impl.DefaultAuditLog.internalAdd(DefaultAuditLog.java:215)
- locked <0x0000000625c66710> (a com.day.cq.audit.impl.DefaultAuditLog)
at com.day.cq.audit.impl.DefaultAuditLog.add(DefaultAuditLog.java:203)
at com.day.cq.wcm.core.impl.event.PageEventAuditListener.handleEvent(PageEventAuditListener.java:91)
at org.apache.felix.eventadmin.impl.handler.EventHandlerProxy.sendEvent(EventHandlerProxy.java:315)
at org.apache.felix.eventadmin.impl.tasks.SyncDeliverTasks.execute(SyncDeliverTasks.java:117)
at org.apache.felix.eventadmin.impl.tasks.AsyncDeliverTasks$TaskExecuter.run(AsyncDeliverTasks.java:109)
at EDU.oswego.cs.dl.util.concurrent.PooledExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Thread.java:722)
"Apache Sling Job Queue CQ Workflows" daemon prio=6 tid=0x000000002cc6e000 nid=0x1db8 waiting for monitor entry [0x0000000048b5f000]
java.lang.Thread.State: BLOCKED (on object monitor)
at org.apache.sling.event.impl.jobs.jcr.PersistenceHandler.lock(PersistenceHandler.java:1080)
- waiting to lock <0x0000000613341df0> (a java.lang.Object)
at org.apache.sling.event.impl.jobs.jcr.JCRJobEvent.lock(JCRJobEvent.java:39)
at org.apache.sling.event.impl.jobs.queues.AbstractJobQueue.executeJob(AbstractJobQueue.java:473)
at org.apache.sling.event.impl.jobs.queues.AbstractParallelJobQueue.start(AbstractParallelJobQueue.java:64)
at org.apache.sling.event.impl.jobs.queues.AbstractJobQueue.runJobQueue(AbstractJobQueue.java:461)
at org.apache.sling.event.impl.jobs.queues.AbstractJobQueue.access$000(AbstractJobQueue.java:50)
at org.apache.sling.event.impl.jobs.queues.AbstractJobQueue$1.run(AbstractJobQueue.java:137)
at java.lang.Thread.run(Thread.java:722)
"Apache Sling Job Queue CQ Workflows" daemon prio=6 tid=0x000000002cc66000 nid=0x1b50 in Object.wait() [0x000000004ab5f000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:503)
at org.apache.sling.event.impl.jobs.queues.TopicRoundRobinJobQueue.take(TopicRoundRobinJobQueue.java:98)
- locked <0x000000066bdaf770> (a java.util.HashMap)
at org.apache.sling.event.impl.jobs.queues.AbstractJobQueue.runJobQueue(AbstractJobQueue.java:454)
at org.apache.sling.event.impl.jobs.queues.AbstractJobQueue.access$000(AbstractJobQueue.java:50)
at org.apache.sling.event.impl.jobs.queues.AbstractJobQueue$1.run(AbstractJobQueue.java:137)
at java.lang.Thread.run(Thread.java:722)
"Apache Sling Job Queue com/day/cq/replication/job/flush8" daemon prio=6 tid=0x0000000020dc5000 nid=0x2070 in Object.wait() [0x000000004aa5f000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:503)
at org.apache.sling.event.impl.jobs.queues.OrderedJobQueue.start(OrderedJobQueue.java:72)
- locked <0x000000066ab02158> (a java.lang.Object)
at org.apache.sling.event.impl.jobs.queues.AbstractJobQueue.runJobQueue(AbstractJobQueue.java:461)
at org.apache.sling.event.impl.jobs.queues.AbstractJobQueue.access$000(AbstractJobQueue.java:50)
at org.apache.sling.event.impl.jobs.queues.AbstractJobQueue$1.run(AbstractJobQueue.java:137)
at java.lang.Thread.run(Thread.java:722)
"Apache Sling Job Queue com/day/cq/replication/job/flush7" daemon prio=6 tid=0x0000000020dc6800 nid=0x15d0 waiting for monitor entry [0x000000004a15f000]
java.lang.Thread.State: BLOCKED (on object monitor)
at org.apache.sling.event.impl.jobs.jcr.PersistenceHandler.lock(PersistenceHandler.java:1080)
- waiting to lock <0x0000000613341df0> (a java.lang.Object)
at org.apache.sling.event.impl.jobs.jcr.JCRJobEvent.lock(JCRJobEvent.java:39)
at org.apache.sling.event.impl.jobs.queues.AbstractJobQueue.executeJob(AbstractJobQueue.java:473)
at org.apache.sling.event.impl.jobs.queues.OrderedJobQueue.start(OrderedJobQueue.java:67)
- locked <0x000000066ab004d8> (a java.lang.Object)
at org.apache.sling.event.impl.jobs.queues.AbstractJobQueue.runJobQueue(AbstractJobQueue.java:461)
at org.apache.sling.event.impl.jobs.queues.AbstractJobQueue.access$000(AbstractJobQueue.java:50)
at org.apache.sling.event.impl.jobs.queues.AbstractJobQueue$1.run(AbstractJobQueue.java:137)
at java.lang.Thread.run(Thread.java:722)
"Apache Sling Job Queue com/day/cq/replication/job/flush6" daemon prio=6 tid=0x0000000020dbb800 nid=0x1414 waiting for monitor entry [0x000000004a05f000]
java.lang.Thread.State: BLOCKED (on object monitor)
at org.apache.sling.event.impl.jobs.jcr.PersistenceHandler.lock(PersistenceHandler.java:1080)
- waiting to lock <0x0000000613341df0> (a java.lang.Object)
at org.apache.sling.event.impl.jobs.jcr.JCRJobEvent.lock(JCRJobEvent.java:39)
at org.apache.sling.event.impl.jobs.queues.AbstractJobQueue.executeJob(AbstractJobQueue.java:473)
at org.apache.sling.event.impl.jobs.queues.OrderedJobQueue.start(OrderedJobQueue.java:67)
- locked <0x000000066ab00218> (a java.lang.Object)
at org.apache.sling.event.impl.jobs.queues.AbstractJobQueue.runJobQueue(AbstractJobQueue.java:461)
at org.apache.sling.event.impl.jobs.queues.AbstractJobQueue.access$000(AbstractJobQueue.java:50)
at org.apache.sling.event.impl.jobs.queues.AbstractJobQueue$1.run(AbstractJobQueue.java:137)
at java.lang.Thread.run(Thread.java:722)
"Apache Sling Job Queue com/day/cq/replication/job/flush5" daemon prio=6 tid=0x0000000020dcc800 nid=0x1d18 in Object.wait() [0x0000000049f5f000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:503)
at org.apache.sling.event.impl.jobs.queues.OrderedJobQueue.start(OrderedJobQueue.java:72)
- locked <0x000000066aafff58> (a java.lang.Object)
at org.apache.sling.event.impl.jobs.queues.AbstractJobQueue.runJobQueue(AbstractJobQueue.java:461)
at org.apache.sling.event.impl.jobs.queues.AbstractJobQueue.access$000(AbstractJobQueue.java:50)
at org.apache.sling.event.impl.jobs.queues.AbstractJobQueue$1.run(AbstractJobQueue.java:137)
at java.lang.Thread.run(Thread.java:722)
"Apache Sling Job Queue com/day/cq/replication/job/flush4" daemon prio=6 tid=0x0000000020db8000 nid=0x1230 in Object.wait() [0x0000000049e5f000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:503)
at org.apache.sling.event.impl.jobs.queues.OrderedJobQueue.start(OrderedJobQueue.java:72)
- locked <0x000000066ab44ea8> (a java.lang.Object)
at org.apache.sling.event.impl.jobs.queues.AbstractJobQueue.runJobQueue(AbstractJobQueue.java:461)
at org.apache.sling.event.impl.jobs.queues.AbstractJobQueue.access$000(AbstractJobQueue.java:50)
at org.apache.sling.event.impl.jobs.queues.AbstractJobQueue$1.run(AbstractJobQueue.java:137)
at java.lang.Thread.run(Thread.java:722)
"Apache Sling Job Queue com/day/cq/replication/job/flush3" daemon prio=6 tid=0x0000000020dc2000 nid=0x1b6c waiting for monitor entry [0x0000000049d5f000]
java.lang.Thread.State: BLOCKED (on object monitor)
at org.apache.sling.event.impl.jobs.jcr.PersistenceHandler.lock(PersistenceHandler.java:1080)
- waiting to lock <0x0000000613341df0> (a java.lang.Object)
at org.apache.sling.event.impl.jobs.jcr.JCRJobEvent.lock(JCRJobEvent.java:39)
at org.apache.sling.event.impl.jobs.queues.AbstractJobQueue.executeJob(AbstractJobQueue.java:473)
at org.apache.sling.event.impl.jobs.queues.OrderedJobQueue.start(OrderedJobQueue.java:67)
- locked <0x000000066ab451a8> (a java.lang.Object)
at org.apache.sling.event.impl.jobs.queues.AbstractJobQueue.runJobQueue(AbstractJobQueue.java:461)
at org.apache.sling.event.impl.jobs.queues.AbstractJobQueue.access$000(AbstractJobQueue.java:50)
at org.apache.sling.event.impl.jobs.queues.AbstractJobQueue$1.run(AbstractJobQueue.java:137)
at java.lang.Thread.run(Thread.java:722)
"Apache Sling Job Queue com/day/cq/replication/job/flush2" daemon prio=6 tid=0x0000000020db6800 nid=0x1fb0 in Object.wait() [0x0000000049c5f000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:503)
at org.apache.sling.event.impl.jobs.queues.OrderedJobQueue.start(OrderedJobQueue.java:72)
- locked <0x000000066ab01828> (a java.lang.Object)
at org.apache.sling.event.impl.jobs.queues.AbstractJobQueue.runJobQueue(AbstractJobQueue.java:461)
at org.apache.sling.event.impl.jobs.queues.AbstractJobQueue.access$000(AbstractJobQueue.java:50)
at org.apache.sling.event.impl.jobs.queues.AbstractJobQueue$1.run(AbstractJobQueue.java:137)
at java.lang.Thread.run(Thread.java:722)
"Apache Sling Job Queue com/day/cq/replication/job/flush12" daemon prio=6 tid=0x0000000020dc0000 nid=0x1594 waiting for monitor entry [0x0000000049b5f000]
java.lang.Thread.State: BLOCKED (on object monitor)
at org.apache.sling.event.impl.jobs.jcr.PersistenceHandler.lock(PersistenceHandler.java:1080)
- waiting to lock <0x0000000613341df0> (a java.lang.Object)
at org.apache.sling.event.impl.jobs.jcr.JCRJobEvent.lock(JCRJobEvent.java:39)
at org.apache.sling.event.impl.jobs.queues.AbstractJobQueue.executeJob(AbstractJobQueue.java:473)
at org.apache.sling.event.impl.jobs.queues.OrderedJobQueue.start(OrderedJobQueue.java:67)
- locked <0x000000066ab01568> (a java.lang.Object)
at org.apache.sling.event.impl.jobs.queues.AbstractJobQueue.runJobQueue(AbstractJobQueue.java:461)
at org.apache.sling.event.impl.jobs.queues.AbstractJobQueue.access$000(AbstractJobQueue.java:50)
at org.apache.sling.event.impl.jobs.queues.AbstractJobQueue$1.run(AbstractJobQueue.java:137)
at java.lang.Thread.run(Thread.java:722)
Wanted to understand why the 2 different outputs.
Also the second output gives a very descriptive summary in tools like thread dump analyzer.

The difference you observe is a difference between normal and forced jstack format. At least when using HotSpot's jstack implementation, the forced format needs to be requested explicitly by -F option. It is used in situations when normal threaddump can not be obtained as the application does not respond. The forced mode uses different mechanism to obtain the at least some data so it misses a lot of important information.
If I understand the question correctly, that particular jstack implementation chooses the output format automatically, presumably based on the application's ability to respond at the given moment. Consult the documentation if this can be turned off or use an alternative way to obtain the threaddump.

Eclipse hung on Mac OS X when clicking on TabItems and then switch to a list

I'm developing an eclipse plugin on Mac OS X 10.7.3. The UI consists of a TabFolder with 4 TabItems, and some other lists. When I switch between the tabs several times and then clicking another list, eclipse will hung. from the Activity Monitor I could see the CPU usage rushed to 100% and Real Mem also keep increasing. The JVM dump is as below:
2012-12-07 10:14:14
Full thread dump Java HotSpot(TM) 64-Bit Server VM (20.4-b02-402 mixed mode):
"Attach Listener" daemon prio=9 tid=7fa8a4906800 nid=0x116fe7000 waiting on condition [00000000]
java.lang.Thread.State: RUNNABLE
"[ThreadPool Manager] - Idle Thread" daemon prio=5 tid=7fa8a396a000 nid=0x116ae1000 in Object.wait() [116ae0000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <7d0133cb8> (a org.eclipse.equinox.internal.util.impl.tpt.threadpool.Executor)
at java.lang.Object.wait(Object.java:485)
at org.eclipse.equinox.internal.util.impl.tpt.threadpool.Executor.run(Executor.java:106)
- locked <7d0133cb8> (a org.eclipse.equinox.internal.util.impl.tpt.threadpool.Executor)
"Worker-3" prio=5 tid=7fa89fe4f000 nid=0x114752000 in Object.wait() [114751000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <7d43e4648> (a org.eclipse.core.internal.jobs.WorkerPool)
at org.eclipse.core.internal.jobs.WorkerPool.sleep(WorkerPool.java:188)
- locked <7d43e4648> (a org.eclipse.core.internal.jobs.WorkerPool)
at org.eclipse.core.internal.jobs.WorkerPool.startJob(WorkerPool.java:220)
at org.eclipse.core.internal.jobs.Worker.run(Worker.java:50)
"org.eclipse.jface.text.reconciler.MonoReconciler" daemon prio=1 tid=7fa8a3a7c000 nid=0x11730c000 in Object.wait() [11730b000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <7d5c6c590> (a org.eclipse.jface.text.reconciler.DirtyRegionQueue)
at org.eclipse.jface.text.reconciler.AbstractReconciler$BackgroundThread.run(AbstractReconciler.java:179)
- locked <7d5c6c590> (a org.eclipse.jface.text.reconciler.DirtyRegionQueue)
"JavaScript indexing" daemon prio=4 tid=7fa8a2aca800 nid=0x11653f000 in Object.wait() [11653e000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <7d576a0c8> (a org.eclipse.wst.jsdt.internal.core.search.indexing.IndexManager)
at java.lang.Object.wait(Object.java:485)
at org.eclipse.wst.jsdt.internal.core.search.processing.JobManager.run(JobManager.java:370)
- locked <7d576a0c8> (a org.eclipse.wst.jsdt.internal.core.search.indexing.IndexManager)
at java.lang.Thread.run(Thread.java:680)
"Java indexing" daemon prio=4 tid=7fa8a1c47800 nid=0x1169de000 in Object.wait() [1169dd000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <7d50c39d8> (a org.eclipse.jdt.internal.core.search.indexing.IndexManager)
at java.lang.Object.wait(Object.java:485)
at org.eclipse.jdt.internal.core.search.processing.JobManager.run(JobManager.java:382)
- locked <7d50c39d8> (a org.eclipse.jdt.internal.core.search.indexing.IndexManager)
at java.lang.Thread.run(Thread.java:680)
"Worker-2" prio=5 tid=7fa89f49a000 nid=0x11641b000 in Object.wait() [11641a000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <7d43e4648> (a org.eclipse.core.internal.jobs.WorkerPool)
at org.eclipse.core.internal.jobs.WorkerPool.sleep(WorkerPool.java:188)
- locked <7d43e4648> (a org.eclipse.core.internal.jobs.WorkerPool)
at org.eclipse.core.internal.jobs.WorkerPool.startJob(WorkerPool.java:220)
at org.eclipse.core.internal.jobs.Worker.run(Worker.java:50)
"Worker-1" prio=5 tid=7fa89f3cc000 nid=0x114dcf000 in Object.wait() [114dce000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <7d43e4648> (a org.eclipse.core.internal.jobs.WorkerPool)
at org.eclipse.core.internal.jobs.WorkerPool.sleep(WorkerPool.java:188)
- locked <7d43e4648> (a org.eclipse.core.internal.jobs.WorkerPool)
at org.eclipse.core.internal.jobs.WorkerPool.startJob(WorkerPool.java:220)
at org.eclipse.core.internal.jobs.Worker.run(Worker.java:50)
"Worker-0" prio=5 tid=7fa89d0b0000 nid=0x116169000 in Object.wait() [116168000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <7d43e4648> (a org.eclipse.core.internal.jobs.WorkerPool)
at org.eclipse.core.internal.jobs.WorkerPool.sleep(WorkerPool.java:188)
- locked <7d43e4648> (a org.eclipse.core.internal.jobs.WorkerPool)
at org.eclipse.core.internal.jobs.WorkerPool.startJob(WorkerPool.java:220)
at org.eclipse.core.internal.jobs.Worker.run(Worker.java:50)
"Worker-JM" prio=5 tid=7fa89e886000 nid=0x113f33000 in Object.wait() [113f32000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <7d4277db8> (a java.util.ArrayList)
at org.eclipse.core.internal.jobs.InternalWorker.run(InternalWorker.java:58)
- locked <7d4277db8> (a java.util.ArrayList)
"Bundle File Closer" daemon prio=5 tid=7fa89b33f000 nid=0x114baf000 in Object.wait() [114bae000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <7d447b468> (a org.eclipse.osgi.framework.eventmgr.EventManager$EventThread)
at java.lang.Object.wait(Object.java:485)
at org.eclipse.osgi.framework.eventmgr.EventManager$EventThread.getNextEvent(EventManager.java:400)
- locked <7d447b468> (a org.eclipse.osgi.framework.eventmgr.EventManager$EventThread)
at org.eclipse.osgi.framework.eventmgr.EventManager$EventThread.run(EventManager.java:336)
"[Timer] - Main Queue Handler" daemon prio=5 tid=7fa89ec29800 nid=0x114ef4000 in Object.wait() [114ef3000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <7d42e9a50> (a java.lang.Object)
at org.eclipse.equinox.internal.util.impl.tpt.timer.TimerImpl.run(TimerImpl.java:141)
- locked <7d42e9a50> (a java.lang.Object)
at java.lang.Thread.run(Thread.java:680)
"Framework Event Dispatcher" daemon prio=5 tid=7fa89b2f9800 nid=0x1141c6000 in Object.wait() [1141c5000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <7d43147f0> (a org.eclipse.osgi.framework.eventmgr.EventManager$EventThread)
at java.lang.Object.wait(Object.java:485)
at org.eclipse.osgi.framework.eventmgr.EventManager$EventThread.getNextEvent(EventManager.java:400)
- locked <7d43147f0> (a org.eclipse.osgi.framework.eventmgr.EventManager$EventThread)
at org.eclipse.osgi.framework.eventmgr.EventManager$EventThread.run(EventManager.java:336)
"Start Level Event Dispatcher" daemon prio=5 tid=7fa89e90b800 nid=0x11464f000 in Object.wait() [11464e000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <7d4314870> (a org.eclipse.osgi.framework.eventmgr.EventManager$EventThread)
at java.lang.Object.wait(Object.java:485)
at org.eclipse.osgi.framework.eventmgr.EventManager$EventThread.getNextEvent(EventManager.java:400)
- locked <7d4314870> (a org.eclipse.osgi.framework.eventmgr.EventManager$EventThread)
at org.eclipse.osgi.framework.eventmgr.EventManager$EventThread.run(EventManager.java:336)
"State Data Manager" daemon prio=5 tid=7fa89b15c000 nid=0x113ce4000 waiting on condition [113ce3000]
java.lang.Thread.State: TIMED_WAITING (sleeping)
at java.lang.Thread.sleep(Native Method)
at org.eclipse.osgi.internal.baseadaptor.StateManager.run(StateManager.java:297)
at java.lang.Thread.run(Thread.java:680)
"Framework Active Thread" prio=5 tid=7fa89b157800 nid=0x113be1000 in Object.wait() [113be0000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <7d4021de0> (a org.eclipse.osgi.framework.internal.core.Framework)
at org.eclipse.osgi.framework.internal.core.Framework.run(Framework.java:1862)
- locked <7d4021de0> (a org.eclipse.osgi.framework.internal.core.Framework)
at java.lang.Thread.run(Thread.java:680)
"Poller SunPKCS11-Darwin" daemon prio=1 tid=7fa89c10c800 nid=0x112448000 waiting on condition [112447000]
java.lang.Thread.State: TIMED_WAITING (sleeping)
at java.lang.Thread.sleep(Native Method)
at sun.security.pkcs11.SunPKCS11$TokenPoller.run(SunPKCS11.java:692)
at java.lang.Thread.run(Thread.java:680)
"Thread-1" daemon prio=6 tid=7fa89c002800 nid=0x7fff734ca960 runnable [7fff68d24000]
java.lang.Thread.State: RUNNABLE
at org.eclipse.swt.internal.cocoa.OS.objc_msgSendSuper(Native Method)
at org.eclipse.swt.widgets.Widget.callSuper(Widget.java:221)
at org.eclipse.swt.widgets.Widget.mouseDownSuper(Widget.java:1093)
at org.eclipse.swt.widgets.List.mouseDownSuper(List.java:765)
at org.eclipse.swt.widgets.Widget.mouseDown(Widget.java:1085)
at org.eclipse.swt.widgets.Control.mouseDown(Control.java:2538)
at org.eclipse.swt.widgets.Display.windowProc(Display.java:5493)
at org.eclipse.swt.internal.cocoa.OS.objc_msgSendSuper(Native Method)
at org.eclipse.swt.widgets.Widget.callSuper(Widget.java:221)
at org.eclipse.swt.widgets.Widget.windowSendEvent(Widget.java:2102)
at org.eclipse.swt.widgets.Shell.windowSendEvent(Shell.java:2299)
at org.eclipse.swt.widgets.Display.windowProc(Display.java:5557)
at org.eclipse.swt.internal.cocoa.OS.objc_msgSendSuper(Native Method)
at org.eclipse.swt.widgets.Display.applicationSendEvent(Display.java:5002)
at org.eclipse.swt.widgets.Display.applicationProc(Display.java:5151)
at org.eclipse.swt.internal.cocoa.OS.objc_msgSend(Native Method)
at org.eclipse.swt.internal.cocoa.NSApplication.sendEvent(NSApplication.java:128)
at org.eclipse.swt.widgets.Display.readAndDispatch(Display.java:3616)
at org.eclipse.e4.ui.internal.workbench.swt.PartRenderingEngine$9.run(PartRenderingEngine.java:1029)
at org.eclipse.core.databinding.observable.Realm.runWithDefault(Realm.java:332)
at org.eclipse.e4.ui.internal.workbench.swt.PartRenderingEngine.run(PartRenderingEngine.java:923)
at org.eclipse.e4.ui.internal.workbench.E4Workbench.createAndRunUI(E4Workbench.java:86)
at org.eclipse.ui.internal.Workbench$5.run(Workbench.java:588)
at org.eclipse.core.databinding.observable.Realm.runWithDefault(Realm.java:332)
at org.eclipse.ui.internal.Workbench.createAndRunWorkbench(Workbench.java:543)
at org.eclipse.ui.PlatformUI.createAndRunWorkbench(PlatformUI.java:149)
at org.eclipse.ui.internal.ide.application.IDEApplication.start(IDEApplication.java:124)
at org.eclipse.equinox.internal.app.EclipseAppHandle.run(EclipseAppHandle.java:196)
at org.eclipse.core.runtime.internal.adaptor.EclipseAppLauncher.runApplication(EclipseAppLauncher.java:110)
at org.eclipse.core.runtime.internal.adaptor.EclipseAppLauncher.start(EclipseAppLauncher.java:79)
at org.eclipse.core.runtime.adaptor.EclipseStarter.run(EclipseStarter.java:353)
at org.eclipse.core.runtime.adaptor.EclipseStarter.run(EclipseStarter.java:180)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.eclipse.equinox.launcher.Main.invokeFramework(Main.java:629)
at org.eclipse.equinox.launcher.Main.basicRun(Main.java:584)
at org.eclipse.equinox.launcher.Main.run(Main.java:1438)
at org.eclipse.equinox.launcher.Main.main(Main.java:1414)
"Low Memory Detector" daemon prio=5 tid=7fa89e80b000 nid=0x111ed4000 runnable [00000000]
java.lang.Thread.State: RUNNABLE
"C2 CompilerThread1" daemon prio=9 tid=7fa89e80a800 nid=0x111dd1000 waiting on condition [00000000]
java.lang.Thread.State: RUNNABLE
"C2 CompilerThread0" daemon prio=9 tid=7fa89e809800 nid=0x111cce000 waiting on condition [00000000]
java.lang.Thread.State: RUNNABLE
"JDWP Command Reader" daemon prio=5 tid=7fa89b0c9000 nid=0x111bcb000 runnable [00000000]
java.lang.Thread.State: RUNNABLE
"JDWP Event Helper Thread" daemon prio=5 tid=7fa89b0c7000 nid=0x111ac8000 runnable [00000000]
java.lang.Thread.State: RUNNABLE
"JDWP Transport Listener: dt_socket" daemon prio=5 tid=7fa89b0c8000 nid=0x1119c5000 runnable [00000000]
java.lang.Thread.State: RUNNABLE
"Signal Dispatcher" daemon prio=9 tid=7fa89b0c5800 nid=0x1118bc000 runnable [00000000]
java.lang.Thread.State: RUNNABLE
"Surrogate Locker Thread (Concurrent GC)" daemon prio=5 tid=7fa89b0c4800 nid=0x1117b9000 waiting on condition [00000000]
java.lang.Thread.State: RUNNABLE
"Finalizer" daemon prio=8 tid=7fa89d041800 nid=0x1114f5000 in Object.wait() [1114f4000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <7d4024710> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:118)
- locked <7d4024710> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:134)
at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:159)
"Reference Handler" daemon prio=10 tid=7fa89d041000 nid=0x1113f2000 in Object.wait() [1113f1000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <7d4020878> (a java.lang.ref.Reference$Lock)
at java.lang.Object.wait(Object.java:485)
at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:116)
- locked <7d4020878> (a java.lang.ref.Reference$Lock)
"main" prio=5 tid=7fa89b001000 nid=0x1099f9000 runnable [00000000]
java.lang.Thread.State: RUNNABLE
"VM Thread" prio=9 tid=7fa89d03c000 nid=0x1112ef000 runnable
"Gang worker#0 (Parallel GC Threads)" prio=9 tid=7fa89b002800 nid=0x10cd72000 runnable
"Gang worker#1 (Parallel GC Threads)" prio=9 tid=7fa89b003000 nid=0x10ce75000 runnable
"Gang worker#2 (Parallel GC Threads)" prio=9 tid=7fa89b003800 nid=0x10cf78000 runnable
"Gang worker#3 (Parallel GC Threads)" prio=9 tid=7fa89b004000 nid=0x10d07b000 runnable
"Concurrent Mark-Sweep GC Thread" prio=9 tid=7fa89b07f800 nid=0x110f90000 runnable
"VM Periodic Task Thread" prio=10 tid=7fa89e81d000 nid=0x111fd7000 waiting on condition
"Exception Catcher Thread" prio=10 tid=7fa89b001800 nid=0x109b99000 runnable
JNI global references: 11239
Does any one have idea on what is happening?

After investigation i finally found the root cause. It's not caused by the Tab but the List. When clicking on the empty area on a list (the list has more rows than entries), the list.getSelection() will still be the current selected item on Windows, but will be empty on Mac. And when retrieving the value by list.getSelection()[0], eclipse will hang instead of throw ArrayIndexOutOfBoundsException(). I've worked around this issue by remembering the current selection with local variable and set the selection when user clicked on empty areas.

Scala REPL ~20secs to startup -- is this normal?

I'm using Scala 2.9.0.final on Ubuntu 11.04. Starting scala from the command line takes about 20 seconds before a prompt appears.
I realize I should expect some delay because it has to lead compiler JAR files, but I have a hard time believing that it should be 20 seconds -- I worry I may have something configured wrong.
EDIT
So I tried jstack and it gives a long output that is above my head. I see one thread waiting on a mutex. But looking at the output from time:
21.96s user 0.66s system 135% cpu 16.711 total
It would seem that it spent a decent amount of cycles on both cores... so maybe it's not just blocking on a mutex for that time?
2011-07-28 11:45:02
Full thread dump OpenJDK 64-Bit Server VM (20.0-b11 mixed mode):
"Attach Listener" daemon prio=10 tid=0x00000000023ea000 nid=0x19a9 waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"Thread-0" prio=10 tid=0x00007f7768304800 nid=0x19a8 runnable [0x00007f776cd92000]
java.lang.Thread.State: RUNNABLE
at scala.collection.generic.Growable$$anonfun$$plus$plus$eq$1.<init>(Growable.scala:48)
at scala.collection.generic.Growable$class.$plus$plus$eq(Growable.scala:48)
at scala.collection.mutable.ListBuffer.$plus$plus$eq(ListBuffer.scala:128)
at scala.collection.TraversableOnce$class.toList(TraversableOnce.scala:242)
at scala.collection.mutable.WrappedArray.toList(WrappedArray.scala:33)
at scala.collection.immutable.List$.apply(List.scala:434)
at scala.Option.toList(Option.scala:231)
at scala.tools.nsc.util.DirectoryClassPath.asURLs(ClassPath.scala:321)
at scala.tools.nsc.util.ClassPath.sortString(ClassPath.scala:284)
at scala.tools.nsc.util.ClassPath.hashCode(ClassPath.scala:289)
at scala.collection.mutable.FlatHashTable$HashUtils$class.elemHashCode(FlatHashTable.scala:349)
at scala.collection.mutable.HashSet.elemHashCode(HashSet.scala:39)
at scala.collection.mutable.FlatHashTable$class.containsEntry(FlatHashTable.scala:109)
at scala.collection.mutable.HashSet.containsEntry(HashSet.scala:39)
at scala.collection.mutable.HashSet.contains(HashSet.scala:55)
at scala.collection.GenSetLike$class.apply(GenSetLike.scala:44)
at scala.collection.mutable.HashSet.apply(HashSet.scala:39)
at scala.collection.SeqLike$$anonfun$distinct$1.apply(SeqLike.scala:505)
at scala.collection.LinearSeqOptimized$class.foreach(LinearSeqOptimized.scala:59)
at scala.collection.immutable.List.foreach(List.scala:45)
at scala.collection.SeqLike$class.distinct(SeqLike.scala:504)
at scala.collection.immutable.List.distinct(List.scala:45)
at scala.tools.util.PathResolver$Calculated$.containers(PathResolver.scala:206)
- locked <0x00000000faba8098> (a scala.tools.util.PathResolver$Calculated$)
at scala.tools.util.PathResolver.containers(PathResolver.scala:228)
at scala.tools.util.PathResolver.result(PathResolver.scala:231)
- locked <0x00000000faba8050> (a scala.tools.util.PathResolver)
at scala.tools.nsc.backend.JavaPlatform$class.classPath(JavaPlatform.scala:18)
at scala.tools.nsc.Global$$anon$2.classPath(Global.scala:54)
- locked <0x00000000faba7ff0> (a scala.tools.nsc.Global$$anon$2)
at scala.tools.nsc.backend.JavaPlatform$class.rootLoader(JavaPlatform.scala:19)
at scala.tools.nsc.Global$$anon$2.rootLoader(Global.scala:54)
at scala.tools.nsc.Global$$anon$2.rootLoader(Global.scala:54)
at scala.tools.nsc.Global.rootLoader(Global.scala:57)
at scala.tools.nsc.symtab.Definitions$definitions$.RootClass(Definitions.scala:138)
- locked <0x00000000faba5620> (a scala.tools.nsc.symtab.Definitions$definitions$)
at scala.tools.nsc.symtab.Definitions$definitions$.EmptyPackage(Definitions.scala:141)
- locked <0x00000000faba5620> (a scala.tools.nsc.symtab.Definitions$definitions$)
at scala.tools.nsc.symtab.Definitions$definitions$.EmptyPackageClass(Definitions.scala:142)
- locked <0x00000000faba5620> (a scala.tools.nsc.symtab.Definitions$definitions$)
at scala.tools.nsc.symtab.Definitions$definitions$.init(Definitions.scala:804)
at scala.tools.nsc.Global$Run.<init>(Global.scala:693)
at scala.tools.nsc.interpreter.IMain.scala$tools$nsc$interpreter$IMain$$_initialize(IMain.scala:120)
at scala.tools.nsc.interpreter.IMain$$anonfun$initialize$1.apply$mcZ$sp(IMain.scala:154)
at scala.tools.nsc.interpreter.IMain$$anonfun$initialize$2.apply(IMain.scala:154)
at scala.tools.nsc.interpreter.IMain$$anonfun$initialize$2.apply(IMain.scala:154)
at scala.concurrent.ThreadRunner$$anon$2$$anonfun$run$2.apply(ThreadRunner.scala:45)
at scala.concurrent.ThreadRunner.scala$concurrent$ThreadRunner$$tryCatch(ThreadRunner.scala:31)
at scala.concurrent.ThreadRunner$$anon$2.run(ThreadRunner.scala:45)
at java.lang.Thread.run(Thread.java:679)
"Low Memory Detector" daemon prio=10 tid=0x00007f77680b9000 nid=0x1996 runnable [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"C2 CompilerThread1" daemon prio=10 tid=0x00007f77680b6800 nid=0x1995 runnable [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"C2 CompilerThread0" daemon prio=10 tid=0x00007f77680b3800 nid=0x1994 runnable [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"Signal Dispatcher" daemon prio=10 tid=0x00007f77680b2000 nid=0x1993 runnable [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"Finalizer" daemon prio=10 tid=0x00007f7768092800 nid=0x1992 in Object.wait() [0x00007f776e27c000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x00000000fb2d82d0> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:133)
- locked <0x00000000fb2d82d0> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:149)
at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:177)
"Reference Handler" daemon prio=10 tid=0x00007f7768091000 nid=0x1991 in Object.wait() [0x00007f776e37d000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x00000000fb2d8370> (a java.lang.ref.Reference$Lock)
at java.lang.Object.wait(Object.java:502)
at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:133)
- locked <0x00000000fb2d8370> (a java.lang.ref.Reference$Lock)
"main" prio=10 tid=0x0000000001db4800 nid=0x198d in Object.wait() [0x00007f7775067000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x00000000fab8d110> (a scala.concurrent.SyncVar)
at java.lang.Object.wait(Object.java:502)
at scala.concurrent.SyncVar.get(SyncVar.scala:22)
- locked <0x00000000fab8d110> (a scala.concurrent.SyncVar)
at scala.concurrent.ThreadRunner$$anonfun$submit$1.apply(ThreadRunner.scala:48)
at scala.Function0$class.apply$mcZ$sp(Function0.scala:34)
at scala.runtime.AbstractFunction0.apply$mcZ$sp(AbstractFunction0.scala:12)
at scala.tools.nsc.interpreter.IMain.global(IMain.scala:162)
- locked <0x00000000fb2d8580> (a scala.tools.nsc.interpreter.ILoop$ILoopInterpreter)
at scala.tools.nsc.interpreter.JLineCompletion.<init>(JLineCompletion.scala:17)
at scala.tools.nsc.interpreter.ILoop.chooseReader(ILoop.scala:791)
at scala.tools.nsc.interpreter.ILoop.process(ILoop.scala:807)
at scala.tools.nsc.MainGenericRunner.runTarget$1(MainGenericRunner.scala:67)
at scala.tools.nsc.MainGenericRunner.process(MainGenericRunner.scala:80)
at scala.tools.nsc.MainGenericRunner$.main(MainGenericRunner.scala:89)
at scala.tools.nsc.MainGenericRunner.main(MainGenericRunner.scala)
"VM Thread" prio=10 tid=0x00007f776808a000 nid=0x1990 runnable
"GC task thread#0 (ParallelGC)" prio=10 tid=0x0000000001dc0000 nid=0x198e runnable
"GC task thread#1 (ParallelGC)" prio=10 tid=0x0000000001dc1800 nid=0x198f runnable
"VM Periodic Task Thread" prio=10 tid=0x00007f77680bb800 nid=0x1997 waiting on condition
JNI global references: 1010

The startup time will be reduced in the upcoming 2.9.1 release. If yu want to you can already check it out by downloading the RC here :
http://www.scala-lang.org/downloads