We recently upgraded our sbt build from 0.13.x to 1.2.8 from then on our build agents are often (not always ) hangs right after
Resolving key references (11281 settings) ...
In thread dump I see following lines
- locked <0x00000000800058c0> (a java.lang.ref.Reference$Lock)
- waiting on <0x00000000800058c0> (a java.lang.ref.Reference$Lock)
Is it a bug? if not what am I doing wrong? Can you please advice?
Environment details
OS Linux aldburg 3.13.0-32-generic #57~precise1-Ubuntu SMP Tue Jul 15 03:51:20 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
SBT launcher version 1.2.8
SBT project version 1.2.8
SBT Plugins we use are
addSbtPlugin("org.scoverage" % "sbt-scoverage" % "1.5.1")
addSbtPlugin("com.typesafe.sbt" % "sbt-site" % "1.3.1")
addSbtPlugin("com.waioeka.sbt" % "cucumber-plugin" % "0.1.7")
addSbtPlugin("com.eed3si9n" % "sbt-unidoc" % "0.4.1")
addSbtPlugin("org.scalastyle" %% "scalastyle-sbt-plugin" % "1.0.0")
addSbtPlugin("io.spray" % "sbt-revolver" % "0.9.0")
addSbtPlugin("org.scalariform" % "sbt-scalariform" % "1.8.2")
addSbtPlugin("com.github.gseitz" % "sbt-release" % "1.0.7")
addSbtPlugin("de.heikoseeberger" % "sbt-header" % "4.1.0")
addSbtPlugin("com.lightbend.sbt" % "sbt-aspectj" % "0.11.0")
addSbtPlugin("com.typesafe.sbt" % "sbt-native-packager" % "1.3.2")
addSbtPlugin("pl.project13.scala" % "sbt-jmh" % "0.3.4")
Thread dump
Full thread dump OpenJDK 64-Bit Server VM (25.111-b14 mixed mode):
"Attach Listener" #51 daemon prio=9 os_prio=0 tid=0x00007fbc40001000 nid=0x10bd waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"pool-4-thread-8" #50 prio=5 os_prio=0 tid=0x00007fbc0c00e000 nid=0xf6c in Object.wait() [0x00007fbc45534000]
java.lang.Thread.State: RUNNABLE
at $0e82cce6ec5c385cb470$.$anonfun$v1$10(build.sbt:58)
at $0e82cce6ec5c385cb470$$$Lambda$2732/1436136891.apply(Unknown Source)
at sbt.internal.util.EvaluateSettings.$anonfun$constant$1(INode.scala:204)
at sbt.internal.util.EvaluateSettings$$Lambda$1613/382762227.apply(Unknown Source)
at sbt.internal.util.EvaluateSettings$MixedNode.evaluate0(INode.scala:221)
at sbt.internal.util.EvaluateSettings$INode.evaluate(INode.scala:164)
- locked <0x0000000083884fc8> (a sbt.internal.util.EvaluateSettings$MixedNode)
at sbt.internal.util.EvaluateSettings.$anonfun$submitEvaluate$1(INode.scala:87)
at sbt.internal.util.EvaluateSettings$$Lambda$1624/2021540695.apply$mcV$sp(Unknown Source)
at sbt.internal.util.EvaluateSettings.sbt$internal$util$EvaluateSettings$$run0(INode.scala:98)
at sbt.internal.util.EvaluateSettings$$anon$3.run(INode.scala:94)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
"pool-4-thread-7" #49 prio=5 os_prio=0 tid=0x00007fbc89f2d800 nid=0xf6b in Object.wait() [0x00007fbc45230000]
java.lang.Thread.State: RUNNABLE
at Dependencies$.<init>(Dependencies.scala:144)
at Dependencies$.<clinit>(Dependencies.scala)
at $0e82cce6ec5c385cb470$.$anonfun$common$10(build.sbt:81)
at $0e82cce6ec5c385cb470$$$Lambda$2744/908744222.apply(Unknown Source)
at sbt.internal.util.EvaluateSettings.$anonfun$constant$1(INode.scala:204)
at sbt.internal.util.EvaluateSettings$$Lambda$1613/382762227.apply(Unknown Source)
at sbt.internal.util.EvaluateSettings$MixedNode.evaluate0(INode.scala:221)
at sbt.internal.util.EvaluateSettings$INode.evaluate(INode.scala:164)
- locked <0x000000008388f6c0> (a sbt.internal.util.EvaluateSettings$MixedNode)
at sbt.internal.util.EvaluateSettings.$anonfun$submitEvaluate$1(INode.scala:87)
at sbt.internal.util.EvaluateSettings$$Lambda$1624/2021540695.apply$mcV$sp(Unknown Source)
at sbt.internal.util.EvaluateSettings.sbt$internal$util$EvaluateSettings$$run0(INode.scala:98)
at sbt.internal.util.EvaluateSettings$$anon$3.run(INode.scala:94)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
"pool-4-thread-6" #48 prio=5 os_prio=0 tid=0x00007fbc89f2c800 nid=0xf6a in Object.wait() [0x00007fbc45332000]
java.lang.Thread.State: RUNNABLE
at $0e82cce6ec5c385cb470$.$anonfun$support$1(build.sbt:163)
at $0e82cce6ec5c385cb470$$$Lambda$2734/1113898680.apply(Unknown Source)
at sbt.internal.util.EvaluateSettings.$anonfun$constant$1(INode.scala:204)
at sbt.internal.util.EvaluateSettings$$Lambda$1613/382762227.apply(Unknown Source)
at sbt.internal.util.EvaluateSettings$MixedNode.evaluate0(INode.scala:221)
at sbt.internal.util.EvaluateSettings$INode.evaluate(INode.scala:164)
- locked <0x00000000838c9798> (a sbt.internal.util.EvaluateSettings$MixedNode)
at sbt.internal.util.EvaluateSettings.$anonfun$submitEvaluate$1(INode.scala:87)
at sbt.internal.util.EvaluateSettings$$Lambda$1624/2021540695.apply$mcV$sp(Unknown Source)
at sbt.internal.util.EvaluateSettings.sbt$internal$util$EvaluateSettings$$run0(INode.scala:98)
at sbt.internal.util.EvaluateSettings$$anon$3.run(INode.scala:94)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
"pool-4-thread-5" #47 prio=5 os_prio=0 tid=0x00007fbc89f2b800 nid=0xf69 in Object.wait() [0x00007fbc44a2a000]
java.lang.Thread.State: RUNNABLE
at Dependencies$Compile$.<init>(Dependencies.scala:42)
at Dependencies$Compile$.<clinit>(Dependencies.scala)
at $0e82cce6ec5c385cb470$.$anonfun$test_server$4(build.sbt:176)
at $0e82cce6ec5c385cb470$$$Lambda$2715/684426930.apply(Unknown Source)
at sbt.internal.util.EvaluateSettings.$anonfun$constant$1(INode.scala:204)
at sbt.internal.util.EvaluateSettings$$Lambda$1613/382762227.apply(Unknown Source)
at sbt.internal.util.EvaluateSettings$MixedNode.evaluate0(INode.scala:221)
at sbt.internal.util.EvaluateSettings$INode.evaluate(INode.scala:164)
- locked <0x0000000083a9c6c0> (a sbt.internal.util.EvaluateSettings$MixedNode)
at sbt.internal.util.EvaluateSettings.$anonfun$submitEvaluate$1(INode.scala:87)
at sbt.internal.util.EvaluateSettings$$Lambda$1624/2021540695.apply$mcV$sp(Unknown Source)
at sbt.internal.util.EvaluateSettings.sbt$internal$util$EvaluateSettings$$run0(INode.scala:98)
at sbt.internal.util.EvaluateSettings$$anon$3.run(INode.scala:94)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
"pool-4-thread-4" #46 prio=5 os_prio=0 tid=0x00007fbc0800d000 nid=0xf68 in Object.wait() [0x00007fbc4502f000]
java.lang.Thread.State: RUNNABLE
at $0e82cce6ec5c385cb470$.$anonfun$test_utils$1(build.sbt:208)
at $0e82cce6ec5c385cb470$$$Lambda$2751/171063899.apply(Unknown Source)
at sbt.internal.util.EvaluateSettings.$anonfun$constant$1(INode.scala:204)
at sbt.internal.util.EvaluateSettings$$Lambda$1613/382762227.apply(Unknown Source)
at sbt.internal.util.EvaluateSettings$MixedNode.evaluate0(INode.scala:221)
at sbt.internal.util.EvaluateSettings$INode.evaluate(INode.scala:164)
- locked <0x0000000084031dd0> (a sbt.internal.util.EvaluateSettings$MixedNode)
at sbt.internal.util.EvaluateSettings.$anonfun$submitEvaluate$1(INode.scala:87)
at sbt.internal.util.EvaluateSettings$$Lambda$1624/2021540695.apply$mcV$sp(Unknown Source)
at sbt.internal.util.EvaluateSettings.sbt$internal$util$EvaluateSettings$$run0(INode.scala:98)
at sbt.internal.util.EvaluateSettings$$anon$3.run(INode.scala:94)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
"pool-4-thread-3" #45 prio=5 os_prio=0 tid=0x00007fbc89f2a800 nid=0xf67 in Object.wait() [0x00007fbc45433000]
java.lang.Thread.State: RUNNABLE
at $0e82cce6ec5c385cb470$.$anonfun$benchmark$1(build.sbt:154)
at $0e82cce6ec5c385cb470$$$Lambda$2797/2015110295.apply(Unknown Source)
at sbt.internal.util.EvaluateSettings.$anonfun$constant$1(INode.scala:204)
at sbt.internal.util.EvaluateSettings$$Lambda$1613/382762227.apply(Unknown Source)
at sbt.internal.util.EvaluateSettings$MixedNode.evaluate0(INode.scala:221)
at sbt.internal.util.EvaluateSettings$INode.evaluate(INode.scala:164)
- locked <0x00000000842c3310> (a sbt.internal.util.EvaluateSettings$MixedNode)
at sbt.internal.util.EvaluateSettings.$anonfun$submitEvaluate$1(INode.scala:87)
at sbt.internal.util.EvaluateSettings$$Lambda$1624/2021540695.apply$mcV$sp(Unknown Source)
at sbt.internal.util.EvaluateSettings.sbt$internal$util$EvaluateSettings$$run0(INode.scala:98)
at sbt.internal.util.EvaluateSettings$$anon$3.run(INode.scala:94)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
"pool-4-thread-2" #44 prio=5 os_prio=0 tid=0x00007fbc89f2a000 nid=0xf66 in Object.wait() [0x00007fbc44f2e000]
java.lang.Thread.State: RUNNABLE
at $0e82cce6ec5c385cb470$.$anonfun$cucumber$2(build.sbt:195)
at $0e82cce6ec5c385cb470$$$Lambda$2794/1142350221.apply(Unknown Source)
at sbt.internal.util.EvaluateSettings.$anonfun$constant$1(INode.scala:204)
at sbt.internal.util.EvaluateSettings$$Lambda$1613/382762227.apply(Unknown Source)
at sbt.internal.util.EvaluateSettings$MixedNode.evaluate0(INode.scala:221)
at sbt.internal.util.EvaluateSettings$INode.evaluate(INode.scala:164)
- locked <0x0000000083f0b728> (a sbt.internal.util.EvaluateSettings$MixedNode)
at sbt.internal.util.EvaluateSettings.$anonfun$submitEvaluate$1(INode.scala:87)
at sbt.internal.util.EvaluateSettings$$Lambda$1624/2021540695.apply$mcV$sp(Unknown Source)
at sbt.internal.util.EvaluateSettings.sbt$internal$util$EvaluateSettings$$run0(INode.scala:98)
at sbt.internal.util.EvaluateSettings$$anon$3.run(INode.scala:94)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
"pool-4-thread-1" #43 prio=5 os_prio=0 tid=0x00007fbc89f28000 nid=0xf65 in Object.wait() [0x00007fbc45130000]
java.lang.Thread.State: RUNNABLE
at $0e82cce6ec5c385cb470$.$anonfun$domain$1(build.sbt:147)
at $0e82cce6ec5c385cb470$$$Lambda$2733/1341548823.apply(Unknown Source)
at sbt.internal.util.EvaluateSettings.$anonfun$constant$1(INode.scala:204)
at sbt.internal.util.EvaluateSettings$$Lambda$1613/382762227.apply(Unknown Source)
at sbt.internal.util.EvaluateSettings$MixedNode.evaluate0(INode.scala:221)
at sbt.internal.util.EvaluateSettings$INode.evaluate(INode.scala:164)
- locked <0x0000000084031e40> (a sbt.internal.util.EvaluateSettings$MixedNode)
at sbt.internal.util.EvaluateSettings.$anonfun$submitEvaluate$1(INode.scala:87)
at sbt.internal.util.EvaluateSettings$$Lambda$1624/2021540695.apply$mcV$sp(Unknown Source)
at sbt.internal.util.EvaluateSettings.sbt$internal$util$EvaluateSettings$$run0(INode.scala:98)
at sbt.internal.util.EvaluateSettings$$anon$3.run(INode.scala:94)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
"Log4j2-TF-1-AsyncLogger[AsyncContext#64bf3bbf]-1" #13 daemon prio=5 os_prio=0 tid=0x00007fbc8a2df800 nid=0xf38 waiting on condition [0x00007fbc6126c000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x000000008000b4e8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
at com.lmax.disruptor.TimeoutBlockingWaitStrategy.waitFor(TimeoutBlockingWaitStrategy.java:38)
at com.lmax.disruptor.ProcessingSequenceBarrier.waitFor(ProcessingSequenceBarrier.java:56)
at com.lmax.disruptor.BatchEventProcessor.processEvents(BatchEventProcessor.java:159)
at com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:125)
at java.lang.Thread.run(Thread.java:745)
"Service Thread" #9 daemon prio=9 os_prio=0 tid=0x00007fbc88213000 nid=0xf36 runnable [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"C1 CompilerThread3" #8 daemon prio=9 os_prio=0 tid=0x00007fbc88205800 nid=0xf35 waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"C2 CompilerThread2" #7 daemon prio=9 os_prio=0 tid=0x00007fbc88201000 nid=0xf34 waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"C2 CompilerThread1" #6 daemon prio=9 os_prio=0 tid=0x00007fbc881ff800 nid=0xf33 waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"C2 CompilerThread0" #5 daemon prio=9 os_prio=0 tid=0x00007fbc881fc800 nid=0xf32 waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"Signal Dispatcher" #4 daemon prio=9 os_prio=0 tid=0x00007fbc881fa800 nid=0xf31 runnable [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"Finalizer" #3 daemon prio=8 os_prio=0 tid=0x00007fbc881d2800 nid=0xf30 in Object.wait() [0x00007fbc62ceb000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x0000000080003910> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143)
- locked <0x0000000080003910> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:164)
at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:209)
"Reference Handler" #2 daemon prio=10 os_prio=0 tid=0x00007fbc881ce000 nid=0xf2e in Object.wait() [0x00007fbc62dec000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x00000000800058c0> (a java.lang.ref.Reference$Lock)
at java.lang.Object.wait(Object.java:502)
at java.lang.ref.Reference.tryHandlePending(Reference.java:191)
- locked <0x00000000800058c0> (a java.lang.ref.Reference$Lock)
at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:153)
"main" #1 prio=5 os_prio=0 tid=0x00007fbc8800a000 nid=0xf20 waiting on condition [0x00007fbc91ed0000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x0000000084031e88> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
at sbt.internal.util.EvaluateSettings.run(INode.scala:72)
at sbt.internal.util.Init.applyInits(Settings.scala:286)
at sbt.internal.util.Init.make(Settings.scala:208)
at sbt.internal.util.Init.make$(Settings.scala:199)
at sbt.Def$.make(Def.scala:20)
at sbt.internal.Load$.$anonfun$apply$5(Load.scala:266)
at sbt.internal.Load$$$Lambda$1561/1579610605.apply(Unknown Source)
at sbt.internal.Load$.timed(Load.scala:1395)
at sbt.internal.Load$.apply(Load.scala:261)
at sbt.internal.Load$.defaultLoad(Load.scala:69)
at sbt.BuiltinCommands$.liftedTree1$1(Main.scala:829)
at sbt.BuiltinCommands$.doLoadProject(Main.scala:829)
at sbt.BuiltinCommands$.$anonfun$loadProjectImpl$2(Main.scala:800)
at sbt.BuiltinCommands$$$Lambda$380/810169941.apply(Unknown Source)
at sbt.Command$.$anonfun$applyEffect$4(Command.scala:142)
at sbt.Command$$$Lambda$330/1723848804.apply(Unknown Source)
at sbt.Command$.$anonfun$applyEffect$2(Command.scala:137)
at sbt.Command$$$Lambda$356/1859823482.apply(Unknown Source)
at sbt.Command$.process(Command.scala:181)
at sbt.MainLoop$.processCommand(MainLoop.scala:151)
at sbt.MainLoop$.$anonfun$next$2(MainLoop.scala:139)
at sbt.MainLoop$$$Lambda$311/198374825.apply(Unknown Source)
at sbt.State$$anon$1.runCmd$1(State.scala:246)
at sbt.State$$anon$1.process(State.scala:250)
at sbt.MainLoop$.$anonfun$next$1(MainLoop.scala:139)
at sbt.MainLoop$$$Lambda$310/1620187937.apply(Unknown Source)
at sbt.internal.util.ErrorHandling$.wideConvert(ErrorHandling.scala:16)
at sbt.MainLoop$.next(MainLoop.scala:139)
at sbt.MainLoop$.run(MainLoop.scala:132)
at sbt.MainLoop$.$anonfun$runWithNewLog$1(MainLoop.scala:110)
at sbt.MainLoop$$$Lambda$303/1924802798.apply(Unknown Source)
at sbt.io.Using.apply(Using.scala:22)
at sbt.MainLoop$.runWithNewLog(MainLoop.scala:104)
at sbt.MainLoop$.runAndClearLast(MainLoop.scala:59)
at sbt.MainLoop$.runLoggedLoop(MainLoop.scala:44)
at sbt.MainLoop$.runLogged(MainLoop.scala:35)
at sbt.StandardMain$.runManaged(Main.scala:138)
at sbt.xMain.run(Main.scala:89)
at xsbt.boot.Launch$$anonfun$run$1.apply(Launch.scala:109)
at xsbt.boot.Launch$.withContextLoader(Launch.scala:128)
at xsbt.boot.Launch$.run(Launch.scala:109)
at xsbt.boot.Launch$$anonfun$apply$1.apply(Launch.scala:35)
at xsbt.boot.Launch$.launch(Launch.scala:117)
at xsbt.boot.Launch$.apply(Launch.scala:18)
at xsbt.boot.Boot$.runImpl(Boot.scala:56)
at xsbt.boot.Boot$.main(Boot.scala:18)
at xsbt.boot.Boot.main(Boot.scala)
"VM Thread" os_prio=0 tid=0x00007fbc881c6800 nid=0xf2d runnable
"GC task thread#0 (ParallelGC)" os_prio=0 tid=0x00007fbc8801f000 nid=0xf21 runnable
"GC task thread#1 (ParallelGC)" os_prio=0 tid=0x00007fbc88020800 nid=0xf22 runnable
"GC task thread#2 (ParallelGC)" os_prio=0 tid=0x00007fbc88022800 nid=0xf27 runnable
"GC task thread#3 (ParallelGC)" os_prio=0 tid=0x00007fbc88024000 nid=0xf28 runnable
"GC task thread#4 (ParallelGC)" os_prio=0 tid=0x00007fbc88026000 nid=0xf29 runnable
"GC task thread#5 (ParallelGC)" os_prio=0 tid=0x00007fbc88027800 nid=0xf2a runnable
"GC task thread#6 (ParallelGC)" os_prio=0 tid=0x00007fbc88029800 nid=0xf2b runnable
"GC task thread#7 (ParallelGC)" os_prio=0 tid=0x00007fbc8802b000 nid=0xf2c runnable
"VM Periodic Task Thread" os_prio=0 tid=0x00007fbc88215800 nid=0xf37 waiting on condition
JNI global references: 5542
Our build server is using different sbt-launcher (0.13.x) than one provided in the agent.
In our case, issue is fixed with the following steps :
Pointing launcher version to sbt-launcher 1.2.8
Removed sbt cache rm -rf ~/.sbt (make sure, take backups of global .sbt files and .credentials file etc..)
Removed ivy cache rm -rf ~/.ivy2/cache
export SBT_OPTS="-Dscala.concurrent.context.maxThreads=1" (out of 5 agents, i have to do this in only one agent)
I am not sure whether it fixes for everyone or not.
Related
I use parallelStream to execute backgroud task in Spring #Async annotation. but the task always blocked then stop. the core code is:
#Async
public void task() {
List<ConfigSymbol> symbols = SymbolUtils.getSaasOpenSymbol();
symbols .parallelStream().forEach(symbol->{
//execute task. 1 minute per loop
}
}
I catch the thread stack to diagnosis the problem. but can not find reason.
why the ForkJoinTask status is waiting .
"SimpleAsyncTaskExecutor-1609" #1777 prio=5 os_prio=0 tid=0x00007fb968006800 nid=0x6f5 in Object.wait() [0x00007fba2d119000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.util.concurrent.ForkJoinTask.externalAwaitDone(ForkJoinTask.java:334)
- locked <0x00000006428a0a38> (a java.util.stream.ForEachOps$ForEachTask)
at java.util.concurrent.ForkJoinTask.doInvoke(ForkJoinTask.java:405)
at java.util.concurrent.ForkJoinTask.invoke(ForkJoinTask.java:734)
at java.util.stream.ForEachOps$ForEachOp.evaluateParallel(ForEachOps.java:159)
at java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateParallel(ForEachOps.java:173)
at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:233)
at java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:485)
at java.util.stream.ReferencePipeline$Head.forEach(ReferencePipeline.java:650)
at com.chainup.stats.service.impl.StatsSymbolRateServiceImpl.getCalculationBTCExchangeRate(StatsSymbolRateServiceImpl.java:141)
at com.chainup.stats.service.impl.StatsSymbolRateServiceImpl.runThreadExchangeRate(StatsSymbolRateServiceImpl.java:67)
at com.chainup.stats.action.StatAct.insertStatsSymbolRate(StatAct.java:442)
at com.chainup.stats.action.StatAct$$FastClassBySpringCGLIB$$74ae31d4.invoke(<generated>)
at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218)
at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:769)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:747)
at org.springframework.aop.aspectj.MethodInvocationProceedingJoinPoint.proceed(MethodInvocationProceedingJoinPoint.java:88)
at org.springframework.cloud.sleuth.instrument.async.TraceAsyncAspect.traceBackgroundThread(TraceAsyncAspect.java:66)
at sun.reflect.GeneratedMethodAccessor277.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethodWithGivenArgs(AbstractAspectJAdvice.java:644)
at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethod(AbstractAspectJAdvice.java:633)
at org.springframework.aop.aspectj.AspectJAroundAdvice.invoke(AspectJAroundAdvice.java:70)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:747)
at org.springframework.aop.aspectj.MethodInvocationProceedingJoinPoint.proceed(MethodInvocationProceedingJoinPoint.java:88)
at com.chainup.stats.interceptor.StatsJobInterceptor.around(StatsJobInterceptor.java:99)
at sun.reflect.GeneratedMethodAccessor273.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethodWithGivenArgs(AbstractAspectJAdvice.java:644)
at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethod(AbstractAspectJAdvice.java:633)
at org.springframework.aop.aspectj.AspectJAroundAdvice.invoke(AspectJAroundAdvice.java:70)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:175)
at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:747)
at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:95)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:747)
at org.springframework.aop.interceptor.AsyncExecutionInterceptor.lambda$invoke$0(AsyncExecutionInterceptor.java:115)
at org.springframework.aop.interceptor.AsyncExecutionInterceptor$$Lambda$790/775437217.call(Unknown Source)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at org.springframework.cloud.sleuth.instrument.async.TraceRunnable.run(TraceRunnable.java:65)
at java.lang.Thread.run(Thread.java:748)
I launch a process from an Ammonite script in this way:
scala.sys.process.Process(Seq("executable", "arg1", "arg2")).run().exitValue()
The launched process terminates ok, but it doesn't know about.
Normally it works ok and process termination is detected ok.
With jstack, I've taken the call stack:
"Thread-2" #15 prio=5 os_prio=0 tid=0x00007fb888023000 nid=0xc0f runnable [0x00007fb8ca696000]
java.lang.Thread.State: RUNNABLE
at java.io.FileInputStream.readBytes(Native Method)
at java.io.FileInputStream.read(FileInputStream.java:255)
at java.io.BufferedInputStream.read1(BufferedInputStream.java:284)
at java.io.BufferedInputStream.read(BufferedInputStream.java:345)
- locked <0x00000000ea926290> (a java.lang.UNIXProcess$ProcessPipeInputStream)
at java.io.FilterInputStream.read(FilterInputStream.java:107)
at scala.sys.process.BasicIO$.loop$1(BasicIO.scala:234)
at scala.sys.process.BasicIO$.transferFullyImpl(BasicIO.scala:242)
at scala.sys.process.BasicIO$.transferFully(BasicIO.scala:223)
at scala.sys.process.BasicIO$.$anonfun$toStdErr$1(BasicIO.scala:212)
at scala.sys.process.BasicIO$.$anonfun$toStdErr$1$adapted(BasicIO.scala:212)
at scala.sys.process.BasicIO$$$Lambda$335/590671716.apply(Unknown Source)
at scala.sys.process.ProcessBuilderImpl$Simple.$anonfun$run$4(ProcessBuilderImpl.scala:79)
at scala.sys.process.ProcessBuilderImpl$Simple$$Lambda$345/1512415799.apply$mcV$sp(Unknown Source)
at scala.sys.process.ProcessImpl$Spawn$$anon$1.run(ProcessImpl.scala:23)
"Thread-1" #14 prio=5 os_prio=0 tid=0x00007fb888021000 nid=0xc0e runnable [0x00007fb8ca797000]
java.lang.Thread.State: RUNNABLE
at java.io.FileInputStream.readBytes(Native Method)
at java.io.FileInputStream.read(FileInputStream.java:255)
at java.io.BufferedInputStream.read1(BufferedInputStream.java:284)
at java.io.BufferedInputStream.read(BufferedInputStream.java:345)
- locked <0x00000000ea9241d0> (a java.lang.UNIXProcess$ProcessPipeInputStream)
at java.io.FilterInputStream.read(FilterInputStream.java:107)
at scala.sys.process.BasicIO$.loop$1(BasicIO.scala:234)
at scala.sys.process.BasicIO$.transferFullyImpl(BasicIO.scala:242)
at scala.sys.process.BasicIO$.transferFully(BasicIO.scala:223)
at scala.sys.process.BasicIO$.$anonfun$toStdOut$1(BasicIO.scala:217)
at scala.sys.process.BasicIO$.$anonfun$toStdOut$1$adapted(BasicIO.scala:217)
at scala.sys.process.BasicIO$$$Lambda$334/1481715712.apply(Unknown Source)
at scala.sys.process.ProcessBuilderImpl$Simple.$anonfun$run$3(ProcessBuilderImpl.scala:76)
at scala.sys.process.ProcessBuilderImpl$Simple$$Lambda$344/1948323324.apply$mcV$sp(Unknown Source)
at scala.sys.process.ProcessImpl$Spawn$$anon$1.run(ProcessImpl.scala:23)
"process reaper" #12 daemon prio=10 os_prio=0 tid=0x00007fb88801c800 nid=0xc0b waiting for monitor entry [0x00007fb8ca8d1000]
java.lang.Thread.State: BLOCKED (on object monitor)
at java.lang.UNIXProcess$ProcessPipeInputStream.processExited(UNIXProcess.java:527)
- waiting to lock <0x00000000ea9241d0> (a java.lang.UNIXProcess$ProcessPipeInputStream)
at java.lang.UNIXProcess.lambda$initStreams$3(UNIXProcess.java:298)
at java.lang.UNIXProcess$$Lambda$342/78086683.run(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
"scala-execution-context-global-10" #10 daemon prio=5 os_prio=0 tid=0x00007fb91a3eb000 nid=0xc08 in Object.wait() [0x00007fb8caad2000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x00000000ea938828> (a scala.sys.process.ProcessImpl$Spawn$$anon$1)
at java.lang.Thread.join(Thread.java:1252)
- locked <0x00000000ea938828> (a scala.sys.process.ProcessImpl$Spawn$$anon$1)
at java.lang.Thread.join(Thread.java:1326)
at scala.sys.process.ProcessImpl$SimpleProcess.$anonfun$exitValue$2(ProcessImpl.scala:241)
at scala.sys.process.ProcessImpl$SimpleProcess.$anonfun$exitValue$2$adapted(ProcessImpl.scala:241)
at scala.sys.process.ProcessImpl$SimpleProcess.exitValue(ProcessImpl.scala:241)
It is locked here:
override def exitValue() = {
try p.waitFor() // wait for the process to terminate
finally inputThread.interrupt() // we interrupt the input thread to notify it that it can terminate
outputThreads foreach (_.join()) // <=== locked here
p.exitValue()
}
More details:
Java 8
Scala 2.12
CentOS 7
How can I make such a simple and basic task more reliable?
Is this a bug from scala.sys.Process?
Workaround is to use the Java API:
val p = Process(Seq(....)).run()
val cmp = p.getClass.getDeclaredField("p")
cmp setAccessible true
val proc = cmp.get(p).asInstanceOf[java.lang.Process]
proc.waitFor()
proc.exitValue()
In this way it doesn't lock anymore.
I'm trying to extend the hamorspam example(https://github.com/h2oai/sparkling-water/blob/master/examples/scripts/hamOrSpam.script.scala
) to make parallel predictions for large dataset using spark's parallel computation power(during the inference stage, not the training phase).
Below is the code I have written for the same. Moreover, it perfectly works fine in single node localmode (for export MASTER="local[*] ``), but fails when I run with export MASTER="local-cluster[2,2,1024] when 2 worker nodes are spawn.(to check the prediction parallelisation)
val data_test = load("smsData.txt") // Should be a large(in GBs) test dataset - using same training data for testing purposes just to test the workflow
val message_test = data.map( r => r(1))
message.take(1000).map(x => isSpam(x, dlModel, hashingTF, idfModel, h2oContext))
So the code fails when executing scala> val table:H2OFrame = resultRDD (
https://github.com/h2oai/sparkling-water/blob/master/examples/scripts/hamOrSpam.script.scala#L110)
I have attached the error from the console below:
17/06/26 20:25:49 WARN TaskSetManager: Lost task 0.0 in stage 6.0 (TID 43, 144.27.27.98, executor 1): java.lang.NoClassDefFoundError: Could not ini
tialize class $line32.$read$
at $line41.$read$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$anonfun$1.apply(<console>:57)
at $line41.$read$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$anonfun$1.apply(<console>:57)
at scala.collection.Iterator$$anon$11.next(Iterator.scala:409)
at scala.collection.Iterator$$anon$11.next(Iterator.scala:409)
at org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIterator.processNext(Unknown Source)
at org.apache.spark.sql.execution.BufferedRowIterator.hasNext(BufferedRowIterator.java:43)
at org.apache.spark.sql.execution.WholeStageCodegenExec$$anonfun$8$$anon$1.hasNext(WholeStageCodegenExec.scala:377)
at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:408)
at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:408)
at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:408)
at org.apache.spark.rdd.RDD$$anonfun$reduce$1$$anonfun$15.apply(RDD.scala:1010)
at org.apache.spark.rdd.RDD$$anonfun$reduce$1$$anonfun$15.apply(RDD.scala:1009)
at org.apache.spark.SparkContext$$anonfun$33.apply(SparkContext.scala:1980)
at org.apache.spark.SparkContext$$anonfun$33.apply(SparkContext.scala:1980)
at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:87)
at org.apache.spark.scheduler.Task.run(Task.scala:99)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:282)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:748)
17/06/26 20:25:49 ERROR TaskSetManager: Task 0 in stage 6.0 failed 4 times; aborting job
org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 6.0 failed 4 times, most recent failure: Lost task 0.3 in stage
6.0 (TID 49, 144.27.27.98, executor 0): java.lang.NoClassDefFoundError: Could not initialize class
at $anonfun$1.apply(<console>:57)
at $anonfun$1.apply(<console>:57)
at scala.collection.Iterator$$anon$11.next(Iterator.scala:409)
at scala.collection.Iterator$$anon$11.next(Iterator.scala:409)
at org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIterator.processNext(Unknown Source)
at org.apache.spark.sql.execution.BufferedRowIterator.hasNext(BufferedRowIterator.java:43)
at org.apache.spark.sql.execution.WholeStageCodegenExec$$anonfun$8$$anon$1.hasNext(WholeStageCodegenExec.scala:377)
at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:408)
at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:408)
at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:408)
at org.apache.spark.rdd.RDD$$anonfun$reduce$1$$anonfun$15.apply(RDD.scala:1010)
at org.apache.spark.rdd.RDD$$anonfun$reduce$1$$anonfun$15.apply(RDD.scala:1009)
at org.apache.spark.SparkContext$$anonfun$33.apply(SparkContext.scala:1980)
at org.apache.spark.SparkContext$$anonfun$33.apply(SparkContext.scala:1980)
at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:87)
at org.apache.spark.scheduler.Task.run(Task.scala:99)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:282)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:748)
Driver stacktrace:
at org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGScheduler$$failJobAndIndependentStages(DAGScheduler.scala:1435)
at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1423)
at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1422)
at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:48)
at org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:1422)
at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:802)
at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:802)
at scala.Option.foreach(Option.scala:257)
at org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:802)
at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.doOnReceive(DAGScheduler.scala:1650)
at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1605)
at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1594)
at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:48)
at org.apache.spark.scheduler.DAGScheduler.runJob(DAGScheduler.scala:628)
at org.apache.spark.SparkContext.runJob(SparkContext.scala:1918)
at org.apache.spark.SparkContext.runJob(SparkContext.scala:1981)
at org.apache.spark.rdd.RDD$$anonfun$reduce$1.apply(RDD.scala:1025)
at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)
at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:112)
at org.apache.spark.rdd.RDD.withScope(RDD.scala:362)
at org.apache.spark.rdd.RDD.reduce(RDD.scala:1007)
at org.apache.spark.h2o.utils.H2OSchemaUtils$.collectMaxArrays(H2OSchemaUtils.scala:229)
at org.apache.spark.h2o.utils.H2OSchemaUtils$.expandedSchema(H2OSchemaUtils.scala:107)
at org.apache.spark.h2o.converters.SparkDataFrameConverter$.toH2OFrame(SparkDataFrameConverter.scala:59)
at org.apache.spark.h2o.H2OContext.asH2OFrame(H2OContext.scala:167)
at org.apache.spark.h2o.H2OContextImplicits.asH2OFrameFromDataFrame(H2OContextImplicits.scala:54)
... 58 elided
Caused by: java.lang.NoClassDefFoundError: Could not initialize class
at $anonfun$1.apply(<console>:57)
at $anonfun$1.apply(<console>:57)
at scala.collection.Iterator$$anon$11.next(Iterator.scala:409)
at scala.collection.Iterator$$anon$11.next(Iterator.scala:409)
at org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIterator.processNext(Unknown Source)
at org.apache.spark.sql.execution.BufferedRowIterator.hasNext(BufferedRowIterator.java:43)
at org.apache.spark.sql.execution.WholeStageCodegenExec$$anonfun$8$$anon$1.hasNext(WholeStageCodegenExec.scala:377)
at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:408)
at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:408)
at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:408)
at org.apache.spark.rdd.RDD$$anonfun$reduce$1$$anonfun$15.apply(RDD.scala:1010)
at org.apache.spark.rdd.RDD$$anonfun$reduce$1$$anonfun$15.apply(RDD.scala:1009)
at org.apache.spark.SparkContext$$anonfun$33.apply(SparkContext.scala:1980)
at org.apache.spark.SparkContext$$anonfun$33.apply(SparkContext.scala:1980)
at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:87)
at org.apache.spark.scheduler.Task.run(Task.scala:99)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:282)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:748)
Any ideas?. Thanks in advance.
I am using Spark's Mllib FPGrowth algorithm to generate frequent itemsets. I'm getting the data from redshift and running the code on it. But the code stalls down with a lot of shuffle occurring between workers with the amount of rows being worked on equal to 5000. The code keeps running for 1-2 hours and then I have to kill it manually. I see the line - Asking to fetch output locations for shuffle 1 on the console and nothing else after this. The same code works fine if I increase the dataset to around 50000 rows. I used jstack to check the thread dump and it looks as if there is a deadlock occurring in FPTree when it fetches Iterator.hasnext(). My configuration is 4 workers with 6 executors with 4 cores and 15GB RAM each.
P.S - I know spark is not meant for small datasets but this just for testing purposes but the thread deadlock looks like a bug
Code :
val data = data1.map { case Row(city : String, cart : Long, p_list : String) =>
p_list
}.rdd
print(data.count)
val transactions: RDD[Array[String]] = data.map(s => s.trim.split(','))
import org.apache.spark.mllib.fpm.AssociationRules
import org.apache.spark.mllib.fpm.FPGrowth.FreqItemset
val fpg = new FPGrowth()
.setMinSupport(0.0001)
.setNumPartitions(24)
val model = fpg.run(transactions)
print(model.freqItemsets.count)
val res = model.generateAssociationRules(0.001)
print(res.count)
The code stalls as soon as it reaches the model.freqItemsets.count statement.
Thread dump -
"Attach Listener" #145 daemon prio=9 os_prio=0 tid=0x00007f7508001000 nid=0x4481 waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"block-manager-slave-async-thread-pool-6" #144 daemon prio=5 os_prio=0 tid=0x00007f7498005000 nid=0x4400 waiting on condition [0x00007f755eee7000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000004022797e8> (a java.util.concurrent.SynchronousQueue$TransferStack)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
"block-manager-slave-async-thread-pool-5" #143 daemon prio=5 os_prio=0 tid=0x00007f7498002800 nid=0x43fe waiting on condition [0x00007f7549dfa000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000004022797e8> (a java.util.concurrent.SynchronousQueue$TransferStack)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
"block-manager-slave-async-thread-pool-4" #142 daemon prio=5 os_prio=0 tid=0x00007f74c801f000 nid=0x43fc waiting on condition [0x00007f755c2ac000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000004022797e8> (a java.util.concurrent.SynchronousQueue$TransferStack)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
"shuffle-server-1" #58 daemon prio=5 os_prio=0 tid=0x00007f7490002000 nid=0x421f runnable [0x00007f7540181000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
- locked <0x0000000404d0c368> (a io.netty.channel.nio.SelectedSelectionKeySet)
- locked <0x0000000405ec34f0> (a java.util.Collections$UnmodifiableSet)
- locked <0x0000000404d0c2d0> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111)
at java.lang.Thread.run(Thread.java:745)
"Executor task launch worker-3" #140 daemon prio=5 os_prio=0 tid=0x00007f74b0004000 nid=0x4183 runnable [0x00007f7540281000]
java.lang.Thread.State: RUNNABLE
at scala.collection.Iterator$$anon$12.hasNext(Iterator.scala:438)
at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:408)
at scala.collection.Iterator$JoinIterator.hasNext(Iterator.scala:219)
at scala.collection.Iterator$$anon$12.hasNext(Iterator.scala:438)
at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:408)
at scala.collection.Iterator$JoinIterator.hasNext(Iterator.scala:219)
at scala.collection.Iterator$$anon$12.hasNext(Iterator.scala:438)
at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:408)
at scala.collection.Iterator$JoinIterator.hasNext(Iterator.scala:219)
at scala.collection.Iterator$$anon$12.hasNext(Iterator.scala:438)
at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:408)
at scala.collection.Iterator$JoinIterator.hasNext(Iterator.scala:219)
at scala.collection.Iterator$$anon$12.hasNext(Iterator.scala:438)
at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:408)
at scala.collection.Iterator$JoinIterator.hasNext(Iterator.scala:219)
at scala.collection.Iterator$$anon$12.hasNext(Iterator.scala:438)
at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:408)
at scala.collection.Iterator$JoinIterator.hasNext(Iterator.scala:219)
at scala.collection.Iterator$$anon$12.hasNext(Iterator.scala:438)
at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:408)
at scala.collection.Iterator$JoinIterator.hasNext(Iterator.scala:219)
at scala.collection.Iterator$$anon$12.hasNext(Iterator.scala:438)
at scala.collection.Iterator$$anon$12.next(Iterator.scala:444)
at scala.collection.Iterator$$anon$11.next(Iterator.scala:409)
at scala.collection.Iterator$JoinIterator.next(Iterator.scala:232)
at scala.collection.Iterator$$anon$12.next(Iterator.scala:444)
at scala.collection.Iterator$$anon$11.next(Iterator.scala:409)
at scala.collection.Iterator$JoinIterator.next(Iterator.scala:232)
at scala.collection.Iterator$$anon$12.next(Iterator.scala:444)
at scala.collection.Iterator$$anon$11.next(Iterator.scala:409)
at scala.collection.Iterator$JoinIterator.next(Iterator.scala:232)
at scala.collection.Iterator$$anon$12.next(Iterator.scala:444)
at scala.collection.Iterator$$anon$11.next(Iterator.scala:409)
at scala.collection.Iterator$JoinIterator.next(Iterator.scala:232)
at scala.collection.Iterator$$anon$12.next(Iterator.scala:444)
at scala.collection.Iterator$$anon$12.next(Iterator.scala:444)
at scala.collection.Iterator$$anon$11.next(Iterator.scala:409)
at org.apache.spark.util.Utils$.getIteratorSize(Utils.scala:1765)
at org.apache.spark.rdd.RDD$$anonfun$count$1.apply(RDD.scala:1134)
at org.apache.spark.rdd.RDD$$anonfun$count$1.apply(RDD.scala:1134)
at org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:1899)
at org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:1899)
at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:70)
at org.apache.spark.scheduler.Task.run(Task.scala:86)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:274)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
"Executor task launch worker-2" #139 daemon prio=5 os_prio=0 tid=0x00007f74b0002800 nid=0x4182 runnable [0x00007f7540382000]
java.lang.Thread.State: RUNNABLE
at scala.collection.Iterator$$anon$12.hasNext(Iterator.scala:438)
at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:408)
at scala.collection.Iterator$JoinIterator.hasNext(Iterator.scala:216)
at scala.collection.Iterator$$anon$12.hasNext(Iterator.scala:438)
at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:408)
at scala.collection.Iterator$JoinIterator.hasNext(Iterator.scala:219)
at scala.collection.Iterator$$anon$12.hasNext(Iterator.scala:438)
at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:408)
at scala.collection.Iterator$JoinIterator.hasNext(Iterator.scala:219)
at scala.collection.Iterator$$anon$12.hasNext(Iterator.scala:438)
at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:408)
at scala.collection.Iterator$JoinIterator.hasNext(Iterator.scala:219)
at scala.collection.Iterator$$anon$12.hasNext(Iterator.scala:438)
at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:408)
at scala.collection.Iterator$JoinIterator.hasNext(Iterator.scala:219)
at scala.collection.Iterator$$anon$12.hasNext(Iterator.scala:438)
at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:408)
at scala.collection.Iterator$JoinIterator.hasNext(Iterator.scala:219)
at scala.collection.Iterator$$anon$12.hasNext(Iterator.scala:438)
at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:408)
at scala.collection.Iterator$JoinIterator.hasNext(Iterator.scala:219)
at scala.collection.Iterator$$anon$12.hasNext(Iterator.scala:438)
at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:408)
at scala.collection.Iterator$JoinIterator.hasNext(Iterator.scala:219)
at scala.collection.Iterator$$anon$12.hasNext(Iterator.scala:438)
at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:408)
at scala.collection.Iterator$JoinIterator.hasNext(Iterator.scala:219)
at scala.collection.Iterator$$anon$12.hasNext(Iterator.scala:438)
at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:408)
at scala.collection.Iterator$JoinIterator.hasNext(Iterator.scala:219)
at scala.collection.Iterator$$anon$12.hasNext(Iterator.scala:438)
at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:408)
at scala.collection.Iterator$JoinIterator.hasNext(Iterator.scala:219)
at scala.collection.Iterator$$anon$12.hasNext(Iterator.scala:438)
at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:408)
at scala.collection.Iterator$JoinIterator.hasNext(Iterator.scala:219)
at scala.collection.Iterator$$anon$12.hasNext(Iterator.scala:438)
at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:408)
at scala.collection.Iterator$JoinIterator.hasNext(Iterator.scala:219)
at scala.collection.Iterator$$anon$12.hasNext(Iterator.scala:438)
at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:408)
at scala.collection.Iterator$JoinIterator.hasNext(Iterator.scala:219)
at scala.collection.Iterator$$anon$12.hasNext(Iterator.scala:438)
at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:408)
at scala.collection.Iterator$JoinIterator.hasNext(Iterator.scala:219)
at scala.collection.Iterator$$anon$12.hasNext(Iterator.scala:438)
at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:408)
at scala.collection.Iterator$JoinIterator.hasNext(Iterator.scala:219)
at scala.collection.Iterator$$anon$12.hasNext(Iterator.scala:438)
at scala.collection.Iterator$$anon$12.hasNext(Iterator.scala:438)
at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:408)
at org.apache.spark.util.Utils$.getIteratorSize(Utils.scala:1763)
at org.apache.spark.rdd.RDD$$anonfun$count$1.apply(RDD.scala:1134)
at org.apache.spark.rdd.RDD$$anonfun$count$1.apply(RDD.scala:1134)
at org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:1899)
at org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:1899)
at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:70)
at org.apache.spark.scheduler.Task.run(Task.scala:86)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:274)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
"Executor task launch worker-1" #138 daemon prio=5 os_prio=0 tid=0x00007f74b0001000 nid=0x4181 runnable [0x00007f755e9e1000]
java.lang.Thread.State: RUNNABLE
at scala.collection.mutable.HashTable$class.findOrAddEntry(HashTable.scala:166)
at scala.collection.mutable.HashMap.findOrAddEntry(HashMap.scala:40)
at scala.collection.mutable.HashMap.put(HashMap.scala:76)
at scala.collection.mutable.HashMap.update(HashMap.scala:81)
at scala.collection.mutable.MapLike$class.getOrElseUpdate(MapLike.scala:194)
at scala.collection.mutable.AbstractMap.getOrElseUpdate(Map.scala:80)
at org.apache.spark.mllib.fpm.FPTree$$anonfun$add$1.apply(FPTree.scala:43)
at org.apache.spark.mllib.fpm.FPTree$$anonfun$add$1.apply(FPTree.scala:40)
at scala.collection.immutable.List.foreach(List.scala:381)
at org.apache.spark.mllib.fpm.FPTree.add(FPTree.scala:40)
at org.apache.spark.mllib.fpm.FPTree$$anonfun$org$apache$spark$mllib$fpm$FPTree$$project$1.apply(FPTree.scala:75)
at org.apache.spark.mllib.fpm.FPTree$$anonfun$org$apache$spark$mllib$fpm$FPTree$$project$1.apply(FPTree.scala:68)
at scala.collection.immutable.List.foreach(List.scala:381)
at scala.collection.generic.TraversableForwarder$class.foreach(TraversableForwarder.scala:35)
at scala.collection.mutable.ListBuffer.foreach(ListBuffer.scala:45)
at org.apache.spark.mllib.fpm.FPTree.org$apache$spark$mllib$fpm$FPTree$$project(FPTree.scala:68)
at org.apache.spark.mllib.fpm.FPTree$$anonfun$extract$1$$anonfun$apply$2.apply(FPTree.scala:108)
at org.apache.spark.mllib.fpm.FPTree$$anonfun$extract$1$$anonfun$apply$2.apply(FPTree.scala:108)
at scala.collection.Iterator$JoinIterator.rhs$lzycompute(Iterator.scala:208)
- locked <0x00000006ce73a440> (a scala.collection.Iterator$JoinIterator)
at scala.collection.Iterator$JoinIterator.rhs(Iterator.scala:208)
at scala.collection.Iterator$JoinIterator.hasNext(Iterator.scala:216)
at scala.collection.Iterator$$anon$12.hasNext(Iterator.scala:438)
at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:408)
at scala.collection.Iterator$JoinIterator.hasNext(Iterator.scala:219)
at scala.collection.Iterator$$anon$12.hasNext(Iterator.scala:438)
at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:408)
at scala.collection.Iterator$JoinIterator.hasNext(Iterator.scala:219)
at scala.collection.Iterator$$anon$12.hasNext(Iterator.scala:438)
at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:408)
at scala.collection.Iterator$JoinIterator.hasNext(Iterator.scala:219)
at scala.collection.Iterator$$anon$12.hasNext(Iterator.scala:438)
at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:408)
at scala.collection.Iterator$JoinIterator.hasNext(Iterator.scala:219)
at scala.collection.Iterator$$anon$12.hasNext(Iterator.scala:438)
at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:408)
at scala.collection.Iterator$JoinIterator.hasNext(Iterator.scala:219)
at scala.collection.Iterator$$anon$12.hasNext(Iterator.scala:438)
at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:408)
at scala.collection.Iterator$JoinIterator.hasNext(Iterator.scala:219)
at scala.collection.Iterator$$anon$12.hasNext(Iterator.scala:438)
at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:408)
at scala.collection.Iterator$JoinIterator.hasNext(Iterator.scala:219)
at scala.collection.Iterator$$anon$12.hasNext(Iterator.scala:438)
at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:408)
at scala.collection.Iterator$JoinIterator.hasNext(Iterator.scala:219)
at scala.collection.Iterator$$anon$12.hasNext(Iterator.scala:438)
at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:408)
at scala.collection.Iterator$JoinIterator.hasNext(Iterator.scala:219)
at scala.collection.Iterator$$anon$12.hasNext(Iterator.scala:438)
at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:408)
at scala.collection.Iterator$JoinIterator.hasNext(Iterator.scala:219)
at scala.collection.Iterator$$anon$12.hasNext(Iterator.scala:438)
at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:408)
at scala.collection.Iterator$JoinIterator.hasNext(Iterator.scala:219)
at scala.collection.Iterator$$anon$12.hasNext(Iterator.scala:438)
at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:408)
at scala.collection.Iterator$JoinIterator.hasNext(Iterator.scala:219)
at scala.collection.Iterator$$anon$12.hasNext(Iterator.scala:438)
at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:408)
at scala.collection.Iterator$JoinIterator.hasNext(Iterator.scala:219)
at scala.collection.Iterator$$anon$12.hasNext(Iterator.scala:438)
at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:408)
at scala.collection.Iterator$JoinIterator.hasNext(Iterator.scala:219)
at scala.collection.Iterator$$anon$12.hasNext(Iterator.scala:438)
at scala.collection.Iterator$$anon$12.hasNext(Iterator.scala:438)
at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:408)
at org.apache.spark.util.Utils$.getIteratorSize(Utils.scala:1763)
at org.apache.spark.rdd.RDD$$anonfun$count$1.apply(RDD.scala:1134)
at org.apache.spark.rdd.RDD$$anonfun$count$1.apply(RDD.scala:1134)
at org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:1899)
at org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:1899)
at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:70)
at org.apache.spark.scheduler.Task.run(Task.scala:86)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:274)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
"org.apache.hadoop.fs.FileSystem$Statistics$StatisticsDataReferenceCleaner" #137 daemon prio=5 os_prio=0 tid=0x00007f74dc8c2800 nid=0x417b in Object.wait() [0x00007f755ede6000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x0000000408920fd8> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143)
- locked <0x0000000408920fd8> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:164)
at org.apache.hadoop.fs.FileSystem$Statistics$StatisticsDataReferenceCleaner.run(FileSystem.java:3063)
at java.lang.Thread.run(Thread.java:745)
"shuffle-client-1" #54 daemon prio=5 os_prio=0 tid=0x00007f74dc818000 nid=0x417a runnable [0x00007f755efe8000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
- locked <0x0000000405ee6968> (a io.netty.channel.nio.SelectedSelectionKeySet)
- locked <0x0000000405f108e0> (a java.util.Collections$UnmodifiableSet)
- locked <0x0000000405ee68d0> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111)
at java.lang.Thread.run(Thread.java:745)
"shuffle-client-0" #53 daemon prio=5 os_prio=0 tid=0x00007f74dc0dd000 nid=0x4179 runnable [0x00007f755ebe4000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
- locked <0x0000000405dc6e08> (a io.netty.channel.nio.SelectedSelectionKeySet)
- locked <0x0000000405dc8ef8> (a java.util.Collections$UnmodifiableSet)
- locked <0x0000000405dc6d60> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111)
at java.lang.Thread.run(Thread.java:745)
"shuffle-client-0" #67 daemon prio=5 os_prio=0 tid=0x00007f74dc094000 nid=0x40d1 runnable [0x00007f755e2b7000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
- locked <0x0000000405e03260> (a io.netty.channel.nio.SelectedSelectionKeySet)
- locked <0x0000000405e03250> (a java.util.Collections$UnmodifiableSet)
- locked <0x0000000405e03208> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310)
at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111)
at java.lang.Thread.run(Thread.java:745)
"Executor task launch worker-0" #66 daemon prio=5 os_prio=0 tid=0x00007f74b8002000 nid=0x40d0 runnable [0x00007f755eae2000]
java.lang.Thread.State: RUNNABLE
at scala.collection.Iterator$$anon$12.hasNext(Iterator.scala:438)
at scala.collection.Iterator$$anon$12.next(Iterator.scala:444)
at scala.collection.Iterator$$anon$11.next(Iterator.scala:409)
at scala.collection.Iterator$JoinIterator.next(Iterator.scala:232)
at scala.collection.Iterator$$anon$12.next(Iterator.scala:444)
at scala.collection.Iterator$$anon$11.next(Iterator.scala:409)
at scala.collection.Iterator$JoinIterator.next(Iterator.scala:232)
at scala.collection.Iterator$$anon$12.next(Iterator.scala:444)
at scala.collection.Iterator$$anon$11.next(Iterator.scala:409)
at scala.collection.Iterator$JoinIterator.next(Iterator.scala:232)
at scala.collection.Iterator$$anon$12.next(Iterator.scala:444)
at scala.collection.Iterator$$anon$11.next(Iterator.scala:409)
at scala.collection.Iterator$JoinIterator.next(Iterator.scala:232)
at scala.collection.Iterator$$anon$12.next(Iterator.scala:444)
at scala.collection.Iterator$$anon$11.next(Iterator.scala:409)
at scala.collection.Iterator$JoinIterator.next(Iterator.scala:232)
at scala.collection.Iterator$$anon$12.next(Iterator.scala:444)
at scala.collection.Iterator$$anon$11.next(Iterator.scala:409)
at scala.collection.Iterator$JoinIterator.next(Iterator.scala:232)
at scala.collection.Iterator$$anon$12.next(Iterator.scala:444)
at scala.collection.Iterator$$anon$11.next(Iterator.scala:409)
at scala.collection.Iterator$JoinIterator.next(Iterator.scala:232)
at scala.collection.Iterator$$anon$12.next(Iterator.scala:444)
at scala.collection.Iterator$$anon$11.next(Iterator.scala:409)
at scala.collection.Iterator$JoinIterator.next(Iterator.scala:232)
at scala.collection.Iterator$$anon$12.next(Iterator.scala:444)
at scala.collection.Iterator$$anon$11.next(Iterator.scala:409)
at scala.collection.Iterator$JoinIterator.next(Iterator.scala:232)
at scala.collection.Iterator$$anon$12.next(Iterator.scala:444)
at scala.collection.Iterator$$anon$11.next(Iterator.scala:409)
at scala.collection.Iterator$JoinIterator.next(Iterator.scala:232)
at scala.collection.Iterator$$anon$12.next(Iterator.scala:444)
at scala.collection.Iterator$$anon$11.next(Iterator.scala:409)
at scala.collection.Iterator$JoinIterator.next(Iterator.scala:232)
at scala.collection.Iterator$$anon$12.next(Iterator.scala:444)
at scala.collection.Iterator$$anon$11.next(Iterator.scala:409)
at scala.collection.Iterator$JoinIterator.next(Iterator.scala:232)
at scala.collection.Iterator$$anon$12.next(Iterator.scala:444)
at scala.collection.Iterator$$anon$11.next(Iterator.scala:409)
at scala.collection.Iterator$JoinIterator.next(Iterator.scala:232)
at scala.collection.Iterator$$anon$12.next(Iterator.scala:444)
at scala.collection.Iterator$$anon$11.next(Iterator.scala:409)
at scala.collection.Iterator$JoinIterator.next(Iterator.scala:232)
at scala.collection.Iterator$$anon$12.next(Iterator.scala:444)
at scala.collection.Iterator$$anon$11.next(Iterator.scala:409)
at scala.collection.Iterator$JoinIterator.next(Iterator.scala:232)
at scala.collection.Iterator$$anon$12.next(Iterator.scala:444)
at scala.collection.Iterator$$anon$11.next(Iterator.scala:409)
at scala.collection.Iterator$JoinIterator.next(Iterator.scala:232)
at scala.collection.Iterator$$anon$12.next(Iterator.scala:444)
at scala.collection.Iterator$$anon$11.next(Iterator.scala:409)
at scala.collection.Iterator$JoinIterator.next(Iterator.scala:232)
at scala.collection.Iterator$$anon$12.next(Iterator.scala:444)
at scala.collection.Iterator$$anon$12.next(Iterator.scala:444)
at scala.collection.Iterator$$anon$11.next(Iterator.scala:409)
at org.apache.spark.util.Utils$.getIteratorSize(Utils.scala:1765)
at org.apache.spark.rdd.RDD$$anonfun$count$1.apply(RDD.scala:1134)
at org.apache.spark.rdd.RDD$$anonfun$count$1.apply(RDD.scala:1134)
at org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:1899)
at org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:1899)
at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:70)
at org.apache.spark.scheduler.Task.run(Task.scala:86)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:274)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
function:
//name: CD_CreateSubClassIndexes
//params:
//idempotence: false
//code:
var codeUnitClasses = db.query("select from (select name from (SELECT expand(classes) FROM #0:1) where superClass = 'CodeUnit') order by name asc");
logger("starting CodeUnit index creation",'info');
for(var i=0; i<codeUnitClasses.length; i++){
var className = codeUnitClasses[i].field("name");
logger("starting CodeUnit index creation for class : " + className,'info');
db.command("CREATE INDEX "+className+".binName ON "+className+" (binName) NOTUNIQUE_HASH_INDEX");
db.command("CREATE INDEX "+className+".isExcluded ON "+className+" (isExcluded) NOTUNIQUE_HASH_INDEX");
db.command("CREATE INDEX "+className+".module ON "+className+" (module) NOTUNIQUE_HASH_INDEX");
db.command("CREATE INDEX "+className+".module_binName ON "+className+" (module,binName) NOTUNIQUE_HASH_INDEX");
db.command("CREATE INDEX "+className+".module_name_new ON "+className+" (module, name) NOTUNIQUE_HASH_INDEX");
db.command("CREATE INDEX "+className+".module_type ON "+className+" (module,type) NOTUNIQUE_HASH_INDEX");
db.command("CREATE INDEX "+className+".name ON "+className+" (name) NOTUNIQUE_HASH_INDEX");
db.command("CREATE INDEX "+className+".orgUnit ON "+className+" (orgUnit) NOTUNIQUE_HASH_INDEX");
db.command("CREATE INDEX "+className+".orgUnit_binName ON "+className+" (orgUnit,binName) NOTUNIQUE_HASH_INDEX");
}
OrientDB javascript function which creates indexes for classes is stuck after following message.
2016-08-09 11:27:12:773 WARNI {db=Insight5} Page in file internal.pcl
with index 7 was placed in wrong free list, this error will be fixed
automatically. [OPaginatedCluster]2016-08-09 11:27:49
I am using:
Windows 7 64 bit, java 8.0.31,
orientdb-community-2.1.20, Xmx4096m, -Dstorage.diskCache.bufferSize=2048
orientdb-server-config.xml:
<properties>
<entry value="50" name="db.pool.min"/>
<entry value="250" name="db.pool.max"/>
<entry value="1000" name="script.pool.maxSize"/>
<entry value="false" name="profiler.enabled"/>
<entry value="INFO" name="log.console.level"/>
<entry value="INFO" name="log.file.level"/>
<entry value="10000" name="storage.record.lockTimeout"/>
</properties>
thread dump:
Full thread dump Java HotSpot(TM) 64-Bit Server VM (25.31-b07 mixed mode):
"OrientDB HTTP Connection /0:0:0:0:0:0:0:1:2480<-/0:0:0:0:0:0:0:1:52523" #32 daemon prio=5 os_prio=0 tid=0x0000000021e5d800 nid=0x1a1c runnable [0x00000000230bf0
java.lang.Thread.State: RUNNABLE
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(SocketInputStream.java:150)
at java.net.SocketInputStream.read(SocketInputStream.java:121)
at java.io.BufferedInputStream.fill(BufferedInputStream.java:246)
at java.io.BufferedInputStream.read(BufferedInputStream.java:265)
- locked <0x00000006c3cdadf8> (a java.io.BufferedInputStream)
at com.orientechnologies.orient.enterprise.channel.text.OChannelText.read(OChannelText.java:61)
at com.orientechnologies.orient.server.network.protocol.http.ONetworkProtocolHttpAbstract.execute(ONetworkProtocolHttpAbstract.java:571)
at com.orientechnologies.common.thread.OSoftThread.run(OSoftThread.java:77)
"OrientDB HTTP Connection /0:0:0:0:0:0:0:1:2480<-/0:0:0:0:0:0:0:1:52494" #23 daemon prio=5 os_prio=0 tid=0x0000000022218800 nid=0x1cfc waiting on condition [0x00
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000006c05f14f0> (a java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:870)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1199)
at java.util.concurrent.locks.ReentrantReadWriteLock$WriteLock.lock(ReentrantReadWriteLock.java:943)
at com.orientechnologies.orient.core.storage.cache.OCachePointer.acquireExclusiveLock(OCachePointer.java:199)
at com.orientechnologies.orient.core.storage.cache.OCacheEntry.acquireExclusiveLock(OCacheEntry.java:79)
at com.orientechnologies.orient.core.storage.impl.local.paginated.OPaginatedCluster.updateFreePagesList(OPaginatedCluster.java:1593)
at com.orientechnologies.orient.core.storage.impl.local.paginated.OPaginatedCluster.updateFreePagesIndex(OPaginatedCluster.java:1549)
at com.orientechnologies.orient.core.storage.impl.local.paginated.OPaginatedCluster.findFreePage(OPaginatedCluster.java:1482)
at com.orientechnologies.orient.core.storage.impl.local.paginated.OPaginatedCluster.updateRecord(OPaginatedCluster.java:784)
at com.orientechnologies.orient.core.storage.impl.local.OAbstractPaginatedStorage.doUpdateRecord(OAbstractPaginatedStorage.java:1960)
at com.orientechnologies.orient.core.storage.impl.local.OAbstractPaginatedStorage.updateRecord(OAbstractPaginatedStorage.java:889)
at com.orientechnologies.orient.core.db.document.ODatabaseDocumentTx.executeSaveRecord(ODatabaseDocumentTx.java:2009)
at com.orientechnologies.orient.core.tx.OTransactionNoTx.saveRecord(OTransactionNoTx.java:160)
at com.orientechnologies.orient.core.db.document.ODatabaseDocumentTx.save(ODatabaseDocumentTx.java:2526)
at com.orientechnologies.orient.core.db.document.ODatabaseDocumentTx.save(ODatabaseDocumentTx.java:122)
at com.orientechnologies.orient.core.record.impl.ODocument.save(ODocument.java:1759)
at com.orientechnologies.orient.core.record.impl.ODocument.save(ODocument.java:1750)
at com.orientechnologies.orient.core.index.OIndexManagerAbstract$1.call(OIndexManagerAbstract.java:132)
at com.orientechnologies.orient.core.db.OScenarioThreadLocal.executeAsDistributed(OScenarioThreadLocal.java:71)
at com.orientechnologies.orient.core.index.OIndexManagerAbstract.save(OIndexManagerAbstract.java:122)
at com.orientechnologies.orient.core.index.OIndexManagerShared.createIndex(OIndexManagerShared.java:167)
at com.orientechnologies.orient.core.index.OIndexManagerProxy.createIndex(OIndexManagerProxy.java:75)
at com.orientechnologies.orient.core.metadata.schema.OClassImpl.createIndex(OClassImpl.java:1759)
at com.orientechnologies.orient.core.sql.OCommandExecutorSQLCreateIndex.execute(OCommandExecutorSQLCreateIndex.java:264)
at com.orientechnologies.orient.core.sql.OCommandExecutorSQLDelegate.execute(OCommandExecutorSQLDelegate.java:90)
at com.orientechnologies.orient.core.storage.impl.local.OAbstractPaginatedStorage.executeCommand(OAbstractPaginatedStorage.java:1433)
at com.orientechnologies.orient.core.storage.impl.local.OAbstractPaginatedStorage.command(OAbstractPaginatedStorage.java:1406)
at com.orientechnologies.orient.core.command.OCommandRequestTextAbstract.execute(OCommandRequestTextAbstract.java:67)
at com.orientechnologies.orient.core.command.script.OScriptDocumentDatabaseWrapper.command(OScriptDocumentDatabaseWrapper.java:114)
at com.orientechnologies.orient.core.command.script.OScriptDocumentDatabaseWrapper.command(OScriptDocumentDatabaseWrapper.java:110)
at java.lang.invoke.LambdaForm$DMH/232925641.invokeVirtual_LL_L(LambdaForm$DMH)
at java.lang.invoke.LambdaForm$NamedFunction.invoke_LL_L(LambdaForm.java:1108)
at java.lang.invoke.LambdaForm$DMH/1971489295.invokeStatic_LL_L(LambdaForm$DMH)
at java.lang.invoke.LambdaForm$NamedFunction.invokeWithArguments(LambdaForm.java:1147)
at java.lang.invoke.LambdaForm.interpretName(LambdaForm.java:625)
at java.lang.invoke.LambdaForm.interpretWithArguments(LambdaForm.java:604)
at java.lang.invoke.LambdaForm$LFI/519569038.interpret_L(LambdaForm$LFI)
at java.lang.invoke.LambdaForm$NamedFunction.invoke_LL_L(LambdaForm.java:1108)
at java.lang.invoke.LambdaForm$DMH/1971489295.invokeStatic_LL_L(LambdaForm$DMH)
at java.lang.invoke.LambdaForm$NamedFunction.invokeWithArguments(LambdaForm.java:1147)
at java.lang.invoke.LambdaForm.interpretName(LambdaForm.java:625)
at java.lang.invoke.LambdaForm.interpretWithArguments(LambdaForm.java:604)
at java.lang.invoke.LambdaForm$LFI/519569038.interpret_L(LambdaForm$LFI)
at java.lang.invoke.LambdaForm$MH/1800412487.exactInvoker(LambdaForm$MH)
at java.lang.invoke.LambdaForm$BMH/1828757853.reinvoke(LambdaForm$BMH)
at java.lang.invoke.LambdaForm$NamedFunction.invoke_LLL_L(LambdaForm.java:1113)
at java.lang.invoke.LambdaForm$DMH/1971489295.invokeStatic_LL_L(LambdaForm$DMH)
at java.lang.invoke.LambdaForm$NamedFunction.invokeWithArguments(LambdaForm.java:1147)
at java.lang.invoke.LambdaForm.interpretName(LambdaForm.java:625)
at java.lang.invoke.LambdaForm.interpretWithArguments(LambdaForm.java:604)
at java.lang.invoke.LambdaForm$LFI/1844169442.interpret_L(LambdaForm$LFI)
at java.lang.invoke.LambdaForm$DMH/2114694065.invokeSpecial_LLLL_L(LambdaForm$DMH)
at java.lang.invoke.LambdaForm$NamedFunction.invoke_LLLL_L(LambdaForm.java:1118)
at java.lang.invoke.LambdaForm$DMH/1971489295.invokeStatic_LL_L(LambdaForm$DMH)
at java.lang.invoke.LambdaForm$NamedFunction.invokeWithArguments(LambdaForm.java:1147)
at java.lang.invoke.LambdaForm.interpretName(LambdaForm.java:625)
at java.lang.invoke.LambdaForm.interpretWithArguments(LambdaForm.java:604)
at java.lang.invoke.LambdaForm$LFI/1844169442.interpret_L(LambdaForm$LFI)
at java.lang.invoke.LambdaForm$MH/1786292209.linkToCallSite(LambdaForm$MH)
at jdk.nashorn.internal.scripts.Script$\^eval\_$9.CD_CreateSubClassIndexes(<eval>:4628)
at java.lang.invoke.LambdaForm$DMH/1971489295.invokeStatic_LL_L(LambdaForm$DMH)
at java.lang.invoke.LambdaForm$MH/1374677625.invokeExact_MT(LambdaForm$MH)
at jdk.nashorn.internal.runtime.ScriptFunctionData.invoke(ScriptFunctionData.java:535)
at jdk.nashorn.internal.runtime.ScriptFunction.invoke(ScriptFunction.java:209)
at jdk.nashorn.internal.runtime.ScriptRuntime.apply(ScriptRuntime.java:378)
at jdk.nashorn.api.scripting.ScriptObjectMirror.callMember(ScriptObjectMirror.java:185)
at jdk.nashorn.api.scripting.NashornScriptEngine.invokeImpl(NashornScriptEngine.java:505)
at jdk.nashorn.api.scripting.NashornScriptEngine.invokeFunction(NashornScriptEngine.java:227)
at com.orientechnologies.orient.core.command.script.OCommandExecutorFunction.executeInContext(OCommandExecutorFunction.java:94)
at com.orientechnologies.orient.core.metadata.function.OFunction.executeInContext(OFunction.java:149)
at com.orientechnologies.orient.server.network.protocol.http.command.all.OServerCommandAbstractLogic.execute(OServerCommandAbstractLogic.java:77)
at com.orientechnologies.orient.server.network.protocol.http.ONetworkProtocolHttpAbstract.service(ONetworkProtocolHttpAbstract.java:180)
at com.orientechnologies.orient.server.network.protocol.http.ONetworkProtocolHttpAbstract.execute(ONetworkProtocolHttpAbstract.java:627)
at com.orientechnologies.common.thread.OSoftThread.run(OSoftThread.java:77)
"OrientDB Write Cache Flush Task (Insight5)" #22 daemon prio=10 os_prio=2 tid=0x000000001e69b800 nid=0x233c runnable [0x0000000020fdf000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000006c0a5c6d0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
"OrientDB WAL Flush Task (Insight5)" #21 daemon prio=5 os_prio=0 tid=0x000000001f512800 nid=0xd44 waiting on condition [0x0000000020e8e000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000006c09edfc8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
"DestroyJavaVM" #19 prio=5 os_prio=0 tid=0x00000000021ea000 nid=0x1eb4 waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"OrientDB ONetworkProtocolHttpDb listen at 0.0.0.0:2480-2490" #16 prio=5 os_prio=0 tid=0x000000001e8bd800 nid=0x15b8 runnable [0x000000002080e000]
java.lang.Thread.State: RUNNABLE
at java.net.DualStackPlainSocketImpl.accept0(Native Method)
at java.net.DualStackPlainSocketImpl.socketAccept(DualStackPlainSocketImpl.java:131)
at java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:404)
at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:199)
- locked <0x00000006c098b330> (a java.net.SocksSocketImpl)
at java.net.ServerSocket.implAccept(ServerSocket.java:545)
at java.net.ServerSocket.accept(ServerSocket.java:513)
at com.orientechnologies.orient.server.network.OServerNetworkListener.run(OServerNetworkListener.java:194)
"OrientDB ONetworkProtocolBinary listen at 0.0.0.0:2424-2430" #14 prio=5 os_prio=0 tid=0x000000001e8c3000 nid=0x16e8 runnable [0x00000000204de000]
java.lang.Thread.State: RUNNABLE
at java.net.DualStackPlainSocketImpl.accept0(Native Method)
at java.net.DualStackPlainSocketImpl.socketAccept(DualStackPlainSocketImpl.java:131)
at java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:404)
at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:199)
- locked <0x00000006c098cf40> (a java.net.SocksSocketImpl)
at java.net.ServerSocket.implAccept(ServerSocket.java:545)
at java.net.ServerSocket.accept(ServerSocket.java:513)
at com.orientechnologies.orient.server.network.OServerNetworkListener.run(OServerNetworkListener.java:194)
"Timer-0" #11 daemon prio=5 os_prio=0 tid=0x000000001e792000 nid=0xd80 in Object.wait() [0x000000001ee3f000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.util.TimerThread.mainLoop(Timer.java:552)
- locked <0x00000006c094a3e8> (a java.util.TaskQueue)
at java.util.TimerThread.run(Timer.java:505)
"Service Thread" #9 daemon prio=9 os_prio=0 tid=0x000000001d6a5000 nid=0x2108 runnable [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"C1 CompilerThread2" #8 daemon prio=9 os_prio=2 tid=0x000000001d63a800 nid=0x1bec waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"C2 CompilerThread1" #7 daemon prio=9 os_prio=2 tid=0x000000001d639000 nid=0x1c64 waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"C2 CompilerThread0" #6 daemon prio=9 os_prio=2 tid=0x000000001d632800 nid=0x19ec waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"Attach Listener" #5 daemon prio=5 os_prio=2 tid=0x000000001c045800 nid=0x14cc runnable [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"Signal Dispatcher" #4 daemon prio=9 os_prio=2 tid=0x000000001c044800 nid=0x2304 waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"Finalizer" #3 daemon prio=8 os_prio=1 tid=0x000000001bfe8000 nid=0x1604 in Object.wait() [0x000000001d62f000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:142)
- locked <0x00000006c0861d78> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:158)
at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:209)
"Reference Handler" #2 daemon prio=10 os_prio=2 tid=0x000000001bfdf000 nid=0x20c0 in Object.wait() [0x000000001d35f000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:502)
at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:157)
- locked <0x00000006c0b40cc8> (a java.lang.ref.Reference$Lock)
"VM Thread" os_prio=2 tid=0x000000001bfdb000 nid=0x2078 runnable
"GC task thread#0 (ParallelGC)" os_prio=0 tid=0x00000000021fe800 nid=0xbfc runnable
"GC task thread#1 (ParallelGC)" os_prio=0 tid=0x0000000002200000 nid=0x768 runnable
"GC task thread#2 (ParallelGC)" os_prio=0 tid=0x0000000002201800 nid=0x264 runnable
"GC task thread#3 (ParallelGC)" os_prio=0 tid=0x0000000002203000 nid=0x1d9c runnable
"VM Periodic Task Thread" os_prio=2 tid=0x000000001d6a6800 nid=0x15e8 waiting on condition
JNI global references: 2513
Heap
PSYoungGen total 928256K, used 76187K [0x000000076ab00000, 0x00000007bce00000, 0x00000007c0000000)
eden space 924672K, 7% used [0x000000076ab00000,0x000000076f1f1ff0,0x00000007a3200000)
from space 3584K, 98% used [0x00000007bbc00000,0x00000007bbf74dc0,0x00000007bbf80000)
to space 10752K, 0% used [0x00000007bc380000,0x00000007bc380000,0x00000007bce00000)
ParOldGen total 235008K, used 130502K [0x00000006c0000000, 0x00000006ce580000, 0x000000076ab00000)
object space 235008K, 55% used [0x00000006c0000000,0x00000006c7f71a28,0x00000006ce580000)
Metaspace used 30289K, capacity 31100K, committed 34304K, reserved 1081344K
class space used 3052K, capacity 3256K, committed 3328K, reserved 1048576K