Strange exception in SBT test - scala

I am facing an issue with SBT (version 0.13.9). One of my ScalaTest tests fails not deterministic with EOFException.
Stack trace:
Exception in thread "Thread-155" Exception in thread "Thread-159" java.io.EOFException
at java.io.ObjectInputStream$BlockDataInputStream.peekByte(ObjectInputStream.java:2601)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1319)
at java.io.ObjectInputStream.readObject(ObjectInputStream.java:371)
at sbt.React.react(ForkTests.scala:114)
at sbt.ForkTests$$anonfun$mainTestTask$1$Acceptor$2$.run(ForkTests.scala:74)
at java.lang.Thread.run(Thread.java:745)
java.io.EOFException
at java.io.ObjectInputStream$BlockDataInputStream.peekByte(ObjectInputStream.java:2601)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1319)
at java.io.ObjectInputStream.readObject(ObjectInputStream.java:371)
at org.scalatest.tools.Framework$ScalaTestRunner$Skeleton$1$React.react(Framework.scala:953)
at org.scalatest.tools.Framework$ScalaTestRunner$Skeleton$1.run(Framework.scala:942)
at java.lang.Thread.run(Thread.java:745)
sbt.ForkMain 59974 failed with exit code 134
I don't see any JVM logs or thread dumps.

Almost certainly this is related to the Java classpath being too long when invoking sbt.ForkMain in certain linux distros. To avoid this I added
javaOptions in Test ++= Seq("-Xms1G","-XX:+CMSClassUnloadingEnabled","-XX:+UseConcMarkSweepGC")
to the build.sbt file. I reckon CMSClassUnloadingEnabled is where the magic happens.

SBT single thread or multi threaded tests forks the JVM and if the host name is not resolved it just fails with this error message. If you look at the logs in the tests, you might see
Uncaught exception when running tests: java.net.ConnectException: Operation timed out
So the easy way to resolve this is to add your hostname to the /etc/hosts and map it to localhost something like this
::1 localhost blahblah
This should resolve the issue.

Related

Getting following error in console while solr connection to Mongo . No error in solr log

Exception in thread "Thread-14" java.lang.NoSuchMethodError: com.mongodb.ReadPreference.secondaryPreferred()Lcom/mongodb/ReadPreference;
at org.apache.solr.handler.dataimport.MongoDataSource.init(MongoDataSource.java:51)
at org.apache.solr.handler.dataimport.DataImporter.getDataSourceInstance(DataImporter.java:397)
at org.apache.solr.handler.dataimport.ContextImpl.getDataSource(ContextImpl.java:100)
at org.apache.solr.handler.dataimport.MongoEntityProcessor.init(MongoEntityProcessor.java:33)
at org.apache.solr.handler.dataimport.EntityProcessorWrapper.init(EntityProcessorWrapper.java:77)
at org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java:434)
at org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java:415)
at org.apache.solr.handler.dataimport.DocBuilder.doFullDump(DocBuilder.java:330)
at org.apache.solr.handler.dataimport.DocBuilder.execute(DocBuilder.java:233)
at org.apache.solr.handler.dataimport.DataImporter.doFullImport(DataImporter.java:424)
at org.apache.solr.handler.dataimport.DataImporter.runCmd(DataImporter.java:483)
at org.apache.solr.handler.dataimport.DataImporter.lambda$runAsync$0(DataImporter.java:466)
at java.lang.Thread.run(Thread.java:745)
There are jar files interdependent (though I took all latest was not compatible). Finally the combination worked is listed below.
solr-dataimporthandler-8.3.0.jar
solr-mongo-importer-1.0.0.jar
mongo-java-driver-2.13.2.jar
It was frustrating.

wiremock standalone server not starting using --print-all-network-traffic

I am trying to run wiremock stand along server using command .
java -jar wiremock-1.57-standalone.jar --port=9090 --verbose --print-all-network-traffic
But it always give me exception .
Exception in thread "main"
Exception: java.util.MissingResourceException thrown from the
UncaughtExceptionHandler in thread "main"
But after removing network command it starts fine.
Could you pleas help me with this.
Thanks
It was issue of old versions i updated jar file from 1.5 to 2.6

typesafe activator ui not launching

play.api.Application$$anon$1: Execution exception[[IllegalArgumentException: req
uirement failed: Source file 'C:\Users\shriv_000\.activator\1.3.6\templates\inde
x.db_e25b80033130c08.tmp' is a directory.]]
at play.api.Application$class.handleError(Application.scala:296) ~[play_
2.11-2.3.9.jar:2.3.9]
at play.api.DefaultApplication.handleError(Application.scala:402) [play_
2.11-2.3.9.jar:2.3.9]
at play.core.server.netty.PlayDefaultUpstreamHandler$$anonfun$14$$anonfu
n$apply$1.applyOrElse(PlayDefaultUpstreamHandler.scala:205) [play_2.11-2.3.9.jar
:2.3.9]
at play.core.server.netty.PlayDefaultUpstreamHandler$$anonfun$14$$anonfu
n$apply$1.applyOrElse(PlayDefaultUpstreamHandler.scala:202) [play_2.11-2.3.9.jar
:2.3.9]
at scala.runtime.AbstractPartialFunction.apply(AbstractPartialFunction.s
cala:36) [scala-library.jar:0.13.8]
Caused by: java.lang.IllegalArgumentException: requirement failed: Source file '
C:\Users\shriv_000\.activator\1.3.6\templates\index.db_e25b80033130c08.tmp' is a
directory.
at scala.Predef$.require(Predef.scala:219) ~[scala-library.jar:0.13.8]
at sbt.IO$.copyFile(IO.scala:584) ~[client-all-2-11-0.3.5.jar:0.3.5]
at sbt.IO$.move(IO.scala:786) ~[client-all-2-11-0.3.5.jar:0.3.5]
at activator.package$RichIO$.createViaTemporary$extension(package.scala:
30) ~[activator-templates-cache-1.0-a0afb008ea619bf9d87dc010156cddffa8a6f880.jar
:1.3.6]
at activator.templates.repository.UriRemoteTemplateRepository$$anonfun$r
esolveIndexTo$1.apply(UriRemoteTemplateRepository.scala:228) ~[activator-templat
es-cache-1.0-a0afb008ea619bf9d87dc010156cddffa8a6f880.jar:1.0-a0afb008ea619bf9d8
7dc010156cddffa8a6f880]
[info] application - onStop received closing down the app
I am at activator 1.3.6
I have seen TypeSafe Activator installation error and it might be different then my problem. I have been able to create and launch a play project.
The solution provided there did not work for me.
I see this problem frequently.
I found a workaround. Try to execute activator as an administrator.

Spark fails on big shuffle jobs with java.io.IOException: Filesystem closed

I often find spark fails with large jobs with a rather unhelpful meaningless exception. The worker logs look normal, no errors, but they get state "KILLED". This is extremely common for large shuffles, so operations like .distinct.
The question is, how do I diagnose what's going wrong, and ideally, how do I fix it?
Given that a lot of these operations are monoidal I've been working around the problem by splitting the data into, say 10, chunks, running the app on each chunk, then running the app on all of the resulting outputs. In other words - meta-map-reduce.
14/06/04 12:56:09 ERROR client.AppClient$ClientActor: Master removed our application: FAILED; stopping client
14/06/04 12:56:09 WARN cluster.SparkDeploySchedulerBackend: Disconnected from Spark cluster! Waiting for reconnection...
14/06/04 12:56:09 WARN scheduler.TaskSetManager: Loss was due to java.io.IOException
java.io.IOException: Filesystem closed
at org.apache.hadoop.hdfs.DFSClient.checkOpen(DFSClient.java:703)
at org.apache.hadoop.hdfs.DFSInputStream.readWithStrategy(DFSInputStream.java:779)
at org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:840)
at java.io.DataInputStream.read(DataInputStream.java:149)
at org.apache.hadoop.io.compress.DecompressorStream.getCompressedData(DecompressorStream.java:159)
at org.apache.hadoop.io.compress.DecompressorStream.decompress(DecompressorStream.java:143)
at org.apache.hadoop.io.compress.DecompressorStream.read(DecompressorStream.java:85)
at java.io.InputStream.read(InputStream.java:101)
at org.apache.hadoop.util.LineReader.fillBuffer(LineReader.java:180)
at org.apache.hadoop.util.LineReader.readDefaultLine(LineReader.java:216)
at org.apache.hadoop.util.LineReader.readLine(LineReader.java:174)
at org.apache.hadoop.mapred.LineRecordReader.next(LineRecordReader.java:209)
at org.apache.hadoop.mapred.LineRecordReader.next(LineRecordReader.java:47)
at org.apache.spark.rdd.HadoopRDD$$anon$1.getNext(HadoopRDD.scala:164)
at org.apache.spark.rdd.HadoopRDD$$anon$1.getNext(HadoopRDD.scala:149)
at org.apache.spark.util.NextIterator.hasNext(NextIterator.scala:71)
at org.apache.spark.InterruptibleIterator.hasNext(InterruptibleIterator.scala:27)
at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)
at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)
at scala.collection.Iterator$class.foreach(Iterator.scala:727)
at scala.collection.AbstractIterator.foreach(Iterator.scala:1157)
at scala.collection.generic.Growable$class.$plus$plus$eq(Growable.scala:48)
at scala.collection.mutable.ListBuffer.$plus$plus$eq(ListBuffer.scala:176)
at scala.collection.mutable.ListBuffer.$plus$plus$eq(ListBuffer.scala:45)
at scala.collection.TraversableOnce$class.to(TraversableOnce.scala:273)
at scala.collection.AbstractIterator.to(Iterator.scala:1157)
at scala.collection.TraversableOnce$class.toList(TraversableOnce.scala:257)
at scala.collection.AbstractIterator.toList(Iterator.scala:1157)
at $line5.$read$$iwC$$iwC$$iwC$$iwC$$anonfun$2.apply(<console>:13)
at $line5.$read$$iwC$$iwC$$iwC$$iwC$$anonfun$2.apply(<console>:13)
at org.apache.spark.rdd.RDD$$anonfun$1.apply(RDD.scala:450)
at org.apache.spark.rdd.RDD$$anonfun$1.apply(RDD.scala:450)
at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:34)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:241)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:232)
at org.apache.spark.rdd.MappedRDD.compute(MappedRDD.scala:31)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:241)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:232)
at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:34)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:241)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:232)
at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:161)
at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:102)
at org.apache.spark.scheduler.Task.run(Task.scala:53)
at org.apache.spark.executor.Executor$TaskRunner$$anonfun$run$1.apply$mcV$sp(Executor.scala:213)
at org.apache.spark.deploy.SparkHadoopUtil$$anon$1.run(SparkHadoopUtil.scala:42)
at org.apache.spark.deploy.SparkHadoopUtil$$anon$1.run(SparkHadoopUtil.scala:41)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)
at org.apache.spark.deploy.SparkHadoopUtil.runAsUser(SparkHadoopUtil.scala:41)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:178)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
As of September 1st 2014, this is an "open improvement" in Spark. Please see https://issues.apache.org/jira/browse/SPARK-3052. As syrza pointed out in the given link, the shutdown hooks are likely done in incorrect order when an executor failed which results in this message. I understand you will have to little more investigation to figure out the main cause of problem (i.e. why your executor failed). If it is a large shuffle, it might be an out-of-memory error which cause executor failure which then caused the Hadoop Filesystem to be closed in their shutdown hook. So, the RecordReaders in running tasks of that executor throw "java.io.IOException: Filesystem closed" exception. I guess it will be fixed in subsequent release and then you will get more helpful error message :)
Something calls DFSClient.close() or DFSClient.abort(), closing the client. The next file operation then results in the above exception.
I would try to figure out what calls close()/abort(). You could use a breakpoint in your debugger, or modify the Hadoop source code to throw an exception in these methods, so you would get a stack trace.
The exception about “file system closed” can be solved if the spark job is running on a cluster. You can set properties like spark.executor.cores , spark.driver.cores and spark.akka.threads to the maximum values w.r.t your resource availability. I had the same problem when my dataset was pretty large with JSON data about 20 million records. I fixed it with the above properties and it ran like a charm. In my case, I set those properties to 25,25 and 20 respectively. Hope it helps!!
Reference Link:
http://spark.apache.org/docs/latest/configuration.html

Error constantly thrown by JBoss

2011-06-02 17:27:01,990 ERROR [org.apache.tomcat.util.net.PoolTcpEndpoint] Socket error caused by remote host /a.b.c.d
java.net.SocketException: Invalid argument
at java.net.PlainSocketImpl.socketSetOption(Native Method)
at java.net.PlainSocketImpl.setOption(PlainSocketImpl.java:264)
at java.net.Socket.setTcpNoDelay(Socket.java:839)
at org.apache.tomcat.util.net.PoolTcpEndpoint.setSocketOptions(PoolTcpEndpoint.java:503)
at org.apache.tomcat.util.net.PoolTcpEndpoint.processSocket(PoolTcpEndpoint.java:515)
at org.apache.tomcat.util.net.MasterSlaveWorkerThread.run(MasterSlaveWorkerThread.java:112)
at java.lang.Thread.run(Thread.java:595)
This is most likely an error in your JVM. If so, its been fixed in the latest VMs.
http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=6378870
What version of Java are you using and on what platform?
Can you provide more info on the issue?