I have installed Zeppelin 0.9.0 on my Ubuntu 20.04 machine.
In interpreters spark.jars I have mongo-spark-connector, mongo-java-driver and bson.
I successfully imported com.mongodb.spark, org.bson.Document and other necessary packages, but when I want to execute
val rdd = MongoSpark.load(sc)
appears error:
java.lang.NoClassDefFoundError: org/bson/conversions/Bson
... 66 elided
Caused by: java.lang.ClassNotFoundException: org.bson.conversions.Bson
at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:581)
at java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:178)
at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:522)
... 66 more
Also, I have spark version 3.1.1, java version 11.0.10, scala version 2.12.10.
I found solution.
I've put the following jars in interpreter/spark/dep folder and it works:
bson-4.3.1.jar
mongodb-driver-core-4.3.1.jar
mongo-java-driver-3.12.10.jar
mongo-spark-connector_2.12-3.0.1.jar
zeppelin-mongodb-0.9.0.jar
Related
When following the json4s instructions on Serialization I keep getting the followinng error:
java.lang.NoClassDefFoundError: java/sql/Timestamp
at org.json4s.reflect.Reflector$.(Reflector.scala:22)
at org.json4s.reflect.Reflector$.(Reflector.scala)
at org.json4s.Extraction$.internalDecomposeWithBuilder(Extraction.scala:160)
at org.json4s.Extraction$.decomposeWithBuilder(Extraction.scala:65)
at org.json4s.native.Serialization$.write(Serialization.scala:43)
at org.json4s.native.Serialization$.write(Serialization.scala:37)
... 34 elided
Caused by: java.lang.ClassNotFoundException: java.sql.Timestamp
at java.base/java.net.URLClassLoader.findClass(URLClassLoader.java:436)
at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:588)
at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:521)
... 40 more
My case class does not have any java.sql.Timemstamp values.
What is needed to setup scala 2.12, and openjdk 12, with json4s properly?
I have setup my sbt in compliance with the instructions and I am able to import java.sql.Timestamp in my project.
Thank you in advance for your consideration and response.
Exception while running python code in Windows 10. I am using Apache Kafka and PySpark.
Python code snippet to read data from Kafka
ssc=StreamingContext(sc,60)
zkQuorum, topic = sys.argv[1:]
kvs=KafkaUtils.createStream(ssc, zkQuorum, "spark-streaming-consumer", {topic: 1})
lines = kvs.map(lambda x: [x[0],x[1]])
lines.pprint()
lines.foreachRDD(SaveRecord)
ssc.start()
ssc.awaitTermination()
Exception while running the code
Exception in thread "streaming-start" java.lang.NoClassDefFoundError: org/apache/spark/internal/Logging$class
at org.apache.spark.streaming.kafka.KafkaReceiver.<init>(KafkaInputDStream.scala:69)
at org.apache.spark.streaming.kafka.KafkaInputDStream.getReceiver(KafkaInputDStream.scala:60)
at org.apache.spark.streaming.scheduler.ReceiverTracker.$anonfun$launchReceivers$1(ReceiverTracker.scala:441)
at scala.collection.TraversableLike.$anonfun$map$1(TraversableLike.scala:237)
at scala.collection.IndexedSeqOptimized.foreach(IndexedSeqOptimized.scala:36)
at scala.collection.IndexedSeqOptimized.foreach$(IndexedSeqOptimized.scala:33)
at scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:198)
at scala.collection.TraversableLike.map(TraversableLike.scala:237)
at scala.collection.TraversableLike.map$(TraversableLike.scala:230)
at scala.collection.mutable.ArrayOps$ofRef.map(ArrayOps.scala:198)
at org.apache.spark.streaming.scheduler.ReceiverTracker.launchReceivers(ReceiverTracker.scala:440)
at org.apache.spark.streaming.scheduler.ReceiverTracker.start(ReceiverTracker.scala:160)
at org.apache.spark.streaming.scheduler.JobScheduler.start(JobScheduler.scala:102)
at org.apache.spark.streaming.StreamingContext.$anonfun$start$1(StreamingContext.scala:583)
at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
at org.apache.spark.util.ThreadUtils$$anon$1.run(ThreadUtils.scala:145)
Caused by: java.lang.ClassNotFoundException: org.apache.spark.internal.Logging$class
at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
... 16 more
This may be due to incompatible version of Scala with Spark. Make sure your Scala Version in Project configuration matches with the Version your Spark Version supports.
Spark requires Scala 2.12; support for Scala 2.11 was removed in Spark 3.0.0
It is also possible that the third party jar (like dstream-twitter for twitter streaming application or your Kafka streaming jar) is built for unsupported version of Scala in your application.
For me dstream-twitter_2.11-2.3.0-SNAPSHOT For Instance didn't work with Spark 3.0, It gave Exception in thread "streaming-start" java.lang.NoClassDefFoundError: org/apache/spark/internal/Logging$class). But when I updated the dtream-twitter jar with scala 2.12 version it solved the issue.
Make sure you get all the Scala Versions correct.
When right-click running a test class, Eclipse failed with
Caused by: java.lang.NoClassDefFoundError: scala/Product$class
at org.scalatest.time.Days$.<init>(Units.scala:291)
at org.scalatest.time.Days$.<clinit>(Units.scala)
at org.scalatest.time.Span$.<init>(Span.scala:585)
at org.scalatest.time.Span$.<clinit>(Span.scala)
at org.scalatest.tools.Runner$.<init>(Runner.scala:779)
at org.scalatest.tools.Runner$.<clinit>(Runner.scala)
at org.scalatest.tools.Runner.main(Runner.scala)
... 6 more
Caused by: java.lang.ClassNotFoundException: scala.Product$class
at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
... 13 more
Yet it ran in the command line with sbt test. The libraries have been updated as described in java.lang.NoClassDefFoundError: scala/Product$class.
This happened with the latest Scala IDE (4.7.0-vfinal-2017-09-29T14:34:02Z-Typesafe) with the patmat project from Coursera's scala course.
What is the cause and how can it be fixed?
Requested info
The Java Build Path is
You have a combination of _2.11 libraries and 2.12.3 Scala library, this won't work. It looks like the _2.11 dependendencies come from SBT (judging from the paths).
You need to either change Scala IDE's Scala version (setting correct scala version on scala ide explains how) or set scalaVersion := "2.12.3" in the SBT project and rerun sbt eclipse.
Please use the Scala library version 2.11 as the other Scala based dependencies like scala-xml and scalatest are based on Scala library version 2.11
I am new to spark application programming, and therefore struggling here with this basic one..
I have scala ide and attached relevant jar files from the latest hadoop and spark distributions. There is just one basic scala object that i am working with -
hadoop - 2.7
spark - 2.0.0
I have attempted this with both scenarios, when hadoop processes are running on my laptop and also when they are not running.. its the same behaviour. Btw, spark shell is not complaining of anything
import org.apache.spark.SparkConf
object SparkAppTest {
def main(args : Array[String]) {
val conf = new SparkConf().setAppName("Spark test")
conf.setMaster("spark://master:7077")
conf.setSparkHome("/hadoop/spark")
conf.set("spark.driver.host","localhost")
}
}
When I am trying to "run" this using eclipse -> run as scala app this is failing with the following error -
Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/hadoop/fs/FSDataInputStream
at org.apache.spark.SparkConf.loadFromSystemProperties(SparkConf.scala:65)
at org.apache.spark.SparkConf.<init>(SparkConf.scala:60)
at org.apache.spark.SparkConf.<init>(SparkConf.scala:55)
at SparkAppTest$.main(SparkAppTest.scala:6)
at SparkAppTest.main(SparkAppTest.scala)
Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.fs.FSDataInputStream
at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
... 5 more
I am trying to run a play framework(2.1) application which is written in Scala but getting some error.
I need to use Play framework 2.1 only.
As per build.sbt file, I am using Scala 2.11.7.
sbt.version=0.13.8 is in project/build.properties.
I have installed Play 2.1.0 and changed my directory to application's root directory and then executed play command.
I got following error:
java.lang.NoClassDefFoundError: scala/StringContext
at sbt.CommandStrings$.<init>(CommandStrings.scala:83)
at sbt.CommandStrings$.<clinit>(CommandStrings.scala)
at sbt.BuiltinCommands$.defaults(Main.scala:202)
at sbt.xMain.run(Main.scala:29)
at xsbt.boot.Launch$.run(Launch.scala:55)
at xsbt.boot.Launch$$anonfun$explicit$1.apply(Launch.scala:45)
at xsbt.boot.Launch$.launch(Launch.scala:69)
at xsbt.boot.Launch$.apply(Launch.scala:16)
at xsbt.boot.Boot$.runImpl(Boot.scala:31)
at xsbt.boot.Boot$.main(Boot.scala:20)
at xsbt.boot.Boot.main(Boot.scala)
Caused by: java.lang.ClassNotFoundException: scala.StringContext
at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
... 11 more
Error during sbt exeenter code herecution: java.lang.NoClassDefFoundError: scala/StringContext
Can someone please tell me what's wrong here?
I am completely new to scala and I just need to run this app at earliest.
I am using 64-bit Cent OS 7(in case someone wants to know about OS).
The easiest way to fix that is to use an older version of SBT. Play 2.1.5 uses 0.12.2 by default, so you can change the file project/build.properties inside your project and use that version:
sbt.version=0.12.2
You can later try newer versions of SBT if you need.
sbt.version=0.12.2 wont work with JDK8 so you might need to downgrade Java.
I used this sbt version and got the following error
eror: error while loading CharSequence, class file '/usr/lib/jvm/java-8-openjdk-amd64/jre/lib/rt.jar(java/lang/CharSequence.class)' is broken for sbt 0.12.2