Spark Runtime Error - ClassDefNotFound: SparkConf - scala

Question

After installing and building Apache Spark (albeit with quite a few warnings), the compilation of our Spark application (using "sbt package") completes successfully. However, when trying to run our application using the spark-submit script, a runtime error results that states that the SparkConf class definition was not found. The SparkConf.scala file is present on our system, but it seems as if it is not being built correctly. Any ideas on how to solve this?
user#compname:~/Documents/TestApp$ /opt/Spark/spark-1.4.0/bin/spark-submit --master local[4] --jars /opt/Spark/spark-1.4.0/jars/elasticsearch-hadoop-2.1.0.Beta2.jar target/scala-2.11/sparkesingest_2.11-1.0.0.jar ~/Desktop/CSV/data.csv es-index localhost
Warning: Local jar /opt/Spark/spark-1.4.0/jars/elasticsearch-hadoop-2.1.0.Beta2.jar does not exist, skipping.
log4j:WARN No appenders could be found for logger (App).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
15/07/01 13:56:58 INFO SparkContext: Running Spark version 1.4.0
15/07/01 13:56:59 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
15/07/01 13:56:59 WARN Utils: Your hostname, compname resolves to a loopback address: 127.0.1.1; using [IP ADDRESS] instead (on interface eth0)
15/07/01 13:56:59 WARN Utils: Set SPARK_LOCAL_IP if you need to bind to another address
15/07/01 13:56:59 INFO SecurityManager: Changing view acls to: user
15/07/01 13:56:59 INFO SecurityManager: Changing modify acls to: user
15/07/01 13:56:59 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(user); users with modify permissions: Set(user)
15/07/01 13:56:59 INFO Slf4jLogger: Slf4jLogger started
15/07/01 13:56:59 INFO Remoting: Starting remoting
15/07/01 13:56:59 INFO Remoting: Remoting started; listening on addresses :[akka.tcp://sparkDriver#[IP ADDRESS]]
15/07/01 13:56:59 INFO Utils: Successfully started service 'sparkDriver' on port 34276.
15/07/01 13:56:59 INFO SparkEnv: Registering MapOutputTracker
15/07/01 13:56:59 INFO SparkEnv: Registering BlockManagerMaster
15/07/01 13:56:59 INFO DiskBlockManager: Created local directory at /tmp/spark-c206e297-c2ef-4bbf-9bd2-de642804bdcd/blockmgr-8d273f32-589e-4f55-98a2-cf0322a05d45
15/07/01 13:56:59 INFO MemoryStore: MemoryStore started with capacity 265.4 MB
15/07/01 13:56:59 INFO HttpFileServer: HTTP File server directory is /tmp/spark-c206e297-c2ef-4bbf-9bd2-de642804bdcd/httpd-f4c3c67a-d058-4aba-bd65-5352feb5f12e
15/07/01 13:56:59 INFO HttpServer: Starting HTTP Server
15/07/01 13:56:59 INFO Utils: Successfully started service 'HTTP file server' on port 33599.
15/07/01 13:56:59 INFO SparkEnv: Registering OutputCommitCoordinator
15/07/01 13:56:59 INFO Utils: Successfully started service 'SparkUI' on port 4040.
15/07/01 13:56:59 INFO SparkUI: Started SparkUI at http://[IP ADDRESS]:4040
15/07/01 13:57:00 ERROR SparkContext: Jar not found at file:/opt/Spark/spark-1.4.0/jars/elasticsearch-hadoop-2.1.0.Beta2.jar
15/07/01 13:57:00 INFO SparkContext: Added JAR file:/home/user/Documents/TestApp/target/scala-2.11/sparkesingest_2.11-1.0.0.jar at http://[IP ADDRESS]:33599/jars/sparkesingest_2.11-1.0.0.jar with timestamp 1435784220028
15/07/01 13:57:00 INFO Executor: Starting executor ID driver on host localhost
15/07/01 13:57:00 INFO Utils: Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 44746.
15/07/01 13:57:00 INFO NettyBlockTransferService: Server created on 44746
15/07/01 13:57:00 INFO BlockManagerMaster: Trying to register BlockManager
15/07/01 13:57:00 INFO BlockManagerMasterEndpoint: Registering block manager localhost:44746 with 265.4 MB RAM, BlockManagerId(driver, localhost, 44746)
15/07/01 13:57:00 INFO BlockManagerMaster: Registered BlockManager
15/07/01 13:57:00 INFO MemoryStore: ensureFreeSpace(143840) called with curMem=0, maxMem=278302556
15/07/01 13:57:00 INFO MemoryStore: Block broadcast_0 stored as values in memory (estimated size 140.5 KB, free 265.3 MB)
15/07/01 13:57:00 INFO MemoryStore: ensureFreeSpace(12635) called with curMem=143840, maxMem=278302556
15/07/01 13:57:00 INFO MemoryStore: Block broadcast_0_piece0 stored as bytes in memory (estimated size 12.3 KB, free 265.3 MB)
15/07/01 13:57:00 INFO BlockManagerInfo: Added broadcast_0_piece0 in memory on localhost:44746 (size: 12.3 KB, free: 265.4 MB)
15/07/01 13:57:00 INFO SparkContext: Created broadcast 0 from textFile at Ingest.scala:159
Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/spark/SparkConf
at org.elasticsearch.spark.rdd.CompatUtils.<clinit>(CompatUtils.java:20)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:274)
at org.elasticsearch.hadoop.util.ObjectUtils.loadClass(ObjectUtils.java:71)
at org.elasticsearch.spark.package$.<init>(package.scala:14)
at org.elasticsearch.spark.package$.<clinit>(package.scala)
at build.Ingest$.main(Ingest.scala:176)
at build.Ingest.main(Ingest.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:664)
at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:169)
at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:192)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:111)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Caused by: java.lang.ClassNotFoundException: org.apache.spark.SparkConf
at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
... 17 more
15/07/01 13:57:00 INFO SparkContext: Invoking stop() from shutdown hook
15/07/01 13:57:00 INFO SparkUI: Stopped Spark web UI at http://[IP ADDRESS]:4040
15/07/01 13:57:00 INFO DAGScheduler: Stopping DAGScheduler
15/07/01 13:57:00 INFO MapOutputTrackerMasterEndpoint: MapOutputTrackerMasterEndpoint stopped!
15/07/01 13:57:00 INFO Utils: path = /tmp/spark-c206e297-c2ef-4bbf-9bd2-de642804bdcd/blockmgr-8d273f32-589e-4f55-98a2-cf0322a05d45, already present as root for deletion.
15/07/01 13:57:00 INFO MemoryStore: MemoryStore cleared
15/07/01 13:57:00 INFO BlockManager: BlockManager stopped
15/07/01 13:57:01 INFO BlockManagerMaster: BlockManagerMaster stopped
15/07/01 13:57:01 INFO SparkContext: Successfully stopped SparkContext
15/07/01 13:57:01 INFO OutputCommitCoordinator$OutputCommitCoordinatorEndpoint: OutputCommitCoordinator stopped!
15/07/01 13:57:01 INFO Utils: Shutdown hook called
15/07/01 13:57:01 INFO Utils: Deleting directory /tmp/spark-c206e297-c2ef-4bbf-9bd2-de642804bdcd
Here is the build.sbt file:
scalaVersion := "2.11.6"
name := "SparkEsIngest"
version := "1.0.0"
libraryDependencies ++= Seq(
"org.apache.spark" %% "spark-core" % "1.4.0" % "provided",
"org.apache.spark" %% "spark-streaming" % "1.4.0" % "provided",
"org.apache.spark" %% "spark-sql" % "1.4.0" % "provided",
"org.elasticsearch" % "elasticsearch-hadoop" % "2.1.0.Beta2" exclude("org.spark-project.akka", "akka-remote_2.10") exclude("org.spark-project.akka", "akka-slf4j_2.10") exclude("org.json4s", "json4s-ast_2.10") exclude("org.apache.spark", "spark-catalyst_2.10") exclude("com.twitter", "chill_2.10") exclude("org.apache.spark", "spark-sql_2.10") exclude("org.json4s", "json4s-jackson_2.10") exclude("org.json4s", "json4s-core_2.10") exclude("org.apache.spark", "spark-core_2.10")
)
if ( System.getenv("QUERY_ES_RESOURCE") != null) {
println("[info] Using lib/es-hadoop-build-snapshot/ unmanagedBase dir")
unmanagedBase <<= baseDirectory { base => base / "lib/es-hadoop-build-snapshot" }
} else {
println("[info] Using lib/ unmanagedBase dir")
unmanagedBase <<= baseDirectory { base => base / "lib" }
}
resolvers += "conjars.org" at "http://conjars.org/repo"
resolvers += "clojars" at "https://clojars.org/repo"

Answer 1

Is spark JAR inside the JAR you submitting ? it seems that you tell sbt you're providing the jar but i don't see "unmanagedJars in Compile += file(...)" in you're sbt, if you're counting that the jar are in the machine i would suggest you wouldn't do that since it mat lead to this kind of problems.
Try unrar you're JAR and see if spark JAR are there, if not use sbt-assembly or other tool of you're choice.

Answer 2

Don't use provided scope as it makes the dependencies available only while compiling and not while running the code. Changed code look like..
libraryDependencies ++= Seq(
"org.apache.spark" %% "spark-core" % "1.4.0",
"org.apache.spark" %% "spark-streaming" % "1.4.0",
"org.apache.spark" %% "spark-sql" % "1.4.0",
"org.elasticsearch" % "elasticsearch-hadoop" % "2.1.0.Beta2" exclude("org.spark-project.akka", "akka-remote_2.10") exclude("org.spark-project.akka", "akka-slf4j_2.10") exclude("org.json4s", "json4s-ast_2.10") exclude("org.apache.spark", "spark-catalyst_2.10") exclude("com.twitter", "chill_2.10") exclude("org.apache.spark", "spark-sql_2.10") exclude("org.json4s", "json4s-jackson_2.10") exclude("org.json4s", "json4s-core_2.10") exclude("org.apache.spark", "spark-core_2.10")
)