Exception while trying to run Spark app with JSON parsing - scala

Question

I have a simple Spark application with Scala and SBT. First I tried to do the following:
run sbt clean package
run spark-submit --class Main ./target/scala-2.11/sparktest_2.11-1.0.jar
but it fails with the following exception:
Exception in thread "main" java.lang.NoClassDefFoundError: com/fasterxml/jackson/module/scala/DefaultScalaModule$
Then I tried the assembly plugin for SBT, but I got the following exception instead:
java.lang.NoSuchMethodError: com.fasterxml.jackson.databind.introspect.POJOPropertyBuilder.addField(Lcom/fasterxml/jackson/databind/introspect/AnnotatedField;Lcom/fasterxml/jackson/databind/PropertyName;ZZZ)V
As I can see, everything looks related to the Jackson lib and to the Scala support. Maybe it's some issue related to versions of the libraries?
My build.sbt looks like this:
name := "SparkTest"
version := "1.0"
scalaVersion := "2.11.4"
scalacOptions := Seq("-unchecked", "-deprecation", "-encoding", "utf8", "-feature")
libraryDependencies ++= {
Seq(
"org.apache.spark" %% "spark-core" % "1.2.1" % "provided",
"com.fasterxml.jackson.core" % "jackson-core" % "2.4.1",
"com.fasterxml.jackson.core" % "jackson-databind" % "2.4.1",
"com.fasterxml.jackson.module" %% "jackson-module-scala" % "2.4.1"
)
}
And my application code is simply this:
import com.fasterxml.jackson.databind.{DeserializationFeature, ObjectMapper}
import com.fasterxml.jackson.module.scala.DefaultScalaModule
import org.apache.spark.{SparkConf, SparkContext}
trait JsonUtil {
val mapper = new ObjectMapper()
mapper.registerModule(DefaultScalaModule)
mapper.configure(DeserializationFeature.FAIL_ON_UNKNOWN_PROPERTIES, false)
}
case class Person(name: String)
object Main extends JsonUtil {
def main(args: Array[String]) {
val conf = new SparkConf().setAppName("Spark Test App")
val sc = new SparkContext(conf)
val inputFile = "/home/user/data/person.json"
val input = sc.textFile(inputFile)
val persons = input.flatMap { line ⇒ {
try {
println(s" [DEBUG] trying to parse '$line'")
Some(mapper.readValue(line, classOf[Person]))
} catch {
case e : Exception ⇒
println(s" [EXCEPTION] ${e.getMessage}")
None
}
}}
println("PERSON LIST:")
for (p ← persons) {
println(s" $p")
}
println("END")
}
}
EDIT: the problem seems to be related to the Spark application. If I run simple application just for testing JSON unmarshalling everything goes OK. But if I try to do the same from the Spark application, then the problem appears as described above. Any ideas?