How to troubleshoot SBT's library dependency warnings? - scala

Question

I'm trying to build a "hello world"-esque app that uses Spark streaming to stream data from a Kafka broker (this works), filters/processes this data, and pushes it to a (local) web browser using the Scalatra web framework and its supported web sockets functionality from Atmosphere. The Kafka/Spark chunk works independently, and the Scalatra/Atmosphere chunk also works independently. It's when I try to bring the two halves together that I run into issues with library dependencies.
The real question: how do I go about selecting library versions for which Spark will play nice with Scalatra?
A bare bones Scalatra/Atmosphere app works fine as follows:
organization := "com.example"
name := "example app"
version := "0.1.0"
scalaVersion := "2.12.2"
val ScalatraVersion = "2.5.+"
libraryDependencies ++= Seq(
"org.json4s" %% "json4s-jackson" % "3.5.2",
"org.scalatra" %% "scalatra" % ScalatraVersion,
"org.scalatra" %% "scalatra-scalate" % ScalatraVersion,
"org.scalatra" %% "scalatra-specs2" % ScalatraVersion % "test",
"org.scalatra" %% "scalatra-atmosphere" % ScalatraVersion,
"org.eclipse.jetty" % "jetty-webapp" % "9.4.6.v20170531" % "provided",
"javax.servlet" % "javax.servlet-api" % "3.1.0" % "provided"
)
enablePlugins(JettyPlugin)
But if I add new dependencies for Spark and Spark streaming, and knock the Scala version down to 2.11 (required for Spark-Kafka streaming):
organization := "com.example"
name := "example app"
version := "0.1.0"
scalaVersion := "2.11.8"
val ScalatraVersion = "2.5.+"
val SparkVersion = "2.2.0"
libraryDependencies ++= Seq(
"org.json4s" %% "json4s-jackson" % "3.5.2",
"org.scalatra" %% "scalatra" % ScalatraVersion,
"org.scalatra" %% "scalatra-scalate" % ScalatraVersion,
"org.scalatra" %% "scalatra-specs2" % ScalatraVersion % "test",
"org.scalatra" %% "scalatra-atmosphere" % ScalatraVersion,
"org.eclipse.jetty" % "jetty-webapp" % "9.4.6.v20170531" % "provided",
"javax.servlet" % "javax.servlet-api" % "3.1.0" % "provided"
)
libraryDependencies ++= Seq(
"org.apache.spark" %% "spark-core" % SparkVersion,
"org.apache.spark" %% "spark-streaming" % SparkVersion,
"org.apache.spark" %% "spark-streaming-kafka-0-8" % SparkVersion
)
enablePlugins(JettyPlugin)
The code compiles, but I get SBT's eviction warning:
[warn] There may be incompatibilities among your library dependencies.
[warn] Here are some of the libraries that were evicted:
[warn] * org.json4s:json4s-jackson_2.11:3.2.11 -> 3.5.3
[warn] Run 'evicted' to see detailed eviction warnings
Then finally, when Jetty tries to run the web server, it fails with this error:
WARN:oejuc.AbstractLifeCycle:main: FAILED org.eclipse.jetty.annotations.ServletContainerInitializersStarter#53fb3dab: java.lang.NoClassDefFoundError: com/sun/jersey/spi/inject/InjectableProvider
java.lang.NoClassDefFoundError: com/sun/jersey/spi/inject/InjectableProvider
How do I get to the bottom of this? I'm new to the Scala world, and the intricacies of dependencies are blowing my mind.

Answer 1

One way to remove the eviction warning is to add the library dependency with the required version using dependencyOverrides
try to add the following in your SBT file and re-build the application
dependencyOverrides += "org.json4s" % "json4s-jackson_2.11" % "3.5.3"
Check SBT documentation here