I'm trying to use the ignite-spark dependency in my sbt project.
Here is the build.sbt file:
name := "App"
version := "1.0"
scalaVersion := "2.10.6"
libraryDependencies += "org.apache.spark" % "spark-core_2.10" % "2.2.0" % "provided"
libraryDependencies += "org.apache.spark" % "spark-streaming_2.10" % "2.2.0" % "provided"
libraryDependencies += "org.apache.spark" % "spark-sql_2.10" % "2.2.0" % "provided"
libraryDependencies += "org.apache.spark" % "spark-mllib_2.10" % "2.2.0" % "provided"
libraryDependencies += "org.xerial.snappy" % "snappy-java" % "1.1.4"
libraryDependencies += "org.apache.spark" % "spark-streaming-kafka-0-8_2.10" % "2.2.0" % "provided"
libraryDependencies += "org.apache.ignite" % "ignite-spark_2.10" % "2.2.0"
libraryDependencies += "org.apache.ignite" % "ignite-spring" % "2.2.0"
But when I try to compile, I get a cross-version suffixes error for the following dependencies:
[error] org.json4s:json4s-ast _2.10, _2.11
[error] org.json4s:json4s-core _2.10, _2.11
java.lang.RuntimeException: Conflicting cross-version suffixes in: org.json4s:json4s-ast, org.json4s:json4s-core
I believe it might be related to this issue https://issues.apache.org/jira/browse/IGNITE-3710
But I thought it was solved already.
Any ideas on how to solve this?
Apparently this is a bug from the ignite-spark_2.10 module.
1) I cloned the ignite project repository: https://github.com/apache/ignite
2) Downgraded it to version 2.2.0
3) Changed the pom.xml from the spark_2.10 module like so:
<dependency>
<groupId>org.json4s</groupId>
<artifactId>json4s-core_2.10</artifactId>
<version>3.5.0</version>
<scope>compile</scope>
</dependency>
4) Compiled and packaged a new Jar.
5) Voilà, it works.
I informed the Ignite community via the mailing list.
This problem was fixed under the following ticket: IGNITE-3084
Try upgrading to Ignite 2.4 or newer. It should resolve the issue.
Related JIRA ticket: IGNITE-6369
Related
I'm new to Scala/Spark, so please be easy on me :)
I'm trying to run an EMR cluster on AWS, running the jar file I packed with sbt package.
When I run the code locally, it is working perfectly fine, but when I'm running it in the AWS EMR cluster, I'm getting an error:
ERROR Client: Application diagnostics message: User class threw exception: java.lang.NoClassDefFoundError: upickle/core/Types$Writer
From what I understand, this error originates in the dependencies of the scala/spark versions.
So I'm using Scala 2.12 with spark 3.0.1, and in AWS I'm using emr-6.2.0.
Here's my build.sbt:
scalaVersion := "2.12.14"
libraryDependencies += "com.amazonaws" % "aws-java-sdk" % "1.11.792"
libraryDependencies += "com.amazonaws" % "aws-java-sdk-core" % "1.11.792"
libraryDependencies += "org.apache.hadoop" % "hadoop-aws" % "3.3.0"
libraryDependencies += "org.apache.hadoop" % "hadoop-common" % "3.3.0"
libraryDependencies += "org.apache.hadoop" % "hadoop-client" % "3.3.0"
libraryDependencies += "org.apache.spark" %% "spark-core" % "3.0.1"
libraryDependencies += "org.apache.spark" %% "spark-sql" % "3.0.1"
libraryDependencies += "com.lihaoyi" %% "upickle" % "1.4.1"
libraryDependencies += "com.lihaoyi" %% "ujson" % "1.4.1"
What am I missing?
Thanks!
If you use sbt package, the generated jar will contain only the code of your project, but not dependencies. You need to use sbt assembly to generate so-called uberjar, that will include dependencies as well.
But in your cases, it's recommended to mark Spark and Hadoop (and maybe AWS) dependencies as Provided - they should be already included into the EMR runtime. Use something like this:
libraryDependencies += "org.apache.spark" %% "spark-core" % "3.0.1" % Provided
My build.sbt looks like this:
import sbt._
name := "spark-jobs"
version := "0.1"
scalaVersion := "2.11.8"
resolvers += "Spark Packages Repo" at "https://dl.bintray.com/spark-packages/maven"
// additional libraries
libraryDependencies ++= Seq(
"org.apache.spark" % "spark-core_2.11" % "2.2.0" % "provided",
"org.apache.spark" % "spark-streaming_2.11" % "2.2.0",
"org.apache.spark" % "spark-sql_2.11" % "2.2.0" % "provided",
"org.apache.spark" % "spark-streaming-kafka-0-10_2.11" % "2.2.0"
)
assemblyMergeStrategy in assembly := {
case PathList("META-INF", xs # _*) => MergeStrategy.discard
case x => MergeStrategy.first
}
This used to work until I decided to see what happens if I add another % "provided" at the end of spark-streaming_2.11. It failed to resolve dependency, I moved on and reverted the change. But, it seems to give me the exception after that as well. Now my build.sbt looks exactly like it used to when everything worked. Still, it gives me this exception :
[error] (*:update) sbt.librarymanagement.ResolveException: unresolved dependency: org.apache.spark#spark-streaming_2.11;2.2.0: org.apache.spark#spark-parent_2.11;2.2.0!spark-parent_2.11.pom(pom.original) origin location must be absolute: file:/home/aswin/.m2/repository/org/apache/spark/spark-parent_2.11/2.2.0/spark-parent_2.11-2.2.0.pom
SBT's behavior is a bit confusing to me. Could someone guide me to as why this could happen? Any good blogs/ resources to understand how exactly SBT works under the hood is also welcome.
Here is my project/assembly.sbt:
addSbtPlugin("com.eed3si9n" % "sbt-assembly" % "0.14.6")
project/build.properties:
sbt.version = 1.0.4
project/plugins.sbt:
resolvers += Resolver.url("artifactory", url("http://scalasbt.artifactoryonline.com/scalasbt/sbt-plugin-releases"))(Resolver.ivyStylePatterns)
resolvers += "Typesafe Repository" at "http://repo.typesafe.com/typesafe/releases/"
Thank you!
If you are in sbt console, just run reload command and try again. After you update your dependencies or sbt plugins, you need to reload the project so that the changes take effect.
By the way, instead of defining the Scala version in your dependencies, you can just use %% operator and it will fetch the appropriate dependency according to your defined scala version.
// additional libraries
libraryDependencies ++= Seq(
"org.apache.spark" %% "spark-core" % "2.2.0" % "provided",
"org.apache.spark" %% "spark-streaming" % "2.2.0",
"org.apache.spark" %% "spark-sql" % "2.2.0" % "provided",
"org.apache.spark" %% "spark-streaming-kafka-0-10" % "2.2.0"
)
I add to my built.sbt: libraryDependencies += "org.apache.httpcomponents" % "httpclient" % "4.5.3" and i refrech sbt, but the library is not imported. anyone has any idea about this issue?
I have the following build.sbt file:
name := "akkaHttp"
version := "1.0"
scalaVersion := "2.12.0"
resolvers ++= Seq("Typesafe Repository" at "http://repo.typesafe.com/typesafe/releases/",
Resolver.bintrayRepo("hseeberger", "maven"))
libraryDependencies ++= {
val AkkaVersion = "2.3.9"
val AkkaHttpVersion = "2.0.1"
val Json4sVersion = "3.2.11"
Seq(
"com.typesafe.akka" %% "akka-slf4j" % AkkaVersion,
"com.typesafe.akka" %% "akka-http-experimental" % AkkaHttpVersion,
"ch.qos.logback" % "logback-classic" % "1.1.2",
"org.json4s" %% "json4s-native" % Json4sVersion,
"org.json4s" %% "json4s-ext" % Json4sVersion,
"de.heikoseeberger" %% "akka-http-json4s" % "1.4.2"
)
}
In it, these dependencies are not met.
Error:Unresolved dependencies:
com.typesafe.akka#akka-slf4j_2.12;2.3.9: not found
com.typesafe.akka#akka-http-experimental_2.12;2.0.1: not found
org.json4s#json4s-native_2.12;3.2.11: not found
org.json4s#json4s-ext_2.12;3.2.11: not found
de.heikoseeberger#akka-http-json4s_2.12;1.4.2: not found
So what should I add to it such that the imports work?
Notice the _2.12 appended to the artifacts? Based on your scalaVersion, SBT tried to download the dependencies built against Scala 2.12.x, but couldn't find any. Try using Scala version 2.11.8.
Different versions of Scala can be binary incompatible, so libraries are cross built against those different versions and published to the repositories. Looks like it hasn't yet happened for those libraries above.
Note that Scala 2.12 is indeed not binary compatible with Scala 2.11.
I am new to JVM development (I am using Scala and SBT) and am having trouble resolving dependencies. Yesterday, I had trouble resolving the org.restlet.2.1.1 dependency and today, I am having trouble with resolving the following:
[error] (*:update) sbt.ResolveException: unresolved dependency: com.mongodb.casbah#casbah_2.9.2;2.1.5-1: not found
[error] unresolved dependency: org.scalatra#scalatra_2.9.2;2.3.0: not found
[error] unresolved dependency: org.scalatra#scalatra-akka2_2.9.2;2.3.0: not found
[error] unresolved dependency: org.scalatra#scalatra-specs2_2.9.2;2.3.0: not found
I am using a giter8 scalatra-mongodb project template from github: click me. Since the project is a little old, it stands to reason that I am trying to obtain outdated versions that no longer exist or are compatible. What does one do in this situation? I tried fiddling with the version numbers in my build.sbt file, but this did not work (and appears to be worse!).
The following is the contents of my build.sbt file:
scalaVersion := "2.9.2"
mainClass := Some("JettyLauncher")
seq(webSettings :_*)
port in container.Configuration := 8080
seq(assemblySettings: _*)
libraryDependencies ++= Seq(
"com.mongodb.casbah" %% "casbah" % "2.8.1-1",
"org.scalatra" %% "scalatra" % "2.2.0",
"org.scalatra" %% "scalatra-akka2" % "2.2.0",
"org.scalatra" %% "scalatra-specs2" % "2.2.0" % "test",
"org.mortbay.jetty" % "servlet-api" % "3.0.20100224" % "provided",
"org.eclipse.jetty" % "jetty-server" % "8.0.0.M3" % "container, compile",
"org.eclipse.jetty" % "jetty-util" % "8.0.0.M3" % "container, compile",
"org.eclipse.jetty" % "jetty-webapp" % "8.0.0.M3" % "container, compile"
)
resolvers ++= Seq(
"Sonatype OSS" at "http://oss.sonatype.org/content/repositories/releases/",
"Sonatype OSS Snapshots" at "http://oss.sonatype.org/content/repositories/snapshots/",
"Typesafe Repository" at "http://repo.typesafe.com/typesafe/releases/",
"Akka Repo" at "http://akka.io/repository/",
"Web plugin repo" at "http://siasia.github.com/maven2"
)
The following is my plugins.sbt file:
addSbtPlugin("com.earldouglas" %% "xsbt-web-plugin" % "0.9.0")
addSbtPlugin("com.eed3si9n" % "sbt-assembly" % "0.7.2")
Note that when I first generated the template, I was receiving missing dependencies for this first plugin. Fortunately, the github page for this plugin gave updated instructions and I am able to get past this dependency.
Anyway, what are the versions of these dependencies that I need to get everything working? In general what is a strategy for resolving these dependencies (right now I have no idea what to do (other than visit the github pages and fiddle with version numbers)?
Thanks for all the help!
I have at least gotten sbt to resolve my dependencies for whatever that is worth (it has classpath issues now...). Anyway, the following is my new and improved build.sbt file:
scalaVersion := "2.10.4"
mainClass := Some("JettyLauncher")
seq(webSettings :_*)
port in container.Configuration := 8080
seq(assemblySettings: _*)
libraryDependencies += "org.mongodb" %% "casbah-core" % "2.7.3"
libraryDependencies += "org.scalatra" %% "scalatra" % "2.2.0-RC3" cross CrossVersion.binary
libraryDependencies += "org.scalatra" %% "scalatra-akka" % "2.2.0-RC3"
libraryDependencies += "org.scalatra" %% "scalatra-specs2" % "2.2.0" % "test"
libraryDependencies += "org.mortbay.jetty" % "servlet-api" % "3.0.20100224" % "provided"
libraryDependencies += "org.eclipse.jetty" % "jetty-server" % "9.0.0.M5" % "container"
libraryDependencies += "org.eclipse.jetty" % "jetty-util" % "9.0.0.M5" % "container"
libraryDependencies += "org.eclipse.jetty" % "jetty-webapp" % "9.0.0.M5" % "container"
resolvers ++= Seq(
"Sonatype releases" at "http://oss.sonatype.org/content/repositories/releases/",
"Sonatype snapshots" at "http://oss.sonatype.org/content/repositories/snapshots/",
"Typesafe Repository" at "http://repo.typesafe.com/typesafe/releases/",
"Akka Repo" at "http://akka.io/repository/",
"Web plugin repo" at "http://siasia.github.com/maven2"
)
All I really did was spend a few hours googling and searching mvn repositories that were compatible with the scala version I am using (2.10.4). I learned that %% will append the scala-version to the dependency name (seems like a nice convention since Scala is always evolving). Once, I got a few dependencies resolved, the rest caved!