I am trying to work on Twitter stream sample project. i am facing problem while defining sbt.
my build.sbt
name := "Tutorial"
version := "0.1.0"
scalaVersion := "2.11.8"
retrieveManaged := true
libraryDependencies ++= Seq(
"org.apache.spark" % "spark-core" % "2.11.0",
"org.apache.spark" % "spark-streaming" % "1.1.0",
"org.apache.spark" % "spark-streaming-twitter" % "1.1.0"
)
Error Log:
[warn] Note: Some unresolved dependencies have extra attributes. Check that these dependencies exist with the requested attributes.
[warn] com.eed3si9n:sbt-assembly:0.9.2 (sbtVersion=0.11.3, scalaVersion=2.11.8)
[warn] com.typesafe.sbteclipse:sbteclipse-plugin:2.2.0 (sbtVersion=0.11.3, scalaVersion=2.11.8)
[warn] com.github.mpeltonen:sbt-idea:1.5.1 (sbtVersion=0.11.3, scalaVersion=2.11.8)
[warn]
[error] {file:/home/muralee1857/scala/workspace/Tutorial/}default-109f4d/*:update: sbt.ResolveException: unresolved dependency: org.apache.spark#spark-core_2.11.8;1.5.1: not found
[error] unresolved dependency: com.eed3si9n#sbt-assembly;0.9.2: not found
[error] unresolved dependency: com.typesafe.sbteclipse#sbteclipse-plugin;2.2.0: not found
[error] unresolved dependency: com.github.mpeltonen#sbt-idea;1.5.1: not found
I would suggest you to define explicitly the packaged versions to the dependency as
libraryDependencies ++= Seq(
"org.apache.spark" % "spark-core_2.10" % "1.1.0",
"org.apache.spark" % "spark-streaming_2.10" % "1.1.0" % "provided",
"org.apache.spark" % "spark-streaming-twitter_2.10" % "1.1.0"
)
You can use %% without defining the packaged version but that will try to download the package with scala version in your system. And sometimes sbt will not find the scala version packaged packages which will create dependency issues.
This should work. note that I am using %% method instead of % here so that right version of the spark library (scala version 2.11) is selected here. ensure you apply the same %% function to others plugins like sbt-assembly, sbt-assembly etc.
libraryDependencies ++= Seq(
"org.apache.spark" %% "spark-core" % "2.11.0",
"org.apache.spark" %% "spark-streaming" % "1.1.0",
"org.apache.spark" %% "spark-streaming-twitter" % "1.1.0"
)
Related
I'm trying to use the ignite-spark dependency in my sbt project.
Here is the build.sbt file:
name := "App"
version := "1.0"
scalaVersion := "2.10.6"
libraryDependencies += "org.apache.spark" % "spark-core_2.10" % "2.2.0" % "provided"
libraryDependencies += "org.apache.spark" % "spark-streaming_2.10" % "2.2.0" % "provided"
libraryDependencies += "org.apache.spark" % "spark-sql_2.10" % "2.2.0" % "provided"
libraryDependencies += "org.apache.spark" % "spark-mllib_2.10" % "2.2.0" % "provided"
libraryDependencies += "org.xerial.snappy" % "snappy-java" % "1.1.4"
libraryDependencies += "org.apache.spark" % "spark-streaming-kafka-0-8_2.10" % "2.2.0" % "provided"
libraryDependencies += "org.apache.ignite" % "ignite-spark_2.10" % "2.2.0"
libraryDependencies += "org.apache.ignite" % "ignite-spring" % "2.2.0"
But when I try to compile, I get a cross-version suffixes error for the following dependencies:
[error] org.json4s:json4s-ast _2.10, _2.11
[error] org.json4s:json4s-core _2.10, _2.11
java.lang.RuntimeException: Conflicting cross-version suffixes in: org.json4s:json4s-ast, org.json4s:json4s-core
I believe it might be related to this issue https://issues.apache.org/jira/browse/IGNITE-3710
But I thought it was solved already.
Any ideas on how to solve this?
Apparently this is a bug from the ignite-spark_2.10 module.
1) I cloned the ignite project repository: https://github.com/apache/ignite
2) Downgraded it to version 2.2.0
3) Changed the pom.xml from the spark_2.10 module like so:
<dependency>
<groupId>org.json4s</groupId>
<artifactId>json4s-core_2.10</artifactId>
<version>3.5.0</version>
<scope>compile</scope>
</dependency>
4) Compiled and packaged a new Jar.
5) Voilà, it works.
I informed the Ignite community via the mailing list.
This problem was fixed under the following ticket: IGNITE-3084
Try upgrading to Ignite 2.4 or newer. It should resolve the issue.
Related JIRA ticket: IGNITE-6369
./build/sbt clean package
gives the below error:
Resolving org.fusesource.jansi#jansi;1.4 ...
[warn] ::::::::::::::::::::::::::::::::::::::::::::::
[warn] :: UNRESOLVED DEPENDENCIES ::
[warn] ::::::::::::::::::::::::::::::::::::::::::::::
[warn] :: ml.dmlc.xgboost#xgboost4j_2.10;0.7: not found
[warn] :: ml.dmlc.xgboost#xgboost4j-spark_2.10;0.7: not found
[warn] ::::::::::::::::::::::::::::::::::::::::::::::
build.sbt looks like below:
name := "xgboostproj"
version := "1.0"
libraryDependencies += "org.apache.spark" %% "spark-core" % "1.6.1"
libraryDependencies += "org.apache.spark" %% "spark-mllib" % "1.6.1"
resolvers += Resolver.mavenLocal
libraryDependencies += "ml.dmlc.xgboost" %% "xgboost4j" % "0.7"
libraryDependencies += "ml.dmlc.xgboost" %% "xgboost4j-spark" % "0.7"
Thanks in Advance!
xgboost jars must be built locally and published to your local maven repository for your set up to work. The instructions for this is published here
Also additionally the dependencies should be like below (the groupId is ml.dmlc)
libraryDependencies += "ml.dmlc" %% "xgboost4j" % "0.7"
libraryDependencies += "ml.dmlc" %% "xgboost4j-spark" % "0.7"
I think the dependencies should be changed to:
libraryDependencies += "ml.dmlc" % "xgboost4j" % "0.7"
libraryDependencies += "ml.dmlc" % "xgboost4j-spark" % "0.7"
Also make sure your sbt is looking at the local maven repo. These were the two things that I needed to fix to make it work. Also make sure the jar files exist in the m2 repo. This is assuming you used mvn install to build the jvm packages.
I'm using spark-cassandra, with the following dependencies:
scalaVersion := "2.11.8"
resolvers += "Spark Packages Repo" at "https://dl.bintray.com/spark-packages/maven"
libraryDependencies ++= Seq(
"org.apache.spark" %% "spark-core" % "2.0.0",
"org.apache.spark" %% "spark-mllib" % "2.0.0",
"datastax" % "spark-cassandra-connector" % "2.0.0-M3"
)
SBT fails to resolve dependencies. What should I change to make it work?
Error:Error while importing SBT project:<br/>...<br/><pre>[info] Resolving org.scala-sbt#run;0.13.8 ...
...
[error] (*:update) sbt.ResolveException: unresolved dependency: datastax#spark-cassandra-connector;2.0.0-M3: not found
[error] (*:ssExtractDependencies) sbt.ResolveException: unresolved dependency: datastax#spark-cassandra-connector;2.0.0-M3: not found
use libraryDependencies += "com.datastax.spark" % "spark-cassandra-connector_2.11" % "2.0.0-M3", as you are using scala 2.11, this should resolve it
maven central repo
I am getting following error when i am trying to import dependencies :
[error] (*:update) sbt.ResolveException: unresolved dependency: com.typesafe.akka#akka-actor_2.11;2.4-SNAPSHOT: not found
sbt.ResolveException: unresolved dependency: com.typesafe.akka#akka-actor_2.11;2.4-SNAPSHOT: not found
at sbt.IvyActions$.sbt$IvyActions$$resolve(IvyActions.scala:291)
at sbt.IvyActions$$anonfun$updateEither$1.apply(IvyActions.scala:188)
at sbt.IvyActions$$anonfun$updateEither$1.apply(IvyActions.scala:165)
at sbt.IvySbt$Module$$anonfun$withModule$1.apply(Ivy.scala:155)
This is how my build.sbt looks like
name := "SBT"
version := "1.0"
scalaVersion := "2.11.7"
resolvers += "Akka Snapshot Repository" at
"http://repo.akka.io/snapshots/"
libraryDependencies += "com.typesafe.akka" %% "akka-actor" %
"2.4-SNAPSHOT"
Please help me where am i going wrong ?
Thank you in Advance
If you want to be on the edge, why don't you use 2.4.1?
In my case it worked without resolvers.
libraryDependencies += "com.typesafe.akka" %% "akka-actor" % "2.4.1"
Or you can use 2.4.0
libraryDependencies += "com.typesafe.akka" %% "akka-actor" % "2.4.0"
I am fairly new to Scala and SBT. I would like to implement a REST server using Jersey JAX-RS and Scala, but have faced the first hurdle trying to configure SBT. I was planning to use the Grizzly HTTP Server. The build.sbt is as follows:
name := "scala-jersey-server"
version := "1.0"
scalaVersion := "2.10.3"
libraryDependencies += "org.glassfish.jersey.containers" % "jersey-container-grizzly2-http" % "2.5.1"
Running sbt update fails with the following messages
[warn] ::::::::::::::::::::::::::::::::::::::::::::::
[warn] :: UNRESOLVED DEPENDENCIES ::
[warn] ::::::::::::::::::::::::::::::::::::::::::::::
[warn] :: org.glassfish.hk2#hk2-utils;2.5.1: not found
[warn] :: org.glassfish.hk2#hk2-locator;2.5.1: not found
[warn] :: javax.validation#validation-api;${javax.validation.version}: not found
[warn] ::::::::::::::::::::::::::::::::::::::::::::::
Looking at the pom.xml for hk2-api, it shows dependency on hk2-utils using the ${project.version} variable, but I'd have expected version 2.2.0-b21 of hk2-utils to be requested by SBT not 2.5.1 (which is JAX-RS version).
<dependency>
<groupId>org.glassfish.hk2</groupId>
<artifactId>hk2-utils</artifactId>
<version>${project.version}</version>
</dependency>
Is this a problem with the configuration of hk2-api's pom.xml, is SBT using an incorrect version or something else?
I think there is an error in the publication chain of your dependency for the 2.5.1 version.
There are at least three possibilities:
1. Use another version
//http://search.maven.org/#search%7Cgav%7C1%7Cg%3A%22org.glassfish.jersey.containers%22%20AND%20a%3A%22jersey-container-grizzly2-http%22
libraryDependencies += "org.glassfish.jersey.containers" % "jersey-container-grizzly2-http" % "2.5"
2. Use another library
3. Exclude the broken dependencies and set working versions
Use http://search.maven.org/ to find existing versions.
libraryDependencies += "org.glassfish.jersey.containers" % "jersey-container-grizzly2-http" % "2.5.1" exclude("org.glassfish.hk2", "hk2-utils") exclude("org.glassfish.hk2", "hk2-locator") exclude("javax.validation", "validation-api")
//http://search.maven.org/#browse%7C703203458
libraryDependencies += "org.glassfish.hk2" % "hk2-utils" % "2.2.0-b27"
//http://search.maven.org/#search%7Cga%7C1%7Ca%3A%22hk2-locator%22
libraryDependencies += "org.glassfish.hk2" % "hk2-locator" % "2.2.0-b27"
//http://search.maven.org/#artifactdetails%7Cjavax.validation%7Cvalidation-api%7C1.1.0.Final%7Cjar
libraryDependencies += "javax.validation" % "validation-api" % "1.1.0.Final"
**I tried the above .sbt file but it was not excluding the jars and was still giving the same ERROR.
So modified the build file. below sample it could create a build successfully.**
name := "KafkaSparkCouchReadWrite"
organization := "my.clairvoyant"
version := "1.0.0-SNAPSHOT"
scalaVersion := "2.11.11"
libraryDependencies ++= Seq(
"org.apache.spark" %% "spark-core" % "2.1.0",
"org.apache.spark" %% "spark-streaming" % "2.1.0",
"org.apache.spark" %% "spark-sql" % "2.1.0",
"com.couchbase.client" %% "spark-connector" % "2.1.0",
"org.glassfish.hk2" % "hk2-utils" % "2.2.0-b27",
"org.glassfish.hk2" % "hk2-locator" % "2.2.0-b27",
"javax.validation" % "validation-api" % "1.1.0.Final"
).map(_.excludeAll(ExclusionRule("org.glassfish.hk2"),ExclusionRule("javax.validation")))