I'm using spark-cassandra, with the following dependencies:
scalaVersion := "2.11.8"
resolvers += "Spark Packages Repo" at "https://dl.bintray.com/spark-packages/maven"
libraryDependencies ++= Seq(
"org.apache.spark" %% "spark-core" % "2.0.0",
"org.apache.spark" %% "spark-mllib" % "2.0.0",
"datastax" % "spark-cassandra-connector" % "2.0.0-M3"
)
SBT fails to resolve dependencies. What should I change to make it work?
Error:Error while importing SBT project:<br/>...<br/><pre>[info] Resolving org.scala-sbt#run;0.13.8 ...
...
[error] (*:update) sbt.ResolveException: unresolved dependency: datastax#spark-cassandra-connector;2.0.0-M3: not found
[error] (*:ssExtractDependencies) sbt.ResolveException: unresolved dependency: datastax#spark-cassandra-connector;2.0.0-M3: not found
use libraryDependencies += "com.datastax.spark" % "spark-cassandra-connector_2.11" % "2.0.0-M3", as you are using scala 2.11, this should resolve it
maven central repo
Related
I am trying to work on Twitter stream sample project. i am facing problem while defining sbt.
my build.sbt
name := "Tutorial"
version := "0.1.0"
scalaVersion := "2.11.8"
retrieveManaged := true
libraryDependencies ++= Seq(
"org.apache.spark" % "spark-core" % "2.11.0",
"org.apache.spark" % "spark-streaming" % "1.1.0",
"org.apache.spark" % "spark-streaming-twitter" % "1.1.0"
)
Error Log:
[warn] Note: Some unresolved dependencies have extra attributes. Check that these dependencies exist with the requested attributes.
[warn] com.eed3si9n:sbt-assembly:0.9.2 (sbtVersion=0.11.3, scalaVersion=2.11.8)
[warn] com.typesafe.sbteclipse:sbteclipse-plugin:2.2.0 (sbtVersion=0.11.3, scalaVersion=2.11.8)
[warn] com.github.mpeltonen:sbt-idea:1.5.1 (sbtVersion=0.11.3, scalaVersion=2.11.8)
[warn]
[error] {file:/home/muralee1857/scala/workspace/Tutorial/}default-109f4d/*:update: sbt.ResolveException: unresolved dependency: org.apache.spark#spark-core_2.11.8;1.5.1: not found
[error] unresolved dependency: com.eed3si9n#sbt-assembly;0.9.2: not found
[error] unresolved dependency: com.typesafe.sbteclipse#sbteclipse-plugin;2.2.0: not found
[error] unresolved dependency: com.github.mpeltonen#sbt-idea;1.5.1: not found
I would suggest you to define explicitly the packaged versions to the dependency as
libraryDependencies ++= Seq(
"org.apache.spark" % "spark-core_2.10" % "1.1.0",
"org.apache.spark" % "spark-streaming_2.10" % "1.1.0" % "provided",
"org.apache.spark" % "spark-streaming-twitter_2.10" % "1.1.0"
)
You can use %% without defining the packaged version but that will try to download the package with scala version in your system. And sometimes sbt will not find the scala version packaged packages which will create dependency issues.
This should work. note that I am using %% method instead of % here so that right version of the spark library (scala version 2.11) is selected here. ensure you apply the same %% function to others plugins like sbt-assembly, sbt-assembly etc.
libraryDependencies ++= Seq(
"org.apache.spark" %% "spark-core" % "2.11.0",
"org.apache.spark" %% "spark-streaming" % "1.1.0",
"org.apache.spark" %% "spark-streaming-twitter" % "1.1.0"
)
./build/sbt clean package
gives the below error:
Resolving org.fusesource.jansi#jansi;1.4 ...
[warn] ::::::::::::::::::::::::::::::::::::::::::::::
[warn] :: UNRESOLVED DEPENDENCIES ::
[warn] ::::::::::::::::::::::::::::::::::::::::::::::
[warn] :: ml.dmlc.xgboost#xgboost4j_2.10;0.7: not found
[warn] :: ml.dmlc.xgboost#xgboost4j-spark_2.10;0.7: not found
[warn] ::::::::::::::::::::::::::::::::::::::::::::::
build.sbt looks like below:
name := "xgboostproj"
version := "1.0"
libraryDependencies += "org.apache.spark" %% "spark-core" % "1.6.1"
libraryDependencies += "org.apache.spark" %% "spark-mllib" % "1.6.1"
resolvers += Resolver.mavenLocal
libraryDependencies += "ml.dmlc.xgboost" %% "xgboost4j" % "0.7"
libraryDependencies += "ml.dmlc.xgboost" %% "xgboost4j-spark" % "0.7"
Thanks in Advance!
xgboost jars must be built locally and published to your local maven repository for your set up to work. The instructions for this is published here
Also additionally the dependencies should be like below (the groupId is ml.dmlc)
libraryDependencies += "ml.dmlc" %% "xgboost4j" % "0.7"
libraryDependencies += "ml.dmlc" %% "xgboost4j-spark" % "0.7"
I think the dependencies should be changed to:
libraryDependencies += "ml.dmlc" % "xgboost4j" % "0.7"
libraryDependencies += "ml.dmlc" % "xgboost4j-spark" % "0.7"
Also make sure your sbt is looking at the local maven repo. These were the two things that I needed to fix to make it work. Also make sure the jar files exist in the m2 repo. This is assuming you used mvn install to build the jvm packages.
I am trying to upgrade my Play application from Scala 2.10 to 2.11 and I get conflicting cross-version suffixes:
[info] Done updating.
[error] Modules were resolved with conflicting cross-version suffixes in {file:/E:/Work/MyProject/development/}root:
[error] com.typesafe.akka:akka-actor _2.11, _2.10
[error] com.typesafe.akka:akka-slf4j _2.11, _2.10
[error] org.scala-stm:scala-stm _2.11, _2.10.0
My build.sbt:
name := """My Project"""
version := "1.0-SNAPSHOT"
lazy val root = (project in file(".")).enablePlugins(PlayScala)
resolvers += "Local Maven Repository" at Path.userHome.asFile.toURI.toURL + ".m2/repository"
resolvers += "scala-sbt.org" at "http://repo.scala-sbt.org/scalasbt/repo"
resolvers += "scala-sbt.org 2" at "http://repo.scala-sbt.org/scalasbt/libs-releases"
resolvers += "scala-sbt.org 3" at "http://repo.scala-sbt.org/scalasbt/plugins-releases"
scalaVersion := "2.11.8"
EclipseKeys.withSource := true
libraryDependencies ++= Seq(
jdbc,
anorm,
cache,
ws,
"com.typesafe.akka" %% "akka-contrib" % "2.3.4",
"com.typesafe.akka" %% "akka-actor" % "2.3.4",
"com.typesafe.akka" %% "akka-slf4j" % "2.3.4",
"com.typesafe.akka" %% "akka-testkit" % "2.3.4" % "test",
"io.argonaut" %% "argonaut" % "6.1",
"org.scalaj" %% "scalaj-http" % "2.3.0",
"my.other.project" % "java-artifact" % "2.2.5")
In the log from sbt compile all of a sudden play_2.10 is resolved:
...
[info] Resolving org.apache.httpcomponents#httpmime;4.0.1 ...
[info] Resolving org.apache.james#apache-mime4j;0.6 ...
[info] Resolving play#play_2.10;2.1.5 ...
[info] Resolving play#sbt-link;2.1.5 ...
[info] Resolving play#play-exceptions;2.1.5 ...
[info] Resolving play#templates_2.10;2.1.5 ...
...
I am using sbt 0.13.5.
I have had this problem before and solving it was a tiresome process.
Is there any way to know which of my dependencies includes play_2.10?
Can I get a dependency tree or something similar from sbt?
There's no native way in sbt, but I had a good experience with third parties, particularly https://github.com/jrudolph/sbt-dependency-graph.
I am getting following error when i am trying to import dependencies :
[error] (*:update) sbt.ResolveException: unresolved dependency: com.typesafe.akka#akka-actor_2.11;2.4-SNAPSHOT: not found
sbt.ResolveException: unresolved dependency: com.typesafe.akka#akka-actor_2.11;2.4-SNAPSHOT: not found
at sbt.IvyActions$.sbt$IvyActions$$resolve(IvyActions.scala:291)
at sbt.IvyActions$$anonfun$updateEither$1.apply(IvyActions.scala:188)
at sbt.IvyActions$$anonfun$updateEither$1.apply(IvyActions.scala:165)
at sbt.IvySbt$Module$$anonfun$withModule$1.apply(Ivy.scala:155)
This is how my build.sbt looks like
name := "SBT"
version := "1.0"
scalaVersion := "2.11.7"
resolvers += "Akka Snapshot Repository" at
"http://repo.akka.io/snapshots/"
libraryDependencies += "com.typesafe.akka" %% "akka-actor" %
"2.4-SNAPSHOT"
Please help me where am i going wrong ?
Thank you in Advance
If you want to be on the edge, why don't you use 2.4.1?
In my case it worked without resolvers.
libraryDependencies += "com.typesafe.akka" %% "akka-actor" % "2.4.1"
Or you can use 2.4.0
libraryDependencies += "com.typesafe.akka" %% "akka-actor" % "2.4.0"
I m trying to run hello slick example from the typesafe activator I create a project in sbt when I type run it gives me following error
warn] ::::::::::::::::::::::::::::::::::::::::::::::
[warn] :: UNRESOLVED DEPENDENCIES ::
[warn] ::::::::::::::::::::::::::::::::::::::::::::::
[warn] :: com.typesafe.slick#slick_2.11;2.0.2: not found
[warn] ::::::::::::::::::::::::::::::::::::::::::::::
[trace] Stack trace suppressed: run last *:update for the full output.
[error] (*:update) sbt.ResolveException: unresolved dependency: com.typesafe.slick#slick_2.11;2.0.2: not found
[error] Total time: 1 s, completed Aug 25, 2014 12:20:42 AM
here is my build.sbt file
name :="hellos"
version :="1.0"
scalaVersion := "2.11.1"
mainClass in Compile := Some("HelloSlick")
libraryDependencies ++= List(
"com.typesafe.slick" %% "slick" % "2.0.2",
"org.slf4j" % "slf4j-nop" % "1.6.4",
"com.h2database" % "h2" % "1.3.170"
)
I m using ubuntu 12.04 and on slick website there is no jar file is present to download so like its not in my system path variable like java and akka is
You need the resolver to the typesafe repository:
libraryDependencies ++= List(
"com.typesafe.slick" %% "slick" % "2.0.2",
"org.slf4j" % "slf4j-nop" % "1.6.4",
"com.h2database" % "h2" % "1.3.170"
)
resolvers += "typesafe" at "http://repo.typesafe.com/typesafe/releases/"
Edit:
It seems that for scala 2.11 you need to specify the slick version to be 2.1:
"com.typesafe.slick" %% "slick" % "2.1.0"