I wrote my build.sbt like this:
name := """scala-hbase"""
version := "1.0"
scalaVersion := "2.11.2"
//scalaVersion := "2.10.4"
/* HBase dependencies */
resolvers ++= Seq(
"Apache Repo" at "https://repository.apache.org/content/repositories/releases",
"Thrift-Repo" at "http://people.apache.org/~rawson/repo",
"ClouderaRepo" at "https://repository.cloudera.com/content/repositories/releases",
"ClouderaRcs" at "https://repository.cloudera.com/artifactory/cdh-releases-rcs",
"Twitter Maven" at "http://maven.twttr.com",
"MVN Repo" at "http://mvnrepository.com/artifact",
"releases" at "http://scala-tools.org/repo-releases"
)
// Hbase 0.94 / Hadoop 2.5.2
libraryDependencies ++= Seq(
//"org.apache.zookeeper" % "zookeeper" % "3.4.6",
//"org.specs2" %% "specs2" % "2.4.5" % "test",
"org.apache.hadoop" % "hadoop-core" % "1.2.1",
"org.apache.hbase" % "hbase" % "0.94.16",
"com.twitter" %% "scalding-args" % "0.10.0",
"com.twitter" %% "scalding-core" % "0.10.0",
"com.twitter" %% "scalding-date" % "0.10.0"
)
// Uncomment to use Akka
//libraryDependencies += "com.typesafe.akka" % "akka-actor_2.11" % "2.3.3"
But I got some unresolved dependencies
[warn] ::::::::::::::::::::::::::::::::::::::::::::::
[warn] :: UNRESOLVED DEPENDENCIES ::
[warn] ::::::::::::::::::::::::::::::::::::::::::::::
[warn] :: com.twitter#scalding-args_2.11;0.10.0: not found
[warn] :: com.twitter#scalding-core_2.11;0.10.0: not found
[warn] :: com.twitter#scalding-date_2.11;0.10.0: not found
[warn] ::::::::::::::::::::::::::::::::::::::::::::::
What version of scalding should i use for scala 2.11 ? Thanks
Scalding 0.13 is now released and it does support Scala 2.11.
https://github.com/twitter/scalding/releases
From google groups
We don't compile for 2.11.0 since its super hard/impossible to have code compatible from 2.9 -> 2.11.
We (at twitter) are still on 2.9.x for the moment so for now we can only support up to the 2.10.x scala's. Hopefully this will change over the next few months.
Also the current scalding version(0.12) is compatible only with scala 2.10(maven)
As #Travis mentioned in the comment, Scalding for Scala version 2.11 is available in the scalding's develop branch(github)
Related
I am trying to work on Twitter stream sample project. i am facing problem while defining sbt.
my build.sbt
name := "Tutorial"
version := "0.1.0"
scalaVersion := "2.11.8"
retrieveManaged := true
libraryDependencies ++= Seq(
"org.apache.spark" % "spark-core" % "2.11.0",
"org.apache.spark" % "spark-streaming" % "1.1.0",
"org.apache.spark" % "spark-streaming-twitter" % "1.1.0"
)
Error Log:
[warn] Note: Some unresolved dependencies have extra attributes. Check that these dependencies exist with the requested attributes.
[warn] com.eed3si9n:sbt-assembly:0.9.2 (sbtVersion=0.11.3, scalaVersion=2.11.8)
[warn] com.typesafe.sbteclipse:sbteclipse-plugin:2.2.0 (sbtVersion=0.11.3, scalaVersion=2.11.8)
[warn] com.github.mpeltonen:sbt-idea:1.5.1 (sbtVersion=0.11.3, scalaVersion=2.11.8)
[warn]
[error] {file:/home/muralee1857/scala/workspace/Tutorial/}default-109f4d/*:update: sbt.ResolveException: unresolved dependency: org.apache.spark#spark-core_2.11.8;1.5.1: not found
[error] unresolved dependency: com.eed3si9n#sbt-assembly;0.9.2: not found
[error] unresolved dependency: com.typesafe.sbteclipse#sbteclipse-plugin;2.2.0: not found
[error] unresolved dependency: com.github.mpeltonen#sbt-idea;1.5.1: not found
I would suggest you to define explicitly the packaged versions to the dependency as
libraryDependencies ++= Seq(
"org.apache.spark" % "spark-core_2.10" % "1.1.0",
"org.apache.spark" % "spark-streaming_2.10" % "1.1.0" % "provided",
"org.apache.spark" % "spark-streaming-twitter_2.10" % "1.1.0"
)
You can use %% without defining the packaged version but that will try to download the package with scala version in your system. And sometimes sbt will not find the scala version packaged packages which will create dependency issues.
This should work. note that I am using %% method instead of % here so that right version of the spark library (scala version 2.11) is selected here. ensure you apply the same %% function to others plugins like sbt-assembly, sbt-assembly etc.
libraryDependencies ++= Seq(
"org.apache.spark" %% "spark-core" % "2.11.0",
"org.apache.spark" %% "spark-streaming" % "1.1.0",
"org.apache.spark" %% "spark-streaming-twitter" % "1.1.0"
)
I followed this link : http://blog.miz.space/tutorial/2016/08/30/how-to-integrate-spark-intellij-idea-and-scala-install-setup-ubuntu-windows-mac/
When I try to compile my project with Intellij, sbt is complaining about unresolved dependencies
[Warn] ===public: tried [Warn]
https://repol.maven.org/maven2/org/apache/spark/spark-core/2.1.1/spark-core-2.1.1.pom
[Warn] Unresolved dependencies path: org.apache.spark:spark-core:2.1.1
My scala version is 2.12.2 and sparkVersion is 2.1.1
Here is what my build.sbt look like :
name := "test" version := "1.0" scalaVersion := "2.12.2"
val sparkVersion = "2.1.1"
libraryDependencies ++= Seq("org.apache.spark" % "spark-core" & sparkVersion)`
thank you
your last line should be
libraryDependencies += "org.apache.spark" % "spark-core_2.10" % sparkVersion
Or
libraryDependencies ++= Seq("org.apache.spark" % "spark-core_2.10" % sparkVersion)
And its better not to use scalaVersion := "2.12.2" as mentioned by #Nonontb and #Cyrelle. Please reduce the version that spark supports for better performance and to avoid unexpected errors.
the following line is from Spark docs
For the Scala API, Spark 2.1.1 uses Scala 2.11. You will need to use a compatible Scala version (2.11.x).
./build/sbt clean package
gives the below error:
Resolving org.fusesource.jansi#jansi;1.4 ...
[warn] ::::::::::::::::::::::::::::::::::::::::::::::
[warn] :: UNRESOLVED DEPENDENCIES ::
[warn] ::::::::::::::::::::::::::::::::::::::::::::::
[warn] :: ml.dmlc.xgboost#xgboost4j_2.10;0.7: not found
[warn] :: ml.dmlc.xgboost#xgboost4j-spark_2.10;0.7: not found
[warn] ::::::::::::::::::::::::::::::::::::::::::::::
build.sbt looks like below:
name := "xgboostproj"
version := "1.0"
libraryDependencies += "org.apache.spark" %% "spark-core" % "1.6.1"
libraryDependencies += "org.apache.spark" %% "spark-mllib" % "1.6.1"
resolvers += Resolver.mavenLocal
libraryDependencies += "ml.dmlc.xgboost" %% "xgboost4j" % "0.7"
libraryDependencies += "ml.dmlc.xgboost" %% "xgboost4j-spark" % "0.7"
Thanks in Advance!
xgboost jars must be built locally and published to your local maven repository for your set up to work. The instructions for this is published here
Also additionally the dependencies should be like below (the groupId is ml.dmlc)
libraryDependencies += "ml.dmlc" %% "xgboost4j" % "0.7"
libraryDependencies += "ml.dmlc" %% "xgboost4j-spark" % "0.7"
I think the dependencies should be changed to:
libraryDependencies += "ml.dmlc" % "xgboost4j" % "0.7"
libraryDependencies += "ml.dmlc" % "xgboost4j-spark" % "0.7"
Also make sure your sbt is looking at the local maven repo. These were the two things that I needed to fix to make it work. Also make sure the jar files exist in the m2 repo. This is assuming you used mvn install to build the jvm packages.
I m trying to run hello slick example from the typesafe activator I create a project in sbt when I type run it gives me following error
warn] ::::::::::::::::::::::::::::::::::::::::::::::
[warn] :: UNRESOLVED DEPENDENCIES ::
[warn] ::::::::::::::::::::::::::::::::::::::::::::::
[warn] :: com.typesafe.slick#slick_2.11;2.0.2: not found
[warn] ::::::::::::::::::::::::::::::::::::::::::::::
[trace] Stack trace suppressed: run last *:update for the full output.
[error] (*:update) sbt.ResolveException: unresolved dependency: com.typesafe.slick#slick_2.11;2.0.2: not found
[error] Total time: 1 s, completed Aug 25, 2014 12:20:42 AM
here is my build.sbt file
name :="hellos"
version :="1.0"
scalaVersion := "2.11.1"
mainClass in Compile := Some("HelloSlick")
libraryDependencies ++= List(
"com.typesafe.slick" %% "slick" % "2.0.2",
"org.slf4j" % "slf4j-nop" % "1.6.4",
"com.h2database" % "h2" % "1.3.170"
)
I m using ubuntu 12.04 and on slick website there is no jar file is present to download so like its not in my system path variable like java and akka is
You need the resolver to the typesafe repository:
libraryDependencies ++= List(
"com.typesafe.slick" %% "slick" % "2.0.2",
"org.slf4j" % "slf4j-nop" % "1.6.4",
"com.h2database" % "h2" % "1.3.170"
)
resolvers += "typesafe" at "http://repo.typesafe.com/typesafe/releases/"
Edit:
It seems that for scala 2.11 you need to specify the slick version to be 2.1:
"com.typesafe.slick" %% "slick" % "2.1.0"
I am fairly new to Scala and SBT. I would like to implement a REST server using Jersey JAX-RS and Scala, but have faced the first hurdle trying to configure SBT. I was planning to use the Grizzly HTTP Server. The build.sbt is as follows:
name := "scala-jersey-server"
version := "1.0"
scalaVersion := "2.10.3"
libraryDependencies += "org.glassfish.jersey.containers" % "jersey-container-grizzly2-http" % "2.5.1"
Running sbt update fails with the following messages
[warn] ::::::::::::::::::::::::::::::::::::::::::::::
[warn] :: UNRESOLVED DEPENDENCIES ::
[warn] ::::::::::::::::::::::::::::::::::::::::::::::
[warn] :: org.glassfish.hk2#hk2-utils;2.5.1: not found
[warn] :: org.glassfish.hk2#hk2-locator;2.5.1: not found
[warn] :: javax.validation#validation-api;${javax.validation.version}: not found
[warn] ::::::::::::::::::::::::::::::::::::::::::::::
Looking at the pom.xml for hk2-api, it shows dependency on hk2-utils using the ${project.version} variable, but I'd have expected version 2.2.0-b21 of hk2-utils to be requested by SBT not 2.5.1 (which is JAX-RS version).
<dependency>
<groupId>org.glassfish.hk2</groupId>
<artifactId>hk2-utils</artifactId>
<version>${project.version}</version>
</dependency>
Is this a problem with the configuration of hk2-api's pom.xml, is SBT using an incorrect version or something else?
I think there is an error in the publication chain of your dependency for the 2.5.1 version.
There are at least three possibilities:
1. Use another version
//http://search.maven.org/#search%7Cgav%7C1%7Cg%3A%22org.glassfish.jersey.containers%22%20AND%20a%3A%22jersey-container-grizzly2-http%22
libraryDependencies += "org.glassfish.jersey.containers" % "jersey-container-grizzly2-http" % "2.5"
2. Use another library
3. Exclude the broken dependencies and set working versions
Use http://search.maven.org/ to find existing versions.
libraryDependencies += "org.glassfish.jersey.containers" % "jersey-container-grizzly2-http" % "2.5.1" exclude("org.glassfish.hk2", "hk2-utils") exclude("org.glassfish.hk2", "hk2-locator") exclude("javax.validation", "validation-api")
//http://search.maven.org/#browse%7C703203458
libraryDependencies += "org.glassfish.hk2" % "hk2-utils" % "2.2.0-b27"
//http://search.maven.org/#search%7Cga%7C1%7Ca%3A%22hk2-locator%22
libraryDependencies += "org.glassfish.hk2" % "hk2-locator" % "2.2.0-b27"
//http://search.maven.org/#artifactdetails%7Cjavax.validation%7Cvalidation-api%7C1.1.0.Final%7Cjar
libraryDependencies += "javax.validation" % "validation-api" % "1.1.0.Final"
**I tried the above .sbt file but it was not excluding the jars and was still giving the same ERROR.
So modified the build file. below sample it could create a build successfully.**
name := "KafkaSparkCouchReadWrite"
organization := "my.clairvoyant"
version := "1.0.0-SNAPSHOT"
scalaVersion := "2.11.11"
libraryDependencies ++= Seq(
"org.apache.spark" %% "spark-core" % "2.1.0",
"org.apache.spark" %% "spark-streaming" % "2.1.0",
"org.apache.spark" %% "spark-sql" % "2.1.0",
"com.couchbase.client" %% "spark-connector" % "2.1.0",
"org.glassfish.hk2" % "hk2-utils" % "2.2.0-b27",
"org.glassfish.hk2" % "hk2-locator" % "2.2.0-b27",
"javax.validation" % "validation-api" % "1.1.0.Final"
).map(_.excludeAll(ExclusionRule("org.glassfish.hk2"),ExclusionRule("javax.validation")))