Sorry I am fairly new to Scala and SBT. Here is my build.sbt file
name := "test_stream"
version := "0.1"
scalaVersion := "2.12.10"
resolvers in ThisBuild += Resolver.bintrayRepo("streetcontxt", "maven")
mainClass in Compile := Some("basepackage.Main")
enablePlugins(JavaAppPackaging)
enablePlugins(DockerPlugin)
libraryDependencies ++= Seq(
"com.typesafe.akka" %% "akka-stream" % "2.6.1",
"com.amazonaws" % "aws-java-sdk-s3" % "1.11.693",
"com.streetcontxt" %% "kcl-akka-stream" % "2.0.3",
"me.maciejb.snappyflows" %% "snappy-flows" % "0.2.0",
"org.xerial.snappy" % "snappy-java" % "1.1.7.3",
"org.apache.hadoop" % "hadoop-common" % "2.10.0",
"org.apache.hadoop" % "hadoop-core" % "1.2.1"
)
And I get the following error:
sbt.librarymanagement.ResolveException: unresolved dependency: org.apache.hadoop#hadoop-common;2.10.0: Resolution failed several times for dependency: org.apache.hadoop#hadoop-common;2.10.0 {compile=[default(compile)]}::
try removing ~/.sbt and ~/.ivy2 and run again
Related
My sbt file looks as follows
organization := "scala"
name := "MyProject"
version := "1.0"
scalaVersion := "2.12.1"
libraryDependencies += "com.github.nscala-time" %% "nscala-time" % "2.20.0"
libraryDependencies += "commons-net" % "commons-net" % "3.6"
libraryDependencies += "commons-validator" % "commons-validator" % "1.6.0"
When I run sbt compile I get this
sbt.librarymanagement.ResolveException: unresolved dependency: commons-validator#commons-validator;1.6.0: not found
However when I change scala version to 2.11.7, sbt compiles fine. What am I missing? How can I make it work for 2.12.1?
According to mvnrepo (https://mvnrepository.com/artifact/commons-validator/commons-validator)
use libraryDependencies += "commons-validator" % "commons-validator" % "1.6"
I mentioned json4s-native 3.3.0 but it refering 3.3.0-SNAPSHOT.
In my project there is 3 modules (common, read and write) each have its scala.sbt.
Before change:
"org.json4s" %% "json4s-native" % Versions.Json4s % "compile",
After change:(Changed all the places in 3 modules)
"org.json4s" %% "json4s-native" % "3.3.0" % "compile",
I run:
sbt
clean
update
Error:
[warn] ::::::::::::::::::::::::::::::::::::::::::::::
[warn] :: UNRESOLVED DEPENDENCIES ::
[warn] ::::::::::::::::::::::::::::::::::::::::::::::
[warn] :: org.json4s#json4s-native_2.10;3.3.0-SNAPSHOT: not found
[warn] ::::::::::::::::::::::::::::::::::::::::::::::
[warn]
[warn] Note: Unresolved dependencies path:
[warn] org.json4s:json4s-native_2.10:3.3.0-SNAPSHOT
My Sales buld.sbt is this:
name := "myapp-sales"
organization in ThisBuild := "com.mycompany"
scalaVersion in ThisBuild := "2.11.7"
//crossScalaVersions in Thisq
//Build := List("2.10.5", scalaVersion.value)
//crossPaths := false
scalacOptions in Compile in ThisBuild ++= Seq("-unchecked", "-deprecation", "-encoding", "utf8", "-language:postfixOps", "-language:implicitConVersions")
lazy val common = Project("myapp-sales-common", file("common"))
lazy val read = Project("myapp-sales-read", file("read")).configs(IntegrationTest).settings(Defaults.itSettings: _*).dependsOn(common, write)
lazy val write = Project("myapp-sales-write", file("write")).configs(IntegrationTest).settings(Defaults.itSettings: _*).dependsOn(common)
conflictWarning in ThisBuild := ConflictWarning.disable
parallelExecution in Test in ThisBuild := false
parallelExecution in IntegrationTest in ThisBuild := false
javacOptions in Compile in ThisBuild ++= Seq("-source", "1.6", "-target", "1.6")
//Remove SNAPSHOT check from the release process (for now until Squants gets a release)
releaseProcess := releaseProcess.value.filterNot(_ == ReleaseTransformations.checkSnapshotDependencies)
My read module build.sbt:
libraryDependencies ++= Seq(
"com.mycompany" %% "myapp-core-read" % myappsales.CoreVersion % "compile", // disable using the Scala version in output paths and artifacts,
"com.mycompany" %% "myapp-core-write" % myappsales.CoreVersion % "compile",
"com.mycompany" %% "myapp-registration-common" % myappsales.RegistrationVersion % "compile",
"com.mycompany" %% "myapp-load-common" % myappsales.LoadVersion % "compile",
"com.mycompany" %% "myapp-core-write-test" % myappsales.CoreVersion % "it, test",
"com.mycompany" %% "myapp-core-test" % myappsales.CoreVersion % "it, test"
)
libraryDependencies ++= Seq(
"com.typesafe" % "config" % myappsales.TypeSafeConfigVersion % "compile",
"org.json4s" %% "json4s-native" % Versions.Json4s % "compile",
"io.spray" % "spray-routing" % Versions.Spray % "compile",
"com.typesafe.akka" %% "akka-actor" % Versions.Akka % "compile",
"com.typesafe.akka" %% "akka-remote" % Versions.Akka % "compile"
exclude ("io.netty", "netty")
)
//Assemby settings
test in assembly := {}
assemblyShadeRules in assembly := Seq(
ShadeRule.rename("play.api.libs.iteratee.**" -> "shade.play.api.libs.iteratee.#1")
.inLibrary("play" % "play-iteratees_2.10" % "2.1-RC2")
.inLibrary("org.reactivemongo" % "reactivemongo_2.10" % "0.8.1-SNAPSHOT"),
ShadeRule.rename("scala.concurrent.stm.**" -> "shade.scala.concurrent.stm.#1")
.inLibrary("org.scala-stm" % "scala-stm_2.10.0" % "0.6")
.inLibrary("play" % "play-iteratees_2.10" % "2.1-RC2")
)
assemblyMergeStrategy in assembly := {
case "application.conf" => MergeStrategy.concat
case path => MergeStrategy.defaultMergeStrategy(path)
}
//Make assembly a publishable artifact
artifact in (
Compile, assembly) := {
val art = (artifact in (Compile, assembly)).value
art.copy(`classifier` = Some("assembly"))
}
addArtifact(artifact in (Compile, assembly), assembly)
Why it still looking for SNAPSHOT version?
There's a fairly standard way of investigating this kind of problems. In your plugins.sbt
addSbtPlugin("net.virtual-void" % "sbt-dependency-graph" % "0.8.2")
Then go sbt dependencyBrowseGraph, this will open your browser and take you to a page where the entire graph of your dependencies including the evicted one can be browsed. Now figure out where the SNAPSHOT is coming from and get rid of it with an exclusion.
Runtime classpath according to 'show runtime:fullClasspath' contains only target/scala-2.11/classes and ~/.ivy2/cache/org.scala-lang/scala-library/jars/scala-library-2.11.7.jar.
compile:fullClasspath contains all libraryDependencies jar locations under ~/.ivy2/cache. Why is this? I am getting java.lang.NoClassDefFoundError on sbt run.
build.sbt:
name := "my-server"
version := "1.0"
scalaVersion := "2.11.7"
libraryDependencies ++= List(
"com.typesafe.slick" %% "slick" % "3.1.0" % "provided",
"com.twitter.finatra" %% "finatra-http" % "2.1.0" % "provided",
"com.roundeights" %% "hasher" % "1.2.0" % "provided",
"com.twitter" %% "util-logging" % "6.29.0" % "provided"
)
resolvers +=
"Twitter" at "http://maven.twttr.com"
resolvers ++= Seq("RoundEights" at "http://maven.spikemark.net/roundeights")
sbt run results:
Exception in thread "main" java.lang.NoClassDefFoundError: com/twitter/logging/Logger
sbt version 0.13.8
Removing "provided" was the fix here - I was using it incorrectly to resolve ambiguous subversions of dependencies (credit to pfn from freenode #scala)
I have written the following sbt file
name := "Test"
version := "1.0"
scalaVersion := "2.11.7"
libraryDependencies ++= Seq(
"org.apache.hadoop" % "hadoop-client" % "2.7.1",
"org.apache.spark" % "spark-core_2.10" % "1.3.0",
"org.apache.avro" % "avro" % "1.7.7",
"org.apache.avro" % "avro-mapred" % "1.7.7"
)
mainClass := Some("com.test.Foo")
I also have the following assembly.sbt file in projects folder
resolvers += Resolver.url("bintray-sbt-plugins", url("http://dl.bintray.com/sbt/sbt-plugin-releases"))(Resolver.ivyStylePatterns)
addSbtPlugin("com.eed3si9n" % "sbt-assembly" % "0.14.0")
when i do sbt assembly i get a huge list of errors
[error] (*:assembly) deduplicate: different file contents found in the following:
[error] /Users/abhishek.srivastava/.ivy2/cache/com.esotericsoftware.kryo/kryo/bundles/kryo-2.21.jar:com/esotericsoftware/minlog/Log$Logger.class
[error] /Users/abhishek.srivastava/.ivy2/cache/com.esotericsoftware.minlog/minlog/jars/minlog-1.2.jar:com/esotericsoftware/minlog/Log$Logger.class
[error] deduplicate: different file contents found in the following:
[error] /Users/abhishek.srivastava/.ivy2/cache/com.esotericsoftware.kryo/kryo/bundles/kryo-2.21.jar:com/esotericsoftware/minlog/Log.class
[error] /Users/abhishek.srivastava/.ivy2/cache/com.esotericsoftware.minlog/minlog/jars/minlog-1.2.jar:com/esotericsoftware/minlog/Log.class
[error] deduplicate: different file contents found in the following:
I was able to resolve the problem. actually there is no need to build a fat jar because the "spark-submit" tool will have everything in the classpath anyway.
thus the right way to build the jar file is
name := "Test"
version := "1.0"
scalaVersion := "2.11.7"
libraryDependencies ++= Seq(
"org.apache.hadoop" % "hadoop-client" % "2.7.1" % "provided",
"org.apache.spark" % "spark-core_2.10" % "1.3.0" % "provided",
"org.apache.avro" % "avro" % "1.7.7" % "provided",
"org.apache.avro" % "avro-mapred" % "1.7.7" % "provided"
)
mainClass := Some("com.test.Foo")
1. use the MergeStrategy, see sbt-assembly
2. exclude the duplicated jars, such as:
lazy val hbaseLibSeq = Seq(
("org.apache.hbase" % "hbase" % hbaseVersion).
excludeAll(
ExclusionRule(organization = "org.slf4j"),
ExclusionRule(organization = "org.mortbay.jetty"),
ExclusionRule(organization = "javax.servlet")),
("net.java.dev.jets3t" % "jets3t" % "0.6.1" % "provided").
excludeAll(ExclusionRule(organization = "javax.servlet"))
)
3. use the provided scope
show dependency tree:
➜ cat ~/.sbt/0.13/plugins/plugins.sbt
addSbtPlugin("net.virtual-void" % "sbt-dependency-graph" % "0.7.5")
addSbtPlugin("com.eed3si9n" % "sbt-assembly" % "0.11.2")
➜ cat ~/.sbt/0.13/global.sbt
net.virtualvoid.sbt.graph.Plugin.graphSettings
➜ sbt dependency-graph
I would like to use the phantom cassandra wrapper in my scala project, but when I try to update my sbt build I get a dependency error.
My build.sbt:
version := "1.0"
scalaVersion := "2.11.2"
seq(lsSettings :_*)
libraryDependencies ++= Seq(
"org.clapper" %% "grizzled-scala" % "1.2",
"commons-io" % "commons-io" % "2.4",
"org.rauschig" % "jarchivelib" % "0.6.0",
"com.google.code.findbugs" % "jsr305" % "3.0.0",
"org.scalatest" % "scalatest_2.11" % "2.2.0" % "test",
"com.github.nscala-time" %% "nscala-time" % "1.2.0",
"org.json4s" %% "json4s-native" % "3.2.10",
"org.scala-lang" % "scala-library" % "2.11.2",
"com.websudos" % "phantom-dsl_2.10" % "1.2.0"
)
resolvers += "grizzled-scala-resolver-0" at "https://oss.sonatype.org/content/repositories/releases"
resolvers += "Typesafe repository releases" at "http://repo.typesafe.com/typesafe/releases/"
I get the following error:
[warn] Note: Some unresolved dependencies have extra attributes. Check that these dependencies exist with the requested attributes.
[warn] com.typesafe.sbt:sbt-pgp:0.8.1 (sbtVersion=0.13, scalaVersion=2.10)
Don't know what I have to do...
edit:
Answer from https://github.com/websudosuk/phantom/issues/119
error is on the pom side, new version 1.2.1 coming soon...
Answer from https://github.com/websudosuk/phantom/issues/119
error is on the pom side, new version 1.2.1 coming soon...