I am trying to make a fat jar by running sbt assembly for my project.
I am getting the following error :
[error] (root/*:assembly) deduplicate: different file contents found in the following:
[error] /Users/xyz/.ivy2/cache/org.apache.hadoop/hadoop-mapreduce-client-core/jars/hadoop-mapreduce-client-core-2.2.0.jar:org/apache/hadoop/filecache/DistributedCache.class
[error] /Users/xyz/.ivy2/cache/org.apache.hadoop/hadoop-core/jars/hadoop-core-2.0.0-mr1-cdh4.7.1.jar:org/apache/hadoop/filecache/DistributedCache.class
DistributedCache of hadoop-mapreduce-client-core is deprecated now.
In my build.sbt I have included :
"org.apache.hadoop" % "hadoop-client" % "2.0.0-mr1-cdh4.7.1" excludeAll(
ExclusionRule(organization = "javax.servlet"))
The dependency is like this :
org.apache.hadoop:hadoop-client:2.2.0
org.apache.hadoop:hadoop-mapreduce-client-app:2.2.0
org.apache.hadoop:hadoop-mapreduce-client-core:2.2.0
How do I handle this?
Thanks in advance!
if you intend to remove the dependency jar of mapreduce-client-app when it is being load when you rely on hadoop-client:2.2.0, just simply add intransitive:
"org.apache.hadoop" % "hadoop-client" % "2.2.0" intransitive()
This will only include the hadoop-client:2.2.0 jar and exclude all its dependencies.
Related
I have installed Flink, Scala and sbt
Flink Version: 1.9.1
Scala Version: 2.10.6
Sbt Version: 1.3.7
I made relevant changes in build.sbt.
Compile command is failing
Here is the relevant information.
Any information is greatly appreciated
**Versions Information
[osboxes#osboxes local]$ scala -version
Scala code runner version 2.10.6 -- Copyright 2002-2013, LAMP/EPFL
[osboxes#osboxes local]$ flink --version
Version: 1.9.1, Commit ID: 4d56de8
[osboxes#osboxes readcsvfile]$ sbt -version
sbt version in this project: 1.3.7
sbt script version: 1.3.7
** build.sbt changes
val flinkVersion = "1.9.1"
val flinkDependencies = Seq(
"org.apache.flink" %% "flink-scala" % flinkVersion % "provided",
"org.apache.flink" %% "flink-streaming-scala" % flinkVersion % "provided")
** Compile Errors
sbt:readCsvfile> compile
[info] Updating
[info] Resolved dependencies
[warn]
[warn] Note: Unresolved dependencies path:
[error] stack trace is suppressed; run last update for the full output
[error] (update) sbt.librarymanagement.ResolveException: Error downloading org.apache.flink:flink-streaming-scala_2.13:1.9.1
[error] Not found
[error] Not found
[error] not found: /home/osboxes/.ivy2/local/org.apache.flink/flink-streaming-scala_2.13/1.9.1/ivys/ivy.xml
[error] not found: https://repo1.maven.org/maven2/org/apache/flink/flink-streaming-scala_2.13/1.9.1/flink-streaming-scala_2.13-1.9.1.pom
[error] Error downloading org.apache.flink:flink-scala_2.13:1.9.1
[error] Not found
[error] Not found
[error] not found: /home/osboxes/.ivy2/local/org.apache.flink/flink-scala_2.13/1.9.1/ivys/ivy.xml
[error] not found: https://repo1.maven.org/maven2/org/apache/flink/flink-scala_2.13/1.9.1/flink-scala_2.13-1.9.1.pom
[error] Total time: 4 s, completed Jan 30, 2020 3:59:12 PM
sbt:readCsvfile>
Few points I want to mention here regarding the SBT dependencies issues are:
Please add scalaVersion := "2.12.11" in build.sbt file like this, which includes the Scala version in your SBT dependencies automatically due to this%%.
name := "flink-streaming-demo"
scalaVersion := "2.12.11"
val flinkVersion = "1.10.0"
libraryDependencies += "org.apache.flink" %% "flink-scala" % flinkVersion % "provided"
libraryDependencies += "org.apache.flink" %% "flink-streaming-scala" % flinkVersion % "provided"
If you want Scala version specific SBT dependencies then use % like this:
libraryDependencies += "org.apache.flink" % "flink-scala_2.12" % flinkVersion % "provided"
libraryDependencies += "org.apache.flink" % "flink-streaming-scala_2.12" % flinkVersion % "provided"
In worst case if all these does not work then simply delete or rename these existing .sbt and .ivy2 hidden folder in your system home directory, where your all dependecies and plugins get sotred after downloading from maven central and then refresh/build the SBT project.
SBT dependency format
libraryDependencies += groupID % artifactID % revision % configuration
Meaning of % and %%
%: A method used to construct an Ivy Module ID from the strings you supply.
%%: When used after the groupID, it automatically adds your project’s Scala version (such as _2.12) to the end of the artifact name.
NOTE: To get more details click here.
summing up the comments since perhaps it is a bit hard to know what you should do
In general, if you get an "Unresolved dependencies" error, look at mvnrepository.com, search for your artifact:
https://mvnrepository.com/artifact/org.apache.flink/flink-scala
This tells you (second column) which Scala versions are supported by it. In this case, the library is available for 2.11.x and 2.12.x.
Thus, you have to use a Scala version compatible with that in your build, in build.sbt:
ThisBuild / scalaVersion := "2.12.10"
I am new to scalatest and scalamock. Here is what I have it in my sbt file
name := "cakepattern"
version := "0.1"
scalaVersion := "2.11.8"
libraryDependencies ++= Seq(
"org.scalatest" %% "scalatest" % "3.0.0" % "test",
"org.scalamock" %% "scalamock-core" % "3.1.1" % "test",
"org.scalamock" %% "scalamock-scalatest-support" % "3.1.1" % "test",
"org.scalacheck" %% "scalacheck" % "1.13.0" % "test"
)
And here is a class that I am trying to mock
package config
import dto.User
import services.AuthServiceComponent
import org.scalatest.mockito.MockitoSugar._
import services.impl.DefaultUserAuthServiceComponent
trait MockAuthServiceComponent extends AuthServiceComponent{
val x = mock[AuthServiceLike]
type AuthService = x.type
override val userAuthService = x
}
When I am do sbt test:compile I get the following error
[error] missing or invalid dependency detected while loading class file 'MockitoSugar.class'.
[error] Could not access term mockito in package org,
[error] because it (or its dependencies) are missing. Check your build definition for
[error] missing or conflicting dependencies. (Re-run with `-Ylog-classpath` to see the problematic classpath.)
[error] A full rebuild may help if 'MockitoSugar.class' was compiled against an incompatible version of org.
[error] missing or invalid dependency detected while loading class file 'MockitoSugar.class'.
[error] Could not access type MockSettings in value org.mockito,
[error] because it (or its dependencies) are missing. Check your build definition for
[error] missing or conflicting dependencies. (Re-run with `-Ylog-classpath` to see the problematic classpath.)
[error] A full rebuild may help if 'MockitoSugar.class' was compiled against an incompatible version of org.mockito.
[error] missing or invalid dependency detected while loading class file 'MockitoSugar.class'.
[error] Could not access type Answer in value org.stubbing,
[error] because it (or its dependencies) are missing. Check your build definition for
[error] missing or conflicting dependencies. (Re-run with `-Ylog-classpath` to see the problematic classpath.)
[error] A full rebuild may help if 'MockitoSugar.class' was compiled against an incompatible version of org.stubbing.
[error] three errors found
[error] (test:compileIncremental) Compilation failed
What am I missing?
[Edit]
So the problem I was having before is resolved but now I get this
Error:scalac: missing or invalid dependency detected while loading class file 'AbstractMockFactory.class'.
Could not access type NoArgTest in trait org.scalatest.Suite,
because it (or its dependencies) are missing. Check your build definition for
missing or conflicting dependencies. (Re-run with `-Ylog-classpath` to see the problematic classpath.)
A full rebuild may help if 'AbstractMockFactory.class' was compiled against an incompatible version of org.scalatest.Suite.
Any suggestions?
Try adding Mockito to your sbt file:
libraryDependencies ++= Seq(
"org.scalatest" %% "scalatest" % "3.0.0" % Test,
"org.scalamock" %% "scalamock-core" % "3.1.1" % Test,
"org.scalamock" %% "scalamock-scalatest-support" % "3.1.1" % Test,
"org.scalacheck" %% "scalacheck" % "1.13.0" % Test,
"org.mockito" % "mockito-all" % "1.10.19" % Test
)
Take care, it's a simple "%", not a double one (it's a java dependency)
Other versions here : https://mvnrepository.com/artifact/org.mockito/mockito-all if 1.10.19 is not compatible with your codebase
EDIT :
Not sure that will help your second problem, but try that in an SBT console :
> clean
> compile
Versions <3.3.0 are not compatible with Scalatest 3+.
I suggest updating to scalamock 3.4.2 (latest one as of writing).
You can always find the newest version on Maven Central.
Also, no need to specify scalamock-core, that gets pulled in automatically.
scalacheck 3.2 depend on scalatest 2.1.3, but you used scalatest 3.0.0, so they are not compatibal. remove the scalatest dependence will resolve the problem.
I was using sbt to compile a scala code, and got the following error message
[error] /user/xyin/Projects/measurement_wspark/src/main/scala/json2csv.scala:41: bad symbolic reference. A signature in DefaultReads.class refers to term time
[error] in package java which is not available.
[error] It may be completely missing from the current classpath, or the version on
[error] the classpath might be incompatible with the version used when compiling DefaultReads.class.
[error] val ts = (json \ "RequestTimestamp").validate[String]
[error] ^
[error] one error found
[error] (compile:compileIncremental) Compilation failed
I was using the play json library to parse json files in my main class. And the content of the build.sbt file is like the following:
name := "json_parser"
version := "1.0"
scalaVersion := "2.10.1"
libraryDependencies += "org.apache.spark" %% "spark-core" % "1.6.1"
resolvers += "Typesafe Repo" at "http://repo.typesafe.com/typesafe/releases/"
libraryDependencies += "com.typesafe.play" %% "play-json" % "2.4.8"
I am totally new to scala and the play framework. Does anyone has any clue about this problem? Thanks!
I'm trying to add parboiled2 as a dependency to my project, and follow the Calculator example but it conflicts with spray.
My current build.sbt file includes:
"io.spray" %% "spray-json" % "1.3.1" withSources() withJavadoc(),
"io.spray" %% "spray-can" % sprayV withSources() withJavadoc(),
"io.spray" %% "spray-routing" % sprayV withSources() withJavadoc(),
"io.spray" %% "spray-testkit" % sprayV % "test" withSources() withJavadoc(),
When I add
"org.parboiled" %% "parboiled" % "2.0.1" withSources() withJavadoc(),
I get
[error] Modules were resolved with conflicting cross-version suffixes in {file:/blar/blar}blar-blar:
[error] com.chuusai:shapeless _2.10.4, _2.10
[error] org.scalamacros:quasiquotes _2.10, _2.10.3
java.lang.RuntimeException: Conflicting cross-version suffixes in: com.chuusai:shapeless, org.scalamacros:quasiquotes
So did some googling, and as usual people suggest using the exclude directive of SBT (which I don't believe makes logical sense as it will inevitably result in problems like ClassNotFoundException and NoSuchMethodError). I tried it nevertheless:
"org.parboiled" %% "parboiled" % "2.0.1" withSources() withJavadoc()
exclude("com.chuusai", "shapeless_2.10.4") exclude("org.scalamacros", "quasiquotes_2.10")
And surprise surprise when I try to run assembly I get
[error] java.lang.ClassNotFoundException: scala.quasiquotes.QuasiquoteCompat$
I also get a load more errors:
[error] bad symbolic reference
. A signature in RuleDSLBasics.class refers to term internal
[error] in package scala.reflect which is not available.
[error] It may be completely missing from the current classpath, or the version on
[error] the classpath might be incompatible with the version used when compiling RuleDSLBasics.class.
[error] error while loading RuleDSLBasics, Missing dependency 'bad symbolic reference. A signature in RuleDSLBasics.class refers to term annotations
[error] in value scala.reflect.internal which is not available.
[error] It may be completely missing from the current classpath, or the version on
[error] the classpath might be incompatible with the version used when compiling RuleDSLBasics.class.', required by ...
If I comment out the Spray dependencies (and code) I can successfully assembly a jar with the Calculator example.
This is a major show stopper for me using parboiled2. Is there some way other than exclude directives in order to make this stuff work? Does anyone have a working build file with both Spray and parboiled2? If someone has managed to get around dependency hell with onejar or ProGuard I'd love it if they could explain just how.
UPDATE:
My build file:
resolvers ++= Seq(
"Concurrent Maven Repo" at "http://conjars.org/repo",
"spray repo" at "http://repo.spray.io"
)
val akkaV = "2.3.6"
val sprayV = "1.3.2"
libraryDependencies ++= Seq(
"org.parboiled" %% "parboiled" % "2.0.1" withSources() withJavadoc(),
// Causes org.scalamacros:quasiquotes _2.10, _2.10.3 cross-version problem
"io.spray" %% "spray-testkit" % sprayV % "test" withSources() withJavadoc(),
// Causes com.chuusai:shapeless _2.10.4, _2.10 cross-version problem
"io.spray" %% "spray-routing" % sprayV withSources() withJavadoc()
)
scalaVersion := "2.10.4"
javaOptions ++= Seq("-target", "1.8", "-source", "1.8")
organization := domain + "." + companyName
Plugins file:
addSbtPlugin("com.eed3si9n" % "sbt-assembly" % "0.12.0")
addSbtPlugin("com.github.mpeltonen" % "sbt-idea" % "1.6.0")
Scala 2.11
It seems "simply" moving 2.11 may solve this, but this isn't always that easy if one has some other libraries that are not 2.11 ready. Nevertheless I tried it, and the second I added parboiled2 as a dependency it broke my build again, sigh, here is the new problem: Parboiled2 causes "missing or invalid dependency detected while loading class file 'Prepender.class'"
The same answer as for Scala 2.11 should work here as well: replace spray-routing with spray-routing-shapeless2.
I would have to look at the whole build.sbt file to understand what is happening but I can use the dependencies you mention without problem with sbt 0.13.7 and scala 2.11.
What sbt version? What Scala version? 2.10 it seems. Why? Are you stuck with those versions?
For reference, here are the build.sbt, plugins.sbt and build.properties files I used to test this scenario.
https://gist.github.com/fedesilva/ddd28a5b592eb88f41ae
Hope that helps
I am trying to create a REST service using Spray Servlet, but does not compile.
Below you can find my build.sbt:
name := "someservice"
version := "0.0.1"
scalaVersion := "2.10.0"
resolvers += "spray repo" at "http://repo.spray.io"
libraryDependencies += "io.spray" % "spray-servlet" % "1.0-M3"
libraryDependencies ++= Seq(
"io.spray" % "spray-servlet" % "1.0-M3",
"io.spray" % "spray-util" % "1.0-M3",
"io.spray" % "spray-http" % "1.1-M7",
"com.typesafe.akka" %% "akka-actor" % "2.1.0",
"org.specs2" %% "specs2" % "1.13" % "test",
"org.eclipse.jetty.orbit" % "javax.servlet" % "3.0.0.v201112011016" artifacts Artifact("javax.servlet", "jar", "jar")
)
There error I get com 'sbt update compile' is:
[error] bad symbolic reference. A signature in package.class refers to term Either
[error] in package scala which is not available.
[error] It may be completely missing from the current classpath, or the version on
[error] the classpath might be incompatible with the version used when compiling
[error] package.class.
[error] bad symbolic reference. A signature in package.class refers to type Future
[error] in package akka.dispatch which is not available.
[error] It may be completely missing from the current classpath, or the version on
[error] the classpath might be incompatible with the version used when
[error] compiling package.class.
[error] two errors found
[error]
(compile:compile) Compilation failed
Do I have to add the scala library to the sbt build file as well or am I missing something else?
tld;dr: try replacing 1.0-M3 with 1.1-M7
It looks like the Spray artifacts you're using were compiled against Scala 2.9.2, which is not binary compatible with Scala 2.10. Usually, Scala artifacts contain the Scala binary version in the artifact name, precisely because major Scala versions are not binary compatible.
Later spray-util milestones were compiled against Scala 2.10.0-RC5 (based on the pom), which technically should work.
Solved by using all the same spray versions (1.1-M7) and upgrading to Scala 2.10.1