Why is the error Conflicting cross-version suffixes? - scala

I'm getting this error when I try to compile a Scala project in sbt.
Modules were resolved with conflicting cross-version suffixes in {file:/home/seven3n/caja/Flujo_de_caja/}flujo_de_caja:
[error] com.typesafe.akka:akka-actor _2.11, _2.10
[error] org.scalaz:scalaz-effect _2.10, _2.11
[error] org.scalaz:scalaz-core _2.10, _2.11
[trace] Stack trace suppressed: run last *:update for the full output.
[error] (*:update) Conflicting cross-version suffixes in: com.typesafe.akka:akka-actor, org.scalaz:scalaz-effect, org.scalaz:scalaz-core
This is my build.sbt file:
scalaVersion := "2.11.0"
resolvers ++= Seq(
"Sonatype snapshots repository" at "https://oss.sonatype.org/content/repositories/snapshots/",
"Spray repository" at "http://repo.spray.io/",
"Typesafe repository" at "http://repo.typesafe.com/typesafe/releases/"
)
libraryDependencies ++= {
val akkaVersion = "2.3.2"
val sprayVersion = "1.3.1-20140423"
val sprayJsonVersion = "1.2.6"
val reactiveMongoVersion = "0.11.0-SNAPSHOT"
val scalaTestVersion = "2.1.5"
val specs2Version = "2.3.11"
val foloneVersion = "0.12-SNAPSHOT"
Seq(
"com.typesafe.akka" %% "akka-actor" % akkaVersion,
"com.typesafe.akka" %% "akka-testkit" % akkaVersion,
"io.spray" %% "spray-can" % sprayVersion,
"io.spray" %% "spray-routing" % sprayVersion,
"io.spray" %% "spray-testkit" % sprayVersion,
"io.spray" %% "spray-json" % sprayJsonVersion,
"org.reactivemongo" % "reactivemongo_2.10" % reactiveMongoVersion,
"org.scalatest" %% "scalatest" % scalaTestVersion % "test",
"org.specs2" %% "specs2" % specs2Version % "test",
"info.folone" % "poi-scala_2.10" % foloneVersion
)
}
Any suggestions?

The conflicts appear because:
you've specified your Scala version to be 2.11
you've explicitly specified the Scala version (2.10) for the reactivemongo and poi-scala libraries.
The fix is to use the %% operator for those two libraries as well.
"org.reactivemongo" %% "reactivemongo" % reactiveMongoVersion,
"info.folone" %% "poi-scala" % foloneVersion
That's the purpose of the %% operator. To append the declared Scala version (2.11 in your case) to the artifact name.

I had the same problem and I simply removed the scalaVersion tag from my sbt file and modified the line
libraryDependencies += "org.apache.spark" %% "spark-core" % "1.6.0"
to
libraryDependencies += "org.apache.spark" % "spark-core_2.11" % "1.6.0"
and the problem went away.

I have tried using %% but it didn't work. I have manually excluded it using
("org.reactivemongo" % "reactivemongo" % reactiveMongoVersion)
.exclude("com.typesafe.akka", "akka-actor_2.10")
.exclude("org.scalaz", "scalaz-effect")
.exclude("org.scalaz", "scalaz-core")

To investigate who is caller you can use a plugin but an easier way to look into target/scala-2.*/resolution-cache/reports/.
There is Ivy's resolution report for each configuration.
Look for *-compile.xml and *-test.xml and search for conflicting library.
You can see similar with
<module organisation="com.github.nscala-time" name="nscala-time_2.11">
...
<caller organisation="com.tumblr" name="colossus-metrics_2.11" conf="compile, runtime" rev="1.2.0" rev-constraint-default="1.2.0" rev-constraint-dynamic="1.2.0" callerrev="0.7.2-RC1"/>
...
</module>
This should tell you the caller of the module.

Related

SCALA and Elastic Search: Add symbol to classpath (Databricks)

:)
I'm getting an error that I have no idea how to fix.. I could not really find outstanding documentation for this SchemaRDD type, and how to use it.
build.sbt contains:
scalaVersion := "2.11.12"
libraryDependencies += "org.apache.spark" %% "spark-sql" % "2.4.0"
libraryDependencies += "org.scalaj" %% "scalaj-http" % "2.4.1"
libraryDependencies += "io.spray" %% "spray-json" % "1.3.5"
libraryDependencies += "com.amazonaws" % "aws-java-sdk-core" % "1.11.534"
libraryDependencies += "com.amazonaws" % "aws-encryption-sdk-java" % "1.3.6"
libraryDependencies += "com.amazonaws" % "aws-java-sdk" % "1.11.550"
libraryDependencies += "com.typesafe" % "config" % "1.3.4"
libraryDependencies += "org.elasticsearch" %% "elasticsearch-spark-1.2" % "2.4.4"
Error:
Symbol 'type org.apache.spark.sql.SchemaRDD' is missing from the classpath.
[error] This symbol is required by 'value org.elasticsearch.spark.sql.package.rdd'.
[error] Make sure that type SchemaRDD is in your classpath and check for conflicting dependencies with `-Ylog-classpath`.
[error] A full rebuild may help if 'package.class' was compiled against an incompatible version of org.apache.spark.sql.
Thank you a lot for all kind of support! :)
Dependency elasticsearch-spark-1.2 is for Spark 1.x, need to use elasticsearch-spark-20 instead. The latest version is built for Spark 2.3
libraryDependencies += "org.elasticsearch" %% "elasticsearch-spark-20" % "7.1.1"

Build.sbt breaks when adding GraphFrames build with scala 2.11

I'm trying to add GraphFrames to my scala spark application, and this was going fine when I added the one based on 2.10. However, as soon as I tried to build it with GraphFrames build with scala 2.11, it breaks.
The problem would be that there are conflicting versions of scala used (2.10 and 2.11). I'm getting the following error:
[error] Modules were resolved with conflicting cross-version suffixes in {file:/E:/Documents/School/LSDE/hadoopcryptoledger/examples/scala-spark-graphx-bitcointransaction/}root:
[error] org.apache.spark:spark-launcher _2.10, _2.11
[error] org.json4s:json4s-ast _2.10, _2.11
[error] org.apache.spark:spark-network-shuffle _2.10, _2.11
[error] com.twitter:chill _2.10, _2.11
[error] org.json4s:json4s-jackson _2.10, _2.11
[error] com.fasterxml.jackson.module:jackson-module-scala _2.10, _2.11
[error] org.json4s:json4s-core _2.10, _2.11
[error] org.apache.spark:spark-unsafe _2.10, _2.11
[error] org.apache.spark:spark-core _2.10, _2.11
[error] org.apache.spark:spark-network-common _2.10, _2.11
However, I can't troubleshoot what causes this.. This is my full build.sbt:
import sbt._
import Keys._
import scala._
lazy val root = (project in file("."))
.settings(
name := "example-hcl-spark-scala-graphx-bitcointransaction",
version := "0.1"
)
.configs( IntegrationTest )
.settings( Defaults.itSettings : _*)
scalacOptions += "-target:jvm-1.7"
crossScalaVersions := Seq("2.11.8")
resolvers += Resolver.mavenLocal
fork := true
jacoco.settings
itJacoco.settings
assemblyJarName in assembly := "example-hcl-spark-scala-graphx-bitcointransaction.jar"
libraryDependencies += "com.github.zuinnote" % "hadoopcryptoledger-fileformat" % "1.0.7" % "compile"
libraryDependencies += "org.apache.spark" %% "spark-core" % "1.5.0" % "provided"
libraryDependencies += "org.apache.spark" %% "spark-graphx" % "1.5.0" % "provided"
libraryDependencies += "org.apache.hadoop" % "hadoop-client" % "2.7.0" % "provided"
libraryDependencies += "javax.servlet" % "javax.servlet-api" % "3.0.1" % "it"
libraryDependencies += "org.apache.hadoop" % "hadoop-common" % "2.7.0" % "it" classifier "" classifier "tests"
libraryDependencies += "org.apache.hadoop" % "hadoop-hdfs" % "2.7.0" % "it" classifier "" classifier "tests"
libraryDependencies += "org.apache.hadoop" % "hadoop-minicluster" % "2.7.0" % "it"
libraryDependencies += "org.apache.spark" % "spark-sql_2.11" % "2.2.0" % "provided"
libraryDependencies += "org.scalatest" %% "scalatest" % "3.0.1" % "test,it"
libraryDependencies += "graphframes" % "graphframes" % "0.5.0-spark2.1-s_2.11"
Can anyone pinpoint which dependency is based on scala 2.10 causing the build to fail?
I found out what the problem was. Apparently, if you use:
libraryDependencies += "org.apache.spark" %% "spark-core" % "1.5.0" % "provided"
It uses the 2.10 version by default. It all worked once I changed the dependencies of spark core and spark graphx to:
libraryDependencies += "org.apache.spark" % "spark-core_2.11" % "2.2.0"
libraryDependencies += "org.apache.spark" % "spark-graphx_2.11" % "2.2.0" % "provided"

Spray microservice assembly deduplicate

I'm using this template to develop a microservice:
http://www.typesafe.com/activator/template/activator-service-container-tutorial
My sbt file is like this:
import sbt._
import Keys._
name := "activator-service-container-tutorial"
version := "1.0.1"
scalaVersion := "2.11.6"
crossScalaVersions := Seq("2.10.5", "2.11.6")
resolvers += "Scalaz Bintray Repo" at "https://dl.bintray.com/scalaz/releases"
libraryDependencies ++= {
val containerVersion = "1.0.1"
val configVersion = "1.2.1"
val akkaVersion = "2.3.9"
val liftVersion = "2.6.2"
val sprayVersion = "1.3.3"
Seq(
"com.github.vonnagy" %% "service-container" % containerVersion,
"com.github.vonnagy" %% "service-container-metrics-reporting" % containerVersion,
"com.typesafe" % "config" % configVersion,
"com.typesafe.akka" %% "akka-actor" % akkaVersion exclude ("org.scala-lang" , "scala-library"),
"com.typesafe.akka" %% "akka-slf4j" % akkaVersion exclude ("org.slf4j", "slf4j-api") exclude ("org.scala-lang" , "scala-library"),
"ch.qos.logback" % "logback-classic" % "1.1.3",
"io.spray" %% "spray-can" % sprayVersion,
"io.spray" %% "spray-routing" % sprayVersion,
"net.liftweb" %% "lift-json" % liftVersion,
"com.typesafe.akka" %% "akka-testkit" % akkaVersion % "test",
"io.spray" %% "spray-testkit" % sprayVersion % "test",
"junit" % "junit" % "4.12" % "test",
"org.scalaz.stream" %% "scalaz-stream" % "0.7a" % "test",
"org.specs2" %% "specs2-core" % "3.5" % "test",
"org.specs2" %% "specs2-mock" % "3.5" % "test",
"com.twitter" %% "finagle-http" % "6.25.0",
"com.twitter" %% "bijection-util" % "0.7.2"
)
}
scalacOptions ++= Seq(
"-unchecked",
"-deprecation",
"-Xlint",
"-Ywarn-dead-code",
"-language:_",
"-target:jvm-1.7",
"-encoding", "UTF-8"
)
crossPaths := false
parallelExecution in Test := false
assemblyJarName in assembly := "santo.jar"
mainClass in assembly := Some("Service")
The project compiles fine!
But when I run assembly, the terminal show me this:
[error] (*:assembly) deduplicate: different file contents found in the following:
[error] /path/.ivy2/cache/io.dropwizard.metrics/metrics-core/bundles/metrics-core-3.1.1.jar:com/codahale/metrics/ConsoleReporter$1.class
[error] /path/.ivy2/cache/com.codahale.metrics/metrics-core/bundles/metrics-core-3.0.1.jar:com/codahale/metrics/ConsoleReporter$1.class
What options do I have to fix it?
Thanks
The issue as it seems transitive dependency of the dependency is resulting with two different versions of metrics-core. The best thing to do would be to used the right library dependency so that you end up with a single version of this library. Please use https://github.com/jrudolph/sbt-dependency-graph , if it is difficult to figure out dependencies.
If it is not possible to get to a single version then you would most likely to go down exclude route . I assume, this only work, if there is compatibility between the all required versions.

Phantom Cassandra driver dependency error

I would like to use the phantom cassandra wrapper in my scala project, but when I try to update my sbt build I get a dependency error.
My build.sbt:
version := "1.0"
scalaVersion := "2.11.2"
seq(lsSettings :_*)
libraryDependencies ++= Seq(
"org.clapper" %% "grizzled-scala" % "1.2",
"commons-io" % "commons-io" % "2.4",
"org.rauschig" % "jarchivelib" % "0.6.0",
"com.google.code.findbugs" % "jsr305" % "3.0.0",
"org.scalatest" % "scalatest_2.11" % "2.2.0" % "test",
"com.github.nscala-time" %% "nscala-time" % "1.2.0",
"org.json4s" %% "json4s-native" % "3.2.10",
"org.scala-lang" % "scala-library" % "2.11.2",
"com.websudos" % "phantom-dsl_2.10" % "1.2.0"
)
resolvers += "grizzled-scala-resolver-0" at "https://oss.sonatype.org/content/repositories/releases"
resolvers += "Typesafe repository releases" at "http://repo.typesafe.com/typesafe/releases/"
I get the following error:
[warn] Note: Some unresolved dependencies have extra attributes. Check that these dependencies exist with the requested attributes.
[warn] com.typesafe.sbt:sbt-pgp:0.8.1 (sbtVersion=0.13, scalaVersion=2.10)
Don't know what I have to do...
edit:
Answer from https://github.com/websudosuk/phantom/issues/119
error is on the pom side, new version 1.2.1 coming soon...
Answer from https://github.com/websudosuk/phantom/issues/119
error is on the pom side, new version 1.2.1 coming soon...

play-json breaks sbt build

Suddenly as of today my project has stopped compiling successfuly. Upon further investigation I've found out the reason is play-json library that I include in dependencies.
Here's my build.sbt:
name := """project-name"""
version := "1.0"
scalaVersion := "2.10.2"
libraryDependencies ++= Seq(
"com.typesafe.akka" %% "akka-actor" % "2.2.1",
"com.typesafe.akka" %% "akka-testkit" % "2.2.1",
"org.scalatest" %% "scalatest" % "1.9.1" % "test",
"org.bouncycastle" % "bcprov-jdk16" % "1.46",
"com.sun.mail" % "javax.mail" % "1.5.1",
"com.typesafe.slick" %% "slick" % "2.0.1",
"org.postgresql" % "postgresql" % "9.3-1101-jdbc41",
"org.slf4j" % "slf4j-nop" % "1.6.4",
"com.drewnoakes" % "metadata-extractor" % "2.6.2",
"com.typesafe.play" %% "play-json" % "2.2.2"
)
resolvers += "Typesafe repository" at "http://repo.typesafe.com/typesafe/releases/"
If I try to create a new project in activator with all the lines except "com.typesafe.play" %% "play-json" % "2.2.2" then it compiles successfully. But once I add play-json I get the folloing error:
[error] References to undefined settings:
[error]
[error] *:playCommonClassloader from echo:run
[error]
[error] docs:managedClasspath from echo:run
[error]
[error] *:playReloaderClassloader from echo:run
[error]
[error] echo:playVersion from echo:echoTracePlayVersion
[error]
[error] *:playRunHooks from echo:playRunHooks
[error] Did you mean echo:playRunHooks ?
[error]
And I keep getting this error even if I remove play-json line. Why is it so? What should I do to fix it?