I am trying to upgrade my Scala version from 2.11.8 to 2.12.10. I made following changes in my sbt file.
"org.apache.spark" %% "spark-core" % "2.4.7" % "provided",
"org.apache.spark" %% "spark-sql" % "2.4.7" % "provided",
"com.holdenkarau" %% "spark-testing-base" % "3.1.2_1.1.0" % "test"
when I am build the sbt file, I am getting following error
[error] Modules were resolved with conflicting cross-version suffixes in ProjectRef(uri("file:/Users/user/IdeaProjects/project/"), "root"):
[error] io.reactivex:rxscala _2.12, _2.11
Tried following possible ways. but no luck.
("io.reactivex" % "rxscala_2.12" % "0.27.0").force().exclude("io.reactivex","rxscala_2.11")
2.Removed Scala verion=2.11.8 from File->project structure->global libraries.
Any Help will be very useful.
It would be best if you could post your entire build.sbt file so it could be reproduced, but as a starting point, the dependency spark-testing-base is pointing to the wrong Spark version. From the documentation:
So you include com.holdenkarau.spark-testing-base [spark_version]_1.0.0 and extend
Based on the information you have provided you should be using:
"com.holdenkarau" %% "spark-testing-base" % "2.4.7_1.1.0" % "test"
Related
I upgraded my test dependencies for my play project and now I get this problem:
object scalacheck is not a member of package org.scalatestplus
[error] import org.scalatestplus.scalacheck.ScalaCheckDrivenPropertyChecks
These are my test dependencies:
"org.scalatest" %% "scalatest" % "3.2.3" % Test,
"org.scalamock" %% "scalamock" % "4.2.0" % "test",
"com.github.alexarchambault" %% "scalacheck-shapeless_1.14" % "1.2.1" % "test",
"org.scalatestplus.play" %% "scalatestplus-play" % "3.1.2" % "test, it",
Do I have some incompatible versioning going on here?
You don't have the right dependency:
"org.scalatestplus" %% "scalacheck-1-15" % "3.2.3.0"
Since you are using scalatest version 3.2.3, you should use scalacheck version 3.2.3.X, for compatability.
Here is its maven location, and this is its github repo.
Scala 2.13 Code run at Scastie.
Scala 2.12 Code run at Scastie.
This question already has an answer here:
sbt unresolved dependency for spark-cassandra-connector 2.0.2
(1 answer)
Closed 5 years ago.
I am trying to set up my first Scala project with IntelliJ Idea on Ubuntu 16.04. I need the Spark library and I think I have installed correctly in my computer, however I am not able to refer it in the project dependencies. In particular, I have added the following code in my build.sbt:
libraryDependencies ++= Seq(
"org.apache.spark" % "spark-core" % "2.1.1",
"org.apache.spark" % "spark-sql" % "2.1.1")
However sbt complains about not finding the correct packages (Unresolved Dependencies error, org.apache.spark#spark-core;2.1.1: not found and org.apache.spark#spark-sql;2.1.1: not found):
I think that the versions of the packages are incorrect (I copied the previous code from the web, just to try).
How can I determine the correct packages versions?
If you use % you have to define the exact version as
libraryDependencies ++= Seq(
"org.apache.spark" % "spark-core_2.10" % "2.1.1",
"org.apache.spark" % "spark-sql_2.10" % "2.1.1")
And if you don't want to define the version and let sbt take the correct version then you need to define %% as
libraryDependencies ++= Seq(
"org.apache.spark" %% "spark-core" % "2.1.1",
"org.apache.spark" %% "spark-sql" % "2.1.1")
you can check of installed version by doing
spark-submit --version
And by going to maven dependency
when I try to run a Main class with sbt, I get this error. What am I missing?
Error:scalac: missing or invalid dependency detected while loading class file 'LDA.class'.
Could not access type Logging in package org.apache.spark,
because it (or its dependencies) are missing. Check your build definition for missing or conflicting dependencies. (Re-run with `-Ylog-classpath` to see the problematic classpath.)
A full rebuild may help if 'LDA.class' was compiled against an incompatible version of org.apache.spark.
My build.sbt looks like this:
"org.apache.spark" %% "spark-core" % "2.0.1" % Provided,
"org.apache.spark" % "spark-mllib_2.11" % "1.3.0"
You are trying to run the old sparkMllib on new Spark Core. i.e. your version of mllib and spark core are totally different.
Try using this:
"org.apache.spark" %% "spark-core_2.11" % "2.0.1",
"org.apache.spark" %% "spark-mllib_2.11" % "2.0.1"
Thant might solve your problem !
I'm trying to add parboiled2 as a dependency to my project, and follow the Calculator example but it conflicts with spray.
My current build.sbt file includes:
"io.spray" %% "spray-json" % "1.3.1" withSources() withJavadoc(),
"io.spray" %% "spray-can" % sprayV withSources() withJavadoc(),
"io.spray" %% "spray-routing" % sprayV withSources() withJavadoc(),
"io.spray" %% "spray-testkit" % sprayV % "test" withSources() withJavadoc(),
When I add
"org.parboiled" %% "parboiled" % "2.0.1" withSources() withJavadoc(),
I get
[error] Modules were resolved with conflicting cross-version suffixes in {file:/blar/blar}blar-blar:
[error] com.chuusai:shapeless _2.10.4, _2.10
[error] org.scalamacros:quasiquotes _2.10, _2.10.3
java.lang.RuntimeException: Conflicting cross-version suffixes in: com.chuusai:shapeless, org.scalamacros:quasiquotes
So did some googling, and as usual people suggest using the exclude directive of SBT (which I don't believe makes logical sense as it will inevitably result in problems like ClassNotFoundException and NoSuchMethodError). I tried it nevertheless:
"org.parboiled" %% "parboiled" % "2.0.1" withSources() withJavadoc()
exclude("com.chuusai", "shapeless_2.10.4") exclude("org.scalamacros", "quasiquotes_2.10")
And surprise surprise when I try to run assembly I get
[error] java.lang.ClassNotFoundException: scala.quasiquotes.QuasiquoteCompat$
I also get a load more errors:
[error] bad symbolic reference
. A signature in RuleDSLBasics.class refers to term internal
[error] in package scala.reflect which is not available.
[error] It may be completely missing from the current classpath, or the version on
[error] the classpath might be incompatible with the version used when compiling RuleDSLBasics.class.
[error] error while loading RuleDSLBasics, Missing dependency 'bad symbolic reference. A signature in RuleDSLBasics.class refers to term annotations
[error] in value scala.reflect.internal which is not available.
[error] It may be completely missing from the current classpath, or the version on
[error] the classpath might be incompatible with the version used when compiling RuleDSLBasics.class.', required by ...
If I comment out the Spray dependencies (and code) I can successfully assembly a jar with the Calculator example.
This is a major show stopper for me using parboiled2. Is there some way other than exclude directives in order to make this stuff work? Does anyone have a working build file with both Spray and parboiled2? If someone has managed to get around dependency hell with onejar or ProGuard I'd love it if they could explain just how.
UPDATE:
My build file:
resolvers ++= Seq(
"Concurrent Maven Repo" at "http://conjars.org/repo",
"spray repo" at "http://repo.spray.io"
)
val akkaV = "2.3.6"
val sprayV = "1.3.2"
libraryDependencies ++= Seq(
"org.parboiled" %% "parboiled" % "2.0.1" withSources() withJavadoc(),
// Causes org.scalamacros:quasiquotes _2.10, _2.10.3 cross-version problem
"io.spray" %% "spray-testkit" % sprayV % "test" withSources() withJavadoc(),
// Causes com.chuusai:shapeless _2.10.4, _2.10 cross-version problem
"io.spray" %% "spray-routing" % sprayV withSources() withJavadoc()
)
scalaVersion := "2.10.4"
javaOptions ++= Seq("-target", "1.8", "-source", "1.8")
organization := domain + "." + companyName
Plugins file:
addSbtPlugin("com.eed3si9n" % "sbt-assembly" % "0.12.0")
addSbtPlugin("com.github.mpeltonen" % "sbt-idea" % "1.6.0")
Scala 2.11
It seems "simply" moving 2.11 may solve this, but this isn't always that easy if one has some other libraries that are not 2.11 ready. Nevertheless I tried it, and the second I added parboiled2 as a dependency it broke my build again, sigh, here is the new problem: Parboiled2 causes "missing or invalid dependency detected while loading class file 'Prepender.class'"
The same answer as for Scala 2.11 should work here as well: replace spray-routing with spray-routing-shapeless2.
I would have to look at the whole build.sbt file to understand what is happening but I can use the dependencies you mention without problem with sbt 0.13.7 and scala 2.11.
What sbt version? What Scala version? 2.10 it seems. Why? Are you stuck with those versions?
For reference, here are the build.sbt, plugins.sbt and build.properties files I used to test this scenario.
https://gist.github.com/fedesilva/ddd28a5b592eb88f41ae
Hope that helps
I've a problem building a module into a project with sbt 0.13.5 and scala 2.9.3, I have the dependency defined to
"com.fasterxml.jackson.module" %% "jackson-module-scala" % "2.2.3" % "provided"
and even when the scalaVersion is explicitly set, throws this error:
[error] Modules were resolved with conflicting cross-version suffixes in {file:/Users/me/Code/project/}module:
[error] com.fasterxml.jackson.module:jackson-module-scala _2.9.2, _2.9.3
This is a provided dependency, that works on other module of the same project with the same resolvers and the same explicit scala version, the only difference there is in the additional dependencies, failing project has
"bouncycastle" % "bcprov-jdk16" % "140" % "provided",
"com.jolbox" % "bonecp" % "0.7.1.RELEASE" % "provided",
"mysql" % "mysql-connector-java" % "5.1.18" % "provided",
"org.liquibase" % "liquibase-core" % "3.0.5" % "provided",
"org.jdbi" % "jdbi" % "2.51" % "provided",
"javax.mail" % "mail" % "1.4" % "provided",
"com.twitter" % "finagle-redis" % "6.6.2" % "provided",
"com.twitter" % "finatra" % "1.3.9" % "provided"
Any idea on how to fix this? It appeared when updated to sbt 0.13, in 0.12 works ok.
Ok, I see what the problem is, finatra 1.3.9 is compiled with scala 2.9.2 and there's a dependency on fasterxml, Tried to fix by cross version mapping but it's a nighmare of dependencies, so, upgrading all dependencies to 2.10 is working so far.