As soon as I bring in a single dependency (using SBT), I see warnings about multiple dependencies.
scalaVersion := "2.10.4"
libraryDependencies ++= Seq(
"org.apache.spark" %% "spark-core" % "1.5.0" % "provided"
)
Warnings:
[warn] Multiple dependencies with the same organization/name but different versions. To avoid conflict, pick one version:
[warn] * org.scala-lang:scala-compiler:(2.10.0, 2.10.4)
[warn] * org.apache.commons:commons-lang3:(3.3.2, 3.0)
[warn] * org.slf4j:slf4j-api:(1.7.10, 1.7.2)
Normally I'm very pedantic about warnings. I want to know why they are there and what I should do to eliminate them. If you let warnings pile up, you quickly have a signal to noise problem.
But how would a novice Scala dev (aka me) know what version to favor?
I'm not really asking how to suppress these warnings, so much as to understand the implications of choosing one version over the other. Seems to me that the source of these warnings is within spark.core, no? How am I to know how to respond?
As soon as I add more dependencies, these warnings pile up and the possibility of a real conflict/problem go up.
I've spent the day trying to find the magic internet search keywords to figure out what to do, but all I'm finding is "how", not "why", if that makes sense.
Thanks.
UPDATE:
Based on this thread, I downgraded from 0.13.8 to 0.13.7. It did get rid of all the noise, though I'm not sure this is really an "answer". But at least I'm able to add all my dependencies without a boatload of warnings, and now sbt-assembly is working as well.
My reading is that minor versions should be forward compatible, so for slf4j it shouldn't matter. The same with the Scala version, moving from 2.10.0 to 2.10.4 shouldn't be a problem (Scala guarantees binary compatibility between minor versions). The only thing that might be a problem here is Apache Commons.
You can use the sbt-dependency-graph plugin to find out more precisely which libraries depend on which versions. Also I think if you run sbt evicted you get more information about conflicting versions.
Related
I keep seeing these messages when I launch sbt shell for my play application:
[warn] * com.typesafe.akka:akka-actor_2.11:2.5.21 is selected over {2.3.13, 2.4.20}
[warn] +- com.typesafe.akka:akka-slf4j_2.11:2.5.21 () (depends on 2.5.21)
[warn] +- com.typesafe.play:play_2.11:2.6.23 () (depends on 2.5.21)
[warn] +- com.typesafe.akka:akka-stream_2.11:2.5.21 () (depends on 2.5.21)
[warn] +- com.typesafe.akka:akka-parsing_2.11:10.0.15 () (depends on 2.4.20)
How do i fix this? Do I need to explicitly set the akka version so it doesn't pick 2.5.21?
I'd recommend to hook up the sbt-dependency-graph sbt plugin and use the following commands to figure out which components depend on which version of akka-actor
sbt <project>/dependencyTree
sbt "<project>/whatDependsOn <organization> <module> <revision>"
sbt <project>/evicted
After coming up with the list of component dependencies, you have a few choices:
Make sure you use the versions of the components (like play) that
require the same version of akka-actor and akka in general. I usually do it by checking POM files for the libraries in question at https://mvnrepository.com/
Force sbt to use a particular version of akka (via dependencyOverrides). It
is generally safe to do for different patch versions (as in
major.minor.patch) for libraries that use semantic versioning
(https://semver.org/). But it may cause problems for different minor and
especially major versions if they are not backwards compatible.
Ignore the conflicts and warnings if they aren't causing problems.
If you use sbt-assembly to create Fat jars, you can use shading to
allow multiple versions of akka to live in the same application via
moving one of them to a different package. It is not always possible though (e.g. when a library loads a class dynamically by name)
write a custom class loader to dynamically load different versions of a library into separate protection domains. It is not trivial and requires quite a bit of code. I wouldn't recommend it for akka.
What you're getting is an eviction warning by sbt, you can check the documentation at this link.
What you could do is to override the dependency used on the sbt file, to avoid runtime errors.
For example:
dependencyOverrides += "com.typesafe.akka" %% "akka-actor" % "2.5.21"
In this way you're specifying to sbt which version of the library to use.
You can check also the sbt's plugin dependency-graphto understand all the dependencies inside your project.
When playing with something in Scala, I typically spend a bunch of time trying combinations of dependency versions, Scala versions, %% vs %, etc. And when it starts working, I am not quite sure why, or for how long...
It would be great if someone could explain the Scala ecosystem's way(s) of dealing with versions of sbt, scala, and libraries. Or perhaps point me to some documentation.
I struggled with this extensively when i started out. These days i start every project with a boiler-plate build.sbt with just scalaVersion and whatever sbt is currently on my machine:
organization := "foo"
version := "0.1"
scalaVersion := "2.10.4"
Pick the latest 2.10 or 2.11, dependening on your need. Most libraries of note are cross-published into both.
Now, as you find libraries you want to use, head over to http://mvnrepository.com/ and search for them there. Look for a _2.10 or _2.11 postfix (depending on your version). If it has neither, you are likely fine.
Once you find your library and the version you want, mavenrepository even provides you the sbt link you need to use in its sbt tab like this:
libraryDependencies += "com.typesafe.play" % "play-test_2.10" % "2.4.0-M3"
And from there you can even explore the dependencies that library will bring along with it. This should cover most of your day to day needs.
I have created a libs folder and placed scalascriptengine-1.3.9-2.11.0.jar in there. After that, I right-clicked on the .jar and selected Add Library.
Then, I created Test.scala:
import java.io.File
import com.googlecode.scalascriptengine.ScalaScriptEngine
object Test {
def main(args: Array[String]): Unit = {
val sourceDir = new File("examples/folder")
val sse = ScalaScriptEngine.onChangeRefresh(sourceDir)
}
}
It correctly recognized ScalaScriptEngine, or at least it did not give any warnings or errors. But it did not compile.
According to the library page I edited my build.sbt:
name := "ScalaScriptEngineTest"
version := "1.0"
libraryDependencies += "com.googlecode.scalascriptengine" %% "scalascriptengine" % "1.3.10"
libraryDependencies += "org.scala-lang" % "scala-compiler" % "2.11.1"
But upon refreshing, I get this: http://pastebin.com/GdirttUJ
What am I missing? I am learning scala and it is the first time I am trying to add a library to IntelliJ Idea...
Short answer:
Change the two dependency entries in your build.sbt as follows:
libraryDependencies +=
"com.googlecode.scalascriptengine" % "scalascriptengine" % "1.3.9-2.10.3"
libraryDependencies += "org.scala-lang" % "scala-compiler" % "2.10.4"
Notice I didn't just change the versions -- I replaced your %% with a single %.
This results in you using a slightly older version of ScalaScriptEngine, and I don't know if that will cause any problems for you.
If you're using sbt build dependencies you don't need to be manually placing jars anywhere.
Explanation:
From your log I see that sbt is looking for the ScalaScriptEngine for Scala 2.10. In fact, it's pretty clear that you're running Scala 2.10.4, even though your sbt file expresses a dependency on the 2.11 compiler, which in fact is consistent with the instructions for using ScalaScriptEngine.
On line 23 of the log you can see exactly where it's looking. If you point your browser part way down that path you'll see that there is a version for Scala 2.11 and another directory, scalascriptengine, without a version qualifier. If you dive down the latter, you'll see it's where they keep all the old versions. There isn't a ScalaScriptEngine 1.3.10 (the one you asked for) compiled for Scala 2.10, so your options seem to be to upgrade to Scala 2.11 (which I don't think currently works if you want to use IntelliJ Idea's tight integration with sbt), or you can use ScalaScriptEngine 1.3.9.
You have basically the same problem with your Scala compiler dependency -- it needs the be the Scala version you're using.
I've confirmed the above solution with Scala 2.10.4. I'm playing it a little fast and loose because there isn't a pre compiled version for 2.10.4, and I gambled that the 2.10.3 build will probably work.
Alternatives:
There may be a cleaner way to solve this, but the way the repository is organized makes me doubt it.
You could build the version of your choice with the compiler of your choice, or persuade the ScalaScriptEngine community to do it for you and put it in The Central Repository, but my guess is that 1.3.10 won't build with anything lower than Scala 2.11.
Finally, if you do want to download jars by hand, you may want to read the "Unmanaged dependencies" section of the sbt documentation. Actually, if you're going to use sbt, just read the whole thing a few times.
I want to add a Java library (e.g. Apache PDFBox) to an sbt project.
This is the Ivy dependency:
dependency org="org.apache.pdfbox" name="pdfbox" rev="1.8.2"
I first tried to do the following:
resolvers += "Sonatype releases" at "http://oss.sonatype.org/content/repositories/releases/"
libraryDependencies += "org.apache.pdfbox" %% "pdfbox" % "1.8.2"
But it gives me errors of the type
[warn] ==== public: tried [warn]
http://repo1.maven.org/maven2/org/apache/pdfbox/pdfbox_2.10/1.8.2/pdfbox_2.10-1.8.2.pom
So I understand that with this syntax I can just manage Scala dependencies. I am sure that there is a way to manage Java dependencies, but how?
I tried to search in Google for "sbt add java dependencies" but did not find (recognize) a relevant result.
You should replace the %% (double percent) with single one.
libraryDependencies += "org.apache.pdfbox" % "pdfbox" % "1.8.2"
The double-percent is a convenience operator, and causes adding the _+scalaVersion postfix inside the path, which is _2.10 in your case. Single percent should fix the problem.
Short answer:
Use
libraryDependencies += "org.apache.pdfbox" % "pdfbox" % "1.8.2"
For java libraries, and
libraryDependencies += "org.scalactic" %% "scalactic" % "3.0.8"
For Scala libraries, where the difference is the double % for the scala library.
Long answer:
Scala is not backward compatible across major version, so a library compiled for scala 2.12.x cannot be used by a project written in scala 2.13.x.
So when writing a scala library, you will need to compile and publish it one time per scala major version you would like to support. When using a library in a project, you would then have to pick the version compiled for the same Scala major version as your are using. Doing this manually would be cumbersome, so SBT has built in support for it.
When publishing a library, you can add the crossScalaVersions key to SBT like
crossScalaVersions := Seq( "2.10.6", "2.11.11", "2.12.3" )
And then publish with sbt +publish. This will make SBT build and publish a version of the library for both scala 2.10.6, 2.11.11 and 2.12.3. Note that the minor number is in-relevant, when it comes to compatibility for libraries. The published libraries, will have the name suffixed with _2.10, _2.11 and _2.12 to show what scala version it is for. An alternative to using the SBT build in support for this, is to use the experimental plugin sbt-projectmatrix as this gives a lot more options, and often faster builds.
When using a library sbt can also help your use the one compiled for the correct scala version, and thats where %% comes into play. When specifying a library without the _ suffix in the name, but instead use %%, then sbt will fill in suffix matching the Scala major version your use, and thereby fetch the correct version of the library.
I am cross building a scala project with sbt 12.1.
crossScalaVersions := Seq("2.9.2", "2.10.0")
However, it can't find the dependencies because they are named _2.10 not _2.10.0. It seems like it's regular to name your library 2.10 instead of 2.10.0 with the exception of scala-language and scala-compiler. For example, scalaz is not found at http://repo1.maven.org/maven2/org/scalaz/scalaz-core_2.10.0/6.0.4/scalaz-core_2.10.0-6.0.4.pom but at http://repo1.maven.org/maven2/org/scalaz/scalaz-core_2.10/6.0.4/scalaz-core_2.10-6.0.4.pom.
Is there an easy way to handle this without writing custom rules for all of my dependencies?
The actual build.sbt is available online.
Since 2.10.x releases are binary compatible between each other, libraries need to be built only with one version of scala library - and they can (and must) drop the .0 part (if you publish with sbt, it is done automatically). When the maintainer of a library releases a library with _2.10.0 tag, it's a mistake and you should consider filing a bug.
By the way, I looked on your build.sbt - running +compile on it works for me (sbt 0.12.1). Do you experience some errors?
To get the Scala version incorporated into the artifact name in the Scala way, you specify the dependency with the %% operator:
libraryDependencies += "io.backchat.jerkson" %% "jerkson" % "0.7.0"
When the exact match is not available, you can specify the Scala version explicitly (but remember compatibility only exists across patch releases of a given major/minor release of Scala):
libraryDependencies += "io.backchat.jerkson" % "jerkson_2.9.2" % "0.7.0"