How do i fix these dependency warnings - scala

I keep seeing these messages when I launch sbt shell for my play application:
[warn] * com.typesafe.akka:akka-actor_2.11:2.5.21 is selected over {2.3.13, 2.4.20}
[warn] +- com.typesafe.akka:akka-slf4j_2.11:2.5.21 () (depends on 2.5.21)
[warn] +- com.typesafe.play:play_2.11:2.6.23 () (depends on 2.5.21)
[warn] +- com.typesafe.akka:akka-stream_2.11:2.5.21 () (depends on 2.5.21)
[warn] +- com.typesafe.akka:akka-parsing_2.11:10.0.15 () (depends on 2.4.20)
How do i fix this? Do I need to explicitly set the akka version so it doesn't pick 2.5.21?

I'd recommend to hook up the sbt-dependency-graph sbt plugin and use the following commands to figure out which components depend on which version of akka-actor
sbt <project>/dependencyTree
sbt "<project>/whatDependsOn <organization> <module> <revision>"
sbt <project>/evicted
After coming up with the list of component dependencies, you have a few choices:
Make sure you use the versions of the components (like play) that
require the same version of akka-actor and akka in general. I usually do it by checking POM files for the libraries in question at https://mvnrepository.com/
Force sbt to use a particular version of akka (via dependencyOverrides). It
is generally safe to do for different patch versions (as in
major.minor.patch) for libraries that use semantic versioning
(https://semver.org/). But it may cause problems for different minor and
especially major versions if they are not backwards compatible.
Ignore the conflicts and warnings if they aren't causing problems.
If you use sbt-assembly to create Fat jars, you can use shading to
allow multiple versions of akka to live in the same application via
moving one of them to a different package. It is not always possible though (e.g. when a library loads a class dynamically by name)
write a custom class loader to dynamically load different versions of a library into separate protection domains. It is not trivial and requires quite a bit of code. I wouldn't recommend it for akka.

What you're getting is an eviction warning by sbt, you can check the documentation at this link.
What you could do is to override the dependency used on the sbt file, to avoid runtime errors.
For example:
dependencyOverrides += "com.typesafe.akka" %% "akka-actor" % "2.5.21"
In this way you're specifying to sbt which version of the library to use.
You can check also the sbt's plugin dependency-graphto understand all the dependencies inside your project.

Related

SBT plugin to forbid use of auto-imported dependencies

I have this line of code in my build.sbt file:
libraryDependencies ++= Seq("com.foo" %% "lib" % "1.2.3")
Imagine that this library depends on "com.bar.lib" lib. Now in my code I can import com.bar.lib._ and it'll work. But I don't want this to compile, so maybe there is SBT plugin out there just for this purpose?
One of libraries I'm using depends on old cats version. I spent really long time to understand why mapN method not works... I just never imported a newer version of cats in the subproject.
SBT offers the intransitive and exclude features to deal with issues like this, as #earldouglas points out. See: https://www.scala-sbt.org/1.x/docs/Library-Management.html
You replied:
I tried to do so, but intransitive() don't import transitive dependencies (so I need to import all of them by hand to make it compile).
Yes, that is what it is for
What I want is something that will warn me about using libraries not directly imported in SBT file.
So you want transitive dependencies on your classpath, but you want the compiler to reject uses of transitive classes in your project code while allowing them in library code?
That is not a sensible approach: at runtime, these transitive dependencies will be on the classpath. The JVM classpath does not distinguish between different kinds of dependencies; such distinction only exists in SBT at build time.
You would be much better served by either
including a newer version of the cats library, overriding the transitive dep or
excluding the transitively included cats library, if it is broken.
However, I think you probably could achieve what you want by setting different dependencies at different build stages:
at Compile stage, include the dependency with intransitive. Your code should compile against your direct dependencies, but fail if you referenced any transitive dependencies
at Runtime stage, include the dependency with its transitive deps
the SBT code might look like this (untested):
(libraryDependencies in Compile) ++= Seq("com.foo" %% "lib" % "1.2.3" intransitive())
(libraryDependencies in Runtime) ++= Seq("com.foo" %% "lib" % "1.2.3")

SBT Scala Version Warning

I started seeing the following warning messages when doing sbt build:
[warn] Scala version was updated by one of library dependencies:
[warn] * org.scala-lang:scala-library:(2.11.1, 2.11.7, 2.11.2, 2.11.6, 2.11.5, 2.11.0) -> 2.11.8
[warn] To force scalaVersion, add the following:
[warn] ivyScala := ivyScala.value map { _.copy(overrideScalaVersion = true) }
[warn] Run 'evicted' to see detailed eviction warnings
Why is this? I'm on sbt 0.13.11!
Did something change with the sbt version? I guess I was on 0.13.5 before and did not see any warning of this sort!
It means your project defines a Scala version smaller than 2.11.8, but sbt decided to increase it to 2.11.8 because one of the dependencies uses that version. It shouldn't matter because these versions are binary compatible. I guess it warns you because you might have deliberately used a smaller version because of some regression, and in that case you would need to force a smaller Scala version using the explanation. In other cases, just ignore the warning or increase your project's Scala version to the latest (2.11.8).
I'm not sure if adding / changing Scala version would be benign. So I just added the following to my build.sbt to get rid of the warnings:
evictionWarningOptions in update :=
EvictionWarningOptions.default
.withWarnTransitiveEvictions(false)
.withWarnDirectEvictions(false)
.withWarnScalaVersionEviction(false)

How do you determine the proper dependency version in a Scala project?

As soon as I bring in a single dependency (using SBT), I see warnings about multiple dependencies.
scalaVersion := "2.10.4"
libraryDependencies ++= Seq(
"org.apache.spark" %% "spark-core" % "1.5.0" % "provided"
)
Warnings:
[warn] Multiple dependencies with the same organization/name but different versions. To avoid conflict, pick one version:
[warn] * org.scala-lang:scala-compiler:(2.10.0, 2.10.4)
[warn] * org.apache.commons:commons-lang3:(3.3.2, 3.0)
[warn] * org.slf4j:slf4j-api:(1.7.10, 1.7.2)
Normally I'm very pedantic about warnings. I want to know why they are there and what I should do to eliminate them. If you let warnings pile up, you quickly have a signal to noise problem.
But how would a novice Scala dev (aka me) know what version to favor?
I'm not really asking how to suppress these warnings, so much as to understand the implications of choosing one version over the other. Seems to me that the source of these warnings is within spark.core, no? How am I to know how to respond?
As soon as I add more dependencies, these warnings pile up and the possibility of a real conflict/problem go up.
I've spent the day trying to find the magic internet search keywords to figure out what to do, but all I'm finding is "how", not "why", if that makes sense.
Thanks.
UPDATE:
Based on this thread, I downgraded from 0.13.8 to 0.13.7. It did get rid of all the noise, though I'm not sure this is really an "answer". But at least I'm able to add all my dependencies without a boatload of warnings, and now sbt-assembly is working as well.
My reading is that minor versions should be forward compatible, so for slf4j it shouldn't matter. The same with the Scala version, moving from 2.10.0 to 2.10.4 shouldn't be a problem (Scala guarantees binary compatibility between minor versions). The only thing that might be a problem here is Apache Commons.
You can use the sbt-dependency-graph plugin to find out more precisely which libraries depend on which versions. Also I think if you run sbt evicted you get more information about conflicting versions.

How to download sbt plugin source jars in a common sbt project?

It's nature to download the dependencies source jars in a sbt project, using sbt gen-idea or idea's autoimport feature
but how can I download the sbt plugin source jars which I declared in project/plugins.sbt
IDEA
IDEA should already do this*.
When you import a project and have Download sbt sources checked in Preferences > Build, Execution, Deployment > Build Tools > sbt then it will run the updateSbtClassifiers sbt task.
* Read the Troubleshooting section as there are several known issues with this.
sbt
As mentioned above, you can run the following sbt task:
sbt> updateSbtClassifiers
This command is somewhat special in that it knows to resolve the classifiers for the plugins. It also uses settings that are scoped to that task. See the caveat about sbt/sbt#3432.
Longer way
It is important to understand that sbt is recursive.
Dependencies declared in build.sbt will be for the proper build.
Dependencies declared in project/plugins.sbt will be for the meta-build.
When you just run updateSbtClassifiers it is running this on the proper build, however the dependencies are actually for the meta-build. That is why I said that this task is a little special.
Another way (which can achieve different results) is to run the updateClassifiers task directly on the meta-build.
First switch over to the meta-build:
sbt> reload plugins
Now that you are in the meta-build run:
sbt:project> updateClassifiers
This will retrieve the src and doc for your dependencies (it may not actually do this for some plugins). Run libraryDependencies to see the dependencies of the meta-build.
To get back to the proper
sbt:project> reload return build run:
Troubleshooting
Download failed
You may see in the sbt logs that it failed to download either the src or doc. For example:
[warn] ::::::::::::::::::::::::::::::::::::::::::::::
[warn] :: FAILED DOWNLOADS ::
[warn] :: ^ see resolution messages for details ^ ::
[warn] ::::::::::::::::::::::::::::::::::::::::::::::
[warn] :: rocks.muki#sbt-graphql;0.5.0!sbt-graphql.jar(doc)
[warn] :: rocks.muki#sbt-graphql;0.5.0!sbt-graphql.jar(src)
[warn] ::::::::::::::::::::::::::::::::::::::::::::::
If you look just above you will see all the resolvers that it tried. You should see one for sbt-plugin-releases which is where most sbt plugins are published to. See predefined resolvers for more details.
If you do not see that it tried sbt-plugin-releases then chances are you have encountered sbt/sbt#3432.
updateSbtClassifiers does not use the correct resolvers. Add the following to your build.sbt file:
updateSbtClassifiers / dependencyResolution := IvyDependencyResolution((updateSbtClassifiers / ivyConfiguration).value)
Sources not attaching
Even if the updateSbtClassifiers successfully downloads and resolves the src and doc IDEA may not attach them.
This seems to be a bug in the sbt-structure plugin. See SCL-13619 for details.
No attempt at downloading
If for some reason this doesn't download the sources (there is no failed download message) then have a look in your .ivy2/exclude_classifiers file. I'm not entirely sure what this file is for but I do know that sbt will exclude anything that is in here.
In my case it had a whole lot of things that I didn't want excluded so I deleted it and then it worked. Delete at your own risk.
You can use reload plugins to go into the project that contains the plugins. If you type libraryDependencies you can (for example) see the list of plugin dependencies. Use reload return to return to your normal sbt console.
More information here: Commands for managing the build definition

Supply provided dependencies with build flag in sbt

I have a project that's main purpose is to provide a collection of algorithms to client. However, it secondarily handles a HTTP interfaces to those algorithms. Since I don't want my clients to inherent the HTTP libraries (unfiltered and dispatch) I have them scoped as provided in sbt. How can I set up my project so that when I sbt assembly, these dependencies (along with a logging implementation) are compiled but when I sbt package or sbt publish they are not?
I think the answer is as simple as telling the assembly plugin to use the compile configuration of jars instead of the runtime dependency (since provided are not on the runtime classpath).
sbt 0.13 syntax:
fullClasspath in assembly := (fullClasspath in Compile).value