I am trying to build my Scala source code with SBT targeting same Scala version but different versions of libraries. My questions are:
Does SBT support this? (seems unlikely)
If not, is there a way to achieve this without messing the build script too much?
Details:
I am building for clusters with different versions of Spark and other libraries. This seems to be a common user case in such a scenario.
Possible solutions I have found:
Use different minor versions of Scala, then dispatch different versions of library with cross-building. Very hackish.
Use dummy directories for multiple projects, redirecting their source paths to the real source directory. This seems most promising but I have not tried it yet.
No sbt does not support this, but if you want to do that then you can use shading.
like if I want to use two different versions of guava that was supported by two different libraries then use guava shading like:
assemblyShadeRules in assembly := Seq(
ShadeRule.rename("com.google.guava**" -> "shadeio.#1").inAll
)
Related
Suppose my application depends on two libraries one of which is Spark.
Spark has the transitive dependency of com.fasterxml.jackson(specific version and not configurable).
Another depending library also has the same transitive dependency, but a newer version(will not work with the version Spark requires).
So basically both versions are required when testing(unit tests) the app.
What would be a recommended way(workaround) to co-exist both versions? if any
You can try shading the problematic dependency with jarjar-abrams. Shading is a process where the bytecode of the library is rewritten to move its classes to a different package, and all references to these classes are rewritten as well to match.
https://eed3si9n.com/jarjar-abrams
Unfortunately it often doesn't work when reflection is involved – I don't know if Jackson uses reflection in a way that's incompatible with this process, but it's worth a shot.
The underlying mechanism used to indicate which version of Scala a library was compiled against is to append _<scala-version> to the library's name. This fairly simple approach allows interoperability with users of Maven, Ant and other build tools.
-- sbt Documentation: Cross-Build Publishing Conventions
While this is a simple approach, the interoperability with Maven and other build tools leaves something to be desired. Because the artifactId is different (e.g. scalatest_2.9.0 and scalatest_2.10.0), Maven treats them as different artifacts. Maven's dependency resolution mechanism is thus compromised and multiple versions of the same artifact (built against different scala versions) can wind up on the classpath.
Why not put the scala version in the classifier? This seems to be one of the primary intended use cases for the classifier:
The classifier allows [Maven] to distinguish artifacts that were built from the same POM but differ in their content. As a motivation for this element, consider for example a project that offers an artifact targeting JRE 1.5 but at the same time also an artifact that still supports JRE 1.4. The first artifact could be equipped with the classifier jdk15 and the second one with jdk14 such that clients can choose which one to use.
-- Maven Documentation: POM Reference
Appending version to the name is a historical decision that was made long time ago so it'll likely not going to change since many libraries are published with the convention already.
Having said that, as Seth noted, there was a discussion to review this topic a few years ago when sbt 0.12 shortened "_2.10.0" postfix to "_2.10" to take advantage of Scala library's binary compatibility between the minor versions. Here's Mark from [0.12] plan:
By cross versioning, I mean the practice of including some part of the Scala version in the module ID to distinguish an artifact generated by compiling the same source code against different Scala versions. I do not mean the ability to build against multiple Scala versions using +task, which will stay; I am just referring to the cross version convention.
[snip]
It has always been a hack to encode this in the inflexible pom.xml format and I think it may be best to move away from this for projects built against Scala 2.10 and later. However, perhaps this is better than any ad hoc solutions that might take its place. I don't see users of other build tools doing this, so I expect nothing would replace it.
Somewhere down the thread Josh suggested:
(1) Scala classifiers. These can be custom strings and can be specified with dependencies. At least, IIRC this should work.
Here's Mark's response:
What do mean by "can be specified with dependencies"? There is only one pom for all of the classifiers, right? How can you declare different dependencies for each classifier?
Here are some more interesting remark on classifiers from Geoff Reedy
I too thought that classifiers would be the perfect way to deal with
this issue especially in light of the suggestion in the maven docs that
classifiers java-1.4 and java-1.5 be used to distiguish between jars
appropriate for the respective platform. The fatal flaw seems to be
transitive dependency management. That is, there's no way to choose the
transitive dependency set based on the classifier used to require the
module. We'd need to be able to say that when you're using this module
with the scala-2.10 classifier it brings its own dependencies using the
scala-2.10 classifier and when used with the 2.9 classifier brings in
its own deps with the scala-2.9 classifier.
I think with the jvm versions it's possible to make this work because
jvm versioning has special support in the profile activation which can
can control dependencies.
Does the Scala version that sbt uses to run matters? Can it be changed?
p.s. I'm not asking about changing the Scala version against which your project is build.
In regards to the current version, looking at the changelogs for sbt, there are several updates that mention a new version of scala, the most recent of which is:
Scala version is bumped to 2.10.5. [...]
As for changing it, I don't believe that's possible, unless you want to build it (and any plugins you are using) from source yourself.
Does it really matter? Not really! Your project itself will use whatever version of scala you specify. There are only two ways I can see that it might matter - if you want to write a custom sbt plugin to us in your project that relies on a feature of a newer scala version, or if you want to use one of those features in your build build files (which are just scala scripts). However, I really can't imagine what you would need to do for a build that you couldn't accomplish with the features of 2.10.x.
scala-2.11 folder appeared after recent update of IDEA and Scala plugin.
What should it be used for?
Usually such directories are used for binary version-dependent code. For example, macros in 2.10 are not source-compatible with macros in 2.11, so if you're building your project for different binary versions and you're using macros, it makes sense to put code which is only valid for the specific version in different source roots. SBT then will use the appropriate directory when compiling for 2.10 or 2.11.
If you're using SBT, though, you would need to set such thing up manually in the build definition. If you're not using SBT, then probably IDEA plugin was updated to handle such things by itself.
Is there a simple way to follow the Scala convention and append the Scala binary version to the archive name and artifactID of Scala libraries in Gradle? Even better if it is possible to compile the library against multiple Scala versions.
The convention that you are referring to isn't built into the Scala plugin, but setting jar.archiveName should do the trick. Likewise, building multiple variants isn't currently a first-class feature, but can be achieved in various ways, for example by declaring a separate source set per variant, with each source set pointing to the same source and resource directories. A proof of concept that also handles variants in dependencies (similar to what's offered by sbt) was presented in my Polyglot Gradle talk at Gradle Exchange 2013.