The underlying mechanism used to indicate which version of Scala a library was compiled against is to append _<scala-version> to the library's name. This fairly simple approach allows interoperability with users of Maven, Ant and other build tools.
-- sbt Documentation: Cross-Build Publishing Conventions
While this is a simple approach, the interoperability with Maven and other build tools leaves something to be desired. Because the artifactId is different (e.g. scalatest_2.9.0 and scalatest_2.10.0), Maven treats them as different artifacts. Maven's dependency resolution mechanism is thus compromised and multiple versions of the same artifact (built against different scala versions) can wind up on the classpath.
Why not put the scala version in the classifier? This seems to be one of the primary intended use cases for the classifier:
The classifier allows [Maven] to distinguish artifacts that were built from the same POM but differ in their content. As a motivation for this element, consider for example a project that offers an artifact targeting JRE 1.5 but at the same time also an artifact that still supports JRE 1.4. The first artifact could be equipped with the classifier jdk15 and the second one with jdk14 such that clients can choose which one to use.
-- Maven Documentation: POM Reference
Appending version to the name is a historical decision that was made long time ago so it'll likely not going to change since many libraries are published with the convention already.
Having said that, as Seth noted, there was a discussion to review this topic a few years ago when sbt 0.12 shortened "_2.10.0" postfix to "_2.10" to take advantage of Scala library's binary compatibility between the minor versions. Here's Mark from [0.12] plan:
By cross versioning, I mean the practice of including some part of the Scala version in the module ID to distinguish an artifact generated by compiling the same source code against different Scala versions. I do not mean the ability to build against multiple Scala versions using +task, which will stay; I am just referring to the cross version convention.
[snip]
It has always been a hack to encode this in the inflexible pom.xml format and I think it may be best to move away from this for projects built against Scala 2.10 and later. However, perhaps this is better than any ad hoc solutions that might take its place. I don't see users of other build tools doing this, so I expect nothing would replace it.
Somewhere down the thread Josh suggested:
(1) Scala classifiers. These can be custom strings and can be specified with dependencies. At least, IIRC this should work.
Here's Mark's response:
What do mean by "can be specified with dependencies"? There is only one pom for all of the classifiers, right? How can you declare different dependencies for each classifier?
Here are some more interesting remark on classifiers from Geoff Reedy
I too thought that classifiers would be the perfect way to deal with
this issue especially in light of the suggestion in the maven docs that
classifiers java-1.4 and java-1.5 be used to distiguish between jars
appropriate for the respective platform. The fatal flaw seems to be
transitive dependency management. That is, there's no way to choose the
transitive dependency set based on the classifier used to require the
module. We'd need to be able to say that when you're using this module
with the scala-2.10 classifier it brings its own dependencies using the
scala-2.10 classifier and when used with the 2.9 classifier brings in
its own deps with the scala-2.9 classifier.
I think with the jvm versions it's possible to make this work because
jvm versioning has special support in the profile activation which can
can control dependencies.
Related
Suppose my application depends on two libraries one of which is Spark.
Spark has the transitive dependency of com.fasterxml.jackson(specific version and not configurable).
Another depending library also has the same transitive dependency, but a newer version(will not work with the version Spark requires).
So basically both versions are required when testing(unit tests) the app.
What would be a recommended way(workaround) to co-exist both versions? if any
You can try shading the problematic dependency with jarjar-abrams. Shading is a process where the bytecode of the library is rewritten to move its classes to a different package, and all references to these classes are rewritten as well to match.
https://eed3si9n.com/jarjar-abrams
Unfortunately it often doesn't work when reflection is involved – I don't know if Jackson uses reflection in a way that's incompatible with this process, but it's worth a shot.
I've created an feature-based Eclipse product where all dependencies respectively external plugins are specified in a dependencies feature project. Some dependencies need different versions of the same plugin. How can I specify several versions of a plugin in a feature.xml? Eclipse seems to always use the current version of a plugin. I've tried to add a dependency with a strict version interval, e.g. [1.6.0.v201011041432,1.6.0.v201011041432], in the feature.xml but Eclipse doesn't accept this format.
I had a similar problem when running a feature based product within Eclipse (Kepler) where multiple versions of the same bundle were involved.
In the end I used a workaround - I changed the symbolic names of the bundles, so they all have different names. If you consequently use Import-Package instead of Require-Bundle this will make no difference in the dependencies you define for bundles or in the OSGi runtime, only in your Feature definition.
This solution is not pretty, but at least it's rather easy to do.
Thinking about that OSGi is intended to target exactly this kind of use case where you have multiple versions of the same dependency/bundle it's rather strange how bad the support in Eclipse is, if you actually have this kind of use case.
Is there a simple way to follow the Scala convention and append the Scala binary version to the archive name and artifactID of Scala libraries in Gradle? Even better if it is possible to compile the library against multiple Scala versions.
The convention that you are referring to isn't built into the Scala plugin, but setting jar.archiveName should do the trick. Likewise, building multiple variants isn't currently a first-class feature, but can be achieved in various ways, for example by declaring a separate source set per variant, with each source set pointing to the same source and resource directories. A proof of concept that also handles variants in dependencies (similar to what's offered by sbt) was presented in my Polyglot Gradle talk at Gradle Exchange 2013.
Since you have only source compatibility between Scala-versions you unfortunately need to compile libraries like scalatest or scalamock for each scala version they support. What puzzles me is that the libraries are provided with loads of artifacts (scalatest_2.9.0, scalatest_2.9.1, scalatest_2.10 and so forth) - one for each scala version, such that the maven repository is littered with many artefacts that are built from the same source. My instinct tells me rather to use one artifact with a classifier for each scala version. (In fact, the maven pom reference mentions that this was sometimes done with jdk14 and jdk15 classifiers for artifacts, which seems similar to me.) So, why did the Scala people go for the many artifact overkill :-) instead?
I may be wrong, but if a classifier's purpose is to "distinguish artifacts that were built from the same POM but differ in their content", then I see a very good reason not to use them for scala versioning: scala versions are not just binary incompatible, they can very well be source incompatible.
By example, when upgrading scala from 2.7 to 2.8, I had to make some significant changes to the code base. If I wanted to keep both a scala 2.7 and 2.8 version at the same time, I would have needed to create a parallel branch, and both branches would definitely not have the same source code.
When I read "from the same POM", I understand that it means from the same source code too, which would clearly not be the case with those two branches of code.
Another more important reason is that a classifier is essentially a single string, which is already used for many things. More or less standard classifiers include "sources", "javadoc" or "resources". The meanings of these classifiers are the same in scala project, and are totally orthogonal to the scala version, as I'll try to show.
Maven's documentation suggests using classifiers such as "jdk15" or "jdk14" to denote which version of the jvm the binary artifact was compiled against.
Given that java code is backward compatible, in principle the artifacts with both classifiers ("jdk15" or "jdk14") are compiled from the same source code. This is why you don't need to duplicate classifiers for "sources" artifact, or in other words you don't need to have a classifier named "sources-jdk14" and "sources-jdk15".
But you cannot apply the same rationale to the scala version: given that you might need different source code whether you compile against scala 2.7 or scala 2.8, you would indeed need two different artifacts with classifiers such as "sources-scala2.7" or "source-scala2.8". So we have composite classifiers already.
As for binary artifacts, you would also not only need to distinguish between the target jvm version (remember, you can compile your scala code to target different jvm versions) but also the scala version it was compiled against. So you would end up with something like "jdk14-scala2.7" or "jdk14-scala2.8" or "jdk15-scala2.7" or "jdk15-scala2.8". Yet another set of composite classifiers.
So the take home message is that the scala version really is a separate way of classifying artifacts, that is totally orthognal to all the existing classifiers.
Yes, we could really use composite classifiers as above (such as "sources-scala2.7") but then we would not be using standard classifiers, which is confusing enough in itself, but would also require to modify all the tooling around classifiers: what if I use a build tool that has no knowledge of scala (only java) but knows how to automatically publish a "source" artifact? Will I need to modify this build tool so that he knows to publish a "sources-scala2.7" artifact instead? On the other hand, if I encode the scala version in the (base) artifact name and give that to the build tool, everything works as usual and I get a an artifact with the "source" classifier.
All in all, and contrary to immediate intuition, encoding the scala version in the name allows for better integration in the existing java build ecosystem.
Scala provides inter-version compatibility of its bytecode output (.class files) only across patch-releases (third component of Major.Minor.Patch version spec).
Maven has no place to properly encode this as a first-class property of the artifact, so it has to be encoded by convention in the name.
Sadly...
I have a library that I wrote in Scala that uses Bouncy Castle and has a whole bunch of dependencies. When I roll a jar, I can either roll a "fat" jar that has all the dependencies (including scala), which weighs in around 19 MB, or I can roll a skinny jar, which doesn't have dependencies, but is only a few hundred KB.
The problem is that I need to include the Bouncy Castle classes/jar with my library, because if its not on the classpath at runtime, all kinds of exceptions get thrown.
So, I think the ideal situation is if there is some way that I can get either Maven or SBT to include some but not all dependencies in the jar that gets rolled. Some dependencies are needed at compile-time, but not at run time, such as the Scala standard libraries. Is there some way to get that to happen?
Thanks!
I would try out the sbt proguard plugin from https://github.com/nuttycom/sbt-proguard-plugin . It should be able to weed out the classes that are not in use.
If it is sufficient to explicitly define which dependencies should be added (one the artifact-level, i.e., single JARs), you can define an assembly (in case of a single project) or an additional assembly project (in case of a multi-module project). Assembly descriptors can explicitly exclude/include artifacts from the dependencies.
Here is some good documentation on this topic (section 8.5.4), here is the official documentation.
Note that you can include all artifacts that belong to one group by using the wildcard notation in dependecySets, e.g. hibernate:*:jar would include all JAR files belonging to the hibernate group.
Covering maven...
Because you declare your project to be dependent upon bouncy castle in your maven pom, anybody using maven to depend upon your library will by default pull in bouncy castle as a transitive dependency.
You should set the appropriate scope on your dependencies, e.g. compile for stuff needed at compile and runtime, test for dependencies only needed in testing and provided for stuff you expect to be provided by the environment.
Whether your library's dependencies are packaged into dependent projects when they are built is a question of how those are projects configured and setting the scopes will influence the default behaviour.
For example, jar type packaging by default does not include dependencies, whereas war will include those in compile scope (but not test or provided). The design aim here was to have packaging plugins behave in the most commonly required way without needing configuration, but of course packaging plugins in maven can be configured to have different behaviour if needed. The plugins themselves which do packaging are well documented at the apache maven site.
If users of your library are unlikely to be using maven to build their projects, an option is to use the shade plugin which will allow you to produce an "uber-jar" which contains all the dependencies you wish. You can configure particular includes or excludes.
This can be a problematic way to deliver, for example where your library includes dependencies which version clash with the direct dependencies of projects using it, i.e. they use a different version of the same libraries yours does.
However if you can it is best that you leave this to maven to manage so that projects using your library can decide whether they want your dependencies or to specify particular versions giving them more flexibility. This is the idiomatic approach.
For more information on dependencies and scopes in maven, see the reference guide published by Sonatype.
I'm not a scala guy, but I have played around with assembling stuff in Java + Maven.
Have you tried looking into creating your own assembly descriptor for the assembly plugin? https://maven.apache.org/plugins/maven-assembly-plugin/assembly.html
You can copy / paste the jar-with-dependencies descriptor then just add some excludes to your < dependencySet >. I'm not a Maven expert, but you should be able to configure it so different profiles kick off different assembly builds.
EDIT: Ack, didn't see my HTML got hidden