Is it possible to include 2.10 packages in 2.11 programs, and vis versa?
Are there any special corner cases I should watch out for?
It's not possible. Major versions are not binary compatible with each other. So for any Scala versions 2.x.y and 2.w.z, where x != w, they will not be compatible. All libraries must be compiled against the same major version, however minor version differences are okay.
Related
I am readying the following sbt page:
https://www.scala-sbt.org/1.x/docs/Cross-Build.html#Cross-building
But i find the documentation not that great, and would like to clarify something which i think i understand but is not made explicit in the doc.
What is the convention for :
CrossVersion.binary (_<scala-binary-version>)
CrossVersion.full (_<scala-version>)
In other words i want a simple example of
_<scala-binary-version>
and of
_
is it :
Binary Version example 2.12
Scala version example 2.12.12
Is that the difference between a binary version and the scala version, where the later would include the compiler access and what not ?
EDIT1
The following example is given in the page:
These are equivalent:
"a" %% "b" % "1.0"
("a" % "b" % "1.0").cross(CrossVersion.binary)
This overrides the defaults to always use the full Scala version instead of the binary Scala version:
("a" % "b" % "1.0").cross(CrossVersion.full)
I only understand what is meant is the second statement, because i have been using the first and know what it does. But it is a guess, and i am just looking for an explicit confirmation with an example, which i believe could benefit any new scala dev in their journey.
So 2.12.12 is a full version (as well as a patch version) and 2.12 is a binary version.
Binary versions are useful because libraries compiled using a different but binary compatible version can be used in your project without any problems. For example, if you are using Scala 2.13.3 you can use a library that was compiled using 2.13.0 or 2.13.4 but not one compiled using 2.12.12.
The full version is useful for things that access the underlying compiler API which doesn't retain binary compatibility, for example, compiler plugins like the kind projector.
BTW, just for fun, 2.13.0-RC1 is another full version but not a patch version.
I'm trying to upgrade the Spark version from 1.6.2 to 2.3.0. I'm currently using Scala version 2.10, but I need to upgrade to Scala version 2.11.x since Scala 2.10 is no longer supported. My question is which sub-version should I upgrade to? I'm struggling to determine which sub-version to use.
I can't find anything that compares the different sub versions of Scala, but I have encountered some entries that recommend using Scala version 2.11.8 and that there was a bug with Scala 2.11.0 with using Spark (not sure how true that is).
What are your experiences and which sub versions do you recommend? Thanks!
As the documentation says, you can use a compatible version (2.11.x).
So just use the latest version (2.11.12 as of today).
I am building my first Spark application, developing with IDEA.
In my cluster, the version of Spark is 2.1.0, and the version of Scala is 2.11.8.
http://spark.apache.org/downloads.html tells me:"Starting version 2.0, Spark is built with Scala 2.11 by default. Scala 2.10 users should download the Spark source package and build with Scala 2.10 support".
So here is my question:What's the meaning of "Scala 2.10 users should download the Spark source package and build with Scala 2.10 support"? Why not use the version of Scala 2.1.1?
Another question:Which version of Scala can I choose?
First a word about the "why".
The reason this subject evens exists is that scala versions are not (generally speacking) binary compatible, although most of the times, source code is compatible.
So you can take Scala 2.10 source and compile it into 2.11.x or 2.10.x versions. But 2.10.x compiled binaries (JARs) can not be run in a 2.11.x environment.
You can read more on the subject.
Spark Distributions
So, the Spark package, as you mention, is built for Scala 2.11.x runtimes.
That means you can not run a Scala 2.10.x JAR of yours, on a cluster / Spark instance that runs with the spark.apache.org-built distribution of spark.
What would work is :
You compile your JAR for scala 2.11.x and keep the same spark
You recompile Spark for Scala 2.10 and keep your JAR as is
What are your options
Compiling your own JAR for Scala 2.11 instead of 2.10 is usually far easier than compiling Spark in and of itself (lots of dependencies to get right).
Usually, your scala code is built with sbt, and sbt can target a specific scala version, see for example, this thread on SO. It is a matter of specifying :
scalaVersion in ThisBuild := "2.10.0"
You can also use sbt to "cross build", that is, build different JARs for different scala versions.
crossScalaVersions := Seq("2.11.11", "2.12.2")
How to chose a scala version
Well, this is "sort of" opinion based. My recommandation would be : chose the scala version that matches your production Spark cluster.
If your production Spark is 2.3 downloaded from https://spark.apache.org/downloads.html, then as they say, it uses Scala 2.11 and that is what you should use too. Using anything else, in my view, just leaves the door open for various incompatibilities down the road.
Stick with what your production needs.
I have a previously working Scala script that when I try to run it on a new PC, the compilation fails.
So I made simple script to test:
#!/bin/sh
exec scala -J-Xmx2g "$0" "$#"
!#
println("test")
And trying to run it I get:
test.scala
error: Compile server encountered fatal condition: java.nio.ByteBuffer.clear()Ljava/nio/ByteBuffer;
java.lang.NoSuchMethodError: java.nio.ByteBuffer.clear()Ljava/nio/ByteBuffer;
at scala.tools.nsc.io.SourceReader.read(SourceReader.scala:61)
at scala.tools.nsc.io.SourceReader.read(SourceReader.scala:40)
at scala.tools.nsc.io.SourceReader.read(SourceReader.scala:49)
at scala.tools.nsc.Global.getSourceFile(Global.scala:395)
at scala.tools.nsc.Global.getSourceFile(Global.scala:401)
at scala.tools.nsc.Global$Run$$anonfun$30.apply(Global.scala:1607)
at scala.tools.nsc.Global$Run$$anonfun$30.apply(Global.scala:1607)
at scala.collection.immutable.List.map(List.scala:284)
at scala.tools.nsc.Global$Run.compile(Global.scala:1607)
at scala.tools.nsc.StandardCompileServer.session(CompileServer.scala:151)
at scala.tools.util.SocketServer$$anonfun$doSession$1$$anonfun$apply$1.apply(SocketServer.scala:74)
at scala.tools.util.SocketServer$$anonfun$doSession$1$$anonfun$apply$1.apply(SocketServer.scala:74)
at scala.util.DynamicVariable.withValue(DynamicVariable.scala:58)
at scala.Console$.withOut(Console.scala:65)
at scala.tools.util.SocketServer$$anonfun$doSession$1.apply(SocketServer.scala:74)
at scala.tools.util.SocketServer$$anonfun$doSession$1.apply(SocketServer.scala:69)
at scala.tools.nsc.io.Socket.applyReaderAndWriter(Socket.scala:49)
at scala.tools.util.SocketServer.doSession(SocketServer.scala:69)
at scala.tools.util.SocketServer.loop$1(SocketServer.scala:85)
at scala.tools.util.SocketServer.run(SocketServer.scala:97)
at scala.tools.nsc.CompileServer$$anonfun$execute$2$$anonfun$apply$mcZ$sp$1.apply$mcZ$sp(CompileServer.scala:218)
at scala.tools.nsc.CompileServer$$anonfun$execute$2$$anonfun$apply$mcZ$sp$1.apply(CompileServer.scala:213)
at scala.tools.nsc.CompileServer$$anonfun$execute$2$$anonfun$apply$mcZ$sp$1.apply(CompileServer.scala:213)
at scala.util.DynamicVariable.withValue(DynamicVariable.scala:58)
at scala.Console$.withOut(Console.scala:53)
at scala.tools.nsc.CompileServer$$anonfun$execute$2.apply$mcZ$sp(CompileServer.scala:213)
at scala.tools.nsc.CompileServer$$anonfun$execute$2.apply(CompileServer.scala:213)
at scala.tools.nsc.CompileServer$$anonfun$execute$2.apply(CompileServer.scala:213)
at scala.util.DynamicVariable.withValue(DynamicVariable.scala:58)
at scala.Console$.withErr(Console.scala:80)
at scala.tools.nsc.CompileServer$.execute(CompileServer.scala:212)
at scala.tools.nsc.CompileServer$.main(CompileServer.scala:180)
at scala.tools.nsc.CompileServer.main(CompileServer.scala)
It seems like Scala is compiling something near my script, but I don't quite know how to debug it and fix it.
TL;DR
Ubuntu's Scala package used to be incompatible with Java 8 (this has been fixed in 2.11.12-4). The solution was to uninstall Ubuntu's Scala package and install one of the official Scala packages. You might still want to do that, this time around, not due to incompatibility with Java, but because Ubuntu's latest packaged Scala version is still 2.11, while Scala's latest version is currently 2.13.
sudo apt remove scala-library scala
wget https://downloads.lightbend.com/scala/2.13.4/scala-2.13.4.deb
sudo dpkg -i scala-2.13.4.deb
Since many people were asking for the reason behind this issue and I was also curious about what caused it, I did some digging...
The root of the problem
In Java 9, Buffer subclasses (including ByteBuffer) were changed to override methods that in the superclass return Buffer to return the respective subtype.
Bug: https://bugs.java.com/bugdatabase/view_bug.do?bug_id=JDK-4774077
Commit: https://github.com/AdoptOpenJDK/openjdk-jdk9/commit/d9d7e875470bf478110b849315b4fff55b4c35cf
This change is not binary backward compatible. If some Java code which calls one these methods directly in one of Buffer's subclasses is compiled with JDK9+, the generated bytecode will not run in JRE8 (even if the returned value is not used at all). This happens because the signature of the method when called will be compiled as java.nio.ByteBuffer.clear()Ljava/nio/ByteBuffer which doesn't exist in JRE8. However, if compiled with JDK8, the signature compiled into bytecode would be java/nio/ByteBuffer.clear:()Ljava/nio/Buffer which exists in the Buffer calss in both JRE8 and JRE9+.
Where did Scala go wrong? (Or did it?)
Scala compiler does use some of the methods affected by the changes above. Particularly, in the SourceReader class where the error in OP's question happened.
Looking at Scala's compatibility matrix, it says that we need at least Scala 2.11.12 to use JDK11, but it doesn't say much explicitly about the opposite direction of compatibility. It does say though that "Scala 2.12+ definitely doesn't work at all on JDK 6 or 7", so we could expect that 2.12+ is still compatible with JDK8, and even more so Scala 2.11.
Why did they break the compatibility then? Couldn't they just compile Scala's source code with an older JDK version? They didn't and they could, so much, that they still do it.
If we download one of the official Scala packages and check the manifest file for scala-compiler.jar, this is what we find:
Scala 2.11.12:
Bundle-Name: Scala Compiler
Bundle-RequiredExecutionEnvironment: JavaSE-1.6, JavaSE-1.7
Bundle-SymbolicName: org.scala-lang.scala-compiler
Bundle-Version: 2.11.12.v20171031-225310-b8155a5502
Created-By: 1.6.0_45 (Sun Microsystems Inc.)
Scala 2.13.4:
Bundle-Name: Scala Compiler
Bundle-RequiredExecutionEnvironment: JavaSE-1.8
Bundle-SymbolicName: org.scala-lang.scala-compiler
Bundle-Version: 2.13.4.v20201117-181115-VFINAL-39148e4
Created-By: 1.8.0_275 (AdoptOpenJDK)
So it seems Scala 2.11 is still being compiled with JDK6 and Scala 2.13 is still being compiled with JDK8. Shouldn't that mean that they are both compatible with JRE8? Yes and indeed they are. Where's the error coming from then?
Where did Ubuntu go wrong?
Ubuntu, as most other Linux distributions do, likes to build its own packages that are made available through its package manager. This is done to ensure that everything works properly within the OS ecosystem, and that often means patching the source code of upstream projects.
Regarding the Scala package in particular, Ubuntu decided to ditch the upstream choices of JDK versions used to compile the Scala source code and has been using newer JDK versions to compile Ubuntu's Scala package for a while.
If we check the manifest file for scala-compiler.jar in Ubuntu's Scala 2.11.12-4, we can see that is was compiled with JDK11:
Created-By: 11.0.2+9-Ubuntu-3ubuntu1 (Oracle Corporation)
Bundle-Name: Scala Distribution
Bundle-SymbolicName: org.scala-ide.scala.compiler;singleton:=true
Bundle-Version: 2.11.12
Didn't you say the issue was resolved in 2.11.12-4? Yes, I did.
Ubuntu's solution for this problem was not to compile Scala with JDK8, but rather to patch Scala's source code to avoid calling the problematic methods directly in the subclasses. This was achieved by casting ByteBuffer (and CharBuffer) to its superclass Buffer before calling these methods. In practice, that meant changing Scala's source code from bytes.clear() to bytes.asInstanceOf[Buffer].clear().asInstanceOf[ByteBuffer] (not sure why they cast it back to ByteBuffer when the result from clear() doesn't seem to be used at all). Here is Ubuntu's patch.
Ubuntu's approach seems a bit dangerous, because other sources of incompatibility could have gone unnoticed and still be there waiting to happen in some very specific situation. Also having their own setup different from the official Scala releases means not having the whole Scala community testing these changes in real-case scenarios.
It works for me by disabling fsc with version 2.11.12:
#!/usr/bin/env -S scala -nc
I have learned that Scala is suffering from a limitation, that all Scala bytecodes needs to be generated from same compiler version. e.g. I cannot have a library built for 2.9 to work with my application which is built by 2.9.1
http://lift.la/blog/scalas-version-fragility-make-the-enterprise
I tried to search from the web for more discussion on this issue but cannot find much updates. Is this issue, as in Scala 2.11.6, fixed in any extend?
In Scala, the 'middle' number in the version string is the major version, so in 2.10.x and 2.11.x, the major version is 10 and 11 respectively.
Major versions are binary compatible. Therefore, if you have a library compiled against Scala 2.11.0, you can safely use it in a project that uses 2.11.6 without recompilation, and vice versa. If your library was compiled for Scala 2.10.5, you would have to compile it newly to use in a Scala 2.11.x project.
If your code doesn't call into deprecated API, it should be source compatible with the subsequent major version.
Most libraries are published for at least two major versions at the same time, so there is quite a bit of elasticity. Take an example, Scalaz, it has its latest artifacts cross-built for Scala 2.9.3, Scala 2.10.x, and Scala 2.11.x.