I try to use my Scala 3 Library in my Scala 2.13 Project.
I get the following exception:
(class scala.tools.tasty.UnpickleException/TASTy signature has wrong version.
expected: {majorVersion: 28, minorVersion: 0}
found : {majorVersion: 28, minorVersion: 1}
The library uses Scala 3.1. The Project uses Scala 2.13.6, with this dependency:
libraryDependencies += ("io.github.pme123" %% "camundala-api" % "0.1.6").cross(CrossVersion.for2_13Use3)
How can I resolve this? I could not find a Compatibility Matrix. My guess is that when using Scala 3.0. in the library it would work.
As mentioned by #Jasper-M, the TASTy reader was updated in Scala 2.13.7 to work with Scala 3.1. Here is the release note where it was mentioned.
So:
Scala 2.13.7 works with Scala 3.1
Scala 2.13.6 works with Scala 3.0
Related
Just made sure that my scala is 2.13.6 and my sbt is 1.5.3 (I use sdkman).
Why? Because I'm switching to Scala 3 and follow the instructions.
Can't compile:
The compiler bridge sources org.scala-sbt:compiler-bridge_2.13:1.2.1:compile could not be retrieved.
Is there a solution to this problem?
build.properties:
sbt.version=1.2.1
This was fixed when I replaced sbt.version in build.properties: with sbt.version=1.5.3 it worked perfectly.
I am building my first Spark application, developing with IDEA.
In my cluster, the version of Spark is 2.1.0, and the version of Scala is 2.11.8.
http://spark.apache.org/downloads.html tells me:"Starting version 2.0, Spark is built with Scala 2.11 by default. Scala 2.10 users should download the Spark source package and build with Scala 2.10 support".
So here is my question:What's the meaning of "Scala 2.10 users should download the Spark source package and build with Scala 2.10 support"? Why not use the version of Scala 2.1.1?
Another question:Which version of Scala can I choose?
First a word about the "why".
The reason this subject evens exists is that scala versions are not (generally speacking) binary compatible, although most of the times, source code is compatible.
So you can take Scala 2.10 source and compile it into 2.11.x or 2.10.x versions. But 2.10.x compiled binaries (JARs) can not be run in a 2.11.x environment.
You can read more on the subject.
Spark Distributions
So, the Spark package, as you mention, is built for Scala 2.11.x runtimes.
That means you can not run a Scala 2.10.x JAR of yours, on a cluster / Spark instance that runs with the spark.apache.org-built distribution of spark.
What would work is :
You compile your JAR for scala 2.11.x and keep the same spark
You recompile Spark for Scala 2.10 and keep your JAR as is
What are your options
Compiling your own JAR for Scala 2.11 instead of 2.10 is usually far easier than compiling Spark in and of itself (lots of dependencies to get right).
Usually, your scala code is built with sbt, and sbt can target a specific scala version, see for example, this thread on SO. It is a matter of specifying :
scalaVersion in ThisBuild := "2.10.0"
You can also use sbt to "cross build", that is, build different JARs for different scala versions.
crossScalaVersions := Seq("2.11.11", "2.12.2")
How to chose a scala version
Well, this is "sort of" opinion based. My recommandation would be : chose the scala version that matches your production Spark cluster.
If your production Spark is 2.3 downloaded from https://spark.apache.org/downloads.html, then as they say, it uses Scala 2.11 and that is what you should use too. Using anything else, in my view, just leaves the door open for various incompatibilities down the road.
Stick with what your production needs.
At the Spark 2.1 docs it's mentioned that
Spark runs on Java 7+, Python 2.6+/3.4+ and R 3.1+. For the Scala API, Spark 2.1.0 uses Scala 2.11. You will need to use a compatible Scala version (2.11.x).
at the Scala 2.12 release news it's also mentioned that:
Although Scala 2.11 and 2.12 are mostly source compatible to facilitate cross-building, they are not binary compatible. This allows us to keep improving the Scala compiler and standard library.
But when I build an uber jar (using Scala 2.12) and run it on Spark 2.1. every thing work just fine.
and I know its not any official source but at the 47 degree blog they mentioned that Spark 2.1 does support Scala 2.12.
How can one explain those (conflicts?) pieces of information ?
Spark does not support Scala 2.12. You can follow SPARK-14220 (Build and test Spark against Scala 2.12) to get up to date status.
update:
Spark 2.4 added an experimental Scala 2.12 support.
Scala 2.12 is officially supported (and required) as of Spark 3. Summary:
Spark 2.0 - 2.3: Required Scala 2.11
Spark 2.4: Supported Scala 2.11 and Scala 2.12, but not really cause almost all runtimes only supported Scala 2.11.
Spark 3: Only Scala 2.12 is supported
Using a Spark runtime that's compiled with one Scala version and a JAR file that's compiled with another Scala version is dangerous and causes strange bugs. For example, as noted here, using a Scala 2.11 compiled JAR on a Spark 3 cluster will cause this error: java.lang.NoSuchMethodError: scala.Predef$.refArrayOps([Ljava/lang/Object;)Lscala/collection/mutable/ArrayOps.
Look at all the poor Spark users running into this very error.
Make sure to look into Scala cross compilation and understand the %% operator in SBT to limit your suffering. Maintaining Scala projects is hard and minimizing your dependencies is recommended.
To add to the answer, I believe it is a typo https://spark.apache.org/releases/spark-release-2-0-0.html has no mention of scala 2.12.
Also, if we look at timings Scala 2.12 was not released untill November 2016 and Spark 2.0.0 was released on July 2016.
References:
https://spark.apache.org/news/index.html
www.scala-lang.org/news/2.12.0/
I am running my scala code from java and it was working fine before when I was using scala 2.9.1 now I upgraded from 2.9.1 to 2.10.4 and I have this issue
java.lang.NoSuchMethodError: scala.Predef$.any2ArrowAssoc(Ljava/lang/Object;)Lscala/Predef$ArrowAssoc
Can anyone tell me whats the problem ?
Thanks
That sounds like you have a dependency clash. That is, you may have updated your code to use Scala 2.10 but you can't updated your dependencies to use Scala 2.10. For those libraries which are Scala base (not Java based) do you have things imported as
"com.foo" %% "name" % "version.number"
and not just name_2.9?
I am building my first Scala/Play application and after I create and import an eclipse project from the Play shell, I get 17 errors.
akka-actor_2.10.jar is cross compiled with an incompatible version of Scala (2.10)
akka-slf4j_2.10.jar is cross compiled with an incompatible version of Scala (2.10)
anorm_2.10.jar ...
play_2.10.jar ...
play_iterates_2.10.jar ...
The list goes on to include the Scala jars, scalaz jars, etc...
I am using:
Eclipse 4.2.2
Scala ide 3.0.0.nightly-2_09
Scala 2.10
Play 2.1
Has someone experienced the same thing?
Is it possible you are using Scala IDE built for for Scala 2.9.x ?
Make sure you use the one for 2.10.x:
http://download.scala-ide.org/sdk/e38/scala210/dev/site/