I'm trying to upgrade the Spark version from 1.6.2 to 2.3.0. I'm currently using Scala version 2.10, but I need to upgrade to Scala version 2.11.x since Scala 2.10 is no longer supported. My question is which sub-version should I upgrade to? I'm struggling to determine which sub-version to use.
I can't find anything that compares the different sub versions of Scala, but I have encountered some entries that recommend using Scala version 2.11.8 and that there was a bug with Scala 2.11.0 with using Spark (not sure how true that is).
What are your experiences and which sub versions do you recommend? Thanks!
As the documentation says, you can use a compatible version (2.11.x).
So just use the latest version (2.11.12 as of today).
Related
We have several Spark applications running on production developed using Spark 2.4.1 (Scala 2.11.12).
For couple of our new Spark jobs,we are considering utilizing features of DeltaLake.For this we need to use Spark 2.4.2 (or higher).
My questions are:
If we upgrade our Spark cluster to 3.0.0, can our 2.4.1 applications still run on the new cluster (without recompile)?
If we need to recompile our previous Spark jobs with Spark 3, are they source compatible or do they need any migration?
There are some breaking changes in Spark 3.0.0, including source incompatible change and binary incompatible changes. See https://spark.apache.org/releases/spark-release-3-0-0.html. And there are also some source and binary incompatible changes between Scala 2.11 and 2.12, so you may also need to update codes because of Scala version change.
However, only do Delta Lake 0.7.0 and above require Spark 3.0.0. If upgrading to Spark 3.0.0 requires a lot of work, you can use Delta Lake 0.6.x or below. You just need to upgrade Spark to 2.4.2 or above in 2.4.x line. They should be source and binary compatible.
You can cross compile projects Spark 2.4 projects with Scala 2.11 and Scala 2.12. The Scala 2.12 JARs should generally work for Spark 3 applications. There are edge cases when using a Spark 2.4/Scala 2.12 JAR won't work properly on a Spark 3 cluster.
It's best to make a clean migration to Spark 3/Scala 2.12 and cut the cord with Spark 2/Scala 2.11.
Upgrading can be a big pain, especially if your project has a lot of dependencies. For example, suppose your project depends on spark-google-spreadsheets, a project that's not built with Scala 2.12. With this dependency, you won't be able to easily upgrade your project to Scala 2.12. You'll need to either compile spark-google-spreadsheets with Scala 2.12 yourself or drop the dependency. See here for more details on how to migrate to Spark 3.
I am new to 'Spark NLP' and I got stuck in version compatibility issues only. That may seems to be silly but still I request you to help me in this:
‘Spark NLP’ is built on top of Apache Spark 2.4.0 and such is the only supported release (mentioned on https://nlp.johnsnowlabs.com/docs/en/quickstart)
Spark 2.4.3 uses Scala 2.12 (mentioned on https://spark.apache.org/docs/latest/)
But maven repo repository just have spark_nlp libararies for scala 2.11 only not after that (see https://repo1.maven.org/maven2/com/johnsnowlabs/nlp/, only spark-nlp_2.11 is present and not spark-nlp_2.12)
Please help
Which version of com.databricks.spark.csv is compatible for Spark 1.6.1, and scala 2.10.5?
I can see
com.databricks_spark-csv_2.10-1.5.0.jar
com.databricks_spark-csv_2.11-1.3.0.jar
already available on my machine, and as per my understanding goes, if I have scala version 2.10, then the first option is the one that I have to use. just wanted to re-confirm.
You should use the first jar as you have scala 2.10 on your machine
com.databricks_spark-csv_2.10-1.5.0.jar
As 2.10 means it is meant for scala 2.10
At the Spark 2.1 docs it's mentioned that
Spark runs on Java 7+, Python 2.6+/3.4+ and R 3.1+. For the Scala API, Spark 2.1.0 uses Scala 2.11. You will need to use a compatible Scala version (2.11.x).
at the Scala 2.12 release news it's also mentioned that:
Although Scala 2.11 and 2.12 are mostly source compatible to facilitate cross-building, they are not binary compatible. This allows us to keep improving the Scala compiler and standard library.
But when I build an uber jar (using Scala 2.12) and run it on Spark 2.1. every thing work just fine.
and I know its not any official source but at the 47 degree blog they mentioned that Spark 2.1 does support Scala 2.12.
How can one explain those (conflicts?) pieces of information ?
Spark does not support Scala 2.12. You can follow SPARK-14220 (Build and test Spark against Scala 2.12) to get up to date status.
update:
Spark 2.4 added an experimental Scala 2.12 support.
Scala 2.12 is officially supported (and required) as of Spark 3. Summary:
Spark 2.0 - 2.3: Required Scala 2.11
Spark 2.4: Supported Scala 2.11 and Scala 2.12, but not really cause almost all runtimes only supported Scala 2.11.
Spark 3: Only Scala 2.12 is supported
Using a Spark runtime that's compiled with one Scala version and a JAR file that's compiled with another Scala version is dangerous and causes strange bugs. For example, as noted here, using a Scala 2.11 compiled JAR on a Spark 3 cluster will cause this error: java.lang.NoSuchMethodError: scala.Predef$.refArrayOps([Ljava/lang/Object;)Lscala/collection/mutable/ArrayOps.
Look at all the poor Spark users running into this very error.
Make sure to look into Scala cross compilation and understand the %% operator in SBT to limit your suffering. Maintaining Scala projects is hard and minimizing your dependencies is recommended.
To add to the answer, I believe it is a typo https://spark.apache.org/releases/spark-release-2-0-0.html has no mention of scala 2.12.
Also, if we look at timings Scala 2.12 was not released untill November 2016 and Spark 2.0.0 was released on July 2016.
References:
https://spark.apache.org/news/index.html
www.scala-lang.org/news/2.12.0/
I have learned that Scala is suffering from a limitation, that all Scala bytecodes needs to be generated from same compiler version. e.g. I cannot have a library built for 2.9 to work with my application which is built by 2.9.1
http://lift.la/blog/scalas-version-fragility-make-the-enterprise
I tried to search from the web for more discussion on this issue but cannot find much updates. Is this issue, as in Scala 2.11.6, fixed in any extend?
In Scala, the 'middle' number in the version string is the major version, so in 2.10.x and 2.11.x, the major version is 10 and 11 respectively.
Major versions are binary compatible. Therefore, if you have a library compiled against Scala 2.11.0, you can safely use it in a project that uses 2.11.6 without recompilation, and vice versa. If your library was compiled for Scala 2.10.5, you would have to compile it newly to use in a Scala 2.11.x project.
If your code doesn't call into deprecated API, it should be source compatible with the subsequent major version.
Most libraries are published for at least two major versions at the same time, so there is quite a bit of elasticity. Take an example, Scalaz, it has its latest artifacts cross-built for Scala 2.9.3, Scala 2.10.x, and Scala 2.11.x.