There are multiple binary incompatible scala 2 versions, however the document says the installation is either via IDE or SBT.
DOWNLOAD SCALA 2
Then, install Scala:...either by installing an IDE such as IntelliJ, or sbt, Scala's build tool.
Spark 3 needs Scala 2.12.
Spark 3.1.2 uses Scala 2.12. You will need to use a compatible Scala version (2.12.x).
Then how can we make sure the scala version is 2.12 if we install sbt?
Or the documentation is not accurate and it should be "to use specific version of scala, need to download specific scala version on your own"?
Updates
As per the answer by mario-galic, in ONE-CLICK INSTALL FOR SCALA it is said:
Installing Scala has always been a task more challenging than necessary, with the potential to drive away beginners. Should I install Scala itself? sbt? Some other build tools? What about a better REPL like Ammonite? Oh and before all that I need to install Java?
To solve this problem, the Scala Center contracted Alexandre Archambault in January 2020 to add a one-click install of Scala through coursier. For example, on Linux, all we now need is:
$ curl -Lo cs https://git.io/coursier-cli-linux && chmod +x cs && ./cs setup
The Scala version is specified in the build.sbt file so SBT will download the appropriate version of Scala as necessary.
I personally use SDKMAN! to install Java and then SBT.
The key concept to understand is the difference between system-wide installation and project-specific version. System-wide installation ends up somewhere on the PATH like
/usr/local/bin/scala
and can be installed in various ways, personally I recommend coursier one-click install for Scala
curl -Lo cs https://git.io/coursier-cli-linux && chmod +x cs && ./cs setup
Project-specific version is specified by scalaVersion sbt settings which downloads Scala to coursier cache location. To see the Scala version and location used by the particular project try show scalaInstance which
inspect scalaInstance
[info] Task: sbt.internal.inc.ScalaInstance
[info] Description:
[info] Defines the Scala instance to use for compilation, running, and testing.
Scala should be binary compatible within minor version so Spark 3 or any other software built against any Scala 2.12.x version should work with any other Scala 2.12.x version where we have major.minor.patch. Note binary compatibility is not guaranteed for internal compiler APIs, so for example when publishing compiler plugins the best practice is to publish it against full specific Scala version. For example notice how kind-projector compiler plugin is published against full Scala version 2.13.6
https://repo1.maven.org/maven2/org/typelevel/kind-projector_2.13.6/
whilst scala-cats application-level library is published against any Scala 2.13.x version
https://repo1.maven.org/maven2/org/typelevel/cats-core_2.13/
Similarly spark is published against any Scala 2.12.x version
https://repo1.maven.org/maven2/org/apache/spark/spark-core_2.12/
Regarding system-wide installation one trick I do for quick switching of versions is to put scala-runners on the PATH and then different versions can be launched via --scala-version argument
scala --scala-version 2.12.14
Using coursier or scala-runners you can even switch JDK quickly via -C--jvm for example
scala --scala-version 2.12.14 -C--jvm=11
For a project, there should be no need to download manually a specific version of Scala. sbt either directly or indirectly via an IDE will download all the dependencies behind the scenes for you, so the only thing to specify is sbt setting scalaVersion.
Using Python as analogy to Scala, and Pipenv as anology to sbt, then python_version in Pipfile is similar to scalaVersion in build.sbt. After executing pipenv shell and pipenv install you end up with project-specific shell environment with project specific Python version and dependencies. sbt similarly downloads project specific Scala version and dependencies based of build.sbt although it has no need for lock files or for modifying your shell environment.
Related
I am a bit confused right now. I have installed IntelliJ and the Scala SDK using the installation wizard provided by IntelliJ. I selected the latest version 2.11.8 but were not able to run it in the terminal.
After that I ran
sudo apt-get install scala
which is apparently version 2.9.2
scala -version
Scala code runner version 2.9.2 -- Copyright 2002-2011, LAMP/EPFL
It appears that I now have installed scala under
/home/<user>/.ivy2/cache
(which comes from IntelliJ and has version 2.11.8) and also
whereis scala
scala: /usr/bin/scala /usr/bin/X11/scala /usr/share/man/man1/scala.1.gz
which comes from the package manager and hast version 2.9.2.
How would I clean up this mess?
This is not really a mess. It's perfectly possible to install a system-wide Scala and one based on your projects. The system-wide one really makes little sense except for you to run shell scripts using scala script.scala. You can also see that those Debian and Ubuntu packages are horribly outdated.
Since Scala is not an end-user application but a compiler or library required by specific project builds, one usually does not need and/or use a global installation. When you build a project using sbt or IntelliJ, your project can freely choose the Scala version it wants to use, whether that's 2.9.2, 2.10.x, 2.11.x or 2.12.0-Milestone... And therefore, these installations are all kept concurrently in a cache such as the Ivy cache you are seeing from IntelliJ/sbt.
$ ls -la ~/.ivy2/cache/org.scala-lang/scala-compiler/jars/
... scala-compiler-2.10.0.jar
... scala-compiler-2.10.2.jar
... scala-compiler-2.10.3.jar
... scala-compiler-2.10.4.jar
... scala-compiler-2.10.5.jar
... scala-compiler-2.10.6.jar
... scala-compiler-2.11.0.jar
... scala-compiler-2.11.1.jar
... scala-compiler-2.11.2.jar
... scala-compiler-2.11.5.jar
... scala-compiler-2.11.6.jar
... scala-compiler-2.11.7.jar
... scala-compiler-2.11.8.jar
... scala-compiler-2.12.0-M2.jar
... scala-compiler-2.12.0-M3.jar
... scala-compiler-2.7.3.jar
... scala-compiler-2.8.0.jar
... scala-compiler-2.8.2.jar
... scala-compiler-2.9.0.jar
... scala-compiler-2.9.2.jar
If you want a global installation because you want to run scripts without setting up a minimal sbt project, I recommend not to use apt (you can uninstall scala again) but simply download the latest version from http://scala-lang.org/download/ - I'm not aware of any .deb package depending on a .deb Scala installation.
I'm currently reading the Play for Scala: Covers Play 2 book by Peter Hilton (publisher: Manning) , and trying to stick to the versions they've outlined in the book (Play version 2.1.1).
I downloaded the new IntelliJ IDEA 14 CE app and installed it. I'm currently importing the project, and it's failing because it wants a version of sbt that is 0.12.4 or greater.
This issue occurred in 13 the other day, but I haven't had time to resolve it, so please, no recommendations to go back to 13 :)
Here is the meat of the issue:
When I type sbt --version on the CLI, it prints sbt launcher version 0.13.6. I have a "greater" version, but it appears that IDEA doesn't recognize this.
In an effort to specify my own SBT launcher JAR, I pointed my custom launcher (in IDEAs global settings) to the one in /usr/local/Cellar/sbt/0.13.6/libexec/sbt-launch.jar, but the build also failed with the same error.
As you may have guessed at this point, I have used Homebrew to install and manage Scala and sbt. When I first started working with Scala back in February, I remember having to make a few small tweaks in my settings to get IDEA to find Scala, but my version of IDEA 14 is totally fresh -- I did not import any previous settings. I downloaded the Scala and sbt plugins and they appear to be properly configured after a cursory look and creation of a new test project.
Here is what I'm failing to understand:
First, and most obvious, why would an sbt version that meets the supposed requirement (0.12.4+) fail with version 0.13.*?
Second, is the bundled sbt with IDEA also greater than this version? Is there possibly an sbt flag somewhere in my system that's pointing it to an earlier version? A side note, I have a few versions of sbt in my "Cellar", but all are greater than 0.13.
Many thanks for your help!
I had a look at the code samples from Manning's website, and the project/build.properties files for each project specify sbt.version=0.12.2. My guess is that the Intellij Scala plugin only supports SBT 0.12.4, and cannot import you project which is configured to use an older version (hence why it recommends that you "update your project definition").
You should try using sbt.version=0.12.4 in the project/build.properties file.
Generally speaking, keep in mind that when an SBT version is specified in project/build.properties, the SBT launcher downloads the specified version and uses it to build the project. In your case, that means that the SBT 0.13.6 launcher will download SBT 0.12.2 and use that version to build the project.
Reason I ask, is because it's possible to specify a Scala version in the build.sbt file (using scalaVersion setting), and once you do that, sbt will automatically download that Scala version to use with the project.
I also seem to remember that despite having Scala 2.11.1 on my system, sbt would compile and run with Scala 2.10 if no version was specified.
So the question is, do I need to install Scala separately if I got sbt installed?
No you don't need it. sbt will download Scala for you.
If you install sbt-extras (basically just a script) you don't even need to download sbt: it will automatically download the sbt launcher you need. Very handy since you just need to specify sbt.version in your build.properties and you're good to go.
Edit: removed my comment about not being able to do sbt console in an empty directory, since both sbt and sbt-extras support it now.
I have recently begun exploring Scala, and have started by installing the Scala IDE in my copy of Eclipse (Indigo). I initially installed the Scala IDE for Scala 2.9, but then noticed that there was a newer release available for Scala 2.10. Installing the newer plug-in over the older one seems to have worked, but...
Scala 2.10 has deprecated the older Scala Actors in favor of Akka Actors. Thus I'm trying to add an import to my toy Scala project:
import akka.actor.Actor
This is flagged in the IDE with the error
not found: object akka
When I look at my Scala project's properties, I indeed do not see any of the akka-* jar files that are mentioned in the Akka documentation.
Do they need to be downloaded and installed separately, even though the Scala IDE plug-in installed the rest of Scala 2.10? Or have package names changed as part of integrating Akka actors in place of the older Scala Actors? (The documentation doesn't say so, but the Scala 2.10 release is fairly recent...)
No, they aren't packaged together.
The easiest way to make sure the Eclipse IDE can see your dependencies (Akka, and anything else referenced in your build.sbt file) is to let sbt do it using the sbteclipse plugin. Here's the instructions I wrote up for co-workers:
Install the "sbteclipse" plugin
This plugin will allow sbt to add the files/references that Eclipse needs to find all the dependencies that you specify in your build.sbt. Otherwise, you will be able to use the IDE, but you will seek all kinds of "object not found" errors.
Just make sure the plugin is being added in your global plugins.sbt file. This file (and it's path) may not exist so you may need to create it at the following location:
~/.sbt $ cd ~/.sbt/0.13/
~/.sbt/0.13 $ mkdir plugins
Edit/create the plugins.sbt file:
~/.sbt/0.13 $ vi plugins/plugins.sbt
then add this line (it may be the only line in the file):
addSbtPlugin("com.typesafe.sbteclipse" % "sbteclipse-plugin" % "2.5.0")
Running sbteclipse
To use this, you just navigate to a scala project on the commandline and run the following. If you already had Eclipse open, go ahead and restart it.
/sites/ewuser (master)$ sbt eclipse
References:
How to initialize a new Scala project in sbt, Eclipse and github
Official sbteclipse plugin
The Akka artifacts are not bundled with the Scala IDE (yet), you will have to add “akka-actor_2.10” and friends to your project’s dependencies.
Download the akka for eclipse from below location
http://downloads.typesafe.com/akka/akka_2.11-2.4.1.zip?_ga=1.167921254.618585520.1450199987
extract the zip
add dependencies from the lib folder into project
Firing up the SBT console it reads :
[info] Building project AYLIEN 1.0 against Scala 2.8.1
[info] using MyProject with sbt 0.7.4 and Scala 2.7.7
How can I make it use MyProject with sbt 0.7.4 and Scala 2.8.1 ? Please pay attenetion that I'm not asking about the Scala version that is used to build my project (it is the 2.8.1 as you can see), but I rather want to make sbt use MyProject with Scala 2.8.1. Apparently sbt uses it's own scala version to work with project definition (MyProject here) which is different than one it uses to actually build the project! or perhaps I'm missing something ... ?
I can see your concern about SBT still using 2.7.7 internally, but it doesn't really matter since SBT downloads that version on its own. You do not have to install 2.7.7 or anything, just forget about it and pretend your environment is pure Scala 2.8.
The configuration file that holds the SBT version setting is: project/build.properties. The content looks like this:
project.organization=com.ab.web
project.name=cool_proj
sbt.version=0.7.4
project.version=1.0
build.scala.versions=2.8.0
project.initialize=false
When you want to move up to the next SBT version, just change 0.7.4 to that version and SBT will update itself. Eventually SBT will use some other Scala version internally, but this will not matter to the user.
SBT 0.7.* won't work with Scala 2.8.* for your project definition.
Mark Harrah is currently working on the next version of SBT which will work with 2.8.*. This means that you can't use any Scala features or functionality that was added after Scala 2.7.7 in your project definition or plugins. Your project itself is free to use 2.8.*.