I've been trying to compile Apache spark with scala-2.11.1 (the latest version at the time). However, each time I try it ends up compiling everything to scala-2.10.*. I don't understand why.
The official documentation suggests that we use maven for compilation after switching to 2.11 using script in the dev/ folder.
What if I wanted to use sbt instead?
You need to enable scala-2.11 profile
>sbt -Dscala-2.11=true
sbt> compile
Related
I am porting a small legacy library from scala 2.12 to scala 2.13. sbt version is 1.3.3. The project is flat and relatively simple. scalaVersion declared in the project is 2.13.1.
I am executing clean and compile tasks, and then publish to both local ivy and to the artifactory.
The process seemingly goes fine and creates the artifact with the _2.13 suffix. When this binary gets executed against scala 2.13 runtime, it fails with MethodNotFound exception. Further introspection shows that the artifact was compiled for 2.12 but not for 2.13.
Does anybody have an idea why a different compiler version was used by sbt, and how to fix this problem?
Just had similar issue, sbt was compiling my project to the wrong Scala version, found this question without answers in google.
So, my problem was actually very simple. Turns out you need to start sbt in a project root directory (where build.sbt is located). I was running it from a directory where all my .scala files were located, so it haven't parsed build.sbt and compiled project using default Scala version (2.12 in my case).
I'm using an external Jar that doesn't work properly with any version of Scala above 2.10.4. However, by looking at the Scala Jar files, I have seen that the version of Scala I'm using is 2.11. Is it possible to downgrade the Scala version to 2.10.4 without replacing the Scala Jar files?
yes it is, run from zeppelin./dev/change_scala_version.sh 2.10, you might need to rebuilt though using mvn clean package -DskipTests -Pspark-2.1 -Phadoop-2.7 -Pyarn -Ppyspark -Psparkr -Pr -Pscala-2.10
I follow the instructions to build Spark with Scala 2.11:
mvn -Dscala-2.11 -DskipTests clean package
Then I launch per instructions:
./sbin/start-master.sh
It fails with two lines in the log file:
Failed to find Spark assembly in /etc/spark-1.2.1/assembly/target/scala-2.10
You need to build Spark before running this program.
Obviously, it's looking for a scala-2.10 build, but I did a scala-2.11 build. I tried the obvious -Dscala-2.11 flag, but that didn't change anything. The docs don't mention anything about how to run in standalone mode with scala 2.11.
Thanks in advance!
Before building you must run the script under:
dev/change-version-to-2.11.sh
Which should replace references to 2.10 with 2.11.
Note that this will not necessarily work as intended with non-GNU sed (e.g. OS X)
Reason I ask, is because it's possible to specify a Scala version in the build.sbt file (using scalaVersion setting), and once you do that, sbt will automatically download that Scala version to use with the project.
I also seem to remember that despite having Scala 2.11.1 on my system, sbt would compile and run with Scala 2.10 if no version was specified.
So the question is, do I need to install Scala separately if I got sbt installed?
No you don't need it. sbt will download Scala for you.
If you install sbt-extras (basically just a script) you don't even need to download sbt: it will automatically download the sbt launcher you need. Very handy since you just need to specify sbt.version in your build.properties and you're good to go.
Edit: removed my comment about not being able to do sbt console in an empty directory, since both sbt and sbt-extras support it now.
Firing up the SBT console it reads :
[info] Building project AYLIEN 1.0 against Scala 2.8.1
[info] using MyProject with sbt 0.7.4 and Scala 2.7.7
How can I make it use MyProject with sbt 0.7.4 and Scala 2.8.1 ? Please pay attenetion that I'm not asking about the Scala version that is used to build my project (it is the 2.8.1 as you can see), but I rather want to make sbt use MyProject with Scala 2.8.1. Apparently sbt uses it's own scala version to work with project definition (MyProject here) which is different than one it uses to actually build the project! or perhaps I'm missing something ... ?
I can see your concern about SBT still using 2.7.7 internally, but it doesn't really matter since SBT downloads that version on its own. You do not have to install 2.7.7 or anything, just forget about it and pretend your environment is pure Scala 2.8.
The configuration file that holds the SBT version setting is: project/build.properties. The content looks like this:
project.organization=com.ab.web
project.name=cool_proj
sbt.version=0.7.4
project.version=1.0
build.scala.versions=2.8.0
project.initialize=false
When you want to move up to the next SBT version, just change 0.7.4 to that version and SBT will update itself. Eventually SBT will use some other Scala version internally, but this will not matter to the user.
SBT 0.7.* won't work with Scala 2.8.* for your project definition.
Mark Harrah is currently working on the next version of SBT which will work with 2.8.*. This means that you can't use any Scala features or functionality that was added after Scala 2.7.7 in your project definition or plugins. Your project itself is free to use 2.8.*.