When playing with something in Scala, I typically spend a bunch of time trying combinations of dependency versions, Scala versions, %% vs %, etc. And when it starts working, I am not quite sure why, or for how long...
It would be great if someone could explain the Scala ecosystem's way(s) of dealing with versions of sbt, scala, and libraries. Or perhaps point me to some documentation.
I struggled with this extensively when i started out. These days i start every project with a boiler-plate build.sbt with just scalaVersion and whatever sbt is currently on my machine:
organization := "foo"
version := "0.1"
scalaVersion := "2.10.4"
Pick the latest 2.10 or 2.11, dependening on your need. Most libraries of note are cross-published into both.
Now, as you find libraries you want to use, head over to http://mvnrepository.com/ and search for them there. Look for a _2.10 or _2.11 postfix (depending on your version). If it has neither, you are likely fine.
Once you find your library and the version you want, mavenrepository even provides you the sbt link you need to use in its sbt tab like this:
libraryDependencies += "com.typesafe.play" % "play-test_2.10" % "2.4.0-M3"
And from there you can even explore the dependencies that library will bring along with it. This should cover most of your day to day needs.
Related
I am facing slowness at many places while working with sbt
Importing SBT Project in Intellij -- approx(8-10 minutes).
Indexing in Intellij of SBT Project.
sbt (In terminal this command takes -- approx(2-3 minutes)).
compile (In sbt shell this command takes -- approx(3-5 minutes)).
5.Whenever I modify build.sbt file then project refresh takes 3-4
minutes.
There are more places i need to check but above specified points i am facing frequently.
Is this problem related to SBT or Scala ?, If yes How to resolve the same
Note : I have good internet connection so this cannot be network issue.
My Scala Class file :
import org.scalatest._
class TaskManagerSpec extends FlatSpec with Matchers {
"An empty tasks list" should "have 0 tasks due today" in {
val tasksDueToday = TaskManager.allTasksDueToday(List())
tasksDueToday should have length 0
}
}
build.sbt
name := "tasky"
version := "0.1.0"
scalaVersion := "2.11.6"
resolvers += "Artima Maven Repository" at "http://repo.artima.com/releases"
libraryDependencies += "org.scalatest" %% "scalatest" % "3.0.0" % "test"
SBT is slow because compiles internal code that is done in Scala and Scala compilation is slow because is a complex language (but once Scala is compiled is a lot faster at runtime)
You can give SBT a boost when using SBT 1.x version with the SBT server. The SBT server allows you to use just one SBT instance shared between command line and the IDE. This is pretty useful, more info here: https://www.scala-sbt.org/1.x/docs/sbt-server.html
Also considere using other build tools that are lighter, like PANTS, that is based on Python which is interpreted and a lot faster. More info at: https://www.pantsbuild.org/
NOTE: the PANTS documentation and community is not as extensive as with SBT but is worth the try, there are amazing things that can be done with PANTS.
NOTE2: if your code base is large it will still take a lot of time to compile/build, so considere to arrange your code and artifacts as incremental/cached pieces/subprojects to see a real boost.
As soon as I bring in a single dependency (using SBT), I see warnings about multiple dependencies.
scalaVersion := "2.10.4"
libraryDependencies ++= Seq(
"org.apache.spark" %% "spark-core" % "1.5.0" % "provided"
)
Warnings:
[warn] Multiple dependencies with the same organization/name but different versions. To avoid conflict, pick one version:
[warn] * org.scala-lang:scala-compiler:(2.10.0, 2.10.4)
[warn] * org.apache.commons:commons-lang3:(3.3.2, 3.0)
[warn] * org.slf4j:slf4j-api:(1.7.10, 1.7.2)
Normally I'm very pedantic about warnings. I want to know why they are there and what I should do to eliminate them. If you let warnings pile up, you quickly have a signal to noise problem.
But how would a novice Scala dev (aka me) know what version to favor?
I'm not really asking how to suppress these warnings, so much as to understand the implications of choosing one version over the other. Seems to me that the source of these warnings is within spark.core, no? How am I to know how to respond?
As soon as I add more dependencies, these warnings pile up and the possibility of a real conflict/problem go up.
I've spent the day trying to find the magic internet search keywords to figure out what to do, but all I'm finding is "how", not "why", if that makes sense.
Thanks.
UPDATE:
Based on this thread, I downgraded from 0.13.8 to 0.13.7. It did get rid of all the noise, though I'm not sure this is really an "answer". But at least I'm able to add all my dependencies without a boatload of warnings, and now sbt-assembly is working as well.
My reading is that minor versions should be forward compatible, so for slf4j it shouldn't matter. The same with the Scala version, moving from 2.10.0 to 2.10.4 shouldn't be a problem (Scala guarantees binary compatibility between minor versions). The only thing that might be a problem here is Apache Commons.
You can use the sbt-dependency-graph plugin to find out more precisely which libraries depend on which versions. Also I think if you run sbt evicted you get more information about conflicting versions.
I have created a libs folder and placed scalascriptengine-1.3.9-2.11.0.jar in there. After that, I right-clicked on the .jar and selected Add Library.
Then, I created Test.scala:
import java.io.File
import com.googlecode.scalascriptengine.ScalaScriptEngine
object Test {
def main(args: Array[String]): Unit = {
val sourceDir = new File("examples/folder")
val sse = ScalaScriptEngine.onChangeRefresh(sourceDir)
}
}
It correctly recognized ScalaScriptEngine, or at least it did not give any warnings or errors. But it did not compile.
According to the library page I edited my build.sbt:
name := "ScalaScriptEngineTest"
version := "1.0"
libraryDependencies += "com.googlecode.scalascriptengine" %% "scalascriptengine" % "1.3.10"
libraryDependencies += "org.scala-lang" % "scala-compiler" % "2.11.1"
But upon refreshing, I get this: http://pastebin.com/GdirttUJ
What am I missing? I am learning scala and it is the first time I am trying to add a library to IntelliJ Idea...
Short answer:
Change the two dependency entries in your build.sbt as follows:
libraryDependencies +=
"com.googlecode.scalascriptengine" % "scalascriptengine" % "1.3.9-2.10.3"
libraryDependencies += "org.scala-lang" % "scala-compiler" % "2.10.4"
Notice I didn't just change the versions -- I replaced your %% with a single %.
This results in you using a slightly older version of ScalaScriptEngine, and I don't know if that will cause any problems for you.
If you're using sbt build dependencies you don't need to be manually placing jars anywhere.
Explanation:
From your log I see that sbt is looking for the ScalaScriptEngine for Scala 2.10. In fact, it's pretty clear that you're running Scala 2.10.4, even though your sbt file expresses a dependency on the 2.11 compiler, which in fact is consistent with the instructions for using ScalaScriptEngine.
On line 23 of the log you can see exactly where it's looking. If you point your browser part way down that path you'll see that there is a version for Scala 2.11 and another directory, scalascriptengine, without a version qualifier. If you dive down the latter, you'll see it's where they keep all the old versions. There isn't a ScalaScriptEngine 1.3.10 (the one you asked for) compiled for Scala 2.10, so your options seem to be to upgrade to Scala 2.11 (which I don't think currently works if you want to use IntelliJ Idea's tight integration with sbt), or you can use ScalaScriptEngine 1.3.9.
You have basically the same problem with your Scala compiler dependency -- it needs the be the Scala version you're using.
I've confirmed the above solution with Scala 2.10.4. I'm playing it a little fast and loose because there isn't a pre compiled version for 2.10.4, and I gambled that the 2.10.3 build will probably work.
Alternatives:
There may be a cleaner way to solve this, but the way the repository is organized makes me doubt it.
You could build the version of your choice with the compiler of your choice, or persuade the ScalaScriptEngine community to do it for you and put it in The Central Repository, but my guess is that 1.3.10 won't build with anything lower than Scala 2.11.
Finally, if you do want to download jars by hand, you may want to read the "Unmanaged dependencies" section of the sbt documentation. Actually, if you're going to use sbt, just read the whole thing a few times.
I come from a background in Ruby, and I use RVM to manage multiple Ruby and gemsets. I have googled around, and I found these two SVM and PVM, not sure what should I use?
Anyone can recommend what should I use to manage multiple scala?
PVM Play Version Manager https://github.com/kaiinkinen/pvm
SVM Scala Version Manager https://github.com/yuroyoro/svm
You don't need a version manager. You need a build tool.
Scala projects work differently than Ruby projects. If you use SBT as a build tool, you specify the Scala version in your build file:
//build.sbt
scalaVersion := "2.11.0" // or some other version
SBT then proceeds to download the specified Scala version for you if it hasn't been downloaded before, and builds and runs your project with this version. If you want, you can even specify which version of SBT you want, and it'll arrange everything for you as well.
This is because Scala, contrary to Ruby, is a compiled language - it must be compiled/built before running. Ruby projects don't have a build process, and can be (attempted to) run on any Ruby version. Scala projects might not build on incompatible versions, let alone run, so you need to specify which Scala version your project is supposed to be built against.
There's also no such thing as gemsets for Scala. For Ruby, gems were originally system-wide libraries and executables, shared by all Ruby scripts on the system. Therefore, gems override each other and you need to maintain gemsets with the specific versions you require for each project. In Scala, a dependency is just a library specifically for your project. They don't override each other, and you just specify which version you need in your build file. SBT then automatically downloads it for you when you build.
This just works:
// myproject1/build.sbt
scalaVersion := "2.10.2"
libraryDependencies += "com.typesafe.akka" %% "akka-actor" % "2.2.0"
// myproject2/build.sbt
scalaVersion := "2.11.0"
libraryDependencies += "com.typesafe.akka" %% "akka-actor" % "2.3.3"
I am cross building a scala project with sbt 12.1.
crossScalaVersions := Seq("2.9.2", "2.10.0")
However, it can't find the dependencies because they are named _2.10 not _2.10.0. It seems like it's regular to name your library 2.10 instead of 2.10.0 with the exception of scala-language and scala-compiler. For example, scalaz is not found at http://repo1.maven.org/maven2/org/scalaz/scalaz-core_2.10.0/6.0.4/scalaz-core_2.10.0-6.0.4.pom but at http://repo1.maven.org/maven2/org/scalaz/scalaz-core_2.10/6.0.4/scalaz-core_2.10-6.0.4.pom.
Is there an easy way to handle this without writing custom rules for all of my dependencies?
The actual build.sbt is available online.
Since 2.10.x releases are binary compatible between each other, libraries need to be built only with one version of scala library - and they can (and must) drop the .0 part (if you publish with sbt, it is done automatically). When the maintainer of a library releases a library with _2.10.0 tag, it's a mistake and you should consider filing a bug.
By the way, I looked on your build.sbt - running +compile on it works for me (sbt 0.12.1). Do you experience some errors?
To get the Scala version incorporated into the artifact name in the Scala way, you specify the dependency with the %% operator:
libraryDependencies += "io.backchat.jerkson" %% "jerkson" % "0.7.0"
When the exact match is not available, you can specify the Scala version explicitly (but remember compatibility only exists across patch releases of a given major/minor release of Scala):
libraryDependencies += "io.backchat.jerkson" % "jerkson_2.9.2" % "0.7.0"