I am trying to run a simple scala program in IntelliJ.
My build.sbt looks like this:
name := "UseThis"
version := "0.1"
scalaVersion := "2.12.4"
libraryDependencies += "org.apache.spark" %% "spark-core" % "1.2.0"
And my import code looks like this:
package WorkWork.FriendsByAge
import com.sun.glass.ui.Window.Level
import com.sun.istack.internal.logging.Logger
import org.apache.spark._
import org.apache.spark.SparkContext._
import org.apache.log4j._
I don't get why the import fails. It tells me the dependency failed to load or wasn't found, but I put the line in the build.sbt as required. Is there some other step I need to have done? I've installed spark. Is it the version note at the end of the build line? I don't even know how to check what version I have of spark.
I'm trying to teach myself Scala (not a noob though, I know Python, R, various flavors of SQL, C#) but my word even setting it up is nigh on impossible, and apparently getting it to even run is too. Any ideas?
Take a look at this page here: Maven Central (Apache Spark core)
Unless you have set up some other repositories, the dependencies that are going to be loaded by sbt usually come from there.
There is a version column with numbers like 2.2.1, then there comes a scala column with numbers like 2.11, 2.10. In order for spark and scala to work together, you have to pick a valid combination from this table.
As of 28.Feb 2018, there are no versions of Spark that work with scala 2.12.4. The latest version of scala for which 1.2.0 works is 2.11. So, you will probably want to set scala version to 2.11.
Also note that the %% syntax in your SBT in
"org.apache.spark" %% "spark-core" % "1.2.0"
will automatically append the suffix _2.12 to the artifact-id-part. Since there is no spark-core_2.12, it cannot resolve the dependency, and you can't import anything in your code.
By the way: there was a big difference between spark 1.2 and spark 1.3, and then there was again a big difference between 1.x and 2.x. Does it really have to be 1.2?
Its because Spark 1.2 is not available for Scala 2.12
https://mvnrepository.com/artifact/org.apache.spark/spark-core
Related
I am trying to load an xml file using scala-xml_2.12-1.0.6.jar but it gives me NullPointerEexception while loading
Following is my line of code to load xml
import scala.xml.XML
val xml = XML.loadFile("sample.xml")
I have decompiled this jar and is method is present in that jar but for some reasons it is unable to find it in code.
I have Scala 2.13.1 on my system but for this project I am using scala 2.12.1 and it is mentioned in mu built.sbt
scalaVersion := "2.12.1"
I have following dependency in my built.sbt for this xml package
libraryDependencies += "org.scala-lang.modules" %% "scala-xml" % "1.0.6"
If I copy and paste the same code to Scala interactive shell( scala 2.13.1) I get following error
import scala.xml.XML
^
error: object xml is not a member of package scala
did you mean Nil?
Can anyone please identify what am i doing wrong?
Thanks in advance.
I'm not sure how you are loading up the Scala REPL, but as mentioned in "How to use third party libraries with Scala REPL?", you should be launching the the REPL from SBT with sbt console. Your .sbt files will also need to be in scope.
The Scala REPL independently does not deal with .sbt files. They are 2 different tools.
Alternatively you could also install Ammonite-REPL which supports Magic Imports. You will be able to import Maven dependencies with import $ivy.
Scala 2.12 comes with scala-xml and you can remove that dependency as long as you run REPL with sbt console and not your native Scala REPL which is already # Scala 2.13. Otherwise you can also switch to Scala 2.13 for your SBT project.
i know that it exist cross-build options to generate various version of a scala API running with different scala version. Let's say i will stay with the same scala version 2.11.12, how can i set my build.sbt to handle multiple version of spark. I have some hint about the "provided" option for dependencies but i'm not sure it is the best way to handle it.
Bonus: what if some spark version are using 2.11 and others 2.12...
Please let us know if you've already ridden through this issue.
"org.apache.spark" %% "spark-core" % "2.3.2"
Use the double % -sign when defining dependencies, sbt then will handle the scala version for you. You should use the same scala version for all your dependencies.
I'm new to Scala Spark and I'm trying to create an example project using Intellij. During Project creation I choose Scala and Sbt with Scala version 2.12 but When I tried adding spark-streaming version 2.3.2 if kept erroring out so I Google'd around and on Apache's website I found the sbt config shown below and I'm still getting the same error.
Error: Could not find or load main class SparkStreamingExample
Caused by: java.lang.ClassNotFoundException: SparkStreamingExample
How can it be determined which version of Scala works with which version of Spark Dependencies?
name := "SparkStreamExample"
version := "0.1"
scalaVersion := "2.11.8"
libraryDependencies ++= Seq(
"org.apache.spark" % "spark-streaming_2.11" % "2.3.2"
)
My Object class is very basic doesn't have much to it...
import org.apache.spark.SparkConf
import org.apache.spark.streaming.StreamingContext
object SparkStreamingExample extends App {
println("SPARK Streaming Example")
}
You can see the version of Scala that is supported by Spark in the Spark documentation.
As of this writing, the documentation says:
Spark runs on Java 8+, Python 2.7+/3.4+ and R 3.1+. For the Scala API, Spark 2.3.2 uses Scala 2.11. You will need to use a compatible Scala version (2.11.x).
Notice that only Scala 2.11.x is supported.
I am trying to run a scala project with spark. I did the following in my file:
package com.sparksql.count
import org.apache.log4j.Level
import org.apache.log4j.Logger
import org.apache.spark.SparkConf
import org.apache.spark._
However it told me that log4j and spark cant be found. I did a bit of research and figured it must have been the sbt. I then went to the SBT and i added in the library dependency as below
name := "SampleLearning"
version := "0.1"
scalaVersion := "2.12.4"
libraryDependencies += "org.apache.spark" %% "spark-core" % "1.2.0"
However, it still cant work and says there is something wrong with librayDependencies. Can anyone help?
Spark doesn't support Scala 2.12. To work with Spark use Scala 2.11
Spark 1.2 hasn't been supported in years. Use either 2.x branch (latest is 2.3.0) or (for legacy application) latest 1.x release (1.6.3) but it is quite outdated today.
I have created a libs folder and placed scalascriptengine-1.3.9-2.11.0.jar in there. After that, I right-clicked on the .jar and selected Add Library.
Then, I created Test.scala:
import java.io.File
import com.googlecode.scalascriptengine.ScalaScriptEngine
object Test {
def main(args: Array[String]): Unit = {
val sourceDir = new File("examples/folder")
val sse = ScalaScriptEngine.onChangeRefresh(sourceDir)
}
}
It correctly recognized ScalaScriptEngine, or at least it did not give any warnings or errors. But it did not compile.
According to the library page I edited my build.sbt:
name := "ScalaScriptEngineTest"
version := "1.0"
libraryDependencies += "com.googlecode.scalascriptengine" %% "scalascriptengine" % "1.3.10"
libraryDependencies += "org.scala-lang" % "scala-compiler" % "2.11.1"
But upon refreshing, I get this: http://pastebin.com/GdirttUJ
What am I missing? I am learning scala and it is the first time I am trying to add a library to IntelliJ Idea...
Short answer:
Change the two dependency entries in your build.sbt as follows:
libraryDependencies +=
"com.googlecode.scalascriptengine" % "scalascriptengine" % "1.3.9-2.10.3"
libraryDependencies += "org.scala-lang" % "scala-compiler" % "2.10.4"
Notice I didn't just change the versions -- I replaced your %% with a single %.
This results in you using a slightly older version of ScalaScriptEngine, and I don't know if that will cause any problems for you.
If you're using sbt build dependencies you don't need to be manually placing jars anywhere.
Explanation:
From your log I see that sbt is looking for the ScalaScriptEngine for Scala 2.10. In fact, it's pretty clear that you're running Scala 2.10.4, even though your sbt file expresses a dependency on the 2.11 compiler, which in fact is consistent with the instructions for using ScalaScriptEngine.
On line 23 of the log you can see exactly where it's looking. If you point your browser part way down that path you'll see that there is a version for Scala 2.11 and another directory, scalascriptengine, without a version qualifier. If you dive down the latter, you'll see it's where they keep all the old versions. There isn't a ScalaScriptEngine 1.3.10 (the one you asked for) compiled for Scala 2.10, so your options seem to be to upgrade to Scala 2.11 (which I don't think currently works if you want to use IntelliJ Idea's tight integration with sbt), or you can use ScalaScriptEngine 1.3.9.
You have basically the same problem with your Scala compiler dependency -- it needs the be the Scala version you're using.
I've confirmed the above solution with Scala 2.10.4. I'm playing it a little fast and loose because there isn't a pre compiled version for 2.10.4, and I gambled that the 2.10.3 build will probably work.
Alternatives:
There may be a cleaner way to solve this, but the way the repository is organized makes me doubt it.
You could build the version of your choice with the compiler of your choice, or persuade the ScalaScriptEngine community to do it for you and put it in The Central Repository, but my guess is that 1.3.10 won't build with anything lower than Scala 2.11.
Finally, if you do want to download jars by hand, you may want to read the "Unmanaged dependencies" section of the sbt documentation. Actually, if you're going to use sbt, just read the whole thing a few times.