I am trying to load an xml file using scala-xml_2.12-1.0.6.jar but it gives me NullPointerEexception while loading
Following is my line of code to load xml
import scala.xml.XML
val xml = XML.loadFile("sample.xml")
I have decompiled this jar and is method is present in that jar but for some reasons it is unable to find it in code.
I have Scala 2.13.1 on my system but for this project I am using scala 2.12.1 and it is mentioned in mu built.sbt
scalaVersion := "2.12.1"
I have following dependency in my built.sbt for this xml package
libraryDependencies += "org.scala-lang.modules" %% "scala-xml" % "1.0.6"
If I copy and paste the same code to Scala interactive shell( scala 2.13.1) I get following error
import scala.xml.XML
^
error: object xml is not a member of package scala
did you mean Nil?
Can anyone please identify what am i doing wrong?
Thanks in advance.
I'm not sure how you are loading up the Scala REPL, but as mentioned in "How to use third party libraries with Scala REPL?", you should be launching the the REPL from SBT with sbt console. Your .sbt files will also need to be in scope.
The Scala REPL independently does not deal with .sbt files. They are 2 different tools.
Alternatively you could also install Ammonite-REPL which supports Magic Imports. You will be able to import Maven dependencies with import $ivy.
Scala 2.12 comes with scala-xml and you can remove that dependency as long as you run REPL with sbt console and not your native Scala REPL which is already # Scala 2.13. Otherwise you can also switch to Scala 2.13 for your SBT project.
Related
I am a noob to Spark and Intellij.I want to run Spark using Scala
I initially installed Scala 2.12 and created the SBT accordingly.Then I got a NoSuchMethod runtime error on
sc = new SparkContext(conf)
From the solution posted NoSuchMethodError when using Sparka and IntelliJ I used Scala version 2.11.3 while creating the project and used the SBT
version := "0.1"
scalaVersion := "2.11.3"
libraryDependencies += "org.apache.spark" % "spark-core_2.11" % "2.0.2"
I am now getting the error
Error:scalac: No 'scala-library*.jar' in Scala compiler classpath in Scala SDK SBT: org.scala-lang:scala-library:2.11.3:jar
This is the library on the External libraries section
I tried creating the project from scratch and Cache Invalidate/Restart option.Same result
I also tried downloading via Maven through File -> Project Structure.Only found spark-core 2.10.Showed the same NoSuchMethod Error
Can anyone identify the problem?
It clearly says that, It could not find scala-library*.jar file
So go to
"Project Structure -> Modules"
And see these jar files
scala-compiler.jar, scala-library.jar, scala-reflect.jar
If they are absent from the Modules add them manually or Reinstall the scala and scala plugin.
Hope this should work!
I'm having problems with my project which I think are due to intellij pulling in a load of scala 2.10 libraries when it created/if I reimport my project.
How can I configure it to either read the version of scala from build.sbt, or manually configure it to compile with scala 2.11.7? I'm running intellij IDEA 14 with the latest updates and it always uses scala 2.10.4.
Make sure you have your Scala Version set in your build.sbt
scalaVersion := "2.11.7"
The rest works seemlessly. Enable "Auto-Import" in IDEA when creating the project.
The 2.10 libraries are used by SBT itself. If you check the Dependencies tab for your modules, you shouldn't see them. If you do, you likely have an error in libraryDependencies in build.sbt.
I downloaded eclipse scala ide from scala-ide.org site and trying to compile my first scala word count program. But its gives error "object not a member of package org" in the following import command
import org.apache.spark.SparkContext
import org.apache.spark.SparkContext._
After some research I found that I need to add the jar file spark-assembly-1.0.0-hadoop2.2.0.jar to overcome this issue
But after doing lot of research I could not locate this jar. Can anyone help here ?
Install the SBT Scala build+dependency tool
Create an empty directory. Name it spark-test or whatever you want to name your project.
Put your source code in the sub-directory src/scala/main. If you have Main.scala in package scalatest, it should be src/scala/main/scalatest/Main.scala
Make a build.sbt file with the following contents
name := """sparktest"""
version := "1.0-SNAPSHOT"
scalaVersion := "2.11.7"
libraryDependencies ++= Seq(
"org.apache.spark" %% "spark-core" % "1.4.0"
)
Configure the SBT Eclipse plugin. Create ~/.sbt/0.13/plugins/plugins.sbt, with:
addSbtPlugin("com.typesafe.sbteclipse" % "sbteclipse-plugin" % "4.0.0")
Generate an Eclipse project with sbt eclipse
Import your project into eclipse.
Run :)
Scala is not a simple language/env to learn. It is important you learn how scala works and then move into spark.
There are tons of material available on web. A proper learning path will be to learn
SBT > SCALA > Use Scala for Spark
The dependency that you mentioned, can be put in he sbt's build.sbt. You can also use maven, but I recommend learning sbt as way of learning scala. Once you have resolved, the dependency using SBT, your simple code should work fine. But still, I recommend doing a "hello world" first than doing a "word count" :-)
Ando to answer your question, in your SBT you should be adding following library,
libraryDependencies += "org.apache.spark" % "spark-assembly_2.10" % "1.1.1"
This was for spark assembly 1.1.1 for hadoop 2.10. I see you need a different version, you can find the proper version at
Maven Repo details for spark/hadoop
Here's the pure eclipse solutions (I had to download and setup eclipse just to answer this question)
Get Scala IDE (it comes with inbuilt Scala Compiler version 2.10.5 and 2.11.6)
Create a new project Scala Wizard > Scala Project
Right click "src" in the scale project , select "Scala Object", give it a name - I gave WordCount
Right click project > Configure > Convert to Maven Project
In the body of word count object (I named the object as WordCount) paste the text from Apache Spark Example which finally looks like
```
import org.apache.spark.SparkContext
import org.apache.spark.SparkContext._
import org.apache.spark.SparkConf
object WordCount {
val sparkConf = new SparkConf().setAppName("SampleTest")
val spark = new SparkContext(sparkConf)
val textFile = spark.textFile("hdfs://...")
val counts = textFile.flatMap(line => line.split(" "))
.map(word => (word, 1))
.reduceByKey(_ + _)
counts.saveAsTextFile("hdfs://...")
}
```
6. Add following to your maven
```
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-core_2.10</artifactId>
<version>1.4.0</version>
</dependency>
```
Right click on "Scala Library Container", and select "Latest 2.10 bundle", click ok
Once I was done with these, there was no error message displayed on my "problems" list of Eclipse...indicating that it compiled as expected.
Obviously, this example won't run as I haven't provided enough info for it to run...but this was just to answer to the question, "how to get the code compiled".
Hope this helps.
I have created a libs folder and placed scalascriptengine-1.3.9-2.11.0.jar in there. After that, I right-clicked on the .jar and selected Add Library.
Then, I created Test.scala:
import java.io.File
import com.googlecode.scalascriptengine.ScalaScriptEngine
object Test {
def main(args: Array[String]): Unit = {
val sourceDir = new File("examples/folder")
val sse = ScalaScriptEngine.onChangeRefresh(sourceDir)
}
}
It correctly recognized ScalaScriptEngine, or at least it did not give any warnings or errors. But it did not compile.
According to the library page I edited my build.sbt:
name := "ScalaScriptEngineTest"
version := "1.0"
libraryDependencies += "com.googlecode.scalascriptengine" %% "scalascriptengine" % "1.3.10"
libraryDependencies += "org.scala-lang" % "scala-compiler" % "2.11.1"
But upon refreshing, I get this: http://pastebin.com/GdirttUJ
What am I missing? I am learning scala and it is the first time I am trying to add a library to IntelliJ Idea...
Short answer:
Change the two dependency entries in your build.sbt as follows:
libraryDependencies +=
"com.googlecode.scalascriptengine" % "scalascriptengine" % "1.3.9-2.10.3"
libraryDependencies += "org.scala-lang" % "scala-compiler" % "2.10.4"
Notice I didn't just change the versions -- I replaced your %% with a single %.
This results in you using a slightly older version of ScalaScriptEngine, and I don't know if that will cause any problems for you.
If you're using sbt build dependencies you don't need to be manually placing jars anywhere.
Explanation:
From your log I see that sbt is looking for the ScalaScriptEngine for Scala 2.10. In fact, it's pretty clear that you're running Scala 2.10.4, even though your sbt file expresses a dependency on the 2.11 compiler, which in fact is consistent with the instructions for using ScalaScriptEngine.
On line 23 of the log you can see exactly where it's looking. If you point your browser part way down that path you'll see that there is a version for Scala 2.11 and another directory, scalascriptengine, without a version qualifier. If you dive down the latter, you'll see it's where they keep all the old versions. There isn't a ScalaScriptEngine 1.3.10 (the one you asked for) compiled for Scala 2.10, so your options seem to be to upgrade to Scala 2.11 (which I don't think currently works if you want to use IntelliJ Idea's tight integration with sbt), or you can use ScalaScriptEngine 1.3.9.
You have basically the same problem with your Scala compiler dependency -- it needs the be the Scala version you're using.
I've confirmed the above solution with Scala 2.10.4. I'm playing it a little fast and loose because there isn't a pre compiled version for 2.10.4, and I gambled that the 2.10.3 build will probably work.
Alternatives:
There may be a cleaner way to solve this, but the way the repository is organized makes me doubt it.
You could build the version of your choice with the compiler of your choice, or persuade the ScalaScriptEngine community to do it for you and put it in The Central Repository, but my guess is that 1.3.10 won't build with anything lower than Scala 2.11.
Finally, if you do want to download jars by hand, you may want to read the "Unmanaged dependencies" section of the sbt documentation. Actually, if you're going to use sbt, just read the whole thing a few times.
I downloaded Scala 2.10.2, unpacked it and run Scala command, I can successfully import akka._.
In another experiment, I create an sbt project with the following line in build.sbt:
scalaVersion := "2.10.2"
A source file import akka._, and sbt complains "akka not found".
What is the difference betwen SBT's Scala 2.10.2 and the one on Scala website? And why does the official Scala already include Akka library, but SBT's Scala does not?
Akka is a part of the Scala Distribution (the zip you downloaded) but not the Scala Standard Library — which is what you get in SBT.