I am trying to learning Scala with Spark. I am following a tutorial but I am having an error, when I try to import the library dependencies of Spark:
libraryDependencies += "org.apache.spark" %% "spark-core" % "2.4.3"
I am getting the following error:
And I have 3 Unkwons artifacts.
What could be the problem here?
My code is so simple, it is just a Hello World.
Probably you need to add to your build.sbt:
resolvers += "spark-core" at "https://mvnrepository.com/artifact/org.apache.spark/spark-core"
Please note that this library is supported only for Scala 2.11 and Scala 2.12.
Related
I've read this article and copipasted the Sort-Merge Join Example, but when I'm trying to build the project I'm getting the following error:
object SparkSession is not a member of package org.apache.spark.sql
I've seen many questions about this error, and the answers were that they used an old version of Spark. However I mentioned in build.sbt version 2.1 of Spark as they use in the example on that website.
Here is my build.sbt:
name := "Simple Project"
version := "1.0"
scalaVersion := "2.11.8"
libraryDependencies += "org.apache.spark" %% "spark-core" % "2.1.0"
What am I missing?
Spark SQL dependency is missing
libraryDependencies += "org.apache.spark" %% "spark-sql" % 2.1.0
I am trying to use json4s but I keep getting the error below when I compile. I thought the library would have the class internally. I am using json4s-ast_2.11-4.0.0-M1.jar, json4s-core_2.12.3.5.3.jar and json4s-jackson_2.9.1-3.0.0.jar.
Error:(64, 25) Symbol 'type org.json4s.JsonAST.JValue' is missing from the classpath.
This symbol is required by 'type org.json4s.JValue'.
Make sure that type JValue is in your classpath and check for conflicting dependencies with `-Ylog-classpath`.
A full rebuild may help if 'package.class' was compiled against an incompatible version of org.json4s.JsonAST.
Imports:
import org.json4s.jackson.JsonMethods._
import org.json4s._
build.sbt
scalaVersion := "2.12.4"
libraryDependencies += "org.json4s" %% "json4s-jackson" % "3.0.0"
libraryDependencies += "org.json4s" %% "json4s-ast" % "4.0.0-M1"
libraryDependencies += "org.json4s" %% "json4s-core" % "3.5.3"
You can't use libraries compiled for different versions of scala in other versions. You need to fix the versioning of your JSON4s. Look at:
http://www.scala-sbt.org/0.13/docs/Cross-Build.html#Using+Cross-Built+Libraries
I am trying to setup IntelliJ for spark 2.11 but it is very daunting and after days I have not been able to compile a simple instruction such as with "spark.read.format" which is not found in main core and sql spark libraries.
I have seen a few posts on the subject but with none resolved. Does anyone have some experience with perhaps a working sample program I can start with?
Could it be that it would be easier with Eclipse?
Many thanks in advance for your answers,
EZ
build project in Intellij using with scala 2.11 and sbt 0.13: then ensure that your plugins.sbt contains as below:
logLevel := Level.Warn
addSbtPlugin("com.eed3si9n" % "sbt-assembly" % "0.13.0")
then your build.sbt must contain as below:
scalaVersion := "2.11.8"
val sparkVersion = "2.1.0"
libraryDependencies += "org.apache.spark" %% "spark-core" % sparkVersion %"provided"
libraryDependencies += "org.apache.spark" %% "spark-sql" % sparkVersion %"provided"
Then write your code, click Terminal in Intellij and type sbt assembly: you can ship that jar to remote cluster, otherwise run from Intelij locally, let me know how it goes
I try to import the scala.util.parsing library and the Eclipse compiler doesn't recognize it.
How can I fix it?
scala.util.parsing was removed from the standard library in Scala 2.11. Add this to your dependencies:
libraryDependencies += "org.scala-lang.modules" %% "scala-parser-combinators" % "1.0.2"
Use this version if you're using Scala 2.13:
libraryDependencies += "org.scala-lang.modules" %% "scala-parser-combinators" % "1.1.2"
The imports are the same, so you can use import scala.util.parsing.json._ for example.
See here for all the published versions of this lib.
I have a Play framework based webapp which has the following defined in its build.sbt file:
....
version := "1.0-SNAPSHOT"
resolvers += "Sonatype Snapshots" at "http://oss.sonatype.org/content/repositories/snapshots/"
resolvers += "Sonatype Releases" at "https://oss.sonatype.org/content/groups/scala-tools"
resolvers += "Novus Releases" at "http://repo.novus.com/releases/"
libraryDependencies ++= Seq(
jdbc,
anorm,
cache,
"com.mongodb.casbah" % "casbah_2.9.0" % "2.2.0-SNAPSHOT",
"com.novus" %% "salat-core" % "1.9.2",
"org.scalatest" % "scalatest_2.10" % "2.0" % "test",
"com.typesafe" % "config" % "1.0.2"
)
....
The Scala version is 2.10.3 and when I try to run a unit test, I run into the following error:
A needed class was not found. This could be due to an error in your runpath. Missing class: scala/reflect/ClassManifest
java.lang.NoClassDefFoundError: scala/reflect/ClassManifest
at com.mongodb.casbah.Imports$.<init>(Implicits.scala:113)
at com.mongodb.casbah.Imports$.<clinit>(Implicits.scala)
at com.mongodb.casbah.MongoConnection.apply(MongoConnection.scala:177)
........
........
I'm completely clueless as to why this is happening? Which additional library is that I'm missing?
You can't mix major scala versions (see, casbah artifact is compiled against scala 2.9.*, whereas scala_test is for 2.10.*, and you're saying you use 2.10 in intellij).
The error says, that compiler can't find class that was cut out from scala library since 2.9.* times and solution is to pick proper scala version (any 2.10.* will fit).
Salat author here. The solution is to fix your deps. The latest stable release of Salat is 1.9.4, and it depends on Casbah 2.6.4 - both are available for 2.9 and 2.10.
Instead of using the casbah driver directly for play I have fallen back into using a plugin called play-salat (https://github.com/leon/play-salat).
The one I use at the moment is "se.radley" % "play-plugins-salat_2.10" % "1.3.0" and works well in Play 2.2.