use spark sql through Scala IDE - scala

Question

i want to try spark sql , i used at first the bin/spark-shell
inserting this code
val sqlcontext=new org.apache.spark.sql.SQLContext(sc)
val data=sc.textFile("hdfs://localhost:9000/cars.csv")
val mapr=data.map (p => p.split(','))
val MyMatchRDD=mapr.map(p =>MyMatch(p(0).toString(),p(1).toString(),p(2).toString(),
p(3).toString(),p(4).toString(),p(5).toString(),p(6).toString(),p(7).toString(),
p(8).toString()))
import sqlcontext.implicits._
val personDF=MyMatchRDD.toDF()
personDF.registerTempTable("Person")
val res = sqlcontext.sql("SELECT * FROM Person")
res.collect().foreach(println)
i didn't get any issue ,all is good.
But when i used the scala ide
i used in pom file (maven)
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-core_2.10</artifactId>
<version>1.3.0</version>
</dependency>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-catalyst_2.10</artifactId>
<version>1.3.0</version>
</dependency>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-sql_2.10</artifactId>
<version>1.3.0</version>
</dependency>
and i used the same code
import org.apache.spark.SparkConf
import org.apache.spark.SparkContext._
import org.apache.spark.SparkContext
import org.apache.spark.sql.SQLContext
import org.slf4j.Logger
import org.slf4j.LoggerFactory;
object SparkSQL {
case class MyMatch( col1: String, col2: String,col3: String, col4 :String ,col5: String,
col6: String,col7 :String ,col8: String,
col9: String)
def main(args:Array[String]) {
val sparkConf = new SparkConf().setAppName("HiveFromSpark").setMaster("local")
val sc = new SparkContext(sparkConf)
val sqlcontext=new org.apache.spark.sql.SQLContext(sc)
val data=sc.textFile("hdfs://localhost:9000/cars.csv")
val mapr=data.map (p => p.split(','))
val MyMatchRDD=mapr.map(p =>MyMatch(p(0).toString(),p(1).toString(),p(2).toString(),p(3).toString(),
p(4).toString(),p(5).toString(),p(6).toString(),p(7).toString(),
p(8).toString()) )
import sqlcontext.implicits._
val personDF=MyMatchRDD.toDF()
personDF.registerTempTable("Person")
val res = sqlcontext.sql("SELECT * FROM Person")
res.collect().foreach(println)
}
}
i got this issue
Exception in thread "main"
scala.reflect.internal.MissingRequirementError: class
org.apache.spark.sql.catalyst.ScalaReflection in JavaMirror with
primordial classloader with boot classpath
[D:\scala-SDK-4.4.1-vfinal-2.11-win32.win32.x86_64\eclipse\plugins\org.scala-
Thanks in advance for your help

Answer 1

You are using an wrong Scala version - Spark's compiled with Scala version 2.10. Check your runtime and compiler Scala version.
Why you're using so old dependencies? Spark has version 2.0.2 right now with Scala 2.11
Recommended actions:
(optional) Change <version>1.3.0</version> to <version>2.0.2</version>
In your Scala compiler, change version to 2.11 (if updated to Spark 2) or 2.10 (if you still use Spark 1)
Make sure you have proper Scala version installed on your machine - 2.11 in case of Spark 2, 2.10 in case of Spark 1. You can check Scala version by typing scala -version in console
Make sure your Scala IDE supports Scala version that was choosen