Cannot Create SparkSession for Scala Without an Error in IntelliJ - scala

Question

I am trying to create a SparkSession so I can use implicits._, but I get errors when running a simple app.
My build.sbt file looks like this:
name := "Reddit-Data-Analyser"
version := "0.1"
scalaVersion := "2.11.12"
fork := true
libraryDependencies += "org.mongodb.scala" %% "mongo-scala-driver" % "2.4.0"
resolvers += "MavenRepository" at "http://central.maven.org/maven2"
libraryDependencies ++= Seq(
"org.apache.spark" %% "spark-core" % "2.3.0",
"org.apache.spark" %% "spark-sql" % "2.3.0"
)
I get unresolved dependency errors on spark-sql, but it appears that the SparkSession class can still load.
My Main.scala looks like this:
import org.apache.spark.sql.SparkSession
object main extends App {
val spark = SparkSession
.builder()
.config("spark.master", "local")
//.config("spark.network.timeout", "10000s") //Not Relevant
//.config("spark.executor.heartbeatInterval", "5000s") //Not Relevant
.getOrCreate()
println("Hello World")
spark.stop()
}
*Edit: I actually was able to get the SparkSession to Run by invalidating caches and restarting (though I already did this many times so I am not sure what changed), now when I do ~run in the SBT console I get the [error] messages and have posted this question here about it: SparkSession logging to console with [error] logs.
Below are my old error messages:
The println does not execute, instead I first get the following ERROR output:
[error] (run-main-7) java.lang.AbstractMethodError
java.lang.AbstractMethodError
at org.apache.spark.internal.Logging$class.initializeLogIfNecessary(Logging.scala:99)
at org.apache.spark.sql.internal.SharedState.initializeLogIfNecessary(SharedState.scala:42)
at org.apache.spark.internal.Logging$class.log(Logging.scala:46)
at org.apache.spark.sql.internal.SharedState.log(SharedState.scala:42)
at org.apache.spark.internal.Logging$class.logInfo(Logging.scala:54)
at org.apache.spark.sql.internal.SharedState.logInfo(SharedState.scala:42)
at org.apache.spark.sql.internal.SharedState.<init>(SharedState.scala:71)
at org.apache.spark.sql.SparkSession$$anonfun$sharedState$1.apply(SparkSession.scala:112)
at org.apache.spark.sql.SparkSession$$anonfun$sharedState$1.apply(SparkSession.scala:112)
at scala.Option.getOrElse(Option.scala:121)
at org.apache.spark.sql.SparkSession.sharedState$lzycompute(SparkSession.scala:112)
at org.apache.spark.sql.SparkSession.sharedState(SparkSession.scala:111)
at org.apache.spark.sql.internal.BaseSessionStateBuilder.build(BaseSessionStateBuilder.scala:284)
at org.apache.spark.sql.SparkSession$.org$apache$spark$sql$SparkSession$$instantiateSessionState(SparkSession.scala:1050)
at org.apache.spark.sql.SparkSession$$anonfun$sessionState$2.apply(SparkSession.scala:130)
at org.apache.spark.sql.SparkSession$$anonfun$sessionState$2.apply(SparkSession.scala:130)
at scala.Option.getOrElse(Option.scala:121)
at org.apache.spark.sql.SparkSession.sessionState$lzycompute(SparkSession.scala:129)
at org.apache.spark.sql.SparkSession.sessionState(SparkSession.scala:126)
at org.apache.spark.sql.SparkSession$Builder$$anonfun$getOrCreate$5.apply(SparkSession.scala:938)
at org.apache.spark.sql.SparkSession$Builder$$anonfun$getOrCreate$5.apply(SparkSession.scala:938)
at scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:99)
at scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:99)
at scala.collection.mutable.HashTable$class.foreachEntry(HashTable.scala:230)
at scala.collection.mutable.HashMap.foreachEntry(HashMap.scala:40)
at scala.collection.mutable.HashMap.foreach(HashMap.scala:99)
at org.apache.spark.sql.SparkSession$Builder.getOrCreate(SparkSession.scala:938)
at controller.main$.delayedEndpoint$controller$main$1(Main.scala:20)
at controller.main$delayedInit$body.apply(Main.scala:11)
at scala.Function0$class.apply$mcV$sp(Function0.scala:34)
at scala.runtime.AbstractFunction0.apply$mcV$sp(AbstractFunction0.scala:12)
at scala.App$$anonfun$main$1.apply(App.scala:76)
at scala.App$$anonfun$main$1.apply(App.scala:76)
at scala.collection.immutable.List.foreach(List.scala:381)
at scala.collection.generic.TraversableForwarder$class.foreach(TraversableForwarder.scala:35)
at scala.App$class.main(App.scala:76)
at controller.main$.main(Main.scala:11)
at controller.main.main(Main.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
[trace] Stack trace suppressed: run last compile:run for the full output.
java.lang.RuntimeException: Nonzero exit code: 1
at scala.sys.package$.error(package.scala:27)
[trace] Stack trace suppressed: run last compile:run for the full output.
[error] (compile:run) Nonzero exit code: 1
[error] Total time: 9 s, completed Mar 14, 2019 9:43:29 PM
8. Waiting for source changes... (press enter to interrupt)
19/03/14 21:43:29 INFO AsyncEventQueue: Stopping listener queue executorManagement.
java.lang.InterruptedException
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2014)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2048)
at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
at org.apache.spark.scheduler.AsyncEventQueue$$anonfun$org$apache$spark$scheduler$AsyncEventQueue$$dispatch$1.apply(AsyncEventQueue.scala:94)
at scala.util.DynamicVariable.withValue(DynamicVariable.scala:58)
at org.apache.spark.scheduler.AsyncEventQueue.org$apache$spark$scheduler$AsyncEventQueue$$dispatch(AsyncEventQueue.scala:83)
at org.apache.spark.scheduler.AsyncEventQueue$$anon$1$$anonfun$run$1.apply$mcV$sp(AsyncEventQueue.scala:79)
at org.apache.spark.util.Utils$.tryOrStopSparkContext(Utils.scala:1319)
at org.apache.spark.scheduler.AsyncEventQueue$$anon$1.run(AsyncEventQueue.scala:78)
19/03/14 21:43:29 INFO AsyncEventQueue: Stopping listener queue appStatus.
java.lang.InterruptedException
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2014)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2048)
at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
at org.apache.spark.scheduler.AsyncEventQueue$$anonfun$org$apache$spark$scheduler$AsyncEventQueue$$dispatch$1.apply(AsyncEventQueue.scala:94)
at scala.util.DynamicVariable.withValue(DynamicVariable.scala:58)
at org.apache.spark.scheduler.AsyncEventQueue.org$apache$spark$scheduler$AsyncEventQueue$$dispatch(AsyncEventQueue.scala:83)
at org.apache.spark.scheduler.AsyncEventQueue$$anon$1$$anonfun$run$1.apply$mcV$sp(AsyncEventQueue.scala:79)
at org.apache.spark.util.Utils$.tryOrStopSparkContext(Utils.scala:1319)
at org.apache.spark.scheduler.AsyncEventQueue$$anon$1.run(AsyncEventQueue.scala:78)
19/03/14 21:43:29 ERROR ContextCleaner: Error in cleaning thread
java.lang.InterruptedException
at java.lang.Object.wait(Native Method)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143)
at org.apache.spark.ContextCleaner$$anonfun$org$apache$spark$ContextCleaner$$keepCleaning$1.apply$mcV$sp(ContextCleaner.scala:181)
at org.apache.spark.util.Utils$.tryOrStopSparkContext(Utils.scala:1319)
at org.apache.spark.ContextCleaner.org$apache$spark$ContextCleaner$$keepCleaning(ContextCleaner.scala:178)
at org.apache.spark.ContextCleaner$$anon$1.run(ContextCleaner.scala:73)

Answer 1

Have you tried something like:
import org.apache.spark.sql.SparkSession
object main extends App {
val spark = SparkSession
.builder()
.appName("myApp")
.config("master", "local[*]")
.getOrCreate()
println("Hello World")
println(spark.version())
spark.stop()
}

Answer 2

So I am not sure exactly what fixed the problem because after doing a number of sbt commands and changes I was eventually able to run my app.
Here is a list of things I did, but I think the sbt command in step #4 may have been the missing piece:
Changed from version := 2.12.5 to version := 2.11.12 in build.sbt. I believe that Apache spark has support for scala 2.12, but IntelliJ or sbt apparently has difficulties retrieving the packages.
Created file build.properties under project root directory and added line sbt.version = 0.13.17, since sbt 1.0 apparently isn't great at working with spark-core repository.
Ran the following sbt commands in this order: reload plugins,update,reload.
One of the last things I tried was running the sbt command package which Creates a jar file containing the files in src/main/resources and the classes compiled from src/main/scala and src/main/java. After doing this (and maybe a full Rebuild/Cache Invalidation) I noticed the missing Scala packages appeared in my External Libraries.
Did Rebuild and Invalidate Cache/Restart several times.