java.lang.InterruptedException when creating SparkSession in Scala - scala

Question

If I clone this gist: https://gist.github.com/jamiekt/cea2dab3ea8de91489b31045b302e011
and then issue sbt run it fails on the line
val spark = SparkSession.builder()
.config(new SparkConf().setMaster("local[*]"))
.enableHiveSupport()
.getOrCreate()
with error:
Java.lang.InterruptedException
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2014)
No clue why this might be happening. Anyone got a suggestion?
Scala version is 2.11.12 (see in build.sbt in the gist)
Spark version is 2.3.0 (again, see in build.sbt)
Java Version
$ java -version
java version "1.8.0_161"

Answer 1

The error is because you have not stopped the sparkSession instance created and the instance is removed from memory without being closed as soon as sbt run completes i.e. after the successful completion of your code.
So all you require is
spark.stop()
at the end of the scope where the instance is created as
object Application extends App{
import DataFrameExtensions_._
val spark = SparkSession.builder().config(new SparkConf().setMaster("local[*]")).enableHiveSupport().getOrCreate()
//import spark.implicits._
//val df = Seq((8, "bat"),(64, "mouse"),(-27, "horse")).toDF("number", "word")
//val groupBy = Seq("number","word")
//val asAt = LocalDate.now()
//val joinedDf = Seq(df.featuresGroup1(_,_), df.featuresGroup2(_,_)).map(_(groupBy, asAt)).joinDataFramesOnColumns(groupBy)
//joinedDf.show
spark.stop()
}
Just before the
Java.lang.InterruptedException at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2014)
You must have following message too
ERROR Utils: uncaught error in thread SparkListenerBus, stopping SparkContext
which gives clue to the cause of the error.