distribute sparkContext error on yarn cluster - scala

Question

My code works in local mode, but with yarn (client or cluster mode), it stops wit this error :
Exception in thread "main" org.apache.spark.SparkException: Job aborted due to stage failure: Task 1 in stage 0.0 failed 4 times, most recent failure: Lost task 1.3 in stage 0.0 (TID 6, hadoopdatanode, executor 1): java.io.IOException: java.lang.NullPointerException
at org.apache.spark.util.Utils$.tryOrIOException(Utils.scala:1353)
at org.apache.spark.rdd.ParallelCollectionPartition.readObject(ParallelCollectionRDD.scala:70)
I don't understand why it works in local mode but not with yarn. The problem comes with the declaration of the sparkContext inside the rdd.foreach.
I need a sparContext inside the executeAlgorithm, and because a sparcontext is not serializable i have to get it inside the rdd.foreach
here is my main object :
def main(args: Array[String]): Unit = {
val sparkConf = new SparkConf().setAppName("scTest")
val sparkContext = new SparkContext(sparkConf)
val sparkSession = org.apache.spark.sql.SparkSession.builder
.appName("sparkSessionTest")
.getOrCreate
val IDList = List("ID1","ID2","ID3")
val IDListRDD = sparkContext.parallelize(IDList)
IDListRDD.foreach(idString => {
val sc = SparkContext.getOrCreate(sparkConf)
executeAlgorithm(idString,sc)
})
Thank you in advance

Answer 1

The rdd.foreach{} block normally should get executed in a executor somewhere in your cluster. In the local mode though, both driver and executor share the same JVM instance accessing each other classes/instances that live in the heap memory
causing an unpredictable behavior. Therefore you can't and you shouldn't make call from an executor node to driver's objects such as SparkContext, RDDs, DataFrames e.t.c please advice the next links for more information:
Apache Spark : When not to use mapPartition and foreachPartition?
Caused by: java.lang.NullPointerException at org.apache.spark.sql.Dataset