NullPointerException on DataFrame where - scala

Question

I have the following method written in Scala:
def fillEmptyCells: Unit = {
val hourIndex = _weather.schema.fieldIndex("Hour")
val dateIndex = _weather.schema.fieldIndex("Date")
val groundSurfaceIndex = _weather.schema.fieldIndex("GroundSurface")
val snowyGroundIndex = _weather.schema.fieldIndex("SnowyGroundSurface")
val precipitationIndex = _weather.schema.fieldIndex("catPrec")
val snowDepthIndex = _weather.schema.fieldIndex("catSnowDepth")
var resultDf : DataFrame = sparkSession.createDataFrame(sparkSession.sparkContext.emptyRDD[Row],_weather.schema)
val days = _weather.select("Date").distinct().rdd
_weather.where("Date = '2014-08-01'").show()
days.foreach(x => {
println(s"Date = '${x.getDate(0)}'")
_weather.where(s"Date = '${x.getDate(0)}'").show()
val day = _weather.where(s"Date = '${x.getDate(0)}'")
val dayValues = day.where("Hour = 6").first()
val grSur = dayValues.getString(groundSurfaceIndex)
val snSur = dayValues.getString(snowyGroundIndex)
val prec = dayValues.getString(precipitationIndex)
val snowDepth = dayValues.getString(snowDepthIndex)
val dayRddMapped = day.rdd.map(y => (y(0), y(1), grSur, snSur, y(4), y(5), y(6), y(7), prec, snowDepth))
.foreach(z => {
resultDf = resultDf.union(Seq(z).toDF())
})
})
resultDf.show(20)
Unit
}
The problem is this line: _weather.where(s"Date = '${x.getDate(0)}'").show() where the NullPointerException occurs. As can be seen at line above, I print the where clause to console (it looks like Date = '2014-06-03') and the line just before foreach takes one of the outputs as parameters and works fine. _weather is a class variable and does not change while this method is running. Debugger shows more stranger things: _weather gets nulled after first iteration.
What is the source of this magic and how can I avoid it?
Moreover, if you have any suggestions according to architecture and code quality, welcome here
Stacktrace:
java.lang.NullPointerException
at org.apache.spark.sql.Dataset.where(Dataset.scala:1344)
at org.[package].WeatherHelper$$anonfun$fillEmptyCells$1.apply(WeatherHelper.scala:148)
at org.[package].WeatherHelper$$anonfun$fillEmptyCells$1.apply(WeatherHelper.scala:146)
at scala.collection.Iterator$class.foreach(Iterator.scala:893)
at scala.collection.AbstractIterator.foreach(Iterator.scala:1336)
at org.apache.spark.rdd.RDD$$anonfun$foreach$1$$anonfun$apply$28.apply(RDD.scala:918)
at org.apache.spark.rdd.RDD$$anonfun$foreach$1$$anonfun$apply$28.apply(RDD.scala:918)
at org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:2062)
at org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:2062)
at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:87)
at org.apache.spark.scheduler.Task.run(Task.scala:108)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:335)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
19/01/10 13:39:35 ERROR Executor: Exception in task 6.0 in stage 10.0 (TID 420)
The class name is WeatherHelper it's just a part of the whole stacktrace which repeats ~20 times.

Answer 1

you cannot use dataframes in RDD code (you use dataframes in days.foreach), th dataframes are null here as it only lives on the driver, but not on the executors