Spark unit test fails due to stage failure - scala

Question

I recently upgraded an application from Spark 1.4.1 to 1.6.0 where the unit tests in my application (in ScalaTest 3.0) suddenly fail, which is not due to API or behavioural changes in Spark.
The weird thing is that each time I run the tests with sbt test a different test fails and always with the following message:
[info] org.apache.spark.SparkException: Job aborted due to stage failure: Task 87 in stage 206.0 failed 1 times, most recent failure: Lost task 87.0 in stage 206.0 (TID 4228, localhost): ExecutorLostFailure (executor driver exited caused by one of the running
tasks) Reason: Executor heartbeat timed out after 148400 ms
[info] Driver stacktrace:
[info] at org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGScheduler$$failJobAndIndependentStages(DAGScheduler.scala:1431)
[info] at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1419)
[info] at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1418)
[info] at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
[info] at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:47)
[info] at org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:1418)
[info] at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:799)
[info] at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:799)
[info] at scala.Option.foreach(Option.scala:236)
[info] at org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:799)
[info] ...
I have set the following in build.sbt:
javaOptions in test += "-Xmx2G"
fork in test := true
parallelExecution in test := false
So, the unit tests are fine but there is something going on that I cannot put my finger on. Does anyone have an idea?

Answer 1

Since this code has been working, I suspect the default memory settings (either executor or driver or overhead) may have changed with the upgrade.
Please post yarn logs for your application id. It will have more details of the error.
Also, please see this link for a similar error https://community.cloudera.com/t5/Advanced-Analytics-Apache-Spark/Executor-Timed-Out/td-p/45097