InterruptedException on .stop() call of StreamingContext without stopping SparkContext - scala

Question

I am having some trouble with stopping a streaming context after a condition has been met inside a foreachRDD. Any time the scc.stop() inside function foo is executed, I get an Interrupted error.
Simplified Code:
def main(){
var sc = new SparkContext(new SparkConf().setAppName("appname").setMaster("local"))
foo(123,sc)
//foo(312,sc) can I call foo again here?
sc.stop()
}
def foo(param1: Integer, sc: SparkContext){
val ssc = new StreamingContext(sc, Seconds(1))
val res = 0
//dummy data, but actual datatypes (but is not relevant to the error I get in this code)
val inputData: mutable.Queue[RDD[Int]] = mutable.Queue()
val inputStream: InputDStream[Int] = ssc.queueStream(inputData)
inputData += sc.makeRDD(List(1, 2))
val rdds_list=some_other_fn(inputstream,param1) //returns DStream
rdds_list.foreachRDD((rdd) => {
def foo1(rdd: RDD[<some_type_2>]) = {
if (condition1) {
println("condition satisfied!") //prints correctly
res = do_stuff(rdd) //executes correctly
println("result: " + res) //executes correctly (and output is as intended)
}else{
println("stopping streaming context!")
ssc.stop(stopSparkContext = false) //error occurs here
}
}
foo(rdd)
})
ssc.start()
ssc.awaitTermination()
res
}
Error log:
**condition satisfied!
result: 124124**
stopping streaming context!
[error] (pool-11-thread-1) java.lang.Error: java.lang.InterruptedException
java.lang.Error: java.lang.InterruptedException
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1155)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.InterruptedException
at java.lang.Object.wait(Native Method)
at java.lang.Thread.join(Thread.java:1252)
at java.lang.Thread.join(Thread.java:1326)
at org.apache.spark.util.AsynchronousListenerBus.stop(AsynchronousListenerBus.scala:160)
at org.apache.spark.streaming.scheduler.JobScheduler.stop(JobScheduler.scala:98)
at org.apache.spark.streaming.StreamingContext.stop(StreamingContext.scala:573)
at org.apache.spark.streaming.StreamingContext.stop(StreamingContext.scala:555)
at edu.gatech.cse8803.main.Main$$anonfun$testClustering$1.foo$1(Main.scala:315)
at edu.gatech.cse8803.main.Main$$anonfun$testClustering$1.apply(Main.scala:318)
at edu.gatech.cse8803.main.Main$$anonfun$testClustering$1.apply(Main.scala:306)
at org.apache.spark.streaming.dstream.DStream$$anonfun$foreachRDD$1.apply(DStream.scala:534)
at org.apache.spark.streaming.dstream.DStream$$anonfun$foreachRDD$1.apply(DStream.scala:534)
at org.apache.spark.streaming.dstream.ForEachDStream$$anonfun$1.apply$mcV$sp(ForEachDStream.scala:42)
at org.apache.spark.streaming.dstream.ForEachDStream$$anonfun$1.apply(ForEachDStream.scala:40)
at org.apache.spark.streaming.dstream.ForEachDStream$$anonfun$1.apply(ForEachDStream.scala:40)
at scala.util.Try$.apply(Try.scala:161)
at org.apache.spark.streaming.scheduler.Job.run(Job.scala:32)
at org.apache.spark.streaming.scheduler.JobScheduler$JobHandler$$anonfun$run$1.apply$mcV$sp(JobScheduler.scala:176)
at org.apache.spark.streaming.scheduler.JobScheduler$JobHandler$$anonfun$run$1.apply(JobScheduler.scala:176)
at org.apache.spark.streaming.scheduler.JobScheduler$JobHandler$$anonfun$run$1.apply(JobScheduler.scala:176)
at scala.util.DynamicVariable.withValue(DynamicVariable.scala:57)
at org.apache.spark.streaming.scheduler.JobScheduler$JobHandler.run(JobScheduler.scala:175)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
I tried using ssc.stop(stopSparkContext = true, stopGracefully = true) but I get this:
WARN scheduler.JobGenerator -
Timed out while stopping the job generator (timeout = 10000)
after foo is called and the program just gets stuck (i.e it does not complete and I have to Ctrl+c it).
Is this the correct way to stop a streaming context? Also, if I wanted to call foo multiple times, should I make any changes? I understand that there should only be one spark context in an application that's why I am trying to re-use them or should I close the SparkContext by setting stopSparkContext as true?
My environment:
sbt v1.0
Scala 2.10.5
Spark 1.3.1
Edit: Looked at other similar questions, tried all their answers - still no luck! :(

Answer 1

It shows that while the spark driver was waiting for the job to finish, you are closing StreamingContext inside the rdd_list which is being handled by the very StreamingContext. This should be closed separately.
And the contexts shouldn't be created and closed with such frequency.
What i would recommend is to do the following...
Initiate and pass StreamingContext from main() to foo(...)
Which will make foo
def foo(param1: Integer, ssc: StreamingContext)
To safely close both of the contexts for a streaming application would be like...
sys.ShutdownHookThread {
//Executes when shutdown signal is received by the app
log.info("Gracefully stopping Spark Context")
sc.stop()
ssc.stop(true, true)
log.info("Application stopped")
}
But if you need to close with programmaticly logic, close StreamingContext with SparkContext.
Which will make your main() look like
def main(){
var sc = new SparkContext(new SparkConf().setAppName("appname").setMaster("local"))
val ssc = new StreamingContext(sc, Seconds(1))
sys.ShutdownHookThread {
//Executes when shutdown signal is received by the app
log.info("Gracefully stopping Spark Context")
sc.stop()
ssc.stop(true, true)
log.info("Application stopped")
}
foo(123,ssc)
sc.stop()
ssc.stop
}