Spark streaming context hangs on stop - scala

Question

I am trying to write a spark streaming program where I want to gracefully shutdown my application in case my application receives a shutdown hook. I wrote the following snippet to accomplish this.
sys.ShutdownHookThread {
println("Gracefully stopping MyStreamJob")
ssc.stop(stopSparkContext = true, stopGracefully = true)
println("Streaming stopped")
sys.exit(0)
}
On calling this code only the first println is called. That is the second println Streaming Stopped is never seen. The last message I receive on the console is:
39790 [shutdownHook1] INFO org.spark-project.jetty.server.handler.ContextHandler - stopped o.s.j.s.ServletContextHandler{/streaming,null}
39791 [shutdownHook1] INFO org.spark-project.jetty.server.handler.ContextHandler - stopped o.s.j.s.ServletContextHandler{/streaming/batch,null}
39792 [shutdownHook1] INFO org.spark-project.jetty.server.handler.ContextHandler - stopped o.s.j.s.ServletContextHandler{/static/streaming,null}
15/10/19 19:59:43 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/static/streaming,null}
I am using spark 1.4.1. I have to kill manually the job using kill -9 for spark to end. Is this the intended behaviour or am I doing something wrong?

Answer 1

Spark added its own call to stop the StreamingContext. See this email thread.
Your code would have worked prior to 1.4, now it will hang as you are experiencing. You can simply remove your hook and the graceful shutdown should happen automatically.
You can now use the following configuration parameter to specify if the shutdown should be graceful:
spark.streaming.stopGracefullyOnShutdown
The SparkContext will be stopped after the graceful shutdown. See:
"Do not stop SparkContext, let its own shutdown hook stop it"