Handle SIGTERM in akka-http - scala

Question

The current (10.1.3) Akka HTTP docs:
https://doc.akka.io/docs/akka-http/current/server-side/graceful-termination.html
talk about graceful termination, using this code sample:
import akka.actor.ActorSystem
import akka.http.scaladsl.server.Directives._
import akka.http.scaladsl.server.Route
import akka.stream.ActorMaterializer
import scala.concurrent.duration._
implicit val system = ActorSystem()
implicit val dispatcher = system.dispatcher
implicit val materializer = ActorMaterializer()
val routes = get {
complete("Hello world!")
}
val binding: Future[Http.ServerBinding] =
Http().bindAndHandle(routes, "127.0.0.1", 8080)
// ...
// once ready to terminate the server, invoke terminate:
val onceAllConnectionsTerminated: Future[Http.HttpTerminated] =
Await.result(binding, 10.seconds)
.terminate(hardDeadline = 3.seconds)
// once all connections are terminated,
// - you can invoke coordinated shutdown to tear down the rest of the system:
onceAllConnectionsTerminated.flatMap { _ ⇒
system.terminate()
}
I am wondering at what point this get called at, the comment states:
// once ready to terminate the server
What does this mean exactly, i.e. who/what determines the server is ready to terminate?
Do I have to put the shutdown code above in some hook function somewhere so that it is invoked on Akka HTTP receiving a SIGTERM?
I’ve tried putting this into the shutdown hook:
CoordinatedShutdown(system).addCancellableJvmShutdownHook{
// once ready to terminate the server, invoke terminate:
val onceAllConnectionsTerminated: Future[Http.HttpTerminated] =
Await.result(binding, 10.seconds)
.terminate(hardDeadline = 3.seconds)
// once all connections are terminated,
// - you can invoke coordinated shutdown to tear down the rest of the system:
onceAllConnectionsTerminated.flatMap { _ ⇒
system.terminate()
}
}
But requests in progress are ended immediately upon sending a SIGTERM (kill ), rather than completing.
I also found a slightly different way of shutdown from https://github.com/akka/akka-http/issues/1210#issuecomment-338825745:
CoordinatedShutdown(system).addTask(
CoordinatedShutdown.PhaseServiceUnbind, "http_shutdown") { () =>
bind.flatMap(_.unbind).flatMap { _ =>
Http().shutdownAllConnectionPools
}.map { _ =>
Done
}
}
Maybe I should using this to handle SIGTERM? I'm not sure..
Thanks!

Answer 1

Resolution taken from this answer here:
https://discuss.lightbend.com/t/graceful-termination-on-sigterm-using-akka-http/1619
CoordinatedShutdown(system).addTask(
CoordinatedShutdown.PhaseServiceUnbind, "http_shutdown") { () =>
bind.flatMap(_.terminate(hardDeadline = 1.minute)).map { _ =>
Done
}
}
For me the main part was to increase akka.coordinated-shutdown.default-phase-timeout as it took longer to finish the processing of the request than the 5 second default. You can also just increase the timeout for that one phase. I had the following message in my logs:
Coordinated shutdown phase [service-unbind] timed out after 5000 milliseconds