How can I send elements/messages to an Akka Sink from an Akka HTTP route? My HTTP route still needs to return a normal HTTP response.
I imagine this requires a stream branch/junction. The normal HTTP routes are flows from HttpRequest -> HttpResponse. I would like to add a branch/junction so that HttpRequests can trigger events to my separate sink as well as generate the normal HttpResponse.
Below is a very simple single route akka-http app. For simplicity, I'm using a simple println sink. My production use case, will obviously involve a less trivial sink.
def main(args: Array[String]): Unit = {
implicit val actorSystem = ActorSystem("my-akka-http-test")
val executor = actorSystem.dispatcher
implicit val materializer = ActorMaterializer()(actorSystem)
// I would like to send elements to this sink in response to HTTP GET operations.
val sink: Sink[Any, Future[Done]] = Sink.foreach(println)
val route: akka.http.scaladsl.server.Route =
path("hello" / Segment) { p =>
get {
// I'd like to send a message to an Akka Sink as well as return an HTTP response.
complete {
s"<h1>Say hello to akka-http. p=$p</h1>"
}
}
}
val httpExt: akka.http.scaladsl.HttpExt = Http(actorSystem)
val bindingFuture = httpExt.bindAndHandle(RouteResult.route2HandlerFlow(route), "localhost", 8080)
println("Server online at http://localhost:8080/")
println("Press RETURN to stop...")
scala.io.StdIn.readLine()
bindingFuture
.flatMap(_.unbind())(executor) // trigger unbinding from the port
.onComplete(_ => Await.result(actorSystem.terminate(), Duration.Inf))(executor) // and shutdown when done
}
EDIT: Or in using the low-level akka-http API, how could I send specific messages to a sink from a specific route handler?
def main(args: Array[String]): Unit = {
implicit val actorSystem = ActorSystem("my-akka-http-test")
val executor = actorSystem.dispatcher
implicit val materializer = ActorMaterializer()(actorSystem)
// I would like to send elements to this sink in response to HTTP GET operations.
val sink: Sink[Any, Future[Done]] = Sink.foreach(println)
val requestHandler: HttpRequest => HttpResponse = {
case HttpRequest(GET, Uri.Path("/"), _, _, _) =>
HttpResponse(entity = HttpEntity(
ContentTypes.`text/html(UTF-8)`,
"<html><body>Hello world!</body></html>"))
case HttpRequest(GET, Uri.Path("/ping"), _, _, _) =>
HttpResponse(entity = "PONG!")
case HttpRequest(GET, Uri.Path("/crash"), _, _, _) =>
sys.error("BOOM!")
case r: HttpRequest =>
r.discardEntityBytes() // important to drain incoming HTTP Entity stream
HttpResponse(404, entity = "Unknown resource!")
}
val serverSource = Http().bind(interface = "localhost", port = 8080)
val bindingFuture: Future[Http.ServerBinding] =
serverSource.to(Sink.foreach { connection =>
println("Accepted new connection from " + connection.remoteAddress)
connection handleWithSyncHandler requestHandler
// this is equivalent to
// connection handleWith { Flow[HttpRequest] map requestHandler }
}).run()
println("Server online at http://localhost:8080/")
println("Press RETURN to stop...")
scala.io.StdIn.readLine()
bindingFuture
.flatMap(_.unbind())(executor) // trigger unbinding from the port
.onComplete(_ => Await.result(actorSystem.terminate(), Duration.Inf))(executor) // and shutdown when done
}
IF you want to send the whole HttpRequest to a sink of yours, I'd say the simplest way is to use the alsoTo combinator. The result would be something along the lines of
val mySink: Sink[HttpRequest, NotUsed] = ???
val handlerFlow = Flow[HttpRequest].alsoTo(mySink).via(RouteResult.route2HandlerFlow(route))
val bindingFuture = Http().bindAndHandle(handlerFlow, "localhost", 8080)
FYI: alsoTo in fact hides a Broadcast stage.
IF instead you need to selectively send a message to a Sink from a specific subroute, you have no other choice but to materialize a new flow for each incoming request. See example below
val sink: Sink[Any, Future[Done]] = Sink.foreach(println)
val route: akka.http.scaladsl.server.Route =
path("hello" / Segment) { p =>
get {
(extract(_.request) & extractMaterializer) { (req, mat) ⇒
Source.single(req).runWith(sink)(mat)
complete {
s"<h1>Say hello to akka-http. p=$p</h1>"
}
}
}
}
Also, keep in mind you can always ditch the high-level DSL completely, and model you whole route using the lower-level streams DSL. This will result in more verbose code - but will give you full control of your stream materialization.
EDIT: example below
val sink: Sink[Any, Future[Done]] = Sink.foreach(println)
val handlerFlow =
Flow.fromGraph(GraphDSL.create() { implicit b =>
import GraphDSL.Implicits._
val partition = b.add(Partition[HttpRequest](2, {
case HttpRequest(GET, Uri.Path("/"), _, _, _) ⇒ 0
case _ ⇒ 1
}))
val merge = b.add(Merge[HttpResponse](2))
val happyPath = Flow[HttpRequest].map{ req ⇒
HttpResponse(entity = HttpEntity(
ContentTypes.`text/html(UTF-8)`,
"<html><body>Hello world!</body></html>"))
}
val unhappyPath = Flow[HttpRequest].map{
case HttpRequest(GET, Uri.Path("/ping"), _, _, _) =>
HttpResponse(entity = "PONG!")
case HttpRequest(GET, Uri.Path("/crash"), _, _, _) =>
sys.error("BOOM!")
case r: HttpRequest =>
r.discardEntityBytes() // important to drain incoming HTTP Entity stream
HttpResponse(404, entity = "Unknown resource!")
}
partition.out(0).alsoTo(sink) ~> happyPath ~> merge
partition.out(1) ~> unhappyPath ~> merge
FlowShape(partition.in, merge.out)
})
val bindingFuture = Http().bindAndHandle(handlerFlow, "localhost", 8080)
This is the solution I used that seems ideal. Akka Http seems like it's designed so that your routes are simple HttpRequest->HttpResponse flows and don't involve any extra branches.
Rather than build everything into a single Akka stream graph, I have a separate QueueSource->Sink graph, and the normal Akka Http HttpRequest->HttpResponse flow just adds elements to the source queue as needed.
object HttpWithSinkTest {
def buildQueueSourceGraph(): RunnableGraph[(SourceQueueWithComplete[String], Future[Done])] = {
val annotateMessage: Flow[String, String, NotUsed] = Flow.fromFunction[String, String](s => s"got message from queue: $s")
val sourceQueue = Source.queue[String](100, OverflowStrategy.dropNew)
val sink: Sink[String, Future[Done]] = Sink.foreach(println)
val annotatedSink = annotateMessage.toMat(sink)(Keep.right)
val queueGraph = sourceQueue.toMat(annotatedSink)(Keep.both)
queueGraph
}
def buildHttpFlow(queue: SourceQueueWithComplete[String],
actorSystem: ActorSystem, materializer: ActorMaterializer): Flow[HttpRequest, HttpResponse, NotUsed] = {
implicit val actorSystemI = actorSystem
implicit val materializerI = materializer
val route: akka.http.scaladsl.server.Route =
path("hello" / Segment) { p =>
get {
complete {
queue.offer(s"got http event p=$p")
s"<h1>Say hello to akka-http. p=$p</h1>"
}
}
}
val routeFlow = RouteResult.route2HandlerFlow(route)
routeFlow
}
def main(args: Array[String]): Unit = {
val actorSystem = ActorSystem("my-akka-http-test")
val executor = actorSystem.dispatcher
implicit val materializer = ActorMaterializer()(actorSystem)
val (queue, _) = buildQueueSourceGraph().run()(materializer)
val httpFlow = buildHttpFlow(queue, actorSystem, materializer)
val httpExt: akka.http.scaladsl.HttpExt = Http(actorSystem)
val bindingFuture = httpExt.bindAndHandle(httpFlow, "localhost", 8080)
println("Server online at http://localhost:8080/")
println("Press RETURN to stop...")
scala.io.StdIn.readLine()
println("Shutting down...")
val serverBinding = Await.result(bindingFuture, Duration.Inf)
Await.result(serverBinding.unbind(), Duration.Inf)
Await.result(actorSystem.terminate(), Duration.Inf)
println("Done. Exiting")
}
}
Related
Hi i'm creating a TCP server in Scala with the lib akka. I want to implement a restart of the tcp socket if my client doesn't send any data in a set interval. I tried with idleTimeout and a SupervisionStrategy but i can't catch the TimeoutException. On the client i see the log "Closing connection due to IO error java.io.IOException: Connection reset by peer"..
How i can resolve that? And restart the stream??
object TCPServer {
def serverLogic(connection: IncomingConnection) (implicit system: ActorSystem): Flow[ByteString, ByteString, NotUsed] = {
val converter: Flow[ByteString, String, NotUsed] = Flow[ByteString].map { (bytes: ByteString) =>
val message = bytes.utf8String
Logging.getLogger(system,this.getClass).debug(s"server received message $message")
message
}
val httpOut: Flow[String, String, NotUsed] = Flow[String].map { string =>
val answer: String = s"hello"
answer
}
val responder: Flow[String, ByteString, NotUsed] = Flow[String].map { string =>
val answer: String = s"Server responded with message [$string]"
ByteString(answer)
}
Flow[ByteString]
.idleTimeout(Duration.apply(1,"minutes"))
.via(converter)
.via(httpOut)
.via(responder)
}
def server(address: String, port: Int)(implicit system: ActorSystem) : Unit = {
val decider: Supervision.Decider = { e =>
LoggerFactory.getLogger(this.getClass).error("Failed ", e)
Supervision.Restart
}
val log = Logging.getLogger(system, this)
import system.dispatcher
val materializerSettings = ActorMaterializerSettings(system).withSupervisionStrategy(decider)
implicit val materializer = ActorMaterializer(materializerSettings)(system)
val connectionHandler: Sink[IncomingConnection, Future[Done]] = Sink.foreach[Tcp.IncomingConnection] { (conn: IncomingConnection) =>
log.debug(s"incomig connection from ${conn.remoteAddress}")
conn.handleWith(serverLogic(conn))
}
val incomingConnections: Source[IncomingConnection, Future[ServerBinding]] = Tcp().bind(address, port)
val binding: Future[ServerBinding] = incomingConnections.to(connectionHandler).run()
binding.onComplete {
case Success(b) =>
log.info("Server started, listening on: " + b.localAddress)
case Failure(e) =>
log.error(s"Server could not bind to $address:$port: ${e.getMessage}")
system.terminate()
}
}
}
Take a look to the doc. available in the akka stream webbsite: https://doc.akka.io/docs/akka/2.6.0/stream/stream-error.html. With this RestartSource you can apply a policy in order to restart TCP server.
I'm trying to establish a websocket connection. I'm trying this in a already created project. And this error occurs.
Type mismatch, expected: Flow[HttpRequest, HttpResponse, Any], actual:
server.Route
The error occurs at route
val binding = Http().bindAndHandle(route, interface, port)
println(s"Server is now online at http://$interface:$port\nPress RETURN
to stop...")
StdIn.readLine()
binding.flatMap(_.unbind()).onComplete(_ => actorSystem.terminate())
println("Server is down...")
val route=path("ws-echo") {
(get) {
val requestHandler: HttpRequest ⇒ HttpResponse = {
case req#HttpRequest( GET, Uri.Path( "/greeter" ), _, _, _ ) ⇒
req.header[UpgradeToWebSocket] match {
case Some( upgrade ) ⇒ upgrade.handleMessages(
greeterWebSocketService
)
case None ⇒ HttpResponse( 400, entity = "Not a valid websocket
request!" )
}
case r: HttpRequest =>
r.discardEntityBytes()
HttpResponse( 404, entity = "Unknown resource!" )
}
}
}
override implicit val system = ActorSystem()
override implicit val executor = system.dispatcher
override implicit val materializer = ActorMaterializer()
override val logger = Logging(system, getClass)
implicit val mat: Materializer
implicit val actorSystem = ActorSystem("akka-system")
val router: server.Route = routes
val config = ConfigFactory.load()
val interface = config.getString("app.interface")
val port = config.getInt("app.port")
Http().bindAndHandle(router, Configuration.httpInterface,
Configuration.httpPort)
I suspect you're getting this error on the line val binding = Http().bindAndHandle(route, interface, port), is that correct?
Indeed bindAndHandle expects a Flow:
def bindAndHandle(
handler: Flow[HttpRequest, HttpResponse, Any],
interface: String,
port: Int = DefaultPortForProtocol,
connectionContext: ConnectionContext = defaultServerHttpContext,
settings: ServerSettings = ServerSettings(system),
log: LoggingAdapter = system.log)(implicit fm: Materializer
): Future[ServerBinding]
So why do so many examples show passing in a Route? The 'magic' here that there's an implicit conversion available to turn a Route into a Flow.
This is basically an implementation detail, you normally shouldn't have to care about this, but the implicit conversion is route2HandlerFlow:
implicit def route2HandlerFlow(
route: Route
)(
implicit routingSettings: RoutingSettings,
parserSettings: ParserSettings,
materializer: Materializer,
routingLog: RoutingLog,
executionContext: ExecutionContext = null,
rejectionHandler: RejectionHandler = RejectionHandler.default,
exceptionHandler: ExceptionHandler = null
): Flow[HttpRequest, HttpResponse, NotUsed]
As you can see, this implicit conversion requires some other implicit values to be available. I suspect you are missing one of those. I think it might be enough to introduce an implicit Materializer, could you give that a try?
I have the following Graph:
case class FlowFactory() {
val reactiveConnection = ???
val serviceRabbitConnection = ???
val switch = KillSwitches.single[Routed]
val stream: RunnableGraph[UniqueKillSwitch] = RunnableGraph.fromGraph(GraphDSL.create(switch) { implicit builder: GraphDSL.Builder[UniqueKillSwitch] => sw =>
import GraphDSL.Implicits._
val in = builder.add(Source.fromPublisher(reactiveConnection.consume(???)))
val context = builder.add(contextFlow(serviceRabbitConnection))
val inflate = builder.add(inflateFlow())
val compute = builder.add(computeFlow())
val out = builder.add(Sink.fromSubscriber(reactiveConnection.publish()))
in ~> context ~> inflate ~> compute ~> sw ~> out
ClosedShape
})
val killSwitch = stream.run()
killSwitch.shutdown()
}
When I shutdown the stream, I also need to kill the following connections : reactiveConnection and serviceRabbitConnection.
How do I achieve that, is there a easy way to override KillSwitch's shutdown() method?
Is there a method that is called when the stream is closed?, like onComplete() or onClose()?
You can perform your callback inside the stream, by attaching an additional sink (Sink.onComplete).
val sink1 = Sink.fromSubscriber(reactiveConnection.publish())
val sink2 = Sink.onComplete{
case Success(_) ⇒ println("success!")
case Failure(e) ⇒ println(s"failure - $e")
}
val out = builder.add(Sink.combine(sink1, sink2)(Broadcast(_)))
I tried to build a small PDF parser with akka-streams (and limited understandings of it yet) and Apache's pdfbox.
One thing which I don't really get, the stream stops exactly after the given number of parallelism which is given into the mapAsync.
so if a PDF-doc has 20 pages and the parallelism is set to 5, the first 5 pages get processed and the rest is ignored, if set to 20, everything is done fine. Anybody an idea what I'm doing wrong?
class PdfParser(ws: WSClient, conf: Configuration, parallelism: Int) {
implicit val system = ActorSystem("image-parser")
implicit val materializer = ActorMaterializer()
def documentPages(doc: PDDocument, key: String): Iterator[Page] = {
val pages: util.List[_] = doc.getDocumentCatalog.getAllPages
val pageList = (for {
i ← 0 until pages.size()
page = pages.get(i)
} yield Page(page, s"$key-$i.jpg")).toIterator
pageList
}
val pageToImage: Flow[Page, Image, NotUsed] = Flow[Page].map { p ⇒
val img = p.content.asInstanceOf[PDPage].convertToImage()
Image(img, p.name)
}
val imageToS3: Flow[Image, String, NotUsed] = Flow[Image].mapAsync(parallelism) { i ⇒
val s3 = S3.fromConfiguration(ws, conf)
val bucket = s3.getBucket("elsa-essays")
val baos = new ByteArrayOutputStream()
ImageIO.write(i.content, "jpg", baos)
val res = bucket add BucketFile(i.name, "image/jpeg", baos.toByteArray)
res.map { _ ⇒
"uploaded"
}.recover {
case e: S3Exception ⇒ e.message
}
}
val sink: Sink[String, Future[String]] = Sink.head[String]
def parse(path: Path, key: String): Future[String] = {
val stream: InputStream = new FileInputStream(path.toString)
val doc = PDDocument.load(stream)
val source = Source.fromIterator(() ⇒ documentPages(doc, key))
val runnable: RunnableGraph[Future[String]] = source.via(pageToImage).via(imageToS3).toMat(sink)(Keep.right)
val res = runnable.run()
res.map { s ⇒
doc.close()
stream.close()
s
}
}
}
The problem is in your Sink. That Sink.head will return one element from your materialized Stream. So the question is, why it´s received more than one value when mapAsync(>1) is used in stream materialization?. Maybe it´s because it uses more than one actor pushing values downstream.
In any case, change your sink to something like:
val sink: Sink[String, Future[String]] = Sink.fold("")((a, b) => b ++ a)
and it will work.
How do you throttle Flow in the latest Akka (2.4.6) ? I'd like to throttle Http client flow to limit number of requests to 3 requests per second. I found following example online but it's for old Akka and akka-streams API changed so much that I can't figure out how to rewrite it.
def throttled[T](rate: FiniteDuration): Flow[T, T] = {
val tickSource: Source[Unit] = TickSource(rate, rate, () => ())
val zip = Zip[T, Unit]
val in = UndefinedSource[T]
val out = UndefinedSink[T]
PartialFlowGraph { implicit builder =>
import FlowGraphImplicits._
in ~> zip.left ~> Flow[(T, Unit)].map { case (t, _) => t } ~> out
tickSource ~> zip.right
}.toFlow(in, out)
}
Here is my best attempt so far
def throttleFlow[T](rate: FiniteDuration) = Flow.fromGraph(GraphDSL.create() { implicit builder =>
import GraphDSL.Implicits._
val ticker = Source.tick(rate, rate, Unit)
val zip = builder.add(Zip[T, Unit.type])
val map = Flow[(T, Unit.type)].map { case (value, _) => value }
val messageExtractor = builder.add(map)
val in = Inlet[T]("Req.in")
val out = Outlet[T]("Req.out")
out ~> zip.in0
ticker ~> zip.in1
zip.out ~> messageExtractor.in
FlowShape.of(in, messageExtractor.out)
})
it throws exception in my main flow though :)
private val queueHttp = Source.queue[(HttpRequest, (Any, Promise[(Try[HttpResponse], Any)]))](1000, OverflowStrategy.backpressure)
.via(throttleFlow(rate))
.via(poolClientFlow)
.mapAsync(4) {
case (util.Success(resp), any) =>
val strictFut = resp.entity.toStrict(5 seconds)
strictFut.map(ent => (util.Success(resp.copy(entity = ent)), any))
case other =>
Future.successful(other)
}
.toMat(Sink.foreach({
case (triedResp, (value: Any, p: Promise[(Try[HttpResponse], Any)])) =>
p.success(triedResp -> value)
case _ =>
throw new RuntimeException()
}))(Keep.left)
.run
where poolClientFlow is Http()(system).cachedHostConnectionPool[Any](baseDomain)
Exception is:
Caused by: java.lang.IllegalArgumentException: requirement failed: The output port [Req.out] is not part of the underlying graph.
at scala.Predef$.require(Predef.scala:219)
at akka.stream.impl.StreamLayout$Module$class.wire(StreamLayout.scala:204)
Here is an attempt that uses the throttle method as mentioned by #Qingwei. The key is to not use bindAndHandle(), but to use bind() and throttle the flow of incoming connections before handling them. The code is taken from the implementation of bindAndHandle(), but leaves out some error handling for simplicity. Please don't do that in production.
implicit val system = ActorSystem("test")
implicit val mat = ActorMaterializer()
import system.dispatcher
val maxConcurrentConnections = 4
val handler: Flow[HttpRequest, HttpResponse, NotUsed] = complete(LocalDateTime.now().toString)
def handleOneConnection(incomingConnection: IncomingConnection): Future[Done] =
incomingConnection.flow
.watchTermination()(Keep.right)
.joinMat(handler)(Keep.left)
.run()
Http().bind("127.0.0.1", 8080)
.throttle(3, 1.second, 1, ThrottleMode.Shaping)
.mapAsyncUnordered(maxConcurrentConnections)(handleOneConnection)
.to(Sink.ignore)
.run()