I forward the incoming messages from Kafka to a webserver via websocket client.
The following code shows, how I am doing it:
import akka.Done
import akka.actor.ActorSystem
import akka.http.scaladsl.Http
import akka.http.scaladsl.model.StatusCodes
import akka.http.scaladsl.model.ws._
import akka.kafka.scaladsl.Consumer
import akka.kafka.{ConsumerSettings, Subscriptions}
import akka.stream.ActorMaterializer
import akka.stream.scaladsl.{Flow, Keep, Sink, Source}
import com.typesafe.scalalogging.Logger
import org.apache.kafka.clients.consumer.ConsumerConfig
import org.apache.kafka.common.serialization.StringDeserializer
import scala.concurrent.{Future, Promise}
final case class WsGraph(logger: Logger, sink: Sink[Message, Future[Done]])(implicit val system: ActorSystem) {
private implicit val materializer = ActorMaterializer()
private implicit val akka = system.settings.config.getConfig("akka.kafka.consumer")
private implicit val executor = system.dispatcher
private val consumerSetup = system.settings.config.getConfig("kafka.consumer.setup")
private val wsSetup = system.settings.config.getConfig("websocket.setup")
private val consumerSettings: ConsumerSettings[String, String] =
ConsumerSettings(akka, new StringDeserializer, new StringDeserializer)
.withBootstrapServers(consumerSetup.getString("bootStrapServers"))
.withGroupId(consumerSetup.getString("groupId"))
.withProperty(ConsumerConfig.AUTO_OFFSET_RESET_CONFIG, "latest")
private val kafkaAsSource: Source[Message, (Consumer.Control, Promise[Option[Message]])] = Consumer
.plainSource(
consumerSettings,
Subscriptions.topics(consumerSetup.getString("topics"))
)
.map(msg => TextMessage(msg.value()))
.concatMat(Source.maybe[Message])(Keep.both)
.mapAsync(Runtime.getRuntime.availableProcessors())(Future(_))
private val socketFlow: Flow[Message, Message, (Consumer.Control, Promise[Option[Message]])] =
Flow.fromSinkAndSourceMat(sink, kafkaAsSource)(Keep.right)
private val (upgradeResponse, (draining, _)) =
Http().singleWebSocketRequest(
WebSocketRequest(wsSetup.getString("server")),
socketFlow)
val create: Future[Either[String, Done]] = upgradeResponse.map { upgrade =>
// just like a regular http request we can access response status which is available via upgrade.response.status
// status code 101 (Switching Protocols) indicates that server support WebSockets
if (upgrade.response.status == StatusCodes.SwitchingProtocols) {
logger.info("Switching protocols")
Right(Done)
} else {
Left(s"Connection failed: ${upgrade.response.status}")
}
}
sys.addShutdownHook {
draining.shutdown()
logger.info("Draining websocket ressource.")
}
}
The problem here is, if the webserver can not be reached, the actor above gets closed.
The question is, how to figure out, if the webserver is not more available, then the actor should restart and try to connect again.
I think your code
private val (upgradeResponse, (draining, _)) =
Http().singleWebSocketRequest(
WebSocketRequest(wsSetup.getString("server")),
socketFlow)
has return type
(Future[WebSocketUpgradeResponse], T)
As you are only using upgradeResponse i.e Future[WebSocketUpgradeResponse]
May be you can try rewriting your code using Recover with Retries
So you have to replace your
val create: Future[Either[String, Done]] = upgradeResponse.map { upgrade =>
// just like a regular http request we can access response status which is available via upgrade.response.status
// status code 101 (Switching Protocols) indicates that server support WebSockets
if (upgrade.response.status == StatusCodes.SwitchingProtocols) {
logger.info("Switching protocols")
Right(Done)
} else {
Left(s"Connection failed: ${upgrade.response.status}")
}
}
with
planB = Source.empty
Source.fromFuture(upgradeResponse).recoverWithRetries(3, {
case ex: RuntimeException => logger.error("Error", ex); planB
}).runWith(Sink.ignore).map {upgrade =>
// just like a regular http request we can access response status which is available via upgrade.response.status
// status code 101 (Switching Protocols) indicates that server support WebSockets
if (upgrade.response.status == StatusCodes.SwitchingProtocols) {
logger.info("Switching protocols")
Right(Done)
} else {
Left(s"Connection failed: ${upgrade.response.status}")
}
}
Here you can add exception handling for the RuntimeException
Please refer https://doc.akka.io/docs/akka/2.5.5/scala/stream/stream-error.html for more details
I hope this will help. Please let me know in case any error.
Thanks
Related
I have a slightly modified example from akka-http docs for WebSocket client-side:
package org.enso
import akka.actor.ActorSystem
import akka.{Done, NotUsed}
import akka.http.scaladsl.Http
import akka.stream.ActorMaterializer
import akka.stream.scaladsl._
import akka.http.scaladsl.model._
import akka.http.scaladsl.model.ws._
import akka.pattern.after
import scala.concurrent.duration._
import scala.concurrent.{Future, Promise}
object SingleWebSocketRequest {
def main(args: Array[String]): Unit = {
implicit val system: ActorSystem = ActorSystem()
implicit val materializer: ActorMaterializer = ActorMaterializer()
import system.dispatcher
// print each incoming text message
val printSink: Sink[Message, Future[Done]] =
Sink.foreach { _ =>
println("foo")
}
val s1: Source[Message, NotUsed] =
Source(Stream.fill(100){
TextMessage("""{"method":"ping","responseSize":4}""")
})
// the Future[Done] is the materialized value of Sink.foreach
// and it is completed when the stream completes
val flow: Flow[Message, Message, Future[Done]] =
Flow.fromSinkAndSourceMat(printSink, s1)(Keep.left)
// upgradeResponse is a Future[WebSocketUpgradeResponse] that
// completes or fails when the connection succeeds or fails
// and closed is a Future[Done] representing the stream completion from above
val (upgradeResponse, closed) =
Http().singleWebSocketRequest(WebSocketRequest("ws://localhost:8080"), flow)
val connected = upgradeResponse.map { upgrade =>
// just like a regular http request we can access response status which is available via upgrade.response.status
// status code 101 (Switching Protocols) indicates that server support WebSockets
if (upgrade.response.status == StatusCodes.SwitchingProtocols) {
Done
} else {
throw new RuntimeException(s"Connection failed: ${upgrade.response.status}")
}
}
// in a real application you would not side effect here
// and handle errors more carefully
connected.onComplete(println)
closed.foreach {_ =>
println("closed")
}
}
}
This code sends requests to a Websocket server-side, which works fine with another JS client. The server-side works fine, sending many responses at once, as evident from this Wireshark dump
E1##ÿ K^B!¹åª&
<·*<·#/{
"response": "aaaa",
"status": "success"
}/{
"response": "aaaa",
"status": "success"
}/{
"response": "aaaa",
"status": "success"
}/...
But printing "foo", which is the side effect of handling server messages in the Sink, only happens every ~5s, which is very slow compared with what is observed in Wireshark.
How would I explain / debug this? Could it be something to do with buffers / back pressure?
Ok, with this Sink everything works nice:
val printSink: Sink[Message, Future[Done]] =
Sink.foreachAsync(1) {
case x: TextMessage => x.toStrict(FiniteDuration(100, "ms")).map { _ =>
logger.debug("foo")
receive = System.currentTimeMillis() :: receive
}
}
As it turns out, whenever working with akka – always consume all the input. This WS client API actually seems strange to me, for the http clients you get a helper method like discardResponseBytes precisely for this case. Anyway – consume the input, always
I am trying to understand, how the following code snippet works:
val flow: Flow[Message, Message, Future[Done]] =
Flow.fromSinkAndSourceMat(printSink, helloSource)(Keep.left)
Two guys gave a very wonderful explanation on this thread. I understand the concept of the Composite flow, but how does it work on the websocket client.
Consider the following code:
import akka.actor.ActorSystem
import akka.{ Done, NotUsed }
import akka.http.scaladsl.Http
import akka.stream.ActorMaterializer
import akka.stream.scaladsl._
import akka.http.scaladsl.model._
import akka.http.scaladsl.model.ws._
import scala.concurrent.Future
object SingleWebSocketRequest {
def main(args: Array[String]) = {
implicit val system = ActorSystem()
implicit val materializer = ActorMaterializer()
import system.dispatcher
// print each incoming strict text message
val printSink: Sink[Message, Future[Done]] =
Sink.foreach {
case message: TextMessage.Strict =>
println(message.text)
}
val helloSource: Source[Message, NotUsed] =
Source.single(TextMessage("hello world!"))
// the Future[Done] is the materialized value of Sink.foreach
// and it is completed when the stream completes
val flow: Flow[Message, Message, Future[Done]] =
Flow.fromSinkAndSourceMat(printSink, helloSource)(Keep.left)
// upgradeResponse is a Future[WebSocketUpgradeResponse] that
// completes or fails when the connection succeeds or fails
// and closed is a Future[Done] representing the stream completion from above
val (upgradeResponse, closed) =
Http().singleWebSocketRequest(WebSocketRequest("ws://echo.websocket.org"), flow)
val connected = upgradeResponse.map { upgrade =>
// just like a regular http request we can access response status which is available via upgrade.response.status
// status code 101 (Switching Protocols) indicates that server support WebSockets
if (upgrade.response.status == StatusCodes.SwitchingProtocols) {
Done
} else {
throw new RuntimeException(s"Connection failed: ${upgrade.response.status}")
}
}
// in a real application you would not side effect here
// and handle errors more carefully
connected.onComplete(println)
closed.foreach(_ => println("closed"))
}
}
It is a websocket client, that send a message to the websocket server and the printSink receives it and print it out.
How can it be, that printSink receives messages, there is no a connection between the Sink and Source.
Is it like a loop?
Stream flow is from left to right, how it comes that the Sink can consume messages from websocket server?
Flow.fromSinkAndSourceMat puts an independent Sink and a Source to a shape of the Flow. Elements going into that Sink do not end up at the Source.
From the Websocket client API perspective, it needs a Source from which requests will be sent to the server and a Sink that it will send the responses to. The singleWebSocketRequest could take a Source and a Sink separately, but that would be a bit more verbose API.
Here is a shorter example that demonstrates the same as in your code snippet but is runnable, so you can play around with it:
import akka._
import akka.actor._
import akka.stream._
import akka.stream.scaladsl._
implicit val sys = ActorSystem()
implicit val mat = ActorMaterializer()
def openConnection(userFlow: Flow[String, String, NotUsed])(implicit mat: Materializer) = {
val processor = Flow[String].map(_.toUpperCase)
processor.join(userFlow).run()
}
val requests = Source(List("one", "two", "three"))
val responses = Sink.foreach(println)
val userFlow = Flow.fromSinkAndSource(responses, requests)
openConnection(userFlow)
I'm trying client-side websocket by following doc at webSocketClientFlow.
sample code is:
import akka.actor.ActorSystem
import akka.Done
import akka.http.scaladsl.Http
import akka.stream.ActorMaterializer
import akka.stream.scaladsl._
import akka.http.scaladsl.model._
import akka.http.scaladsl.model.ws._
import scala.concurrent.Future
object WebSocketClientFlow {
def main(args: Array[String]) = {
implicit val system = ActorSystem()
implicit val materializer = ActorMaterializer()
import system.dispatcher
// Future[Done] is the materialized value of Sink.foreach,
// emitted when the stream completes
val incoming: Sink[Message, Future[Done]] =
Sink.foreach[Message] {
case message: TextMessage.Strict =>
println(message.text)
}
// send this as a message over the WebSocket
val outgoing = Source.single(TextMessage("hello world!"))
// flow to use (note: not re-usable!)
val webSocketFlow = Http().webSocketClientFlow(WebSocketRequest("ws://echo.websocket.org"))
// the materialized value is a tuple with
// upgradeResponse is a Future[WebSocketUpgradeResponse] that
// completes or fails when the connection succeeds or fails
// and closed is a Future[Done] with the stream completion from the incoming sink
val (upgradeResponse, closed) =
outgoing
.viaMat(webSocketFlow)(Keep.right) // keep the materialized Future[WebSocketUpgradeResponse]
.toMat(incoming)(Keep.both) // also keep the Future[Done]
.run()
// just like a regular http request we can access response status which is available via upgrade.response.status
// status code 101 (Switching Protocols) indicates that server support WebSockets
val connected = upgradeResponse.flatMap { upgrade =>
if (upgrade.response.status == StatusCodes.SwitchingProtocols) {
Future.successful(Done)
} else {
throw new RuntimeException(s"Connection failed: ${upgrade.response.status}")
}
}
// in a real application you would not side effect here
connected.onComplete(println)
closed.foreach(_ => println("closed"))
}
}
after had connection upgraded, how to use the connection send message to websocket server side?
I noticed from the doc:
The Flow that is returned by this method can only be materialized once. For each request a new flow must be acquired by calling the method again.
still confused, why we need construct the flow many times since an upgraded connection alrady ready.
You can create an actor based source and send new messages over the established websocket connection.
val req = WebSocketRequest(uri = "ws://127.0.0.1/ws")
val webSocketFlow = Http().webSocketClientFlow(req)
val messageSource: Source[Message, ActorRef] =
Source.actorRef[TextMessage.Strict](bufferSize = 10, OverflowStrategy.fail)
val messageSink: Sink[Message, NotUsed] =
Flow[Message]
.map(message => println(s"Received text message: [$message]"))
.to(Sink.ignore)
val ((ws, upgradeResponse), closed) =
messageSource
.viaMat(webSocketFlow)(Keep.both)
.toMat(messageSink)(Keep.both)
.run()
val connected = upgradeResponse.flatMap { upgrade =>
if (upgrade.response.status == StatusCodes.SwitchingProtocols) {
Future.successful(Done)
} else {
throw new RuntimeException(s"Connection failed: ${upgrade.response.status}")
}
}
ws ! TextMessage.Strict("Hello World")
ws ! TextMessage.Strict("Hi")
ws ! TextMessage.Strict("Yay!")
`
I'm using Akka 2.4.4 and trying to move from Apache HttpAsyncClient (unsuccessfully).
Below is simplified version of code that I use in my project.
The problem is that it hangs if I send more than 1-3 requests to the flow. So far after 6 hours of debugging I couldn't even locate the problem. I don't see exceptions, error logs, events in Decider. NOTHING :)
I tried reducing connection-timeout setting to 1s thinking that maybe it's waiting for response from the server but it didn't help.
What am I doing wrong ?
import akka.actor.ActorSystem
import akka.http.scaladsl.Http
import akka.http.scaladsl.model.headers.Referer
import akka.http.scaladsl.model.{HttpRequest, HttpResponse}
import akka.http.scaladsl.settings.ConnectionPoolSettings
import akka.stream.Supervision.Decider
import akka.stream.scaladsl.{Sink, Source}
import akka.stream.{ActorAttributes, Supervision}
import com.typesafe.config.ConfigFactory
import scala.collection.immutable.{Seq => imSeq}
import scala.concurrent.{Await, Future}
import scala.concurrent.duration.Duration
import scala.util.Try
object Main {
implicit val system = ActorSystem("root")
implicit val executor = system.dispatcher
val config = ConfigFactory.load()
private val baseDomain = "www.google.com"
private val poolClientFlow = Http()(system).cachedHostConnectionPool[Any](baseDomain, 80, ConnectionPoolSettings(config))
private val decider: Decider = {
case ex =>
ex.printStackTrace()
Supervision.Stop
}
private def sendMultipleRequests[T](items: Seq[(HttpRequest, T)]): Future[Seq[(Try[HttpResponse], T)]] =
Source.fromIterator(() => items.toIterator)
.via(poolClientFlow)
.log("Logger")(log = myAdapter)
.recoverWith {
case ex =>
println(ex)
null
}
.withAttributes(ActorAttributes.supervisionStrategy(decider))
.runWith(Sink.seq)
.map { v =>
println(s"Got ${v.length} responses in Flow")
v.asInstanceOf[Seq[(Try[HttpResponse], T)]]
}
def main(args: Array[String]) {
val headers = imSeq(Referer("https://www.google.com/"))
val reqPair = HttpRequest(uri = "/intl/en/policies/privacy").withHeaders(headers) -> "some req ID"
val requests = List.fill(10)(reqPair)
val qwe = sendMultipleRequests(requests).map { case responses =>
println(s"Got ${responses.length} responses")
system.terminate()
}
Await.ready(system.whenTerminated, Duration.Inf)
}
}
Also what's up with proxy support ? Doesn't seem to work for me either.
You need to consume the body of the response fully so that the connection is made available for subsequent requests. If you don't care about the response entity at all, then you can just drain it to a Sink.ignore, something like this:
resp.entity.dataBytes.runWith(Sink.ignore)
By the default config, when using a host connection pool, the max connections is set to 4. Each pool has it's own queue where requests wait until one of the open connections becomes available. If that queue ever goes over 32 (default config, can be changed, must be a power of 2) then yo will start seeing failures. In your case, you only do 10 requests, so you don't hit that limit. But by not consuming the response entity you don't free up the connection and everything else just queues in behind, waiting for the connections to free up.
I'm trying to use akka-http in order to make http requests to a single host (e.g. "akka.io"). The problem is that the created flow (Http().cachedHostConnectionPool) starts emitting responses only after N http requests are made, where N is equal to max-connections.
import scala.util.Failure
import scala.util.Success
import com.typesafe.config.ConfigFactory
import akka.actor.ActorSystem
import akka.http.scaladsl.Http
import akka.http.scaladsl.model.HttpRequest
import akka.http.scaladsl.model.Uri.apply
import akka.http.scaladsl.settings.ConnectionPoolSettings
import akka.stream.ActorMaterializer
import akka.stream.scaladsl.Sink
import akka.stream.scaladsl.Source
object ConnectionPoolExample extends App {
implicit val system = ActorSystem()
implicit val executor = system.dispatcher
implicit val materializer = ActorMaterializer()
val config = ConfigFactory.load()
val connectionPoolSettings = ConnectionPoolSettings(config).withMaxConnections(10)
lazy val poolClientFlow = Http().cachedHostConnectionPool[Unit]("akka.io", 80, connectionPoolSettings)
val fakeSource = Source.fromIterator[Unit] { () => Iterator.continually { Thread.sleep(1000); () } }
val requests = fakeSource.map { _ => println("Creating request"); HttpRequest(uri = "/") -> (()) }
val responses = requests.via(poolClientFlow)
responses.runForeach {
case (tryResponse, jsonData) =>
tryResponse match {
case Success(httpResponse) =>
httpResponse.entity.dataBytes.runWith(Sink.ignore)
println(s"status: ${httpResponse.status}")
case Failure(e) => {
println(e)
}
}
}
}
The output looks like this:
Creating request
Creating request
Creating request
Creating request
Creating request
Creating request
Creating request
Creating request
Creating request
Creating request
status: 200 OK
Creating request
status: 200 OK
Creating request
status: 200 OK
...
I am failing to find any configuration parameters which would allow emitting responses as soon as they are ready and not when the pool is out of free connections.
Thanks!
The reason is that you block the client from doing other work by calling Thread.sleep—that method is simply forbidden inside reactive programs. The proper and simpler approach is to use Source.tick.