I am currently playing around with akka streams and tried the following example.
Get the first element from kafka when requesting a certain HTTP endpoint.
This is the code I wrote and its working.
get {
path("ticket" / IntNumber) { ticketNr =>
val future = Consumer.plainSource(consumerSettings, Subscriptions.topics("tickets"))
.take(1)
.completionTimeout(5 seconds)
.runWith(Sink.head)
onComplete(future) {
case Success(record) => complete(HttpEntity(ContentTypes.`text/html(UTF-8)`, record.value()))
case _ => complete(HttpResponse(StatusCodes.NotFound))
}
}
}
I am just wondering if this is the ideomatic way of working with (akka) streams.
So is there a more "direct" way of connecting the kafka stream to the HTTP response stream?
For example, when POSTing I do this:
val kafkaTicketsSink = Flow[String]
.map(new ProducerRecord[Array[Byte], String]("tickets", _))
.toMat(Producer.plainSink(producerSettings))(Keep.right)
post {
path("ticket") {
(entity(as[Ticket]) & extractMaterializer) { (ticket, mat) => {
val f = Source.single(ticket).map(t => t.description).runWith(kafkaTicketsSink)(mat)
onComplete(f) { _ =>
val locationHeader = headers.Location(s"/ticket/${ticket.id}")
complete(HttpResponse(StatusCodes.Created, headers = List(locationHeader)))
}
}
}
}
}
Maybe this can also be improved??
You could keep a single, backpressured stream alive using Sink.queue. You can pull an element from the materialized queue every time a request is received. This should give you back one element if available, and backpressure otherwise.
Something along the lines of:
val queue = Consumer.plainSource(consumerSettings, Subscriptions.topics("tickets"))
.runWith(Sink.queue())
get {
path("ticket" / IntNumber) { ticketNr =>
val future: Future[Option[ConsumerRecord[String, String]]] = queue.pull()
onComplete(future) {
case Success(Some(record)) => complete(HttpEntity(ContentTypes.`text/html(UTF-8)`, record.value()))
case _ => complete(HttpResponse(StatusCodes.NotFound))
}
}
}
More info on Sink.queue can be found in the docs.
Related
Because I do some "complex" operations, I think my Actor became asyncronous. The main problem I think is that I use Await.result inside the method which return responses.
actor:
def process(subscribers: Set[ActorRef]): Receive = {
case Join(ref) => context become process(subscribers + ref)
case Leave(ref) => context become process(subscribers - ref)
case Push(request) =>
val filteredSubscribers = (subscribers - sender())
.filter(s => exists(s, request)) // just some actor filters
filteredSubscribers.foreach { subscriber =>
// here I have a Map with each actor requests
val actorOptions = getActorOptions(subscriber)
subscriber ? getResponse(actorOptions, request)
}
}
The problem is inside getResponse (I think).
getResponse(actorOptions: JsValue, request: SocketRequest): JsValue = {
(actorOptions \ "dashboardId").asOpt[Int] match {
case Some(id) => {
val response = widgetsService.getByDashboadId(id) map { widgets =>
val widgetsResponse: List[Future[String]] = widgets.map(w => {
widgetsService.getDataById(w.id) map {
data => s"""{ "widgetId": ${w.id}, "data": $data }"""
}
})
var responses: List[String] = List.empty
widgetsResponse.foreach(f => {
f.onComplete {
case Success(value) => responses = value :: responses
case Failure(e) => println(s"Something happened: ${e.getMessage}")
}
})
// first time when I use Await.result
// used to populate the responses list with data from all futures
Await.result(Future.sequence(widgetsResponse), Duration.Inf)
Json.parse(s"""{
"dashboardId": $id,
"widgets": [${response.mkString(", ")}]
}""".stripMargin)
}
// second time when I use Await.result
// used to return a JsValue instead of a Future[JsValue]
Await.result(response, Duration.Inf)
}
case None => buildDefaultJson // return default json value, unimportant for this example
}
}
Due of that, In frontend, if I have 2 sockets clients, the response for the second will be send only after first.
I found that I can obtain a "fake" increase of performance if I embrance the getResponse in a future inside of my Actor.
filteredSubscribers.foreach { subscriber =>
val actorOptions = getActorOptions(subscriber)
Future(subscriber ? getResponse(actorOptions, request))
}
So, for both subscribers the action will be started in same time, but when the first will reach the Await.result, the second will be locked until first is done.
I need to avoid using Await.result there, but I don't know how to get the results of a list of futures, without using for-comprehension (because is a dynamically list) for first time where I use it.
Because Akka ask operator (?) return a Future[Any], I tried that my getResponse method to return directly a JsValue to be mapped then in Future[JsValue]. If I remove the second Await.result and my method will return Future[JsValue], then the actor will return a Future[Future[JsValue]] which I don't think is too right.
After some more researches and solutions found on so, my code become:
Future.sequence(widgetsResponse) map { responses =>
Json.parse(
s"""
|{
|"dashboardId": $id,
|"tableSourceId": $tableSourceId,
|"widgets": [ ${responses.mkString(", ")}]
|}""".stripMargin
)
}
getResponse returns a Future[JsValue] now, removing both Await.result cases, and actor case become:
filteredSubscribers.foreach { subscriber =>
val actorOptions = getActorOptions(subscriber)
getResponse(actorOptions, request) map { data =>
subscriber ? data
}
}
I don't know why, still have a synchronous behavior. Damn, this can be due of my subscribers type: Set[ActorRef]? I tried to use parallel foreach and this looks like solving my problem:
filteredSubscribers.par.foreach { subscriber =>
val actorOptions = getActorOptions(subscriber)
getResponse(actorOptions, request) map { data =>
subscriber ? data
}
}
i am using akka http one of my routes is interacting with an external service via akka http client side api and the httpRequest is continuously running i am unable to make it work
here is my use case -> i am interacting with a janus server and doing a long poll get request as soon as the server responded back with an 'keepAlive' or an "event" i am requesting again and so on the server keeps on responding
all of this is happening inside an actor and i have an akka htttp route which is intiailising the first request
here is my code
final case class CreateLongPollRequest(sessionId:BigInt)
class LongPollRequestActor (config: Config) extends Actor {
def receive = {
case CreateLongPollRequest(sessionId) =>
senderRef = Some(sender())
val uri: String = "localhost:8080/" + sessionId
val request = HttpRequest(HttpMethods.GET, uri)
val responseFuture = Http(context.system).singleRequest(request)
responseFuture
.onComplete {
case Success(res)
Unmarshal(res.entity.toStrict(40 seconds)).value.map { result =>
val responseStr = result.data.utf8String
log.info("Actor LongPollRequestActor: long poll responseStr {}",responseStr)
senderRef match {
case Some(ref) =>
ref ! responseStr
case None => log.info("Actor LongPollRequestActor: sender ref is null")
}
}
case Failure(e) =>log.error(e)
}
}
}
final case class JanusLongPollRequest(sessionId: BigInt)
class JanusManagerActor(childMaker: List[ActorRefFactory => ActorRef]) extends Actor {
var senderRef: Option[akka.actor.ActorRef] = None
val longPollRequestActor = childMaker(1)(context)
def receive: PartialFunction[Any, Unit] = {
case JanusLongPollRequest(sessionId)=>
senderRef = Some(sender)
keepAlive(sessionId,senderRef)
}
def keepAlive(sessionId:BigInt,sender__Ref: Option[ActorRef]):Unit= {
val senderRef = sender__Ref
val future = ask(longPollRequestActor, CreateLongPollRequest(sessionId)).mapTo[String] //.pipeTo(sender)
if (janus.equals("keepalive")) {
val janusRequestResponse = Future {
JanusSessionRequestResponse(janus = janus)
}
senderRef match {
case Some(sender_ref) =>
janusRequestResponse.pipeTo(sender_ref)
}
keepAlive(sessionId,senderRef)
}
else if (janus.equals("event")) {
//some fetching of values from server
val janusLongPollRequestResponse = Future {
JanusLongPollRequestResponse(janus = janus,sender=sender, transaction=transaction,pluginData=Some(pluginData))
}
senderRef match {
case Some(sender_ref) =>
janusLongPollRequestResponse.pipeTo(sender_ref)
}
keepAlive(sessionId,senderRef)
}
def createLongPollRequest: server.Route =
path("create-long-poll-request") {
post {
entity(as[JsValue]) {
json =>
val sessionID = json.asJsObject.fields("sessionID").convertTo[String]
val future = ask(janusManagerActor, JanusLongPollRequest(sessionID)).mapTo[JanusSessionRequestResponse]
onComplete(future) {
case Success(sessionDetails) =>
log.info("janus long poll request created")
val jsonResponse = JsObject("longpollDetails" -> sessionDetails.toJson)
complete(OK, routeResponseMessage.getResponse(StatusCodes.OK.intValue, ServerMessages.JANUS_SESSION_CREATED, jsonResponse))
case Failure(ex) =>
failWith(ex)
}
}
}
now the above route createLongPollRequest worked fine for the first time I can see the response and for the next attempts i am getting a dead letter as follows
[INFO] [akkaDeadLetter][07/30/2021 12:13:53.587] [demo-Janus-ActorSystem-akka.actor.default-dispatcher-6] [akka://demo-Janus-ActorSystem/deadLetters] Message [com.ifkaar.lufz.janus.models.janus.JanusSessionRequestResponse] from Actor[akka://demo-Janus-ActorSystem/user/ActorManager/ManagerActor#-721316187] to Actor[akka://demo-Janus-ActorSystem/deadLetters] was not delivered. [4] dead letters encountered. If this is not an expected behavior then Actor[akka://demo-Janus-ActorSystem/deadLetters] may have terminated unexpectedly. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'.
probably this is causing the issue after the first iteration
responseFuture.pipeTo(sender()
IS there a way where i can get a response in my akkahttp route when ever my backend server responds?
The Actor should only reply once to the CreateLongPollRequest and it should only do this when it has valid data. If the poll fails the Actor should just issue another poll request.
It is difficult to give more help without the details of the Actor.
I am trying to build a money transaction system using akka-http for REST API and akka actors for AccountActors.
post {
(path("accounts" / "move-money") & entity(as[MoveMoneyRequest])) { moveMoneyRequest =>
complete(
(bankActor ? moveMoneyRequest).map(x => MoveMoneyResponse("Money Transfer Successful!"))
)
}
}
The bankActor is created inside a main app
val bankActor = mainActorSystem.actorOf(Props(classOf[BankingActor], accountService), name = "bankActor")
Inside BankActor, we have:
def receive: Receive = LoggingReceive {
case req: MoveMoneyRequest =>
val fromAcc = createAccountActor(Some(req.fromAccount))
val toAcc = createAccountActor(Some(req.toAccount))
fromAcc ? DebitAccount(req.tranferAmount)
become(awaitFrom(fromAcc, toAcc, req.tranferAmount, sender))
}
private def createAccountActor(accountNum: Option[String]): ActorRef = {
actorOf(Props(classOf[AccountActor], accountNum, accountService))
}
Question: Now, for the first API call everytime, it's successful but seems the actor dies/shuts down and the ? (ask) does not find the actor as the message does not reach the receive method. Do I need to make the ask call different?
The correct directive to deal with futures is onComplete, for example
post {
(path("accounts" / "move-money") & entity(as[MoveMoneyRequest])) { moveMoneyRequest =>
val fut = (bankActor ? moveMoneyRequest).map(x => MoveMoneyResponse("Money Transfer Successful!"))
onComplete(fut){
case util.Success(_) => complete(StatusCodes.OK)
case util.Failure(ex) => complete(StatusCodes.InternalServerError)
}
}
}
More details in the docs.
I'm using secure social and akka to send messages to a web client via a web socket using play frame work 2.3
Websocket
def feedWS = WebSocket.tryAccept[JsValue] { implicit request =>
SecureSocial.currentUser[PDUser].map {
maybeUser => maybeUser match {
case Some(user) =>
Right(Feed.start(user.id))
case None => {
Left(Redirect("/login"))
}
}
}
But the map returns a future and Feed.start returns an future too. Something like
Object serving ws and connecting to an actor that pumps messages out (actor works fine)
def start(user: String): Future[(Iteratee[JsValue, _], Enumerator[JsValue])] = {
(actorFor(user) ? connect(user)).map {
case ConnectedE(enumerator) => {
val iteratee = Iteratee.foreach[JsValue] { event =>
val things = FromJson.aList(event)
actorFor(user) ! Subscribe(things)
}.map { _ =>
log.info("Disconnected")
actorFor(user) ! UnSubscribe
}
(iteratee, enumerator)
}
}
}
How do I compose them, if indeed I can or does it need an await?
At the moment I have a future of a future.
The below code does streaming back to client, in, what I gather is a more idiomatic way than using Java's IO Streams. It, however, has an issue: connection is kept open after stream is done.
def getImage() = Action { request =>
val imageUrl = "http://hereandthere.com/someimageurl.png"
Ok.stream({ content: Iteratee[Array[Byte], Unit] =>
WS.url(imageUrl).withHeaders("Accept"->"image/png").get { response => content }
return
}).withHeaders("Content-Type"->"image/png")
}
this is intended for streaming large (>1 mb) files from internal API to requester.
The question is, why does it keep the connection open? Is there something it expects from upstream server? I tested the upstream server using curl, and the connection does close - it just doesn't close when passed through this proxy.
The reason that the stream doesn't finish is because an EOF isn't sent to the iteratee that comes back from WS.get() call. Without this explicit EOF, the connection stays open - as it's in chunked mode, and potentially a long-running, comet-like connection.
Here's the fixed code:
Ok.stream({ content: Iteratee[Array[Byte], Unit] =>
WS.url(imageUrl)
.withHeaders("Accept"->"image/png")
.get { response => content }
.onRedeem { ii =>
ii.feed(Input.EOF)
}
}).withHeaders("Content-Type"->"image/png")
Here is a modified version for play 2.1.0. See https://groups.google.com/forum/#!msg/play-framework/HwoRR-nipCc/gUKs9NexCx4J
Thanks Anatoly G for sharing.
def proxy = Action {
val url = "..."
Async {
val iterateePromise = Promise[Iteratee[Array[Byte], Unit]]
val resultPromise = Promise[ChunkedResult[Array[Byte]]]
WS.url(url).get { responseHeaders =>
resultPromise.success {
new Status(responseHeaders.status).stream({ content: Iteratee[Array[Byte], Unit] =>
iterateePromise.success(content)
}).withHeaders(
"Content-Type" -> responseHeaders.headers.getOrElse("Content-Type", Seq("application/octet-stream")).head,
"Connection" -> "Close")
}
Iteratee.flatten(iterateePromise.future)
}.onComplete {
case Success(ii) => ii.feed(Input.EOF)
case Failure(t) => resultPromise.failure(t)
}
resultPromise.future
}
}
Update for play 2.2.x:
def proxy = Action.async {
val url = "http://localhost:9000"
def enumerator(chunks: Iteratee[Array[Byte], Unit] => _) = {
new Enumerator[Array[Byte]] {
def apply[C](i: Iteratee[Array[Byte], C]): Future[Iteratee[Array[Byte], C]] = {
val doneIteratee = Promise[Iteratee[Array[Byte], C]]()
chunks(i.map {
done =>
doneIteratee.success(Done[Array[Byte], C](done)).asInstanceOf[Unit]
})
doneIteratee.future
}
}
}
val iterateePromise = Promise[Iteratee[Array[Byte], Unit]]()
val resultPromise = Promise[SimpleResult]()
WS.url(url).get {
responseHeaders =>
resultPromise.success(new Status(responseHeaders.status).chunked(
enumerator({
content: Iteratee[Array[Byte], Unit] => iterateePromise.success(content)
}
)).withHeaders(
"Content-Type" -> responseHeaders.headers.getOrElse("Content-Type", Seq("application/octet-stream")).head,
"Connection" -> "Close"))
Iteratee.flatten(iterateePromise.future)
}.onComplete {
case Success(ii) => ii.feed(Input.EOF)
case Failure(t) => throw t
}
resultPromise.future
}
if anyone has a better solution, it interests me greatly!