While using Akka HttpRequest and pipe the request to an actor, i couldn't identify the response.
The actor will handle each message that will receive but it doesn't know which request used to get this response. Is there any way to identify each request to match the response with ?
Note: i don't have the server to resend any part of request body again.
Thanks in advance
MySelf.scala
import akka.actor.{ Actor, ActorLogging }
import akka.http.scaladsl.Http
import akka.http.scaladsl.model._
import akka.stream.{ ActorMaterializer, ActorMaterializerSettings }
import akka.util.ByteString
class Myself extends Actor with ActorLogging {
import akka.pattern.pipe
import context.dispatcher
final implicit val materializer: ActorMaterializer =
ActorMaterializer(ActorMaterializerSettings(context.system))
def receive = {
case HttpResponse(StatusCodes.OK, headers, entity, _) =>
entity.dataBytes.runFold(ByteString(""))(_ ++ _).foreach { body =>
log.info("Got response, body: " + body.utf8String)
}
case resp # HttpResponse(code, _, _, _) =>
log.info("Request failed, response code: " + code)
resp.discardEntityBytes()
}
}
Main.scala
import akka.actor.{ActorSystem, Props}
import akka.http.scaladsl.Http
import akka.http.scaladsl.model._
import akka.stream.ActorMaterializer
object HttpServerMain extends App {
import akka.pattern.pipe
// import system.dispatcher
implicit val system = ActorSystem()
implicit val materializer = ActorMaterializer()
// needed for the future flatMap/onComplete in the end
implicit val executionContext = system.dispatcher
val http = Http(system)
val myActor = system.actorOf(Props[MySelf])
http.singleRequest(HttpRequest(uri = "http://akka.io"))
.pipeTo(myActor)
http.singleRequest(HttpRequest(uri = "http://akka.io/another-request"))
.pipeTo(myActor)
Thread.sleep(2000)
system.terminate()
You can simply use map to transform the Future and add some kind of ID (usually called correlation ID for such purposes) to it before you pipe it to myActor:
http.singleRequest(HttpRequest(uri = "http://akka.io"))
.map(x => (1, x)).pipeTo(myActor)
You'll need to change you pattern match blocks to take a tupple:
case (id, HttpResponse(StatusCodes.OK, headers, entity, _)) =>
If you can't/don't want to change your pattern match block for some reason you can use same approach, but instead add a unique HTTP header into your completed request (using copy) with something like this (not checked if compiles):
// make a unique header name that you are sure will not be
// received from http response:
val correlationHeader: HttpHeader = ... // mycustomheader
// Basically hack the response to add your header:
http.singleRequest(HttpRequest(uri = "http://akka.io"))
.map(x => x.copy(headers = correlationHeader +: headers)).pipeTo(myActor)
// Now you can check your header to see which response that was:
case HttpResponse(StatusCodes.OK, headers, entity, _) =>
headers.find(_.is("mycustomheader")).map(_.value).getOrElse("NA")
This is more of a hack though compared to previous option because you are modifying a response.
I think you cannot do that directly using pipeTo because it essentially just adds andThen call to your Future. One option is tomap and then send a (request, response) tuple to actor:
val request = HttpRequest(uri = "http://akka.io")
http.singleRequest(request).map {
response => myActor ! (request, response)
}
class Myself extends Actor with ActorLogging {
...
def receive = {
case (request, HttpResponse(StatusCodes.OK, headers, entity, _)) =>
...
case (request, resp # HttpResponse(code, _, _, _)) =>
log.info(request.toString)
...
}
}
Related
I am currently building a client-side WebSockets consumer using Akka-HTTP. Instead of trying to do the parsing in the Sink, I wanted to wrap the code in a function which emits the outcome from the Sink, and then use this output (from the function) later for further processing (more parsing...etc.).
I am currently able to print every message from the Sink; however, the return type of the function remains to be Unit. My objective is to Emit a String from the function, for each item that lands in the sink, and then use the returned string to do further parsing. I have the code I have so far (Note: it's mostly boiler plate).
import java.util.concurrent.atomic.AtomicInteger
import akka.Done
import akka.actor.ActorSystem
import akka.http.scaladsl.Http
import akka.http.scaladsl.model.StatusCodes
import akka.http.scaladsl.model.ws.{Message, TextMessage, WebSocketRequest, WebSocketUpgradeResponse}
import akka.http.scaladsl.settings.ClientConnectionSettings
import akka.stream.Materializer
import akka.stream.scaladsl.{Flow, Keep, Sink, Source}
import akka.util.ByteString
import scala.concurrent.ExecutionContext.Implicits.global
import scala.concurrent.Future
import scala.util.{Failure, Success, Try}
object client extends App {
def parseData(uri: String)(implicit system: ActorSystem, materializer: Materializer): Unit = {
val defaultSettings = ClientConnectionSettings(system)
val pingCounter = new AtomicInteger()
val customWebsocketSettings = defaultSettings.websocketSettings.withPeriodicKeepAliveData(
() => ByteString(s"debug-${pingCounter.incrementAndGet()}")
)
val customSettings = defaultSettings.withWebsocketSettings(customWebsocketSettings)
val outgoing = Source.maybe[Message]
val sink: Sink[Message, Future[Done]] = Sink.foreach[Message] {
case message: TextMessage.Strict => message.text // I Want to emit/stream this message as a String from the function (or get a handle on it from the outside)
case _ => println("Other")
}
val webSocketFlow: Flow[Message, Message, Future[WebSocketUpgradeResponse]] =
Http().webSocketClientFlow(WebSocketRequest(uri), settings = customSettings)
val (upgradeResponse, closed) =
outgoing
.viaMat(webSocketFlow)(Keep.right)
.toMat(sink)(Keep.both)
.run()
val connected = upgradeResponse.flatMap { upgrade =>
if (upgrade.response.status == StatusCodes.SwitchingProtocols) {
Future.successful(Done)
} else {
throw new RuntimeException(
s"Connection failed: ${upgrade.response.status}"
)
}
}
connected.onComplete {
case Success(value) => value
case Failure(exception) => throw exception
}
closed.onComplete { _ =>
println("Retrying...")
parseData(uri)
}
upgradeResponse.onComplete {
case Success(value) => println(value)
case Failure(exception) => throw exception
}
}
}
And in a seperate object, I would like to do the parsing, so something like:
import akka.actor.ActorSystem
import akka.stream.Materializer
import api.client.parseData
object Application extends App {
implicit val system: ActorSystem = ActorSystem()
implicit val materializer: Materializer = Materializer(system)
val uri = "ws://localhost:8080/foobar"
val res = parseData(uri) // I want to handle the function output here
// parse(res)
println(res)
Is there a way I can get a handle on the Sink from outside the function, or do I need to do any parsing in the Sink. I am mainly trying to not overcomplicate the Sink.
Update: I am also considering if adding another Flow element to the stream (which handles the parsing) is a better practice than getting values outside of the stream.
Adding a flow element seems to solve your problem while being totally idiomatic.
What you have to keep in mind is that the sinks semantic is meant to describe how to "terminate" the stream, so while it can describe very complex computations, it will always return a single value which is returned only once the stream ends.
Said differently, a sink does not return a value per stream element, it returns a value per whole stream.
Getting started with Akka Streams I want to perform a simple computation. Extending the basic QuickStart https://doc.akka.io/docs/akka/2.5/stream/stream-quickstart.html with a call to a restful web api:
val source: Source[Int, NotUsed] = Source(1 to 100)
source.runForeach(println)
already works nicely to print the numbers. But when trying to create an Actor to perform the HTTP request (is this actually necessary?) according to https://doc.akka.io/docs/akka/2.5.5/scala/stream/stream-integrations.html
import akka.pattern.ask
implicit val askTimeout = Timeout(5.seconds)
val words: Source[String, NotUsed] =
Source(List("hello", "hi"))
words
.mapAsync(parallelism = 5)(elem => (ref ? elem).mapTo[String])
// continue processing of the replies from the actor
.map(_.toLowerCase)
.runWith(Sink.ignore)
I cannot get it to compile as the ? operator is not defined. As ar as I know this one would only be defined inside an actor.
I also do not understand yet where exactly inside mapAsync my custom actor needs to be called.
edit
https://blog.colinbreck.com/backoff-and-retry-error-handling-for-akka-streams/ contains at least parts of an example.
It looks like it is not mandatory to create an actor i.e.
implicit val system = ActorSystem()
implicit val ec = system.dispatcher
implicit val materializer = ActorMaterializer()
val source = Source(List("232::03::14062::19965186", "232::03::14062::19965189"))
.map(cellKey => {
val splits = cellKey.split("::")
val mcc = splits(0)
val mnc = splits(1)
val lac = splits(2)
val ci = splits(3)
CellKeySource(cellKey, mcc, mnc, lac, ci)
})
.limit(2)
.mapAsyncUnordered(2)(ck => getResponse(ck.cellKey, ck.mobileCountryCode, ck.mobileNetworkCode, ck.locationArea, ck.cellKey)("<<myToken>>"))
def getResponse(cellKey: String, mobileCountryCode:String, mobileNetworkCode:String, locationArea:String, cellId:String)(token:String): Future[String] = {
RestartSource.withBackoff(
minBackoff = 10.milliseconds,
maxBackoff = 30.seconds,
randomFactor = 0.2,
maxRestarts = 2
) { () =>
val responseFuture: Future[HttpResponse] =
Http().singleRequest(HttpRequest(uri = s"https://www.googleapis.com/geolocation/v1/geolocate?key=${token}", entity = ByteString(
// TODO use proper JSON objects
s"""
|{
| "cellTowers": [
| "mobileCountryCode": $mobileCountryCode,
| "mobileNetworkCode": $mobileNetworkCode,
| "locationAreaCode": $locationArea,
| "cellId": $cellId,
| ]
|}
""".stripMargin)))
Source.fromFuture(responseFuture)
.mapAsync(parallelism = 1) {
case HttpResponse(StatusCodes.OK, _, entity, _) =>
Unmarshal(entity).to[String]
case HttpResponse(statusCode, _, _, _) =>
throw WebRequestException(statusCode.toString() )
}
}
.runWith(Sink.head)
.recover {
case _ => throw StreamFailedAfterMaxRetriesException()
}
}
val done: Future[Done] = source.runForeach(println)
done.onComplete(_ ⇒ system.terminate())
is already the (partial) answer for the question i.e. how to integrate Akka-streams + akka-http. However, it does not work, i.e. only throws error 400s and never terminates.
i think you already found an api how to call akka-http client
regarding your first code snippet which doesn't work. i think there happened some misunderstanding of the example itself. you expected the code in the example to work after just copied. but the intension of the doc was to demonstrate just an example/concept, how you can delegate some long running task out of the stream flow and then consuming the result when it's ready. for this was used ask call to akka actor, because call to ask method returns a Future. probably the authors of the doc just omitted the definition of actor. you can try this one example:
import java.lang.System.exit
import akka.NotUsed
import akka.actor.{Actor, ActorRef, ActorSystem, Props}
import akka.pattern.ask
import akka.stream.ActorMaterializer
import akka.stream.scaladsl.{Sink, Source}
import akka.util.Timeout
import scala.concurrent.ExecutionContext.Implicits.global
import scala.concurrent.duration._
import scala.language.higherKinds
object App extends scala.App {
implicit val sys: ActorSystem = ActorSystem()
implicit val mat: ActorMaterializer = ActorMaterializer()
val ref: ActorRef = sys.actorOf(Props[Translator])
implicit val askTimeout: Timeout = Timeout(5.seconds)
val words: Source[String, NotUsed] = Source(List("hello", "hi"))
words
.mapAsync(parallelism = 5)(elem => (ref ? elem).mapTo[String])
.map(_.toLowerCase)
.runWith(Sink.foreach(println))
.onComplete(t => {
println(s"finished: $t")
exit(1)
})
}
class Translator extends Actor {
override def receive: Receive = {
case msg => sender() ! s"$msg!"
}
}
You must import ask pattern from akka.
import akka.pattern.ask
Edit: OK, sorry, I can see that you have already imported. What is ref in your code? ActorRef?
I'm trying to implement an NNTP client that streams a list of commands to the server and parsing the results back. I'm facing several problems :
the NNTP protocol doesn't have an "unique" delimiter that could be use to frame results. Some commands return multi-line responses. How to handle that with streams ?
how to "map" the command issued with the server response and wait the end of server response before sending the next command ? (Throttling is not relevant here)
how to stop the stream processing on disconnection ? (Actually, the program never returns)
Here is my current implementation :
import akka.stream._
import akka.stream.scaladsl._
import akka.{ NotUsed, Done }
import akka.actor.ActorSystem
import akka.util.ByteString
import scala.concurrent._
import scala.concurrent.duration._
import java.nio.file.Paths
import scala.io.StdIn
import scala.concurrent.ExecutionContext.Implicits.global
import scala.util.Success
import scala.util.Failure
object AutomatedClient extends App {
implicit val system = ActorSystem("NewsClientTest")
implicit val materializer = ActorMaterializer()
// MODEL //
final case class Command(query: String)
final case class CommandResult(
resultCode: Int,
resultStatus: String,
resultList: Option[List[String]])
final case class ParseException(message: String) extends RuntimeException
// COMMAND HANDLING FUN //
// out ->
val sendCommand: Command => ByteString = c => ByteString(c.query + "\r\n")
// in <-
val parseCommandResultStatus: String => (Int, String) = s =>
(s.take(3).toInt, s.drop(3).trim)
val parseCommandResultList: List[String] => List[String] = l =>
l.foldLeft(List().asInstanceOf[List[String]]){
case (acc, ".") => acc
case (acc, e) => e.trim :: acc
}.reverse
val parseCommandResult: ByteString => Future[CommandResult] = b => Future {
val resultLines = b.decodeString("UTF-8").split("\r\n")
resultLines.length match {
case 0 => throw new ParseException("empty result")
case 1 =>
val (code, text) = parseCommandResultStatus(resultLines.head)
new CommandResult(code, text, None)
case _ =>
val (code, text) = parseCommandResultStatus(resultLines.head)
new CommandResult(code, text, Some(parseCommandResultList(resultLines.tail.toList)))
}
}
// STREAMS //
// Flows
val outgoing: Flow[Command, ByteString, NotUsed] = Flow fromFunction sendCommand
val incoming: Flow[ByteString, Future[CommandResult], NotUsed] = Flow fromFunction parseCommandResult
val protocol = BidiFlow.fromFlows(incoming, outgoing)
// Sink
val print: Sink[Future[CommandResult], _] = Sink.foreach(f =>
f.onComplete {
case Success(r) => println(r)
case Failure(r) => println("error decoding command result")
})
// Source
val testSource: Source[Command, NotUsed] = Source(List(
new Command("help"),
new Command("list"),
new Command("quit")
))
val (host, port) = ("localhost", 1119)
Tcp()
.outgoingConnection(host, port)
.join(protocol)
.runWith(testSource, print)
}
And here is the result output :
CommandResult(200,news.localhost NNRP Service Ready - newsmaster#localhost (posting ok),None)
CommandResult(100,Legal Commands,Some(List(article [<messageid>|number], authinfo type value, body [<messageid>|number], date, group newsgroup, head [<messageid>|number], help, last, list [active wildmat|active.times|counts wildmat], list [overview.fmt|newsgroups wildmat], listgroup newsgroup, mode reader, next, post, stat [<messageid>|number], xhdr field [range], xover [range], xpat field range pattern, xfeature useragent <client identifier>, xfeature compress gzip [terminator], xzver [range], xzhdr field [range], quit, 480 Authentication Required*, 205 Goodbye)))
We can see that the second CommandResult contains the result of "list" command and "quit" command.
I am trying to make a GET request to a REST web service using Akka Http Client.
I am not able to figure out how do I set a cookie on the request before I make the GET.
I searched the web and I found ways to read the cookie on the server side. but I could not find anything which showed me how to set the cookie on the client side request.
Based on my own research I tried the following approach to set a cookie on http request
import akka.actor.ActorSystem
import akka.http.scaladsl.Http
import akka.http.scaladsl.model._
import akka.http.scaladsl.unmarshalling.Unmarshal
import akka.stream.scaladsl.{Sink, Source}
import akka.http.scaladsl.marshallers.sprayjson.SprayJsonSupport
import akka.http.scaladsl.model.headers.HttpCookie
import akka.stream.ActorMaterializer
import spray.json._
import scala.util.{Failure, Success}
case class Post(postId: Int, id: Int, name: String, email: String, body: String)
trait JsonSupport extends SprayJsonSupport with DefaultJsonProtocol {
implicit val postFormat = jsonFormat5(Post.apply)
}
object AkkaHttpClient extends JsonSupport{
def main(args: Array[String]) : Unit = {
val cookie = headers.`Set-Cookie`(HttpCookie(name="foo", value="bar"))
implicit val system = ActorSystem("my-Actor")
implicit val actorMaterializer = ActorMaterializer()
implicit val executionContext = system.dispatcher
val mycookie = HttpCookie(name="foo", value="bar")
val httpClient = Http().outgoingConnection(host = "jsonplaceholder.typicode.com")
val request = HttpRequest(uri = Uri("/comments"), headers = List(cookie))
val flow = Source.single(request)
.via(httpClient)
.mapAsync(1)(r => Unmarshal(r.entity).to[List[Post]])
.runWith(Sink.head)
flow.andThen {
case Success(list) => println(s"request succeded ${list.size}")
case Failure(_) => println("request failed")
}.andThen {
case _ => system.terminate()
}
}
}
But this gives an error
[WARN] [08/05/2016 10:50:11.134] [my-Actor-akka.actor.default-dispatcher-3] [akka.actor.ActorSystemImpl(my-Actor)]
HTTP header 'Set-Cookie: foo=bar' is not allowed in requests
The idiomatic way to construct any header for an akka-http client is by
using akka.http.scaladsl.model.headers.
In your case it would be
val cookieHeader = akka.http.scaladsl.model.headers.Cookie("name","value")
HttpRequest(uri = Uri("/comments"), headers = List(cookieHeader, ...))
The outgoing header must be 'Cookie' not 'Set-Cookie':
val cookie = HttpCookiePair("foo", "bar")
val headers: immutable.Seq[HttpHeader] = if (cookies.isEmpty) immutable.Seq.empty else immutable.Seq(Cookie(cookies))
val request = HttpRequest(uri = uri).withHeadersAndEntity(headers, HttpEntity(msg))
So I have a function with this signature (akka.http.model.HttpResponse):
def apply(query: Seq[(String, String)], accept: String): HttpResponse
I simply get a value in a test like:
val resp = TagAPI(Seq.empty[(String, String)], api.acceptHeader)
I want to check its body in a test something like:
resp.entity.asString == "tags"
My question is how I can get the response body as string?
import akka.http.scaladsl.unmarshalling.Unmarshal
implicit val system = ActorSystem("System")
implicit val materializer = ActorFlowMaterializer()
val responseAsString: Future[String] = Unmarshal(entity).to[String]
Since Akka Http is streams based, the entity is streaming as well. If you really need the entire string at once, you can convert the incoming request into a Strict one:
This is done by using the toStrict(timeout: FiniteDuration)(mat: Materializer) API to collect the request into a strict entity within a given time limit (this is important since you don't want to "try to collect the entity forever" in case the incoming request does actually never end):
import akka.stream.ActorFlowMaterializer
import akka.actor.ActorSystem
implicit val system = ActorSystem("Sys") // your actor system, only 1 per app
implicit val materializer = ActorFlowMaterializer() // you must provide a materializer
import system.dispatcher
import scala.concurrent.duration._
val timeout = 300.millis
val bs: Future[ByteString] = entity.toStrict(timeout).map { _.data }
val s: Future[String] = bs.map(_.utf8String) // if you indeed need a `String`
You can also try this one also.
responseObject.entity.dataBytes.runFold(ByteString(""))(_ ++ _).map(_.utf8String) map println
Unmarshaller.stringUnmarshaller(someHttpEntity)
works like a charm, implicit materializer needed as well
Here is simple directive that extracts string from request's body
def withString(): Directive1[String] = {
extractStrictEntity(3.seconds).flatMap { entity =>
provide(entity.data.utf8String)
}
}
Unfortunately in my case, Unmarshal to String didn't work properly complaining on: Unsupported Content-Type, supported: application/json. That would be more elegant solution, but I had to use another way. In my test I used Future extracted from entity of the response and Await (from scala.concurrent) to get the result from the Future:
Put("/post/item", requestEntity) ~> route ~> check {
val responseContent: Future[Option[String]] =
response.entity.dataBytes.map(_.utf8String).runWith(Sink.lastOption)
val content: Option[String] = Await.result(responseContent, 10.seconds)
content.get should be(errorMessage)
response.status should be(StatusCodes.InternalServerError)
}
If you need to go through all lines in a response, you can use runForeach of Source:
response.entity.dataBytes.map(_.utf8String).runForeach(data => println(data))
Here is my working example,
import akka.actor.ActorSystem
import akka.http.scaladsl.Http
import akka.http.scaladsl.model._
import akka.stream.ActorMaterializer
import akka.util.ByteString
import scala.concurrent.Future
import scala.util.{ Failure, Success }
def getDataAkkaHTTP:Unit = {
implicit val system = ActorSystem()
implicit val materializer = ActorMaterializer()
// needed for the future flatMap/onComplete in the end
implicit val executionContext = system.dispatcher
val url = "http://localhost:8080/"
val responseFuture: Future[HttpResponse] = Http().singleRequest(HttpRequest(uri = url))
responseFuture.onComplete {
case Success(res) => {
val HttpResponse(statusCodes, headers, entity, _) = res
println(entity)
entity.dataBytes.runFold(ByteString(""))(_ ++ _).foreach (body => println(body.utf8String))
system.terminate()
}
case Failure(_) => sys.error("something wrong")
}
}