akka HttpResponse read body as String scala - scala

So I have a function with this signature (akka.http.model.HttpResponse):
def apply(query: Seq[(String, String)], accept: String): HttpResponse
I simply get a value in a test like:
val resp = TagAPI(Seq.empty[(String, String)], api.acceptHeader)
I want to check its body in a test something like:
resp.entity.asString == "tags"
My question is how I can get the response body as string?

import akka.http.scaladsl.unmarshalling.Unmarshal
implicit val system = ActorSystem("System")
implicit val materializer = ActorFlowMaterializer()
val responseAsString: Future[String] = Unmarshal(entity).to[String]

Since Akka Http is streams based, the entity is streaming as well. If you really need the entire string at once, you can convert the incoming request into a Strict one:
This is done by using the toStrict(timeout: FiniteDuration)(mat: Materializer) API to collect the request into a strict entity within a given time limit (this is important since you don't want to "try to collect the entity forever" in case the incoming request does actually never end):
import akka.stream.ActorFlowMaterializer
import akka.actor.ActorSystem
implicit val system = ActorSystem("Sys") // your actor system, only 1 per app
implicit val materializer = ActorFlowMaterializer() // you must provide a materializer
import system.dispatcher
import scala.concurrent.duration._
val timeout = 300.millis
val bs: Future[ByteString] = entity.toStrict(timeout).map { _.data }
val s: Future[String] = bs.map(_.utf8String) // if you indeed need a `String`

You can also try this one also.
responseObject.entity.dataBytes.runFold(ByteString(""))(_ ++ _).map(_.utf8String) map println

Unmarshaller.stringUnmarshaller(someHttpEntity)
works like a charm, implicit materializer needed as well

Here is simple directive that extracts string from request's body
def withString(): Directive1[String] = {
extractStrictEntity(3.seconds).flatMap { entity =>
provide(entity.data.utf8String)
}
}

Unfortunately in my case, Unmarshal to String didn't work properly complaining on: Unsupported Content-Type, supported: application/json. That would be more elegant solution, but I had to use another way. In my test I used Future extracted from entity of the response and Await (from scala.concurrent) to get the result from the Future:
Put("/post/item", requestEntity) ~> route ~> check {
val responseContent: Future[Option[String]] =
response.entity.dataBytes.map(_.utf8String).runWith(Sink.lastOption)
val content: Option[String] = Await.result(responseContent, 10.seconds)
content.get should be(errorMessage)
response.status should be(StatusCodes.InternalServerError)
}
If you need to go through all lines in a response, you can use runForeach of Source:
response.entity.dataBytes.map(_.utf8String).runForeach(data => println(data))

Here is my working example,
import akka.actor.ActorSystem
import akka.http.scaladsl.Http
import akka.http.scaladsl.model._
import akka.stream.ActorMaterializer
import akka.util.ByteString
import scala.concurrent.Future
import scala.util.{ Failure, Success }
def getDataAkkaHTTP:Unit = {
implicit val system = ActorSystem()
implicit val materializer = ActorMaterializer()
// needed for the future flatMap/onComplete in the end
implicit val executionContext = system.dispatcher
val url = "http://localhost:8080/"
val responseFuture: Future[HttpResponse] = Http().singleRequest(HttpRequest(uri = url))
responseFuture.onComplete {
case Success(res) => {
val HttpResponse(statusCodes, headers, entity, _) = res
println(entity)
entity.dataBytes.runFold(ByteString(""))(_ ++ _).foreach (body => println(body.utf8String))
system.terminate()
}
case Failure(_) => sys.error("something wrong")
}
}

Related

akka stream integrating akka-htpp web request call into stream

Getting started with Akka Streams I want to perform a simple computation. Extending the basic QuickStart https://doc.akka.io/docs/akka/2.5/stream/stream-quickstart.html with a call to a restful web api:
val source: Source[Int, NotUsed] = Source(1 to 100)
source.runForeach(println)
already works nicely to print the numbers. But when trying to create an Actor to perform the HTTP request (is this actually necessary?) according to https://doc.akka.io/docs/akka/2.5.5/scala/stream/stream-integrations.html
import akka.pattern.ask
implicit val askTimeout = Timeout(5.seconds)
val words: Source[String, NotUsed] =
Source(List("hello", "hi"))
words
.mapAsync(parallelism = 5)(elem => (ref ? elem).mapTo[String])
// continue processing of the replies from the actor
.map(_.toLowerCase)
.runWith(Sink.ignore)
I cannot get it to compile as the ? operator is not defined. As ar as I know this one would only be defined inside an actor.
I also do not understand yet where exactly inside mapAsync my custom actor needs to be called.
edit
https://blog.colinbreck.com/backoff-and-retry-error-handling-for-akka-streams/ contains at least parts of an example.
It looks like it is not mandatory to create an actor i.e.
implicit val system = ActorSystem()
implicit val ec = system.dispatcher
implicit val materializer = ActorMaterializer()
val source = Source(List("232::03::14062::19965186", "232::03::14062::19965189"))
.map(cellKey => {
val splits = cellKey.split("::")
val mcc = splits(0)
val mnc = splits(1)
val lac = splits(2)
val ci = splits(3)
CellKeySource(cellKey, mcc, mnc, lac, ci)
})
.limit(2)
.mapAsyncUnordered(2)(ck => getResponse(ck.cellKey, ck.mobileCountryCode, ck.mobileNetworkCode, ck.locationArea, ck.cellKey)("<<myToken>>"))
def getResponse(cellKey: String, mobileCountryCode:String, mobileNetworkCode:String, locationArea:String, cellId:String)(token:String): Future[String] = {
RestartSource.withBackoff(
minBackoff = 10.milliseconds,
maxBackoff = 30.seconds,
randomFactor = 0.2,
maxRestarts = 2
) { () =>
val responseFuture: Future[HttpResponse] =
Http().singleRequest(HttpRequest(uri = s"https://www.googleapis.com/geolocation/v1/geolocate?key=${token}", entity = ByteString(
// TODO use proper JSON objects
s"""
|{
| "cellTowers": [
| "mobileCountryCode": $mobileCountryCode,
| "mobileNetworkCode": $mobileNetworkCode,
| "locationAreaCode": $locationArea,
| "cellId": $cellId,
| ]
|}
""".stripMargin)))
Source.fromFuture(responseFuture)
.mapAsync(parallelism = 1) {
case HttpResponse(StatusCodes.OK, _, entity, _) =>
Unmarshal(entity).to[String]
case HttpResponse(statusCode, _, _, _) =>
throw WebRequestException(statusCode.toString() )
}
}
.runWith(Sink.head)
.recover {
case _ => throw StreamFailedAfterMaxRetriesException()
}
}
val done: Future[Done] = source.runForeach(println)
done.onComplete(_ ⇒ system.terminate())
is already the (partial) answer for the question i.e. how to integrate Akka-streams + akka-http. However, it does not work, i.e. only throws error 400s and never terminates.
i think you already found an api how to call akka-http client
regarding your first code snippet which doesn't work. i think there happened some misunderstanding of the example itself. you expected the code in the example to work after just copied. but the intension of the doc was to demonstrate just an example/concept, how you can delegate some long running task out of the stream flow and then consuming the result when it's ready. for this was used ask call to akka actor, because call to ask method returns a Future. probably the authors of the doc just omitted the definition of actor. you can try this one example:
import java.lang.System.exit
import akka.NotUsed
import akka.actor.{Actor, ActorRef, ActorSystem, Props}
import akka.pattern.ask
import akka.stream.ActorMaterializer
import akka.stream.scaladsl.{Sink, Source}
import akka.util.Timeout
import scala.concurrent.ExecutionContext.Implicits.global
import scala.concurrent.duration._
import scala.language.higherKinds
object App extends scala.App {
implicit val sys: ActorSystem = ActorSystem()
implicit val mat: ActorMaterializer = ActorMaterializer()
val ref: ActorRef = sys.actorOf(Props[Translator])
implicit val askTimeout: Timeout = Timeout(5.seconds)
val words: Source[String, NotUsed] = Source(List("hello", "hi"))
words
.mapAsync(parallelism = 5)(elem => (ref ? elem).mapTo[String])
.map(_.toLowerCase)
.runWith(Sink.foreach(println))
.onComplete(t => {
println(s"finished: $t")
exit(1)
})
}
class Translator extends Actor {
override def receive: Receive = {
case msg => sender() ! s"$msg!"
}
}
You must import ask pattern from akka.
import akka.pattern.ask
Edit: OK, sorry, I can see that you have already imported. What is ref in your code? ActorRef?

Identify Akka HttpRequest and HttpResponse?

While using Akka HttpRequest and pipe the request to an actor, i couldn't identify the response.
The actor will handle each message that will receive but it doesn't know which request used to get this response. Is there any way to identify each request to match the response with ?
Note: i don't have the server to resend any part of request body again.
Thanks in advance
MySelf.scala
import akka.actor.{ Actor, ActorLogging }
import akka.http.scaladsl.Http
import akka.http.scaladsl.model._
import akka.stream.{ ActorMaterializer, ActorMaterializerSettings }
import akka.util.ByteString
class Myself extends Actor with ActorLogging {
import akka.pattern.pipe
import context.dispatcher
final implicit val materializer: ActorMaterializer =
ActorMaterializer(ActorMaterializerSettings(context.system))
def receive = {
case HttpResponse(StatusCodes.OK, headers, entity, _) =>
entity.dataBytes.runFold(ByteString(""))(_ ++ _).foreach { body =>
log.info("Got response, body: " + body.utf8String)
}
case resp # HttpResponse(code, _, _, _) =>
log.info("Request failed, response code: " + code)
resp.discardEntityBytes()
}
}
Main.scala
import akka.actor.{ActorSystem, Props}
import akka.http.scaladsl.Http
import akka.http.scaladsl.model._
import akka.stream.ActorMaterializer
object HttpServerMain extends App {
import akka.pattern.pipe
// import system.dispatcher
implicit val system = ActorSystem()
implicit val materializer = ActorMaterializer()
// needed for the future flatMap/onComplete in the end
implicit val executionContext = system.dispatcher
val http = Http(system)
val myActor = system.actorOf(Props[MySelf])
http.singleRequest(HttpRequest(uri = "http://akka.io"))
.pipeTo(myActor)
http.singleRequest(HttpRequest(uri = "http://akka.io/another-request"))
.pipeTo(myActor)
Thread.sleep(2000)
system.terminate()
You can simply use map to transform the Future and add some kind of ID (usually called correlation ID for such purposes) to it before you pipe it to myActor:
http.singleRequest(HttpRequest(uri = "http://akka.io"))
.map(x => (1, x)).pipeTo(myActor)
You'll need to change you pattern match blocks to take a tupple:
case (id, HttpResponse(StatusCodes.OK, headers, entity, _)) =>
If you can't/don't want to change your pattern match block for some reason you can use same approach, but instead add a unique HTTP header into your completed request (using copy) with something like this (not checked if compiles):
// make a unique header name that you are sure will not be
// received from http response:
val correlationHeader: HttpHeader = ... // mycustomheader
// Basically hack the response to add your header:
http.singleRequest(HttpRequest(uri = "http://akka.io"))
.map(x => x.copy(headers = correlationHeader +: headers)).pipeTo(myActor)
// Now you can check your header to see which response that was:
case HttpResponse(StatusCodes.OK, headers, entity, _) =>
headers.find(_.is("mycustomheader")).map(_.value).getOrElse("NA")
This is more of a hack though compared to previous option because you are modifying a response.
I think you cannot do that directly using pipeTo because it essentially just adds andThen call to your Future. One option is tomap and then send a (request, response) tuple to actor:
val request = HttpRequest(uri = "http://akka.io")
http.singleRequest(request).map {
response => myActor ! (request, response)
}
class Myself extends Actor with ActorLogging {
...
def receive = {
case (request, HttpResponse(StatusCodes.OK, headers, entity, _)) =>
...
case (request, resp # HttpResponse(code, _, _, _)) =>
log.info(request.toString)
...
}
}

Route akka-http request through a proxy

I am rewriting some application layer code in scala from using scalaj to akka-http
in order to reduce the number of third party dependencies in the project (we already use akka for other things in the same project.) The code simply wraps common types of request to an underlying general request provided by the library
Mostly it has been fine, but I am stuck on the problem of optionally adding a proxy to a request.
Requests should either be direct to the destination or via a proxy, determined by a parameter at runtime.
In my scalaj implementation, I have the following helper class and methods
object HttpUtils {
private def request(
host: Host,
method: HttpMethod,
params: Map[String, String],
postData: Option[String],
timeout: Duration,
headers: Seq[(String, String)],
proxy: Option[ProxyConfig]
): HttpResponse[String] = {
// most general request builder. Other methods in the object fill in parameters and wrap this in a Future
val baseRequest = Http(host.url)
val proxiedRequest = addProxy(proxy, baseRequest)
val fullRequest = addPostData(postData)(proxiedRequest)
.method(method.toString)
.params(params)
.headers(headers)
.option(HttpOptions.connTimeout(timeout.toMillis.toInt))
.option(HttpOptions.readTimeout(timeout.toMillis.toInt))
fullRequest.asString // scalaj for send off request and block until response
}
// Other methods ...
private def addProxy(proxy: Option[ProxyConfig], request: HttpRequest): HttpRequest =
proxy.fold(request)((p: ProxyConfig) => request.proxy(p.host, p.port))
}
case class ProxyConfig(host: String, port: Int)
Is there a way to build a similar construct with akka-http?
Akka HTTP does have proxy support that, as of version 10.0.9, is still unstable. Keeping in mind that the API could change, you could do something like the following to handle optional proxy settings:
import java.net.InetSocketAddress
import akka.actor.ActorSystem
import akka.stream.ActorMaterializer
import akka.http.scaladsl.{ClientTransport, Http}
implicit val system = ActorSystem()
implicit val materializer = ActorMaterializer()
case class ProxyConfig(host: String, port: Int)
val proxyConfig = Option(ProxyConfig("localhost", 8888))
val clientTransport =
proxyConfig.map(p => ClientTransport.httpsProxy(InetSocketAddress.createUnresolved(p.host, p.port)))
.getOrElse(ClientTransport.TCP)
val settings = ConnectionPoolSettings(system).withTransport(clientTransport)
Http().singleRequest(HttpRequest(uri = "https://google.com"), settings = settings)
In Akka Http 10.2.0, use bindflow for a Flow[HttpRequest, HttpResponse, NotUsed] defined by a RunnableGraph with Flowshape. Insided the RunnableGraph, an Http() outgoingConnection is used to connect to the remote proxy. Some example code:
import akka.actor.typed.ActorSystem
import akka.actor.typed.scaladsl.Behaviors
import akka.http.scaladsl.Http
import akka.http.scaladsl.model.{HttpRequest, HttpResponse}
import akka.stream._
import akka.stream.scaladsl.{Broadcast, Flow, GraphDSL, Merge}
import scala.concurrent.ExecutionContextExecutor
import scala.concurrent.duration.DurationInt
import scala.io.StdIn
import scala.util.{Failure, Success}
object Main {
def main(args: Array[String]) {
implicit val system: ActorSystem[Nothing] = ActorSystem(Behaviors.empty, "testproxy")
implicit val executionContext: ExecutionContextExecutor = system.executionContext
system.log.info("TestAkkaHttpProxy Main started...")
val remoteHost = "xxx.xxx.xxx.x"
val remotePort = 8000
val proxyHost = "0.0.0.0"
val proxyPort = 8080
val gateway = Flow.fromGraph(GraphDSL.create() { implicit b =>
import GraphDSL.Implicits._
// Broadcast for flow input
val broadcaster = b.add(Broadcast[HttpRequest](1))
// Merge for flow output
val responseMerge = b.add(Merge[HttpResponse](1))
// outgoing client for remote proxy
val remote = Http().outgoingConnection(remoteHost, remotePort)
// filter out header that creates Akka Http warning
val requestConvert = Flow[HttpRequest]
.map(req => { req.mapHeaders(headers => headers.filter(h => h.isNot("timeout-access")))
})
// connect graph
broadcaster.out(0) ~> requestConvert ~> remote ~> responseMerge
// expose ports
FlowShape(broadcaster.in, responseMerge.out)
})
// Akka Http server that binds to Flow (for remote proxy)
Http().newServerAt(proxyHost, proxyPort).bindFlow(gateway)
.onComplete({
case Success(binding) ⇒
println(s"Server is listening on 0.0.0.0:8080")
binding.addToCoordinatedShutdown(hardTerminationDeadline = 10.seconds)
case Failure(e) ⇒
println(s"Binding failed with ${e.getMessage}")
system.terminate()
})
system.log.info("Press RETURN to stop...")
StdIn.readLine()
system.terminate()
}
}

Akka Http Client Set Cookie on a HttpRequest

I am trying to make a GET request to a REST web service using Akka Http Client.
I am not able to figure out how do I set a cookie on the request before I make the GET.
I searched the web and I found ways to read the cookie on the server side. but I could not find anything which showed me how to set the cookie on the client side request.
Based on my own research I tried the following approach to set a cookie on http request
import akka.actor.ActorSystem
import akka.http.scaladsl.Http
import akka.http.scaladsl.model._
import akka.http.scaladsl.unmarshalling.Unmarshal
import akka.stream.scaladsl.{Sink, Source}
import akka.http.scaladsl.marshallers.sprayjson.SprayJsonSupport
import akka.http.scaladsl.model.headers.HttpCookie
import akka.stream.ActorMaterializer
import spray.json._
import scala.util.{Failure, Success}
case class Post(postId: Int, id: Int, name: String, email: String, body: String)
trait JsonSupport extends SprayJsonSupport with DefaultJsonProtocol {
implicit val postFormat = jsonFormat5(Post.apply)
}
object AkkaHttpClient extends JsonSupport{
def main(args: Array[String]) : Unit = {
val cookie = headers.`Set-Cookie`(HttpCookie(name="foo", value="bar"))
implicit val system = ActorSystem("my-Actor")
implicit val actorMaterializer = ActorMaterializer()
implicit val executionContext = system.dispatcher
val mycookie = HttpCookie(name="foo", value="bar")
val httpClient = Http().outgoingConnection(host = "jsonplaceholder.typicode.com")
val request = HttpRequest(uri = Uri("/comments"), headers = List(cookie))
val flow = Source.single(request)
.via(httpClient)
.mapAsync(1)(r => Unmarshal(r.entity).to[List[Post]])
.runWith(Sink.head)
flow.andThen {
case Success(list) => println(s"request succeded ${list.size}")
case Failure(_) => println("request failed")
}.andThen {
case _ => system.terminate()
}
}
}
But this gives an error
[WARN] [08/05/2016 10:50:11.134] [my-Actor-akka.actor.default-dispatcher-3] [akka.actor.ActorSystemImpl(my-Actor)]
HTTP header 'Set-Cookie: foo=bar' is not allowed in requests
The idiomatic way to construct any header for an akka-http client is by
using akka.http.scaladsl.model.headers.
In your case it would be
val cookieHeader = akka.http.scaladsl.model.headers.Cookie("name","value")
HttpRequest(uri = Uri("/comments"), headers = List(cookieHeader, ...))
The outgoing header must be 'Cookie' not 'Set-Cookie':
val cookie = HttpCookiePair("foo", "bar")
val headers: immutable.Seq[HttpHeader] = if (cookies.isEmpty) immutable.Seq.empty else immutable.Seq(Cookie(cookies))
val request = HttpRequest(uri = uri).withHeadersAndEntity(headers, HttpEntity(msg))

How to perform a simple json post with spray-json in spray?

I'm trying to perform a simple json post with spray. But it seems that i can get an http entity for a json object that can be Marshall.
here is my error:
[error]
...../IdeaProjects/PoolpartyConnector/src/main/scala/org/iadb/poolpartyconnector/thesaurusoperation/ThesaurusCacheService.scala:172:
could not find implicit value for evidence parameter of type
spray.httpx.marshalling.Marshaller[spray.json.JsValue]
[error] val request =
Post(s"$thesaurusapiEndpoint/$coreProjectId/suggestFreeConcept?",
suggestionJsonBody)
and the code that comes with it:
override def createSuggestedFreeConcept(suggestedPrefLabel: String, lang: String, scheme: String, b: Boolean): String = {
import system.dispatcher
import spray.json._
val pipeline = addCredentials(BasicHttpCredentials("superadmin", "poolparty")) ~> sendReceive
val label = LanguageLiteral(suggestedPrefLabel, lang)
val suggestion = SuggestFreeConcept(List(label), b, Some(List(scheme)), None, None,None, None)
val suggestionJsonBody = suggestion.toJson
val request = Post(s"$thesaurusapiEndpoint/$coreProjectId/suggestFreeConcept?", suggestionJsonBody)
val res = pipeline(request)
getSuggestedFromFutureHttpResponse(res) match {
case None => ""
case Some(e) => e
}
}
Please, does any one has an idea of what is going on with the implicit marshaller. I though spray Json would come with implicit marshaller.
I assume you already have a custom Json Protocol somewhere so that suggestion.toJson works correctly?
Try the following:
val body = HttpEntity(`application/json`, suggestionJsonBody.prettyPrint)
val request = Post(s"$thesaurusapiEndpoint/$coreProjectId/suggestFreeConcept?", body)
you could also use compactPrint rather than prettyPrint, in either case, it turns the Json into a string containing the json information.
Here is how i solved it:
override def createSuggestedFreeConcepts(suggestedPrefLabels: List[LanguageLiteral], scheme: String, checkDuplicates: Boolean): List[String] = {
import system.dispatcher
import spray.httpx.marshalling._
import spray.httpx.SprayJsonSupport._
val pipeline = addCredentials(BasicHttpCredentials("superadmin", "poolparty")) ~> sendReceive
suggestedPrefLabels map { suggestedPrefLabel =>
val suggestion = SuggestFreeConcept(List(suggestedPrefLabel), checkDuplicates, Some(List(Uri(scheme))), None, None, None, None)
val request = Post(s"$thesaurusapiEndpoint/$coreProjectId/suggestFreeConcept", marshal(suggestion))
val res = pipeline(request)
getSuggestedFromFutureHttpResponse(res) match {
case None => ""
case Some(e) => e
}
}
}
the key is:
import spray.httpx.marshalling._ import spray.httpx.SprayJsonSupport._
and
val request =
Post(s"$thesaurusapiEndpoint/$coreProjectId/suggestFreeConcept",
marshal(suggestion))
I marshall suggestion. The explanation is not super super straightforward. But by fetching around in the doc, it is explained.