I am very new to Akka HTTP so please accept my apologies beforehand for the very elementary question.
In the following piece of code, I want to retrieve the entity from the HTTP request (entity will be plain text), get the text out of the entity, and return it as a response.
implicit val system = ActorSystem("ActorSystem")
implicit val materializer = ActorMaterializer
import system.dispatcher
val requestHandler: Flow[HttpRequest, HttpResponse, _] = Flow[HttpRequest].map {
case HttpRequest(HttpMethods.POST, Uri.Path("/api"), _, entity, _) =>
val entityAsText = ... // <- get entity content as text
HttpResponse(
StatusCodes.OK,
entity = HttpEntity(
ContentTypes.`text/plain(UTF-8)`,
entityAsText
)
)
}
Http().bindAndHandle(requestHandler, "localhost", 8080)
How can I get the string content of the entity?
Many thanks in advance!
One approach is to call toStrict on the RequestEntity, which loads the entity into memory, and mapAsync on the Flow:
import scala.concurrent.duration._
val requestHandler: Flow[HttpRequest, HttpResponse, _] = Flow[HttpRequest].mapAsync(1) {
case HttpRequest(HttpMethods.GET, Uri.Path("/api"), _, entity, _) =>
val entityAsText: Future[String] = entity.toStrict(1 second).map(_.data.utf8String)
entityAsText.map { text =>
HttpResponse(
StatusCodes.OK,
entity = HttpEntity(
ContentTypes.`text/plain(UTF-8)`,
text
)
)
}
}
Adjust the timeout on the former and the parallelism level on the latter as needed.
Another approach would be to use the already Unmarshaller which is in scope (most likely it will be the akka.http.scaladsl.unmarshalling.PredefinedFromEntityUnmarshallers#stringUnmarshaller):
val entityAsText: Future[String] = Unmarshal(entity).to[String]
This approach ensures the consistent usage of the provided Unmarshaller for String and you wouldn't have to cope with timeouts.
Related
I am new to learning Scala, Akka Streams and Akka HTTP, so apologies beforehand if the question is too basic.
I want to do an HTTP request inside an HTTP request, just like in the following piece of code:
implicit val system = ActorSystem("ActorSystem")
implicit val materializer = ActorMaterializer
import system.dispatcher
val requestHandler: Flow[HttpRequest, HttpResponse, _] = Flow[HttpRequest].map {
case HttpRequest(HttpMethods.GET, Uri.Path("/api"), _, _, _) =>
val responseFuture = Http().singleRequest(HttpRequest(uri = "http://www.google.com"))
responseFuture.onComplete {
case Success(response) =>
response.discardEntityBytes()
println(s"The request was successful")
case Failure(ex) =>
println(s"The request failed with: $ex")
}
//Await.result(responseFuture, 10 seconds)
println("Reached HttpResponse")
HttpResponse(
StatusCodes.OK
)
}
Http().bindAndHandle(requestHandler, "localhost", 8080)
But in the above case the result looks like this, meaning that Reached HttpResponse is reached first before completing the request:
Reached HttpResponse
The request was successful
I tried using Await.result(responseFuture, 10 seconds) (currently commented out) but it made no difference.
What am I missing here? Any help will be greatly appreciated!
Many thanks in advance!
map is a function that takes request and produces a response:
HttpRequest => HttpResponse
The challenge is that response is a type of Future. Therefore, you need a function that deals with it. The function that takes HttpRequest and returns Future of HttpResponse.
HttpRequest => Future[HttpResponse]
And voila, mapAsync is exactly what you need:
val requestHandler: Flow[HttpRequest, HttpResponse, _] = Flow[HttpRequest].mapAsync(2) {
case HttpRequest(HttpMethods.GET, Uri.Path("/api"), _, _, _) =>
Http().singleRequest(HttpRequest(uri = "http://www.google.com")).map (resp => {
resp.discardEntityBytes()
println(s"The request was successful")
HttpResponse(StatusCodes.OK)
})
}
I'am trying to implement Akka http client for post request where Authorization will be given in header from postman. I am not able to authorize, following is my code
def main(args: Array[String]) {
implicit val system: ActorSystem = ActorSystem()
implicit val materializer: ActorMaterializer = ActorMaterializer()
implicit val executionContext: ExecutionContextExecutor = system.dispatcher
val requestHeader = scala.collection.immutable.Seq(RawHeader("Authorization", "admin"))
val requestHandler: HttpRequest => HttpResponse = {
case HttpRequest(POST, Uri.Path("/GetTrackerData"), requestHeader, entity, _) =>
val chunk = Unmarshal(entity).to[DeviceLocationData]
val deviceLocationData = Await.result(chunk, 1.second)
val responseArray = "Authorized"
HttpResponse(entity = HttpEntity(ContentTypes.`application/json`, responseArray)
)
case r: HttpRequest =>
println(r.uri.toString())
r.discardEntityBytes() // important to drain incoming HTTP Entity stream
HttpResponse(404, entity = "Unknown resource!")
}
val bindingFuture = Http().bindAndHandleSync(requestHandler, "0.0.0.0", 7070)
println(s"iot engine api live at 0.0.0.0:7070")
sys.addShutdownHook({
println("ShutdownHook called")
bindingFuture
.flatMap(_.unbind()) // trigger unbinding from the port
.onComplete(_ => system.terminate()) // and shutdown when done
})
}
Whatever value I give from postman. It serves the request. What I am skipping ?
My use case is that result showed be displayed only after authorization
You are pattern matching on HttpRequest.
The requestHeader you use there is not the one you specified earlier but will be the headers from the HttpRequest itself.
One way to resolve it could be checking for the values in the headers:
case HttpRequest(HttpMethods.POST, Uri.Path("/GetTrackerData"), headers, entity, _)
if (headers.exists(h => h.name == "Authorization" && h.value == "admin")) =>
Here is one problem:
case HttpRequest(POST, Uri.Path("/GetTrackerData"), requestHeader, entity, _) =>
This does not match the current value of requestHeader, it creates a new value requestHeader containing the current headers of the HttpRequest. So this match will not check the Authorization field of the header. You will have to do this manually.
More generally, it would be worth looking at the Route support in Akka HTTP as a cleaner and more powerful way of implementing a server.
Getting started with Akka Streams I want to perform a simple computation. Extending the basic QuickStart https://doc.akka.io/docs/akka/2.5/stream/stream-quickstart.html with a call to a restful web api:
val source: Source[Int, NotUsed] = Source(1 to 100)
source.runForeach(println)
already works nicely to print the numbers. But when trying to create an Actor to perform the HTTP request (is this actually necessary?) according to https://doc.akka.io/docs/akka/2.5.5/scala/stream/stream-integrations.html
import akka.pattern.ask
implicit val askTimeout = Timeout(5.seconds)
val words: Source[String, NotUsed] =
Source(List("hello", "hi"))
words
.mapAsync(parallelism = 5)(elem => (ref ? elem).mapTo[String])
// continue processing of the replies from the actor
.map(_.toLowerCase)
.runWith(Sink.ignore)
I cannot get it to compile as the ? operator is not defined. As ar as I know this one would only be defined inside an actor.
I also do not understand yet where exactly inside mapAsync my custom actor needs to be called.
edit
https://blog.colinbreck.com/backoff-and-retry-error-handling-for-akka-streams/ contains at least parts of an example.
It looks like it is not mandatory to create an actor i.e.
implicit val system = ActorSystem()
implicit val ec = system.dispatcher
implicit val materializer = ActorMaterializer()
val source = Source(List("232::03::14062::19965186", "232::03::14062::19965189"))
.map(cellKey => {
val splits = cellKey.split("::")
val mcc = splits(0)
val mnc = splits(1)
val lac = splits(2)
val ci = splits(3)
CellKeySource(cellKey, mcc, mnc, lac, ci)
})
.limit(2)
.mapAsyncUnordered(2)(ck => getResponse(ck.cellKey, ck.mobileCountryCode, ck.mobileNetworkCode, ck.locationArea, ck.cellKey)("<<myToken>>"))
def getResponse(cellKey: String, mobileCountryCode:String, mobileNetworkCode:String, locationArea:String, cellId:String)(token:String): Future[String] = {
RestartSource.withBackoff(
minBackoff = 10.milliseconds,
maxBackoff = 30.seconds,
randomFactor = 0.2,
maxRestarts = 2
) { () =>
val responseFuture: Future[HttpResponse] =
Http().singleRequest(HttpRequest(uri = s"https://www.googleapis.com/geolocation/v1/geolocate?key=${token}", entity = ByteString(
// TODO use proper JSON objects
s"""
|{
| "cellTowers": [
| "mobileCountryCode": $mobileCountryCode,
| "mobileNetworkCode": $mobileNetworkCode,
| "locationAreaCode": $locationArea,
| "cellId": $cellId,
| ]
|}
""".stripMargin)))
Source.fromFuture(responseFuture)
.mapAsync(parallelism = 1) {
case HttpResponse(StatusCodes.OK, _, entity, _) =>
Unmarshal(entity).to[String]
case HttpResponse(statusCode, _, _, _) =>
throw WebRequestException(statusCode.toString() )
}
}
.runWith(Sink.head)
.recover {
case _ => throw StreamFailedAfterMaxRetriesException()
}
}
val done: Future[Done] = source.runForeach(println)
done.onComplete(_ ⇒ system.terminate())
is already the (partial) answer for the question i.e. how to integrate Akka-streams + akka-http. However, it does not work, i.e. only throws error 400s and never terminates.
i think you already found an api how to call akka-http client
regarding your first code snippet which doesn't work. i think there happened some misunderstanding of the example itself. you expected the code in the example to work after just copied. but the intension of the doc was to demonstrate just an example/concept, how you can delegate some long running task out of the stream flow and then consuming the result when it's ready. for this was used ask call to akka actor, because call to ask method returns a Future. probably the authors of the doc just omitted the definition of actor. you can try this one example:
import java.lang.System.exit
import akka.NotUsed
import akka.actor.{Actor, ActorRef, ActorSystem, Props}
import akka.pattern.ask
import akka.stream.ActorMaterializer
import akka.stream.scaladsl.{Sink, Source}
import akka.util.Timeout
import scala.concurrent.ExecutionContext.Implicits.global
import scala.concurrent.duration._
import scala.language.higherKinds
object App extends scala.App {
implicit val sys: ActorSystem = ActorSystem()
implicit val mat: ActorMaterializer = ActorMaterializer()
val ref: ActorRef = sys.actorOf(Props[Translator])
implicit val askTimeout: Timeout = Timeout(5.seconds)
val words: Source[String, NotUsed] = Source(List("hello", "hi"))
words
.mapAsync(parallelism = 5)(elem => (ref ? elem).mapTo[String])
.map(_.toLowerCase)
.runWith(Sink.foreach(println))
.onComplete(t => {
println(s"finished: $t")
exit(1)
})
}
class Translator extends Actor {
override def receive: Receive = {
case msg => sender() ! s"$msg!"
}
}
You must import ask pattern from akka.
import akka.pattern.ask
Edit: OK, sorry, I can see that you have already imported. What is ref in your code? ActorRef?
I have the following problem.
I am querying a server for some data and getting it back as HttpEntity.Chunked.
The response String looks like this with up to 10.000.000 lines like this:
[{"name":"param1","value":122343,"time":45435345},
{"name":"param2","value":243,"time":4325435},
......]
Now I want to get the incoming data into and Array[String] where each String is a line from the response, because later on it should be imported into an apache spark dataframe.
Currently I am doing it likes this:
//For the http request
trait StartHttpRequest {
implicit val system: ActorSystem
implicit val materializer: ActorMaterializer
def httpRequest(data: String, path: String, targetPort: Int, host: String): Future[HttpResponse] = {
val connectionFlow: Flow[HttpRequest, HttpResponse, Future[OutgoingConnection]] = {
Http().outgoingConnection(host, port = targetPort)
}
val responseFuture: Future[HttpResponse] =
Source.single(RequestBuilding.Post(uri = path, entity = HttpEntity(ContentTypes.`application/json`, data)))
.via(connectionFlow)
.runWith(Sink.head)
responseFuture
}
}
//result of the request
val responseFuture: Future[HttpResponse] = httpRequest(.....)
//convert to string
responseFuture.flatMap { response =>
response.status match {
case StatusCodes.OK =>
Unmarshal(response.entity).to[String]
}
}
//and then something like this, but with even more stupid stuff
responseFuture.onSuccess { str:String =>
masterActor! str.split("""\},\{""")
}
My question is, what would be a better way to get the result into an array?
How can I unmarshall the response entity directly? Because .to[Array[String]] for example did not work. And because there are so many lines coming, could I do it with a stream, to be more efficent?
Answering your questions out of order:
How can I unmarshall the response entity directly?
There is an existing question & answer related to unmarshalling an Array of case classes.
what would be a better way to get the result into an array?
I would take advantage of the Chunked nature and use streams. This allows you to do string processing and json parsing concurrently.
First you need a container class and parser:
case class Data(name : String, value : Int, time : Long)
object MyJsonProtocol extends DefaultJsonProtocol {
implicit val dataFormat = jsonFormat3(Data)
}
Then you have to do some manipulations to get the json objects to look right:
//Drops the '[' and the ']' characters
val dropArrayMarkers =
Flow[ByteString].map(_.filterNot(b => b == '['.toByte || b == ']'.toByte))
val preppendBrace =
Flow[String].map(s => if(!s.startsWith("{")) "{" + s else s)
val appendBrace =
Flow[String].map(s => if(!s.endsWith("}")) s + "}" else s)
val parseJson =
Flow[String].map(_.parseJson.convertTo[Data])
Finally, combine these Flows to convert a Source of ByteString into a Source of Data objects:
def strSourceToDataSource(source : Source[ByteString,_]) : Source[Data, _] =
source.via(dropArrayMarkers)
.via(Framing.delimiter(ByteString("},{"), 256, true))
.map(_.utf8String)
.via(prependBrace)
.via(appendBrace)
.via(parseJson)
This source can then be drained into an Seq of Data objects:
val dataSeq : Future[Seq[Data]] =
responseFuture flatMap { response =>
response.status match {
case StatusCodes.OK =>
strSourceToDataSource(response.entity.dataBytes).runWith(Sink.seq)
}
}
So I have a function with this signature (akka.http.model.HttpResponse):
def apply(query: Seq[(String, String)], accept: String): HttpResponse
I simply get a value in a test like:
val resp = TagAPI(Seq.empty[(String, String)], api.acceptHeader)
I want to check its body in a test something like:
resp.entity.asString == "tags"
My question is how I can get the response body as string?
import akka.http.scaladsl.unmarshalling.Unmarshal
implicit val system = ActorSystem("System")
implicit val materializer = ActorFlowMaterializer()
val responseAsString: Future[String] = Unmarshal(entity).to[String]
Since Akka Http is streams based, the entity is streaming as well. If you really need the entire string at once, you can convert the incoming request into a Strict one:
This is done by using the toStrict(timeout: FiniteDuration)(mat: Materializer) API to collect the request into a strict entity within a given time limit (this is important since you don't want to "try to collect the entity forever" in case the incoming request does actually never end):
import akka.stream.ActorFlowMaterializer
import akka.actor.ActorSystem
implicit val system = ActorSystem("Sys") // your actor system, only 1 per app
implicit val materializer = ActorFlowMaterializer() // you must provide a materializer
import system.dispatcher
import scala.concurrent.duration._
val timeout = 300.millis
val bs: Future[ByteString] = entity.toStrict(timeout).map { _.data }
val s: Future[String] = bs.map(_.utf8String) // if you indeed need a `String`
You can also try this one also.
responseObject.entity.dataBytes.runFold(ByteString(""))(_ ++ _).map(_.utf8String) map println
Unmarshaller.stringUnmarshaller(someHttpEntity)
works like a charm, implicit materializer needed as well
Here is simple directive that extracts string from request's body
def withString(): Directive1[String] = {
extractStrictEntity(3.seconds).flatMap { entity =>
provide(entity.data.utf8String)
}
}
Unfortunately in my case, Unmarshal to String didn't work properly complaining on: Unsupported Content-Type, supported: application/json. That would be more elegant solution, but I had to use another way. In my test I used Future extracted from entity of the response and Await (from scala.concurrent) to get the result from the Future:
Put("/post/item", requestEntity) ~> route ~> check {
val responseContent: Future[Option[String]] =
response.entity.dataBytes.map(_.utf8String).runWith(Sink.lastOption)
val content: Option[String] = Await.result(responseContent, 10.seconds)
content.get should be(errorMessage)
response.status should be(StatusCodes.InternalServerError)
}
If you need to go through all lines in a response, you can use runForeach of Source:
response.entity.dataBytes.map(_.utf8String).runForeach(data => println(data))
Here is my working example,
import akka.actor.ActorSystem
import akka.http.scaladsl.Http
import akka.http.scaladsl.model._
import akka.stream.ActorMaterializer
import akka.util.ByteString
import scala.concurrent.Future
import scala.util.{ Failure, Success }
def getDataAkkaHTTP:Unit = {
implicit val system = ActorSystem()
implicit val materializer = ActorMaterializer()
// needed for the future flatMap/onComplete in the end
implicit val executionContext = system.dispatcher
val url = "http://localhost:8080/"
val responseFuture: Future[HttpResponse] = Http().singleRequest(HttpRequest(uri = url))
responseFuture.onComplete {
case Success(res) => {
val HttpResponse(statusCodes, headers, entity, _) = res
println(entity)
entity.dataBytes.runFold(ByteString(""))(_ ++ _).foreach (body => println(body.utf8String))
system.terminate()
}
case Failure(_) => sys.error("something wrong")
}
}