Scala and Akka HTTP: Request inside a request & issue with threads - scala

I am new to learning Scala, Akka Streams and Akka HTTP, so apologies beforehand if the question is too basic.
I want to do an HTTP request inside an HTTP request, just like in the following piece of code:
implicit val system = ActorSystem("ActorSystem")
implicit val materializer = ActorMaterializer
import system.dispatcher
val requestHandler: Flow[HttpRequest, HttpResponse, _] = Flow[HttpRequest].map {
case HttpRequest(HttpMethods.GET, Uri.Path("/api"), _, _, _) =>
val responseFuture = Http().singleRequest(HttpRequest(uri = "http://www.google.com"))
responseFuture.onComplete {
case Success(response) =>
response.discardEntityBytes()
println(s"The request was successful")
case Failure(ex) =>
println(s"The request failed with: $ex")
}
//Await.result(responseFuture, 10 seconds)
println("Reached HttpResponse")
HttpResponse(
StatusCodes.OK
)
}
Http().bindAndHandle(requestHandler, "localhost", 8080)
But in the above case the result looks like this, meaning that Reached HttpResponse is reached first before completing the request:
Reached HttpResponse
The request was successful
I tried using Await.result(responseFuture, 10 seconds) (currently commented out) but it made no difference.
What am I missing here? Any help will be greatly appreciated!
Many thanks in advance!

map is a function that takes request and produces a response:
HttpRequest => HttpResponse
The challenge is that response is a type of Future. Therefore, you need a function that deals with it. The function that takes HttpRequest and returns Future of HttpResponse.
HttpRequest => Future[HttpResponse]
And voila, mapAsync is exactly what you need:
val requestHandler: Flow[HttpRequest, HttpResponse, _] = Flow[HttpRequest].mapAsync(2) {
case HttpRequest(HttpMethods.GET, Uri.Path("/api"), _, _, _) =>
Http().singleRequest(HttpRequest(uri = "http://www.google.com")).map (resp => {
resp.discardEntityBytes()
println(s"The request was successful")
HttpResponse(StatusCodes.OK)
})
}

Related

Scala and Akka HTTP: How to get the entity content as string

I am very new to Akka HTTP so please accept my apologies beforehand for the very elementary question.
In the following piece of code, I want to retrieve the entity from the HTTP request (entity will be plain text), get the text out of the entity, and return it as a response.
implicit val system = ActorSystem("ActorSystem")
implicit val materializer = ActorMaterializer
import system.dispatcher
val requestHandler: Flow[HttpRequest, HttpResponse, _] = Flow[HttpRequest].map {
case HttpRequest(HttpMethods.POST, Uri.Path("/api"), _, entity, _) =>
val entityAsText = ... // <- get entity content as text
HttpResponse(
StatusCodes.OK,
entity = HttpEntity(
ContentTypes.`text/plain(UTF-8)`,
entityAsText
)
)
}
Http().bindAndHandle(requestHandler, "localhost", 8080)
How can I get the string content of the entity?
Many thanks in advance!
One approach is to call toStrict on the RequestEntity, which loads the entity into memory, and mapAsync on the Flow:
import scala.concurrent.duration._
val requestHandler: Flow[HttpRequest, HttpResponse, _] = Flow[HttpRequest].mapAsync(1) {
case HttpRequest(HttpMethods.GET, Uri.Path("/api"), _, entity, _) =>
val entityAsText: Future[String] = entity.toStrict(1 second).map(_.data.utf8String)
entityAsText.map { text =>
HttpResponse(
StatusCodes.OK,
entity = HttpEntity(
ContentTypes.`text/plain(UTF-8)`,
text
)
)
}
}
Adjust the timeout on the former and the parallelism level on the latter as needed.
Another approach would be to use the already Unmarshaller which is in scope (most likely it will be the akka.http.scaladsl.unmarshalling.PredefinedFromEntityUnmarshallers#stringUnmarshaller):
val entityAsText: Future[String] = Unmarshal(entity).to[String]
This approach ensures the consistent usage of the provided Unmarshaller for String and you wouldn't have to cope with timeouts.

How to implement Akka HTTP Request Post Client Headers Authorization

I'am trying to implement Akka http client for post request where Authorization will be given in header from postman. I am not able to authorize, following is my code
def main(args: Array[String]) {
implicit val system: ActorSystem = ActorSystem()
implicit val materializer: ActorMaterializer = ActorMaterializer()
implicit val executionContext: ExecutionContextExecutor = system.dispatcher
val requestHeader = scala.collection.immutable.Seq(RawHeader("Authorization", "admin"))
val requestHandler: HttpRequest => HttpResponse = {
case HttpRequest(POST, Uri.Path("/GetTrackerData"), requestHeader, entity, _) =>
val chunk = Unmarshal(entity).to[DeviceLocationData]
val deviceLocationData = Await.result(chunk, 1.second)
val responseArray = "Authorized"
HttpResponse(entity = HttpEntity(ContentTypes.`application/json`, responseArray)
)
case r: HttpRequest =>
println(r.uri.toString())
r.discardEntityBytes() // important to drain incoming HTTP Entity stream
HttpResponse(404, entity = "Unknown resource!")
}
val bindingFuture = Http().bindAndHandleSync(requestHandler, "0.0.0.0", 7070)
println(s"iot engine api live at 0.0.0.0:7070")
sys.addShutdownHook({
println("ShutdownHook called")
bindingFuture
.flatMap(_.unbind()) // trigger unbinding from the port
.onComplete(_ => system.terminate()) // and shutdown when done
})
}
Whatever value I give from postman. It serves the request. What I am skipping ?
My use case is that result showed be displayed only after authorization
You are pattern matching on HttpRequest.
The requestHeader you use there is not the one you specified earlier but will be the headers from the HttpRequest itself.
One way to resolve it could be checking for the values in the headers:
case HttpRequest(HttpMethods.POST, Uri.Path("/GetTrackerData"), headers, entity, _)
if (headers.exists(h => h.name == "Authorization" && h.value == "admin")) =>
Here is one problem:
case HttpRequest(POST, Uri.Path("/GetTrackerData"), requestHeader, entity, _) =>
This does not match the current value of requestHeader, it creates a new value requestHeader containing the current headers of the HttpRequest. So this match will not check the Authorization field of the header. You will have to do this manually.
More generally, it would be worth looking at the Route support in Akka HTTP as a cleaner and more powerful way of implementing a server.

Akka streams Source.repeat stops after 100 requests

I am working on the below stream processing system to grab frames from one source, process, and send to another. I'm using a combination of akka-streams and akka-http through their scapa api. The pipeline is very short but I can't seem to locate where the system decides to stop after precisely 100 requests to the endpoint.
object frameProcessor extends App {
implicit val system: ActorSystem = ActorSystem("VideoStreamProcessor")
val decider: Supervision.Decider = _ => Supervision.Restart
implicit val materializer: ActorMaterializer = ActorMaterializer()
implicit val dispatcher: ExecutionContextExecutor = system.dispatcher
val http = Http(system)
val sourceConnectionFlow: Flow[HttpRequest, HttpResponse, Future[Http.OutgoingConnection]] = http.outgoingConnection(sourceUri)
val byteFlow: Flow[HttpResponse, Future[ByteString], NotUsed] =
Flow[HttpResponse].map(_.entity.dataBytes.runFold(ByteString.empty)(_ ++ _))
Source.repeat(HttpRequest(uri = sourceUri))
.via(sourceConnectionFlow)
.via(byteFlow)
.map(postFrame)
.runWith(Sink.ignore)
.onComplete(_ => system.terminate())
def postFrame(imageBytes: Future[ByteString]): Unit = {
imageBytes.onComplete{
case Success(res) => system.log.info(s"post frame. ${res.length} bytes")
case Failure(_) => system.log.error("failed to post image!")
}
}
}
Fore reference, I'm using akka-streams version 2.5.19 and akka-http version 10.1.7. No error is thrown, no error codes on the source server where the frames come from, and the program exits with error code 0.
My application.conf is as follows:
logging = "DEBUG"
Always 100 units processed.
Thanks!
Edit
Added logging to the stream like so
.onComplete{
case Success(res) => {
system.log.info(res.toString)
system.terminate()
}
case Failure(res) => {
system.log.error(res.getMessage)
system.terminate()
}
}
Received a connection reset exception but this is inconsistent. The stream completes with Done.
Edit 2
Using .mapAsync(1)(postFrame) I get the same Success(Done) after precisely 100 requests. Additionally, when I check the nginx server access.log and error.log there are only 200 responses.
I had to modify postFrame as follows to run mapAsync
def postFrame(imageBytes: Future[ByteString]): Future[Unit] = {
imageBytes.onComplete{
case Success(res) => system.log.info(s"post frame. ${res.length} bytes")
case Failure(_) => system.log.error("failed to post image!")
}
Future(Unit)
}
I believe I have found the answer on on the Akka docs using delayed restarts with a backoff operator. Instead of sourcing direct from an unstable remote connection, I use RestartSource.withBackoff and not RestartSource.onFailureWithBackoff. The modified stream looks like;
val restartSource = RestartSource.withBackoff(
minBackoff = 100.milliseconds,
maxBackoff = 1.seconds,
randomFactor = 0.2
){ () =>
Source.single(HttpRequest(uri = sourceUri))
.via(sourceConnectionFlow)
.via(byteFlow)
.mapAsync(1)(postFrame)
}
restartSource
.runWith(Sink.ignore)
.onComplete{
x => {
println(x)
system.terminate()
}
}
I was not able to find the source of the problem but it seems this will work.

Play framework / Akka Streams: Detecting when a WebSocket has closed

When handling WebSockets with Akka Streams directly, I didn't find a proper way to know when the client disconnects (either normally or due to a crash or timeout). I'm using a basic example like the one from the official documentation:
import play.api.mvc._
import akka.stream.scaladsl._
def socket = WebSocket.accept[String, String] { request =>
// Log events to the console
val in = Sink.foreach[String](println)
// Send a single 'Hello!' message and then leave the socket open
val out = Source.single("Hello!").concat(Source.maybe)
Flow.fromSinkAndSource(in, out)
}
I need to know when a client is no longer connected.
Use watchTermination:
def socket = WebSocket.accept[String, String] { request =>
val in = Sink.foreach[String](println)
val out = Source.single("Hello!").concat(Source.maybe)
Flow.fromSinkAndSource(in, out)
.watchTermination() { (_, fut) =>
fut onComplete {
case Success(_) =>
println("Client disconnected")
case Failure(t) =>
println(s"Disconnection failure: ${t.getMessage}")
}
}
}

Spray Client for Jenkins hangs waiting for large response

I'm writing a Spray client for the Jenkins build server. I can query a project with up to 18 build results but a project with 31 has the behaviour that my client never receives the response. I can do this successfully with curl so I'm at a loss for what to look at next.
I do not believe this is about a chunked transfer encoding as I don't see that header in the successful response - although I may be mistaken in my understand of what should happen.
object JenkinsBuildStatus extends App {
implicit val system = ActorSystem("JenkinsBuildStatus")
import system.dispatcher // execution context for futures
val log = Logging(system, getClass)
val pipeline: HttpRequest => Future[HttpResponse] = (
addCredentials(BasicHttpCredentials("username","foobar"))
~> addHeader("Accept", MediaRanges.`*/*`.value)
~> addHeader("User-Agent", "curl/7.37.1")
~> logRequest(log)
~> sendReceive
~> logResponse(log)
)
val response: Future[HttpResponse] = pipeline(Get("https://jenkinsbuilds.corp.com/jenkins/job/Project-Java1.8/api/json/"))
response onComplete {
case Success(data) => {
log.info (data.toString())
shutdown()
}
case Failure(error) => {
log.error(error, "Failure with web client")
shutdown()
}
}
def shutdown(): Unit = {
IO(Http).ask(Http.CloseAll)(1.second).await
system.shutdown()
}
}