I'm using Spray API(spray-client) to hit an external URL and I'm getting gzipped HttpResponse. How do I unzip this HttpResponse to get its entity(json, in my case)?
val future: Future[HttpResponse] = (IO(Http) ? Get(uri)).mapTo[HttpResponse]
val response = Await.result(future, Duration.inf)
val json = response.entity
Here, json is gzipped. How do I unzip it?
You need to use pipelining and the decode directive. Like in this example.
Modifying that example your code would look something like this:
val pipeline: HttpRequest => Future[String] = (
sendReceive
~> decode(Gzip)
~> unmarshal[String]
)
val response: Future[String] =
pipeline(Get(uri))
You can then do Await on the response if you don't want the benefits of Futures.
On a side note you can use spray-json and create an object for your response and then unmarshal the http response directly into a case class without having to deal with the json.
Related
I have an existing Akka HTTP HttpRequest and I want to add two headers to it.
val req: HttpRequest = ???
val hs: Seq[HttpHeader] = Seq(RawHeader("a", "b"))
req.addHeaders(hs)
Expected:
a new HttpRequest object with the additional headers
Actual:
.addHeaders expects a java.lang.Iterable and does not compile.
What is the recommended way of doing this in Scala?
There is a workaround, but it's a bit cludgy:
req.withHeaders(req.headers ++ hs)
Running Scala 2.12.8 and Akka HTTP 10.1.7.
Another workaround that is maybe a tiny sliver less cludgy. This is approximately how addHeaders is defined in the source. I unfortunately have no clue why addHeaders is not exposed in the scala api.
req.mapHeaders(_ ++ hs)
One alternative is to use foldLeft and addHeader:
val req: HttpRequest = ???
val hs: Seq[HttpHeader] = Seq(RawHeader("a", "b"))
hs.foldLeft(req)((r, h) => r.addHeader(h))
You can copy the existing HttpRequest to a new HttpRequest with headers
val req: HttpRequest = ???
val hs: Seq[HttpHeader] = Seq(RawHeader("a", "b"))
val reqWithHeaders: HttpRequest = req.copy(headers=hs)
I'm trying to send a get request to my localhost using Akka HTTP and I get the following exception:
EntityStreamSizeException: actual entity size (Some(10166731700)) exceeded content length limit (8388608 bytes)! You can configure this by setting akka.http.[server|client].parsing.max-content-length or calling HttpEntity.withSizeLimit before materializing the dataBytes stream.)
Basically, the file on my localhost which I'm trying to request is very large.
I tried to solve it by using withoutSizeLimit, but it's not working:
val request = Get("http://localhost:8080/oa-ok.ntriples")
val limitedRequest = request.withEntity(request.entity.withoutSizeLimit())
val responseFuture = Http().singleRequest(limitedRequest)
I also tried to use withSizeLimit and it didn't help. Any ideas?
Use withoutSizeLimit on the response entity, not the request entity. For example:
val responseFuture: Future[HttpResponse] =
Http().singleRequest(HttpRequest(uri = "http://localhost:8080/oa-ok.ntriples"))
val responseSource: Source[ByteString, NotUsed] =
Source.fromFuture(responseFuture)
.flatMapConcat(_entity.withoutSizeLimit.dataBytes)
responseSource is an Akka Streams Source of the response entity.
I have an iterator of mongodb query results and I want to stream those results to http response without loading the whole results set into memory.
Is it possible to complete akka http response with an iterator instead of a collection or future?
Given an Iterator of data:
type Data = ???
val dataIterator : () => Iterator[Data] = ???
You will first need a function to convert Data to ByteString representation, and the ContentType (e.g. json, binary, csv, xml, ...) of the representation:
import akka.util.ByteString
import akka.http.scaladsl.model.ContentType
val dataToByteStr : Data => ByteString = ???
//see akka.http.scaladsl.model.ContentTypes for possible values
val contentType : ContentType = ???
The Iterator and converter function can now be used to create an HttpResponse that will stream the results back to the http client without holding the entire set of Data in memory:
import akka.http.scaladsl.model.HttpEntity.{Chunked, ChunkStreamPart}
import akka.http.scaladsl.model.ResponseEntity
import akka.stream.scaladsl.Source
import akka.http.scaladsl.model.HttpResponse
val chunks : Source[ChunkStreamPart,_] =
Source.fromIterator(dataIterator)
.map(dataToByteStr)
.map(ChunkStreamPart.apply)
val entity : ResponseEntity = Chunked.fromData(contentType, chunks)
val httpResponse : HttpResponse = HttpResponse(entity=entity)
Note: Since a new Iterator is produced each time from dataIterator you don't have to create a new httpResponse for each incoming request; the same response can be used for all requests.
Take a look to Alpakka MongoDB connector. It allows to create one Source from a Mongo collection like:
val source: Source[Document, NotUsed] = MongoSource(numbersColl.find())
val rows: Future[Seq[Document]] = source.runWith(Sink.seq)
Or you maybe want your own source implementation as a GraphStage for example.
I have to send following curl POST request in my akka code where it is sending both json payload and file in the request:
curl \
-F "payload=</tmp/upload_file_payload.json" \
-F "file=#/tmp/file.pdf" \
-v https://host/api
I am trying to implement the same in akka-http, but not sure exactly how to do it. I found some example here and here, but it is not working as it is, so I have written following code, which also have some error, but seems like I am close:
val httpEntity = HttpEntity(MediaTypes.`application/octet-stream`, file, 100000)
// val httpEntity = HttpEntity.fromPath(ContentType.apply(MediaTypes.`application/octet-stream`), Paths.get(file.getAbsolutePath))
val fileFormData = Multipart.FormData.BodyPart.Strict("file", httpEntity, Map.empty)
val jsonFormData = Multipart.FormData.BodyPart.Strict("payload", payload, Map.empty)
// Multipart.FormData.Strict(scala.collection.immutable.Seq(jsonFormData, fileFormData)).toEntity()
val entity = Multipart.FormData( Source(List(jsonFormData, fileFormData))).toEntity()
val httpRequest = HttpRequest(HttpMethods.POST, uri = uri, entity = entity)
But this code is not compiling.
In between, when I made the code compile, I was getting error:
411 Length Requered
I tried added Content-Length header with request but no avail.
Finally after looking at the akka-http code and more attempts, finally I got this working as following, explanation in comments:
val httpEntity = HttpEntity(MediaTypes.`application/octet-stream`, file, 100000).toStrict(10.seconds)(mat) // I had to convert this into strict: which adds Content-Length and tells it is not streaming
val fileFormData = Multipart.FormData.BodyPart.Strict("file", Await.result(httpEntity, 10.seconds), Map.empty) // used Await here to get httpEntity from Future, this might be made better
val jsonFormData = Multipart.FormData.BodyPart.Strict("payload", payload, Map.empty)
val entity = Multipart.FormData(jsonFormData, fileFormData).toEntity() // Corrected this signature
val httpRequest = HttpRequest(HttpMethods.POST, uri = uri, entity = entity)
To avoid 411 error I had to make FormData strict, whcih itself adds required Content-Length header.
But their documentation looks assuming I'm already familiar with Scala, Akka and Spray itself. I mean I couldn't find out how to do this simple basic thing, that I would love to have as one snippet of code in their home page...
The only thing I could find is how to build a request with their spray-httpx:
import spray.httpx.RequestBuilder._
val req = Get("http://url")
The object doesn't have operation to send itself to anywhere, so I'm sure I'm supposed to use Akka things to do it, but their documentation doesn't show the process. Please tell me how to do it. If spray-can do the same thing, I know it can, I would prefer the way.
There is an example here: http://spray.io/documentation/1.1-SNAPSHOT/spray-client/
import spray.http._
import spray.client.pipelining._
implicit val system = ActorSystem()
import system.dispatcher // execution context for futures
val pipeline: HttpRequest => Future[HttpResponse] = sendReceive
val response: Future[HttpResponse] = pipeline(Get("http://spray.io/"))
and even simpler example here: https://github.com/spray/spray/wiki/spray-client
val conduit = new HttpConduit("github.com")
val responseFuture = conduit.sendReceive(HttpRequest(GET, uri = "/"))
In both cases you have to process the result like you normally process a Future, e.g.:
for {response <- responseFuture} yield { someFunction(response) }