How can i handle multipart post request with akka http? - scala

I wont to handle multipart request.
If I accept a request using such a route
val routePutData = path("api" / "putFile" / Segment) {
subDir => {
entity(as[String]) { (str) => {
complete(str)
}
}
}}
I get the following text(i try to send log4j config):
Content-Disposition: form-data; name="file"; filename="log4j.properties"
Content-Type: application/binary
log4j.rootLogger=INFO, stdout
log4j.appender.stdout=org.apache.log4j.ConsoleAppender
log4j.appender.stdout.Target=System.out
log4j.appender.stdout.layout=org.apache.log4j.PatternLayout
log4j.appender.stdout.layout.ConversionPattern=%d{yyyy-MM-dd hh:mm:ss} %t %-5p %c{1} - %m%n
----gc0pMUlT1B0uNdArYc0p--
How can i get array of bytes from file i send and file name?
I try to use entity(as[Multipart.FormData]), and formFields directive, but it didn't help.

You should keep up with the akka docs, but I think that there were not enought examples in the file uploading section. Anyway, you don't need to extract entity as a string or byte arrays, akka already has a directive, called fileUpload. This takes a parameter called fieldName which is the key to look for in the multipart request, and expects a function to know what to do given the metadata and the content of the file. Something like this:
post {
extractRequestContext { ctx =>
implicit val mat = ctx.materializer
fileUpload(fieldName = "myfile") {
case (metadata, byteSource) =>
val fileName = metadata.fileName
val futureBytes = byteSource
.mapConcat[Byte] { byteString =>
collection.immutable.Iterable.from(
byteString.iterator
)
}
.toMat(Sink.fold(Array.emptyByteArray) {
case (arr, newLine) => arr :+ newLine
}
)(Keep.right)
.run()
val filePath = Files.createFile(Paths.get(s"/DIR/TO/SAVE/FILE/$fileName"))
onSuccess(futureBytes.map(bytes => Files.write(filePath, bytes))) { _ =>
complete(s"wrote file to: ${filePath.toUri.toString}")
}
}
}
}

While the above solution looks good, there is also the storeUploadedFile directive to achieve the same with less code, sth like:
path("upload") {
def tempDestination(fileInfo: FileInfo): File = File.createTempFile(fileInfo.fileName, ".tmp.server")
storeUploadedFile("myfile", tempDestination) {
case (metadataFromClient: FileInfo, uploadedFile: File) =>
println(s"Server stored uploaded tmp file with name: ${uploadedFile.getName} (Metadata from client: $metadataFromClient)")
complete(HttpResponse(StatusCodes.OK))
}
}

Related

Akka Http route test with formFields causing UnsupportedRequestContentTypeRejection

I have a GET request with parameters and a formField.
It works when I use a client like Insomnia/Postman to send the req.
But the route test below fails with the error:
UnsupportedRequestContentTypeRejection(Set(application/x-www-form-urlencoded, multipart/form-data))
(Rejection created by unmarshallers. Signals that the request was rejected because the requests content-type is unsupported.)
I have tried everything I can think of to fix it but it still returns the same error.
It is the formField that causes the problem, for some reason when called by the test it doesnt like the headers.
Is it something to do with withEntity ?
Code:
path("myurl" ) {
get {
extractRequest { request =>
parameters('var1.as[String], 'var2.as[String], 'var3.as[String], 'var4.as[String]) { (var1, var2, var3, var4) =>
formField('sessionid.as[String]) { (sessionid) =>
complete {
request.headers.foreach(a => println("h=" + a))
}
}
}
}
}
}
Test:
// TESTED WITH THIS - Fails with UnsupportedRequestContentTypeRejection(Set(application/x-www-form-urlencoded, multipart/form-data))
class GETTest extends FreeSpec with Matchers with ScalatestRouteTest {
val get = HttpRequest(HttpMethods.GET, uri = "/myurl?var1=456&var2=123&var3=789&var4=987")
.withEntity("sessionid:1234567890")
.withHeaders(
scala.collection.immutable.Seq(
RawHeader("Content-Type", "application/x-www-form-urlencoded"), // same problem if I comment out these 2 Content-Type lines
RawHeader("Content-Type", "multipart/form-data"),
RawHeader("Accept", "Application/JSON")
)
)
get ~> route ~> check {
status should equal(StatusCodes.OK)
}
The exception is thrown before the formField line.
Full exception:
ScalaTestFailureLocation: akka.http.scaladsl.testkit.RouteTest$$anonfun$check$1 at (RouteTest.scala:57)
org.scalatest.exceptions.TestFailedException: Request was rejected with rejection UnsupportedRequestContentTypeRejection(Set(application/x-www-form-urlencoded, multipart/form-data))
at akka.http.scaladsl.testkit.TestFrameworkInterface$Scalatest$class.failTest(TestFrameworkInterface.scala:24)
}
You could either use:
val get = HttpRequest(HttpMethods.GET, uri = "/myurl?var1=456&var2=123&var3=789&var4=987", entity = FormData("sessionid" -> "1234567.890").toEntity)
or
val get = Get("/myurl?var1=456&var2=123&var3=789&var4=987", FormData("sessionid" -> "1234567.890"))

How to upload files and get formfields in akka-http

I am trying to upload a file via akka-http, and have gotten it to work with the following snippet
def tempDestination(fileInfo: FileInfo): File =
File.createTempFile(fileInfo.fileName, ".tmp")
val route =
storeUploadedFile("csv", tempDestination) {
case (metadata, file) =>
//Do my operation on the file.
complete("File Uploaded. Status OK")
}
But I'd also want to send a param1/param2 in the posted form.
I tried the following, and it works, but I am having to send the parameters via the URL (http://host:port/csv-upload?userid=arvind)
(post & path("csv-upload")) {
storeUploadedFile("csv", tempDestination) {
case (metadata, file) =>
parameters('userid) { userid =>
//logic for processing the file
complete(OK)
}
}
}
The restriction on the file size is around 200-300 MB. I added the following property to my conf
akka{
http{
parsing{
max-content-length=200m
}
}
}
Is there a way, I can get the parameters via the formFields directive ?
I tried the following
fileUpload("csv") {
case (metadata, byteSource) =>
formFields('userid) { userid =>
onComplete(byteSource.runWith(FileIO.toPath(Paths.get(metadata.fileName)))) {
case Success(value) =>
logger.info(s"${metadata}")
complete(StatusCodes.OK)
case Failure(exception) =>
complete("failure")
But, with the above code, I hit the following exception
java.lang.IllegalStateException: Substream Source cannot be materialized more than once
at akka.stream.impl.fusing.SubSource$$anon$13.setCB(StreamOfStreams.scala:792)
at akka.stream.impl.fusing.SubSource$$anon$13.preStart(StreamOfStreams.scala:802)
at akka.stream.impl.fusing.GraphInterpreter.init(GraphInterpreter.scala:306)
at akka.stream.impl.fusing.GraphInterpreterShell.init(ActorGraphInterpreter.scala:593)
Thanks,
Arvind
I got this working with sth like:
path("upload") {
formFields(Symbol("payload")) { payload =>
println(s"Server received request with additional payload: $payload")
def tempDestination(fileInfo: FileInfo): File = File.createTempFile(fileInfo.fileName, ".tmp.server")
storeUploadedFile("binary", tempDestination) {
case (metadataFromClient: FileInfo, uploadedFile: File) =>
println(s"Server stored uploaded tmp file with name: ${uploadedFile.getName} (Metadata from client: $metadataFromClient)")
complete(Future(FileHandle(uploadedFile.getName, uploadedFile.getAbsolutePath, uploadedFile.length())))
}
}
}
Full example:
https://github.com/pbernet/akka_streams_tutorial/blob/master/src/main/scala/akkahttp/HttpFileEcho.scala

Play Framework - edit Sec-WebSocket-Protocol in Web socket Response Header

Edit the Web socket header response sent from server to client.
I am creating a websocket server application using playframework. Right now the websocket response from the server is
taken care by Play. Following is the response header,
Request Header:
(UpgradeToWebSocket,),
(Host,localhost:8083),
(Connection,Upgrade),
(Upgrade,websocket),
(Sec-WebSocket-Version,13),
(Accept-Encoding,gzip, deflate, br),
(Accept-Language,en-US,en;q=0.9),
(Sec-WebSocket-Key,ZvfzpVo3EX4DFA4BRcgRIA==)
def chatSystem(): WebSocket = WebSocket.acceptOrResult[String, String] { request =>
Future.successful{
AuthenticationService.doBasicAuthentication(request.headers) match {
case Results.Ok => Right(ActorFlow.actorRef { out => ChatServiceActor.props(out) })
case _ => Left(Unauthorized)
}
}
}
I want to validate Sec-WebSocket-Protocol if it is present in the request header or add the same with value in the server response if it is not present.
I used the following code:
// Defined at http://tools.ietf.org/html/rfc6455#section-4.2.2
val MagicGuid = "258EAFA5-E914-47DA-95CA-C5AB0DC85B11"
def websocketAcceptForKey(key: String): String = {
val sha1 = MessageDigest.getInstance("sha1")
val salted = key + MagicGuid
val hash = sha1.digest(salted.asciiBytes)
val acceptKey: String = Base64.rfc2045().encodeToString(hash, false)
acceptKey
}
Use it like the following:
val wsKey: Optional[HttpHeader] = request.getHeader("Sec-WebSocket-Key")
val wsAccept = if (wsKey.isPresent) Some(RawHeader("Sec-WebSocket-Accept", websocketAcceptForKey(wsKey.get.value()))) else None

Scala/Spray/Akka unable to leverage mapRequest

I am new to Scala/Spray/AKKA so please excuse this dumb requests.
I have the following Directive and it is being called as the first
logger line ("inside") is showing up in logs.
However, anything inside mapRequest{} is skipped over. The logging line ("headers:") isn't showing up
private def directiveToGetHeaders(input: String) : Directive0 = {
logger.info("inside")
mapRequest { request =>
val headList: List[HttpHeader] = request.headers
logger.info("headers: " + headList.size)
request
}
}
I am not sure what I did wrong. My goal is to pull out all the HTTP headers. Any tip/pointer much appreciated. Thanks
-v
You can use extractRequest directive for getting headers.
private def directiveToGetHeaders(input: String) : Directive0 = {
logger.info("inside")
extractRequest { request =>
val headList: Seq[HttpHeader] = request.headers
logger.info("headers: " + headList.size)
complete(HttpResponse())
}
}

Akka-Http client: How to get binary data from an http response?

I call an API to get a zip file response. The API responds correctly but I am unable to get the byte array from response because the future that should complete on getting the ByteString never completes:
val authorization = akka.http.javadsl.model.headers.Authorization.basic("xxxxx", "xxxxxx")
val query = Map("fed" -> "xxxx", "trd" -> "yyy", "id" -> "zzz")
val request = HttpRequest(HttpMethods.GET, Uri("https://xxxx.yyyy.com/ggg/ttt.php").withQuery(Query(params = query))).addHeader(authorization)
val responseFut = http.singleRequest(request)
responseFut1.map(response => {
println("*******************************")
println(response)
response.status match {
case akka.http.javadsl.model.StatusCodes.OK => {
println("^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^" + response._3)
val entityFut = response.entity.toStrict(60.seconds)
val byteStringFut = entityFut.flatMap(entity => {
entity.dataBytes.runFold(ByteString.empty)(_ ++ _)
})
println("#############################")
try {
byteStringFut.map(x => {
//this never prints =======================================problem
println("----------------------------" + x.toArray[Byte])
})
}catch{
case e: Exception => println("Error: " + e)
}
}
case _ => {}
}
})
If I print out the response this is what it looks like:
*******************************
HttpResponse(200 OK,List(Date: Fri, 08 Sep 2017 20:58:43 GMT, Server: Apache/2.4.18 (Ubuntu), Content-Disposition: attachment; filename="xxxxx.zip", Pragma: public, Cache-Contr
ol: public, must-revalidate, Content-Transfer-Encoding: binary),HttpEntity.Chunked(application/x-zip),HttpProtocol(HTTP/1.1))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^HttpEntity.Chunked(application/x-zip)
#############################
So response is coming back fine, but I still cannot get the binary data for the zip file.
We use akka-http in other places to call APIs that return json response and this approach seems to work fine there.
Why doesnt it work here? What am I doing wrong?
Any advice is appreciated.
Thanks.
Update
Adding byteStringFut.failed.foreach(println(_)) shows this exception: akka.http.scaladsl.model.EntityStreamException: HTTP chunk size exceeds the configured limit of 1048576 bytes
It looks like something went wrong and an exception was thrown in the async computation. You can check the exception by inspecting Future in the following way:
byteStringFut.failed.foreach(println)