Can someone help me get a hold on the spray code to better understand how to spray ?
I'm the context of sending a file as multipart data.
The code that was suggested to me is :
import akka.actor.ActorSystem
import spray.client.pipelining._
import spray.http.{MediaTypes, BodyPart, MultipartFormData}
object UploadFileExample extends App {
implicit val system = ActorSystem("simple-spray-client")
import system.dispatcher // execution context for futures below
val pipeline = sendReceive
val payload = MultipartFormData(Seq(BodyPart(new File("/tmp/test.pdf"), "datafile", MediaTypes.`application/pdf`)))
val request =
Post("http://localhost:8080/file-upload", payload)
pipeline(request).onComplete { res =>
println(res)
system.shutdown()
}
}
Which is fine and works of course.
However i want to understand what is under the hood so i can do things by myself:
Here are the confusion coming from this code:
BodyPart(new File("/tmp/test.pdf"), "datafile", MediaTypes.`application/pdf`)
is the first issue, indeed, BodyPart only has one apply method that is closely match:
def apply(file: File, fieldName: String, contentType: ContentType): BodyPart =
apply(HttpEntity(contentType, HttpData(file)), fieldName, Map.empty.updated("filename", file.getName))
it takes a contentType and not a MediaType.
However i found in contentType
object ContentType {
private[http] case object `; charset=` extends SingletonValueRenderable
def apply(mediaType: MediaType, charset: HttpCharset): ContentType = apply(mediaType, Some(chars et))
implicit def apply(mediaType: MediaType): ContentType = apply(mediaType, None) }
But ContentType is not in the scope of the Main (see first code at the top.) Hence i don't know where that implicit conversion comes from?
Then the last thing that i do not understand here is
val payload = MultipartFormData(Seq(BodyPart(new File("/tmp/test.pdf"),
"datafile", MediaTypes.`application/pdf`)))
Post("http://localhost:8080/file-upload", payload)
The problem here is that, it is based on the RequestBuilding (as can be found in the RequestBuilding Source)
val Post = new RequestBuilder(POST)
with an object that contain the apply method:
def apply[T: Marshaller](uri: String, content: Option[T]): HttpRequest = apply(Uri(uri), content)
....
....
def apply[T: Marshaller](uri: Uri, content: Option[T]): HttpRequest = {
val ctx = new CollectingMarshallingContext {
override def startChunkedMessage(entity: HttpEntity, ack: Option[Any],
headers: Seq[HttpHeader])(implicit sender: ActorRef) =
sys.error("RequestBuilding with marshallers producing chunked requests is not supported")
}
content match {
case None ⇒ HttpRequest(method, uri)
case Some(value) ⇒ marshalToEntityAndHeaders(value, ctx) match {
case Right((entity, headers)) ⇒ HttpRequest(method, uri, headers.toList, entity)
case Left(error) ⇒ throw error
}
}
}
MultiPartFormData is not a Marshaler , hence i do not understand how does it works.
Can someone help me figure that out ?
Many thanks,
M
Related
I am calling the following helper function from an action to make a syncronous HTTP-post with a special key-value pair appended to the URL-parameters in formdata:
def synchronousPost(url: String, formdata: Map[String, Seq[String]], timeout: Duration=Duration.create(30, TimeUnit.SECONDS)): String = {
import ExecutionContext.Implicits.global
val params = formdata.map { case (k, v) => "%s=%s".format(k, URLEncoder.encode(v.head, "UTF-8")) }.mkString("&")
val future: Future[WSResponse] = ws.url(url).
withHttpHeaders(("Content-Type", "application/x-www-form-urlencoded")).
post(params)
try {
Await.result(future, timeout)
future.value.get.get.body
} catch {
case ex: Exception =>
Logger.error(ex.toString)
ex.toString
}
}
It is called like this:
def act = Action { request =>
request.body.asFormUrlEncoded match {
case Some(formdata) =>
synchronousPost(url, formdata + ("handshake" -> List("ok"))) match {
Actually it is copy-pasted from some old gist and I am quite sure it can be rewritten in a cleaner way.
How can the line val params = ... be rewritten in a cleaner way? It seems to be low level.
The original formdata comes in from request.body.asFormUrlEncoded and I simply need to append the handshake parameter to the formdata-map and send the original request back to the sender to do the handshake.
Since formdata is a Map[String, Seq[String]], a data type for which a default WSBodyWritable is provided, you can simply use it as the request body directly:
val future: Future[WSResponse] = ws.url(url)
.withHttpHeaders(("Content-Type", "application/x-www-form-urlencoded"))
.post(formdata)
Incidentally, it's considered bad form to use Await when it's easy to make Play controllers return a Future[Result] using Action.async, e.g:
def asyncPost(url: String, formdata: Map[String, Seq[String]]): Future[String] = {
ws.url(url)
.withHttpHeaders(("Content-Type", "application/x-www-form-urlencoded"))
.post(formdata)
.map(_.body)
}
def action = Action.async { request =>
request.body.asFormUrlEncoded match {
case Some(formdata) =>
asyncPost(url, formdata + ("handshake" -> List("ok"))).map { body =>
Ok("Here is the body: " + body)
} recover {
case e =>
Logger.error(e)
InternalServerError(e.getMessage)
}
case None => Future.successful(BadRequest("No form data given"))
}
}
I am using akka-http and trying to log a request on a specific path using logrequest :
path(Segment / "account") { id =>
logRequest("users/account", Logging.InfoLevel) {
post {
entity(as[Account]) { account => ???
complete(HttpResponse(StatusCodes.NoContent))
}
}
}
however on my log I see something like
HttpRequest(HttpMethod(POST),https://localhost:9009/api/users/123/account,List(Host: localhost:9009, User-Agent: akka-http/10.0.6, Timeout-Access: <function1>),HttpEntity.Chunked(application/json),HttpProtocol(HTTP/1.1))
what I am looking for is the exact request including the body (json) as it was sent by the requestor.
The "HttpEntity.Chunked(application/json)" segment of the log is the output of HttpEntity.Chunked#toString. To get the entire request body, which is implemented as a stream, you need to call HttpEntity#toStrict to convert the Chunked request entity into a Strict request entity. You can make this call in a custom route:
def logRequestEntity(route: Route, level: LogLevel)
(implicit m: Materializer, ex: ExecutionContext) = {
def requestEntityLoggingFunction(loggingAdapter: LoggingAdapter)(req: HttpRequest): Unit = {
val timeout = 900.millis
val bodyAsBytes: Future[ByteString] = req.entity.toStrict(timeout).map(_.data)
val bodyAsString: Future[String] = bodyAsBytes.map(_.utf8String)
bodyAsString.onComplete {
case Success(body) =>
val logMsg = s"$req\nRequest body: $body"
loggingAdapter.log(level, logMsg)
case Failure(t) =>
val logMsg = s"Failed to get the body for: $req"
loggingAdapter.error(t, logMsg)
}
}
DebuggingDirectives.logRequest(LoggingMagnet(requestEntityLoggingFunction(_)))(route)
}
To use the above, pass your route to it:
val loggedRoute = logRequestEntity(route, Logging.InfoLevel)
Given a function with this signature:
def parser[A](otherParser: BodyParser[A]): BodyParser[A]
How can I write the function in such a way that the request body is examined and verified before it is passed to otherParser?
For simplicity let's say that I want to verify that a header ("Some-Header", perhaps) has a value that matches the body exactly. So if I have this action:
def post(): Action(parser(parse.tolerantText)) { request =>
Ok(request.body)
}
When I make a request like curl -H "Some-Header: hello" -d "hello" http://localhost:9000/post it should return "hello" in the response body with a status of 200. If my request is curl -H "Some-Header: hello" -d "hi" http://localhost:9000/post it should return a 400 with no body.
Here's what I've tried.
This one does not compile because otherParser(request).through(flow) expects flow to output a ByteString. The idea here was that the flow could notify the accumulator whether or not to continue processing via the Either output. I'm not sure how to let the accumulator know the status of the previous step.
def parser[A](otherParser: BodyParser[A]): BodyParser[A] = BodyParser { request =>
val flow: Flow[ByteString, Either[Result, ByteString], NotUsed] = Flow[ByteString].map { bytes =>
if (request.headers.get("Some-Header").contains(bytes.utf8String)) {
Right(bytes)
} else {
Left(BadRequest)
}
}
val acc: Accumulator[ByteString, Either[Result, A]] = otherParser(request)
// This fails to compile because flow needs to output a ByteString
acc.through(flow)
}
I also attempted to use filter. This one does compile and the response body that gets written is correct. However it always returns a 200 Ok response status.
def parser[A](otherParser: BodyParser[A]): BodyParser[A] = BodyParser { request =>
val flow: Flow[ByteString, ByteString, akka.NotUsed] = Flow[ByteString].filter { bytes =>
request.headers.get("Some-Header").contains(bytes.utf8String)
}
val acc: Accumulator[ByteString, Either[Result, A]] = otherParser(request)
acc.through(flow)
}
I came up with a solution using a GraphStageWithMaterializedValue. This concept was borrowed from Play's maxLength body parser. The key difference between my first attempt in my question (that doesn't compile) is that instead of attempting to mutate the stream I should use the materialized value to convey information about the state of processing. While I had created a Flow[ByteString, Either[Result, ByteString], NotUsed] it turns out what I needed was a Flow[ByteString, ByteString, Future[Boolean]].
So with that, my parser function ends up looking like this:
def parser[A](otherParser: BodyParser[A]): BodyParser[A] = BodyParser { request =>
val flow: Flow[ByteString, ByteString, Future[Boolean]] = Flow.fromGraph(new BodyValidator(request.headers.get("Some-Header")))
val parserSink: Sink[ByteString, Future[Either[Result, A]]] = otherParser.apply(request).toSink
Accumulator(flow.toMat(parserSink) { (statusFuture: Future[Boolean], resultFuture: Future[Either[Result, A]]) =>
statusFuture.flatMap { success =>
if (success) {
resultFuture.map {
case Left(result) => Left(result)
case Right(a) => Right(a)
}
} else {
Future.successful(Left(BadRequest))
}
}
})
}
The key line is this one:
val flow: Flow[ByteString, ByteString, Future[Boolean]] = Flow.fromGraph(new BodyValidator(request.headers.get("Some-Header")))
The rest kind of falls into place once you are able to create this flow. Unfortunately BodyValidator is pretty verbose and feels somewhat boiler-platey. In any case, it's mostly pretty easy to read. GraphStageWithMaterializedValue expects you to implement def shape: S (S is FlowShape[ByteString, ByteString] here) to specify the input type and output type of this graph. It also expects you to imlpement def createLogicAndMaterializedValue(inheritedAttributes: Attributes): (GraphStageLogic, M) (M is a Future[Boolean] here) to define what the graph should actually do. Here's the full code of BodyValidator (I'll explain in more detail below):
class BodyValidator(expected: Option[String]) extends GraphStageWithMaterializedValue[FlowShape[ByteString, ByteString], Future[Boolean]] {
val in = Inlet[ByteString]("BodyValidator.in")
val out = Outlet[ByteString]("BodyValidator.out")
override def shape: FlowShape[ByteString, ByteString] = FlowShape.of(in, out)
override def createLogicAndMaterializedValue(inheritedAttributes: Attributes): (GraphStageLogic, Future[Boolean]) = {
val status = Promise[Boolean]()
val bodyBuffer = new ByteStringBuilder()
val logic = new GraphStageLogic(shape) {
setHandler(out, new OutHandler {
override def onPull(): Unit = pull(in)
})
setHandler(in, new InHandler {
def onPush(): Unit = {
val chunk = grab(in)
bodyBuffer.append(chunk)
push(out, chunk)
}
override def onUpstreamFinish(): Unit = {
val fullBody = bodyBuffer.result()
status.success(expected.map(ByteString(_)).contains(fullBody))
completeStage()
}
override def onUpstreamFailure(e: Throwable): Unit = {
status.failure(e)
failStage(e)
}
})
}
(logic, status.future)
}
}
You first want to create an Inlet and Outlet to set up the inputs and outputs for your graph
val in = Inlet[ByteString]("BodyValidator.in")
val out = Outlet[ByteString]("BodyValidator.out")
Then you use these to define shape.
def shape: FlowShape[ByteString, ByteString] = FlowShape.of(in, out)
Inside createLogicAndMaterializedValue you need to initialize the value you intend to materialze. Here I've used a promise that can be resolved when I have the full data from the stream. I also create a ByteStringBuilder to track the data between iterations.
val status = Promise[Boolean]()
val bodyBuffer = new ByteStringBuilder()
Then I create a GraphStageLogic to actually set up what this graph does at each point of processing. Two handler are being set. One is an InHandler for dealing with data as it comes from the upstream source. The other is an OutHandler for dealing with data to send downstream. There's nothing really interesting in the OutHandler so I'll ignore it here besides to say that it is necessary boiler plate in order to avoid an IllegalStateException. Three methods are overridden in the InHandler: onPush, onUpstreamFinish, and onUpstreamFailure. onPush is called when new data is ready from upstream. In this method I simply grab the next chunk of data, write it to bodyBuffer and push the data downstream.
def onPush(): Unit = {
val chunk = grab(in)
bodyBuffer.append(chunk)
push(out, chunk)
}
onUpstreamFinish is called when the upstream finishes (surprise). This is where the business logic of comparing the body with the header happens.
override def onUpstreamFinish(): Unit = {
val fullBody = bodyBuffer.result()
status.success(expected.map(ByteString(_)).contains(fullBody))
completeStage()
}
onUpstreamFailure is implemented so that when something goes wrong, I can mark the materialized future as failed as well.
override def onUpstreamFailure(e: Throwable): Unit = {
status.failure(e)
failStage(e)
}
Then I just return the GraphStageLogic I've created and status.future as a tuple.
I left the Akka world for a few months, and apparently I've lost my mojo. I am trying to write a web service that returns either an XML or a JSON document, based on the Accept header.
However, I can't get the Marshallers to work (returns 406, only accepts text/plain). This is what I have:
trait MyMarshallers extends DefaultJsonProtocol with SprayJsonSupport with ScalaXmlSupport {
implicit def ec: ExecutionContext
implicit val itemJsonFormat = jsonFormat3(MyPerson)
def marshalCatalogItem(obj: MyPerson): NodeSeq =
<MyPerson>
<id>
{obj.ID}
</id>
<name>
{obj.Name}
</name>
<age>
{obj.Age}
</age>
</MyPerson>
def marshalCatalogItems(items: Iterable[MyPerson]): NodeSeq =
<Team>
{items.map(marshalCatalogItem)}
</Team>
implicit def catalogXmlFormat = Marshaller.opaque[Iterable[MyPerson], NodeSeq](marshalCatalogItems)
implicit def catalogItemXmlFormat = Marshaller.opaque[MyPerson, NodeSeq](marshalCatalogItem)
implicit val catalogMarshaller: ToResponseMarshaller[Iterable[MyPerson]] = Marshaller.oneOf(
Marshaller.withFixedContentType(MediaTypes.`application/json`) { catalog ⇒
HttpResponse(entity = HttpEntity(ContentType(MediaTypes.`application/json`),
catalog.map(i ⇒ MyPerson(i.ID, i.Name, i.Age))
.toJson.compactPrint))
}
,
Marshaller.withOpenCharset(MediaTypes.`application/xml`) { (catalog, charset) ⇒
HttpResponse(entity = HttpEntity.CloseDelimited(ContentType(MediaTypes.`application/xml`, HttpCharsets.`UTF-8`),
Source.fromFuture(Marshal(catalog.map(i => MyPerson(i.ID, i.Name, i.Age)))
.to[NodeSeq])
.map(ns ⇒ ByteString(ns.toString()))
)
)
}
)
}
and my route looks like this:
class MyService extends MyMarshallers {
implicit val system = ActorSystem("myService")
implicit val materializer = ActorMaterializer()
implicit val ec: ExecutionContext = system.dispatcher
...
def route = ...
(get & path("teams")) {
parameters('name.as[String]) { id =>
complete {
getTeam(id)
}
}
...
}
And if I request /, I get plain text, but if I request application/xml or application/json, I get a 406.
How does AKKA-HTTP decide what content types it will accept? I just updated everything to Akka 2.4.6.
Sorry, I figured it out. The above code had two issues, one of them fixed the problem:
Issue 1 - changed to use fromIterator instead of fromFuture in the
XML marshaller.
Issue 2 - my getTeam() method was returning an Iterable. I changed that to a Seq.
Now everything works.
I've made ActionRefiner to read language of current request from parameter in url:
class LangRequest[A](val lang: Lang, request: Request[A]) extends WrappedRequest[A](request)
def LangAction(lang: String) = new ActionRefiner[Request, LangRequest] {
def refine[A](input: Request[A]) = Future.successful {
val availLangs: List[String] = Play.current.configuration.getStringList("play.i18n.langs").get.toList
if (!availLangs.contains(lang))
Left {
input.acceptLanguages.head match {
case Lang(value, _) if availLangs.contains(value) => Redirect(controllers.routes.Application.index(value))
case _ => Redirect(controllers.routes.Application.index(availLangs.head))
}
}
else Right {
new LangRequest(Lang(lang), input)
}
}
}
and try to use it in action like this:
def login(lng: String) =
LangAction(lng) {
implicit request =>
Ok("Ok")
}
And here I get this "missing parameter type" error for implicit request
What I'm doing wrong?
Thanks in advance
Note: I am new to Play/Scala. There may be a simpler solution.
Likely you had same scenario as me.
Initially tried to implement code similar to examples in https://www.playframework.com/documentation/2.6.x/ScalaActionsComposition#Putting-it-all-together
The difficulty is in implementing shorter action chains than their 'putting it all together'. For example I just needed to refine action based on an ID, but I didn't need any auth.
So rather than having userAction andThen ItemAction(itemId) andThen PermissionCheckAction, I just needed ItemAction(itemId)
I ran into same error as you trying to naively remove the other 2 action funcs.
I believe the issue is that userAction in guide specifies/define what body-parser the request will use.
By removing this part, your request doesn't know what type of body-parser, so it doesn't know the [A] of request[A], hence complaining about type
My Fix: Use an action class (rather than just function), that can have a body-parser passed into constructor
class LeagueRequest[A](val league: League, request: Request[A]) extends WrappedRequest[A](request)
class LeagueAction(val parser: BodyParser[AnyContent], leagueId: String)(implicit val ec: ExecutionContext) extends ActionBuilder[LeagueRequest, AnyContent] with ActionRefiner[Request, LeagueRequest]{
def executionContext = ec
override def refine[A](input: Request[A]) = Future.successful {
inTransaction(
(for {
leagueIdLong <- IdParser.parseLongId(leagueId, "league")
league <- AppDB.leagueTable.lookup(leagueIdLong).toRight(NotFound(f"League id $leagueId does not exist"))
out <- Right(new LeagueRequest(league, input))
} yield out)
)
}
}
with my controller having
def get(leagueId: String) = (new LeagueAction(parse.default, leagueId)).async { implicit request =>
Future(Ok(Json.toJson(request.league)))
}
I could not manage to avoid OP's error using action composition functions, rather than class that extended ActionBuilder.