How do I do a full proxy with Play Framework?
I want to keep the headers and body intact for both the request and the response. Basically, a transparent proxying layer to both the client and server.
Note: I've got something working. Will post it when SO allows me to.
This is what I end up with.
With my (not comprehensive) testing this works for all methods with various body types.
Note my use of _.head. I haven't dug into why headers have the type Map[String, Seq[String]]. I might be dropping duplicate header contents (e.g. having more than on Content-Type in the header). Perhaps joining the Seq[String] with ; is the better way.
import play.api.libs.ws._
import play.api.libs.iteratee.Enumerator
import play.api.mvc._
def proxy(proxyUrl: String) = Action.async(BodyParsers.parse.raw) { request =>
// filter out the Host and potentially any other header we don't want
val headers: Seq[(String, String)] = request.headers.toSimpleMap.toSeq.filter {
case (headerStr, _) if headerStr != "Host" => true
case _ => false
}
val wsRequestBase: WSRequestHolder = WS.url(s"http://localhost:9000/$proxyUrl") // set the proxy path
.withMethod(request.method) // set our HTTP method
.withHeaders(headers : _*) // Set our headers, function takes var args so we need to "explode" the Seq to var args
.withQueryString(request.queryString.mapValues(_.head).toSeq: _*) // similarly for query strings
// depending on whether we have a body, append it in our request
val wsRequest: WSRequestHolder = request.body.asBytes() match {
case Some(bytes) => wsRequestBase.withBody(bytes)
case None => wsRequestBase
}
wsRequest
.stream() // send the request. We want to process the response body as a stream
.map { case (responseHeader: WSResponseHeaders, bodyStream: Enumerator[Array[Byte]]) => // we want to read the raw bytes for the body
// Content stream is an enumerator. It's a 'stream' that generates Array[Byte]
new Result(
new ResponseHeader(responseHeader.status),
bodyStream
)
.withHeaders(responseHeader.headers.mapValues(_.head).toSeq: _*)
}
}
routes file entry will look something like this:
GET /proxy/*proxyUrl #controllers.Application.proxy(proxyUrl: String)
You will need to other lines to support other methods (e.g. POST)
Feel free to suggest edits.
Related
I'm creating a simple message delivery service using Akka stream. The service is just like mail delivery, where elements from source include destination and content like:
case class Message(destination: String, content: String)
and the service should deliver the messages to appropriate sink based on the destination field. I created a DeliverySink class to let it have a name:
case class DeliverySink(name: String, sink: Sink[String, Future[Done]])
Now, I instantiated two DeliverySink, let me call them sinkX and sinkY, and created a map based on their name. In practice, I want to provide a list of sink names and the list should be configurable.
The challenge I'm facing is how to dynamically choose an appropriate sink based on the destination field.
Eventually, I want to map Flow[Message] to a sink. I tried:
val sinkNames: List[String] = List("sinkX", "sinkY")
val sinkMapping: Map[String, DeliverySink] =
sinkNames.map { name => name -> DeliverySink(name, ???)}.toMap
Flow[Message].map { msg => msg.content }.to(sinks(msg.destination).sink)
but, obviously this doesn't work because we can't reference msg outside of map...
I guess this is not a right approach. I also thought about using filter with broadcast, but if the destination scales to 100, I cannot type every routing. What is a right way to achieve my goal?
[Edit]
Ideally, I would like to make destinations dynamic. So, I cannot statically type all destinations in filter or routing logic. If a destination sink has not been connected, it should create a new sink dynamically too.
If You Have To Use Multiple Sinks
Sink.combine would directly suite your existing requirements. If you attach an appropriate Flow.filter before each Sink then they'll only receive the appropriate messages.
Don't Use Multiple Sinks
In general I think it is bad design to have the structure, and content, of streams contain business logic. Your stream should be a thin veneer for back-pressured concurrency on top of business logic which is in ordinary scala/java code.
In this particular case, I think it would be best to wrap your destination routing inside of a single Sink and the logic should be implemented inside of a separate function. For example:
val routeMessage : (Message) => Unit =
(message) =>
if(message.destination equalsIgnoreCase "stdout")
System.out println message.content
else if(message.destination equalsIgnoreCase "stderr")
System.err println message.content
val routeSink : Sink[Message, _] = Sink foreach routeMessage
Note how much easier it is to now test my routeMessage since it isn't inside of the stream: I don't need any akka testkit "stuff" to test routeMessage. I can also move the function to a Future or a Thread if my concurrency design were to change.
Many Destinations
If you have many destinations you can use a Map. Suppose, for example, you are sending your messages to AmazonSQS. You could define a function to convert a Queue Name to Queue URL and use that function to maintain a Map of already created names:
type QueueName = String
val nameToRequest : (QueueName) => CreateQueueRequest = ??? //implementation unimportant
type QueueURL = String
val nameToURL : (AmazonSQS) => (QueueName) => QueueURL = {
val nameToURL = mutable.Map.empty[QueueName, QueueURL]
(sqs) => (queueName) => nameToURL.get(queueName) match {
case Some(url) => url
case None => {
sqs.createQueue(nameToRequest(queueName))
val url = sqs.getQueueUrl(queueName).getQueueUrl()
nameToURL put (queueName, url)
url
}
}
}
Now you can use this non-stream function inside of a singular Sink:
val sendMessage : (AmazonSQS) => (Message) => Unit =
(sqs) => (message) =>
sqs sendMessage {
(new SendMessageRequest())
.withQueueUrl(nameToURL(sqs)(message.destination))
.withMessageBody(message.content)
}
val sqs : AmazonSQS = ???
val messageSink = Sink foreach sendMessage(sqs)
Side Note
For destination you probably want to use something other than String. A coproduct is usually better because they can be used with case statements and you'll get helpful compiler errors if you miss one of the possibilities:
sealed trait Destination
object Out extends Destination
object Err extends Destination
object SomethingElse extends Destination
case class Message(destination: Destination, content: String)
//This function won't compile because SomethingElse doesn't have a case
val routeMessage : (Message) => Unit =
(message) => message.destination match {
case Out =>
System.out.println(message.content)
case Err =>
System.err.println(message.content)
}
Given your requirement, maybe you want to consider multiplexing your stream source into substreams using groubBy:
import akka.actor.ActorSystem
import akka.stream.ActorMaterializer
import akka.stream.scaladsl._
import akka.util.ByteString
import akka.{NotUsed, Done}
import akka.stream.IOResult
import scala.concurrent.Future
import java.nio.file.Paths
import java.nio.file.StandardOpenOption._
implicit val system = ActorSystem("sys")
implicit val materializer = ActorMaterializer()
import system.dispatcher
case class Message(destination: String, content: String)
case class DeliverySink(name: String, sink: Sink[ByteString, Future[IOResult]])
val messageSource: Source[Message, NotUsed] = Source(List(
Message("a", "uuu"), Message("a", "vvv"),
Message("b", "xxx"), Message("b", "yyy"), Message("b", "zzz")
))
val sinkA = DeliverySink("sink-a", FileIO.toPath(
Paths.get("/path/to/sink-a.txt"), options = Set(CREATE, WRITE)
))
val sinkB = DeliverySink("sink-b", FileIO.toPath(
Paths.get("/path/to/sink-b.txt"), options = Set(CREATE, WRITE)
))
val sinkMapping: Map[String, DeliverySink] = Map("a" -> sinkA, "b" -> sinkB)
val totalDests = 2
messageSource.map(m => (m.destination, m)).
groupBy(totalDests, _._1).
fold(("", List.empty[Message])) {
case ((_, list), (dest, msg)) => (dest, msg :: list)
}.
mapAsync(parallelism = totalDests) {
case (dest: String, msgList: List[Message]) =>
Source(msgList.reverse).map(_.content).map(ByteString(_)).
runWith(sinkMapping(dest).sink)
}.
mergeSubstreams.
runWith(Sink.ignore)
Not sure if I'm getting this whole routing DSL thing right but here's the question. I want to do a post to external service such as:
val post = pathPrefix("somePath") {
post {
//get the response mapped to my Output object
}
}
Then I want the response (which is a Json) to be mapped to an object matching the fields for example Output (assuming I have my JsonProtocol set up). How is this done?
You are using HTTP server directives to "retrieve" something "externally". This is what typically an HTTP client does.
For this sort of things, you can use akka http client api.
For example:
val response = Http().singleRequest(HttpRequest(uri = "http://akka.io"))
response onComplete {
case Success(res) =>
val entity = Unmarshal(res.entity).to[YourDomainObject]
// use entity here
case Failure(ex) => // do something here
}
However, this requires some Unmarshaller (to deserialize the received json). Take also a look at Json Support, as it helps you define marshallers easily:
case class YourDomainObject(id: String, name: String)
implicit val YourDomainObjectFormat = jsonFormat2(YourDomainObject)
I think what you are trying to ask is how to get the body i.e in JSOn format to the Case class that you have
Here is a quick example:
path("createBot" / Segment) { tag: String =>
post {
decodeRequest {
entity(as[CaseClassName]) { caseclassInstance: CaseClassName =>
val updatedAnswer = doSomeStuff(caseclassInstance)
complete {
"Done"
}
}
}
You can find more detailed example from here : https://github.com/InternityFoundation/Stackoverflowbots/blob/master/src/main/scala/in/internity/http/RestService.scala#L56
I hope it answers your question.
I am using Akka Http to make requests to a 3rd party API. The responses are "application/json", and I would like to use Akka Http to convert them to a custom case class. I would like to do something like this:
val request = RequestBuilding.Get("https://service.com/v1/api/items")
val response : Future[ItemsResponse] = http.singleRequest(request).flatMap({ response =>
Unmarshal(response.entity).to[ItemsResponse]
})
This fails to compile, because I am missing an implicit unmarshaller of type akka.http.scaladsl.unmarshalling.Unmarshaller[akka.http.scaladsl.model.ResponseEntity, com.mycompany.models.ItemsResponse].
It's unclear to me what the idiomatic way to do this with akka http is. I am aware that I could use spray-json, but I'd like to understand how to do this without importing another library. It seems possible with Akka Http, but the documentation isn't clear (to me at least).
The simplest way is to use spray-json as it comes as part of Akka HTTP:
import spray.json._
import akka.http.scaladsl.marshallers.sprayjson.SprayJsonSupport
// change 2 to the number of attributes of ItemsResponse
implicit val ItemsResponseFormat = jsonFormat2(ItemsResponse)
This should make your existing code compile.
I think your question is valid, and there are cases where avoiding extra dependencies makes sense. Mine is from making an authentication library, where I don't want to impose my JSON library preferences to the users of such library. The library needs JSON unmarshalling for understanding a token info response.
To the code! :)
case class TokenInfo private (uid: String, realm: String, scope: Seq[String])
object TokenInfo {
private
def parseOpt(s: String): Option[TokenInfo] = {
util.parsing.json.JSON.parseFull(s) match {
case Some(map: Map[String,Any] #unchecked) =>
val tmp: Map[String,Any] = map.collect {
case (k# "uid",x: String) => k -> x
case (k# "realm",x: String) => k -> x
case (k# "scope",x: Seq[String] #unchecked) => k -> x
// other keys are ignored
}.toMap
if (tmp.size == 3) {
Some( TokenInfo( tmp("uid").asInstanceOf[String], tmp("realm").asInstanceOf[String], tmp("scope").asInstanceOf[Seq[String]]) )
} else {
None
}
case _ => None
}
}
implicit
val unm: FromEntityUnmarshaller[TokenInfo] = {
PredefinedFromEntityUnmarshallers.stringUnmarshaller.map{ s => parseOpt(s).getOrElse{
throw new RuntimeException(s"Unknown TokenInfo: $s")
}}
}
}
I chose to use util.parsing.json which comes within Scala. The other option was simply regex's, but in that case I'm either fixing the expected order of fields, or the code might get complex.
I'm building a micro-service using Play Framework 2.3.x using Scala (I'm a beginner in both) but I can't figure out a way to stream my request body.
Here is the problem:
I need an endpoint /transform where I can receive a huge TSV file that I will parse and render in another format: simple transformation. The problem is that every single command in my controller is ran "too late". It waits to receive the full file before starting the code.
Example:
def transform = Action.async {
Future {
Logger.info("Too late")
Ok("A response")
}
}
I want to be able to read line-by-line the request body during its upload and process already the request without having to wait for the file to be received completely.
Any hint would be welcome.
This answer applies to Play 2.5.x and higher since it uses the Akka streams API that replaced Play's Iteratee-based streaming in that version.
Basically, you can create a body parser that returns a Source[T] that you can pass to Ok.chunked(...). One way to do this is to use Accumulator.source[T] in the body parser. For example, an action that just returned data sent to it verbatim might look like this:
def verbatimBodyParser: BodyParser[Source[ByteString, _]] = BodyParser { _ =>
// Return the source directly. We need to return
// an Accumulator[Either[Result, T]], so if we were
// handling any errors we could map to something like
// a Left(BadRequest("error")). Since we're not
// we just wrap the source in a Right(...)
Accumulator.source[ByteString]
.map(Right.apply)
}
def stream = Action(verbatimBodyParser) { implicit request =>
Ok.chunked(request.body)
}
If you want to do something like transform a TSV file you can use a Flow to transform the source, e.g:
val tsvToCsv: BodyParser[Source[ByteString, _]] = BodyParser { req =>
val transformFlow: Flow[ByteString, ByteString, NotUsed] = Flow[ByteString]
// Chunk incoming bytes by newlines, truncating them if the lines
// are longer than 1000 bytes...
.via(Framing.delimiter(ByteString("\n"), 1000, allowTruncation = true))
// Replace tabs by commas. This is just a silly example and
// you could obviously do something more clever here...
.map(s => ByteString(s.utf8String.split('\t').mkString(",") + "\n"))
Accumulator.source[ByteString]
.map(_.via(transformFlow))
.map(Right.apply)
}
def convert = Action(tsvToCsv) { implicit request =>
Ok.chunked(request.body).as("text/csv")
}
There may be more inspiration in the Directing the Body Elsewhere section of the Play docs.
I have been having issues with downloading media from the media uri provided on the mms messages.
val url = https://api.twilio.com/2010-04-01/Accounts/xx/Messages/xx/Media/xx
the media url provided is in the above structure,
new URL(url) #> new File("file.png") !! //this fails, due to multiple redirects
When I open the URI in browser the redirect ends up in
http://media.twiliocdn.com.s3-external-1.amazonaws.com/xx/xx
1st url -> 2nd url -> above url ;so,all in all 2 redirects
And if I try the snippet posted above with the new url, it works. I am sure its because of the multiple redirects, the snippet didnt work in the first place.
Been using play framework with scala, can I get any source example to download the file. Any help or pointers is appreciated. Tried various examples but still could not solve the issue.
Some findings =>
Accessing Twilio MMS images
anything similar for scala?
Update: #millhouse
def fileDownloader(urls: String, location: String) = {
import play.api.Play.current
import scala.concurrent.ExecutionContext.Implicits.global
// Make the request
val futureResponse: Future[(WSResponseHeaders, Enumerator[Array[Byte]])] =
WS.url(urls).withFollowRedirects(true).getStream()
futureResponse.flatMap {
case (headers, body) =>
val file = new File(location)
val outputStream = new FileOutputStream(file)
// The iteratee that writes to the output stream
val iteratee = Iteratee.foreach[Array[Byte]] { bytes =>
outputStream.write(bytes)
}
// Feed the body into the iteratee
(body |>>> iteratee).andThen {
case result =>
// Close the output stream whether there was an error or not
outputStream.close()
// Get the result or rethrow the error
result.get
}.map(_ => file)
}
}
This is the approach I had been using till now(works), as explained in the play docs. But I needed a sync approach, meaning I would need to carry out another step on successful file download. Sorry, for not clarifying out ahead.
Update 2 : Solved in this manner,
def fileDownloader(urls: String, location: String) = {
import play.api.Play.current
import scala.concurrent.ExecutionContext.Implicits.global
// Make the request
val futureResponse: Future[(WSResponseHeaders, Enumerator[Array[Byte]])] =
WS.url(urls).withFollowRedirects(true).getStream()
val downloadedFile: Future[File] = futureResponse.flatMap {
case (headers, body) =>
val file = new File(location)
val outputStream = new FileOutputStream(file)
// The iteratee that writes to the output stream
val iteratee = Iteratee.foreach[Array[Byte]] { bytes =>
outputStream.write(bytes)
}
// Feed the body into the iteratee
(body |>>> iteratee).andThen {
case result =>
// Close the output stream whether there was an error or not
outputStream.close()
// Get the result or rethrow the error
result.get
}.map(_ => file)
}
downloadedFile.map{ fileIn =>
//things needed to do
}
}
Thanks,
I haven't used the Twilio MMS API but it should be very straightforward to get the Play Framework HTTP client to follow redirects, using the documented option to the client:
val url = "https://api.twilio.com/2010-04-01/Accounts/xx/Messages/xx/Media/xx"
ws.url(url).withFollowRedirects(true).get().map { response =>
val theBytes:Array[Byte] = response.bodyAsBytes // Play 2.4 and lower
// ... save it
}
Note that the above code works for Play 2.4.x and lower; the bodyAsBytes method of WSResponse returns an Array[Byte]. If you're on the current cutting-edge and using Play 2.5.x, bodyAsBytes gives you an Akka ByteString with lots of nice functional methods, but you probably just want to call toArray on it if all you want is to store the data:
ws.url(url).withFollowRedirects(true).get().map { response =>
val theBytes:Array[Byte] = response.bodyAsBytes.toArray // Play 2.5
// ... save it
}