I'm trying to serve an HTML page using Spray. It's fairly easy using getFromResource and getFromResourceDirectory, but I additionally need to pass some query parameters so that some Javascript on the page knows what to do. Is that possible ? All my prior attempts consisted in this kind of things
val route = path("show-repo") { serveResourceWithParams(SHOW_REPO) } ~ getFromResourceDirectory("web")
def serveWithParams(page: String, params: (String, String)*) = {
val url = page + (if (params.isEmpty) "" else "?") + params.map { case (k, v) => s"$k=$v" }.mkString("&")
getFromResource(url)
}
but I now realize it was a bit naive
I would create a new case class with members of type Array[Byte] and List[(String, String)].
case class FileWithParams(file: Array[Byte], params: List[(String, String)])
If you know the file won't be huge, you can read it in one shot with
FileUtils.readAllBytes
Assuming you're using(which you need to run Spray) https://github.com/sirthias/parboiled/blob/master/parboiled-core/src/main/java/org/parboiled/common/FileUtils.java
Then manually read the file contents, create the case class, and marshall it as you would normally do.
Of course, my preferred way would be to split this up into two async requests and wait for both to complete. You'll probably be waiting on the file route to complete anyway.
Related
In Http4s 0.16.6a, I had the following service.
import org.http4s.server.staticcontent._
import org.http4s._
object StaticFiles {
val basepath = ...
def apply(): HttpService = Service.lift(request => {
val file = basepath + request.uri.path
StaticFile.fromString(file, Some(request)).fold {
Pass.now // aka fallthrough
} {
NoopCacheStrategy.cache(request.pathInfo, _)
}
})
}
It takes the path from the url and tries to work out if a static file can be served. So a GET request to /index.html would try and load it with fromFile and if it can't be found, fallthrough or "Pass". When composed with other services using ||, it meant the total function (from lift) would be treated a bit like a partial function (from apply).
I can't seem to convert this over to Http4s 0.18.x.
The Http4s docs suggest the following:
import cats.effect._
import org.http4s._
import org.http4s.dsl.io._
import java.io.File
val service = HttpService[IO] {
case request # GET -> Root / "index.html" =>
StaticFile.fromFile(new File("relative/path/to/index.html"), Some(request))
.getOrElseF(NotFound()) // In case the file doesn't exist
}
Which is the basic form of what I'm trying to do, only I'd like to generify it a little and not create a partial function for every file I want to serve. i.e. avoid this:
case request # GET -> Root / "index.html" => ???
case request # GET -> Root / "fileA.html" => ???
case request # GET -> Root / "fileB.html" => ???
So, my questions are:
Is there the concept of Pass and passthough when using lift in 0.18?
How to I use the NooopCacheStretegy with lift?
Ultimately, how to I convert the code above to 0.18?
My endeavours so far have lead to this abomination (which obvs doesn't compile):
def apply(): HttpService[IO] = HttpService.lift(request => {
val target = basepath + request.uri.path
StaticFile.fromString[IO](target, Some(request)).fold {
// passthrough
IO.pure(???)
} {
// process request
response => NoopCacheStrategy[IO].cache(request.pathInfo, response).unsafeRunSync()
}
})
Note that I'm trying to use HttpService.lift not OptionT.liftF (as recommended). Mostly because I have no clue how to!
So as far as I can tell, the concept of Pass has been replaced by OptionT in 0.18.x, with None playing the role of Pass. However, you don't have access to the OptionT with that overload. Instead, the assumption is that since you're passing a partial function to HttpService, the requests the function is defined on are precisely the ones you want this service to provide a response for.
You could try making it work with OptionT.lift, but I wouldn't recommend it either! Instead, I'd make a partial function that's only defined on arguments when your static files exists. The way http4s allows you to define the criteria needed to hit an endpoint through pattern-matching on requests is extremely powerful, and you're completely ignoring that option in both of your solutions.
As far as NoopCacheStrategy is concerned, I'm guessing the problem you ran into was that the return type of StaticFile.fromX is now IO[Response[IO]] and NoopCacheStrategy takes a Response[IO]. This is easily handled via flatMap.
Which is to say that this is what I came up with to translate your code to 0.18.x:
import java.nio.file.{Files, Paths}
import cats.effect.IO
import org.http4s.{HttpService, StaticFile}
import org.http4s.dsl.io._
import org.http4s.server.staticcontent.NoopCacheStrategy
val service = HttpService[IO] {
case request # _ -> _ / file if Files.exists(Paths.get(file)) =>
StaticFile
.fromString(file, Some(request))
.getOrElseF(NotFound())
.flatMap(resp => NoopCacheStrategy[IO].cache(request.pathInfo, resp))
}
A slight annoyance is that we're actually handling the case where no such file exists twice, once in the if clause of the case statement and once with getOrElseF. In practice, that NotFound should never be reached. I figure one can live with this.
And as an example of what I mean by the power of http4s' pattern matching on requests, by adjusting the case statement it's very easy to make sure this will only...
match on GET requests: case request # GET -> _ / file if ...
match top-level files, nothing in subdirectories: case request # _ -> Root / file if ...
match HTML files: case request # _ -> _ / file ~ html if Files.exists(Paths.get(s"$file.html"))
You can even write your own custom extractor that checks whether you can serve a given file name to end up with something like case request # _ -> _ / ValidStaticFile(file). That way you don't have to cram all your logic into the case statement.
I can't seem to format as a comment so posted as a new answer...
How about this?
def apply(): HttpService[IO] = Kleisli.apply(request => {
val basepath = ...
val target = location + request.uri.path
StaticFile.fromString[IO](target, Some(request))
})
I am trying to prevent empty values being inserted into my mongoDB collection. The field in question looks like this:
MongoDB Field
"stadiumArr" : [
"Old Trafford",
"El Calderon",
...
]
Sample of (mapped) case class
case class FormData(_id: Option[BSONObjectID], stadiumArr: Option[List[String]], ..)
Sample of Scala form
object MyForm {
val form = Form(
mapping(
"_id" -> ignored(Option.empty[BSONObjectID]),
"stadiumArr" -> optional(list(text)),
...
)(FormData.apply)(FormData.unapply)
)
}
I am also using the Repeated Values functionality in Play Framework like so:
Play Template
#import helper._
#(myForm: Form[models.db.FormData])(implicit request: RequestHeader, messagesProvider: MessagesProvider)
#repeatWithIndex(myForm("stadiumArr"), min = 5) { (stadium, idx) =>
#inputText(stadium, '_label -> ("stadium #" + (idx + 1)))
}
This ensures that whether there are at least 5 values or not in the array; there will still be (at least) 5 input boxes created. However if one (or more) of the input boxes are empty when the form is submitted an empty string is still being added as value in the array, e.g.
"stadiumArr" : [
"Old Trafford",
"El Calderon",
"",
"",
""
]
Based on some other ways of converting types from/to the database; I've tried playing around with a few solutions; such as:
implicit val arrayWrite: Writes[List[String]] = new Writes[List[String]] {
def writes(list: List[String]): JsValue = Json.arr(list.filterNot(_.isEmpty))
}
.. but this isn't working. Any ideas on how to prevent empty values being inserted into the database collection?
Without knowing specific versions or libraries you're using it's hard to give you an answer, but since you linked to play 2.6 documentation I'll assume that's what you're using there. The other assumption I'm going to make is that you're using reactive-mongo library. Whether or not you're using the play plugin for that library or not is the reason why I'm giving you two different answers here:
In that library, with no plugin, you'll have defined a BSONDocumentReader and a BSONDocumentWriter for your case class. This might be auto-generated for you with macros or not, but regardless how you get it, these two classes have useful methods you can use to transform the reads/writes you have to another one. So, let's say I defined a reader and writer for you like this:
import reactivemongo.bson._
case class FormData(_id: Option[BSONObjectID], stadiumArr: Option[List[String]])
implicit val formDataReaderWriter = new BSONDocumentReader[FormData] with BSONDocumentWriter[FormData] {
def read(bson: BSONDocument): FormData = {
FormData(
_id = bson.getAs[BSONObjectID]("_id"),
stadiumArr = bson.getAs[List[String]]("stadiumArr").map(_.filterNot(_.isEmpty))
)
}
def write(formData: FormData) = {
BSONDocument(
"_id" -> formData._id,
"stadiumArr" -> formData.stadiumArr
)
}
}
Great you say, that works! You can see in the reads I went ahead and filtered out any empty strings. So even if it's in the data, it can be cleaned up. That's nice and all, but let's notice I didn't do the same for the writes. I did that so I can show you how to use a useful method called afterWrite. So pretend the reader/writer weren't the same class and were separate, then I can do this:
val initialWriter = new BSONDocumentWriter[FormData] {
def write(formData: FormData) = {
BSONDocument(
"_id" -> formData._id,
"stadiumArr" -> formData.stadiumArr
)
}
}
implicit val cleanWriter = initialWriter.afterWrite { bsonDocument =>
val fixedField = bsonDocument.getAs[List[String]]("stadiumArr").map(_.filterNot(_.isEmpty))
bsonDocument.remove("stadiumArr") ++ BSONDocument("stadiumArr" -> fixedField)
}
Note that cleanWriter is the implicit one, that means when the insert call on the collection happens, it will be the one chosen to be used.
Now, that's all a bunch of work, if you're using the plugin/module for play that lets you use JSONCollections then you can get by with just defining play json Reads and Writes. If you look at the documentation you'll see that the reads trait has a useful map function you can use to transform one Reads into another.
So, you'd have:
val jsonReads = Json.reads[FormData]
implicit val cleanReads = jsonReads.map(formData => formData.copy(stadiumArr = formData.stadiumArr.map(_.filterNot(_.isEmpty))))
And again, because only the clean Reads is implicit, the collection methods for mongo will use that.
NOW, all of that said, doing this at the database level is one thing, but really, I personally think you should be dealing with this at your Form level.
val form = Form(
mapping(
"_id" -> ignored(Option.empty[BSONObjectID]),
"stadiumArr" -> optional(list(text)),
...
)(FormData.apply)(FormData.unapply)
)
Mainly because, surprise surprise, form has a way to deal with this. Specifically, the mapping class itself. If you look there you'll find a transform method you can use to filter out empty values easily. Just call it on the mapping you need to modify, for example:
"stadiumArr" -> optional(
list(text).transform(l => l.filter(_.nonEmpty), l => l.filter(_.nonEmpty))
)
To explain a little more about this method, in case you're not used to reading the signatures in the scaladoc.
def
transform[B](f1: (T) ⇒ B, f2: (B) ⇒ T): Mapping[B]
says that by calling transform on some mapping of type Mapping[T] you can create a new mapping of type Mapping[B]. In order to do this you must provide functions that convert from one to the other. So the code above causes the list mapping (Mapping[List[String]]) to become a Mapping[List[String]] (the type did not change here), but when it does so it removes any empty elements. If I break this code down a little it might be more clear:
def convertFromTtoB(list: List[String]): List[String] = list.filter(_.nonEmpty)
def convertFromBtoT(list: List[String]): List[String] = list.filter(_.nonEmpty)
...
list(text).transform(convertFromTtoB, convertFromBtoT)
You might wondering why you need to provide both, the reason is because when you call Form.fill and the form is populated with values, the second method will be called so that the data goes into the format the play form is expecting. This is more obvious if the type actually changes. For example, if you had a text area where people could enter CSV but you wanted to map it to a form model that had a proper List[String] you might do something like:
def convertFromTtoB(raw: String): List[String] = raw.split(",").filter(_.nonEmpty)
def convertFromBtoT(list: List[String]): String = list.mkString(",")
...
text.transform(convertFromTtoB, convertFromBtoT)
Note that when I've done this in the past sometimes I've had to write a separate method and just pass it in if I didn't want to fully specify all the types, but you should be able to work from here given the documentation and type signature for the transform method on mapping.
The reason I suggest doing this in the form binding is because the form/controller should be the one with the concern of dealing with your user data and cleaning things up I think. But you can always have multiple layers of cleaning and whatnot, it's not bad to be safe!
I've gone for this (which always seems obvious when it's written and tested):
implicit val arrayWrite: Writes[List[String]] = new Writes[List[String]] {
def writes(list: List[String]): JsValue = Json.toJson(list.filterNot(_.isEmpty).toIndexedSeq)
}
But I would be interested to know how to
.map the existing Reads rather than redefining from scratch
as #cchantep suggests
I am using Alpakka-FTP, but maybe I'm looking for a general akka-stream pattern. The FTP connector can list files or retrieve them:
def ls(host: String): Source[FtpFile, NotUsed]
def fromPath(host: String, path: Path): Source[ByteString, Future[IOResult]]
Ideally, I would like to create a stream like this:
LIST
.FETCH_ITEM
.FOREACH(do something)
But I'm unable to create such a stream with the two functions I wrote above. I feel like I should be able to get there using a Flow, something like
Ftp.ls
.via(some flow that uses the Ftp.fromPath above)
.runWith(Sink.foreach(do something))
Is this possible, given only the ls and fromPath functions above?
EDIT:
I am able to work it out using one actor and mapAsync, but I still feel it should be more straightforward.
class Downloader extends Actor {
override def receive = {
case ftpFile: FtpFile =>
Ftp.fromPath(Paths.get(ftpFile.path), settings)
.toMat(FileIO.toPath(Paths.get("testHDF.txt")))(Keep.right)
.run() pipeTo sender
}
}
val downloader = as.actorOf(Props(new Downloader))
Ftp.ls("test_path", settings)
.mapAsync(1)(ftpFile => (downloader ? ftpFile) (3.seconds).mapTo[IOResult])
.runWith(Sink.foreach(res => println("got it!" + res)))
You should be able to use flatMapConcat for this purpose. Your specific example could be re-written to
Ftp.ls("test_path", settings).flatMapConcat{ ftpFile =>
Ftp.fromPath(Paths.get(ftpFile.path), settings)
}.runWith(FileIO.toPath(Paths.get("testHDF.txt")))
Documentation here.
Context: I'm trying to write a Process1[ByteVector, spray.http.HttpResponsePart] with output ChunkedResponseStart(bytes), MessageChunk(bytes), MessageChunk(bytes), ..., ChunkedResponseEnd. I haven't yet fully wrapped my head around scalaz-stream and its vocabulary.
How to write a Process that can handle first n chunks differently?
I've come up with this (strings as an example):
val headerChunk = process1.chunk[String](5).map(_.reduce(_ + _))
val headerChunkAndRest: Process1[String, String] =
headerChunk.take(1) ++ process1.id
io.linesR(Files.newInputStream(Paths.get("testdata/fahrenheit.txt")))
.pipe(headerChunkAndRest)
.to(io.stdOutLines)
.run.run
What is an idiomatic and, possibly, a generally composable way to write headerChunkAndRest?
General Considerations
There are several ways to do this, strongly depending on the details of your needs. You can use the following helper methods that are part of scalaz-streams:
foldWithIndex This gives you the current index of the chunk as a number. You can discriminate based on that index
zipWithState You can add a state from one invocation of your method to the next and use this state to track if you are still parsing headers or a if you have reached the body. In the next step you can then use this state to handle header and body different
repartition Use this to group all header and all body elements together. You can then process them in the next step.
zipWithNext This function always presents you the previous element grouped with the current element. You can use this to detect, when you are switching from header to body and react accordingly.
Possibly you should re-think, what you really need. For exactly your question, it would be zipwithIndex and then map. But if you re-think your problem, you will probably end with repartition or zipWithState.
Example code
Let's make a simple example: A HTTP client, that separates the HTTP header elements from the body (HTTP, not HTML). In the header a things like cookies, in the body is the real "content", like an image or the HTTP sources.
A simple HTTP client could look like this:
import scalaz.stream._
import scalaz.concurrent.Task
import java.net.InetSocketAddress
import java.nio.channels.AsynchronousChannelGroup
implicit val AG = nio.DefaultAsynchronousChannelGroup
def httpGetRequest(hostname : String, path : String = "/"): Process[Nothing, String] =
Process(
s"GET $path HTTP/1.1",
s"Host: $hostname",
"Accept: */*",
"User-Agent: scalaz-stream"
).intersperse("\n").append(Process("\n\n"))
def simpleHttpClient(hostname : String, port : Int = 80, path : String = "/")(implicit AG: AsynchronousChannelGroup) : Process[Task, String] =
nio.connect(new InetSocketAddress(hostname, port)).flatMap(_.run(httpGetRequest(hostname, path).pipe(text.utf8Encode))).pipe(text.utf8Decode).pipe(text.lines())
Now we can use this code to separate header lines from the rest. In HTTP, the header is structured in lines. It is separated from the body by an empty line. So first, let's count the number of lines in the header:
val demoHostName="scala-lang.org" // Hope they won't mind...
simpleHttpClient(demoHostName).zipWithIndex.takeWhile(! _._1.isEmpty).runLast.run
// res3: Option[(String, Int)] = Some((Content-Type: text/html,8))
When I ran this, there were 8 lines in the header. Let's first define an enumeration, so classify the parts of the response:
object HttpResponsePart {
sealed trait EnumVal
case object HeaderLine extends EnumVal
case object HeaderBodySeparator extends EnumVal
case object Body extends EnumVal
val httpResponseParts = Seq(HeaderLine, HeaderBodySeparator, Body)
}
And then let's use zipWithIndex plus map to classify the parts of the response:
simpleHttpClient(demoHostName).zipWithIndex.map{
case (line, idx) if idx < 9 => (line, HeaderLine)
case (line, idx) if idx == 10 => (line, HeaderBodySeparator)
case (line, _) => (line, Body)
}.take(15).runLog.run
For me, this works fine. But of course, the amount of header lines can change at any time without notice. It is much more robust to use a very simple parser that considers the structure of the response. for this I use zipWithState:
simpleHttpClient(demoHostName).zipWithState(HeaderLine : EnumVal){
case (line, HeaderLine) if line.isEmpty => HeaderBodySeparator
case (_, HeaderLine) => HeaderLine
case (_, HeaderBodySeparator) => Body
case (line, Body) => Body
}.take(15).runLog.run
You can see, that both approaches use a similar structure and both approaches should lead to the same result. The fine thing is, both approaches are easily reusable. You can just swap out the source, e.g. with a file, and don't have to change anything. Same with the processing after the classification. The .take(15).runLog.run is exactly the same in both approaches.
I'm writing a Play 2.3.2 application in Scala.
In my application I'm writing a method that call an other controller method like the following:
def addTagToUser = CorsAction.async { request =>
implicit val userRestFormat = UserFormatters.restFormatter
implicit val inputFormat = InputFormatters.restFormatter
implicit val outputWriter = OutputFormatters.restWriter
//update the tag of a user
def updateTagToUserDB(value: JsValue): Future[Boolean] = {
val holder : WSRequestHolder = WS.url("http://localhost:9000/recommendation/ advise")
val complexHolder = holder.withHeaders("Content-Type" -> "application/json")
complexHolder.post(value).map(response => response.status match {//handle the response
case 200 => true
case _ => false
}
)
}
val jsonData = request.body.asJson //get the json data
jsonData match {
case Some(x) => x.validate[Input] match {
case JsSuccess(input, _) => updateTagToUserDB(x).flatMap(status => status match {
case true => Future{Ok}
case _ => Future{InternalServerError("Error on update the users tags")}
})
case e: JsError => Future{BadRequest("json bad formed")}
}
case None => Future{BadRequest("need a json value")}
}
}
But in this code I've the problem that the url is create static, Is possible to get the absolute uri of a Controller method in Play??
How can I make that??
As mentioned in reverse routing section of Play docs, you can achieve this with the following method call:
routes.Application.advise()
Note that routes exists in controllers so if you are in controllers package you can simply access reverse routes with routes.ControllerName.methodName.
From other parts of the code you need to use the fully qualified package, i.e. controllers.reverse.Application.advise().
If controller method takes a parameter you need to pass the desired argument and get the actual route, for example routes.Application.someMethod(10).
Reverse routing is a powerful asset in Play toolbox which frees you from repeating yourself. It's future proof in a sense that if you change your route, the change will be reflected automatically to the whole application.
Alternative
This approach may not be the best approach.
Redirecting to another controller makes sense, but sending a request to another controller which resides just inside the same web app is overkill and unnecessary. It would be more wise if your web app serves responses to outside not to request a response from itself.
You can easily avoid it by putting the common logic somewhere else and use it from both controllers. According to best practices a good controller is a thin one! By better layering life will be much easier.