Upgrading Http4s to 0.18: StaticFile and fallthrough/pass - scala

In Http4s 0.16.6a, I had the following service.
import org.http4s.server.staticcontent._
import org.http4s._
object StaticFiles {
val basepath = ...
def apply(): HttpService = Service.lift(request => {
val file = basepath + request.uri.path
StaticFile.fromString(file, Some(request)).fold {
Pass.now // aka fallthrough
} {
NoopCacheStrategy.cache(request.pathInfo, _)
}
})
}
It takes the path from the url and tries to work out if a static file can be served. So a GET request to /index.html would try and load it with fromFile and if it can't be found, fallthrough or "Pass". When composed with other services using ||, it meant the total function (from lift) would be treated a bit like a partial function (from apply).
I can't seem to convert this over to Http4s 0.18.x.
The Http4s docs suggest the following:
import cats.effect._
import org.http4s._
import org.http4s.dsl.io._
import java.io.File
val service = HttpService[IO] {
case request # GET -> Root / "index.html" =>
StaticFile.fromFile(new File("relative/path/to/index.html"), Some(request))
.getOrElseF(NotFound()) // In case the file doesn't exist
}
Which is the basic form of what I'm trying to do, only I'd like to generify it a little and not create a partial function for every file I want to serve. i.e. avoid this:
case request # GET -> Root / "index.html" => ???
case request # GET -> Root / "fileA.html" => ???
case request # GET -> Root / "fileB.html" => ???
So, my questions are:
Is there the concept of Pass and passthough when using lift in 0.18?
How to I use the NooopCacheStretegy with lift?
Ultimately, how to I convert the code above to 0.18?
My endeavours so far have lead to this abomination (which obvs doesn't compile):
def apply(): HttpService[IO] = HttpService.lift(request => {
val target = basepath + request.uri.path
StaticFile.fromString[IO](target, Some(request)).fold {
// passthrough
IO.pure(???)
} {
// process request
response => NoopCacheStrategy[IO].cache(request.pathInfo, response).unsafeRunSync()
}
})
Note that I'm trying to use HttpService.lift not OptionT.liftF (as recommended). Mostly because I have no clue how to!

So as far as I can tell, the concept of Pass has been replaced by OptionT in 0.18.x, with None playing the role of Pass. However, you don't have access to the OptionT with that overload. Instead, the assumption is that since you're passing a partial function to HttpService, the requests the function is defined on are precisely the ones you want this service to provide a response for.
You could try making it work with OptionT.lift, but I wouldn't recommend it either! Instead, I'd make a partial function that's only defined on arguments when your static files exists. The way http4s allows you to define the criteria needed to hit an endpoint through pattern-matching on requests is extremely powerful, and you're completely ignoring that option in both of your solutions.
As far as NoopCacheStrategy is concerned, I'm guessing the problem you ran into was that the return type of StaticFile.fromX is now IO[Response[IO]] and NoopCacheStrategy takes a Response[IO]. This is easily handled via flatMap.
Which is to say that this is what I came up with to translate your code to 0.18.x:
import java.nio.file.{Files, Paths}
import cats.effect.IO
import org.http4s.{HttpService, StaticFile}
import org.http4s.dsl.io._
import org.http4s.server.staticcontent.NoopCacheStrategy
val service = HttpService[IO] {
case request # _ -> _ / file if Files.exists(Paths.get(file)) =>
StaticFile
.fromString(file, Some(request))
.getOrElseF(NotFound())
.flatMap(resp => NoopCacheStrategy[IO].cache(request.pathInfo, resp))
}
A slight annoyance is that we're actually handling the case where no such file exists twice, once in the if clause of the case statement and once with getOrElseF. In practice, that NotFound should never be reached. I figure one can live with this.
And as an example of what I mean by the power of http4s' pattern matching on requests, by adjusting the case statement it's very easy to make sure this will only...
match on GET requests: case request # GET -> _ / file if ...
match top-level files, nothing in subdirectories: case request # _ -> Root / file if ...
match HTML files: case request # _ -> _ / file ~ html if Files.exists(Paths.get(s"$file.html"))
You can even write your own custom extractor that checks whether you can serve a given file name to end up with something like case request # _ -> _ / ValidStaticFile(file). That way you don't have to cram all your logic into the case statement.

I can't seem to format as a comment so posted as a new answer...
How about this?
def apply(): HttpService[IO] = Kleisli.apply(request => {
val basepath = ...
val target = location + request.uri.path
StaticFile.fromString[IO](target, Some(request))
})

Related

Converting an `Option[A]` to an Ok() or NotFound() inside an Http4s API

I've got an API that looks like this:
object Comics {
...
def impl[F[_]: Applicative]: Comics[F] = new Comics[F] {
def getAuthor(slug: Authors.Slug): F[Option[Authors.Author]] =
...
and a routing that looks like this:
object Routes {
def comicsRoutes[F[_]: Sync](comics: Comics[F]): HttpRoutes[F] = {
val dsl = new Http4sDsl[F] {}
import dsl._
HttpRoutes.of[F] {
case GET -> Root / "comics" / authorSlug =>
comics
.getAuthor(Authors.Slug(authorSlug))
.flatMap {
case Some(author) => Ok(author)
case None => NotFound()
}
So when there is a None, it gets converted to a 404. Since there are several routes, the .flatMap { ... } gets duplicated.
Question: How do I move this into a separate .orNotFound helper function specific to my project?
My attempt:
To make things simple for me (and avoid parameterisation over F initially), I've tried to define this inside comicsRoutes:
def comicsRoutes[F[_]: Sync](comics: Comics[F]): HttpRoutes[F] = {
val dsl = new Http4sDsl[F] {}
import dsl._
def orNotFound[A](opt: Option[A]): ???[A] =
opt match {
case Some(value) => Ok(value)
case None => NotFound()
}
HttpRoutes.of[F] {
case GET -> Root / "comics" / authorSlug =>
comics
.getAuthor(Authors.Slug(authorSlug))
.flatMap(orNotFound)
But what's ??? here? It doesn't seem to be Response or Status. Also, the .flatMap { ... } was made under import dsl._, but I'd like to move this further out. What would a good place be? Does it go into the routes file, or do I put it in a separate ExtendedSomething extension file? (I expect that ??? and Something might be related, but I'm a little confused as to what the missing types are.)
(Equally importantly, how do I find out what ??? is here? I was hoping ??? at the type level might give me a "typed hole", and VSCode's hover function provides very sporadic documentation value.)
The type returned by Http4sDsl[F] for your actions is F[Response[F]].
It has to be wrapped with F because your are using .flatMap on F. Response is parametrized with F because it will produce the result returned to caller using F.
To find that out you can use IntelliJ and then generate the annotation by IDE (Alt+Enter and then "Add type annotation to value definition"). You can also:
preview implicits to check that Ok object imported from Statuses trait is provided extension methods with http4sOkSyntax implicit conversion (Ctrl+Alt+Shift+Plus sign, you can press it a few times to expand implicits more, and Ctrl+Alt+Shift+Minut to hide them again)
find http4sOkSyntax by pressing Shift twice to open find window, and then pressing it twice again to include non-project symbols,
from there navigate with Ctrl+B through OkOps to EntityResponseGenerator class which is providing you the
functionality you used (in apply) returning F[Resposne[F]].
So if you want to move things around/extract them, pay attention to what implicits are required to instantiate the DSL and extension methods.
(Shortcuts differ between Mac OS - which sometime use Cmd instead of Ctrl - and non-Mac OS systems so just check them in documentation if you have an issue).
Thanks to Mateusz, I learned that ??? should be F[Response[F]].
To make this helper function work fully, two more type-related problems occurred:
Since value: A is polymorphic, Http4s expects an implicit EntityEncoder[F, A] in order to serialize an arbitrary value. (This was not a problem with the original { case ... } match, since the type was concrete and not polymorphic.
Adding this implicit annotation was, for some reason, not enough. Doing .flatMap(orNotFound) fails type inference. Doing .flatMap(orNotFound[Authors.Slug]) fixes this.
(Thanks to keynmol for pointing out the other two.)
Having all three changes, this results in:
def comicsRoutes[F[_]: Sync](comics: Comics[F]): HttpRoutes[F] = {
val dsl = new Http4sDsl[F] {}
import dsl._
def orNotFound[A](opt: Option[A])(implicit ee: EntityEncoder[F, A]): F[Response[F]] =
opt match {
case Some(value) => Ok(value)
case None => NotFound()
}
HttpRoutes.of[F] {
case GET -> Root / "comics" / authorSlug =>
comics
.getAuthor(Authors.Slug(authorSlug))
.flatMap(orNotFound[Authors.Author])
...

scala ZIO foreachPar

I'm new to parallel programming and ZIO, i'm trying to get data from an API, by parallel requests.
import sttp.client._
import zio.{Task, ZIO}
ZIO.foreach(files) { file =>
getData(file)
Task(file.getName)
}
def getData(file: File) = {
val data: String = readData(file)
val request = basicRequest.body(data).post(uri"$url")
.headers(content -> "text", char -> "utf-8")
.response(asString)
implicit val backend: SttpBackend[Identity, Nothing, NothingT] = HttpURLConnectionBackend()
request.send().body
resquest.Response match {
case Success(value) => {
val src = new PrintWriter(new File(filename))
src.write(value.toString)
src.close()
}
case Failure(exception) => log error
}
when i execute the program sequentially, it work as expected,
if i tried to run parallel, by changing ZIO.foreach to ZIO.foreachPar.
The program is terminating prematurely, i get that, i'm missing something basic here,
any help is appreciated to help me figure out the issue.
Generally speaking I wouldn't recommend mixing synchronous blocking code as you have with asynchronous non-blocking code which is the primary role of ZIO. There are some great talks out there on how to effectively use ZIO with the "world" so to speak.
There are two key points I would make, one ZIO lets you manage resources effectively by attaching allocation and finalization steps and two, "effects" we could say are "things which actually interact with the world" should be wrapped in the tightest scope possible*.
So lets go through this example a bit, first of all, I would not suggest using the default Identity backed backend with ZIO, I would recommend using the AsyncHttpClientZioBackend instead.
import sttp.client._
import zio.{Task, ZIO}
import zio.blocking.effectBlocking
import sttp.client.asynchttpclient.zio.AsyncHttpClientZioBackend
// Extract the common elements of the request
val baseRequest = basicRequest.post(uri"$url")
.headers(content -> "text", char -> "utf-8")
.response(asString)
// Produces a writer which is wrapped in a `Managed` allowing it to be properly
// closed after being used
def managedWriter(filename: String): Managed[IOException, PrintWriter] =
ZManaged.fromAutoCloseable(UIO(new PrintWriter(new File(filename))))
// This returns an effect which produces an `SttpBackend`, thus we flatMap over it
// to extract the backend.
val program = AsyncHttpClientZioBackend().flatMap { implicit backend =>
ZIO.foreachPar(files) { file =>
for {
// Wrap the synchronous reading of data in a `Task`, but which allows runs this effect on a "blocking" threadpool instead of blocking the main one.
data <- effectBlocking(readData(file))
// `send` will return a `Task` because it is using the implicit backend in scope
resp <- baseRequest.body(data).send()
// Build the managed writer, then "use" it to produce an effect, at the end of `use` it will automatically close the writer.
_ <- managedWriter("").use(w => Task(w.write(resp.body.toString)))
} yield ()
}
}
At this point you will just have the program which you will need to run using one of the unsafe methods or if you are using a zio.App through the main method.
* Not always possible or convenient, but it is useful because it prevents resource hogging by yielding tasks back to the runtime for scheduling.
When you use a purely functional IO library like ZIO, you must not call any side-effecting functions (like getData) except when calling factory methods like Task.effect or Task.apply.
ZIO.foreach(files) { file =>
Task {
getData(file)
file.getName
}
}

Akka-HTTP compilation error using custom requireParam() directive

I developed custom generic directive, which will provide param of given type, if it exists, or else reject with my custom exception.
import akka.http.scaladsl.common.NameReceptacle
import akka.http.scaladsl.server.Directives._
import akka.http.scaladsl.server.directives.ParameterDirectives.ParamDefAux
import akka.http.scaladsl.server.{Directive1, Route}
class MyCustomException(msg: String) extends Exception(msg)
def requireParam[T](name: NameReceptacle[T])
(implicit pdef: ParamDefAux[NameReceptacle[T], Directive1[T]]): Directive1[T] =
parameter(name).recover { _ =>
throw new MyCustomException(s"${name.name} is missed!")
}
It works ok, if I want to create route, using two parameters, for example:
val negSumParams: Route =
(requireParam("param1".as[Int]) & requireParam("param2".as[Int])) {
(param1, param2) =>
complete((-param1-param2).toString)
}
But if I try to use exactly one parameter, this doesn't compile:
val negParamCompilationFail: Route =
requireParam("param".as[Int]) {
param => // scalac complains about missing type param here
complete((-param).toString)
}
If I use it with pass directive, it works:
val negParamWithPass: Route =
(pass & requireParam("param".as[Int])) { // this pass usage looks hacky
param =>
complete((-param).toString)
}
If I write requireParam() return type explicitly, it works too:
val negParamWithExplicitType: Route =
(requireParam("param".as[Int]): Directive1[Int]) { // DRY violation
param =>
complete((-param).toString)
}
Why do I need these tricks? Why can't it work just with requireParam("param".as[Int])?
Scala version 2.12.1, Akka-HTTP 10.0.10.
This error happens due to the Directive companion object apply method. IT allows to create a Route from a function with parameter (T => Route) => Route:
object Directive {
/**
* Constructs a directive from a function literal.
*/
def apply[T: Tuple](f: (T ⇒ Route) ⇒ Route): Directive[T] =
new Directive[T] { def tapply(inner: T ⇒ Route) = f(inner) }
}
But the T parameter must be a tuple. In your case, the compiler can not build the Route. Your requireParam("param".as[Int]) returns a Directive1[Int] so the apply method doesn´t work because Int is not a Tuple.
To make this work you shoul use tapply method directly:
(requireParam("param1".as[Int])).tapply((param1) =>
complete((-param1._1).toString))
and
val negSumParams2: Route =
(requireParam("param1".as[Int]) & requireParam("param2".as[Int])).tapply {
case (param1, param2) =>
complete((-param1-param2).toString)
}
So it seems that every Directive tries to convert its param to TupleX. For example:
path("order" / IntNumber) returns a Directive[Tuple1[Int]] instead of Directive1[Int]. In your case requireParam("param1".as[Int]) returns Directive1[Int]
Maybe there is a better solution and to avoid tapply
Johannes from Lightbend answered to this question here: https://groups.google.com/forum/#!topic/akka-user/NmQvcrz5sJg Answer is:
You discovered one of the reasons for the magnet pattern. If you use
requireParam("param".as[Int]) { abc => ... } then the { abc => }
block is mistaken as the implicit argument of requireParam. So,
either you are ok with that and require users to use extra parentheses
((requireParam("param".as[Int])) { abc => ... }) or you use the
magnet pattern at this level as well. You can read about the magnet
pattern on the old spray blog
(http://spray.io/blog/2012-12-13-the-magnet-pattern/) or just look
into akka-http sources.
A better way to implement the feature would be just to use the
existing implementation ;) and install a custom rejection handler that
produces whatever output you would like. See
https://doc.akka.io/docs/akka-http/10.0.10/scala/http/routing-dsl/rejections.html#customizing-rejection-handling
for how to do that.

Show static content with parameters with Spray

I'm trying to serve an HTML page using Spray. It's fairly easy using getFromResource and getFromResourceDirectory, but I additionally need to pass some query parameters so that some Javascript on the page knows what to do. Is that possible ? All my prior attempts consisted in this kind of things
val route = path("show-repo") { serveResourceWithParams(SHOW_REPO) } ~ getFromResourceDirectory("web")
def serveWithParams(page: String, params: (String, String)*) = {
val url = page + (if (params.isEmpty) "" else "?") + params.map { case (k, v) => s"$k=$v" }.mkString("&")
getFromResource(url)
}
but I now realize it was a bit naive
I would create a new case class with members of type Array[Byte] and List[(String, String)].
case class FileWithParams(file: Array[Byte], params: List[(String, String)])
If you know the file won't be huge, you can read it in one shot with
FileUtils.readAllBytes
Assuming you're using(which you need to run Spray) https://github.com/sirthias/parboiled/blob/master/parboiled-core/src/main/java/org/parboiled/common/FileUtils.java
Then manually read the file contents, create the case class, and marshall it as you would normally do.
Of course, my preferred way would be to split this up into two async requests and wait for both to complete. You'll probably be waiting on the file route to complete anyway.

Obtain the URI of a Controller method call

I'm writing a Play 2.3.2 application in Scala.
In my application I'm writing a method that call an other controller method like the following:
def addTagToUser = CorsAction.async { request =>
implicit val userRestFormat = UserFormatters.restFormatter
implicit val inputFormat = InputFormatters.restFormatter
implicit val outputWriter = OutputFormatters.restWriter
//update the tag of a user
def updateTagToUserDB(value: JsValue): Future[Boolean] = {
val holder : WSRequestHolder = WS.url("http://localhost:9000/recommendation/ advise")
val complexHolder = holder.withHeaders("Content-Type" -> "application/json")
complexHolder.post(value).map(response => response.status match {//handle the response
case 200 => true
case _ => false
}
)
}
val jsonData = request.body.asJson //get the json data
jsonData match {
case Some(x) => x.validate[Input] match {
case JsSuccess(input, _) => updateTagToUserDB(x).flatMap(status => status match {
case true => Future{Ok}
case _ => Future{InternalServerError("Error on update the users tags")}
})
case e: JsError => Future{BadRequest("json bad formed")}
}
case None => Future{BadRequest("need a json value")}
}
}
But in this code I've the problem that the url is create static, Is possible to get the absolute uri of a Controller method in Play??
How can I make that??
As mentioned in reverse routing section of Play docs, you can achieve this with the following method call:
routes.Application.advise()
Note that routes exists in controllers so if you are in controllers package you can simply access reverse routes with routes.ControllerName.methodName.
From other parts of the code you need to use the fully qualified package, i.e. controllers.reverse.Application.advise().
If controller method takes a parameter you need to pass the desired argument and get the actual route, for example routes.Application.someMethod(10).
Reverse routing is a powerful asset in Play toolbox which frees you from repeating yourself. It's future proof in a sense that if you change your route, the change will be reflected automatically to the whole application.
Alternative
This approach may not be the best approach.
Redirecting to another controller makes sense, but sending a request to another controller which resides just inside the same web app is overkill and unnecessary. It would be more wise if your web app serves responses to outside not to request a response from itself.
You can easily avoid it by putting the common logic somewhere else and use it from both controllers. According to best practices a good controller is a thin one! By better layering life will be much easier.