URL decoding with unfiltered - scala

I'm working with Unfiltered 0.6.8 (using the Jetty connector) and have encountered an odd behaviour: path segments aren't URL decoded.
The following bit of code is my minimal test case:
import unfiltered.request._
import unfiltered.response._
object Test extends App with unfiltered.filter.Plan {
def intent = {
case reg # Path(Seg(test :: Nil)) =>
println(test)
ResponseString(test)
}
unfiltered.jetty.Http.local(8080).filter(Test).run()
}
Querying http://localhost:8080/some_string yields the expected result: some_string, both on the client and server side.
On the other hand, http://localhost:8080/some%20string yields some%20string on both client and server, rather than the some string I was expecting.
Working around the issue is trivial (java.net.URLDecoder#decode(String, String)), but I'd like to know if:
I'm forgetting something trivial and making a fool of myself.
unfiltered has a kit to deal with the hassle automatically.
if none of the above, is there a particular reason for this behaviour, or should I file a bug report?
As a side note, the unfiltered tag doesn't exist and I do not have enough reputation to create it, which is why I defaulted to scala.

Strange, I'm seeing this behavior too. There is nothing in the Seg object that does any sort of url decoding prior to splitting up the path segments and I don't see anything else in the framework for this either. I ran into a post that detailed a solution using a custom extractor as follows:
object Decode {
import java.net.URLDecoder
import java.nio.charset.Charset
trait Extract {
def charset: Charset
def unapply(raw: String) =
Try(URLDecoder.decode(raw, charset.name())).toOption
}
object utf8 extends Extract {
val charset = Charset.forName("utf8")
}
}
Then you could use it like this:
case reg # Path(Seg(Decode.utf8(test) :: Nil)) =>
println(test)
ResponseString(test)
Or like this if you wanted the entire path decoded:
case reg # Path(Decode.utf8(Seg(test :: Nil))) =>
println(test)
ResponseString(test)
Thankfully the framework is flexible and open to extension like this, so you certainly have options.

Related

(Scala TYPE-SAFETY) take a string as a parameter only if it has .txt extension in it else compile error it?

I'm thinking if its possible to create some sort of data structure that would take a string as a parameter and if that string does not contains .txt or something like that it would give me a compile error?
I can be specific on my problem - I am trying to take a file as a parameter(Source) to a function but I am told that currently I am passing any kind of file and it only should take text files so it is not done correctly. How do i approach this?
-Thanks!! ^^
Yes, this is not a problem. The big question is should you do it though. If your code is a library of sorts then under certain circumstances this might make sense. Check out Uri.uri from the http4s project
In your case the implementation could look something like this:
import scala.language.experimental.macros
import scala.reflect.macros.whitebox.Context
class Macros(val c: Context) {
import c.universe._
def txtFile(fileName: c.Expr[String]): Tree =
fileName.tree match {
case Literal(Constant(s: String)) if s.endsWith(".txt") =>
fileName.tree
case _ =>
c.abort(
c.enclosingPosition,
s"Supplied parameter is not a text file"
)
}
}
object MyObject {
def txtFile(fileName: String): String = macro Macros.txtFile
}
You'd use this like:
val x: String = txtFile("abc.tx") // Fails to compile
val y: String = txtFile("abc.txt") // Compiles
Again, it can be done, but you probably want to go with Option :-)
Assuming that String is coming from somewhere other than the code base (web request, cmd line), you'll need to pass that String to something that accepts Strings. Thus you'll have a point at which you're converting a String to your special filename datatype. So I can't see any way you'll be able to protect against some kind of runtime check.
If you want a runtime check that will avoid an error, you could return an Option, returning None if the String isn't a valid filename
You could use the refined library:
type TxtSuffix = EndsWith[W.`".txt"`.T]
type TxtFile = String Refined TxtSuffix
In order to construct a TxtFile, you need to give it a string that ends with ".txt":
val file: TxtFile = refineMV[TxtSuffix]("some.pdf") // compiler error
error: Predicate failed: "some.pdf".endsWith(".txt")

Akka Http get query returning empty response

I'm fumbling my way through akka-http; I've got a single route that compiles:
val route = get {
pathPrefix("compass") {
path("route") {
parameters(('srclat.as[Double], 'srclon.as[Double],
'destlat.as[Double], 'destlon.as[Double])) {
(srclat, srclon, destlat, destlon) =>
complete(getRoute((LatLong(srclat, srclon),
LatLong(destlat, destlon))))
}
}
}
}
And I've verified that the parameters are being extracted correctly. When I call the method (with valid lat / longs), I'm expecting to receive an array of coordinates representing a (physical) route, but instead I receive a route object with an empty list of coordinates. Here's the signature of the method being run by complete:
// the Route here is a domain object, not an akka-http Route
def getRoute(coordinates: (LatLong, LatLong)):
Future[Option[Route]] = ???
And starting the server itself looks something like this:
val bindingFuture = Http().bindAndHandle(service.route, "0.0.0.0",
port)
I'm using akka and akka-streams 2.5.4 and akka-http 10.0.9, with Circe support from hseeberger (version 1.18.0). If anyone knows what I'm doing wrong here, please let me know...any help would be appreciated!
instead I receive a route object with an empty list of coordinates.
I think the problem is not in the code shown, but somewhere inside the getRoute function.
I have a hunch that you might be making changes to an immutable case class, and returning a previous copy, rather than the updated version?
e.g. code like the following would give the bug you describe:
import scala.concurrent.Future
import scala.concurrent.ExecutionContext.Implicits.global
trait LatLong
case class Route(steps: List[String]) // for example
def getRoute(coordinates: (LatLong, LatLong)): Future[Option[Route]] = {
val myRoute = new Route(Nil)
val steps = List("one", "two", "three")
myRoute.copy(steps = steps) // BUG HERE, new copy discarded
Future(Some(myRoute))
}
If that doesn't explain things, please could you show more of the getRoute function?
A quick test to narrow things down might be to change getRoute temporarily to return a hardcoded non-empty Route, and check that it comes back OK over HTTP.
This was completely my fault; due to a cut-and-paste error, the source coordinates were being sent in as both source and destination, so the empty Route object was legitimate. Thanks to those who took their time looking into my screw-up!

Play Scala No Json deserializer found for type (String, String). Try to implement an implicit Reads or Format for this type

These Json serializers in Play with Scala are driving me nuts.
I have read dozens of posts and the tutorials and the documentation. Tried four different ways of implementing Reads / Writes / Format overrides and all to no avail.
So I backed off the custom type and decided to go uber simple:
def suggest = Action(parse.json) {
request =>
request.body.validate[(String, String)].map {
case (suggestion, categories) => Ok("You suggested " + suggestion + " for categories " + categories)
}.recoverTotal {
e => BadRequest(JsError.toFlatJson(e))
}
}
And the error comes back as noted in the subject.
Do I really need to provide a custom Reads / Writes / Format implementation for such a basic body?
A sample input body could be:
{"suggestion":"add generics", "categories":"request;language;updates"}
What simple thing am I missing?
Play! gives you a LOT of ways to work with Json. From the looks of your code, you're going down the Tuple road. Which essentially lets you convert Json into a Tuple. You're also using 'reads' which has the downside that you don't get precise error reporting (ie: if invalid json was passed you would know its invalid... but you wouldn't necessarily know why it was invalid). If you wanted more error handling then you need to start using the 'validate' method (details here: http://www.playframework.com/documentation/2.1.1/ScalaJsonCombinators).
Another way you could go is to map Json to case classes doing something like:
import play.api.libs.json._
case class MyClass(
suggestion: String,
categories: String
)
object MyClass {
implicit val readsMyClass: Reads[MyClass] = new Reads[MyClass] {
def reads(json: JsValue): JsResult[MyClass] = {
for {
suggestion <- (json \ "suggestion").validate[String]
categories <- (json \ "categories").validate[String]
} yield MyClass(json,categories)
}
}
}
This seems like a lot of code so Play 2.1 introduced Json 'Inception' which I have yet to try (http://www.playframework.com/documentation/2.1.1/ScalaJsonInception).
Finally, if you want the Json validation but don't necessary need to marshall/unmarshall case classes, then you can try the 'coast-to-coast' method. This will keep all your json as JsObject types but still give you validation. Sample code here: https://github.com/mandubian/play2-json-demo/blob/master/json-coast-to-coast/app/controllers/Application.scala
Hope this helps.
So I added this:
implicit val rds = (
(__ \ 'suggestion).read[String] and
(__ \ 'categories).read[String]
) tupled
And that seems to work.
Curious, though, is this really the best way to do this? It seems like a LOT of code if you have many many types to serialize / deserialize.

Can't convert unicode symbols to cyrillic

I have a bunch of documents persisted in Apache Lucene with some names in russian, and when I'm trying to print them out it looks like this "\u0410\u0441\u043f\u0430\u0440", but not in cyrillic symbols. The project is in Scala. I've tried to fix this with Apache Commons unescapeJava method, but it didn't help. Are there any other options?
Updated:
Project is writen with Spray framework and returns json like this.
{
"id" : 0,
"name" : "\u0410\u0441\u043f\u0430\u0440"
}
I'm going to try to infer exactly what you are doing.
You are using Spray, so I gather that you are using its json library "spray-json"
So I suppose that you have some instance of spray.json.JsObject, and that what you posted in your question is what you get as the output when printing this instance.
Your json object is correct, the value of the name field has no embeded escaping, it is actually the conversion to string that escapes some unicode characters.
See the definition of printString here:
https://github.com/spray/spray-json/blob/master/src/main/scala/spray/json/JsonPrinter.scala
I will also assume that when you tried to use unescapeJava, you applied it on the value of the name field, creating a new spray.json.JsObject instance that you then printed as before. Given that your json object does not actually have any escaping, this did absolutly nothing, and then when printing it the printer does the escaping as before, and you're back to square one.
As a side note, it's worth mentioning that the json spec does not mandate how characters are encoded: they can either be stored as their literal value, or as a unicode escape. By example the string "abc" could be described as just "abc", or as "\u0061\u0062\u0063". Either form is correct. It just happens that the author of spray-json decided to use the latter form for all non-ascii characters.
So now you ask, what can I do to work around this? You could ask the spray-json author to add an option that let's you specify that you don't want any unicode escaping.
But I imagine that you want a solution right now.
The simplest thing to do is to just convert your object to a string (via JsValue.toString or JsValue.compactPrint or JsValue.prettyPrint), and then pass the result to unescapeJava. At least this will give you back your cyrillic original characters.
But this is a bit gross, and actually quite dangerous as some characters are not safe to unescape inside a string literal. By example: \n will be unescaped to an actual return, and \u0022 will be unescaped to ". You can easily see how it will break your json document.
But at the very least it will allow to confirm my theory (remember that I have been making assumptions about what exactly you are doing).
Now for a proper fix: you could simply extend JsonPrinter and override its printString method to remove the unicode escapting. Something like this (untested):
trait NoUnicodeEscJsonPrinter extends JsonPrinter {
override protected def printString(s: String, sb: StringBuilder) {
#tailrec
def printEscaped(s: String, ix: Int) {
if (ix < s.length) {
s.charAt(ix) match {
case '"' => sb.append("\\\"")
case '\\' => sb.append("\\\\")
case x if 0x20 <= x && x < 0x7F => sb.append(x)
case '\b' => sb.append("\\b")
case '\f' => sb.append("\\f")
case '\n' => sb.append("\\n")
case '\r' => sb.append("\\r")
case '\t' => sb.append("\\t")
case x => sb.append(x)
}
printEscaped(s, ix + 1)
}
}
sb.append('"')
printEscaped(s, 0)
sb.append('"')
}
}
trait NoUnicodeEscPrettyPrinter extends PrettyPrinter with NoUnicodeEscJsonPrinter
object NoUnicodeEscPrettyPrinter extends NoUnicodeEscPrettyPrinter
trait NoUnicodeEscCompactPrinter extends CompactPrinter with NoUnicodeEscJsonPrinter
object NoUnicodeEscCompactPrinter extends NoUnicodeEscCompactPrinter
Then you can do:
val json: JsValue = ...
val jsonString: String = NoUnicodeEscPrettyPrinter( json )
jsonString will contain your json document in pretty-print format and without any unicde escaping.
This problem appears to be corrected in spray-json 1.3.2: https://github.com/spray/spray-json/issues/46
I ran into a similar problem with Arabic characters using Akka HTTP 1.0, which depends on 1.3.1. By upgrading to 1.3.2, my problem was resolved.

How to convert a scala.xml.Elem into a JsExp in Lift?

My latest problem is one that I already have a solution for, it just
feels like there should be a better way.
The problem:
I want to send a PartialUpdate to a comet service, and I need to XML
escape the string, so that when it is used on the client it gets the
correct results. I currently have:
override def lowPriority = {
case v: List[TaskOwner] => {
partialUpdate(
taskOwners.foldLeft(JsCrVar("table", Call("$", Str("table#userTable"))) &
Call("table.dataTable().fnClearTable"))((r, c) => {
r & Call("table.dataTable().fnAddData",
JsArray(Str(Text(c.name).toString),
Str(Text(c.initials).toString),
Str(makeDeleteButton(c).toString)),
Num(0))
}) & Call("table.dataTable().fnDraw"))
}
}
And this works fine, however the Str(Text(c.name).toString) feels
quite wordy to me. Now, I can, of course, create a pair of implicit
conversion functions for this, but it seems like this should have
already been done somewhere, I just don't know where. And so, in the
spirit of reducing the code that I have written, I ask if anyone knows
a better way to do this, or if the implicit conversion already exist
somewhere?
I have seen reference to a solution here. However the code is summarized as:
def xmlToJson(xml: Elem): JsExp = {
// code to map XML responses to JSON responses. Handles tricky things like always returning
// js arrays for some fields even if only 1 element appears in the XML
}
A possibly better way of escaping the names is, instead of:
JsArray(Str(Text(c.name).toString),
Str(Text(c.initials).toString),
Str(makeDeleteButton(c).toString))
to use
JsArray(Str(c.name.asHtml.toString),
Str(c.initials.asHtml.toString),
Str(makeDeleteButton(c).toString))
This can be further reduced by using an implicit within the class like:
implicit def elemToJsExp(elem: NodeSeq): JsExp = Str(elem.toString)
…
JsArray(c.name.asHtml,
c.initials.asHtml,
makeDeleteButton (c))
I don't know what Str does, but maybe you mean Str(xml.Utility.escape(c.name))?
Well, how about:
def JsStrArray(strings: String*) = JsArray(strings map xml.Utility.escape map Str : _*)
And then just use
JsStrArray(c.name, c.initials, makeDeleteButton(c).toString)
Mmmmm. It might incorrectly escape the result of makeDeleteButton. Anyway, you can play with it and see what looks good.