In a PUT request from a client device, I would like the application/json body to look like {"content": []} in order to clear a list I'm maintaining for the client. However, in a Play Form such key-value pair where the value is an empty array disappears after form parsing.
You can see that in regular (non-Form) Play JSON parsing, the empty array is maintained as expected.
[info] Starting scala interpreter...
Welcome to Scala 2.11.12 (OpenJDK 64-Bit Server VM, Java 1.8.0_212).
Type in expressions for evaluation. Or try :help.
scala> import play.api.libs.json._
import play.api.libs.json._
scala> val json = Json.parse("""{"content": []}""")
json: play.api.libs.json.JsValue = {"content":[]}
However, in Play's form parsing, the behavior is different.
private[data] object FormUtils {
def fromJson(prefix: String = "", js: JsValue): Map[String, String] = js match {
case JsObject(fields) =>
val prefix2 = Option(prefix).filterNot(_.isEmpty).map(_ + ".").getOrElse("")
fields.iterator
.map { case (key, value) => fromJson(prefix2 + key, value) }
.foldLeft(Map.empty[String, String])(_ ++ _)
case JsArray(values) =>
values.zipWithIndex.iterator
.map { case (value, i) => fromJson(s"$prefix[$i]", value) }
.foldLeft(Map.empty[String, String])(_ ++ _)
case JsNull => Map.empty
case JsUndefined() => Map.empty
case JsBoolean(value) => Map(prefix -> value.toString)
case JsNumber(value) => Map(prefix -> value.toString)
case JsString(value) => Map(prefix -> value.toString)
}
}
https://github.com/playframework/playframework/blob/826c76ee967d8ec35b76b9a8b594bfaa676a9510/core/play/src/main/scala/play/api/data/Form.scala#L397
The general idea of the function is a recursive traversal of the incoming JSValue objects and arrays where null, undefined, booleans, numbers, strings, and empty objects and arrays serve as base cases.
In the JsArray case, if values is empty then it returns Map.empty[String, String]. As a result, both [] and {"content": []} will be parsed to Map.empty[String, String]. For a form, the [] case seems reasonable, but {"content": []} is a legitimate key-value pair that should be kept in the form. Note that the same issue occurs with an empty JsObject.
Does this seem like a reasonable issue? If so, any workarounds that you can think of? I understand that it could be difficult for Play to change this since users of the framework might already be depending on this behavior.
I think that is correct behavior, since as far as forms are concerned there is no concept of an Array.
A form simply consists of a set of key-value-pairs and an array is just flattened to fit into this scheme, a la path.to.array[0] = value.
The form, however, can then be mapped to a model which actually has a sequence of values.
Say we have an array of one dimensional javascript objects contained in a file Array.json for which the key schema isn't known, that is the keys aren't known until the file is read.
Then we wish to output a CSV file with a header or first entry which is a comma delimited set of keys from all of the objects.
Each next line of the file should contain the comma separated values which correspond to each key from the file.
Array.json
[
abc:123,
xy:"yz",
s12:13,
],
...
[
abc:1
s:133,
]
A valid output:
abc,xy,s12,s
123,yz,13,
1,,,133
I'm teaching myself 'functional style' programming but I'm thinking that this problem doesn't lend itself well to a functional solution.
I believe that this problem requires some state to be kept for the output header and that subsequently each line depends on that header.
I'm looking to solve the problem in a single pass. My goals are efficiency for a large data set, minimal traversals, and if possible, parallelizability. If this isn't possible then can you give a proof or reasoning to explain why?
EDIT: Is there a way to solve the problem like this functionally?:
Say you pass through the array once, in some particular order. Then
from the start the header set looks like abc,xy,s12 for the first
object. With CSV entry 123,yz,13 . Then on the next object we add an
additional key to the header set so abc,xy,s12,s would be the header
and the CSV entry would be 1,,,133 . In the end we wouldn't need to
pass through the data set a second time. We could just append extra
commas to the result set. This is one way we could approach a single
pass....
Are there functional tools ( functions ) designed to solve problems like this, and what should I be considering? [ By functional tools I mean Monads,FlatMap, Filters, etc. ] . Alternatively, should I be considering things like Futures ?
Currently I've been trying to approach this using Java8, but am open to solutions from Scala, etc. Ideally I would be able to determine if Java8s' functional approach can solve the problem since that's the language I'm currently working in.
Since the csv output will change with every new line of input, you must hold that in memory before writing it out. If you consider creating an output text format from an internal representation of a csv file another "pass" over the data (the internal representation of the csv is practically a Map[String,List[String]] which you must traverse to convert it to text) then it's not possible to do this in a single pass.
If, however, this is acceptable, then you can use a Stream to read a single item from your json file, merge that into the csv file, and do this until the stream is empty.
Assuming, that the internal representation of the csv file is
trait CsvFile {
def merge(line: Map[String, String]): CsvFile
}
And you can represent a single item as
trait Item {
def asMap: Map[String, String]
}
You can implement it using foldLeft:
def toCsv(items: Stream[Item]): CsvFile =
items.foldLeft(CsvFile(Map()))((csv, item) => csv.merge(item.asMap))
or use recursion to get the same result
#tailrec def toCsv(items: Stream[Item], prevCsv: CsvFile): CsvFile =
items match {
case Stream.Empty => prevCsv
case item #:: rest =>
val newCsv = prevCsv.merge(item.asMap)
toCsv(rest, newCsv)
}
Note: Of course you don't have to create types for CsvFile or Item, you can use Map[String,List[String]] and Map[String,String] respectively
UPDATE:
As more detail was requested for the CsvFile trait/class, here's an example implementation:
case class CsvFile(lines: Map[String, List[String]], rowCount: Int = 0) {
def merge(line: Map[String, String]): CsvFile = {
val orig = lines.withDefaultValue(List.fill(rowCount)(""))
val current = line.withDefaultValue("")
val newLines = (lines.keySet ++ line.keySet) map {
k => (k, orig(k) :+ current(k))
}
CsvFile(newLines.toMap, rowCount + 1)
}
}
This could be one approach:
val arr = Array(Map("abc" -> 123, "xy" -> "yz", "s12" -> 13), Map("abc" -> 1, "s" -> 133))
val keys = arr.flatMap(_.keys).distinct // get the distinct keys for header
arr.map(x => keys.map(y => x.getOrElse(y,""))) // get an array of rows
Its completely OK to have state in functional programming. But having mutable state or mutating state is not allowed in functional programming.
Functional programming advocates creating new changed state instead of mutating the state in place.
So, its Ok to read and access state created in the program until and unless you are mutating or side effecting.
Coming to the point.
val list = List(List("abc" -> "123", "xy" -> "yz"), List("abc" -> "1"))
list.map { inner => inner.map { case (k, v) => k}}.flatten
list.map { inner => inner.map { case (k, v) => v}}.flatten
REPL
scala> val list = List(List("abc" -> "123", "xy" -> "yz"), List("abc" -> "1"))
list: List[List[(String, String)]] = List(List((abc,123), (xy,yz)), List((abc,1)))
scala> list.map { inner => inner.map { case (k, v) => k}}.flatten
res1: List[String] = List(abc, xy, abc)
scala> list.map { inner => inner.map { case (k, v) => v}}.flatten
res2: List[String] = List(123, yz, 1)
or use flatMap instead of map and flatten
val list = List(List("abc" -> "123", "xy" -> "yz"), List("abc" -> "1"))
list.flatMap { inner => inner.map { case (k, v) => k}}
list.flatMap { inner => inner.map { case (k, v) => v}}
In functional programming, mutable state is not allowed. But immutable states/values are fine.
Assuming that you have read your json file in to a value input:List[Map[String,String]], the codes below will solve your problem:
val input = List(Map("abc"->"123", "xy"->"yz" , "s12"->"13"), Map("abc"->"1", "s"->"33"))
val keys = input.map(_.keys).flatten.toSet
val keyvalues = input.map(kvs => keys.map(k => (k->kvs.getOrElse(k,""))).toMap)
val values = keyvalues.map(_.values)
val result = keys.mkString(",") + "\n" + values.map(_.mkString(",")).mkString("\n")
Is there a way to loop for each member variable of a class? It's trivial with Lists and arrays but I have to construct a case class with each json field mapping to a member variable to use the Play framework's Reads/validator API
Did you mean something like this:
case class SomeClass(a: Int, b: Int)
SomeClass(1,2).productIterator.foreach{ a => println(a)}
this will give you an output of: 1 2
Or if you are trying to construct an object from json. You can define reads in your case class which deserialises json to your object :
override def reads(json: JsValue): JsResult[SomeClass] = JsSuccess(SomeClass(
(json \ "a").as[Int],
(json \ "b").as[Int]
)
)
then to use the deserialisation:
val json = Json.obj() //Your json
json.validate[SomeClass]
json.fold(
errors => {
Json.obj("status" -> "Not OK", "message" -> JsError.toJson(errors))
},
preset => {
Json.obj("status" -> "OK")
}
)
If you want compare Json and get difference, may be better use JsObject methods?
For example fieldSet return all fields as a set. You can use diff on previous and current field set to get changed fields. This is fast solution and no any specific classes.
I have nested url parameters being passed to an endpoint, and I need these represented in as a JsValue. My initial assumption was that Play would parse them in a way similar to Rails, however parameters seem to only be split by & and =. Example:
Query params: ?test[testkey]=testvalue&test[newkey]=newvalue
Actual:
Map(
"test[testkey]" -> "testvalue" ,
"test[newkey]" -> "newvalue
)
Expected:
Map(
"test" -> Map(
"testkey" -> "testvalue" ,
"newkey" -> "newvalue"
)
)
Note that the end goal here is to be able to convert this into a JsObject.
I've started writing this myself, however simply porting the function from Rack is very un-scala-y and I feel like there has to be a quick way to get this that I am simply missing.
UPDATE
I am trying to find a generic solution which mimics the parsing that Rails uses (ie, with nested objects, lists, etc), and not just one level deep objects.
Just for fun, one option is to do something like:
import scala.util.matching.Regex
val pattern = new Regex("""(\w+)\[(\w+)\]""")
val qs : Map[String, Map[String, List[Seq[String]]]] = request.queryString.toList.map {
case (k, v) =>
pattern findFirstIn k match {
case Some(pattern(key, value)) => (key, value, v)
}
}.groupBy(_._1).mapValues(value => value.groupBy(_._2).mapValues {
value => value.map(x => x._3)
})
To convert this is to a JsValue, we can simply invoke:
import play.api.libs.json.Json
Json.toJson(qs)
This assumes that all your url params look like map[key]=value. You would have to modify the code a little to accommodate the standard key=value pattern.
I'm using the Specs2 JSONMatcher to validate that a GET request is being correctly converted from its internal representation (there are some manipulations we do before generating the JSON). What I need to do is, make sure that an element in the JSON array matches the corresponding object from our repository.
What I've tried:
val response = response.entity.asString // Spray's way of getting the JSON response
repository.all.map { obj =>
resp must */ ("id" -> obj.id)
resp must */ ("state" -> generateState(obj)
}
The problem is that the */ matcher just finds that "state": "whatever" (assuming generateState returns "whatever") exists somewhere in the JSON document, not necessarily in the same one matched by the ID
I tried using the indices but the repository.all method doesn't always return the elements in the same order, so there's no way of matching by index.
What I'd like to do is, iterate over the elements of the JSON array and match each one separately. Say an /## operator (or something) that takes matchers for each element:
resp /## { elem =>
val id = elem("id")
val obj = repository.lookup(id)
elem /("state" -> generateState(obj))
}
Does anyone have a way to do this or something similar?
Probably the easiest thing to do for now (until a serious refactoring of JsonMatchers) is to do some parsing and recursively use a JsonMatchers in a Matcher[String]:
"""{'db' : { 'id' : '1', 'state' : 'ok_1'} }""" must /("db" -> stateIsOk)
// a string matcher for the json string in 'db'
def stateIsOk: Matcher[String] = { json: String =>
// you need to provide a way to access the 'id' field
// then you can go on using a json matcher for the state
findId(json) must beSome { id: String =>
val obj = repository.lookup(id)
json must /("state" -> generate(obj))
}
}
// I am using my own parse function here
def findId(json: String): Option[String] =
parse(json).flatMap { a =>
findDeep("id", a).collect { case JSONArray(List(v)) => v.toString }
}
// dummy system
def generate(id: String) = "ok_"+id
case object repository {
def lookup(id: String) = id
}
What I did in the end is use responseAs[JArray], JArray#arr and JObject#values to convert the JSON structures into Lists and Maps, and then used the standard List and Map matchers. Much more flexible.