I am trying to build a validator for Json structure, and at this point I already have something as follows:
Reads definitions
case class SubTaskConfigElement(name: String)
case class MultiSelectConfig(subTasks: Seq[SubTaskConfigElement])
implicit val subTaskConfigElementReads: Reads[SubTaskConfigElement] =
(__ \ "name").read[String](minLength[String](0)).map(SubTaskConfigElement)
implicit val multiSelectConfigReads: Reads[MultiSelectConfig] = (
(__ \ "subTasks").read[Seq[SubTaskConfigElement]]
).map(MultiSelectConfig)
And I have a unit test as follows:
val configJson = Json.parse(
"""
|{
| "subTasks": [
| { "name": "Sub Task 1" },
| { "name": "Sub Task 2" },
| { "name": "Sub Task 3" }
| ]
|}
""".stripMargin)
val valid = configJson.validate[MultiSelectConfig] getOrElse JsError
logger.info(valid + "")
valid must beAnInstanceOf[MultiSelectConfig]
valid.subTasks must beAnInstanceOf[List[SubTaskConfigElement]]
In the last line of the test I am getting an error when I execute the test:
[error] /app/process-street/test/validation/widget/config/FormFieldWidgetSpec.scala:29: value subTasks is not a member of Serializable
[error] valid.subTasks must beAnInstanceOf[Seq[SubTaskConfigElement]]
IntelliJ also identifies it as a problem with: "Cannot resolve symbol subTasks"
Why is it happening? What so I miss?
Thanks.
The issue is this line:
val valid = configJson.validate[MultiSelectConfig] getOrElse JsError
The type of valid is inferred as Serializable, because that's the common parent type of MultiSelectConfig and JsError. Both are case classes and case classes automatically inherit from `Serializable.
Related
I am currently getting decoding failures for String types where the JSON file has nulls.
case class User(
updatedAt: String,
name: String
)
Json looks like:
{
"updated_at": null,
"name": "John"
}
In my tests are have:
"parse User from json file" in {
val jsonString = Source.fromURL(getClass.getResource("/user.json")).mkString
val expected = User(
null,
"John"
)
parser.decode[User](jsonString) must_=== Right(expected)
}
I have the following implicits:
implicit val customConfig: Configuration = Configuration.default.withSnakeCaseMemberNames
implicit val userDe: Decoder[User] = deriveConfiguredDecoder[User]
implicit val userEn: Encoder[User] = deriveConfiguredEncoder[User]
When I run my tests, I get this test failure:
[error] Actual: Left(DecodingFailure(String, List(DownField(updated_at))))
[error] Expected: Right(User(null,John))
What is wrong with my setup here?
During JSON parsing, I want to catch an exception for optional sequential files which schema differed from my case class.
Let me elaborate
I have the following case class:
case class SimpleFeature(
column: String,
valueType: String,
nullValue: String,
func: Option[String])
case class TaskConfig(
taskInfo: TaskInfo,
inputType: String,
training: Table,
testing: Table,
eval: Table,
splitStrategy: SplitStrategy,
label: Label,
simpleFeatures: Option[List[SimpleFeature]],
model: Model,
evaluation: Evaluation,
output: Output)
And this is part of JSON file I want to point attention to:
"simpleFeatures": [
{
"column": "pcat_id",
"value": "categorical",
"nullValue": "DUMMY"
},
{
"column": "brand_code",
"valueType": "categorical",
"nullValue": "DUMMY"
}
]
As you can see the first element has an error in the schema and while parsing, I want to raise an error. At the same time, I want to keep optional behavior in case there is no object to parse.
One idea that I've been researching for a while - to create the custom serializer and manually check fields, but not sure I'm on the right track
object JSONSerializer extends CustomKeySerializer[SimpleFeatures](format => {
case jsonObj: JObject => {
case Some(simplFeatures (jsonObj \ "simpleFeatures")) => {
// Extraction logic goes here
}
}
})
I might be not quite proficient in Scala and json4s so any advice is appreciated.
json4s version
3.2.10
scala version
2.11.12
jdk version
1.8.0
I think you need to extend CustomSerializer class since CustomKeySerializer it is used to implement custom logic for JSON keys:
import org.json4s.{CustomSerializer, MappingException}
import org.json4s.JsonAST._
import org.json4s.JsonDSL._
import org.json4s.jackson.JsonMethods._
case class SimpleFeature(column: String,
valueType: String,
nullValue: String,
func: Option[String])
case class TaskConfig(simpleFeatures: Option[Seq[SimpleFeature]])
object Main extends App {
implicit val formats = new DefaultFormats {
override val strictOptionParsing: Boolean = true
} + new SimpleFeatureSerializer()
class SimpleFeatureSerializer extends CustomSerializer[SimpleFeature](_ => ( {
case jsonObj: JObject =>
val requiredKeys = Set[String]("column", "valueType", "nullValue")
val diff = requiredKeys.diff(jsonObj.values.keySet)
if (diff.nonEmpty)
throw new MappingException(s"Fields [${requiredKeys.mkString(",")}] are mandatory. Missing fields: [${diff.mkString(",")}]")
val column = (jsonObj \ "column").extract[String]
val valueType = (jsonObj \ "valueType").extract[String]
val nullValue = (jsonObj \ "nullValue").extract[String]
val func = (jsonObj \ "func").extract[Option[String]]
SimpleFeature(column, valueType, nullValue, func)
}, {
case sf: SimpleFeature =>
("column" -> sf.column) ~
("valueType" -> sf.valueType) ~
("nullValue" -> sf.nullValue) ~
("func" -> sf.func)
}
))
// case 1: Test single feature
val singleFeature = """
{
"column": "pcat_id",
"valueType": "categorical",
"nullValue": "DUMMY"
}
"""
val singleFeatureValid = parse(singleFeature).extract[SimpleFeature]
println(singleFeatureValid)
// SimpleFeature(pcat_id,categorical,DUMMY,None)
// case 2: Test task config
val taskConfig = """{
"simpleFeatures": [
{
"column": "pcat_id",
"valueType": "categorical",
"nullValue": "DUMMY"
},
{
"column": "brand_code",
"valueType": "categorical",
"nullValue": "DUMMY"
}]
}"""
val taskConfigValid = parse(taskConfig).extract[TaskConfig]
println(taskConfigValid)
// TaskConfig(List(SimpleFeature(pcat_id,categorical,DUMMY,None), SimpleFeature(brand_code,categorical,DUMMY,None)))
// case 3: Invalid json
val invalidSingleFeature = """
{
"column": "pcat_id",
"value": "categorical",
"nullValue": "DUMMY"
}
"""
val singleFeatureInvalid = parse(invalidSingleFeature).extract[SimpleFeature]
// throws MappingException
}
Analysis: the main question here is how to gain access to the keys of jsonObj in order to check whether there is an invalid or missing key, one way to achieve that is through jsonObj.values.keySet. For the implementation, first we assign the mandatory fields to the requiredKeys variable, then we compare the requiredKeys with the ones that are currently present with requiredKeys.diff(jsonObj.values.keySet). If the difference is not empty that means there is a missing mandatory field, in this case we throw an exception including the necessary information.
Note1: we should not forget to add the new serializer to the available formats.
Note2: we throw an instance of MappingException, which is already used by json4s internally when parsing JSON string.
UPDATE
In order to force validation of Option fields you need to set strictOptionParsing option to true by overriding the corresponding method:
implicit val formats = new DefaultFormats {
override val strictOptionParsing: Boolean = true
} + new SimpleFeatureSerializer()
Resources
https://nmatpt.com/blog/2017/01/29/json4s-custom-serializer/
https://danielasfregola.com/2015/08/17/spray-how-to-deserialize-entities-with-json4s/
https://www.programcreek.com/scala/org.json4s.CustomSerializer
You can try using play.api.libs.json
"com.typesafe.play" %% "play-json" % "2.7.2",
"net.liftweb" % "lift-json_2.11" % "2.6.2"
You just need to define case class and formatters.
Example:
case class Example(a: String, b: String)
implicit val formats: DefaultFormats.type = DefaultFormats
implicit val instancesFormat= Json.format[Example]
and then just do :
Json.parse(jsonData).asOpt[Example]
In case the above gives some errors: Try adding "net.liftweb" % "lift-json_2.11" % "2.6.2" too in your dependency.
Scala newbie here.
I use Play to provide a json API for reading and writing a directory like structure. Therefore I use Scalaz.Tree, which provides ways of traversing, updating and rebuilding the Tree.
Formatting the Tree into json works well:
case class File(id: String = BSONObjectID.generate.toString(), name: String, noteBookId: String = null)
implicit val fileFormat: Format[File] = Json.format[File]
implicit def treeWrites: Writes[Tree[File]] =
new Writes[Tree[File]] {
def writes(o: Tree[File]) = o match {
case Node(file, children) => Json.obj(
"name" -> file.name,
"id" -> file.id,
"children" -> JsArray(children.map(Json.toJson(_))),
"notebookId" -> file.noteBookId
)
}
}
Reading json into a Tree however, fails
implicit def treeReads: Reads[Tree[File]] = (
//(__ \ "children").lazyRead(Reads.seq[File](treeReads)) and
(__ \ "children").read[Tree[File]] and
(__ \ "name").read[String] and
(__ \ "notebookid").read[String] and // <-- this is line 41, where the error message points at!!
(__ \ "id").read[String]
)(apply _)
implicit val treeFormat: Format[Tree[File]] = Format(treeReads, treeWrites)
The error I am getting:
[error] /home/dikken/Development/core-service-spaas/app/models/dirTree.scala:41: overloaded method value apply with alternatives:
[error] [B](f: B => (scalaz.Tree[model.treedir.File], String, String, String))(implicit fu: play.api.libs.functional.ContravariantFunctor[play.api.libs.json.Reads])play.api.libs.json.Reads[B] <and>
[error] [B](f: (scalaz.Tree[model.treedir.File], String, String, String) => B)(implicit fu: play.api.libs.functional.Functor[play.api.libs.json.Reads])play.api.libs.json.Reads[B]
[error] cannot be applied to ((=> Nothing) => scalaz.Tree[Nothing])
[error] (__ \ "id").read[String] and
[error] ^
[error] one error found
[error] (compile:compile) Compilation failed
Does this mean I have to pattern match on a case where I have a Tree of Nothing? And how excatly should I do that?
Any help appreciated! Tx!
I'm going to assume that apply _ is actually File.apply _, which cannot work here. File.apply accepts the parameters of the case class File (of which there are three). With JSON combinators, it is trying to pass the four parameters above to File.apply, which does not mix. It also does not produce a Tree[File]. What you need to do is replace File.apply with a method that accepts (children, notebookid, name, id) as parameters, and produces a Tree[File].
Here's a somewhat crude one that will work:
def jsonToTree(children: Seq[Tree[File]], name: String, notebookid: String, id: String): Tree[File] =
Tree.node(File(id, name, notebookid), children.toStream)
The Reads will now look more like this:
implicit def treeReads: Reads[Tree[File]] = (
(__ \ "children").lazyRead[Seq[Tree[File]]](Reads.seq(treeReads)).orElse(Reads.pure(Nil)) and
(__ \ "name").read[String] and
(__ \ "notebookid").read[String] and
(__ \ "id").read[String]
)(jsonToTree _)
You were closer with the commented out line as well. Because this is a recursive structure, we need to use lazyRead.
Testing:
val js = Json.parse("""{
"id": "1",
"name": "test",
"notebookid": "abc",
"children": [
{
"id": "2",
"name": "test222",
"notebookid": "ijk"
},
{
"id": "3",
"name": "test333",
"notebookid": "xyz"
}
]
}""")
scala> val tree = js.as[Tree[File]]
tree: scalaz.Tree[File] = <tree>
scala> tree.rootLabel
res8: File = File(1,test,abc)
scala> tree.subForest
res9: Stream[scalaz.Tree[File]] = Stream(<tree>, ?)
This can also be done (certainly in different ways) without combinators, as well (provided there is an implicit Reads[File] available):
implicit def treeReads: Reads[Tree[File]] = new Reads[Tree[File]] {
def reads(js: JsValue): JsResult[Tree[File]] = {
js.validate[File] map { case file =>
(js \ "children").validate[Stream[Tree[File]]].fold(
_ => Tree.leaf(file),
children => Tree.node(file, children)
)
}
}
}
I have a JSArray like below from server (I cannot change as it belongs to others):
[ {"name": "US", "id": 0, "translations" : {"name: {"es": "Estados unidos", "fr": "Etats-unis"}}},
{"name": "UK", "id": 1, "translations" : {"name: {"es": "Estados Kdda", "fr": "dsfjas"}}},
...
]
I need to extract all the name like US, UK but not name in translations and also id.
I tried several ways and it always have problem. Below are what I tried.
I first tried
case class Country(name: String, id:String)
implicit object CountryReads extends Reads[Country] {
def reads(json: JsValue) = Country(
(json \ "name"),
(json \ "id")
)
}
val countries = Json.parse(result) match { //here result is Json String
case JsArray(Seq(t)) => Some(t.as[Seq[Country]])
case _ => None
}
But I get compiling error as below:
[error] C:\git9\hss\app\models\LivingSocial.scala:80: type mismatch;
[error] found : play.api.libs.json.JsValue
[error] required: play.api.libs.json.JsResult[MyCountry]
[error] def reads(json: JsValue) = (json \ "currencyCode")
[error] ^
[error] one error found
[error] (compile:compile) Compilation failed
then I tried:
val jResult = Json.parse(result)
(jResult \\ "name").foreach { name =>
println(name.as[String])
}
I get error in println() as "\" will recursively pull names under translation also.
Any good way to do it?
case class Country(name: String, id: Int, translations: Map[String, Map[String, String]])
object Country {
implicit val format = Json.format[Country]
}
val js = Json.parse(yourString)
val names: JsResult[Seq[String]] = js.validate[Seq[Country]].map { countries => countries.map(_.name) }
At that point you can deal with the JsResult since you'll need error handling in case the JSON doesn't conform to your expectations.
You can change your read to look like this :
implicit val CountryReads: Reads[Country] = (
(JsPath \ "name").read[String] and
(JsPath \ "name").read[String]
)(Country.apply _)
this is a correct way to create a reader according to play documentation
Given a JSON string like this:
{"Locations":
{"list":
[
{"description": "some description", "name": "the name", "id": "dev123"},
{"description": "other description", "name": "other name", "id": "dev59"}
]
}
}
I'd like to return a list of "id"s from a function parsing the above string. JSON.parseFull() (from scala.util.parsing.json) gives me a result of type Option[Any]. Scala REPL shows it as Some(Map(Locations -> Map(list -> List(Map(id -> dev123, ... and as a beginner in Scala I'm puzzled as to which way to approach it.
Scala API docs suggest "to treat it as a collection or monad and use map, flatMap, filter, or foreach". Top-level element is an Option[Any] however that should be Some with a Map that should contain a single key "Locations", that should contain a single key "list" that finally is a List. What would be an idiomatic way in Scala to write a function retrieving the "id"s?
First of all, you should cast json from Any to right type:
val json = anyJson.asInstanceOf[Option[Map[String,List[Map[String,String]]]]]
And then you may extract ids from Option using map method:
val ids = json.map(_("Locations")("list").map(_("id"))).getOrElse(List())
Because Any is everywhere is the returned result, you'll have to cast. Using one of my earlier answers:
class CC[T] { def unapply(a:Any):Option[T] = Some(a.asInstanceOf[T]) }
object M extends CC[Map[String, Any]]
object L extends CC[List[Any]]
object S extends CC[String]
object D extends CC[Double]
object B extends CC[Boolean]
for {
Some(M(map)) <- List(JSON.parseFull(jsonString))
M(locMap) = map("Locations")
L(list) = locMap("list")
description <- list
M(desc) = description
S(id) = desc("id")
} yield id
// res0: List[String] = List(dev123, dev59)
For this type of tasks, you should take a look at Rapture.io. I'm also a scala beginner, but from what I've searched for, this seems to have the friendliest syntax. Here's a short example, taken from a gist:
import rapture.io._
// Let's parse some JSON
val src: Json = Json.parse("""
{
"foo": "Hello world",
"bar": {
"baz": 42
}
}
""")
// We can now access the value bar.baz
val x: Json = src.bar.baz
// And get it as an integer
val y: Int = x.get[Int]
// Alternatively, we can use an extractor to get the values we want:
val json""" { "bar": { "baz": $x }, "foo": $z }""" = src
// Now x = 42 and z = "Hello world".
Is this what you need? (using lift-json)
scala> import net.liftweb.json._
import net.liftweb.json._
scala> implicit val formats = DefaultFormats
formats: net.liftweb.json.DefaultFormats.type = net.liftweb.json.DefaultFormats$#79e379
scala> val jsonString = """{"Locations":
{"list":
[
{"description": "some description", "name": "the name", "id": "dev123"},
{"description": "other description", "name": "other name", "id": "dev59"}
]
}
}"""
jsonString: java.lang.String =
{"Locations":
{"list":
[
{"description": "some description", "name": "the name", "id": "dev123"},
{"description": "other description", "name": "other name", "id": "dev59"}
]
}
}
scala> Serialization.read[Map[String, Map[String, List[Map[String, String]]]]](jsonString)
res43: Map[String,Map[String,List[Map[String,String]]]] = Map(Locations -> Map(list -> List(Map(description -> some desc
ription, name -> the name, id -> dev123), Map(description -> other description, name -> other name, id -> dev59))))
scala> val json = parse(jsonString)
json: net.liftweb.json.package.JValue = JObject(List(JField(Locations,JObject(List(JField(list,JArray(List(JObject(List(
JField(description,JString(some description)), JField(name,JString(the name)), JField(id,JString(dev123)))), JObject(Lis
t(JField(description,JString(other description)), JField(name,JString(other name)), JField(id,JString(dev59))))))))))))
scala> json \\ "id"
res44: net.liftweb.json.JsonAST.JValue = JObject(List(JField(id,JString(dev123)), JField(id,JString(dev59))))
scala> compact(render(res44))
res45: String = {"id":"dev123","id":"dev59"}
In a branch of SON of JSON, this will work. Note that I'm not using the parser. Not that it doesn't exist. It's just that creating an JSON object using the builder methods is easier:
scala> import nl.typeset.sonofjson._
import nl.typeset.sonofjson._
scala> var all = obj(
| locations = arr(
| obj(description = "foo", id = "807",
| obj(description = "bar", id = "23324"
| )
| )
scala> all.locations.map(_.id).as[List[String]]
res2: List[String] = List(23324, 807)
Or use a for comprehension:
scala> (for (location <- all.locations) yield location.id).as[List[String]]
res4: List[String] = List(23324, 807)