Suppose you have a JSON which looks like this:
[{"foo": 1, "bar": 2}, {"foo": 3, "bar": {"baz": 4}}]
It seems natural to try to represent this using a Scala sum type:
sealed trait Item
case class IntItem(foo: Int, bar: Int) extends Item
case class Baz(baz: Int)
case class BazItem(foo: Int, bar: Baz) extends Item
My question is: is it possible to use Jackson's Scala module to serialize the JSON above into a List[Item]?
My attempt:
val string = "[{\"foo\": 1, \"bar\": 2}, {\"foo\": 3, \"bar\": {\"baz\": 4}}]"
val mapper = new ObjectMapper() with ScalaObjectMapper
mapper.registerModule(DefaultScalaModule)
mapper.configure(DeserializationFeature.FAIL_ON_UNKNOWN_PROPERTIES, false)
mapper.readValue[List[Item]](string)
The exception:
Exception in thread "main" com.fasterxml.jackson.databind.JsonMappingException: Can not construct instance of ...Item, problem: abstract types either need to be mapped to concrete types, have custom deserializer, or be instantiated with additional type information
at [Source: [{"foo": 1, "bar": {"baz": 2}}, {"foo": 3, "bar": {"baz": 4}}]; line: 1, column: 2] (through reference chain: com.fasterxml.jackson.module.scala.deser.BuilderWrapper[0])
That makes it fairly clear what the problem is, but I'm not sure how best to fix it.
As #Dima pointed out, I don't think there exists a generic solution that covers all the cases. Moreover I'm not sure it can exist at all because the difference might be hidden arbitrary deep and I suspect someone smart enough can create a halting problem from that. However many specific cases can be solved.
First of all, if you control both sides (serialization and deserialization), you should consider using JsonTypeIdResolver annotation with some of TypeIdResolver subclasses that will put name of the type in the JSON itself.
If you can't use JsonTypeIdResolver, probably the only solution is to roll out your custom JsonDeserializer as the error suggests. The example you provided in your question can be handled by something like this:
sealed trait Item
case class IntItem(foo: Int, bar: Int) extends Item
case class Baz(baz: Int)
case class BazItem(foo: Int, bar: Baz) extends Item
import com.fasterxml.jackson.core._
import com.fasterxml.jackson.databind._
import com.fasterxml.jackson.databind.module.SimpleModule
import com.fasterxml.jackson.databind.util.TokenBuffer
import com.fasterxml.jackson.databind.deser.std.StdDeserializer
import com.fasterxml.jackson.databind.node._
import com.fasterxml.jackson.databind.exc._
import java.io.IOException
class ItemDeserializer() extends StdDeserializer[Item](classOf[Item]) {
#throws[IOException]
#throws[JsonProcessingException]
def deserialize(jp: JsonParser, ctxt: DeserializationContext): Item = {
// 1) Buffer current state of the JsonParser
// 2) Use firstParser (from the buffer) to parser whole sub-tree into a generic JsonNode
// 3) Analyze tree to find out the real type to be parser
// 3) Using the buffer roll back history and create objectParser to parse the sub-tree as known type
val tb = new TokenBuffer(jp, ctxt)
tb.copyCurrentStructure(jp)
val firstParser = tb.asParser
firstParser.nextToken
val curNode = firstParser.getCodec.readTree[JsonNode](firstParser)
val objectParser = tb.asParser
objectParser.nextToken()
val bar = curNode.get("bar")
if (bar.isInstanceOf[IntNode]) {
objectParser.readValueAs[IntItem](classOf[IntItem])
}
else if (bar.isInstanceOf[ObjectNode]) {
objectParser.readValueAs[BazItem](classOf[BazItem])
}
else {
throw ctxt.reportBadDefinition[JsonMappingException](classOf[Item], "Unknown subtype of Item") // Jackson 2.9
//throw InvalidDefinitionException.from(jp, "Unknown subtype of Item", ctxt.constructType(classOf[Item])) // Jackson 2.8
}
}
}
and then you can use it as following
def test() = {
import com.fasterxml.jackson.module.scala._
import com.fasterxml.jackson.module.scala.experimental._
val mapper = new ObjectMapper() with ScalaObjectMapper
mapper.registerModule(DefaultScalaModule)
// add our custom ItemDeserializer
val module = new SimpleModule
module.addDeserializer(classOf[Item], new ItemDeserializer)
mapper.registerModule(module)
mapper.configure(DeserializationFeature.FAIL_ON_UNKNOWN_PROPERTIES, false)
val string = "[{\"foo\": 1, \"bar\": 2}, {\"foo\": 3, \"bar\": {\"baz\": 4}}]"
val list = mapper.readValue[List[Item]](string)
println(list.mkString(", "))
}
which prints
IntItem(1,2), BazItem(3,Baz(4))
The main trick in the ItemDeserializer is to use TokenBuffer to parse JSON twice: first time to analyze the JSON-tree and find out as what type it should be parsed as, second time to actually parse the object of a known type.
Related
case class Person(name: String,
override val age: Int,
override val address: String
) extends Details(age, address)
class Details(val age: Int, val address: String)
val person = Person("Alex", 33, "Europe")
val details = person.asInstanceOf[Details] // ???
println(details) // I want only Details class fields
I have these 2 classes. In reality, both have a lot of fields. Somewhere, I need only field of superclass, taken from Person class.
There is a nice way to get only super class values and not mapping them field by field?
*I'm pretty sure I'll have some problems with json writes for class Details (which is not a case class and have not a singleton object, but this is another subject)
If I get your question correctly, then you might be asking me runtime polymorphism or dynamic method dispatch from java.
If so, you may have to create both the class and not case class
class Details( val age: Int, val address: String)
class Person(name: String,
override val age: Int,
override val address: String
) extends Details(age, address) {
}
Now create the object of person and reference to superclass (Details)
val detail:Details = new Person("Alex", 33, "Europe")
println(detail.address)
println(detail.age)
This way you will be able to get the only address and age
Another way is like , why can't we create the Details a separate entity like:
case class Details( age: Int, address: String)
case class Person(name: String,
details: Details
)
val detail = Person("Alex", Details(10,"Europe") )
Output:
println(detail.details)
Details(10,Europe)
I will post a solution that leverages scala macro system (old kind, not the newest introduced with Scala 3.0). It could be an overkill for you...
BTW, if you want to access to only parent values (for example for getting key, value pair), you can:
given a type tag, get all parents;
from them, extract all the accessors (vals);
for each val, get its value;
and finally returns a list with all accessors taken
So, I try to solve each point step by step.
First of all, we have to write the macro definition as:
object Macros {
def accessors[T](element : T): String = macro MacrosImpl.accessors[T]
}
object MacrosImpl {
def accessors[T: c.WeakTypeTag](c: whitebox.Context): c.Expr[String] = ...
}
for the first point, we can leverage the reflection macroprogramming API using c.universe:
import c.universe._
val weakType = weakTypeTag[T] //thanks to the WeakTypeTag typeclass
val parents = weakType.tpe.baseClasses
for the second point, we can iterate over the parent classes and then take only the public accessors:
val accessors = parents
.map(weakType.tpe.baseType(_))
.flatMap(_.members)
.filter(_.isPublic)
.filter(_.isMethod)
.map(_.asMethod)
.filter(_.isAccessor)
.toSet
So, for example, if the we write Macros.accessors[Details](person), accessors will yield age and address.
To take the value, we can leverage quasiqouting. So, first we take only the values name:
val names = accessors
.map(_.fullName)
.map(_.split("\\."))
.map(_.reverse.head)
Then we convert them into a TermName:
val terms = names.map(TermName(_))
And finally, we convert each term to a key value tuple containing the val name and its value:
val accessorValues = terms
.map(name => c.Expr[(String, Any)](q"(${name.toString}, ${element}.${name})"))
.toSeq
The last step consist in convert a Seq[Expr[(String, Any)] into a Expr[Seq[(String, Any)]. A way to do that, could be leveraging recursion, reify, and splicing expression:
def seqToExprs(seq: Seq[Expr[(String, Any)]]): c.Expr[Seq[(String, Any)]] =
seq.headOption match {
case Some(head) =>
c.universe.reify(
Seq((head.splice._1, head.splice._2)) ++
seqToExprs(seq.tail).splice
)
case _ => c.Expr[Seq[(String, Any)]](q"Seq.empty")
}
So now I decide to return a String representation (but you can manipulate it as you want):
val elements = seqToExprs(accessorValues)
c.Expr[String](q"${elements}.mkString")
You can use it as:
import Macros._
class A(val a : Int)
class B(val b : Int) extends A(b)
class C(val c: Int) extends B(c)
//println(typeToString[List[Set[List[Double]]]])
val c = new C(10)
println(accessors[C](c)) // prints (a, 10)(b, 10)(c, 10)
println(accessors[B](c)) // prints (a, 10)(b, 10)
println(accessors[A](c)) // prints (a, 10)
And, using your example:
// Your example:
case class Person(name: String,
override val age: Int,
override val address: String
) extends Details(age, address)
class Details(val age: Int, val address: String)
val person = Person("Alex", 33, "Europe")
println(accessors[Details](person)) // prints (address,Europe)(age,33)
println(accessors[Person](person)) // prints (address,Europe)(age,33)(name,Alex)
Here there is a repository with the macro implemented.
Scala 3.0 introduce a safer and cleaner macro system, if you use it and you want to go further you can read these articles:
macros tips and tricks
short tutorial
another tutorial
I've created new Play-Scala project and added my first model class (case class).
I have a mock repository for my model class and when I try to add Seq variable in the repository class, project's compilation fails.
If I create companion object for my model class and add Seq variable inside this object, compilation is succeed. Without any Seq definition compilation is fine too.
Error text is very strange for me: Error:(7, 25) illegal character '\u2028'
I use IntelliJ Idea and file encoding is UTF-8.
My classes code:
case class Teacher(id: Long, name: String) extends BaseItem
object Teacher {
val list = Seq(1, 2, 3) //this compiles
}
class TeacherRepository extends BaseRepository[Teacher] with BaseTeacherRepository {
val int = List(1, 2, 3
) //this one fails
override def list: Seq[Teacher] = ???
override def getById(id: Long): Option[Teacher] = ???
override def getByLanguage(language: String): Seq[Teacher] = ???
}
Edited: It seems like IDE does not show line separators before closing bracket in the second case.
Valid code should be
val int = List(1, 2, 3)
Why not just format your code correctly?
class TeacherRepository extends BaseRepository[Teacher] with BaseTeacherRepository {
val int = List(1, 2, 3)
...
}
If I have a Scala Map
val map = Map("a" -> true,
"b" -> "hello"
"c" -> 5)
Is there a way I can convert this to an object (case class, regular class, anything really) such that I can access the field like this:
val obj = map.toObj
println(obj.a)
println(obj.b)
Without knowing what the parameters will be ahead of time?
Or even better, can it be an instance variable of the class I'm currently working with?
So I could actually just have
println(a)
println(b)
Have a look at Dynamic:
import scala.language.dynamics
class Wrapper(m: Map[String, Any]) extends Dynamic {
def selectDynamic(name: String) = {
m(name)
}
}
object Demo {
def main(args: Array[String]) {
val map = Map("a" -> true,
"b" -> "hello",
"c" -> 5)
val w = new Wrapper(map)
println(w.a)
println(w.b)
}
}
Dynamic gives you "properties on demand".
But it's not type-safe. In the snippet above, if you try to access a non-existing key in the map, a NoSuchElementException will be thrown. And you get the Any type, so you have to use asInstanceOf at the caller's side, for example.
Dynamic has further methods, so have a closer look at the documentation of Dynamic
I hope you understand by asking this question that you will loose all compile time guarantees when you rely on runtime data. If this is really what you want then you can rely on Dynamic:
object X extends App {
import scala.language.dynamics
class D(fields: Map[String, Any]) extends Dynamic {
def selectDynamic(str: String): Any =
fields.getOrElse(str, throw new NoSuchFieldException(str))
}
val fields = Map[String, Any]("a" -> true, "b" -> "hello")
val obj = new D(fields)
println(obj.a)
println(obj.b)
}
Dynamic is a compiler feature that translates all field/method calls to a call to the *Dynamic* methods. Because the compiler can't know anything about your program (how should it?), the return type you get here is Any and when you call a field/method that does not exist you get an exception at runtime instead of a compile time error. When some fields/methods are known to compile time you can combine Dynamic with macros to get at least some compile time checking (such a method is described in the linked answer).
Beside from that, that is the only syntax you can enable in Scala. If return types are important to you, you can at least add them as type parameter:
object X extends App {
import scala.language.dynamics
class D(fields: Map[String, Any]) extends Dynamic {
def selectDynamic[A : reflect.ClassTag](str: String): A =
fields.get(str) match {
case Some(f: A) => f
case _ => throw new NoSuchFieldException(str)
}
}
val fields = Map[String, Any]("a" -> true, "b" -> "hello")
val obj = new D(fields)
println(obj.a[Boolean])
println(obj.b[String])
}
I would like to extend Scala's implementation of Enumeration with a custom field, say label. That new field should be accessible via the values of that enumeration. Furthermore, that custom field should be part of various implementations of Enumeration.
I am aware of the following questions at Stackoverflow:
How to add a method to Enumeration in Scala?
How do I create an enum in scala that has an extra field
Overriding Scala Enumeration Value
Scala doesn't have enums - what to use instead of an enum
However, none of them solves my issues:
The first issue is that I am able to add a custom field. However, I cannot access that additional field via the Values returned by Enumeration.values. The following code works and prints 2nd enumeration value:
object MyEnum extends Enumeration {
type MyEnum = MyVal
val VALUE_ONE = MyVal()
val VALUE_TWO = MyVal(Some("2nd enumeration value"))
val VALUE_THREE = MyVal(Some("3rd value"))
case class MyVal(label: Option[String] = None) extends Val(nextId)
}
import MyEnum._
println(VALUE_TWO.label.get)
Note that I access the label via one of the values. The following code does not work:
for (value <- MyEnum.values) println(value.label)
The error message is as follows: error: value label is not a member of MyEnum.Value
Obviously, instead of MyEnum.MyVal, MyEnum.Val is used. The latter does not define label, while my custom value would provide field label.
The second issue is that it seems to be possible to introduce a custom Value and Val, respectively, in the context of an Enumeration only. Thus, as far as I know, it is not possible to use such a field across different enums. At least, the following code does not compile:
case class MyVal(label: Option[String] = None) extends Enumeration.Val(nextId)
object MyEnum extends Enumeration {
type MyEnum = MyVal
val VALUE_ONE = MyVal()
val VALUE_TWO = MyVal(Some("2nd enumeration value"))
}
object MySecondEnum extends Enumeration {
type MySecondEnum = MyVal
val VALUE_ONE = MyVal()
val VALUE_TWO = MyVal(Some("2nd enumeration value"))
}
Due to the fact that class Val is protected, case class MyVal cannot access Val -- MyVal is not defined in the context of an enumeration.
Any idea how to solve the above issues?
The first issue is addressed by a recent question, my answer to which got no love.
For that use case, I would write a custom widgets method with the useful type, but my linked answer, which just introduces an implicit conversion, seems pretty handy. I don't know why it's not the canonical solution.
For the second issue, your derived MyVal should just implement a trait.
Sample:
scala> trait Labelled { def label: Option[String] }
defined trait Labelled
scala> object A extends Enumeration { case class AA(label: Option[String]) extends Val with Labelled ; val X = AA(Some("one")) }
defined object A
scala> object B extends Enumeration { case class BB(label: Option[String]) extends Val with Labelled ; val Y = BB(None) }
defined object B
scala> val labels = List(A.X, B.Y)
labels: List[Enumeration#Val with Product with Labelled] = List(X, Y)
scala> labels map (_.label)
res0: List[Option[String]] = List(Some(one), None)
I'm trying to use "pickling" serialization is Scala, and I see the same example demonstrating it:
import scala.pickling._
import json._
val pckl = List(1, 2, 3, 4).pickle
Unpickling is just as easy as pickling:
val lst = pckl.unpickle[List[Int]]
This example raises some question. First of all, it skips converting of object to string. Apparently you need to call pckl.value to get json string representation.
Unpickling is even more confusing. Deserialization is an act of turning string (or bytes) into an object. How come this "example" demonstrates deserialization if there is no string/binry representation of object?
So, how do I deserialize simple object with pickling library?
Use the type system and case classes to achieve your goals. You can unpickle to some superior type in your hierarchy (up to and including AnyRef). Here is an example:
trait Zero
case class One(a:Int) extends Zero
case class Two(s:String) extends Zero
object Test extends App {
import scala.pickling._
import json._
// String that can be sent down a wire
val wire: String = Two("abc").pickle.value
// On the other side, just use a case class
wire.unpickle[Zero] match {
case One(a) => println(a)
case Two(s) => println(s)
case unknown => println(unknown.getClass.getCanonicalName)
}
}
Ok, I think I understood it.
import scala.pickling._
import json._
var str = Array(1,2,3).pickle.value // this is JSON string
println(str)
val x = str.unpickle[Array[Int]] // unpickle from string
will produce JSON string:
{
"tpe": "scala.Array[scala.Int]",
"value": [
1,
2,
3
]
}
So, the same way we pickle any type, we can unpickle string. Type of serialization is regulated by implicit formatter declared in "json." and can be replaced by "binary."
It does look like you will be starting with a pickle to unpickle to a case class. But the JSON string can be fed to the JSONPickle class to get the starting pickle.
Here's an example based on their array-json test
package so
import scala.pickling._
import json._
case class C(arr: Array[Int]) { override def toString = s"""C(${arr.mkString("[", ",", "]")})""" }
object PickleTester extends App {
val json = """{"arr":[ 1, 2, 3 ]}"""
val cPickle = JSONPickle( json )
val unpickledC: C = cPickle.unpickle[C]
println( s"$unpickledC, arr.sum = ${unpickledC.arr.sum}" )
}
The output printed is:
C([1,2,3]), arr.sum = 6
I was able to drop the "tpe" in from the test as well as the .stripMargin.trim on the input JSON from the test. It works all in one line, but I thought it might be more apparent split up. It's unclear to me if that "tpe" from the test is supposed to provide a measure of type safety for the incoming JSON.
Looks like the only other class they support for pickling is a BinaryPickle unless you want to roll your own. The latest scala-pickling snapshot jar requires quasiquotes to compile the code in this answer.
I tried someting more complicated this morning and discovered that the "tpe" is required for non-primatives in the incoming JSON - which points out that the serialized string really must be compatible with the pickler( which I mixed into the above code ):
case class J(a: Option[Boolean], b: Option[String], c: Option[Int]) { override def toString = s"J($a, $b, $c)" }
...
val jJson = """{"a": {"tpe": "scala.None.type"},
| "b":{"tpe": "scala.Some[java.lang.String]","x":"donut"},
| "c":{"tpe": "scala.Some[scala.Int]","x":47}}"""
val jPickle = JSONPickle( jJson.stripMargin.trim )
val unpickledJ: J = jPickle.unpickle[J]
println( s"$unpickledJ" )
...
where naturually, I had to use .value on a J(None, Some("donut"), Some(47)) to figure out how to create the jJson input value to prevent the unpickling from throwing an exception.
The output for J is like:
J(None, Some(donut), Some(47))
Looking at this test, it appears that if the incoming JSON is all primatives or case classes (or combinations) that the JSONPickle magic works, but some other classes like Options require extra "tpe" type information to unpickle correctly.