Scala pickling: how? - scala

I'm trying to use "pickling" serialization is Scala, and I see the same example demonstrating it:
import scala.pickling._
import json._
val pckl = List(1, 2, 3, 4).pickle
Unpickling is just as easy as pickling:
val lst = pckl.unpickle[List[Int]]
This example raises some question. First of all, it skips converting of object to string. Apparently you need to call pckl.value to get json string representation.
Unpickling is even more confusing. Deserialization is an act of turning string (or bytes) into an object. How come this "example" demonstrates deserialization if there is no string/binry representation of object?
So, how do I deserialize simple object with pickling library?

Use the type system and case classes to achieve your goals. You can unpickle to some superior type in your hierarchy (up to and including AnyRef). Here is an example:
trait Zero
case class One(a:Int) extends Zero
case class Two(s:String) extends Zero
object Test extends App {
import scala.pickling._
import json._
// String that can be sent down a wire
val wire: String = Two("abc").pickle.value
// On the other side, just use a case class
wire.unpickle[Zero] match {
case One(a) => println(a)
case Two(s) => println(s)
case unknown => println(unknown.getClass.getCanonicalName)
}
}

Ok, I think I understood it.
import scala.pickling._
import json._
var str = Array(1,2,3).pickle.value // this is JSON string
println(str)
val x = str.unpickle[Array[Int]] // unpickle from string
will produce JSON string:
{
"tpe": "scala.Array[scala.Int]",
"value": [
1,
2,
3
]
}
So, the same way we pickle any type, we can unpickle string. Type of serialization is regulated by implicit formatter declared in "json." and can be replaced by "binary."

It does look like you will be starting with a pickle to unpickle to a case class. But the JSON string can be fed to the JSONPickle class to get the starting pickle.
Here's an example based on their array-json test
package so
import scala.pickling._
import json._
case class C(arr: Array[Int]) { override def toString = s"""C(${arr.mkString("[", ",", "]")})""" }
object PickleTester extends App {
val json = """{"arr":[ 1, 2, 3 ]}"""
val cPickle = JSONPickle( json )
val unpickledC: C = cPickle.unpickle[C]
println( s"$unpickledC, arr.sum = ${unpickledC.arr.sum}" )
}
The output printed is:
C([1,2,3]), arr.sum = 6
I was able to drop the "tpe" in from the test as well as the .stripMargin.trim on the input JSON from the test. It works all in one line, but I thought it might be more apparent split up. It's unclear to me if that "tpe" from the test is supposed to provide a measure of type safety for the incoming JSON.
Looks like the only other class they support for pickling is a BinaryPickle unless you want to roll your own. The latest scala-pickling snapshot jar requires quasiquotes to compile the code in this answer.
I tried someting more complicated this morning and discovered that the "tpe" is required for non-primatives in the incoming JSON - which points out that the serialized string really must be compatible with the pickler( which I mixed into the above code ):
case class J(a: Option[Boolean], b: Option[String], c: Option[Int]) { override def toString = s"J($a, $b, $c)" }
...
val jJson = """{"a": {"tpe": "scala.None.type"},
| "b":{"tpe": "scala.Some[java.lang.String]","x":"donut"},
| "c":{"tpe": "scala.Some[scala.Int]","x":47}}"""
val jPickle = JSONPickle( jJson.stripMargin.trim )
val unpickledJ: J = jPickle.unpickle[J]
println( s"$unpickledJ" )
...
where naturually, I had to use .value on a J(None, Some("donut"), Some(47)) to figure out how to create the jJson input value to prevent the unpickling from throwing an exception.
The output for J is like:
J(None, Some(donut), Some(47))
Looking at this test, it appears that if the incoming JSON is all primatives or case classes (or combinations) that the JSONPickle magic works, but some other classes like Options require extra "tpe" type information to unpickle correctly.

Related

Fail-fast json4s serialisation of sealed trait and object enum when missing serializer

Set up
I'm using json4s 3.2.11 and Scala 2.11.
I have an enumeration defined using sealed trait, and a custom serializer for it:
import org.json4s.CustomSerializer
import org.json4s.JsonAST.JString
import org.json4s.DefaultFormats
import org.json4s.jackson.Serialization
sealed trait Foo
case object X extends Foo
case object Y extends Foo
object FooSerializer
extends CustomSerializer[Foo](
_ =>
({
case JString("x") => X
case JString("y") => Y
}, {
case X => JString("x")
case Y => JString("y")
})
)
This is great, and works well when added to the formats:
{
implicit val formats = DefaultFormats + FooSerializer
Serialization.write(X) // "x"
}
This is great!
Problem
If the serializer is not added to the formats, json4s will use reflection to create a default representation of the fields, which is extremely unhelpful for these objects that don't have fields. It does this silently, seemingly without a way to control it.
{
implicit val formats = DefaultFormats
Serialization.write(X) // {}
}
This is a problematic, as there's no indication of what's gone wrong until much later. This invalid/useless data might be sent around the network or written to databases, if tests don't happen to catch it. And, this may be exposed publicly from a library, meaning downstream users have to remember it as well.
NB. this is different to read, which throws an exception on failure, since the Foo trait doesn't have any useful constructors:
{
implicit val formats = DefaultFormats
Serialization.read[Foo]("\"x\"")
}
org.json4s.package$MappingException: No constructor for type Foo, JString(x)
at org.json4s.Extraction$ClassInstanceBuilder.org$json4s$Extraction$ClassInstanceBuilder$$constructor(Extraction.scala:417)
at org.json4s.Extraction$ClassInstanceBuilder.org$json4s$Extraction$ClassInstanceBuilder$$instantiate(Extraction.scala:468)
at org.json4s.Extraction$ClassInstanceBuilder$$anonfun$result$6.apply(Extraction.scala:515)
...
Question
Is there a way to either disable the default {} formatting for these objects, or to "bake" in the formatting to the object itself?
For instance, having write throw an exception like read would be fine, as it would flag the problem to the caller immediately.
There is an old open issue which seems to ask similar question where one of the contributors suggests to
you need to create a custom deserializer or serializer
which makes it sound there is no out-of-the-box way to alter the default behaviour.
Method 1: Disallow default formats via Scalastyle
Try disallowing import of org.json4s.DefaultFormats using Scalastyle IllegalImportsChecker
<check level="error" class="org.scalastyle.scalariform.IllegalImportsChecker" enabled="true">
<parameters>
<customMessage>Import from illegal package: Please use example.DefaultFormats instead of org.json4s.DefaultFormats</customMessage>
<parameter name="illegalImports"><![CDATA[org.json4s.DefaultFormats]]></parameter>
</parameters>
</check>
and provide custom DefaultFormats like so
package object example {
val DefaultFormats = Serialization.formats(NoTypeHints) + FooSerializer
}
which would allow us to serialise ADTs like so
import example.DefaultFormats
implicit val formats = DefaultFormats
case class Bar(foo: Foo)
println(Serialization.write(Bar(X)))
println(Serialization.write(X))
println(Serialization.write(Y))
which should output
{"foo":"x"}
"x"
"y"
If we try to import org.json4s.DefaultFormats, then Scalastyle should raise the following error:
Import from illegal package: Please use example.DefaultFormats instead of org.json4s.DefaultFormats
Method 2: Bake in serialisation for non-nested values
Perhaps we could "bake in" the formatting into objects by defining write method in Foo which delegates to Serialization.write like so
sealed trait Foo {
object FooSerializer extends CustomSerializer[Foo](_ =>
({
case JString("x") => X
case JString("y") => Y
}, {
case X => JString("x")
case Y => JString("y")
})
)
def write: String =
Serialization.write(this)(DefaultFormats + FooSerializer)
}
case object X extends Foo
case object Y extends Foo
Note how we hardcoded passing FooSerializer format to write. Now we can serialise with
println(X.write)
println(Y.write)
which should output
"x"
"y"
Method 3: Provide custom DefaultFormats alongside org.json4s.DefaultFormats
We could also try defining custom DefaultFormats in our own package like so
package example
object DefaultFormats extends DefaultFormats {
override val customSerializers: List[Serializer[_]] = List(FooSerializer)
}
which would allow us to serialise ADTs like so
import example.DefaultFormats
implicit val formats = DefaultFormats
case class Bar(foo: Foo)
println(Serialization.write(Bar(X)))
println(Serialization.write(X))
println(Serialization.write(Y))
which should output
{"foo":"x"}
"x"
"y"
Having two default formats, org.json4s.DefaultFormats and example.DefaultFormats, would at least make the user have to choose between the two, if say, they use IDE to auto-import them.

Scala: reflection against named arguments

I'm trying to pass named arguments to a function from a regular Scala object like string/list/map, where the name of the argument and it's value are both variable (in my case from parsed user input). Is there a way to do this in Scala? I'm in principal looking for a short program in scala, similar to this in python:
def surprise(animal, color):
print('Oh, a ' + color + ' ' + animal + '!')
arguments = {'animal': 'cat', 'color': 'black'}
surprise(**arguments)
Since python can unpack dictionaries into named arguments, this results in
Oh, a black cat!
I've been searching for this functionality in scala, but I could not find it. Can anyone give me an example on how to accomplish this in scala?
Thanks in advance!
I would say, it is not that easy as in python, but I will try to propose couple of solutions. You need to extract your parameters from json (or other user input) with types and with order. It could be done using, for example, helper case class and play-json:
def surprise(animal: String, color: String): Unit = {
println(s"Oh, a $color $animal!")
}
import play.api.libs.json.{JsValue, Json}
case class FuncArguments(animal: String, color: String)
implicit val funcArgumentsFormat = Json.format[FuncArguments]
implicit def jsValueToFuncArguments(json: JsValue): FuncArguments =
json.as[FuncArguments]
def surprise2(json: JsValue): Unit = {
(surprise _).tupled(FuncArguments.unapply(json).get)
}
case class has same signature as your method. implicit val funcArgumentsFormatis play-json format to extract your data into case class (unsafe, because of as. Will throw Exception in case of missing required argument names/types in json), implicit def jsValueToFuncArguments converts your json into case class (theoretically also unsafe, because of Option.get, but do not think that you can get exception here). And helper function surprise2 to convert json into arguments.
Another approach would be, for example, to use some reflection:
import play.api.libs.json.{JsValue, Json}
class SomeClass {
def surprise(animal: String, color: String): Unit = {
println(s"Oh, a $color $animal!")
}
}
def surprise3(json: JsValue): Unit = {
val method = classOf[SomeClass].getMethods.find(_.getName == "surprise").get
val params = method.getParameters
val values = params.map(_.getName).map(name => (json \ name).as[String])
val ref = new SomeClass
method.invoke(ref, values: _*)
}
In this example we have an assumption that all fields have the same type String, your function is inside class. Play-json is used for parsing. You get method, than argument names, than extract values for arguments (in same order as arguments) and than just apply them to function.
And calls:
val json = Json.obj("animal" -> "cat", "color" -> "black")
surprise2(json)
surprise3(json)

Transforming Scala case class into JSON

I have two case classes. The main one, Request, contains two maps.
The first map has a string for both key and value.
The second map has a string key, and value which is an instance of the second case class, KVMapList.
case class Request (var parameters:MutableMap[String, String] = MutableMap[String, String](), var deps:MutableMap[String, KVMapList] = MutableMap[String, KVMapList]())
case class KVMapList(kvMap:MutableMap[String, String], list:ListBuffer[MutableMap[String, String]])
The requirement is to transform Request into a JSON representation.
The following code is trying to do this:
import com.fasterxml.jackson.annotation.PropertyAccessor
import com.fasterxml.jackson.module.scala.experimental.ScalaObjectMapper
import com.fasterxml.jackson.annotation.JsonAutoDetect.Visibility
import com.fasterxml.jackson.databind.ObjectMapper
def test(req:Request):String {
val mapper = new ObjectMapper() with ScalaObjectMapper
mapper.setVisibility(PropertyAccessor.ALL, Visibility.ANY)
var jsonInString: String = null
try {
jsonInString = mapper.writeValueAsString(request)
}
catch {
=case e: IOException => {
e.printStackTrace
}
jsonString
}
This however is not working. Even when the Request class is populated, the output is :
{"parameters":{"underlying":{"some-value":""},"empty":false,"traversableAgain":true},"deps":{"sizeMapDefined":false,"empty":false,"traversableAgain":true}}
Using the JSON object mapper with corresponding Java classes is straightforward, but have not yet got it working in Scala. Any assistance is very much appreciated.
Jackson is more of a bad old memory in Scala to some degree. You should use a native Scala library for JSON processing, particularly one really good at compile time derivation of JSON serializers, such as circe.
I'm aware this doesn't directly answer your question, but after using circe I would never go back to anything else.
import io.circe.generic.auto._
import io.circe.parser._
import io.circe.syntax._
val req = new Request(...)
val json = req.asJson.noSpaces
val reparsed = decode[Request](json)
On a different note, using mutable maps inside case classes is as non-idiomatic as it gets, and it should be quite trivial to implement immutable ops for your maps using the auto-generated copy method.
case class Request(parameters: Map[String, String] {
def +(key: String, value: String): Request = {
this.copy(parameters = parameters + (key -> value))
}
}
You should really avoid mutability wherever possible, and it looks like avoiding it here wouldn't be much work at all.
I am not sure what this ScalaObjectMapper does, doesn't look like it is useful.
If you add mapper.registerModule(DefaultScalaModule) in the beginning, it should work ... assuming that by MutableMap you mean mutable.Map, and not some sort of home-made class (because, if you do, you'd have to provide a serializer for it yourself).
(DefaultScalaModule is in jackson-module-scala library. Just add it to your build if you don't already have it).

circe type field not showing

When encoding to Json with circe we really want the type field to show e.g.
scala> val fooJson = foo.asJson
fooJson: io.circe.Json =
{
"this_is_a_string" : "abc",
"another_field" : 123,
"type" : "Foo"
}
This is taken from the release notes which previously mentions that you can configure the encoding like this:
implicit val customConfig: Configuration =
Configuration.default.withSnakeCaseKeys.withDefaults.withDiscriminator("type")
Also other information about circe here suggests that without any configuration you should get some class type information in the encoding json.
Am I missing something? How do you get the class type to show?
UPDATE 30/03/2017: Follow up to OP's comment
I was able to make this work, as shown in the linked release notes.
Preparation step 1: add additional dependency to build.sbt
libraryDependencies += "io.circe" %% "circe-generic-extras" % "0.7.0"
Preparation step 2: setup dummy sealed trait hierarchy
import io.circe.{ Decoder, Encoder }
import io.circe.parser._, io.circe.syntax._
import io.circe.generic.extras.Configuration
import io.circe.generic.extras.auto._
import io.circe.generic.{ semiauto => boring } // <- This is the default generic derivation behaviour
import io.circe.generic.extras.{ semiauto => fancy } // <- This is the new generic derivation behaviour
implicit val customConfig: Configuration = Configuration.default.withDefaults.withDiscriminator("type")
sealed trait Stuff
case class Foo(thisIsAString: String, anotherField: Int = 13) extends Stuff
case class Bar(thisIsAString: String, anotherField: Int = 13) extends Stuff
object Foo {
implicit val decodeBar: Decoder[Bar] = fancy.deriveDecoder
implicit val encodeBar: Encoder[Bar] = fancy.deriveEncoder
}
object Bar {
implicit val decodeBar: Decoder[Bar] = boring.deriveDecoder
implicit val encodeBar: Encoder[Bar] = boring.deriveEncoder
}
Actual code using this:
val foo: Stuff = Foo("abc", 123)
val bar: Stuff = Bar("xyz", 987)
val fooString = foo.asJson.noSpaces
// fooString: String = {"thisIsAString":"abc","anotherField":123,"type":"Foo"}
val barString = bar.asJson.noSpaces
// barString: String = {"thisIsAString":"xyz","anotherField":987,"type":"Bar"}
val bar2 = for{
json <- parse(barString)
bar2 <- json.as[Stuff]
} yield bar2
// bar2: scala.util.Either[io.circe.Error,Stuff] = Right(Bar(xyz,987))
val foo2 = for{
json <- parse(fooString)
foo2 <- json.as[Stuff]
} yield foo2
// foo2: scala.util.Either[io.circe.Error,Stuff] = Right(Foo(abc,123))
So, provided you import the extra dependency (which is where Configuration comes from), it looks like it works.
Finally, as a sidenote, it does seem that there is some disconnection between Circe's DESIGN.md and practice, for which I am actually happy.
Original answer:
I am not sure this is supposed to be supported, by design.
Taken from Circe's DESIGN.md:
Implicit scope should not be used for configuration. Lots of people have asked for a way to configure generic codec derivation to use e.g. a type field as the discriminator for sealed trait hierarchies, or to use snake case for member names. argonaut-shapeless supports this quite straightforwardly with a JsonCoproductCodec type that the user can provide implicitly.
I don't want to criticize this approach—it's entirely idiomatic Scala, and it often works well in practice—but I personally don't like using implicit values for configuration, and I'd like to avoid it in circe until I am 100% convinced that there's no alternative way to provide this functionality.
What this means concretely: You'll probably never see an implicit argument that isn't a type class instance—i.e. that isn't a type constructor applied to a type in your model—in circe, and configuration of generic codec derivation is going to be relatively limited (compared to e.g. argonaut-shapeless) until we find a nice way to do this kind of thing with type tags or something similar.
In particular, customConfig: Configuration seems to be exactly the type of argument that the last paragraph refers to (e.g. an implicit argument that isn't a type class instance)
I am sure that #travis-brown or any other Circe's main contributors could shed some more light on this, in case there was in fact a way of doing this - and I would be very happy to know it! :)

Type-safe generic case class updates in Scala

I'm attempting to write some code that tracks changes to a record and applies them at a later date. In a dynamic language I'd do this by simply keeping a log of List[(String, Any)] pairs, and then simply applying these as an update to the original record when I finally decide to commit the changes.
I need to be able to introspect over the updates, so a list of update functions isn't appropriate.
In Scala this is fairly trivial using reflection, however I'd like to implement a type-safe version.
My first attempt was to try with shapeless. This works well if we know specific types.
import shapeless._
import record._
import syntax.singleton._
case class Person(name:String, age:Int)
val bob = Person("Bob", 31)
val gen = LabelledGeneric[Person]
val updated = gen.from( gen.to(bob) + ('age ->> 32) )
// Result: Person("Bob", 32)
However I can't figure out how to make this work generically.
trait Record[T]
def update( ??? ):T
}
Given the way shapeless handles this, I'm not sure if this would even be possible?
If I accept a lot of boilerplate, as a poor mans version I could do something along the lines of the following.
object Contact {
sealed trait Field[T]
case object Name extends Field[String]
case object Age extends Field[Int]
}
// A typeclass would be cleaner, but too verbose for this simple example.
case class Contact(...) extends Record[Contact, Contact.Field] {
def update[T]( field:Contact.Field[T], value:T ) = field match {
case Contact.Name => contact.copy( name = value )
case Contact.Age => contact.copy( age = value )
}
}
However this isn't particularly elegant and requires a lot of boilerplate. I could probably write my own macro to handle this, however it seems like a fairly common thing - is there a way to handle this with Shapeless or a similar macro library already?
How about using the whole instance of the class as an update?
case class Contact(name: String, age: Int)
case class ContactUpdate(name: Option[String] = None, age: Option[Int] = None)
object Contact {
update(target: Contact, delta: ContactUpdate) = Contact(
delta.name.getOrElse(target.name)
target.age.getOrElse(delta.age)
)
}
// also, optionally this:
object ContactUpdate {
apply(name: String) = ContactUpdate(name = Option(name))
apply(age: Int) = ContactUpdate(age = Option(age))
}
I think, if you want the really type-safe solution, this is the cleanest and most readable, and also, possibly the least pain to implement, as you don't need to deal with Records, lenses and individual field descriptors, just ContactUpdate(name="foo") creates an update, and updates.map(Contact.update(target, _)) applies them all in sequence.