Decoding a sealed trait in Argonaut based on JSON structure? - scala

Given the following example:
sealed trait Id
case class NewId(prefix: String, id: String) extends Id
case class RevisedId(prefix: String, id: String, rev: String) extends Id
case class User(key: Id, name: String)
val json = """
{
"key": {
"prefix": "user",
"id": "Rt01",
"rev": "0-1"
},
"name": "Bob Boberson"
}
"""
implicit val CodecUser: CodecJson[User] = casecodec2(User.apply, User.unapply)("key", "name")
implicit val CodecId: CodecJson[Id] = ???
json.decodeOption[User]
I need to write a CodecJson for Id that will decode an object when it has the proper structure.
Adding a discriminator field of some sort is a common suggestion for this, but I don't want to change the JSON I'm already producing/consuming with spray-json and json4s.
In those libraries your encoders/decoders are basically just PartialFunction[JValue, A] and PartialFunction[A, JValue]. If your value isn't defined in the domain it's a failure. It's a really simple, elegant solution I think. In addition to that you've got Extractors for the JSON types, so it's easy to match an object on fields/structure.
Rapture goes a step further, making field order unimportant and ignoring the presence of non-matching fields, so you could just do something like:
case json"""{ "prefix": $prefix, "id": $id, "rev": $rev }""" =>
RevisedId(prefix, id, rev)
That's really simple/powerful.
I'm having trouble figuring out how to do something similar with argonaut. This is the best I've come up with so far:
val CodecNewId = casecodec2(NewId.apply, NewId.unapply)("prefix", "id")
val CodecRevisedId = casecodec3(RevisedId.apply, RevisedId.unapply)("prefix", "id", "rev")
implicit val CodecId: CodecJson[Id] =
CodecJson.derived[Id](
EncodeJson {
case id: NewId => CodecNewId(id)
case id: IdWithRev => RevisedId(id)
},
DecodeJson[Id](c => {
val q = RevisedId(c).map(a => a: Id)
q.result.fold(_ => CodecNewId(c).map(a => a: Id), _ => q)
})
)
So there's a few problems with that. I have to define extra codecs I don't intend to use. Instead of using the case-class extractors in the EncodeJson for CodecJson[Id], I'm delegating to the other encoders I've defined. Just just doesn't feel very straight-forward or clean for classes that only have 2 or 3 fields.
The code for the DecodeJson section is also pretty messy. Aside from an extra type-cast in the ifEmpty side of the fold it's identical to the code in DecodeJson.|||.
Does anyone have a more idiomatic way to write a basic codec for Sum-types in argonaut that doesn't require a discriminator and can instead match on the structure of the json?

This is the best I've been able to come up with. It doesn't have that fundamental feel of elegance that the partial functions do, but it is a good bit more terse and easier to decipher than my first attempt.
val CodecNewId = casecodec2(NewId.apply, NewId.unapply)("prefix", "id")
val CodecRevisedId = casecodec3(RevisedId.apply, RevisedId.unapply)("prefix", "id", "rev")
implicit val CodecId: CodecJson[Id] = CodecJson(
{
case id: NewId => CodecNewId(id)
case id: RevisedId => CodecRevisedId(id)
},
(CodecRevisedId ||| CodecNewId.map(a => a: Id))(_))
We're still using the "concrete" codecs for each sub-type. But we've gotten away from the CodecJson.derive call, we don't need to wrap our encode function in an EncodeJson, and we can map our DecodeJson function instead of type-casting so we can go back to using ||| instead of copying it's implementation, which makes the code a lot more readable.
This is definitely a usable solution, if not quite what I'd hoped for.

Related

How to "reads" into a Scala Case Class given a Json object with key names that start with a capital letter

This question is based upon Scala 2.12.12
scalaVersion := "2.12.12"
using play-json
"com.typesafe.play" %% "play-json" % "2.9.1"
If I have a Json object that looks like this:
{
"UpperCaseKey": "some value",
"AnotherUpperCaseKey": "some other value"
}
I know I can create a case class like so:
case class Yuck(UpperCaseKey: String, AnotherUpperCaseKey: String)
and follow that up with this chaser:
implicit val jsYuck = Json.format[Yuck]
and that will, of course, give me both reads[Yuck] and writes[Yuck] to and from Json.
I'm asking this because I have a use case where I'm not the one deciding the case of the keys and I've being handed a Json object that is full of keys that start with an uppercase letter.
In this use case I will have to read and convert millions of them so performance is a concern.
I've looked into #JsonAnnotations and Scala's transformers. The former doesn't seem to have much documentation for use in Scala at the field level and the latter seems to be a lot of boilerplate for something that might be very simple another way if I only knew how...
Bear in mind as you answer this that some Keys will be named like this:
XXXYyyyyZzzzzz
So the predefined Snake/Camel case conversions will not work.
Writing a custom conversion seems to be an option yet unsure how to do that with Scala.
Is there a way to arbitrarily request that the Json read will take Key "XXXYyyyZzzz" and match it to a field labeled "xxxYyyyZzzz" in a Scala case class?
Just to be clear I may also need to convert, or at least know how, a Json key named "AbCdEf" into field labeled "fghi".
Just use provided PascalCase.
case class Yuck(
upperCaseKey: String,
anotherUpperCaseKey: String)
object Yuck {
import play.api.libs.json._
implicit val jsonFormat: OFormat[Yuck] = {
implicit val cfg = JsonConfiguration(naming = JsonNaming.PascalCase)
Json.format
}
}
play.api.libs.json.Json.parse("""{
"UpperCaseKey": "some value",
"AnotherUpperCaseKey": "some other value"
}""").validate[Yuck]
// => JsSuccess(Yuck(some value,some other value),)
play.api.libs.json.Json.toJson(Yuck(
upperCaseKey = "foo",
anotherUpperCaseKey = "bar"))
// => JsValue = {"UpperCaseKey":"foo","AnotherUpperCaseKey":"bar"}
I think that the only way play-json support such a scenario, is defining your own Format.
Let's assume we have:
case class Yuck(xxxYyyyZzzz: String, fghi: String)
So we can define Format on the companion object:
object Yuck {
implicit val format: Format[Yuck] = {
((__ \ "XXXYyyyZzzz").format[String] and (__ \ "AbCdEf").format[String]) (Yuck.apply(_, _), yuck => (yuck.xxxYyyyZzzz, yuck.fghi))
}
}
Then the following:
val jsonString = """{ "XXXYyyyZzzz": "first value", "AbCdEf": "second value" }"""
val yuck = Json.parse(jsonString).validate[Yuck]
println(yuck)
yuck.map(yuckResult => Json.toJson(yuckResult)).foreach(println)
Will output:
JsSuccess(Yuck(first value,second value),)
{"XXXYyyyZzzz":"first value","AbCdEf":"second value"}
As we can see, XXXYyyyZzzz was mapped into xxxYyyyZzzz and AbCdEf into fghi.
Code run at Scastie.
Another option you have, is to usd JsonNaming, as #cchantep suggested in the comment. If you define:
object Yuck {
val keysMap = Map("xxxYyyyZzzz" -> "XXXYyyyZzzz", "fghi" -> "AbCdEf")
implicit val config = JsonConfiguration(JsonNaming(keysMap))
implicit val fotmat = Json.format[Yuck]
}
Running the same code will output the same. Code ru nat Scastie.

How to remove null values in list of objects using Circe

I'm trying to encode a list of objects using Circe, something that looks similar to:
val test = Seq(MyObject("hello", None, 1, 2, None)
I'm trying to parse this using Circe:
test.asJson
But this creates the JSON object:
[
{
name: "hello",
someVal: null,
someNum: 1,
anotherNum: 2,
anotherVal: null
}
]
I've tried running asJson with .dropNullValues, but that doesn't seem to access the null values inside of the object. Is there a way to drop the null values inside of the objects?
I'm expecting something more like this:
[
{
name: "hello",
someNum: 1,
anotherNum: 2
}
]
Circe provides the function deepDropNullValues in the Json class.
Example: test.asJson.deepDropNullValues
You see field: null because circe turns Option[T] to t.asJson on Some[T] and JsonNull on None, and default case class encoder just puts all fields to the JsonObject. In a way that circe uses to encode sealed trait family, it may use these null fields to distinguish classes like
sealed trait Foo
case class Bar(a: Option[String])
case class Baz(a: Option[String], b: Option[String])
So, if you really want to drop this information and need one-way conversion with information loss, you can map resulting Json to drop all and every null field with code like that:
implicit val fooEnc: Encoder[Foo] = deriveEncoder[Foo].mapJsonObject{jsonObj => jsonObj.filter{case (k,v) => !v.isNull}}
However, you should write such a custom codec for any class you want to drop null fields. To post-process your json, you can use fold on resulting json:
val json: Json = ???
json.fold[Json](
Json.Null,
Json.fromBoolean,
{_.asJson},
{_.asJson},
{_.asJson},
{jsonObj => jsonObj.filter{case (k,v) => !v.isNull}.asJson}
)
or implement a custom folder.
The answer that was suggested by #iva-kmm is good, but it can be done better!
This mechanic is already implemented in circe, just call it:
implicit val fooEnc: Encoder[Foo] = deriveEncoder[Foo].mapJson(_.dropNullValues) // or _.deepDropNullValues

Creating a `Decoder` for arbitrary JSON

I am building a GraphQL endpoint for an API using Finch, Circe and Sangria. The variables that come through in a GraphQL query are basically an arbitrary JSON object (let's assume there's no nesting). So for example, in my test code as Strings, here are two examples:
val variables = List(
"{\n \"foo\": 123\n}",
"{\n \"foo\": \"bar\"\n}"
)
The Sangria API expects a type for these of Map[String, Any].
I've tried a bunch of ways but have so far been unable to write a Decoder for this in Circe. Any help appreciated.
The Sangria API expects a type for these of Map[String, Any]
This is not true. Variables for an execution in sangria can be of an arbitrary type T, the only requirement that you have an instance of InputUnmarshaller[T] type class for it. All marshalling integration libraries provide an instance of InputUnmarshaller for correspondent JSON AST type.
This means that sangria-circe defines InputUnmarshaller[io.circe.Json] and you can import it with import sangria.marshalling.circe._.
Here is a small and self-contained example of how you can use circe Json as a variables:
import io.circe.Json
import sangria.schema._
import sangria.execution._
import sangria.macros._
import sangria.marshalling.circe._
val query =
graphql"""
query ($$foo: Int!, $$bar: Int!) {
add(a: $$foo, b: $$bar)
}
"""
val QueryType = ObjectType("Query", fields[Unit, Unit](
Field("add", IntType,
arguments = Argument("a", IntType) :: Argument("b", IntType) :: Nil,
resolve = c ⇒ c.arg[Int]("a") + c.arg[Int]("b"))))
val schema = Schema(QueryType)
val vars = Json.obj(
"foo" → Json.fromInt(123),
"bar" → Json.fromInt(456))
val result: Future[Json] =
Executor.execute(schema, query, variables = vars)
As you can see in this example, I used io.circe.Json as variables for an execution. The execution would produce following result JSON:
{
"data": {
"add": 579
}
}
Here's a decoder that works.
type GraphQLVariables = Map[String, Any]
val graphQlVariablesDecoder: Decoder[GraphQLVariables] = Decoder.instance { c =>
val variablesString = c.downField("variables").focus.flatMap(_.asString)
val parsedVariables = variablesString.flatMap { str =>
val variablesJsonObject = io.circe.jawn.parse(str).toOption.flatMap(_.asObject)
variablesJsonObject.map(j => j.toMap.transform { (_, value: Json) =>
val transformedValue: Any = value.fold(
(),
bool => bool,
number => number.toDouble,
str => str,
array => array.map(_.toString),
obj => obj.toMap.transform((s: String, json: Json) => json.toString)
)
transformedValue
})
}
parsedVariables match {
case None => left(DecodingFailure(s"Unable to decode GraphQL variables", c.history))
case Some(variables) => right(variables)
}
}
We basically parse the JSON, turn it into a JsonObject, then transform the values within the object fairly simplistically.
Although the above answers work for the specific case of Sangria, I'm interested in the original question: What's the best approach in Circe (which generally assumes that all types are known up front) for dealing with arbitrary chunks of Json?
It's fairly common when encoding/decoding Json that 95% of the Json is specified, but the last 5% is some type of "additional properties" chunk which can be any Json object.
Solutions I've played with:
Encode/Decode the free-form chunk as Map[String,Any]. This means you'll have to introduce implicit encoders/decoders for Map[String, Any], which can be done, but is dangerous as that implicit can be pulled into places you didn't intend.
Encode/Decode the free-form chunk as Map[String, Json]. This is the easiest approach and is supported out of the box in Circe. But now the Json serialization logic has leaked out into your API (often you'll want to keep the Json stuff completely wrapped, so you can swap in other non-json formats later).
Encode/Decode to a String, where the string is required to be a valid Json chunk. At least you haven't locked your API into a specific Json library, but it doesn't feel very nice to have to ask your users to create Json chunks in this manual way.
Create a custom trait hierarchy to hold the data (e.g sealed trait Free; FreeInt(i: Int) extends Free; FreeMap(m: Map[String, Free] extends Free; ...). Now you you can create specific encoders/decoders for it. But what you've really done is replicate the Json type hierarchy that already exists in Circe.
I'm leaning more toward option 3. Since it's the most flexible, and will introduce the least dependencies in the API. But none of them are entirely satisfying. Any other ideas?

Conversion from bson.Document to JsObject and applying reads on it with joda.DateTime

I am currently integrating part of our system with MongoDB and we decided to use the official scala driver for it.
We have case class with joda.DateTime as parameters:
case class Schema(templateId: Muid,
createdDate: DateTime,
updatedDate: DateTime,
columns: Seq[Column])
We also defined format for it:
implicit lazy val checklistSchemaFormat : Format[Schema] = (
(__ \ "templateId").format[Muid] and
(__ \ "createdDate").format[DateTime] and
(__ \ "updatedDate").format[DateTime] and
(__ \ "columns").format[Seq[Column]]
)((Schema.apply _), unlift(Schema.unapply))
When I serialize this object to json and write to mongo, the createdDate and updatedDate getting converted to Long (which is technically fine). And this is how we do it:
val bsonDoc = Document(Json.toJson(schema).toString())
collection(DbViewSchemasCollectionName).replaceOne(filter, bsonDoc, new UpdateOptions().upsert(true)).subscribe(new Observer[UpdateResult] {
override def onNext(result: UpdateResult): Unit = logger.info(s"Successfully updates checklist template schema with result: $result")
override def onError(e: Throwable): Unit = logger.info(s"Failed to update checklist template schema with error: $e")
override def onComplete(): Unit = {}
})
as a result Mongo has this type of object:
{
"_id": ObjectId("56fc4247eb3c740d31b04f05"),
"templateId": "gaQP3JIB3ppJtro9rO9BAw",
"createdDate": NumberLong(1459372615507),
"updatedDate": NumberLong(1459372615507),
"columns": [
...
]
}
Now, I am trying to read it like so:
collection(DbViewSchemasCollectionName).find(filter).first().head() map { document =>
ChecklistSchema.checklistSchemaFormat reads Json.parse(document.toJson()) match {
case JsSuccess(value, _) => {
Some(value)
}
case JsError(e) => throw JsResultException(e)
}
} recover { case e =>
logger.info("ERROR " + e)
None
}
And at this point the reads always failing, since the createdDate and updatedDate now look like this:
"createdDate" : { "$numberLong" : "1459372615507" }, "updatedDate" :
{"$numberLong" : "1459372615507" }
How do I deal with this situation? Is there any easier conversion between bson.Document and JsObject? Or I am completely digging into the wrong direction...
Thanks,
You can use the following approach to resolve your issue.
Firstly, I used json4s for reading the writing json to case classes
example:
case class User(_id: Option[Int], username: String, firstName: String, createdDate: DateTime , updatedDate: DateTime )
// A small wapper to convert case class to Document
def toBson[A <: AnyRef](x : A):Document = {
val json = write[A](x)
Document(json) }
def today() = DateTime.now
val user = User(Some(212),"binkabir","lacmalndl", today , today)
val bson = toBson(user)
usersCollection.insertOne(bson).subscribe((x: Completed) => println(x))
val f = usersCollection.find(equal("_id",212)).toFuture()
f.map(_.head.toJson).map( x => read[User](x)).foreach(println)
The code above will create a user case class, convert to Document, save to mongo db, query the db and print the returned User case class
I hope this makes sense!
To answer your second question (bson.Document <-> JsObject) - YES; this is a solved problem, check out Play2-ReactiveMongo, which makes it seem like you're storing/retrieving JsObject instances - plus it's fully asynchronous and super-easy to get going in a Play application.
You can even go a step further and use a library like Mondrian (full disclosure: I wrote it!) to get the basic CRUD operations on top of ReactiveMongo for your Play-JSON domain case-classes.
Obviously I'm biased, but I think these solutions are a great fit if you've already defined your models as case classes in Play - you can forget about the whole BSONDocument family and stick to Json._ and JsObject etc that you already know well.
EDIT:
At the risk of further downvotes, I will demonstrate how I would go about storing and retrieving the OP's Schema object, using Mondrian. I'm going to show pretty-much everything for completeness; bear in mind you've actually already done most of this. Your final code will have fewer lines than your current code, as you'd expect when you use a library.
Model Objects
There's a Column class here that's never mentioned, for simplicity let's just say that's:
case class Column (name:String, width:Int)
Now we can get on with the Schema, which is just:
import com.themillhousegroup.mondrian._
case class Schema(_id: Option[MongoId],
createdDate: DateTime,
updatedDate: DateTime,
columns: Seq[Column]) extends MongoEntity
So far, we've just implemented the MongoEntity trait, which just required the templateId field to be renamed and given the required type.
JSON Converters
import com.themillhousegroup.mondrian._
import play.api.libs.json._
import play.api.libs.functional.syntax._
object SchemaJson extends MongoJson {
implicit lazy val columnFormat = Json.format[Column]
// Pick one - easy:
implicit lazy val schemaFormat = Json.format[Schema]
// Pick one - backwards-compatible (uses "templateId"):
implicit lazy val checklistSchemaFormat : Format[Schema] = (
(__ \ "templateId").formatNullable[MongoId] and
(__ \ "createdDate").format[DateTime] and
(__ \ "updatedDate").format[DateTime] and
(__ \ "columns").format[Seq[Column]]
)((Schema.apply _), unlift(Schema.unapply))
}
The JSON converters are standard Play-JSON stuff; we pick up the MongoId Format by extending MongoJson. I've shown two different ways of defining the Format for Schema. If you have clients out in the wild using templateId (or if you prefer it) then use the second, more verbose declaration.
Service layer
For brevity I'll skip the application-configuration, you can read the Mondrian README.md for that. Let's define the SchemaService that is responsible for persistence operations on Schema instances:
import com.themillhousegroup.mondrian._
import SchemaJson._
class SchemaService extends TypedMongoService[Schema]("schemas")
That's it. We've linked the model object, the name of the MongoDB collection ("schemas") and (implicitly) the necessary converters.
Saving a Schema and finding a Schema based on some criteria
Now we start to realize the value of Mondrian. save and findOne are standard operations - we get them for free in our Service, which we inject into our controllers in the standard way:
class SchemaController #Inject (schemaService:SchemaService) extends Controller {
...
// Returns a Future[Boolean]
schemaService.save(mySchema).map { saveOk =>
...
}
...
...
// Define a filter criteria using standard Play-JSON:
val targetDate = new DateTime()
val criteria = Json.obj("createdDate" -> Json.obj("$gt" ->targetDate.getMillis))
// Returns a Future[Option[Schema]]
schemaService.findOne(criteria).map { maybeFoundSchema =>
...
}
}
So there we go. No sign of the BSON family, just the Play JSON that, as you say, we all know and love. You'll only need to reach for the Mongo documentation when you need to construct a JSON query (that $gt stuff) although in some cases you can use Mondrian's overloaded findOne(example:Schema) method if you are just looking for a simple object match, and avoid even that :-)

Generic synchronisation design

We are building some sync functionality using two-way json requests and this algorithm. All good and we have it running in prototype mode. Now I am trying to genericise the code, as we will be synching for several tables in the app. It would be cool to be able to define a class as "extends Synchable" and get the additional attributes and sync processing methods with a few specialisations/overrides. I have got this far:
abstract class Synchable [T<:Synchable[T]] (val ruid: String, val lastSyncTime: String, val isDeleted:Int) {
def contentEquals(Target: T): Boolean
def updateWith(target: T)
def insert
def selectSince(clientLastSyncTime: String): List[T]
def findByRuid(ruid: String): Option[T]
implicit val validator: Reads[T]
def process(clientLastSyncTime: String, updateRowList: List[JsObject]) = {
for (syncRow <- updateRowList) {
val validatedSyncRow = syncRow.validate[Synchable]
validatedSyncRow.fold(
valid = { result => // valid row
findByRuid(result.ruid) match { //- do we know about it?
case Some(knownRow) => knownRow.updateWith(result)
case None => result.insert
}
}... invalid, etc
I am new to Scala and know I am probably missing things - WIP!
Any pointers or suggestions on this approach would be much appreciated.
Some quick ones:
Those _ parameters you pass in and then immediately assign to vals: why not do it in one hit? e.g.
abstract class Synchable( val ruid: String = "", val lastSyncTime: String = "", val isDeleted: Int = 0) {
which saves you a line and is clearer in intent as well I think.
I'm not sure about your defaulting of Strings to "" - unless there's a good reason (and there often is), I think using something like ruid:Option[String] = None is more explicit and lets you do all sorts of nice monad-y things like fold, map, flatMap etc.
Looking pretty cool otherwise - the only other thing you might want to do is strengthen the typing with a bit of this.type magic so you'll prevent incorrect usage at compile-time. With your current abstract class, nothing prevents me from doing:
class SynchableCat extends Synchable { ... }
class SynchableDog extends Synchable { ... }
val cat = new SynchableCat
val dog = new SynchableDog
cat.updateWith(dog) // This won't end well
But if you just change your abstract method signatures to things like this:
def updateWith(target: this.type)
Then the change ripples down through the subclasses, narrowing down the types, and the compiler will omit a (relatively clear) error if I try the above update operation.