Is it possible to serialize as known default format in custom serializer? - scala

I have to deserialize JSON response that can have one of the fields set to different objects (with just one common field). Real-life model is rather complex, but for example we can represent it by two case classes extending sealed trait:
sealed trait Item {
val itemType: String
}
case class FirstItem(
itemType: String = "FirstItem",
firstProperties: SomeComplexType
) extends Item
case class SecondItem(
itemType: String = "SecondItem",
secondProperties: SomeOtherComplexType,
secondName: String,
secondSize: Int
) extends Item
Since Json4s does not know how to handle that object I wrote custom serializer:
object ItemSerializer extends CustomSerializer[Item](_ => ({
case i: JObject =>
implicit val formats: Formats = DefaultFormats
(i \ "itemType").extract[String] match {
case "FirstType" => i.extract[FirstItem]
case "SecondItem" => i.extract[SecondItem]
}
}, {
case x: Item => x match {
case f: FirstItem => JObject() //TODO
case s: SecondItem => JObject() //TODO
}
}))
First part - deserialization is not perfect as it depends strongly on type-field, but its fine for my needs. The problem is the second part - serialization. In examples I've found people usually write down every field step by step, but usually, they serialize some simple objects. In my case this object has multiple levels and over 60-80 fields in total so it would result in rather messy and hard to read code. So I'm wondering if there is a better way to do it, as both FirstItem and SecondItem can be deserialized using only DefaultFormats. Is there any way to tell Json4s that if object matches the given type it should be serialized with default format?

It took me some digging in various examples and turns out that it is is possible and super simple. There is an org.json4s.Extraction.decompose() method that handles everything e.g.:
object ItemSerializer extends CustomSerializer[Item](_ => ({
case i: JObject =>
implicit val formats: Formats = DefaultFormats
(i \ "itemType").extract[String] match {
case "FirstType" => i.extract[FirstItem]
case "SecondItem" => i.extract[SecondItem]
}
}, {
case x: Item =>
implicit val formats: Formats = DefaultFormats
x match {
case f: FirstItem => Extraction.decompose(f)
case s: SecondItem => Extraction.decompose(s)
}
}))
However, my solution to the described problem was wrong. I don't need to specify extra serializers. What I needed was just a companion object for trait, that contains constructors for each format of data and Json4s handles everything perfectly, e.g.:
object Item{
def apply(
itemType: String,
firstProperties: SomeComplexType
): Item = FirstItem(itemType, firstProperties)
def apply(
itemType: String,
secondProperties: SomeOtherComplexType,
secondName: String, secondSize: Int
): Item = SecondItem(itemType, secondProperties, secondName, secondSize)
}

Related

Encoder for update endpoint of a Rest API

I'm implementing a client for the Keystone API of Openstack. For the Users I have the following classes:
import java.time.OffsetDateTime
import io.circe.derivation.{deriveDecoder, deriveEncoder, renaming}
import io.circe.{Decoder, Encoder}
object User {
object Update {
implicit val encoder: Encoder[Update] = deriveEncoder(renaming.snakeCase)
}
case class Update(
name: Option[String] = None,
password: Option[String] = None,
defaultProjectId: Option[String] = None,
enabled: Option[Boolean] = None,
)
implicit val decoder: Decoder[User] = deriveDecoder(renaming.snakeCase)
}
final case class User(
id: String,
name: String,
domainId: String,
defaultProjectId: Option[String] = None,
passwordExpiresAt: Option[OffsetDateTime] = None,
enabled: Boolean = true,
)
Where User.Update contains the possible fields to update a user. Updates are done using PATCH, or in other words they are partial. The encoders are being used in a class which has the methods to create, update, delete, get, and list the users. This service class uses the encoders in a http4s EntityEncoder with:
import io.circe.{Encoder, Printer}
import org.http4s.{EntityDecoder, circe}
val jsonPrinter: Printer = Printer.noSpaces.copy(dropNullValues = true)
implicit def jsonEncoder[A: Encoder]: EntityEncoder[F, A] = circe.jsonEncoderWithPrinterOf(jsonPrinter)
My problem is how to implement the update for defaultProjectId. In the final json sent to the server the following cases are possible:
Keep the current value of defaultProjectId (the json object does not contain the field default_project_id:
{
(...)
}
Change the defaultProjectId to an-id:
{
(...),
"default_project_id: "an-id",
(...)
}
Unset the defaultProjectId:
{
(...),
"default_project_id: null,
(...)
}
The current implementation: defaultProjectId: Option[String] = None + dropNullValues in the printer, models correctly the cases 1 and 2, but prevents case 3.
Ideally I would have an ADT like:
sealed trait Updatable[+T]
case object KeepExistingValue extends Updatable[Nothing]
case object Unset extends Updatable[Nothing]
case class ChangeTo[T](value: T) extends Updatable[T]
Usage example (probably in the future all fields would be Updatables):
case class Update(
name: Option[String] = None,
password: Option[String] = None,
defaultProjectId: Updatable[String] = KeepExistingValue,
enabled: Option[Boolean] = None,
)
But I can't find a clean way to encode this ADT. Attempted solutions and their problems (they all require not using the printer with dropNullValues in the update method):
Unset is special:
// The generic implementation of Updatable
implicit def updatableEncoder[T](implicit valueEncoder: Encoder[T]): Encoder[Updatable[T]] = {
case KeepExistingValue => Json.Null
case Unset => Json.fromString(Unset.getClass.getName) // Or another arbitrary value
case ChangeTo(value) => valueEncoder(value)
}
// In the service class
def nullifyUnsets(obj: JsonObject): JsonObject = obj.mapValues {
case json if json.asString.contains(Unset.getClass.getName) => Json.Null
case json => json
}
def update(id: String, update: Update): F[Model] = {
// updateEncoder is of type Encoder[Update]
updateEncoder(update).dropNullValues.mapObject(nullifyUnsets)
(...)
}
Pros:
Using the dropNullValues nicely handles the KeepExistingValue case.
If the user invokes dropNullValues to derive the encoder the code still works.
Cons:
Because of dropNullValues the Unset case is meh.
We iterate twice on the Json Object field/values, once for dropNullValues another for mapObject.
Json.fromString(Unset.getClass.getName) is arbitrary and can collide with a legit value for T, although very unlikely.
KeepExistingValue is special:
// The generic implementation of Updatable
implicit def updatableEncoder[T](implicit valueEncoder: Encoder[T]): Encoder[Updatable[T]] = {
case KeepExistingValue => Json.fromString(Unset.getClass.getName) // Or another arbitrary value
case Unset => Json.Null
case ChangeTo(value) => valueEncoder(value)
}
// In the service class
def dropKeepExistingValues(obj: JsonObject): JsonObject = obj.filter{
case (_, json) => !json.asString.contains(Unset.getClass.getName)
}
def update(id: String, update: Update): F[Model] = {
// updateEncoder is of type Encoder[Update]
updateEncoder(update).mapObject(dropKeepExistingValues)
(...)
}
Pros:
Simpler implementation, updatableEncoder implementation maps more directly to the needed Json.
Just one pass over the Json Object field/values.
Cons:
If the programmer invokes dropNullValues to derive the encoder then the code stops working.
Json.fromString(Unset.getClass.getName) is arbitrary and can collide with a legit value for T, although very unlikely.
I'm sure I'm not the first one to hit this problem, but I can't search for it, the best I got is this comment.

How to write a custom decoder for [Option[Option[A]] in Circe?

I had written a Reads converter in play-json for Option[Option[A]] that had the following behavior:
//given this case class
case class MyModel(field: Option[Option[String]])
//this JSON -- maps to --> this MyModel:
//"{ \"field\": \"value\" }" --> MyModel(field = Some(Some("value")))
//"{ \"field\": null, ... }" --> MyModel(field = Some(None))
//"{ }" --> MyModel(field = None)
So, providing the value mapped to Some[Some[A]], providing null mapped to Some[None] (i.e. Some[Option.empty[A]]), and not providing the value mapped to just None (i.e. Option.empty[Option[A]]). Here's the play-json converter:
def readOptOpt[A](implicit r: Reads[A]): Reads[Option[Option[A]]] = {
Reads[Option[Option[A]]] { json =>
path.applyTillLast(json).fold(
identity,
_.fold(_ => JsSuccess(None), {
case JsNull => JsSuccess(Some(None))
case js => r.reads(js).repath(path).map(a => Some(Some(a)))
})
)
}
}
Now I am converting my play-json code to Circe, but I can't figure out how to write a Decoder[Option[Option[A]] that has the same behavior. That is, I need
def optOptDecoder[A](implicit d: Decoder[A]): Decoder[Option[Option[A]] = ??? //help!
Any ideas on how I can make this work? Thanks
I figured this out:
There were two problems:
1) How to deal with the case where the field was completely missing from the JSON. Turns out you have to use Decoder.reattempt in your custom decoder, following Circe's decodeOption code, which works.
2) How to have the compiler recognize cases of Option[Option[A]] when your decoder code is sitting in a helper object (or wherever). Turns out if you're using semi-auto derivation, you can create an implicit in the companion object and that will override the defaults:
//companion object
object MyModel {
implicit def myModelOptOptDecoder[A](implicit d: Decoder[A]): Decoder[Option[Option[A]]] =
MyHelperObject.optOptDecoder
implicit val myModelDecoder: Decoder[MyModel] = deriveDecoder
}
Anyway, I don't think this will be much help to anybody in the future, so unless I get any upvotes in the next few hours I think I'll just delete this.
Edit2: Okay it was answered so I won't delete it. Stay strong, esoteric circe question, stay strong...
An Option[Option[A]] is a bit odd. I understand and mostly agree with the reasoning, but I think it's weird enough that it may warrant just replacing it with your own class (and writing a decoder for that). Something like:
sealed trait OptionalNull[+A] {
def toOption: Option[Option[A]]
}
object NotPresent extends OptionalNull[Nothing] {
override def toOption = None
}
object PresentButNull extends OptionalNull[Nothing] {
override def toOption = Some(None)
}
case class PresentNotNull[A](value: A) extends OptionalNull[A] {
override def toOption = Some(Some(value))
}
This has the additional benefit of not having to worry about implicit precedence and stuff like that. Might simplify your decoder.
Here is another solution I found (This is not my gist):
sealed trait UpdateOrDelete[+A]
case object Delete extends UpdateOrDelete[Nothing]
final case class UpdateOptionalFieldWith[A](value: A) extends UpdateOrDelete[A]
object UpdateOrDelete {
implicit def optionalDecoder[A](implicit decodeA: Decoder[A]): Decoder[UpdateOptionalField[A]] =
Decoder.withReattempt {
// We're trying to decode a field but it's missing.
case c: FailedCursor if !c.incorrectFocus => Right(None)
case c =>
Decoder.decodeOption[A].tryDecode(c).map {
case Some(a) => Some(UpdateOptionalFieldWith(a))
case None => Some(Delete)
}
}
// Random UUID to _definitely_ avoid collisions
private[this] val marker: String = s"$$marker-${UUID.randomUUID()}-marker$$"
private[this] val markerJson: Json = Json.fromString(marker)
implicit def optionalEncoder[A](implicit encodeA: Encoder[A]): Encoder[UpdateOptionalField[A]] =
Encoder.instance {
case Some(Delete) => Json.Null
case Some(UpdateOptionalFieldWith(a)) => encodeA(a)
case None => markerJson
}
def filterMarkers[A](encoder: Encoder.AsObject[A]): Encoder.AsObject[A] =
encoder.mapJsonObject { obj =>
obj.filter {
case (_, value) => value =!= markerJson
}
}
}

required: spray.httpx.marshalling.ToResponseMarshallable Error

Hey im pretty new to Spray and reactive mongo .
Im trying to return a list of result as json but i'm having some issue with converting the result to list of json.
this is my model
import reactivemongo.bson.BSONDocumentReader
import reactivemongo.bson.BSONObjectID
import reactivemongo.bson.Macros
case class Post(id: BSONObjectID, likes: Long, message: String, object_id: String, shares: Long)
object Post {
implicit val reader: BSONDocumentReader[Post] = Macros.reader[Post]
}
the Mongo method
def getAll(): Future[List[Post]] ={
val query = BSONDocument(
"likes" -> BSONDocument(
"$gt" -> 27))
collection.find(query).cursor[Post].collect[List]()
}
and this is the route
val route1 =
path("posts") {
val res: Future[List[Post]]= mongoService.getAll()
onComplete(res) {
case Success(value) => complete(value)
case Failure(ex) => complete(ex.getMessage)
}
}
error
type mismatch; found : List[com.example.model.Post] required: spray.httpx.marshalling.ToResponseMarshallable
thanks,
miki
You'll need to define how a Post will be serialized, which you can do via a spray-json Protocol (see the docs for more detailed information). It's quite easy to do so, but before that, you'll also need to define a format for the BSONObjectId type, since there's no built-in support for that type in spray-json (alternatively, if object_id is a string representation of the BSONObjectId, think about removing the id property from your Post class or change it to be a String):
// necessary imports
import spray.json._
import spray.httpx.SprayJsonSupport._
implicit object BSONObjectIdProtocol extends RootJsonFormat[BSONObjectID] {
override def write(obj: BSONObjectID): JsValue = JsString(obj.stringify)
override def read(json: JsValue): BSONObjectID = json match {
case JsString(id) => BSONObjectID.parse(id) match {
case Success(validId) => validId
case _ => deserializationError("Invalid BSON Object Id")
}
case _ => deserializationError("BSON Object Id expected")
}
}
Now, we're able to define the actual protocol for the Post class:
object PostJsonProtocol extends DefaultJsonProtocol {
implicit val format = jsonFormat5(Post.apply)
}
Furthermore, we'll also need to make sure that we have the defined format in scope:
import PostJsonProtocol._
Now, everything will compile as expected.
One more thing: have a look at the docs about the DSL structure of spray. Your mongoService.getAll() isn't within a complete block, which might not reflect your intentions. This ain't an issue yet, but probably will be if your route get more complex. To fix this issue simply put the future into the onComplete call or make it lazy:
val route1 =
path("posts") {
onComplete(mongoService.getAll()) {
case Success(value) => complete(value)
case Failure(ex) => complete(ex.getMessage)
}
}

How to represent optional fields in spray-json?

I have an optional field on my requests:
case class SearchRequest(url: String, nextAt: Option[Date])
My protocol is:
object SearchRequestJsonProtocol extends DefaultJsonProtocol {
implicit val searchRequestFormat = jsonFormat(SearchRequest, "url", "nextAt")
}
How do I mark the nextAt field optional, such that the following JSON objects will be correctly read and accepted:
{"url":"..."}
{"url":"...", "nextAt":null}
{"url":"...", "nextAt":"2012-05-30T15:23Z"}
I actually don't really care about the null case, but if you have details, it would be nice. I'm using spray-json, and was under the impression that using an Option would skip the field if it was absent on the original JSON object.
Works for me (spray-json 1.1.1 scala 2.9.1 build)
import cc.spray.json._
import cc.spray.json.DefaultJsonProtocol._
// string instead of date for simplicity
case class SearchRequest(url: String, nextAt: Option[String])
// btw, you could use jsonFormat2 method here
implicit val searchRequestFormat = jsonFormat(SearchRequest, "url", "nextAt")
assert {
List(
"""{"url":"..."}""",
"""{"url":"...", "nextAt":null}""",
"""{"url":"...", "nextAt":"2012-05-30T15:23Z"}""")
.map(_.asJson.convertTo[SearchRequest]) == List(
SearchRequest("...", None),
SearchRequest("...", None),
SearchRequest("...", Some("2012-05-30T15:23Z")))
}
You might have to create an explicit format (warning: psuedocodish):
object SearchRequestJsonProtocol extends DefaultJsonProtocol {
implicit object SearchRequestJsonFormat extends JsonFormat[SearchRequest] {
def read(value: JsValue) = value match {
case JsObject(List(
JsField("url", JsString(url)),
JsField("nextAt", JsString(nextAt)))) =>
SearchRequest(url, Some(new Instant(nextAt)))
case JsObject(List(JsField("url", JsString(url)))) =>
SearchRequest(url, None)
case _ =>
throw new DeserializationException("SearchRequest expected")
}
def write(obj: SearchRequest) = obj.nextAt match {
case Some(nextAt) =>
JsObject(JsField("url", JsString(obj.url)),
JsField("nextAt", JsString(nextAt.toString)))
case None => JsObject(JsField("url", JsString(obj.url)))
}
}
}
Use NullOptions trait to disable skipping nulls:
https://github.com/spray/spray-json#nulloptions
Example:
https://github.com/spray/spray-json/blob/master/src/test/scala/spray/json/ProductFormatsSpec.scala
Don't know if this will help you but you can give that field a default value in the case class definition, so if the field is not in the json, it will assign the default value to it.
Easy.
import cc.spray.json._
trait MyJsonProtocol extends DefaultJsonProtocol {
implicit val searchFormat = new JsonWriter[SearchRequest] {
def write(r: SearchRequest): JsValue = {
JsObject(
"url" -> JsString(r.url),
"next_at" -> r.nextAt.toJson,
)
}
}
}
class JsonTest extends FunSuite with MyJsonProtocol {
test("JSON") {
val search = new SearchRequest("www.site.ru", None)
val marshalled = search.toJson
println(marshalled)
}
}
For anyone who is chancing upon this post and wants an update to François Beausoleil's answer for newer versions of Spray (circa 2015+?), JsField is deprecated as a public member of JsValue; you should simply supply a list of tuples instead of JsFields. Their answer is spot-on, though.

Scala Lift - Mongodb save data as json object

I have the following model
class Recording private() extends MongoRecord[Recording] with ObjectIdPk[Recording] {
def meta = Recording
object data extends StringField(this, 50)
}
I'm currently saving a Json object as a string in the "data" field, I've used JsonObject field before but only with predefined object structures. In this case the json object being saved can have any structure or data fields so a predefined data structure is not an option.
Say I have:
{"name" : "James", "value" : "Hai!"}
Or
{"result" : 1, "handle" : "lorem_ipsum"}
I need to be able to save both as a json object in the same field, "data".
Is there a way I can do this?
Thanks in advance for any help, much appreciated :)
What might work for you is storing the data as a JValue rather than a String. You could use a JsonObjectField wrapping a case class that contains a JValue, which would allow an arbitrary structure, but it will give you an extra level of nesting in Mongo. To get around that, how about creating a custom field just to hold a JValue?
One stab at it:
abstract class JValueObjectField[OwnerType <: BsonRecord[OwnerType]](rec: OwnerType)
extends Field[JValue,OwnerType]
with MandatoryTypedField[JValue]
with MongoFieldFlavor[JValue] {
def owner = rec
def defaultValue = JNothing
def toForm: Box[NodeSeq] = Empty
implicit val formats = owner.meta.formats
def asJValue: JValue = valueBox openOr JNothing
def setFromJValue(jv: JValue): Box[JValue] = Full(jv)
def setFromString(in: String): Box[JValue] = tryo(JsonParser.parse(in)) match {
case Full(jv: JValue) => setFromJValue(jv)
case f: Failure => setBox(f)
case other => setBox(Failure("Error parsing String into a JValue: "+in))
}
def setFromAny(in: Any): Box[JValue] = in match {
case dbo: DBObject => setFromDBObject(dbo)
case value: JValue => setBox(Full(value))
case Some(value: JValue) => setBox(Full(value))
case Full(value: JValue) => setBox(Full(value))
case (value: JValue) :: _ => setBox(Full(value))
case s: String => setFromString(s)
case Some(s: String) => setFromString(s)
case Full(s: String) => setFromString(s)
case null|None|Empty => setBox(defaultValueBox)
case f: Failure => setBox(f)
case o => setFromString(o.toString)
}
def asDBObject: DBObject = JObjectParser.parse(asJValue.asInstanceOf[JObject])
def setFromDBObject(dbo: DBObject): Box[JValue] =
setFromJValue(JObjectParser.serialize(dbo))
}
...which looks a lot: all I've done is cut and paste from JValueObjectField with one parameter removed and fixed at JValue. There may be a smarter way to do that.
You can then use this in your model:
object data extends JValueObjectField(this)
I'd populate it using the lift-json DSL:
val json = ("name" -> "Bob") ~ ("result" -> 1)
Recording.createRecord.data(json).save
This will give you something in your MongoDB document like this:
"data" : {
"name" : "Bob",
"result" : 1
}
BTW, the Lift Mailing list is a good way to get a better answer: that just happens to be where most of the Lift-related people seem to congregate.
class Recording private() extends MongoRecord[Recording] with ObjectIdPk[Recording] {
def meta = Recording
object data extends JObjectField(this)
}