I am trying to parse MongoDB Extended JSON using Circe JSON Parser, works fine in most of the cases except for special datatypes, for eg. in below case class i have priorityOrder which is of long datatype.
case class relinfo(id:String,assetId:String,insureeId:String,queue:String,priorityOrder:Long) extends baseDomain
But when it is converted to MongoDB JSON format, it is converted to special mongo format described below (check priorityOrder field)
{
"_id" : "4abf009d-64b1-496c-b0e8-9061f5e183a0",
"id" : "4abf009d-64b1-496c-b0e8-9061f5e183a0",
"assetId" : "e26d5310-ab0c-4672-9971-4babd3420302",
"insureeId" : "cdee05a1-a09c-4e10-81df-c3f112298cc3",
"queue" : "Low",
"priorityOrder" : {
"$numberLong" : "1930926795621"
}
}
The challenge is during de-serialization process, if i try to take this JSON and convert back to concrete object type using circe parser then it fails to map priorityOrder attribute, is there any way i can write custom encoder/decoder that will treat long data type in a special manner. The custom encoder/decoder will read value from "$numberLong" nested type and convert that value to Long datatype.
I get this exception from circe parser
Left(DecodingFailure(Long, List(El(DownField(priorityOrder),true,false))))
I was able to figure out solution to this problem by creating custom decoder for long datatype. Here is the code for individuals in similar boat
implicit val decodeLong: Decoder[Long] = new Decoder[Long] {
final def apply(c: HCursor): Decoder.Result[Long] =
{
val longval = c.downField("$numberLong").as[String] match
{
case Right(x) => x.toLong
case _ => throw new Exception("Unable to find $numberLong")
}
Right(longval)
}
}
Related
I have a JSON object that I've transformed that I need to filter down to only a subset of its original keys. I've looked through the docs for the Json object in circe but it doesn't appear to expose any API around filtering the object. Do I have to use a cursor for this? I considered creating a decoder from a case class however my keys have a special character . in them. Here is some more code/data for context.
{
"field.nested.this": "value",
"field.nested.that": "value",
"field.nested.where": "value"
}
What's the best approach to create a new JSON instance that doesn't contain the field.nested.that field?
I'm not sure if this is what you need:
object Circe extends App {
import io.circe._
import io.circe.literal._
import io.circe.syntax._
//I'm using a json literal here.
//If you have a runtime string from an external source
// you would need to parse it with `io.circe.parser.parse` first
val json: Json = json"""
{
"field.nested.this": "value",
"field.nested.that": "value",
"field.nested.where": "value"
}
"""
val maybeJsonFiltered =
json.asObject.map(_.filterKeys(_ != "field.nested.that").asJson)
println(maybeJsonFiltered)
// Some({
// "field.nested.this" : "value",
// "field.nested.where" : "value"
// })
}
Alternatively you could also parse it as a map (json.as[Map[String, String]]) or a custom case class with only the fields you need, and encode them back to json. You will probably need a #JsonKey annotation for all your fields with ..
When parsing a Json String, there are cases where we are interested in parsing just a subset of the fields (the ones we know we need) and delay any semantic parsing/mapping of the rest of the fields.
E.g. we want inspect the key in a key-value Event for (say) load-balancing, but at that point in time we don't have enough info to completely parse the values (we assume that some application further down the road will know what to do with the values):
val json = """{ "key": {"a": true, "b": 123}, "value": [1,2,3,4] }"""
case class Key(a: Boolean, b: Int)
case class Event(key: Key, value: ???) // value can be any valid Json
import org.json4s._
import org.json4s.native.JsonMethods._
implicit val formats = DefaultFormats
val parsedJson = parse(json)
parsedJson.extract[Event]
The question is how to represent the fields that we don't yet know (or care) to parse? What do we add as the type for value?
Note: One solution is to change the event to be case class Event(key: Key). This will work, but it will completely ignore the value, which we ideally want to keep so we can dispatch it properly to another service. So this solution will not work for us.
parse() parses JSON into json4s AST and returns JValue.
AFAIK You can not parse JSON partially because parsing to AST includes JSON validation and for that you need to parse the whole JSON string to AST tree.
But you can partially extract from AST. Here you have two options.
First. Make value field a JValue to defer extracting. You can do it later by calling extract() on this JValue instance.
case class Event(key: Key, value: JValue)
val event = parsedJson.extract[Event]
val value = event.value.extract[Seq[Int]]
Second. Split Event to two classes and extract them separately.
case class EventKey(key: Key)
case class EventValue(value: Seq[Int])
val parsedJson = parse(json)
val eventKey = parsedJson.extract[EventKey]
val eventValue = parsedJson.extract[EventValue]
I am building a GraphQL endpoint for an API using Finch, Circe and Sangria. The variables that come through in a GraphQL query are basically an arbitrary JSON object (let's assume there's no nesting). So for example, in my test code as Strings, here are two examples:
val variables = List(
"{\n \"foo\": 123\n}",
"{\n \"foo\": \"bar\"\n}"
)
The Sangria API expects a type for these of Map[String, Any].
I've tried a bunch of ways but have so far been unable to write a Decoder for this in Circe. Any help appreciated.
The Sangria API expects a type for these of Map[String, Any]
This is not true. Variables for an execution in sangria can be of an arbitrary type T, the only requirement that you have an instance of InputUnmarshaller[T] type class for it. All marshalling integration libraries provide an instance of InputUnmarshaller for correspondent JSON AST type.
This means that sangria-circe defines InputUnmarshaller[io.circe.Json] and you can import it with import sangria.marshalling.circe._.
Here is a small and self-contained example of how you can use circe Json as a variables:
import io.circe.Json
import sangria.schema._
import sangria.execution._
import sangria.macros._
import sangria.marshalling.circe._
val query =
graphql"""
query ($$foo: Int!, $$bar: Int!) {
add(a: $$foo, b: $$bar)
}
"""
val QueryType = ObjectType("Query", fields[Unit, Unit](
Field("add", IntType,
arguments = Argument("a", IntType) :: Argument("b", IntType) :: Nil,
resolve = c ⇒ c.arg[Int]("a") + c.arg[Int]("b"))))
val schema = Schema(QueryType)
val vars = Json.obj(
"foo" → Json.fromInt(123),
"bar" → Json.fromInt(456))
val result: Future[Json] =
Executor.execute(schema, query, variables = vars)
As you can see in this example, I used io.circe.Json as variables for an execution. The execution would produce following result JSON:
{
"data": {
"add": 579
}
}
Here's a decoder that works.
type GraphQLVariables = Map[String, Any]
val graphQlVariablesDecoder: Decoder[GraphQLVariables] = Decoder.instance { c =>
val variablesString = c.downField("variables").focus.flatMap(_.asString)
val parsedVariables = variablesString.flatMap { str =>
val variablesJsonObject = io.circe.jawn.parse(str).toOption.flatMap(_.asObject)
variablesJsonObject.map(j => j.toMap.transform { (_, value: Json) =>
val transformedValue: Any = value.fold(
(),
bool => bool,
number => number.toDouble,
str => str,
array => array.map(_.toString),
obj => obj.toMap.transform((s: String, json: Json) => json.toString)
)
transformedValue
})
}
parsedVariables match {
case None => left(DecodingFailure(s"Unable to decode GraphQL variables", c.history))
case Some(variables) => right(variables)
}
}
We basically parse the JSON, turn it into a JsonObject, then transform the values within the object fairly simplistically.
Although the above answers work for the specific case of Sangria, I'm interested in the original question: What's the best approach in Circe (which generally assumes that all types are known up front) for dealing with arbitrary chunks of Json?
It's fairly common when encoding/decoding Json that 95% of the Json is specified, but the last 5% is some type of "additional properties" chunk which can be any Json object.
Solutions I've played with:
Encode/Decode the free-form chunk as Map[String,Any]. This means you'll have to introduce implicit encoders/decoders for Map[String, Any], which can be done, but is dangerous as that implicit can be pulled into places you didn't intend.
Encode/Decode the free-form chunk as Map[String, Json]. This is the easiest approach and is supported out of the box in Circe. But now the Json serialization logic has leaked out into your API (often you'll want to keep the Json stuff completely wrapped, so you can swap in other non-json formats later).
Encode/Decode to a String, where the string is required to be a valid Json chunk. At least you haven't locked your API into a specific Json library, but it doesn't feel very nice to have to ask your users to create Json chunks in this manual way.
Create a custom trait hierarchy to hold the data (e.g sealed trait Free; FreeInt(i: Int) extends Free; FreeMap(m: Map[String, Free] extends Free; ...). Now you you can create specific encoders/decoders for it. But what you've really done is replicate the Json type hierarchy that already exists in Circe.
I'm leaning more toward option 3. Since it's the most flexible, and will introduce the least dependencies in the API. But none of them are entirely satisfying. Any other ideas?
I am currently integrating part of our system with MongoDB and we decided to use the official scala driver for it.
We have case class with joda.DateTime as parameters:
case class Schema(templateId: Muid,
createdDate: DateTime,
updatedDate: DateTime,
columns: Seq[Column])
We also defined format for it:
implicit lazy val checklistSchemaFormat : Format[Schema] = (
(__ \ "templateId").format[Muid] and
(__ \ "createdDate").format[DateTime] and
(__ \ "updatedDate").format[DateTime] and
(__ \ "columns").format[Seq[Column]]
)((Schema.apply _), unlift(Schema.unapply))
When I serialize this object to json and write to mongo, the createdDate and updatedDate getting converted to Long (which is technically fine). And this is how we do it:
val bsonDoc = Document(Json.toJson(schema).toString())
collection(DbViewSchemasCollectionName).replaceOne(filter, bsonDoc, new UpdateOptions().upsert(true)).subscribe(new Observer[UpdateResult] {
override def onNext(result: UpdateResult): Unit = logger.info(s"Successfully updates checklist template schema with result: $result")
override def onError(e: Throwable): Unit = logger.info(s"Failed to update checklist template schema with error: $e")
override def onComplete(): Unit = {}
})
as a result Mongo has this type of object:
{
"_id": ObjectId("56fc4247eb3c740d31b04f05"),
"templateId": "gaQP3JIB3ppJtro9rO9BAw",
"createdDate": NumberLong(1459372615507),
"updatedDate": NumberLong(1459372615507),
"columns": [
...
]
}
Now, I am trying to read it like so:
collection(DbViewSchemasCollectionName).find(filter).first().head() map { document =>
ChecklistSchema.checklistSchemaFormat reads Json.parse(document.toJson()) match {
case JsSuccess(value, _) => {
Some(value)
}
case JsError(e) => throw JsResultException(e)
}
} recover { case e =>
logger.info("ERROR " + e)
None
}
And at this point the reads always failing, since the createdDate and updatedDate now look like this:
"createdDate" : { "$numberLong" : "1459372615507" }, "updatedDate" :
{"$numberLong" : "1459372615507" }
How do I deal with this situation? Is there any easier conversion between bson.Document and JsObject? Or I am completely digging into the wrong direction...
Thanks,
You can use the following approach to resolve your issue.
Firstly, I used json4s for reading the writing json to case classes
example:
case class User(_id: Option[Int], username: String, firstName: String, createdDate: DateTime , updatedDate: DateTime )
// A small wapper to convert case class to Document
def toBson[A <: AnyRef](x : A):Document = {
val json = write[A](x)
Document(json) }
def today() = DateTime.now
val user = User(Some(212),"binkabir","lacmalndl", today , today)
val bson = toBson(user)
usersCollection.insertOne(bson).subscribe((x: Completed) => println(x))
val f = usersCollection.find(equal("_id",212)).toFuture()
f.map(_.head.toJson).map( x => read[User](x)).foreach(println)
The code above will create a user case class, convert to Document, save to mongo db, query the db and print the returned User case class
I hope this makes sense!
To answer your second question (bson.Document <-> JsObject) - YES; this is a solved problem, check out Play2-ReactiveMongo, which makes it seem like you're storing/retrieving JsObject instances - plus it's fully asynchronous and super-easy to get going in a Play application.
You can even go a step further and use a library like Mondrian (full disclosure: I wrote it!) to get the basic CRUD operations on top of ReactiveMongo for your Play-JSON domain case-classes.
Obviously I'm biased, but I think these solutions are a great fit if you've already defined your models as case classes in Play - you can forget about the whole BSONDocument family and stick to Json._ and JsObject etc that you already know well.
EDIT:
At the risk of further downvotes, I will demonstrate how I would go about storing and retrieving the OP's Schema object, using Mondrian. I'm going to show pretty-much everything for completeness; bear in mind you've actually already done most of this. Your final code will have fewer lines than your current code, as you'd expect when you use a library.
Model Objects
There's a Column class here that's never mentioned, for simplicity let's just say that's:
case class Column (name:String, width:Int)
Now we can get on with the Schema, which is just:
import com.themillhousegroup.mondrian._
case class Schema(_id: Option[MongoId],
createdDate: DateTime,
updatedDate: DateTime,
columns: Seq[Column]) extends MongoEntity
So far, we've just implemented the MongoEntity trait, which just required the templateId field to be renamed and given the required type.
JSON Converters
import com.themillhousegroup.mondrian._
import play.api.libs.json._
import play.api.libs.functional.syntax._
object SchemaJson extends MongoJson {
implicit lazy val columnFormat = Json.format[Column]
// Pick one - easy:
implicit lazy val schemaFormat = Json.format[Schema]
// Pick one - backwards-compatible (uses "templateId"):
implicit lazy val checklistSchemaFormat : Format[Schema] = (
(__ \ "templateId").formatNullable[MongoId] and
(__ \ "createdDate").format[DateTime] and
(__ \ "updatedDate").format[DateTime] and
(__ \ "columns").format[Seq[Column]]
)((Schema.apply _), unlift(Schema.unapply))
}
The JSON converters are standard Play-JSON stuff; we pick up the MongoId Format by extending MongoJson. I've shown two different ways of defining the Format for Schema. If you have clients out in the wild using templateId (or if you prefer it) then use the second, more verbose declaration.
Service layer
For brevity I'll skip the application-configuration, you can read the Mondrian README.md for that. Let's define the SchemaService that is responsible for persistence operations on Schema instances:
import com.themillhousegroup.mondrian._
import SchemaJson._
class SchemaService extends TypedMongoService[Schema]("schemas")
That's it. We've linked the model object, the name of the MongoDB collection ("schemas") and (implicitly) the necessary converters.
Saving a Schema and finding a Schema based on some criteria
Now we start to realize the value of Mondrian. save and findOne are standard operations - we get them for free in our Service, which we inject into our controllers in the standard way:
class SchemaController #Inject (schemaService:SchemaService) extends Controller {
...
// Returns a Future[Boolean]
schemaService.save(mySchema).map { saveOk =>
...
}
...
...
// Define a filter criteria using standard Play-JSON:
val targetDate = new DateTime()
val criteria = Json.obj("createdDate" -> Json.obj("$gt" ->targetDate.getMillis))
// Returns a Future[Option[Schema]]
schemaService.findOne(criteria).map { maybeFoundSchema =>
...
}
}
So there we go. No sign of the BSON family, just the Play JSON that, as you say, we all know and love. You'll only need to reach for the Mongo documentation when you need to construct a JSON query (that $gt stuff) although in some cases you can use Mondrian's overloaded findOne(example:Schema) method if you are just looking for a simple object match, and avoid even that :-)
We are in the process of migrating an existing REST service from Spring/Java to Spray using ReactiveMongo. One of the requirements for the migration (the first phase of it anyway), is that all inputs and outputs must match the current system. The issue with this is the business objects allow null values - both at rest in the datastore, and when returned in GET methods on the service. Fields can be missing as input to the service for PUT/POST, but the corresponding values must still be written as null to the datastore and returned the same.
Normally 'not required' fields aren't an issue for Scala/Spray through the use of Option, but the issue I'm having is actually writing the values of the Option fields as null when persisting, and setting the fields as None when reading the same null from Mongo.
In the research I've been doing, I have not been able to find a way to do this.
Here are snippets of my code:
UserPersistent
case class UserPersistent(id: Option[String], name: Option[String])
PersistentUser
object PersistentUser {
implicit object PersistentUserReader extends BSONDocumentReader[UserPersistent] {
def read(doc: BSONDocument): UserPersistent = UserPersistent(
id = doc.getAs[String]("_id"),
name = doc.getAs[String]("name")
)
}
implicit object PersistentUserWriter extends BSONDocumentWriter[UserPersistent] {
override def write(persisted: UserPersistent): BSONDocument = {
BSONDocument(
"_id" -> persisted.id,
"name" -> persisted.name
)
}
}
}
I have tried the following on the write() side, and although the code compiles and runs, it throws a NullPointerException when executed
"name" -> {
val nnn = persisted.name match {
case Some(n) => n
case _ => null
}
nnn
}
I have used OptionFormat for the 'presentation' of the data, which returns nulls (but for everything), but I need to take care of the Mongo side of this.
Surely there's a way to do this - what am I missing?
Try This:
object PersistentUser {
implicit val reader: BSONDocumentReader[UserPersistent] = Macros.reader[UserPersistent]
implicit val writer: BSONDocumentWriter[UserPersistent] = Macros.writer[UserPersistent]
}