When parsing a Json String, there are cases where we are interested in parsing just a subset of the fields (the ones we know we need) and delay any semantic parsing/mapping of the rest of the fields.
E.g. we want inspect the key in a key-value Event for (say) load-balancing, but at that point in time we don't have enough info to completely parse the values (we assume that some application further down the road will know what to do with the values):
val json = """{ "key": {"a": true, "b": 123}, "value": [1,2,3,4] }"""
case class Key(a: Boolean, b: Int)
case class Event(key: Key, value: ???) // value can be any valid Json
import org.json4s._
import org.json4s.native.JsonMethods._
implicit val formats = DefaultFormats
val parsedJson = parse(json)
parsedJson.extract[Event]
The question is how to represent the fields that we don't yet know (or care) to parse? What do we add as the type for value?
Note: One solution is to change the event to be case class Event(key: Key). This will work, but it will completely ignore the value, which we ideally want to keep so we can dispatch it properly to another service. So this solution will not work for us.
parse() parses JSON into json4s AST and returns JValue.
AFAIK You can not parse JSON partially because parsing to AST includes JSON validation and for that you need to parse the whole JSON string to AST tree.
But you can partially extract from AST. Here you have two options.
First. Make value field a JValue to defer extracting. You can do it later by calling extract() on this JValue instance.
case class Event(key: Key, value: JValue)
val event = parsedJson.extract[Event]
val value = event.value.extract[Seq[Int]]
Second. Split Event to two classes and extract them separately.
case class EventKey(key: Key)
case class EventValue(value: Seq[Int])
val parsedJson = parse(json)
val eventKey = parsedJson.extract[EventKey]
val eventValue = parsedJson.extract[EventValue]
Related
I have a file, each row is a json array.
I reading each line of the file, and trying to convert the rows into a json array, and then for each element I am converting to a case class using json spray.
I have this so far:
for (line <- source.getLines().take(10)) {
val jsonArr = line.parseJson.convertTo[JsArray]
for (ele <- jsonArr.elements) {
val tryUser = Try(ele.convertTo[User])
}
}
How could I convert this entire process into a single line statement?
val users: Seq[User] = source.getLines.take(10).map(line => line.parseJson.convertTo[JsonArray].elements.map(ele => Try(ele.convertTo[User])
The error is:
found : Iterator[Nothing]
Note: I used Scala 2.13.6 for all my examples.
There is a lot to unpack in these few lines of code. First of all, I'll share some code that we can use to generate some meaningful input to play around with.
object User {
import scala.util.Random
private def randomId: Int = Random.nextInt(900000) + 100000
private def randomName: String = Iterator
.continually(Random.nextInt(26) + 'a')
.map(_.toChar)
.take(6)
.mkString
def randomJson(): String = s"""{"id":$randomId,"name":"$randomName"}"""
def randomJsonArray(size: Int): String =
Iterator.continually(randomJson()).take(size).mkString("[", ",", "]")
}
final case class User(id: Int, name: String)
import scala.util.{Try, Success, Failure}
import spray.json._
import DefaultJsonProtocol._
implicit val UserFormat = jsonFormat2(User.apply)
This is just some scaffolding to define some User domain object and come up with a way to generate a JSON representation of an array of such objects so that we can then use a JSON library (spray-json in this case) to parse it back into what we want.
Now, going back to your question. This is a possible way to massage your data into its parsed representation. It may not fit 100% what your are trying to do, but there's some nuance in the data types involved and how they work:
val parsedUsers: Iterator[Try[User]] =
for {
line <- Iterator.continually(User.randomJsonArray(4)).take(10)
element <- line.parseJson.convertTo[JsArray].elements
} yield Try(element.convertTo[User])
First difference: notice that I use the for comprehension in a form in which the "outcome" of an iteration is not a side effect (for (something) { do something }) but an actual value for (something) yield { return a value }).
Second difference: I explicitly asked for an Iterator[Try[User]] rather than a Seq[User]. We can go very down into a rabbit hole on the topic of why the types are what they are here, but the simple explanation is that a for ... yield expression:
returns the same type as the one in the first line of the generation -- if you start with a val ns: Iterator[Int]; for (n<- ns) ... you'll get an iterator at the end
if you nest generators, they need to be of the same type as the "outermost" one
You can read more on for comprehensions on the Tour of Scala and the Scala Book.
One possible way of consuming this is the following:
for (user <- parsedUsers) {
user match {
case Success(user) => println(s"parsed object $user")
case Failure(error) => println(s"ERROR: '${error.getMessage}'")
}
As for how to turn this into a "one liner", for comprehensions are syntactic sugar applied by the compiler which turns every nested call into a flatMap and the final one into map, as in the following example (which yields an equivalent result as the for comprehension above and very close to what the compiler does automatically):
val parsedUsers: Iterator[Try[User]] = Iterator
.continually(User.randomJsonArray(4))
.take(10)
.flatMap(line =>
line.parseJson
.convertTo[JsArray]
.elements
.map(element => Try(element.convertTo[User]))
)
One note that I would like to add is that you should be mindful of readability. Some teams prefer for comprehensions, others manually rolling out their own flatMap/map chains. Coders discretion is advised.
You can play around with this code here on Scastie (and here is the version with the flatMap/map calls).
I'm trying to encode a list of objects using Circe, something that looks similar to:
val test = Seq(MyObject("hello", None, 1, 2, None)
I'm trying to parse this using Circe:
test.asJson
But this creates the JSON object:
[
{
name: "hello",
someVal: null,
someNum: 1,
anotherNum: 2,
anotherVal: null
}
]
I've tried running asJson with .dropNullValues, but that doesn't seem to access the null values inside of the object. Is there a way to drop the null values inside of the objects?
I'm expecting something more like this:
[
{
name: "hello",
someNum: 1,
anotherNum: 2
}
]
Circe provides the function deepDropNullValues in the Json class.
Example: test.asJson.deepDropNullValues
You see field: null because circe turns Option[T] to t.asJson on Some[T] and JsonNull on None, and default case class encoder just puts all fields to the JsonObject. In a way that circe uses to encode sealed trait family, it may use these null fields to distinguish classes like
sealed trait Foo
case class Bar(a: Option[String])
case class Baz(a: Option[String], b: Option[String])
So, if you really want to drop this information and need one-way conversion with information loss, you can map resulting Json to drop all and every null field with code like that:
implicit val fooEnc: Encoder[Foo] = deriveEncoder[Foo].mapJsonObject{jsonObj => jsonObj.filter{case (k,v) => !v.isNull}}
However, you should write such a custom codec for any class you want to drop null fields. To post-process your json, you can use fold on resulting json:
val json: Json = ???
json.fold[Json](
Json.Null,
Json.fromBoolean,
{_.asJson},
{_.asJson},
{_.asJson},
{jsonObj => jsonObj.filter{case (k,v) => !v.isNull}.asJson}
)
or implement a custom folder.
The answer that was suggested by #iva-kmm is good, but it can be done better!
This mechanic is already implemented in circe, just call it:
implicit val fooEnc: Encoder[Foo] = deriveEncoder[Foo].mapJson(_.dropNullValues) // or _.deepDropNullValues
Hi the Decode Person example in the documentation is great if the JSON has a key and value and you can use the key name to extract its value, but what about if the string that makes up the key is arbitrary but meaningful.
for Fxample one open cryptocurrency api can give historic prices of coins and the structure of the JSON returned is different depending on the base currency of the coin I'm asking for and the various quote currencies I want it priced in.. for example lets say I want the price at a particular date of 'DOGE' in 'AUD' and 'XRP' the returned JSON looks like
{"DOGE":{"AUD":0.008835,"XRP":0.004988}}
I can't navigate to base and get its value and then prices and get them as the JSON is not stuctured that way, I need to look for 'DOGE' as a Key then in the Object retrned know that there will be a 'AUD' key and 'XRP' key. And of course that will be different for every result depending on my query.
Of course I know these keys as I create the search based on them but how can I use Argonaut to parse this JSON? Can I somehow create a Decode that closes over my key names?
Any help or guidance is appreciated, thanks.
Since you don't know what the property names are going to be ahead of time, you can't create a codec and decode the raw JSON directly to a Scala class.
You want to parse the raw JSON as a generic argonaut.Json object, then you can pattern match or use fold to examine the contents. For example:
val rawJson: String = ...
val parsed: Either[String, argonaut.Json] = argonaut.Parse.parse(rawJson)
You can see the methods available on argonaut's Json object by inspecting the source code.
As per Fried Brice's answer I did go down the parse route then mapped the resulting Either to produce my data type see code snippet below, suggestions, improvements welcome.
def parseHistoricPriceJSON(rawJson: String, fromcurrency: Currency, toCurrencies: List[Currency]): Either[String, PricedAsset] = {
import argonaut._, Argonaut._
import monocle.macros.syntax.lens._
val parsed: Either[String, Json] = Parse.parse(rawJson)
val myTocurrs = Currency("XRP") :: toCurrencies
parsed.right.map(outer => {
val cursor = outer.cursor
val ps = for {
toC <- myTocurrs
prices <- cursor.downField(fromcurrency.sym)
price <- prices.downField(toC.sym)
thep <- price.focus.number
} yield (toC, thep.toDouble.get)
PricedAsset(fromcurrency, ps)
})
}
case class Currency(sym: String) extends AnyVal {
def show = sym
}
case class PricedAsset(base:Currency, quotePrices: List[(Currency,Double)])
I am building a GraphQL endpoint for an API using Finch, Circe and Sangria. The variables that come through in a GraphQL query are basically an arbitrary JSON object (let's assume there's no nesting). So for example, in my test code as Strings, here are two examples:
val variables = List(
"{\n \"foo\": 123\n}",
"{\n \"foo\": \"bar\"\n}"
)
The Sangria API expects a type for these of Map[String, Any].
I've tried a bunch of ways but have so far been unable to write a Decoder for this in Circe. Any help appreciated.
The Sangria API expects a type for these of Map[String, Any]
This is not true. Variables for an execution in sangria can be of an arbitrary type T, the only requirement that you have an instance of InputUnmarshaller[T] type class for it. All marshalling integration libraries provide an instance of InputUnmarshaller for correspondent JSON AST type.
This means that sangria-circe defines InputUnmarshaller[io.circe.Json] and you can import it with import sangria.marshalling.circe._.
Here is a small and self-contained example of how you can use circe Json as a variables:
import io.circe.Json
import sangria.schema._
import sangria.execution._
import sangria.macros._
import sangria.marshalling.circe._
val query =
graphql"""
query ($$foo: Int!, $$bar: Int!) {
add(a: $$foo, b: $$bar)
}
"""
val QueryType = ObjectType("Query", fields[Unit, Unit](
Field("add", IntType,
arguments = Argument("a", IntType) :: Argument("b", IntType) :: Nil,
resolve = c ⇒ c.arg[Int]("a") + c.arg[Int]("b"))))
val schema = Schema(QueryType)
val vars = Json.obj(
"foo" → Json.fromInt(123),
"bar" → Json.fromInt(456))
val result: Future[Json] =
Executor.execute(schema, query, variables = vars)
As you can see in this example, I used io.circe.Json as variables for an execution. The execution would produce following result JSON:
{
"data": {
"add": 579
}
}
Here's a decoder that works.
type GraphQLVariables = Map[String, Any]
val graphQlVariablesDecoder: Decoder[GraphQLVariables] = Decoder.instance { c =>
val variablesString = c.downField("variables").focus.flatMap(_.asString)
val parsedVariables = variablesString.flatMap { str =>
val variablesJsonObject = io.circe.jawn.parse(str).toOption.flatMap(_.asObject)
variablesJsonObject.map(j => j.toMap.transform { (_, value: Json) =>
val transformedValue: Any = value.fold(
(),
bool => bool,
number => number.toDouble,
str => str,
array => array.map(_.toString),
obj => obj.toMap.transform((s: String, json: Json) => json.toString)
)
transformedValue
})
}
parsedVariables match {
case None => left(DecodingFailure(s"Unable to decode GraphQL variables", c.history))
case Some(variables) => right(variables)
}
}
We basically parse the JSON, turn it into a JsonObject, then transform the values within the object fairly simplistically.
Although the above answers work for the specific case of Sangria, I'm interested in the original question: What's the best approach in Circe (which generally assumes that all types are known up front) for dealing with arbitrary chunks of Json?
It's fairly common when encoding/decoding Json that 95% of the Json is specified, but the last 5% is some type of "additional properties" chunk which can be any Json object.
Solutions I've played with:
Encode/Decode the free-form chunk as Map[String,Any]. This means you'll have to introduce implicit encoders/decoders for Map[String, Any], which can be done, but is dangerous as that implicit can be pulled into places you didn't intend.
Encode/Decode the free-form chunk as Map[String, Json]. This is the easiest approach and is supported out of the box in Circe. But now the Json serialization logic has leaked out into your API (often you'll want to keep the Json stuff completely wrapped, so you can swap in other non-json formats later).
Encode/Decode to a String, where the string is required to be a valid Json chunk. At least you haven't locked your API into a specific Json library, but it doesn't feel very nice to have to ask your users to create Json chunks in this manual way.
Create a custom trait hierarchy to hold the data (e.g sealed trait Free; FreeInt(i: Int) extends Free; FreeMap(m: Map[String, Free] extends Free; ...). Now you you can create specific encoders/decoders for it. But what you've really done is replicate the Json type hierarchy that already exists in Circe.
I'm leaning more toward option 3. Since it's the most flexible, and will introduce the least dependencies in the API. But none of them are entirely satisfying. Any other ideas?
I've got a case class Game which I have no trouble serializing/deserializing using json4s.
case class Game(name: String,publisher: String,website: String, gameType: GameType.Value)
In my app I use mapperdao as my ORM. Because Game uses a Surrogate Id I do not have id has part of its constructor.
However, when mapperdao returns an entity from the DB it supplies the id of the persisted object using a trait.
Game with SurrogateIntId
The code for the trait is
trait SurrogateIntId extends DeclaredIds[Int]
{
def id: Int
}
trait DeclaredIds[ID] extends Persisted
trait Persisted
{
#transient
private var mapperDaoVM: ValuesMap = null
#transient
private var mapperDaoDetails: PersistedDetails = null
private[mapperdao] def mapperDaoPersistedDetails = mapperDaoDetails
private[mapperdao] def mapperDaoValuesMap = mapperDaoVM
private[mapperdao] def mapperDaoInit(vm: ValuesMap, details: PersistedDetails) {
mapperDaoVM = vm
mapperDaoDetails = details
}
.....
}
When I try to serialize Game with SurrogateIntId I get empty parenthesis returned, I assume this is because json4s doesn't know how to deal with the attached trait.
I need a way to serialize game with only id added to its properties , and almost as importantly a way to do this for any T with SurrogateIntId as I use these for all of my domain objects.
Can anyone help me out?
So this is an extremely specific solution since the origin of my problem comes from the way mapperDao returns DOs, however it may be helpful for general use since I'm delving into custom serializers in json4s.
The full discussion on this problem can be found on the mapperDao google group.
First, I found that calling copy() on any persisted Entity(returned from mapperDao) returned the clean copy(just case class) of my DO -- which is then serializable by json4s. However I did not want to have to remember to call copy() any time I wanted to serialize a DO or deal with mapping lists, etc. as this would be unwieldy and prone to errors.
So, I created a CustomSerializer that wraps around the returned Entity(case class DO + traits as an object) and gleans the class from generic type with an implicit manifest. Using this approach I then pattern match my domain objects to determine what was passed in and then use Extraction.decompose(myDO.copy()) to serialize and return the clean DO.
// Entity[Int, Persisted, Class[T]] is how my DOs are returned by mapperDao
class EntitySerializer[T: Manifest] extends CustomSerializer[Entity[Int, Persisted, Class[T]]](formats =>(
{PartialFunction.empty} //This PF is for extracting from JSON and not needed
,{
case g: Game => //Each type is one of my DOs
implicit val formats: Formats = DefaultFormats //include primitive formats for serialization
Extraction.decompose(g.copy()) //get plain DO and then serialize with json4s
case u : User =>
implicit val formats: Formats = DefaultFormats + new LinkObjectEntitySerializer //See below for explanation on LinkObject
Extraction.decompose(u.copy())
case t : Team =>
implicit val formats: Formats = DefaultFormats + new LinkObjectEntitySerializer
Extraction.decompose(t.copy())
...
}
The only need for a separate serializer is in the event that you have non-primitives as parameters of a case class being serialized because the serializer can't use itself to serialize. In this case you create a serializer for each basic class(IE one with only primitives) and then include it into the next serializer with objects that depend on those basic classes.
class LinkObjectEntitySerializer[T: Manifest] extends CustomSerializer[Entity[Int, Persisted, Class[T]]](formats =>(
{PartialFunction.empty},{
//Team and User have Set[TeamUser] parameters, need to define this "dependency"
//so it can be included in formats
case tu: TeamUser =>
implicit val formats: Formats = DefaultFormats
("Team" -> //Using custom-built representation of object
("name" -> tu.team.name) ~
("id" -> tu.team.id) ~
("resource" -> "/team/") ~
("isCaptain" -> tu.isCaptain)) ~
("User" ->
("name" -> tu.user.globalHandle) ~
("id" -> tu.user.id) ~
("resource" -> "/user/") ~
("isCaptain" -> tu.isCaptain))
}
))
This solution is hardly satisfying. Eventually I will need to replace mapperDao or json4s(or both) to find a simpler solution. However, for now, it seems to be the fix with the least amount of overhead.