Trouble serializing optional value class instances with json4s - scala

I'm trying to serialize a case class with optional value-class field to JSON using json4s. So far, I'm not able to get the optional value-class field rendered correctly (see the snippet below with examples).
I tried json-native and json-jackson libraries, results are identical.
Here's a short self-contained test
import org.json4s.DefaultFormats
import org.scalatest.FunSuite
import org.json4s.native.Serialization._
class JsonConversionsTest extends FunSuite {
implicit val jsonFormats = DefaultFormats
test("optional value-class instance conversion") {
val json = writePretty(Foo(Option(Id(123)), "foo-name", Option("foo-opt"), Id(321)))
val actual =
"""
|{
| "id":{
| "value":123
| },
| "name":"foo-name",
| "optField":"foo-opt",
| "nonOptId":321
|}
|""".stripMargin.trim
assert(json === actual)
val correct =
"""
|{
| "id": 123,
| "name":"foo-name",
| "optField":"foo-opt",
| "nonOptId":321
|}
|""".stripMargin.trim
assert(json !== correct)
}
}
case class Id(value: Int) extends AnyVal
case class Foo(id: Option[Id], name: String, optField: Option[String], nonOptId: Id)
I'm using scala 2.12 and the latest json4s-native version:
"org.json4s" %% "json4s-native" % "3.6.7"

It looks very similar to this issue, doesn't seem to be fixed or commented on.
A custom serializer would save your day.
object IdSerializer extends CustomSerializer[Id] ( format => (
{ case JInt(a) => Id(a.toInt) },
{ case a: Id => JInt(a.value) }
))
implicit val formats = DefaultFormats + IdSerializer
val json = writePretty(Foo(Option(Id(123)), "foo-name", Option("foo-opt"), Id(321)))

In any case try other options like using of jsoniter-scala instead. It is much safe and efficient than json4s especially in serialization of prettified JSON.
Add/replace dependencies:
libraryDependencies ++= Seq(
"com.github.plokhotnyuk.jsoniter-scala" %% "jsoniter-scala-core" % "0.55.2" % Compile,
"com.github.plokhotnyuk.jsoniter-scala" %% "jsoniter-scala-macros" % "0.55.2" % Provided // required only in compile-time
)
Derive a codec for the top-level data structure and use it to serialize into a string:
import com.github.plokhotnyuk.jsoniter_scala.macros._
import com.github.plokhotnyuk.jsoniter_scala.core._
case class Id(value: Int) extends AnyVal
case class Foo(id: Option[Id], name: String, optField: Option[String], nonOptId: Id)
implicit val codec: JsonValueCodec[Foo] = JsonCodecMaker.make(CodecMakerConfig())
val json = writeToString(Foo(Option(Id(123)), "foo-name", Option("foo-opt"), Id(321)), WriterConfig(indentionStep = 2))
val correct =
"""{
| "id": 123,
| "name": "foo-name",
| "optField": "foo-opt",
| "nonOptId": 321
|}""".stripMargin
assert(json == correct)
There are also more efficient options to store into byte array, java.nio.ByteBuffer or java.io.OutputStream immediately.

Related

What is Scala 3 equivalent to this Scala 2 code that uses Enumeration and play-json?

I have some code that works in Scala 2.{10,11,12,13} that I'm now trying to convert to Scala 3. Scala 3 does Enumeration differently than Scala 2. I'm trying to figure out how to convert the following code that interacts with play-json so that it will work with Scala 3. Any tips or pointers to code from projects that have already crossed this bridge?
// Scala 2.x style code in EnumUtils.scala
import play.api.libs.json._
import scala.language.implicitConversions
// see: http://perevillega.com/enums-to-json-in-scala
object EnumUtils {
def enumReads[E <: Enumeration](enum: E): Reads[E#Value] =
new Reads[E#Value] {
def reads(json: JsValue): JsResult[E#Value] = json match {
case JsString(s) => {
try {
JsSuccess(enum.withName(s))
} catch {
case _: NoSuchElementException =>
JsError(s"Enumeration expected of type: '${enum.getClass}', but it does not appear to contain the value: '$s'")
}
}
case _ => JsError("String value expected")
}
}
implicit def enumWrites[E <: Enumeration]: Writes[E#Value] = new Writes[E#Value] {
def writes(v: E#Value): JsValue = JsString(v.toString)
}
implicit def enumFormat[E <: Enumeration](enum: E): Format[E#Value] = {
Format(EnumUtils.enumReads(enum), EnumUtils.enumWrites)
}
}
// ----------------------------------------------------------------------------------
// Scala 2.x style code in Xyz.scala
import play.api.libs.json.{Reads, Writes}
object Xyz extends Enumeration {
type Xyz = Value
val name, link, unknown = Value
implicit val enumReads: Reads[Xyz] = EnumUtils.enumReads(Xyz)
implicit def enumWrites: Writes[Xyz] = EnumUtils.enumWrites
}
As an option you can switch to jsoniter-scala.
It supports enums for Scala 2 and Scala 3 out of the box.
Also it has handy derivation of safe and efficient JSON codecs.
Just need to add required libraries to your dependencies:
libraryDependencies ++= Seq(
// Use the %%% operator instead of %% for Scala.js and Scala Native
"com.github.plokhotnyuk.jsoniter-scala" %% "jsoniter-scala-core" % "2.13.5",
// Use the "provided" scope instead when the "compile-internal" scope is not supported
"com.github.plokhotnyuk.jsoniter-scala" %% "jsoniter-scala-macros" % "2.13.5" % "compile-internal"
)
And then derive a codec and use it:
import com.github.plokhotnyuk.jsoniter_scala.core._
import com.github.plokhotnyuk.jsoniter_scala.macros._
implicit val codec: JsonValueCodec[Xyz.Xyz] = JsonCodecMaker.make
println(readFromString[Xyz.Xyz]("\"name\""))
BTW, you can run the full code on Scastie: https://scastie.scala-lang.org/Evj718q6TcCZow9lRhKaPw

get annotations from class in scala 3 macros

i am writing a macro to get annotations from a 'Class'
inline def getAnnotations(clazz: Class[?]): Seq[Any] = ${ getAnnotationsImpl('clazz) }
def getAnnotationsImpl(expr: Expr[Class[?]])(using Quotes): Expr[Seq[Any]] =
import quotes.reflect.*
val cls = expr.valueOrError // error: value value is not a member of quoted.Expr[Class[?]]
val tpe = TypeRepr.typeConstructorOf(cls)
val annotations = tpe.typeSymbol.annotations.map(_.asExpr)
Expr.ofSeq(annotations)
but i get an error when i get class value from expr parameter
#main def test(): Unit =
val cls = getCls
val annotations = getAnnotations(cls)
def getCls: Class[?] = Class.forName("Foo")
is it possible to get annotations of a Class at compile time by this macro ?!
By the way, eval for Class[_] doesn't work even in Scala 2 macros: c.eval(c.Expr[Class[_]](clazz)) produces
java.lang.ClassCastException:
scala.reflect.internal.Types$ClassNoArgsTypeRef cannot be cast to java.lang.Class.
Class[_] is too runtimy thing. How can you extract its value from its tree ( Expr is a wrapper over tree)?
If you already have a Class[?] you should use Java reflection rather than Scala 3 macros (with Tasty reflection).
Actually, you can try to evaluate a tree from its source code (hacking multi-staging programming and implementing our own eval instead of forbidden staging.run). It's a little similar to context.eval in Scala 2 macros (but we evaluate from a source code rather than from a tree).
import scala.quoted.*
object Macro {
inline def getAnnotations(clazz: Class[?]): Seq[Any] = ${getAnnotationsImpl('clazz)}
def getAnnotationsImpl(expr: Expr[Class[?]])(using Quotes): Expr[Seq[Any]] = {
import quotes.reflect.*
val str = expr.asTerm.pos.sourceCode.getOrElse(
report.errorAndAbort(s"No source code for ${expr.show}")
)
val cls = Eval[Class[?]](str)
val tpe = TypeRepr.typeConstructorOf(cls)
val annotations = tpe.typeSymbol.annotations.map(_.asExpr)
Expr.ofSeq(annotations)
}
}
import dotty.tools.dotc.core.Contexts.Context
import dotty.tools.dotc.{Driver, util}
import dotty.tools.io.{VirtualDirectory, VirtualFile}
import java.net.URLClassLoader
import java.nio.charset.StandardCharsets
import dotty.tools.repl.AbstractFileClassLoader
object Eval {
def apply[A](str: String): A = {
val content =
s"""
|package $$generated
|
|object $$Generated {
| def run = $str
|}""".stripMargin
val sourceFile = util.SourceFile(
VirtualFile(
name = "$Generated.scala",
content = content.getBytes(StandardCharsets.UTF_8)),
codec = scala.io.Codec.UTF8
)
val files = this.getClass.getClassLoader.asInstanceOf[URLClassLoader].getURLs
val depClassLoader = new URLClassLoader(files, null)
val classpathString = files.mkString(":")
val outputDir = VirtualDirectory("output")
class DriverImpl extends Driver {
private val compileCtx0 = initCtx.fresh
val compileCtx = compileCtx0.fresh
.setSetting(
compileCtx0.settings.classpath,
classpathString
).setSetting(
compileCtx0.settings.outputDir,
outputDir
)
val compiler = newCompiler(using compileCtx)
}
val driver = new DriverImpl
given Context = driver.compileCtx
val run = driver.compiler.newRun
run.compileSources(List(sourceFile))
val classLoader = AbstractFileClassLoader(outputDir, depClassLoader)
val clazz = Class.forName("$generated.$Generated$", true, classLoader)
val module = clazz.getField("MODULE$").get(null)
val method = module.getClass.getMethod("run")
method.invoke(module).asInstanceOf[A]
}
}
package mypackage
import scala.annotation.experimental
#experimental
class Foo
Macro.getAnnotations(Class.forName("mypackage.Foo")))
// new scala.annotation.internal.SourceFile("/path/to/src/main/scala/mypackage/Foo.scala"), new scala.annotation.experimental()
scalaVersion := "3.1.3"
libraryDependencies += scalaOrganization.value %% "scala3-compiler" % scalaVersion.value
How to compile and execute scala code at run-time in Scala3?
(compile time of the code expanding macros is the runtime of macros)
Actually, there is even a way to evaluate a tree itself (not its source code). Such functionality exists in Scala 3 compiler but is deliberately blocked because of phase consistency principle. So this to work, the code expanding macros should be compiled with a compiler patched
https://github.com/DmytroMitin/dotty-patched
scalaVersion := "3.2.1"
libraryDependencies += scalaOrganization.value %% "scala3-staging" % scalaVersion.value
// custom Scala settings
managedScalaInstance := false
ivyConfigurations += Configurations.ScalaTool
libraryDependencies ++= Seq(
scalaOrganization.value % "scala-library" % "2.13.10",
scalaOrganization.value %% "scala3-library" % "3.2.1",
"com.github.dmytromitin" %% "scala3-compiler-patched-assembly" % "3.2.1" % "scala-tool"
)
import scala.quoted.{Expr, Quotes, staging, quotes}
object Macro {
inline def getAnnotations(clazz: Class[?]): Seq[String] = ${impl('clazz)}
def impl(expr: Expr[Class[?]])(using Quotes): Expr[Seq[String]] = {
import quotes.reflect.*
given staging.Compiler = staging.Compiler.make(this.getClass.getClassLoader)
val tpe = staging.run[Any](expr).asInstanceOf[TypeRepr]
val annotations = Expr(tpe.typeSymbol.annotations.map(_.asExpr.show))
report.info(s"annotations=${annotations.show}")
annotations
}
}
Normally, for expr: Expr[A] staging.run(expr) returns a value of type A. But Class is specific. For expr: Expr[Class[_]] inside macros it returns a value of type dotty.tools.dotc.core.Types.CachedAppliedType <: TypeRepr. That's why I had to cast.
In Scala 2 this also would be c.eval(c.Expr[Any](/*c.untypecheck*/(clazz))).asInstanceOf[Type].typeSymbol.annotations because for Class[_] c.eval returns scala.reflect.internal.Types$ClassNoArgsTypeRef <: Type.
https://github.com/scala/bug/issues/12680

Raise exception while parsing JSON with the wrong schema on the Optional field

During JSON parsing, I want to catch an exception for optional sequential files which schema differed from my case class.
Let me elaborate
I have the following case class:
case class SimpleFeature(
column: String,
valueType: String,
nullValue: String,
func: Option[String])
case class TaskConfig(
taskInfo: TaskInfo,
inputType: String,
training: Table,
testing: Table,
eval: Table,
splitStrategy: SplitStrategy,
label: Label,
simpleFeatures: Option[List[SimpleFeature]],
model: Model,
evaluation: Evaluation,
output: Output)
And this is part of JSON file I want to point attention to:
"simpleFeatures": [
{
"column": "pcat_id",
"value": "categorical",
"nullValue": "DUMMY"
},
{
"column": "brand_code",
"valueType": "categorical",
"nullValue": "DUMMY"
}
]
As you can see the first element has an error in the schema and while parsing, I want to raise an error. At the same time, I want to keep optional behavior in case there is no object to parse.
One idea that I've been researching for a while - to create the custom serializer and manually check fields, but not sure I'm on the right track
object JSONSerializer extends CustomKeySerializer[SimpleFeatures](format => {
case jsonObj: JObject => {
case Some(simplFeatures (jsonObj \ "simpleFeatures")) => {
// Extraction logic goes here
}
}
})
I might be not quite proficient in Scala and json4s so any advice is appreciated.
json4s version
3.2.10
scala version
2.11.12
jdk version
1.8.0
I think you need to extend CustomSerializer class since CustomKeySerializer it is used to implement custom logic for JSON keys:
import org.json4s.{CustomSerializer, MappingException}
import org.json4s.JsonAST._
import org.json4s.JsonDSL._
import org.json4s.jackson.JsonMethods._
case class SimpleFeature(column: String,
valueType: String,
nullValue: String,
func: Option[String])
case class TaskConfig(simpleFeatures: Option[Seq[SimpleFeature]])
object Main extends App {
implicit val formats = new DefaultFormats {
override val strictOptionParsing: Boolean = true
} + new SimpleFeatureSerializer()
class SimpleFeatureSerializer extends CustomSerializer[SimpleFeature](_ => ( {
case jsonObj: JObject =>
val requiredKeys = Set[String]("column", "valueType", "nullValue")
val diff = requiredKeys.diff(jsonObj.values.keySet)
if (diff.nonEmpty)
throw new MappingException(s"Fields [${requiredKeys.mkString(",")}] are mandatory. Missing fields: [${diff.mkString(",")}]")
val column = (jsonObj \ "column").extract[String]
val valueType = (jsonObj \ "valueType").extract[String]
val nullValue = (jsonObj \ "nullValue").extract[String]
val func = (jsonObj \ "func").extract[Option[String]]
SimpleFeature(column, valueType, nullValue, func)
}, {
case sf: SimpleFeature =>
("column" -> sf.column) ~
("valueType" -> sf.valueType) ~
("nullValue" -> sf.nullValue) ~
("func" -> sf.func)
}
))
// case 1: Test single feature
val singleFeature = """
{
"column": "pcat_id",
"valueType": "categorical",
"nullValue": "DUMMY"
}
"""
val singleFeatureValid = parse(singleFeature).extract[SimpleFeature]
println(singleFeatureValid)
// SimpleFeature(pcat_id,categorical,DUMMY,None)
// case 2: Test task config
val taskConfig = """{
"simpleFeatures": [
{
"column": "pcat_id",
"valueType": "categorical",
"nullValue": "DUMMY"
},
{
"column": "brand_code",
"valueType": "categorical",
"nullValue": "DUMMY"
}]
}"""
val taskConfigValid = parse(taskConfig).extract[TaskConfig]
println(taskConfigValid)
// TaskConfig(List(SimpleFeature(pcat_id,categorical,DUMMY,None), SimpleFeature(brand_code,categorical,DUMMY,None)))
// case 3: Invalid json
val invalidSingleFeature = """
{
"column": "pcat_id",
"value": "categorical",
"nullValue": "DUMMY"
}
"""
val singleFeatureInvalid = parse(invalidSingleFeature).extract[SimpleFeature]
// throws MappingException
}
Analysis: the main question here is how to gain access to the keys of jsonObj in order to check whether there is an invalid or missing key, one way to achieve that is through jsonObj.values.keySet. For the implementation, first we assign the mandatory fields to the requiredKeys variable, then we compare the requiredKeys with the ones that are currently present with requiredKeys.diff(jsonObj.values.keySet). If the difference is not empty that means there is a missing mandatory field, in this case we throw an exception including the necessary information.
Note1: we should not forget to add the new serializer to the available formats.
Note2: we throw an instance of MappingException, which is already used by json4s internally when parsing JSON string.
UPDATE
In order to force validation of Option fields you need to set strictOptionParsing option to true by overriding the corresponding method:
implicit val formats = new DefaultFormats {
override val strictOptionParsing: Boolean = true
} + new SimpleFeatureSerializer()
Resources
https://nmatpt.com/blog/2017/01/29/json4s-custom-serializer/
https://danielasfregola.com/2015/08/17/spray-how-to-deserialize-entities-with-json4s/
https://www.programcreek.com/scala/org.json4s.CustomSerializer
You can try using play.api.libs.json
"com.typesafe.play" %% "play-json" % "2.7.2",
"net.liftweb" % "lift-json_2.11" % "2.6.2"
You just need to define case class and formatters.
Example:
case class Example(a: String, b: String)
implicit val formats: DefaultFormats.type = DefaultFormats
implicit val instancesFormat= Json.format[Example]
and then just do :
Json.parse(jsonData).asOpt[Example]
In case the above gives some errors: Try adding "net.liftweb" % "lift-json_2.11" % "2.6.2" too in your dependency.

Can't get json4s to render my case class?

I set up a SBT console like...
import org.json4s._
import org.json4s.native.JsonMethods._
import org.json4s.JsonDSL._
case class TagOptionOrNull(tag: String, optionUuid: Option[java.util.UUID], uuid: java.util.UUID)
val t1 = new TagOptionOrNull("t1", Some(java.util.UUID.randomUUID), java.util.UUID.randomUUID)
val t2 = new TagOptionOrNull("t2", None, null)
I'm trying to see json4s's behavior around null vs Option[UUID]. But I can't figure out the invocation to get it to make my case class a String of JSON.
scala> implicit val formats = DefaultFormats
formats: org.json4s.DefaultFormats.type = org.json4s.DefaultFormats$#614275d5
scala> compact(render(t1))
<console>:23: error: type mismatch;
found : TagOptionOrNull
required: org.json4s.JValue
(which expands to) org.json4s.JsonAST.JValue
compact(render(t1))
What am I missing?
Serialization.write should be able to serialise case class like so
import org.json4s.native.Serialization.write
implicit val formats = DefaultFormats ++ JavaTypesSerializers.all
println(write(t1))
which should output
{"tag":"t1","optionUuid":"95645021-f60c-4708-8bf3-9d5609559fdb","uuid":"19cc4979-5836-4edf-aedd-dcb3e96f17d6"}
Note to serialise UUID we need JavaTypeSerializers formats from
libraryDependencies += "org.json4s" %% "json4s-ext" % version

Marshalling list of case class objects

I want to return list of json objects based on my case class objects.
Following is my spray router, which returns list of 'Appointment' objects.
trait GatewayService extends HttpService with SLF4JLogging {
import com.sml.apigw.protocols.AppointmentProtocol._
import spray.httpx.SprayJsonSupport._
implicit def executionContext = actorRefFactory.dispatcher
val router =
pathPrefix("api" / "v1") {
path("appointments") {
get {
complete {
val a = new Appointment("1", "2")
val l = List(a, a, a, a)
l
}
}
}
}
}
}
Following is the 'AppointmentProtocol'
import spray.json.DefaultJsonProtocol
case class Appointment(id: String, patient: String)
object AppointmentProtocol extends DefaultJsonProtocol {
implicit val appointmentFormat = jsonFormat2(Appointment.apply)
}
It gives the compilation error 'expression type of List[Appointment] doesn't confirms to expected type toResponseMarshallable'
Maybe you missed something in your example, but my brain-based compiler is telling me that your code should throw a compilation error since any directive in Spray requires a result of type Route, as i can see you have a List[Appointment]. Please read this article on Routes. Your route structure should be completed with complete, so i assume the this way would solve your issue:
get {
val a = new Appointment("1", "2")
val l = List(a, a, a, a)
complete(l)
}
Please note a complete directive which wraps the list. This should help, otherwise please provide the tree with resolved implicits by compiling your code with the flag -Xprint:typer, that should show where the problem with implicits is.
I think that you should use this library spray-json depending on your version of spay this will be your import and do not forget to add the imports to your code:
import MyJsonProtocol._
import spray.json._
for that be sure that you have imported:
libraryDependencies += "io.spray" %% "spray-json" % "1.3.2"
THen you can conver an object whith this, I also have problems having the case class in the same file but this was using testing:
case class Color(name: String, red: Int, green: Int, blue: Int)
object MyJsonProtocol extends DefaultJsonProtocol {
implicit val colorFormat = jsonFormat4(Color)
}
import MyJsonProtocol._
import spray.json._
val json = Color("CadetBlue", 95, 158, 160).toJson
val color = json.convertTo[Color]
and the list:
val jsonAst = List(1, 2, 3).toJson
This are extracted examples from the spray-json github project