Hi I am trying to extract a JSON using reflection
import net.liftweb.json._
case class Bike(make: String, price: Int) {
def this(price: Int) = this("Trek", price)
}
val cls = Class.forName("Bike")
val manifest = Manifest.classType(cls)
val parsedData =net.liftweb.json.JsonParser.parse(json)
JsonParser.parse(""" {"price":350} """).extract[manifest]
however I am getting this error:
not found: type manifest
JsonParser.parse(""" {"price":350} """).extract[manifest]
^
although manifest is from type Manifest
You can extract directly into a case class
val json = "the json";
val bike = parse(json).extract[Bike];
JSON parsing is done through reflection.
If the class is a runtime construct, create a TypeInfo instance and pass that to the extract method.
There is a variation of the extract() method that might work for you, if you provide it with a TypeInfo instance.
See here: https://github.com/lift/lift/blob/master/framework/lift-base/lift-json/src/main/scala/net/liftweb/json/Extraction.scala#L178
Related
Given a simple Scala case class like this:
package com.foo.storage.schema
case class Person(name: String, age: Int)
it's possible to create a Spark schema from a case class as follows:
import org.apache.spark.sql._
import com.foo.storage.schema.Person
val schema = Encoders.product[Person].schema
I wonder if it's possible to access the schema from a case class in Python/PySpark. I would hope to do something like this [Python]:
jvm = sc._jvm
py4j_class = jvm.com.foo.storage.schema.Person
jvm.org.apache.spark.sql.Encoders.product(py4j_class)
This throws an error com.foo.storage.schema.Person._get_object_id does not exist in the JVM. The Encoders.product is a generic in Scala, and I'm not entirely sure how to specify the type using Py4J. Is there a way to use the case class to create a PySpark schema?
I've found there's no clean / easy way to do this using generics, also not as a pure Scala function. What I ended up doing is making a companion object for the case class that can fetch the schema.
Solution
package com.foo.storage.schema
case class Person(name: String, age: Int)
object Person {
def getSchema = Encoders.product[Person].schema
}
This function can be called from Py4J, but will return a JavaObject. It can be converted with a helper function like this:
from pyspark.sql.types import StructType
import json
def java_schema_to_python(j_schema):
json_schema = json.loads(ddl.json())
return StructType.fromJson(json_schema)
Finally, we can extract our schema:
j_schema = jvm.com.foo.storage.Person.getSchema()
java_schema_to_python(j_schema)
Alternative solution
I found there is one more way to do this, but I like the first one better. You can make a generic function that infers the type of the argument in Scala, and uses that to infer the type:
object SchemaConverter {
def getSchemaFromType[T <: Product: TypeTag](obj: T): StructType = {
Encoders.product[T].schema
}
}
Which can be called like this:
val schema = SchemaConverter.getSchemaFromType(Person("Joe", 42))
I didn't like this method since it requires you to create a dummy instance of the case class. Haven't tested it, but I think the function above could be called using Py4J too.
Say I have a model like this:
case class Items(typeOfItems: TypeOfItems)
object Items{
implicit val format = Json.format[Items]
}
sealed trait TypeOfItems extends Product with Serializable
object TypeOfItems {
final case class Toy(typeOfItem : String, name : String, price : Int) extends TypeOfItems
final case class Car(typeOfItem : String, model: String, price : Int ) extends TypeOfItems
}
I cannot do any serialising or deserialising. I get the following errors:
No instance of play.api.libs.json.Format is available for model.TypeOfItems in the implicit scope (Hint: if declared in the same file, make sure it's declared before)
[error] implicit val format = Json.format[Items]
[error] ^
[error] No Json serializer found for type model.Items. Try to implement an implicit Writes or Format for this type.
[error] Ok(Json.toJson(Items(Toy("Toy", "Doll", 2))))
I am not sure how to specify the formats for TypeOfItems. How do I do that?
In Play 2.7 sealed traits are supported in play-json.
As already mentioned in comments all you are missing is:
object TypeOfItems{
implicit val format = Json.format[TypeOfItems]
}
See here the requirements in the documentation: ScalaJsonAutomated
For Play 2.6 see this answer: Noise free JSON format for sealed traits with Play 2.2 library (as the title suggests there are also solutions for older versions)
I have a JSON string and I created a function which parses this JSON as an object using Scala case class. I wrote the below code to parse it in a generic way. However, It gave me an error:
def getJsonObj[T](jsonString:String): T = {
implicit val formats: DefaultFormats.type = DefaultFormats
parse(jsonString).extract[T]
}
Error can be found below:
Error:(19, 32) No Manifest available for T.
parse(jsonString).extract[T] Error:(19, 32) not enough arguments for
method extract: (implicit formats: org.json4s.Formats, implicit mf:
scala.reflect.Manifest[T])T. Unspecified value parameter mf.
parse(jsonString).extract[T]
I found this No Manifest available for Type But I don't know how to fix it in my code. Also, I found this,Spark Scala - How to construct Scala Map from nested JSON (Scala: "No manifest available for type T") But I need to pass the case class to the function in a generic way. It seems a common problem but I can't solve it using the available answers as I am new in Scala.
Another point, How can I add try-catch to see if it's correctly parsed or not?
I think this answer solves your question, Scala: “No manifest available for type T”. It easily solved by implicitly passing on the manifest for the type method. I add an example of the code and a simple function for error handling.
val jsonStr: String = """{"airports":[{"name":"sfo","score":1},{"name":"phx","score":1},{"name":"sjc","score":1}]}"""
case class AirPortScores(name: String, score: Double)
case class JsonRulesHandler(airports: List[AirPortScores])
val json: JsonRulesHandler = getJsonObj[JsonRulesHandler](jsonStr)
println(json)
def getJsonObj[T](jsonString:String)(implicit m: Manifest[T]): T = {
extractFrom(jsonString) match {
case Success(jsonParsed) ⇒
jsonParsed
case Failure(exc) ⇒
throw new IllegalArgumentException(exc)
}
}
private def extractFrom[T](jsonString:String)(implicit m: Manifest[T]): Try[T] = {
implicit val formats: DefaultFormats.type = DefaultFormats
Try {
parse(jsonString).extract[T]
}
}
I use this library https://christopherdavenport.github.io/fuuid/ for creating ID of custom object and persist them into databse.
I have a simple case class which is my model:
import io.chrisdavenport.fuuid.FUUID
case class Bet(
betId: Option[FUUID],
home: String,
away: String,
stake: BigDecimal,
betType: String)
I used FUUID here as an Option parameter. I have also a routes created with Http4s which should take json from input and map it into model:
class BettingRoutes[F[_] : Async](service: BettingService[F]) extends Http4sDsl[F] {
def routes: HttpRoutes[F] = HttpRoutes.of[F] {
case req#PUT -> Root / "bets" =>
for {
bet <- req.as[Bet]
created <- service.put(bet)
response <- Created(created)
} yield response
}
}
I also added some implicits to encode and decode from Circe:
object jsons {
implicit def circeDecoder[A[_] : Sync, B: Decoder]: EntityDecoder[A, B] = jsonOf[A, B]
implicit def circeEncoder[A[_] : Sync, B: Encoder]: EntityEncoder[A, B] = jsonEncoderOf[A, B]
}
The problem is - when I want to compile project, I got an errors like this in route class:
Error:(23, 22) Cannot decode into a value of type model.Bet, because no EntityDecoder[F, model.Bet] instance could be found.
bet <- req.as[Bet]
Error:(23, 22) not enough arguments for method as: (implicit F: cats.Functor[F], implicit decoder: org.http4s.EntityDecoder[F,model.Bet])F[model.Bet].
Unspecified value parameter decoder.
bet <- req.as[Bet]
Error:(25, 28) Cannot convert from model.Bet to an Entity, because no EntityEncoder[F, model.Bet] instance could be found.
response <- Created(created)
etc. I investigated it and it appears because of using FUUID. I changed all FUUID classes to Long and after this just to java's UUID and then everything compile correctly without errors. The problem is only with FUUID and probably with conversion of it. I tried to use Circe Integration as It was shown in FUUID link above, but it did not help. Do you know how to fix this code to compile everything with fuuid and circe?
I am new to cats and connected libs, so maybe it is a simple mistake, but it is not trivial for me now.
In order to have EntityDecoder[F, Bet] via jsons.circeDecoder we firstly need Decoder[Bet]. It can be auto-generated by Circe if we have decoders for all fields. The thing is there is Decoder[UUID] but no Decoder[FUUID].
So just define necessary implicit
implicit val fuuidDecoder: Decoder[FUUID] = Decoder[UUID].map(FUUID.fromUUID)
Similarly for encoders
implicit val fuuidEncoder: Encoder[FUUID] = Encoder[UUID].contramap(FUUID.Unsafe.toUUID)
I'm trying to use "pickling" serialization is Scala, and I see the same example demonstrating it:
import scala.pickling._
import json._
val pckl = List(1, 2, 3, 4).pickle
Unpickling is just as easy as pickling:
val lst = pckl.unpickle[List[Int]]
This example raises some question. First of all, it skips converting of object to string. Apparently you need to call pckl.value to get json string representation.
Unpickling is even more confusing. Deserialization is an act of turning string (or bytes) into an object. How come this "example" demonstrates deserialization if there is no string/binry representation of object?
So, how do I deserialize simple object with pickling library?
Use the type system and case classes to achieve your goals. You can unpickle to some superior type in your hierarchy (up to and including AnyRef). Here is an example:
trait Zero
case class One(a:Int) extends Zero
case class Two(s:String) extends Zero
object Test extends App {
import scala.pickling._
import json._
// String that can be sent down a wire
val wire: String = Two("abc").pickle.value
// On the other side, just use a case class
wire.unpickle[Zero] match {
case One(a) => println(a)
case Two(s) => println(s)
case unknown => println(unknown.getClass.getCanonicalName)
}
}
Ok, I think I understood it.
import scala.pickling._
import json._
var str = Array(1,2,3).pickle.value // this is JSON string
println(str)
val x = str.unpickle[Array[Int]] // unpickle from string
will produce JSON string:
{
"tpe": "scala.Array[scala.Int]",
"value": [
1,
2,
3
]
}
So, the same way we pickle any type, we can unpickle string. Type of serialization is regulated by implicit formatter declared in "json." and can be replaced by "binary."
It does look like you will be starting with a pickle to unpickle to a case class. But the JSON string can be fed to the JSONPickle class to get the starting pickle.
Here's an example based on their array-json test
package so
import scala.pickling._
import json._
case class C(arr: Array[Int]) { override def toString = s"""C(${arr.mkString("[", ",", "]")})""" }
object PickleTester extends App {
val json = """{"arr":[ 1, 2, 3 ]}"""
val cPickle = JSONPickle( json )
val unpickledC: C = cPickle.unpickle[C]
println( s"$unpickledC, arr.sum = ${unpickledC.arr.sum}" )
}
The output printed is:
C([1,2,3]), arr.sum = 6
I was able to drop the "tpe" in from the test as well as the .stripMargin.trim on the input JSON from the test. It works all in one line, but I thought it might be more apparent split up. It's unclear to me if that "tpe" from the test is supposed to provide a measure of type safety for the incoming JSON.
Looks like the only other class they support for pickling is a BinaryPickle unless you want to roll your own. The latest scala-pickling snapshot jar requires quasiquotes to compile the code in this answer.
I tried someting more complicated this morning and discovered that the "tpe" is required for non-primatives in the incoming JSON - which points out that the serialized string really must be compatible with the pickler( which I mixed into the above code ):
case class J(a: Option[Boolean], b: Option[String], c: Option[Int]) { override def toString = s"J($a, $b, $c)" }
...
val jJson = """{"a": {"tpe": "scala.None.type"},
| "b":{"tpe": "scala.Some[java.lang.String]","x":"donut"},
| "c":{"tpe": "scala.Some[scala.Int]","x":47}}"""
val jPickle = JSONPickle( jJson.stripMargin.trim )
val unpickledJ: J = jPickle.unpickle[J]
println( s"$unpickledJ" )
...
where naturually, I had to use .value on a J(None, Some("donut"), Some(47)) to figure out how to create the jJson input value to prevent the unpickling from throwing an exception.
The output for J is like:
J(None, Some(donut), Some(47))
Looking at this test, it appears that if the incoming JSON is all primatives or case classes (or combinations) that the JSONPickle magic works, but some other classes like Options require extra "tpe" type information to unpickle correctly.