I am trying to parse list of dictionaries(which is in string) inside scala. Basically i want to build another list so that i can traverse through the list using a for loop.
When i have one single list of dictionaries it works fine.
class CC[T] { def unapply(a:Any):Option[T] = Some(a.asInstanceOf[T]) }
object M extends CC[Map[String, Any]]
object A extends CC[List[Any]] //for s3
object I extends CC[Double]
object S extends CC[String]
object E extends CC[String]
object F extends CC[String]
object G extends CC[Map[String, Any]]
val jsonString =
"""
{
"index": 1,
"source": "a",
"name": "v",
"s3": [{
"path": "s3://1",
"bucket": "p",
"key": "r"
}]
}
""".stripMargin
//println(List(JSON.parseFull(jsonString)))
val result = for {
Some(M(map)) <- List(JSON.parseFull(jsonString))
//L(text) = map("text")
//M(texts) <- text
I(index) = map("index")
S(source) = map("source")
N(name) = map("name")
A(s3q)=map("s3")
G(s3data) <- s3q
F(path) = s3data("path")
} yield {
(index.toInt,source,name,path)
}
But when i aded another list, it gives error stating "java.lang.ClassCastException: scala.collection.immutable.$colon$colon cannot be cast to scala.collection.immutable.Map"
class CC[T] { def unapply(a:Any):Option[T] = Some(a.asInstanceOf[T]) }
object M extends CC[Map[String, Any]]
object A extends CC[List[Any]] //for s3
object I extends CC[Double]
object S extends CC[String]
object E extends CC[String]
object F extends CC[String]
object G extends CC[Map[String, Any]]
val jsonString =
"""
[{
"index": 1,
"source": "a",
"name": "v",
"s3": [{
"path": "s3://1",
"bucket": "p",
"key": "r"
}]
},{
"index": 1,
"source": "a",
"name": "v",
"s3": [{
"path": "s3://1",
"bucket": "p",
"key": "r"
}]
}]
""".stripMargin
//println(List(JSON.parseFull(jsonString)))
val result = for {
Some(M(map)) <- List(JSON.parseFull(jsonString))
//L(text) = map("text")
//M(texts) <- text
I(index) = map("index")
S(source) = map("source")
N(name) = map("name")
A(s3q)=map("s3")
G(s3data) <- s3q
F(path) = s3data("path")
} yield {
(index.toInt,source,name,path)
}
Related
I am trying to extract the following avro record
{
"StateName": "Alabama",
"Capital": "Montgomery",
"Counties": [{
"CountyName": "Baldwin",
"CountyPopulation": 200000,
"Cities": [{
"CityName": "Daphne",
"CityPopulation": 20000
},
{
"CityName": "Foley",
"CityPopulation": 14000
}
]
}, {
"CountyName": "Calhoun",
"CountyPopulation": 100000,
"Cities": [{
"CityName": "Anniston",
"CityPopulation": 23000
},
{
"CityName": "Glencoe",
"CityPopulation": 5000
}
]
}]
}
and modify them and create new individual record like this(Extract each county and create new records based on county like this)
{
"StateName": "Alabama",
"Capital": "Montgomery",
"CountyName": "Baldwin",
"CountyPopulation": 200000,
"Cities": [{
"CityName": "Daphne",
"CityPopulation": 20000
},
{
"CityName": "Foley",
"CityPopulation": 14000
}
]
}
I am trying to extract the records using the json4s. Taken the reference from https://nmatpt.com/blog/2017/01/29/json4s-custom-serializer/
val StateName = avroRecord.get("StateName").asInstanceOf[Utf8].toString
val Capital = avroRecord.get("Capital").asInstanceOf[Utf8].toString
val CountyArray = avroRecord.get("Counties").toString
val jsonData = parse(CountyArray, useBigDecimalForDouble = true)
val CountyList = jsonData match {
case JArray(_) =>
jsonData.extract[List[CountyArrayRecord]]
case JObject(_) =>
List(jsonData.extract[CountyArrayRecord])
List()
}
Custom serializer
implicit val formats: Formats = Serialization.formats(NoTypeHints) + new TestSerializer
class TestSerializer extends CustomSerializer[CountyArrayRecord](format => (
{ case jsonObj: JObject =>
val countyName = (jsonObj \ "CountyName").extract[String]
val countyPopulation = (jsonObj \ "CountyPopulation").extract[Int]
val cities = (jsonObj \ "Cities").extract[List[GenericRecord]]
CountyArrayRecord(countyName, countyPopulation, cities)
}
)
)
Once extracted trying to create list new records using avro4s.Taken reference from this https://github.com/sksamuel/avro4s#avro-records
val returnList = CountyList.map { CountyListRecord =>
val record = FinalCountyRecord (StateName, Capital, CountyListRecord.CountyName, CountyListRecord.CountyPopulation, CountyListRecord.Cities)
val format = RecordFormat[FinalCountyRecord]
format.to(record)
}
returnList
But this does not seem to work since county list has another list(Cities) inside.
I have the JSon response as given below.
If metadata's Organic=true then label='true-Organic', else label='non-Organic'
in the end => return List or Map[modelId,label]
import net.liftweb.json.{DefaultFormats, _}
object test1 extends App {
val json_response =
"""{
"requestId": "91ee60d5f1b45e#316",
"error": null,
"errorMessages": [
],
"entries": [
{
"modelId":"RT001",
"sku": "SKU-ASC001",
"store": "New Jersey",
"ttlInSeconds": 8000,
"metadata": {
"manufactured_date": "2019-01-22T01:25Z",
"organic": "true"
}
},
{
"modelId":"RT002",
"sku": "SKU-ASC002",
"store": "livingstone",
"ttlInSeconds": 8000,
"metadata": {
"manufactured_date": "2019-10-03T01:25Z",
"organic": "false"
}
}
] }"""
tried like this :
val json = parse(json_response)
implicit val formats = DefaultFormats
var map = Map[String, String]()
case class Sales(modelId: String, sku: String, store: String, ttlInSeconds: Int, metadata:
Map[String, String])
case class Response(entries: List[Sales])
val response = json.extract[Response]
After this, not sure how to proceed.
This is a straightforward map operation on the entries field:
response.entries.map{ e =>
e.modelId ->
if (e.metadata.get("organic").contains("true")) {
"true-Organic"
} else {
"non-Organic"
}
}
This will return List[(String, String)], but you can call toMap to turn this into a Map if required.
For instance I have a simple author's JSON:
{
"name": "string",
"articles": [
"string",
"string",
"string"
]
}
I am defining Json reading format:
implicit object authorFormat extends RootJsonFormat[Author] {
override def write(a: Author) = ???
override def read(json: JsValue) = json.asJsObject.getFields("name", "articles") match {
case Seq(JsString(name), JsArray(articles)) => ... //Build author
}
}
This pattern matching guarantees "articles" is an array(of JsValues). But it is not guarantees "articles" is an array of strings. How to fix it?
Try deserialising articles to List[String] like so
articles.map(_.convertTo[String]).toList
if jsonFormat2 is not an option. Here is a working example
import spray.json._
import DefaultJsonProtocol._
object MyApp extends App {
case class Author(name: String, articles: List[String])
implicit object authorFormat extends RootJsonFormat[Author] {
override def write(a: Author) = ???
override def read(json: JsValue) = json.asJsObject.getFields("name", "articles") match {
case Seq(JsString(name), JsArray(articles)) => Author(name, articles.map(_.convertTo[String]).toList)
}
}
val raw =
"""
|{
| "name": "string",
| "articles": [
| "string,
| "string",
| "string"
| ]
|}
""".stripMargin
val json = raw.parseJson
val author = json.convertTo[Author]
println(author)
}
which outputs
Author(string,List(string, string, string))
whilst if articles contains a non-string like so
val raw =
"""
|{
| "name": "string",
| "articles": [
| 1,
| "string",
| "string"
| ]
|}
""".stripMargin
we get error
Exception in thread "main" spray.json.DeserializationException: Expected String as JsString, but got 1
How do you create a jackson custom serializer and use it in your program? The serializer is used to serialize data from a kafka stream, because my job fails if it encounters a null.
I tried the following to create a serializer.
import org.json4s._
import org.json4s.jackson.JsonMethods._
case class Person(
val user: Option[String]
)
object PersonSerializer extends CustomSerializer[Person](formats => ( {
case JObject(JField("user", JString(user)) :: Nil) => Person(Some(user))
case JObject(JField("user", null) :: Nil) => Person(None)
},
{
case Person(Some(user)) => JObject(JField("user", JString(user)) :: Nil)
case Person(None) => JObject(JField("user", JString(null)) :: Nil)
}))
I am trying to use it this way.
object ConvertJsonTOASTDeSerializer extends App
{
case class Address(street : String, city : String)
case class PersonAddress(name : String, address : Address)
val testJson1 =
"""
{ "user": null,
"address": {
"street": "Bulevard",
"city": "Helsinki",
"country": {
"code": "CD" }
},
"children": [
{
"name": "Mary",
"age": 5,
"birthdate": "2004-09-04T18:06:22Z"
},
{
"name": "Mazy",
"age": 3
}
]
}
"""
implicit var formats : Formats = DefaultFormats + PersonSerializer
val output = parse(testJson1).as[Person]
println(output.user)
}
I am getting an error saying that
Error:(50, 35) No JSON deserializer found for type com.examples.json4s.Person. Try to implement an implicit Reader or JsonFormat for this type.
val output = parse(testJson1).as[Person]
Not sure if I answer your question. I provide the runnable code:
import org.json4s._
import org.json4s.jackson.JsonMethods._
case class Person(
user: Option[String],
address: Address,
children: List[Child]
)
case class Address(
street: String,
city: String,
country: Country
)
case class Country(
code: String
)
case class Child(
name: String,
age: Int
)
val s =
"""
{ "user": null,
"address": {
"street": "Bulevard",
"city": "Helsinki",
"country": {
"code": "CD" }
},
"children": [
{
"name": "Mary",
"age": 5,
"birthdate": "2004-09-04T18:06:22Z"
},
{
"name": "Mazy",
"age": 3
}
]
}
"""
implicit val formats : Formats = DefaultFormats
parse(s).extract[Person] // Person(None,Address(Bulevard,Helsinki,Country(CD)),List(Child(Mary,5), Child(Mazy,3)))
How does one control the deserialization for spray-json? For example, I have a class defined as:
case class A (Name:String, Value:String)
And I would like to deserialize the following JSON into a List of A objects:
{
"one": "1",
"two": "2"
}
and it should become:
List(A("one", "1"), A("two", "2"))
The problem is that the default JSON representation of that List is this one, which I do not want:
[
{ "Name": "one", "Value": "1" },
{ "Name": "two", "Value": "2" }
]
How can I accomplish this?
You can write your own custom deserializer for the structure you are looking for:
case class A(Name:String, Value:String)
implicit object ListAFormat extends RootJsonReader[List[A]] {
override def read(json: JsValue): List[A] = {
json.asJsObject.fields.toList.collect {
case (k, JsString(v)) => A(k, v)
}
}
}
import spray.json._
def main(args: Array[String]): Unit = {
val json =
"""
|{
| "one": "1",
| "two": "2"
|}
""".stripMargin
val result = json.parseJson.convertTo[List[A]]
println(result)
}
Prints:
List(A(one,1), A(two,2))