I need to change from one List[String] to List[MyObject] in scala.
For example,JSON input is like below
employee: {
name: "test",
employeeBranch: ["CSE", "IT", "ECE"]
}
Output should be like this,
Employee: {
Name: "test",
EmployeeBranch:[{"branch": "CSE"}, {"branch": "IT"}, {"branch": "ECE"}]
}
Input case class:
Class Office(
name: Option[String],
employeeBranch: Option[List[String]])
Output case class:
Class Output(
Name: Option[String],
EmployeeBranch: Option[List[Branch]])
case class Branch(
branch: Option[String])
This is the requirement.
It is hard to answer without knowing details of the particular JSON library, but an Object is probably represented as a Map. So to convert a List[String] to a List[Map[String, String]] you can do this:
val list = List("CSE", "IT", "ECE")
val map = list.map(x => Map("branch" -> x))
This gives
List(Map(branch -> CSE), Map(branch -> IT), Map(branch -> ECE))
which should convert to the JSON you want.
Related
I have a dataset with following case class type:
case class AddressRawData(
addressId: String,
customerId: String,
address: String
)
I want to convert it to:
case class AddressData(
addressId: String,
customerId: String,
address: String,
number: Option[Int], //i.e. it is optional
road: Option[String],
city: Option[String],
country: Option[String]
)
Using a parser function:
def addressParser(unparsedAddress: Seq[AddressData]): Seq[AddressData] = {
unparsedAddress.map(address => {
val split = address.address.split(", ")
address.copy(
number = Some(split(0).toInt),
road = Some(split(1)),
city = Some(split(2)),
country = Some(split(3))
)
}
)
}
I am new to scala and spark. Could anyone please let me know how can this be done?
You were on the right path! There are multiple ways of doing this of course. But as you're already on the way by making some case classes, and you've started making a parsing function an elegant solution is by using the Dataset's map function. From the docs, this map function signature is the following:
def map[U](func: (T) ⇒ U)(implicit arg0: Encoder[U]): Dataset[U]
Where T is the starting type (AddressRawData in your case) and U is the type you want to get to (AddressData in your case). So the input of this map function is a function that transforms a AddressRawData to a AddressData. That could perfectly be the addressParser you've started making!
Now, your current addressParser has the following signature:
def addressParser(unparsedAddress: Seq[AddressData]): Seq[AddressData]
In order to be able to feed it to that map function, we need to make this signature:
def newAddressParser(unparsedAddress: AddressRawData): AddressData
Knowing all of this, we can work further! An example would be the following:
import spark.implicits._
import scala.util.Try
// Your case classes
case class AddressRawData(addressId: String, customerId: String, address: String)
case class AddressData(
addressId: String,
customerId: String,
address: String,
number: Option[Int],
road: Option[String],
city: Option[String],
country: Option[String]
)
// Your addressParser function, adapted to be able to feed into the Dataset.map
// function
def addressParser(rawAddress: AddressRawData): AddressData = {
val addressArray = rawAddress.address.split(", ")
AddressData(
rawAddress.addressId,
rawAddress.customerId,
rawAddress.address,
Try(addressArray(0).toInt).toOption,
Try(addressArray(1)).toOption,
Try(addressArray(2)).toOption,
Try(addressArray(3)).toOption
)
}
// Creating a sample dataset
val rawDS = Seq(
AddressRawData("1", "1", "20, my super road, beautifulCity, someCountry"),
AddressRawData("1", "1", "badFormat, some road, cityButNoCountry")
).toDS
val parsedDS = rawDS.map(addressParser)
parsedDS.show
+---------+----------+--------------------+------+-------------+----------------+-----------+
|addressId|customerId| address|number| road| city| country|
+---------+----------+--------------------+------+-------------+----------------+-----------+
| 1| 1|20, my super road...| 20|my super road| beautifulCity|someCountry|
| 1| 1|badFormat, some r...| null| some road|cityButNoCountry| null|
+---------+----------+--------------------+------+-------------+----------------+-----------+
As you see, thanks to the fact that you had already foreseen that parsing can go wrong, it was easily possible to use scala.util.Try to try and get the pieces of that raw address and add some robustness in there (the second line contains some null values where it could not parse the address string.
Hope this helps!
I have this following json input where I am trying to parse the name field in-order
scala> result
res6: play.api.libs.json.JsValue = {"L0":
{"name":"FASHION","id":"50000"},"L1":{"name":"ACCESSORIES AND TRAVEL","id":"51000"},"L2":{"name":"FASHION ACCESSORIES","id":"51001"},"L3":{"name":"MENS FASHION ACCESSORIES","id":"51100"},"L4":{"name":"MENS HATS","id":"51204"}}
scala> result \\ "name"
res5: Seq[play.api.libs.json.JsValue] = List("ACCESSORIES AND TRAVEL", "MENS HATS", "MENS FASHION ACCESSORIES", "FASHION ACCESSORIES", "FASHION")
What I am trying is to get those names in-order like
List("FASHION", "ACCESSORIES AND TRAVEL", "FASHION ACCESSORIES", "MENS FASHION ACCESSORIES", "MENS HATS")
Is there a way to achieve that with play Json library?
With Play JSON I always use case classes. So your example would look like:
import play.api.libs.json._
val json = """{"L0":
{"name":"FASHION","id":"50000"},"L1":{"name":"ACCESSORIES AND TRAVEL","id":"51000"},"L2":{"name":"FASHION ACCESSORIES","id":"51001"},"L3":{"name":"MENS FASHION ACCESSORIES","id":"51100"},"L4":{"name":"MENS HATS","id":"51204"}}
"""
case class Element(id: String, name: String)
object Element {
implicit val jsonFormat: Format[Element] = Json.format[Element]
}
Json.parse(json).validate[Map[String, Element]] match {
case JsSuccess(elems, _) => println(elems.toList.sortBy(_._1).map(e => e._2.name))
case other => println(s"Handle exception $other")
}
What this gives you, is that you can sort the result by the key - the info that is lost in your solution.
I've been working with this example from the Elastic4s manual. It is working fine until it attempts to retrieve a document that does not have a field specified in the case class.
In this example from the manual, let's say one result only had name and was missing the location field. It would yield this error:
java.util.NoSuchElementException: key not found: location
I'm looking for a good approach to deal with search results that have varying fields.
Code sample:
case class Character(name: String, location: String)
implicit object CharacterHitAs extends HitAs[Character] {
override def as(hit: RichSearchHit): Character = {
Character(hit.sourceAsMap("name").toString, hit.sourceAsMap("location").toString) }}
val resp = client.execute {
search in "gameofthrones" / "characters" query "kings landing"
}.await
val characters :Seq[Character] = resp.as[Character]
When developing a case class with optional parameters, use Option:
case class Character(name: String, location: Option[String])
Character("Tyrion Lannister", None)
Then all you have to do is modify your data extractor to pass a None Option if it doesn't find the data:
val tyrion = Map("location" -> "King's Landing", "name" -> "Cersei Lannister")
val cersei = Map("father" -> "Tywin Lannister?", "name" -> "Cersei Lannister")
val jaime = Map("father" -> "Tywin Lannister", "location" -> "Tower of the Hand")
val characters = List(tyrion, cersei, jaime)
case class Character(name: String, location: Option[String])
characters.map(x => Character(x.getOrElse("name", "A CHARACTER HAS NO NAME"), x.get("location")))
The result of characters.map(...) is this:
res0: List[Character] = List(
Character(Cersei Lannister,Some(King's Landing)),
Character(Cersei Lannister,None),
Character(A CHARACTER HAS NO NAME NAME,Some(Tower of the Hand)))
From the source code for RichSearchHit, sourceAsMap should return a Map object:
def sourceAsMap: Map[String, AnyRef] = if (java.sourceAsMap == null) Map.empty else java.sourceAsMap.asScala.toMap
Given that you're using a Map shorthand, you should be able to convert your code to:
case class Character(name: String, location: Option[String])
implicit object CharacterHitAs extends HitAs[Character] {
override def as(hit: RichSearchHit): Character = {
Character(hit.sourceAsMap.getOrElse("name", "A CHARACTER HAS NO NAME"), hit.sourceAsMap.get("location")) }}
I have a Json object stored in Mongo like below. It is 'flat', i.e. no nested elements:
{
"key1" : "val1",
"key2" : "val2",
....
"keyn" : "valn"
}
I have fetched it as a JsArray. I also have a case class:
case class IndividualProduct(key1: String, key2: String, ... , key_n: String) {}
In total the Json will have over 40 key/value pairs. Is there a neat way to parse the JsArray into the case class without verbosely referencing the keys?
thanks in advance - Future[Thanks]
import play.api.libs.json._
implicit val reader = Json.reads[IndividualProduct]
val ip = Json.fromJson[IndividualProduct](fetchedJsObj)
That's not a JsArray, but rather a Map[String, String].
So if you have a json like the one you showed, here's what can work:
val json = getYourJsonFromDB()
val kv = json.as[Map[String, String]]
Now you'll be able to do something like this:
val valueForKey13 = kv.get("key13") //returns an Option[String]
Hope this helps
I was able to perform simple validations on simple json structures like this one:
object RestTest extends Controller {
case class Address(street: String,
number: Int)
case class Person(name: String,
age: Int,
address: Address)
implicit val address = Json.reads[Address]
implicit val rds = Json.reads[Person]
def restTest = Action(parse.json) {
request =>
request.body.validate[Person].map {
case person => Ok(Json.obj("e" -> 0, "message" -> ("The name is: " + person.name + " and he lives in " + person.address.street)))
}.recoverTotal(e => Ok("e" -> 1)
}
}
Now I have the following structure that contains arrays, but I wasn't able to validate it correctly so far. I have tried many different ways, but I keep receiving compilation errors.
case class SecondStructure(index: Int)
case class EntryStructure(field1: String,
muSecondJsonArray: List[SecondStructure])
case class MyJsonArray(allEntries: List[EntryStructure])
How can I validate this json?
Thanks
First of all, ensure you are using the latest Play 2.1.1 releases. There was an issue with earlier versions when validating case classes with a single field. After that, it should all work - please see below for an example:
object JsonTest {
case class SecondStructure(index: Int)
case class EntryStructure(field1: String, muSecondJsonArray: List[SecondStructure])
case class MyJsonArray(allEntries: List[EntryStructure])
// Use the macro "inception" feature to automatically build your Readers.
implicit val ssReads = Json.reads[SecondStructure]
implicit val esReads = Json.reads[EntryStructure]
implicit val arrayReads = Json.reads[MyJsonArray]
// Defining an example instance...
val testArray = MyJsonArray(
List(
EntryStructure("foo", List(SecondStructure(1), SecondStructure(2))),
EntryStructure("bar", List(SecondStructure(3), SecondStructure(4)))))
// And the equivilant JSON structure...
val testJson = Json.obj("allEntries" ->
Json.arr(
Json.obj("field1" -> "foo", "muSecondJsonArray" -> Json.arr(
Json.obj("index" -> 1), Json.obj("index" -> 2))),
Json.obj("field1" -> "bar", "muSecondJsonArray" -> Json.arr(
Json.obj("index" -> 3), Json.obj("index" -> 4)))))
testJson.validate[MyJsonArray].map {
case foo if foo == testArray => println("Okay, we're good!")
}
}