I've been working with this example from the Elastic4s manual. It is working fine until it attempts to retrieve a document that does not have a field specified in the case class.
In this example from the manual, let's say one result only had name and was missing the location field. It would yield this error:
java.util.NoSuchElementException: key not found: location
I'm looking for a good approach to deal with search results that have varying fields.
Code sample:
case class Character(name: String, location: String)
implicit object CharacterHitAs extends HitAs[Character] {
override def as(hit: RichSearchHit): Character = {
Character(hit.sourceAsMap("name").toString, hit.sourceAsMap("location").toString) }}
val resp = client.execute {
search in "gameofthrones" / "characters" query "kings landing"
}.await
val characters :Seq[Character] = resp.as[Character]
When developing a case class with optional parameters, use Option:
case class Character(name: String, location: Option[String])
Character("Tyrion Lannister", None)
Then all you have to do is modify your data extractor to pass a None Option if it doesn't find the data:
val tyrion = Map("location" -> "King's Landing", "name" -> "Cersei Lannister")
val cersei = Map("father" -> "Tywin Lannister?", "name" -> "Cersei Lannister")
val jaime = Map("father" -> "Tywin Lannister", "location" -> "Tower of the Hand")
val characters = List(tyrion, cersei, jaime)
case class Character(name: String, location: Option[String])
characters.map(x => Character(x.getOrElse("name", "A CHARACTER HAS NO NAME"), x.get("location")))
The result of characters.map(...) is this:
res0: List[Character] = List(
Character(Cersei Lannister,Some(King's Landing)),
Character(Cersei Lannister,None),
Character(A CHARACTER HAS NO NAME NAME,Some(Tower of the Hand)))
From the source code for RichSearchHit, sourceAsMap should return a Map object:
def sourceAsMap: Map[String, AnyRef] = if (java.sourceAsMap == null) Map.empty else java.sourceAsMap.asScala.toMap
Given that you're using a Map shorthand, you should be able to convert your code to:
case class Character(name: String, location: Option[String])
implicit object CharacterHitAs extends HitAs[Character] {
override def as(hit: RichSearchHit): Character = {
Character(hit.sourceAsMap.getOrElse("name", "A CHARACTER HAS NO NAME"), hit.sourceAsMap.get("location")) }}
Related
I am using case classes to extract json with json4s's extract method. Unfortunately, the Natural Earth source data I am using isn't consistent about casing... at some resolutions a field is called iso_a2 and at some it's ISO_A2. I can only make json4s accept the one that matches the field in the case class:
object TopoJSON {
case class Properties(ISO_A2: String)
...
// only accepts capitalised version.
Is there any way to make json4s ignore case and accept both?
There is no way to make it case insensitive using the configuration properties, but a similar result can be achieved by either lowercasing or uppercasing the field names in the parsed JSON.
For example, we have our input:
case class Properties(iso_a2: String)
implicit val formats = DefaultFormats
val parsedLower = parse("""{ "iso_a2": "test1" }""")
val parsedUpper = parse("""{ "ISO_A2": "test2" }""")
We can lowercase all field names using a short function:
private def lowercaseAllFieldNames(json: JValue) = json transformField {
case (field, value) => (field.toLowerCase, value)
}
or make it for specific fields only:
private def lowercaseFieldByName(fieldName: String, json: JValue) = json transformField {
case (field, value) if field == fieldName => (fieldName.toLowerCase, value)
}
Now, to extract the case class instances:
val resultFromLower = lowercaseAllFieldNames(parsedLower).extract[Properties]
val resultFromUpper = lowercaseAllFieldNames(parsedUpper).extract[Properties]
val resultByFieldName = lowercaseFieldByName("ISO_A2", parsedUpper).extract[Properties]
// all produce expected items:
// Properties(test1)
// Properties(test2)
// Properties(test2)
There are many nice libraries for writing/reading Scala case classes to/from CSV files. I'm looking for something that goes beyond that, which can handle nested cases classes. For example, here a Match has two Players:
case class Player(name: String, ranking: Int)
case class Match(place: String, winner: Player, loser: Player)
val matches = List(
Match("London", Player("Jane",7), Player("Fred",23)),
Match("Rome", Player("Marco",19), Player("Giulia",3)),
Match("Paris", Player("Isabelle",2), Player("Julien",5))
)
I'd like to effortlessly (no boilerplate!) write/read matches to/from this CSV:
place,winner.name,winner.ranking,loser.name,loser.ranking
London,Jane,7,Fred,23
Rome,Marco,19,Giulia,3
Paris,Isabelle,2,Julien,5
Note the automated header line using the dot "." to form the column name for a nested field, e.g. winner.ranking. I'd be delighted if someone could demonstrate a simple way to do this (say, using reflection or Shapeless).
[Motivation. During data analysis it's convenient to have a flat CSV to play around with, for sorting, filtering, etc., even when case classes are nested. And it would be nice if you could load nested case classes back from such files.]
Since a case-class is a Product, getting the values of the various fields is relatively easy. Getting the names of the fields/columns does require using Java reflection.
The following function takes a list of case-class instances and returns a list of rows, each is a list of strings. It is using a recursion to get the values and headers of child case-class instances.
def toCsv(p: List[Product]): List[List[String]] = {
def header(c: Class[_], prefix: String = ""): List[String] = {
c.getDeclaredFields.toList.flatMap { field =>
val name = prefix + field.getName
if (classOf[Product].isAssignableFrom(field.getType)) header(field.getType, name + ".")
else List(name)
}
}
def flatten(p: Product): List[String] =
p.productIterator.flatMap {
case p: Product => flatten(p)
case v: Any => List(v.toString)
}.toList
header(classOf[Match]) :: p.map(flatten)
}
However, constructing case-classes from CSV is far more involved, requiring to use reflection for getting the types of the various fields, for creating the values from the CSV strings and for constructing the case-class instances.
For simplicity (not saying the code is simple, just so it won't be further complicated), I assume that the order of columns in the CSV is the same as if the file was produced by the toCsv(...) function above.
The following function starts by creating a list of "instructions how to process a single CSV row" (the instructions are also used to verify that the column headers in the CSV matches the the case-class properties). The instructions are then used to recursively produce one CSV row at a time.
def fromCsv[T <: Product](csv: List[List[String]])(implicit tag: ClassTag[T]): List[T] = {
trait Instruction {
val name: String
val header = true
}
case class BeginCaseClassField(name: String, clazz: Class[_]) extends Instruction {
override val header = false
}
case class EndCaseClassField(name: String) extends Instruction {
override val header = false
}
case class IntField(name: String) extends Instruction
case class StringField(name: String) extends Instruction
case class DoubleField(name: String) extends Instruction
def scan(c: Class[_], prefix: String = ""): List[Instruction] = {
c.getDeclaredFields.toList.flatMap { field =>
val name = prefix + field.getName
val fType = field.getType
if (fType == classOf[Int]) List(IntField(name))
else if (fType == classOf[Double]) List(DoubleField(name))
else if (fType == classOf[String]) List(StringField(name))
else if (classOf[Product].isAssignableFrom(fType)) BeginCaseClassField(name, fType) :: scan(fType, name + ".")
else throw new IllegalArgumentException(s"Unsupported field type: $fType")
} :+ EndCaseClassField(prefix)
}
def produce(instructions: List[Instruction], row: List[String], argAccumulator: List[Any]): (List[Instruction], List[String], List[Any]) = instructions match {
case IntField(_) :: tail => produce(tail, row.drop(1), argAccumulator :+ row.head.toString.toInt)
case StringField(_) :: tail => produce(tail, row.drop(1), argAccumulator :+ row.head.toString)
case DoubleField(_) :: tail => produce(tail, row.drop(1), argAccumulator :+ row.head.toString.toDouble)
case BeginCaseClassField(_, clazz) :: tail =>
val (instructionRemaining, rowRemaining, constructorArgs) = produce(tail, row, List.empty)
val newCaseClass = clazz.getConstructors.head.newInstance(constructorArgs.map(_.asInstanceOf[AnyRef]): _*)
produce(instructionRemaining, rowRemaining, argAccumulator :+ newCaseClass)
case EndCaseClassField(_) :: tail => (tail, row, argAccumulator)
case Nil if row.isEmpty => (Nil, Nil, argAccumulator)
case Nil => throw new IllegalArgumentException("Not all values from CSV row were used")
}
val instructions = BeginCaseClassField(".", tag.runtimeClass) :: scan(tag.runtimeClass)
assert(csv.head == instructions.filter(_.header).map(_.name), "CSV header doesn't match target case-class fields")
csv.drop(1).map(row => produce(instructions, row, List.empty)._3.head.asInstanceOf[T])
}
I've tested this using:
case class Player(name: String, ranking: Int, price: Double)
case class Match(place: String, winner: Player, loser: Player)
val matches = List(
Match("London", Player("Jane", 7, 12.5), Player("Fred", 23, 11.1)),
Match("Rome", Player("Marco", 19, 13.54), Player("Giulia", 3, 41.8)),
Match("Paris", Player("Isabelle", 2, 31.7), Player("Julien", 5, 16.8))
)
val csv = toCsv(matches)
val matchesFromCsv = fromCsv[Match](csv)
assert(matches == matchesFromCsv)
Obviously this should be optimized and hardened if you ever want to use this for production...
simple case class:
case class country(name: String, townPopulation: Map[String,Int])
with simple example:
scala> val germany = country("Germany",Map("Berlin" -> 100000, "Saale" -> 4000))
germany: country = country(Germany,Map(Berlin -> 100000, Saale -> 4000))
scala> germany.townPopulation("Berlin")
res77: Int = 100000
scala> germany.townPopulation("blergh")
java.util.NoSuchElementException: key not found: blergh
at scala.collection.MapLike$class.default(MapLike.scala:228)
at scala.collection.AbstractMap.default(Map.scala:59)
at scala.collection.MapLike$class.apply(MapLike.scala:141)
at scala.collection.AbstractMap.apply(Map.scala:59)
... 42 elided
I would like to return 0 for towns that dont exist, this can be done when declaring a val:
scala> val germany = country("Germany",Map("Berlin" -> 100000, "Saale" -> 4000).withDefaultValue(0))
germany: country = country(Germany,Map(Berlin -> 100000, Saale -> 4000))
scala> germany.townPopulation("fdhjkjhkhjdfg")
res79: Int = 0
but I can not figure out how to do it in one place, at least not when it is a case class, I would like something simple as the following, but I am obviously doing it wrong:
scala> case class country(name: String, townPopulation: Map[String,Int].withDefaultValue(0))
<console>:1: error: ')' expected but '.' found.
case class country(name: String, townPopulation: Map[String,Int].withDefaultValue(0))
^
<console>:1: error: ';' expected but ')' found.
case class country(name: String, townPopulation: Map[String,Int].withDefaultValue(0))
Is there a short simple path to a solution that has 0 as defaultValue always?
I see few possible ways:
add auxiliary method which encapsulate logic of default value
def population(town : String) : Int = townPopulation.getOrElse(town, 0)
add method to companion object with same purpose
def withDefault(name: String, townPopulation: Map[String, Int]) : country =
country(name, townPopulation.withDefaultValue(0))
Use map.get(), which returns an Option:
println germany.townPopulation.get("blergh").getOrElse(0)
// or, more concisely:
println germany.townPopulation.getOrElse("blergh", 0)
Ah, on re-reading your question, you want to hard-code the default value in the case class. I guess you'll have to mess with the apply() method.
val germany = country("Germany",
Map("Berlin" -> 100000, "Saale" -> 4000)
.withDefaultValue(0))
Edit (after OP's answer):
My bad! Should have read more thoroughly your question.
As stated in this SO question: You do not have the option of changing the way the default constructor stores its parameters (e.g. by modifying the parameters before they are stored as vals) […].
An alternate solution would be to declare a regular class along with its companion object:
class Country(val name: String, val townPopulation: Map[String, Int])
case object Country {
def apply(name: String, townPopulation: Map[String, Int]) =
new Country(name, townPopulation.withDefaultValue(0))
}
This would allow you to declare countries using the nice syntax:
val germany = Country("Germany", Map("Berlin" -> 100000, "Saale" -> 4000))
assert(germany.townPopulation("Berlin") == 100000)
assert(germany.townPopulation("Blergh") == 0)
But note that, as it is not a case class you won't get the usual case class perks such as:
// Compiler will give you
// "object Country is not a case class,
// nor does it have an unapply/unapplySeq member"
//germany match {
// case Country(a, b) => println("It won't compile! Not a case class")
//}
Depending on your use case, you could go the long hard road and implement methods unapply and unapplySeq to retrieve such behavior if needed!
I was able to perform simple validations on simple json structures like this one:
object RestTest extends Controller {
case class Address(street: String,
number: Int)
case class Person(name: String,
age: Int,
address: Address)
implicit val address = Json.reads[Address]
implicit val rds = Json.reads[Person]
def restTest = Action(parse.json) {
request =>
request.body.validate[Person].map {
case person => Ok(Json.obj("e" -> 0, "message" -> ("The name is: " + person.name + " and he lives in " + person.address.street)))
}.recoverTotal(e => Ok("e" -> 1)
}
}
Now I have the following structure that contains arrays, but I wasn't able to validate it correctly so far. I have tried many different ways, but I keep receiving compilation errors.
case class SecondStructure(index: Int)
case class EntryStructure(field1: String,
muSecondJsonArray: List[SecondStructure])
case class MyJsonArray(allEntries: List[EntryStructure])
How can I validate this json?
Thanks
First of all, ensure you are using the latest Play 2.1.1 releases. There was an issue with earlier versions when validating case classes with a single field. After that, it should all work - please see below for an example:
object JsonTest {
case class SecondStructure(index: Int)
case class EntryStructure(field1: String, muSecondJsonArray: List[SecondStructure])
case class MyJsonArray(allEntries: List[EntryStructure])
// Use the macro "inception" feature to automatically build your Readers.
implicit val ssReads = Json.reads[SecondStructure]
implicit val esReads = Json.reads[EntryStructure]
implicit val arrayReads = Json.reads[MyJsonArray]
// Defining an example instance...
val testArray = MyJsonArray(
List(
EntryStructure("foo", List(SecondStructure(1), SecondStructure(2))),
EntryStructure("bar", List(SecondStructure(3), SecondStructure(4)))))
// And the equivilant JSON structure...
val testJson = Json.obj("allEntries" ->
Json.arr(
Json.obj("field1" -> "foo", "muSecondJsonArray" -> Json.arr(
Json.obj("index" -> 1), Json.obj("index" -> 2))),
Json.obj("field1" -> "bar", "muSecondJsonArray" -> Json.arr(
Json.obj("index" -> 3), Json.obj("index" -> 4)))))
testJson.validate[MyJsonArray].map {
case foo if foo == testArray => println("Okay, we're good!")
}
}
I'm a Scala beginner and this piece of code makes me struggle.
Is there a way to do pattern matching to make sure everything i pass to Data is of the correct type? As you can see i have quite strange datatypes...
class Data (
val recipient: String,
val templateText: String,
val templateHtml: String,
val blockMaps: Map[String,List[Map[String,String]]],
templateMap: Map[String,String]
)
...
val dataParsed = JSON.parseFull(message)
dataParsed match {
case dataParsed: Map[String, Any] => {
def e(s: String) = dataParsed get s
val templateText = e("template-text")
val templateHtml = e("template-html")
val recipient = e("email")
val templateMap = e("data")
val blockMaps = e("blkdata")
val dependencies = new Data(recipient, templateText, templateHtml, blockMaps, templateMap)
Core.inject ! dependencies
}
...
I guess your problem is you want to be able to patten match the map that you get from parseFull(), but Map doesn't have an unapply.
So you could pattern match every single value, providing a default if it is not of the correct type:
val templateText: Option[String] = e("template-text") match {
case s: String => Some(s)
case _ => None
}
Or temporarily put all the data into some structure that can be pattern matched:
val data = (e("template-text"), e("template-html"), e("email"), e("data"),
e("blkdata"))
val dependencies: Option[Data] = data match {
case (templateText: String,
templateHtml: String,
blockMaps: Map[String,List[Map[String,String]]],
templateMap: Map[String,String]) =>
Some(new Data(recipient, templateText, templateHtml, blockMaps, templateMap))
case _ => None
}