Substitute value of an attribute based on conditions in scala - scala

I am working on an API module in scala which queries DB and returns the fetched data in JSON format as the response.
For a particular attribute in the JSON(whose type is Option[String]) , I need to alter the value fetched from DB based on certain conditions.
Currently , my JSON response looks like this:
.....
{
"id": "1",
"name": "abc",
"osType": "windows",
"osVersion": "10.0.14393"
},
{
"id": "2",
"name": "xyz",
"osType": "ubuntu",
"osVersion": "18.04"
},
{
"id": "3",
"name": "pqr",
"osType": "windows",
"osVersion": "6.1.7601"
},
......
I need to substitute the following values for osVersion for records with "osType": "windows"
if osVersion = 10.0.14393 , then Windows Server 2016
if osVersion starts with 6.1 , then Windows Server 2008 R2
So that desired output looks like this:
.....
{
"id": "1",
"name": "abc",
"osType": "windows",
"osVersion": "Windows Server 2016"
},
{
"id": "2",
"name": "xyz",
"osType": "ubuntu",
"osVersion": "18.04"
},
{
"id": "3",
"name": "pqr",
"osType": "windows",
"osVersion": "Windows Server 2008 R2"
},
......
I tried using if-else and pattern-matching , but am unable to substitute correctly as the type is Option[String].
This is how I tried:
(1)
def translateOSVersion (osType :Option[String] ,osVersion : Option[String]):Option[String] ={
if (osType == "windows"){
val version = if (osVersion =="10.0.14393")
{
Some("Windows Server 2016")
}
else {osVersion}
version
}
}
val oSVersion = translateOSVersion(ostr("osType") ,ostr("osVersion") )
<modelJSON>(id, name, ostr("osType"), oSVersion)
(2)
def translateOSVersion (osType :Option[String] ,osVersion : Option[String]) ={
if (osType == "windows"){
val version = if (osVersion =="10.0.14393") Some("Windows Server 2016")
else osVersion
}
}
val oSVersion = Some(translateOSVersion(ostr("osType") ,ostr("osVersion") ).toString)
<modelJSON>(id, name, ostr("osType"), oSVersion)
(3)
def translateOSVersion (osType :Option[String] ,osVersion : Option[String]) ={
if (osType == "windows"){
osVersion match {
case Some("10.0.14393") => "Windows Server 2016"
case x =>x
}
}
}
val oSVersion = Some(translateOSVersion(ostr("osType") ,ostr("osVersion") ).toString)
<modelJSON>(id, name, ostr("osType"), oSVersion)
All the approaches do not result in desired result.
Kindly help here.

Something like this should work:
def translateOSVersion(osType: Option[String], osVersion: Option[String]): Option[String] =
(for {
ot <- osType if ot == "windows"
ov <- osVersion
} yield {
if (ov == "10.0.14393") {
"Windows Server 2016"
} else if (ov.startsWith("6.1")) {
"Windows Server 2008 R2"
} else {
ov
}
}) orElse osVersion
This is how it works:
The for statement creates a new Option[String] representing the updated version number, or None if there is no change. It works by extracting the two Option values into ot for the OS Type and ov for the OS Version. If either of the Options is None it will return None, and it will also return None if the OS type is not "windows". If all these tests pass, the yield expression computes the new version based on these values. In this case the for will return Some(x) where x is the value of the yield expression.
The orElse expression will return the first value if it is Some(x), otherwise it will return the second value. So if the result of the for is Some(x) then it is returned as the result of the function. If the for returns None then the original value is returned as the result.
This is a more prosaic version that is probably more suitable, though less fun!
def translateOSVersion(osType: Option[String], osVersion: Option[String]): Option[String] =
if (osType.contains("windows")) {
if (osVersion.contains("10.0.14393")) {
Some("Windows Server 2016")
} else if (osVersion.exists(_.startsWith("6.1"))) {
Some("Windows Server 2008 R2")
} else {
osVersion
}
} else {
osVersion
}

Related

Scala play json - update all values with the same key

Let's say I have a JsValue in the form:
{
"businessDetails" : {
"name" : "Business",
"phoneNumber" : "+44 0808 157 0192"
},
"employees" : [
{
"name" : "Employee 1",
"phoneNumber" : "07700 900 982"
},
{
"name" : "Employee 2",
"phoneNumber" : "+44(0)151 999 2458"
}
]
}
I was wondering if there is a way to do an update on every value belonging to a key with a certain name inside a JsValue regardless of its complexity?
Ideally I'd like to map on every phone number to ensure that a (0) is removed if there is one.
I have come across play-json-zipper updateAll but I'm getting unresolved dependency issues when adding the library to my sbt project.
Any help either adding the play-json-zipper library or implementing this in ordinary play-json would be much appreciated.
Thanks!
From what I can see in play-json-zipper project page, you might forgot to add resolver resolvers += "mandubian maven bintray" at "http://dl.bintray.com/mandubian/maven"
If it won't help, and you would like to proceed with custom implementation: play-json does not provide folding or traversing API over JsValue out of the box, so it can be implemented recursively in the next way:
/**
* JSON path from the root. Int - index in array, String - field
*/
type JsPath = Seq[Either[Int,String]]
type JsEntry = (JsPath, JsValue)
type JsTraverse = PartialFunction[JsEntry, JsValue]
implicit class JsPathOps(underlying: JsPath) {
def isEndsWith(field: String): Boolean = underlying.lastOption.contains(Right(field))
def isEndsWith(index: Int): Boolean = underlying.lastOption.contains(Left(index))
def /(field: String): JsPath = underlying :+ Right(field)
def /(index: Int): JsPath = underlying :+ Left(index)
}
implicit class JsValueOps(underlying: JsValue) {
/**
* Traverse underlying json based on given partially defined function `f` only on scalar values, like:
* null, string or number.
*
* #param f function
* #return updated json
*/
def traverse(f: JsTraverse): JsValue = {
def traverseRec(prefix: JsPath, value: JsValue): JsValue = {
val lifted: JsValue => JsValue = value => f.lift(prefix -> value).getOrElse(value)
value match {
case JsNull => lifted(JsNull)
case boolean: JsBoolean => lifted(boolean)
case number: JsNumber => lifted(number)
case string: JsString => lifted(string)
case array: JsArray =>
val updatedArray = array.value.zipWithIndex.map {
case (arrayValue, index) => traverseRec(prefix / index, arrayValue)
}
JsArray(updatedArray)
case `object`: JsObject =>
val updatedFields = `object`.fieldSet.toSeq.map {
case (field, fieldValue) => field -> traverseRec(prefix / field, fieldValue)
}
JsObject(updatedFields)
}
}
traverseRec(Nil, underlying)
}
}
which can be used in the next way:
val json =
s"""
|{
| "businessDetails" : {
| "name" : "Business",
| "phoneNumber" : "+44(0) 0808 157 0192"
| },
| "employees" : [
| {
| "name" : "Employee 1",
| "phoneNumber" : "07700 900 982"
| },
| {
| "name" : "Employee 2",
| "phoneNumber" : "+44(0)151 999 2458"
| }
| ]
|}
|""".stripMargin
val updated = Json.parse(json).traverse {
case (path, JsString(phone)) if path.isEndsWith("phoneNumber") => JsString(phone.replace("(0)", ""))
}
println(Json.prettyPrint(updated))
which will produce desired result:
{
"businessDetails" : {
"name" : "Business",
"phoneNumber" : "+44 0808 157 0192"
},
"employees" : [ {
"name" : "Employee 1",
"phoneNumber" : "07700 900 982"
}, {
"name" : "Employee 2",
"phoneNumber" : "+44151 999 2458"
} ]
}
Hope this helps!

spark: how to merge rows to array of jsons

Input:
id1 id2 name value epid
"xxx" "yyy" "EAN" "5057723043" "1299"
"xxx" "yyy" "MPN" "EVBD" "1299"
I want:
{ "id1": "xxx",
"id2": "yyy",
"item_specifics": [
{
"name": "EAN",
"value": "5057723043"
},
{
"name": "MPN",
"value": "EVBD"
},
{
"name": "EPID",
"value": "1299"
}
]
}
I tried the following two solutions from How to aggregate columns into json array? and how to merge rows into column of spark dataframe as vaild json to write it in mysql:
pi_df.groupBy(col("id1"), col("id2"))
//.agg(collect_list(to_json(struct(col("name"), col("value"))).alias("item_specifics"))) // => not working
.agg(collect_list(struct(col("name"),col("value"))).alias("item_specifics"))
But I got:
{ "name":"EAN","value":"5057723043", "EPID": "1299", "id1": "xxx", "id2": "yyy" }
How to fix this? Thanks
For Spark < 2.4
You can create 2 dataframes, one with name and value and other with epic as name and epic value as value and union them together. Then aggregate them as collect_set and create a json. The code should look like this.
//Creating Test Data
val df = Seq(("xxx","yyy" ,"EAN" ,"5057723043","1299"), ("xxx","yyy" ,"MPN" ,"EVBD", "1299") )
.toDF("id1", "id2", "name", "value", "epid")
df.show(false)
+---+---+----+----------+----+
|id1|id2|name|value |epid|
+---+---+----+----------+----+
|xxx|yyy|EAN |5057723043|1299|
|xxx|yyy|MPN |EVBD |1299|
+---+---+----+----------+----+
val df1 = df.withColumn("map", struct(col("name"), col("value")))
.select("id1", "id2", "map")
val df2 = df.withColumn("map", struct(lit("EPID").as("name"), col("epid").as("value")))
.select("id1", "id2", "map")
val jsonDF = df1.union(df2).groupBy("id1", "id2")
.agg(collect_set("map").as("item_specifics"))
.withColumn("json", to_json(struct("id1", "id2", "item_specifics")))
jsonDF.select("json").show(false)
+---------------------------------------------------------------------------------------------------------------------------------------------+
|json |
+---------------------------------------------------------------------------------------------------------------------------------------------+
|{"id1":"xxx","id2":"yyy","item_specifics":[{"name":"MPN","value":"EVBD"},{"name":"EAN","value":"5057723043"},{"name":"EPID","value":"1299"}]}|
+---------------------------------------------------------------------------------------------------------------------------------------------+
For Spark = 2.4
It provides a array_union method. It might be helpful in doing it without union. I haven't tried it though.
val jsonDF = df.withColumn("map1", struct(col("name"), col("value")))
.withColumn("map2", struct(lit("epid").as("name"), col("epid").as("value")))
.groupBy("id1", "id2")
.agg(collect_set("map1").as("item_specifics1"),
collect_set("map2").as("item_specifics2"))
.withColumn("item_specifics", array_union(col("item_specifics1"), col("item_specifics2")))
.withColumn("json", to_json(struct("id1", "id2", "item_specifics2")))
You're pretty close. I believe you're looking for something like this:
val pi_df2 = pi_df.withColumn("name", lit("EPID")).
withColumnRenamed("epid", "value").
select("id1", "id2", "name","value")
pi_df.select("id1", "id2", "name","value").
union(pi_df2).withColumn("item_specific", struct(col("name"), col("value"))).
groupBy(col("id1"), col("id2")).
agg(collect_list(col("item_specific")).alias("item_specifics")).
write.json(...)
The union should bring back epid into item_specifics
Here is what you need to do
import scala.util.parsing.json.JSONObject
import scala.collection.mutable.WrappedArray
//Define udf
val jsonFun = udf((id1 : String, id2 : String, item_specifics: WrappedArray[Map[String, String]], epid: String)=> {
//Add epid to item_specifics json
val item_withEPID = item_specifics :+ Map("epid" -> epid)
val item_specificsArray = item_withEPID.map(m => ( Array(Map("name" -> m.keys.toSeq(0), "value" -> m.values.toSeq(0))))).map(m => m.map( mi => JSONObject(mi).toString().replace("\\",""))).flatten.mkString("[",",","]")
//Add id1 and id2 to output json
val m = Map("id1"-> id1, "id2"-> id2, "item_specifics" -> item_specificsArray.toSeq )
JSONObject(m).toString().replace("\\","")
})
val pi_df = Seq( ("xxx","yyy","EAN","5057723043","1299"), ("xxx","yyy","MPN","EVBD","1299")).toDF("id1","id2","name","value","epid")
//Add epid as part of group by column else the column will not be available after group by and aggregation
val df = pi_df.groupBy(col("id1"), col("id2"), col("epid")).agg(collect_list(map(col("name"), col("value")) as "map").as("item_specifics")).withColumn("item_specifics",jsonFun($"id1",$"id2",$"item_specifics",$"epid"))
df.show(false)
scala> df.show(false)
+---+---+----+--------------------------------------------------------------------------------------------------------------------------------------------------------------------+
|id1|id2|epid|item_specifics |
+---+---+----+--------------------------------------------------------------------------------------------------------------------------------------------------------------------+
|xxx|yyy|1299|{"id1" : "xxx", "id2" : "yyy", "item_specifics" : [{"name" : "MPN", "value" : "EVBD"},{"name" : "EAN", "value" : "5057723043"},{"name" : "epid", "value" : "1299"}]}|
+---+---+----+--------------------------------------------------------------------------------------------------------------------------------------------------------------------+
Content of item_specifics column/ output
{
"id1": "xxx",
"id2": "yyy",
"item_specifics": [{
"name": "MPN",
"value": "EVBD"
}, {
"name": "EAN",
"value": "5057723043"
}, {
"name": "epid",
"value": "1299"
}]
}

Array query in Scala and ReactiveMongo?

I have a MongoDB collection whose documents look like this:
{
"name" : "fabio",
"items" : [
{
"id" : "1",
"word" : "xxxx"
},
{
"id" : "2",
"word" : "yyyy"
}
]
}
Now, given one name and one id, I want to retrieve "name" and the corresponding "word".
I query it like this and it seems to work:
val query = BSONDocument("name" -> name, "items.id" -> id)
But then, how do I access the value of "word"? I can get the name using this reader:
The reader for this object is like this:
implicit object UserReader extends BSONDocumentReader[User] {
def read(doc: BSONDocument): User = {
val name = doc.getAs[String]("name").get
// how do I retrive the value of "word"?
User(id, word)
}
}
But I am very confused about "word".
Additionally, because I am only interested in two fields, how should I filter the query? The following doesn't seem to work.
val filter = BSONDocument("name" -> 1, "items.$.word" -> 1)
Thanks for your help!

How to serialize/deserialize dynamic field names using Play's json

I'm using Play framework 2.2.2.
I'm trying to handle json request like this one
[
{
"id" : "123",
"language" : "en",
"text" : "This is an example of a text",
"Metadata_IP" : "192.168.20.34",
"Metadata_date" : "2001-07-04T12:08:56.235-0700"
},
{
"id" : "124",
"language" : "en",
"text" : "Some more text here",
"Metadata_IP" : "192.168.20.31",
"Metadata_date" : "2001-07-04T12:09:56.235-0700",
"Metadata_name" : "someone"
}
]
The Metadata_ field are dynamic field meaning the user can send what ever he want (eg. Metadata_color, etc...)
What is the best way to handle this?
Can I use Readers with deserialize it to case class? How can I do this? I guess the dynamic field will be Map[String, String], but how should I make the reader parse this?
Thanks
Something like this could work:
implicit object jsObjToKeyValueSeq extends Reads[Seq[(String, String)]] {
override def reads(json: JsValue) = json match {
case js: JsObject =>
JsSuccess(js.fields.collect { case (key, JsString(value)) => key -> value })
case x => JsError(s"Unexpected json: $x")
}
}
We have faced the exact problem and solved it using a custom implementation. The solution is detailed here
Example:
Scala class
case class Person(name: String, age: String, customFields: Map[String,String])
Default Json representation of above class will be:
{
"name": "anil",
"age": "30",
"customFields": {
"field1": "value1",
"field2": "value2"
}
}
But what we wanted was:
{
"name": "anil",
"age": "30",
"field1": "value1",
"field2": "value2"
}
This was not very straight forward. While this could be possible using play framework, we didn’t want to complicate things too much. Finally we found a way to do it by returning a Map[String, String] which represents each class (it’s fields & values) using reflection and handle the behavior for custom fields separately.
case class Person(name: String, age: String, customFields:CustomFields)
case class CustomFields(valueMap: Map[String,String])
def covertToMap(ref: AnyRef) =
ref.getClass.getDeclaredFields.foldLeft(Map[String, Any]()){
(map, field) => {
field.setAccessible(true)
val value = field.get(ref)
value match {
case c: CustomFields => {
map ++ c.valueMap
}
case _ => {
map + (field.getName -> value)
}
}
}
}
Use the covertToMap() to convert any case class to a Map and then convert this map to normal Json using jackson json4s.
val json = Serialization.write(covertToMap(person))
Complete source code is available here

Creating a json in Play/Scala with same keys but different values

Here's what I want to achieve:
{ "user-list" : {
"user" : [
"username" : "foo"
},
{
"username" : "bar"
}
]
}
}
Im using play-framework and scala.
Thanks!
As previous commenters already pointed out, it is not obvious how to help you, given that your json code is invalid (try JSONLint) and that we don't know where it comes from (string? (case) classes from a database? literals?) and what you want to do with it.
Valid json code close to yours would be:
{
"user-list": {
"user": [
{ "username": "foo" },
{ "username": "bar" }
]
}
}
Depending on how much additional information your structure contains, the following might be sufficient (V1):
{
"user-list": [
{ "username": "foo" },
{ "username": "bar" }
]
}
Or even (V2):
{ "user-list": ["foo", "bar"] }
Following the Play documentation, you should be able to generate V1 with:
val jsonObject = Json.toJson(
Map(
"user-list" -> Seq(
toJson(Map("username" -> toJson("foo"))),
toJson(Map("username" -> toJson("bar")))
)
)
)
and V2 with:
val jsonObject = Json.toJson(
Map(
"user-list" -> Seq(toJson("foo"), toJson("bar"))
)
)