I'm trying to make use of circe for json parsing in scala. Can you please help me parse 'pocs' from the data in the case class as well? here is the code:
import io.circe.Decoder
import io.circe.generic.semiauto.deriveDecoder
import io.circe.parser
val json: String =
"""
{
"segmements": [
{
"tableName": "X",
"segmentName": "XX",
"pocs": [
"aa#aa.com",
"bb#bb.com"
]
},
{
"tableName": "Y",
"segmentName": "YY",
"pocs": [
"aa#aa.com",
"bb#bb.com"
]
}
]
}
"""
final case class TableInfo(tableName: String, segmentName: String)
object TableInfo {
implicit final val TableInfoDecoder: Decoder[TableInfo] = deriveDecoder
}
val result = for {
data <- parser.parse(json)
obj <- data.asObject.toRight(left = new Exception("Data was not an object"))
segmements <- obj("segmements").toRight(left = new Exception("Json didn't had the
segments key"))
r <- segmements.as[List[TableInfo]]
} yield r
println(result)
scastie link: https://scastie.scala-lang.org/BalmungSan/eVEvBulOQwGzg5hIJroAoQ/3
Just add parameter typed as collection of String:
final case class TableInfo(tableName: String, segmentName: String, pocs: Seq[String])
scastie
Related
How to get the return value from For loop and pass it to .body(StringBody(session => in Gatling using Scala
I have created a method with for loop to generate String Array in gatling with scala
def main(args: Array[String]): Unit = {
var AddTest: Array[String] = Array[String]()
for (i <- 0 to 3) {
val TestBulk: String =
s"""{ "name": "Perftest ${Random.alphanumeric.take(6).mkString}",
"testID": "00000000-0000-0000-0000-000000006017",
"typeId": "00000000-0000-0000-0000-000000011001",
"statusId": "00000000-0000-0000-0000-000000005058"};"""
AddTest = TestBulk.split(",")
// val TestBulk2: Nothing = AddTest.mkString(",").replace(';', ',')
// println(TestBulk)
}
}
now I want to pass the return value to .body(StringBody(session =>
.exec(http("PerfTest Bulk Json")
.post("/PerfTest/bulk")
.body(StringBody(session =>
s"""[(Value from the for loop).mkString(",").replace(';', ',')
]""".stripMargin)).asJson
Please help me with the possibilities
Please let me know if
You don't need for loop (or var or split) for this. You also do not have ; anywhere, so last replace is pointless.
val ids = """
"testId": "foo",
"typeId": "bar",
"statusId": "baz"
"""
val data = (1 to 3)
.map { _ => Random.alphanumeric.take(6).mkString }
.map { r => s""""name": "Perftest $r"""" }
.map { s => s"{ $s, $ids }" }
.mkString("[", ",", "]")
exec("foo").post("/bar").body(_ => StringBody(data)).asJson
(I added [ and ] around your generated string to make it look like valid json).
Alternatively, you probably have some library that converts maps and lists to json out-of-the box (I don't know gatling, but there must be something), a bit cleaner way to do this would be with something like this:
val ids = Map(
"testId" -> "foo",
"typeId" -> "bar",
"statusId" -> "baz"
)
val data = (1 to 3)
.map { _ => Random.alphanumeric.take(6).mkString }
.map { r => ids + ("name" -> s"Perftest $r") }
exec("foo").post("/bar").body(_ => StringBody(toJson(data))).asJson
This Worked for me
Thanks to
#dima .
I build this with your suggested method.
import scala.util.Random
import math.Ordered.orderingToOrdered
import math.Ordering.Implicits.infixOrderingOps
import play.api.libs.json._
import play.api.libs.json.Writes
import play.api.libs.json.Json.JsValueWrapper
val data1 = (1 to 2)
.map {r => Json.toJson(Map(
"name" -> Json.toJson(s"Perftest${Random.alphanumeric.take(6).mkString}"),
"domainId"->Json.toJson("343RDFDGF4RGGFG"),
"typeId"->Json.toJson("343RDFDGF4RGGFG"),
"statusId"->Json.toJson("343RDFDGF4RGGFG"),
"excludedFromAutoHyperlinking" ->Json.toJson(true)))}
println(Json.toJson(data1))```
I have two JSONs with the exact same keys.
val json1 =
"""{
'name': 'Henry',
'age' : 26,
'activities' : {
'school': 'basketball club',
'after-school': 'chess'
}
}"""
val json2 =
"""{
'name': 'David',
'age' : 23,
'activities' : {
'school': 'baseball club',
'after-school': 'programming'
}
}"""
I would like the difference between the two JSONs, for example, such as:
name = Henry, David
age = 23, 26
activities.school= basketball club, baseball club
activities.after-school=chess, programming
It doesn't have to follow the above format but, I would like to get the keys and values that are differing.
You could try diffson, a circe based library: https://github.com/gnieh/diffson
Example:
import diffson._
import diffson.lcs._
import diffson.circe._
import diffson.jsonpatch._
import diffson.jsonpatch.lcsdiff._
import io.circe._
import io.circe.parser._
import cats._
import cats.implicits._
implicit val lcs = new Patience[Json]
val json1 = parse("""{
| "a": 1,
| "b": true,
| "c": ["test", "plop"]
|}""".stripMargin)
val json2 = parse("""{
| "a": 6,
| "c": ["test2", "plop"],
| "d": false
|}""".stripMargin)
val patch =
for {
json1 <- json1
json2 <- json2
} yield diff(json1, json2)
Would return:
[{
"op":"replace",
"path":"/a",
"value":6
},{
"op":"remove",
"path":"/b"
},{
"op":"replace",
"path":"/c/0",
"value":"test2"
},{
"op":"add",
"path":"/d",
"value":false
}]
Basically you can concatenate "add" section values for your exact purpose.
As was already suggested by #dk14 you can use diffson librarry: https://github.com/gnieh/diffson - but JsonPatch structure which it provides might be not very convenient for your use case, so it can be converted into another to get result in desired format.
Please, see some code example below:
import diffson.jsonpatch.{Add, JsonPatch, Remove, Replace}
import diffson.jsonpointer.{Part, Pointer}
import io.circe.Json
// Model representing plain json diff at certain path, that can be rendered at more human readable format
case class JsonPathDiff(path: Pointer, left: Option[Json], right: Option[Json]) {
def readableString: String = {
val pathReadableString: String = {
def partToString(part: Part): String = part.fold(identity, _.toString)
path.parts.toList.map(partToString).mkString(".")
}
def jsonReadableValue(json: Option[Json]): String = json.map(_.toString()).getOrElse("")
val leftValue = jsonReadableValue(left)
val rightValue = jsonReadableValue(right)
s"$pathReadableString = $leftValue , $rightValue"
}
}
// Model representing overall difference between two JSON's
case class JsonDiff(diff: List[JsonPathDiff]) {
def readableString: String = diff.map(_.readableString).mkString("\n")
}
object JsonDiff {
def fromPatch(patch: JsonPatch[Json]): JsonDiff = {
val paths = patch.ops.collect {
case Add(path, value) => JsonPathDiff(path, None, Some(value))
case Remove(path, old) => JsonPathDiff(path, old, None)
case Replace(path, value, old) => JsonPathDiff(path, old, Some(value))
}
JsonDiff(paths)
}
}
def main(args: Array[String]): Unit = {
import diffson._
import diffson.circe._
import diffson.jsonpatch._
import diffson.jsonpatch.lcsdiff.remembering._
import diffson.lcs._
import io.circe._
import io.circe.parser._
val json1 =
s"""{
"name": "Henry",
"age" : 26,
"activities" : {
"school": "basketball club",
"after-school": "chess"
}
}"""
val json2 =
s"""{
"name": "David",
"age" : 23,
"activities" : {
"school": "baseball club",
"after-school": "programming"
}
}"""
implicit val lcs = new Patience[Json]
val patch: Either[ParsingFailure, JsonPatch[Json]] =
for {
json1 <- parse(json1)
json2 <- parse(json2)
} yield diff(json1, json2)
val jsonDiff = JsonDiff.fromPatch(patch.right.get) // Using `get` for sake of example, avoid in real production code
println(jsonDiff.readableString)
}
which will produce next result:
activities.after-school = "chess" , "programming"
activities.school = "basketball club" , "baseball club"
age = 26 , 23
name = "Henry" , "David"
Hope this helps!
I have a JSON string like
{
"key1": "value1",
"definition": {
// JSON content here
}
}
"definition" key in JSON can contains JSONArray, JSONObject.
For example, it can have
"key2" : ""
or
"key2" : {}
or
"key2" : []
To accommodate this, I have created corresponding Scala class like
import com.google.gson.JsonObject
class JsonData {
var key1: String = _
var definition: JsonObject = _
}
While mapping JSON string to class JsonData, I am getting "definition" in JsonData instance as empty.
Sample code:
import com.fasterxml.jackson.databind.{DeserializationFeature, ObjectMapper}
import com.fasterxml.jackson.module.scala.DefaultScalaModule
import com.fasterxml.jackson.module.scala.experimental.ScalaObjectMapper
import com.google.gson.JsonObject
object TestMe {
val mapper = new ObjectMapper with ScalaObjectMapper
mapper.registerModule(DefaultScalaModule)
mapper.configure(DeserializationFeature.FAIL_ON_UNKNOWN_PROPERTIES, false)
def main(args: Array[String]): Unit = {
val jsonString = "{\"key1\": \"value1\",\"definition\": {\"key2\" : \"abc\"}}"
val data = mapper.readValue[JsonData](jsonString)
println(data.definition.getAsJsonObject()) //empty
val jsonString1 = "{\"key1\": \"value1\",\"definition\": {\"key2\" : {\"key3\" : \"\"}}}"
val data1 = mapper.readValue[JsonData](jsonString1)
println(data1.definition.getAsJsonObject()) //empty
val jsonString2 = "{\"key1\": \"value1\",\"definition\": {\"key2\" : [\"a\",\"b\"]}}"
val data2 = mapper.readValue[JsonData](jsonString2)
println(data2.definition.getAsJsonObject()) //empty
}
class JsonData {
var key1: String = _
var definition: JsonObject = _
}
}
How can I read JSON string and map it to class which has one of its attribute type of JsonObject?
Versions:
Scala : 2.11
Jackson-core = 2.6.x;
Gson = 2.6.x;
Jackson-databind = 2.6.x;
Jackson-module-scala = 2.6.5;
I would use com.fasterxml.jackson.databind.JsonNode instead of using Google's Gson JsonObject class. Using Jackson's own classes should make it pretty trivial.
Although you may just map to a Map[String, Any] instead for this kind of flexibility, unless you really need it to still be in Json.
Sample Json
"alternateId": [
{
"type": "POPID",
"value": "1-7842-0759-001"
},
{
"type": "CAMID",
"value": "CAMID 0000-0002-7EC1-02FF-O-0000-0000-2"
},
{
"type": "ProgrammeUuid",
"value": "1ddb01e2-6146-4e10-bba9-dde40d0ad886"
}
]
I want to update a existing dataframe with two columns, those two columns are POPID and CAMID . These two values needs to be parsed from json structure
I dont know how to parse this structure , Can you help me on what do i need to change on fetchField method. As per above json POPID is placed first and CAMID is placed second, but in real jsons, it can be placed at one of those 3 places inside alternateId.
val fetchCAMID_udf = udf(fetchCAMID _)
val fetchPOPID_udf = udf(fetchPOPID _)
var updatedDf = //Data frame initialize
updatedDf = updatedDf.withColumn("CAMID", fetchCAMID_udf(col("alternate_id")))
updatedDf = updatedDf.withColumn("POPID", fetchPOPID_udf(col("alternate_id")))
updatedDf .show(10,false)
def fetchCAMID(jsonStr: String): String = {
var CAMID: String = fetchField(jsonStr, "CAMID")
CAMID
}
def fetchPOPID(jsonStr: String): String = {
fetchField(jsonStr, "POPID")
}
def fetchField(jsonStr: String, fieldName: String): String = {
try {
implicit val formats = DefaultFormats
val extractedField = jsonStr match {
case "(unknown)" => jsonStr
case _ => {
val json = JsonMethods.parse(jsonStr)
val resultExtracted = (json \\ fieldName)
val result = resultExtracted match {
case _: JString => resultExtracted.extract[String]
case _: JInt => resultExtracted.extract[Int].toString
case _: JObject => "(unknown)"
}
result
}
}
extractedField
}
catch{
case e: Exception =>{
log.error(s"Fetch field failed. Field name: $fieldName . Json: $jsonStr")
"(unknown)"
}
}
}
Change your fetchField function as the following
def fetchField(jsonStr: String, fieldName: String): String = {
try {
val typeAndValue = (JsonMethods.parse("{"+jsonStr+"}") \ "alternateId" \ "type" \\ classOf[JString]).zip(JsonMethods.parse("{"+jsonStr+"}") \ "alternateId" \ "value" \\ classOf[JString])
typeAndValue.filter(_._1 == fieldName).map(_._2).toList(0)
}catch{
case e: Exception =>{
"(unknown)"
}
}
}
and you get the CAMID and POPID populated
you can read the JSON using Spark and get it using regular spark operations
val df=spark.read.option("multiLine",true).json("test.json")
df.select($"alternateId".getItem(0).as("pop"),$"alternateId".getItem(1).as("cam")).select($"pop.value".as("POPID"),$"cam.value".as("CAMID")).show()
+---------------+--------------------+
| POPID| CAMID|
+---------------+--------------------+
|1-7842-0759-001|CAMID 0000-0002-7...|
+---------------+--------------------+
I want to convert a jvalue to json string. Here is what my code look like:
import org.json4s._
import org.json4s.JsonDSL._
import org.json4s.native.JsonMethods._
import org.json4s.DefaultFormats._
object Json4sTest {
def main(arg: Array[String]) {
var json = parse("""{"name":"luca", "id": "1q2w3e4r5t", "age": 26, "inner": { "age": 27 }, "url":"http:// www.nosqlnocry.wordpress.com"}""")
// println(json)
val a: List[Map[String, JValue]] = List(Map("inner/age" -> 35, "age" -> 27), Map("name" -> "foo"))
val r = jsonFieldUpdater(json, a)
println(r)
}
def jsonFieldUpdater(json: JValue, list: List[Map[String, JValue]]): JValue =
//
}
}
gets me the result as :
JObject(List((name,JString(foo)), (id,JString(1q2w3e4r5t)), (age,JInt(27)), (inner,JObject(List((age,JInt(35))))), (url,JString(http:// www.nosqlnocry.wordpress.com))))
I am looking for a Json String output as :
{"name":"luca", "id": "1q2w3e4r5t", "age": 27, "inner": { "age": 35 }, "url":"http:// www.nosqlnocry.wordpress.com"}
If you want to print string representation of JValue, you can do it by:
println(compact(render(r)))
This way the String received by method compact(render(r)) looks like this:
{"lotto":{"lotto-id":5,"winning-numbers":[2,45,34,23,7,5,3],"winners":[{"winner-id":23,"numbers":[2,45,34,23,3,5]},{"winner-id":54,"numbers":[52,3,12,11,18,22]}]}}
Or you can use println(pretty(render(r))) to obtain pretty String like this:
{
"lotto":{
"lotto-id":5,
"winning-numbers":[2,45,34,23,7,5,3],
"winners":[{
"winner-id":23,
"numbers":[2,45,34,23,3,5]
},{
"winner-id":54,
"numbers":[52,3,12,11,18,22]
}]
}
}
Examples used from documentation.