I have a json string:
val message = "{\"me\":\"a\",
\"version\":\"1.0\",
\"message_metadata\": \"{
\"event_type\":\"UpdateName\",
\"start_date\":\"1515\"}\"
}"
I want to extract the value of the field event_type from this json string.
I have used below code to extract the value:
val mapper = new ObjectMapper
val root = mapper.readTree(message)
val metadata =root.at("/message_metadata").asText()
val root1 = mapper.readTree(metadata)
val event_type =root1.at("/event_type").asText()
print("eventType:" + event_type.toString) //UpdateName
This works fine and I get the value as UpdateName. But I when I want to get the event type in a single line as below:
val mapper = new ObjectMapper
val root = mapper.readTree(message)
val event_type =root.at("/message_metadata/event_type").asText()
print("eventType:" + event_type.toString) //Empty string
Here event type returns a empty sting. This might be because of the message_metadata has Json object as a string value. Is there a way I can get the value of event_type in a single line?
The problem is that your JSON message contains an object who's message_metadata field itself contains JSON, so it must be decoded separately. I'd suggest that you don't put JSON into JSON but only encode the data structure once.
Example:
val message = "{\"me\":\"a\",
\"version\":\"1.0\",
\"message_metadata\": {
\"event_type\":\"UpdateName\",
\"start_date\":\"1515\"
}
}"
You can parse your JSON using case classes and then get your event_type field from there.
case class Json(me: String, version: String, message_metadata: Message)
case class Message(event_type: String, start_date: String)
object Mapping {
def main(args: Array[String]): Unit = {
import com.fasterxml.jackson.databind.ObjectMapper
import com.fasterxml.jackson.module.scala.DefaultScalaModule
import com.fasterxml.jackson.module.scala.experimental.ScalaObjectMapper
val objectMapper = new ObjectMapper() with ScalaObjectMapper
objectMapper.registerModule(DefaultScalaModule)
val str = "{\n \"me\": \"a\",\n \"version\": \"1.0\",\n \"message_metadata\": {\n \"event_type\": \"UpdateName\",\n \"start_date\": \"1515\"\n }\n}"
val json = objectMapper.readValue(str, classOf[Json])
//to print event_type
println(json.message_metadata.event_type)
//output: UpdateName
}
}
You can even convert a JSON to Scala Case Class and then get the particular field from the case class.
Please find a working and detailed answer which I have provided using generics here.
Related
I am trying to deserialize this JSON using Jackson library -
{
"name": "abc",
"ageInInt": 30
}
To the case class Person
case class Person(name: String, #JsonProperty(value = "ageInInt")#JsonAlias(Array("ageInInt")) age: Int)
but I am getting -
No usable value for age
Did not find value which can be converted into int
org.json4s.package$MappingException: No usable value for age
Did not find value which can be converted into int
Basically, I want to deserialize the json with the different key fields ageInInt to age.
here is the complete code -
val json =
"""{
|"name": "Tausif",
|"ageInInt": 30
|}""".stripMargin
implicit val format = DefaultFormats
println(Serialization.read[Person](json))
You need to register DefaultScalaModule to your JsonMapper.
import com.fasterxml.jackson.databind.json.JsonMapper
import com.fasterxml.jackson.module.scala.DefaultScalaModule
import com.fasterxml.jackson.core.`type`.TypeReference
import com.fasterxml.jackson.annotation.JsonProperty
val mapper = JsonMapper.builder()
.addModule(DefaultScalaModule)
.build()
case class Person(name: String, #JsonProperty(value = "ageInInt") age: Int)
val json =
"""{
|"name": "Tausif",
|"ageInInt": 30
|}""".stripMargin
val person: Person = mapper.readValue(json, new TypeReference[Person]{})
println(person) // Prints Person(Tausif,30)
I'am trying to convert prepareStament(object uses for sending SQL statement to the database ) to Json with scala.
So far, I've discovered that the best way to convert an object to Json in scala is to do it with the net.liftweb library.
But when I tried it, I got an empty json.
this is the code
import java.sql.DriverManager
import net.liftweb.json._
import net.liftweb.json.Serialization.write
object Main {
def main (args: Array[String]): Unit = {
implicit val formats = DefaultFormats
val jdbcSqlConnStr = "sqlserverurl**"
val conn = DriverManager.getConnection(jdbcSqlConnStr)
val statement = conn.prepareStatement("exec select_all")
val piedPierJSON2= write(statement)
println(piedPierJSON2)
}
}
this is the result
{}
I used an object I created , and the conversion worked.
case class Person(name: String, address: Address)
case class Address(city: String, state: String)
val p = Person("Alvin Alexander", Address("Talkeetna", "AK"))
val piedPierJSON3 = write(p)
println(piedPierJSON3)
This is the result
{"name":"Alvin Alexander","address":{"city":"Talkeetna","state":"AK"}}
I understood where the problem was, PrepareStament is an interface, and none of its subtypes are serializable...
I'm going to try to wrap it up and put it in a different class.
White Space in scala keys are not retained
case class c1(
state :String,
`Card No`:String
)
Request payload
{
"state": "HH",
"Card No": "c1234",
"xxxx": "xxxx"
}
When used that case class in code after converting from gson to Json
it becomes
{"state":"HH","Card$u0020No":"c1234","xxx":"xxx"}
I haven't tried out gson for JSON processing. I use Play JSON play.api.libs.json framework.
The below works in play JSON:
import play.api.libs.json._
case class MyClass(State: String, `Card No`: String)
// Play can convert any case class into a JsValue, using macros.
// you will need an implicit `Writes` for `MyClass`
implicit private val MyClassWrites = Json.writes[MyClass]
val myobj = MyClass(State = "Bangalore", `Card No` = "Bl##1234")
val json = Json.toJson(myobj)
println(json)
Gives the below JSON:
{"State":"Bangalore","Card No":"Bl##1234"}
Will update this once I checkout gson lib.
Update: With Gson
I would like you not having backquotes to define case class values, rather have it in the standard format - maybe as camel cases. This would help keep the code cleaner.
Like below:
case class MyClassA (state: String, cardNo: String)
and use the gson FieldNamingPolicy to get the right formatting while building the JSON.
You can create a gson builder and do something like this
val status = MyClassA(State = "Bangalore", cardNo = "Bl##1234")
val gson = new GsonBuilder()
.setFieldNamingPolicy(FieldNamingPolicy.UPPER_CAMEL_CASE_WITH_SPACES)
.setPrettyPrinting().create()
println(gson.toJson(status))
FieldNamingPolicy.UPPER_CAMEL_CASE_WITH_SPACES will ensure that the first letter of the field name is capitalized when serialized to its JSON form and the words will be separated by a space.
above will produce the below JSON.
{
"State": "Bangalore",
"Card No": "Bl##1234"
}
Also, you can use #SerializedName annotation on any field name to specify the serialized name.
Using annotation
Using the SerializedName as below:
import com.google.gson.annotations.SerializedName
import com.google.gson.GsonBuilder
import com.google.gson.FieldNamingPolicy
import com.google.gson.GsonBuilder._
case class MyClassA (
State: String,
#(SerializedName #scala.annotation.meta.field)("card no")
cardNo: String
)
object Test {
def main(args: Array[String]): Unit = {
val status = MyClassA(State = "Bangalore", CARD = "Bl##1234")
val gson = new GsonBuilder()
.setPrettyPrinting().create()
println(gson.toJson(status))
}
}
UPD: below is the library dependency I am using:
libraryDependencies += "com.google.code.gson" % "gson" % "2.8.6"
You need to use the meta-annotations in scala.annotation.meta. Refer this git issue
I'm trying to map string to double from an ArrayBuffer that I parsed through Playframework but I keep getting the following error:
Exception in thread "main" java.lang.NumberFormatException: For input string: ""0.04245800""
I'm not sure why it's doing this and I'm new to Scala coming from Python background.
import org.apache.http.client.methods.HttpGet
import play.api.libs.json._
import org.apache.http.impl.client.DefaultHttpClient
object main extends App {
val url = "https://api.binance.com/api/v1/aggTrades?symbol=ETHBTC"
val httpClient = new DefaultHttpClient()
val httpResponse = httpClient.execute(new HttpGet(url))
val entity = httpResponse.getEntity()
val content = ""
if (entity !=null) {
val inputStream = entity.getContent()
val result = io.Source.fromInputStream(inputStream).getLines.mkString
inputStream.close
println("REST API: " + url)
val json: JsValue = Json.parse(result)
var prices = (json\\"p")
println(prices.map(_.toString()).map(_.toDouble))
}
}
If you know for sure your list contains only strings you can cast them like this, and use the 'original' value to get the Double value from:
println(prices.map(_.as[JsString].value.toDouble))
As JsString is not a String you cannot call toDouble on that.
Just for completeness: If you are not certain your list contains only strings you should add an instanceof check or pattern matching.
I need to fetch the date from a file.
Below is my spark program:
import org.apache.spark.sql.SparkSession
import scala.xml.XML
import java.text.SimpleDateFormat
object Active6Month {
def main(args:Array[String]){
val format = new SimpleDateFormat("yyyy-MM-dd'T'hh:mm:ss.SSS")
val format1 = new SimpleDateFormat("yyyy-MM")
val spark = SparkSession.builder.appName("Active6Months").master("local").getOrCreate()
val data = spark.read.textFile("D:\\BGH\\StackOverFlow\\Posts.xml").rdd
val date = data.filter{line => {
line.toString().trim().startsWith("<row")
}}.filter{line=>{
line.contains("PostTypeId=\"1\"")
}}.map{line=>{
val xml = XML.loadString(line)
var closedDate = format1.format(format.parse(xml.attribute("ClosedDate").toString())).toString()
(closedDate,1)
}}.reduceByKey(_+_)
date.foreach(println)
spark.stop
}
}
And I am getting this error:
java.text.ParseException: Unparseable date: "Some(2014-05-14T14:40:25.950)"
The format of date in file is perfect i.e:
CreationDate="2014-05-13T23:58:30.457"
But in error it shows the String "Some" attached to it.
And my other question is why same working in below code:
val date = data.filter{line => {
line.toString().trim().startsWith("<row")
}}.filter{line=>{
line.contains("PostTypeId=\"1\"")
}}.flatMap{line=>{
val xml = XML.loadString(line)
xml.attribute("ClosedDate")
}}.map{line=>{
(format1.format(format.parse(line.toString())).toString(),1)
}}.reduceByKey(_+_)
My guess is that xml.attribute("ClosedDate").toString() is actually returning a string containing Some attached to it. Have you debugged that to make sure?
Maybe you shouldn't use toString(), but instead, get the attribute value, by using the proper method.
Or you can do it the "ugly" way and include "Some" in the pattern:
val format = new SimpleDateFormat("'Some('yyyy-MM-dd'T'hh:mm:ss.SSS')'")
Your second approach works because (and that's a guess because I don't code in Scala), probably the xml.attribute("ClosedDate") method returns an object, and calling toString() on this object returns the string with "Some" attached to it (why? ask the API authors). But when you use map on this object, it sets the line variable to the correct value (without the "Some" part).