how to manipulate my dataframe in spark? - scala

Question

I have an nested json rdd stream comming in from a kafka topic.
the data looks like this:
{
"time":"sometext1","host":"somehost1","event":
{"category":"sometext2","computerName":"somecomputer1"}
}
I turned this into a dataframe and the schema looks like
root
|-- event: struct (nullable = true)
| |-- category: string (nullable = true)
| |-- computerName: string (nullable = true)
|-- time: string (nullable = true)
|-- host: string (nullable = true)
Im trying to save it to a hive table on hdfs with a schema like this
category:string
computerName:string
time:string
host:string
This is my first time working with spark and scala. I would appretiate if someone could help me.
Thanks

Answer 1

// Creating Rdd
val vals = sc.parallelize(
"""{"time":"sometext1","host":"somehost1","event": {"category":"sometext2","computerName":"somecomputer1"}}""" ::
Nil)
// Creating Schema
val schema = (new StructType)
.add("time", StringType)
.add("host", StringType)
.add("event", (new StructType)
.add("category", StringType)
.add("computerName", StringType))
import sqlContext.implicits._
val jsonDF = sqlContext.read.schema(schema).json(vals)
jsonDF.printSchema
root
|-- time: string (nullable = true)
|-- host: string (nullable = true)
|-- event: struct (nullable = true)
| |-- category: string (nullable = true)
| |-- computerName: string (nullable = true)
// selecting columns
val df = jsonDF.select($"event.*",$"time",
$"host")
df.printSchema
root
|-- category: string (nullable = true)
|-- computerName: string (nullable = true)
|-- time: string (nullable = true)
|-- host: string (nullable = true)
df.show
+---------+-------------+---------+---------+
| category| computerName| time| host|
+---------+-------------+---------+---------+
|sometext2|somecomputer1|sometext1|somehost1|
+---------+-------------+---------+---------+