How to convert wrapped array to dataset in spark scala? - scala

Question

Hi I am new to spark scala. I have this structure in json file which I need to convert it to dataset. I am being unable to do this because of the nested data.
I tried to do something like this which I got from some post but it does not work. Can someone please suggest me the solution?
spark.read.json(path).map(r=>r.getAs[mutable.WrappedArray[String]]("readings"))

Answer 1

Your JSON format is invalid for spark to convert into dataframe. json informations that needs to be converted into dataframe/dataset row should be a line.
So the first step for you to do is read the json file and convert into valid json format. You can use wholeTextFiles api and some replacements.
val rdd = sc.wholeTextFiles("path to your json text file")
val validJson = rdd.map(_._2.replace(" ", "").replace("\n", ""))
Second step is to covert the valid json data into dataframe or dataset. Here I am using dataframe
val dataFrame = sqlContext.read.json(validJson)
which should give you
+--------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
|did |readings |
+--------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
|d7cc92c24be32d5d419af1277289313c|[[WrappedArray([aa1111111111111111c1111111111111112222222222e,AppleiOS,-46,49,ITU++], [09dfs1111111111111c1111111111111112222222222e,AppleiOS,-50,45,ITU++]),1506770544]]|
+--------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
root
|-- did: string (nullable = true)
|-- readings: array (nullable = true)
| |-- element: struct (containsNull = true)
| | |-- clients: array (nullable = true)
| | | |-- element: struct (containsNull = true)
| | | | |-- cid: string (nullable = true)
| | | | |-- clientOS: string (nullable = true)
| | | | |-- rssi: long (nullable = true)
| | | | |-- snRatio: long (nullable = true)
| | | | |-- ssid: string (nullable = true)
| | |-- ts: long (nullable = true)
Now selecting WrappedArray is easy step as
dataFrame.select("readings.clients")
which should give you
+------------------------------------------------------------------------------------------------------------------------------------------------------------+
|clients |
+------------------------------------------------------------------------------------------------------------------------------------------------------------+
|[WrappedArray([aa1111111111111111c1111111111111112222222222e,AppleiOS,-46,49,ITU++], [09dfs1111111111111c1111111111111112222222222e,AppleiOS,-50,45,ITU++])]|
+------------------------------------------------------------------------------------------------------------------------------------------------------------+
I hope the answer is helpful
Updated
Dataframe and datasets are almost the same except that datasets are type safety with encoders used, and that datasets are optimized than dataframes.
Long story short, you can change the dataframe to dataset by creating case classes. For your case you would need three case classes.
case class client(cid: String, clientOS: String, rssi: Long, snRatio: Long, ssid: String)
case class reading(clients: Array[client], ts: Long)
case class dataset(did: String, readings: Array[reading])
And then cast the dataframe to dataset as
val dataSet = sqlContext.read.json(validJson).as[dataset]
You should have dataset in your hand :)

Answer 2

You cannot create DataSet with the following code
spark.read.json(path).map(r => r.getAs[WrappedArray[String]]("readings"))
Check the schema of clients type for the DF created upon reading the JSON.
spark.read.json(path).printSchema
root
|-- did: string (nullable = true)
|-- readings: array (nullable = true)
| |-- element: struct (containsNull = true)
| | |-- clients: array (nullable = true)
| | | |-- element: struct (containsNull = true)
| | | | |-- cid: string (nullable = true)
| | | | |-- clientOS: string (nullable = true)
| | | | |-- rssi: long (nullable = true)
| | | | |-- snRatio: long (nullable = true)
| | | | |-- ssid: string (nullable = true)
| | |-- ts: long (nullable = true)
You can get the scala.collection.mutable.WrappedArrayobject with the below code
spark.read.json(path).first.getAs[WrappedArray[(String,String,Long,Long,String)]]("readings")
If you need a create the dataframe use the below.
spark.read.json(path).select("readings.clients")