How to convert nested object from rdd row to some custom object - scala

Question

I'm trying to learn some scala/spark and trying to practice using some basic spark integration example. So my problem is that I have a Mongo db running locally. I'm pulling some data and making an rdd from it. The data in db has a structure like that:
{
"_id": 0,
"name": "aimee Zank",
"scores": [
{
"score": 1.463179736705023,
"type": "exam"
},
{
"score": 11.78273309957772,
"type": "quiz"
},
{
"score": 35.8740349954354,
"type": "homework"
}
]
}
Here is some code:
val conf: SparkConf = new SparkConf().setMaster("local[*]").setAppName("simple-app")
val sparkSession = SparkSession.builder()
.appName("example-spark-scala-read-and-write-from-mongo")
.config(conf)
.config("spark.mongodb.output.uri", "mongodb://sproot:12345#172.18.0.3:27017/spdb.students")
.config("spark.mongodb.input.uri", "mongodb://sproot:12345#172.18.0.3:27017/spdb.students")
.getOrCreate()
// Reading Mongodb collection into a dataframe
val df = MongoSpark.load(sparkSession)
val dataRdd: RDD[Row] = df.rdd
dataRdd.foreach(row => println(row.getValuesMap[Any](row.schema.fieldNames)))
The code above provides me this:
Map(_id -> 0, name -> aimee Zank, scores -> WrappedArray([1.463179736705023,exam], [11.78273309957772,quiz], [35.8740349954354,homework]))
Map(_id -> 1, name -> Aurelia Menendez, scores -> WrappedArray([60.06045071030959,exam], [52.79790691903873,quiz], [71.76133439165544,homework]))
At the end I have a problem converting this data to:
case class Student(id: Long, name: String, scores: Scores)
case class Scores(#JsonProperty("scores") scores: List[Score])
case class Score (
#JsonProperty("score") score: Double,
#JsonProperty("type") scoreType: String
)
To conclude - the problem is that I cannot convert some data from RDD to the Student object. The most problematic place for me is that 'scores' nested object.
Please help me to understand how this should be done.

Answer 1

Played a bit more with it and ended up with the following solution:
object MainClass {
def main(args: Array[String]): Unit = {
val conf: SparkConf = new SparkConf().setMaster("local[*]").setAppName("simple-app")
val sparkSession = SparkSession.builder()
.appName("example-spark-scala-read-and-write-from-mongo")
.config(conf)
.config("spark.mongodb.output.uri", "mongodb://sproot:12345#172.18.0.3:27017/spdb.students")
.config("spark.mongodb.input.uri", "mongodb://sproot:12345#172.18.0.3:27017/spdb.students")
.getOrCreate()
val objectMapper = new ObjectMapper()
objectMapper.registerModule(DefaultScalaModule)
// Reading Mongodb collection into a dataframe
val df = MongoSpark.load(sparkSession)
val dataRdd: RDD[Row] = df.rdd
val students: List[Student] =
dataRdd
.collect()
.map(row => Student(row.getInt(0), row.getString(1), createScoresObject(row))).toList
println()
}
def createScoresObject(row: Row): Scores = {
Scores(getAllScoresFromWrappedArray(row).map(x => Score(x.getDouble(0), x.getString(1))).toList)
}
def getAllScoresFromWrappedArray(row: Row): mutable.WrappedArray[GenericRowWithSchema] = {
getScoresWrappedArray(row).map(x => x.asInstanceOf[GenericRowWithSchema])
}
def getScoresWrappedArray(row: Row): mutable.WrappedArray[AnyVal] = {
row.getAs[mutable.WrappedArray[AnyVal]](2)
}
}
case class Student(id: Long, name: String, scores: Scores)
case class Scores(scores: List[Score])
case class Score (score: Double, scoreType: String)
But I would be glad to know if there is some elegant solution.