How to print the list of map values using scala - scala

Question

I have written the following code
val list = List(
Map("empid" -> "12", "empName" -> "Rohan", "depId" -> "201"),
Map("empid" -> "13", "empName" -> "swathi", "depId" -> "202")
).flatten.toMap
val mapRDD= sc.parallelize(Seq(list))
val columns=mapRDD.take(1).flatMap(a=>a.keys)
val columnval=mapRDD.take(2).flatMap(a=>a.keys)
val resultantDF=mapRDD.map{value=>
val list=value.values.toList
(list(0),list(1),list(2))
}.toDF(columns:_*)
resultantDF.show()
i am expecting the below output,
+-----+-------+-----+
|empid|empName|depId|
+-----+-------+-----+
| 12| Rohan| 201|
| 13|SWATHI|202 |
but i am getting only,
+-----+-------+-----+
|empid|empName|depId|
+-----+-------+-----+
| 13|SWATHI|202
Please let me know where i am doing the mistake.

Answer 1

The problem lies in your first line only,
scala> val list = List(
| Map("empid" -> "12", "empName" -> "Rohan", "depId" -> "201"),
| Map("empid" -> "13", "empName" -> "swathi", "depId" -> "202")
| ).flatten.toMap
// list: scala.collection.immutable.Map[String,String] = Map(empid -> 13, empName -> swathi, depId -> 202)
Your list actually ends up becoming a Map. And a Map can have only 1 value for each key.
Let's do the first line step by step,
So, first you created a list of maps,
scala> val listOfMaps = List(
| Map("empid" -> "12", "empName" -> "Rohan", "depId" -> "201"),
| Map("empid" -> "13", "empName" -> "swathi", "depId" -> "202")
| )
// list: List[scala.collection.immutable.Map[String,String]] = List(Map(empid -> 12, empName -> Rohan, depId -> 201), Map(empid -> 13, empName -> swathi, depId -> 202))
Then, you flattened the maps inside the listOfMaps which will result in a list of key-value pairs.
scala> val flattenedListOfMaps = listOfMaps.flatten
// flattenedListOfMaps: List[(String, String)] = List((empid,12), (empName,Rohan), (depId,201), (empid,13), (empName,swathi), (depId,202))
Now, you are converting it to a Map using toMap, which will keep on overriding the values of keys and result in a Map with unique keys,
scala> scala> val yourMap = flattenedListOfMaps.toMap
// yourMap: scala.collection.immutable.Map[String,String] = Map(empid -> 13, empName -> swathi, depId -> 202)

Answer 2

As already pointed out in the previous answer and comment, at the moment your list variable is actually a map (which is confusing at least).
What you probably want initially as input is a list.
Hence what you need is:
1.
get rid of .flatten.toMap:
val list = List(
Map("empid" -> "12", "empName" -> "Rohan", "depId" -> "201"),
Map("empid" -> "13", "empName" -> "swathi", "depId" -> "202")
)
2.
Also when calling sc.parallelize you don't need to create a separate Seq from original input (in fact, otherwise you would have a compile error).
So you also need to change it like this:
val mapRDD = sc.parallelize(list)
After making only those two changes you will receive expected result, i.e. 2 records shown in console output.