how to pivot a table represented as scala List of Maps by taking one column and converting it to row? - scala

Question

I have following table data represented as scala's List[Map[String, Any]]
Following is scala code for creating such list of map
val inputData = List(
Map(
"d1_name" -> "t1",
"d1_id" -> 1,
"d2_name" -> "p1",
"d2_id" -> 11,
"value1" -> 5,
"value2" -> 12,
"value3" -> 30
),
Map(
"d1_name" -> "t1",
"d1_id" -> 1,
"d2_name" -> "p2",
"d2_id" -> 22,
"value1" -> 10,
"value2" -> 14,
"value3" -> 300
),
Map(
"d1_name" -> "t2",
"d1_id" -> 2,
"d2_name" -> "p3",
"d2_id" -> 33,
"value1" -> 15,
"value2" -> 16,
"value3" -> 33
),
Map(
"d1_name" -> "t3",
"d1_id" -> 3,
"d2_name" -> "p7",
"d2_id" -> 7,
"value1" -> 5,
"value2" -> 8,
"value3" -> 17
)
)
I want to convert shown input table to following pivoted output table in scala, the operation is to pivot "d1_name" column over value3 data, also dropping "d1_id" column in final output.
(i am expecting same List[Map[String, Any]] output type)
following is pictorial representation of output.

Answer 1

Try like this:
Use GroupBy with the column d1_name and for each element in the grouped list get modify data as required.
inputData.groupBy(_.getOrElse("d1_name", "")).flatMap(e => {
e._2.map(list => {
Map(
"d2_name" -> list.getOrElse("d2_name", ""),
"d2_id" -> list.getOrElse("d2_id", ""),
"value1" -> list.getOrElse("value1", 0),
"value2" -> list.getOrElse("value2", 0),
"value3" -> list.getOrElse("value3", 0),
if (e._1 == "t1") {"t1-value3" -> list.getOrElse("value3", 0)} else {"t1-value3" -> 0},
if (e._1 == "t2") {"t2-value3" -> list.getOrElse("value3", 0)} else {"t2-value3" -> 0},
if (e._1 == "t3") {"t3-value3" -> list.getOrElse("value3", 0)} else {"t3-value3" -> 0}
)
}
)
})
// List(
Map(value3 -> 30, t3-value3 -> 0, t2-value3 -> 0, d2_name -> p1, value1 -> 5, t1-value3 -> 30, value2 -> 12, d2_id -> 11),
Map(value3 -> 300, t3-value3 -> 0, t2-value3 -> 0, d2_name -> p2, value1 -> 10, t1-value3 -> 300, value2 -> 14, d2_id -> 22),
Map(value3 -> 17, t3-value3 -> 17, t2-value3 -> 0, d2_name -> p7, value1 -> 5, t1-value3 -> 0, value2 -> 8, d2_id -> 7),
Map(value3 -> 33, t3-value3 -> 0, t2-value3 -> 33, d2_name -> p3, value1 -> 15, t1-value3 -> 0, value2 -> 16, d2_id -> 33)
)
Hope this helps you.
---EDIT----
To generate the keys dynamically
val groupedInput = inputData.groupBy(_.getOrElse("d1_name", ""))
val uniqueKeys = groupedInput.keySet
groupedInput.flatMap(e => {
e._2.map(list => {
val value3= uniqueKeys.map(key => if (e._1 == key) {key+"-value3" -> list.getOrElse("value3", 0)} else {key+"-value3" -> 0})
Map(
List("d2_name" -> list.getOrElse("d2_name", ""),
"d2_id" -> list.getOrElse("d2_id", ""),
"value1" -> list.getOrElse("value1", 0),
"value2" -> list.getOrElse("value2", 0),
"value3" -> list.getOrElse("value3", 0)).++(value3) :_*
)
}
)
})

Answer 2

This version pivots "value3" based on the value of "d1_name", and deletes "d1_id". It leaves other values untouched. In a proper implementation these would be taken out as parameters, and there would be more error checking.
val defaults = // Compute default "table" for new pivot values
inputData
.map(_("d1_name").toString + "-" + "value3" -> 0)
.toMap
val pivot = // Add new pivot values to existing "table"
inputData.map { map =>
val d1Name = map("d1_name") + "-" + "value3"
map ++ defaults + (d1Name -> map("value3")) - "d1_name" - "d1_id"
}