Pivot spark dataframe array of kv pairs into individual columns - scala

Question

I have following schema:
root
|-- id: string (nullable = true)
|-- date: timestamp (nullable = true)
|-- config: struct (nullable = true)
| |-- entry: array (nullable = true)
| | |-- element: struct (containsNull = true)
| | | |-- key: string (nullable = true)
| | | |-- value: string (nullable = true)
There will not be more than 3 key-value pairs (k1,k2,k3) in the array and I would like to make value from each key into its own column while the corresponding data would come from the value from the same kv pair.
+--------+----------+----------+----------+---------+
|id |date |k1 |k2 |k3 |
+--------+----------+----------+----------+---------+
| id1 |2019-08-12|id1-v1 |id1-v2 |id1-v3 |
| id2 |2019-08-12|id2-v1 |id2-v2 |id2-v3 |
+--------+----------+----------+----------+---------+
So far I tried something like this:
sourceDF.filter($"someColumn".contains("SOME_STRING"))
.select($"id", $"date", $"config.entry" as "kvpairs")
.withColumn($"kvpairs".getItem(0).getField("key").toString(), $"kvpairs".getItem(0).getField("value"))
.withColumn($"kvpairs".getItem(1).getField("key").toString(), $"kvpairs".getItem(1).getField("value"))
.withColumn($"kvpairs".getItem(2).getField("key").toString(), $"kvpairs".getItem(2).getField("value"))
But in this case, the column names are shown as kvpairs[0][key], kvpairs[1][key] and kvpairs[2][key] as shown below:
+--------+----------+---------------+---------------+---------------+
|id |date |kvpairs[0][key]|kvpairs[1][key]|kvpairs[2][key]|
+--------+----------+---------------+---------------+---------------+
| id1 |2019-08-12| id1-v1 | id1-v2 | id1-v3 |
| id2 |2019-08-12| id2-v1 | id2-v2 | id2-v3 |
+--------+----------+---------------+---------------+---------------+
Two questions:
Is my approach right? Is there a better and easier way to pivot this
such that I get one row per array with the 3 kv pairs as 3 columns? I want to handle cases where order of the kv pairs may differ.
If the above approach is fine, how do I alias the column name to the data of the "key" element in the array?

Answer 1

Using multiple withColumn together with getItem will not work since the order of the kv pairs may differ. What you can do instead is explode the array and then use pivot as follows:
sourceDF.filter($"someColumn".contains("SOME_STRING"))
.select($"id", $"date", explode($"config.entry") as "exploded")
.select($"id", $"date", $"exploded.*")
.groupBy("id", "date")
.pivot("key")
.agg(first("value"))
The usage of first inside the aggregation here assumes there will be a single value for each key. Otherwise collect_list or collect_set can be used.
Result:
+---+----------+------+------+------+
|id |date |k1 |k2 |k2 |
+---+----------+------+------+------+
|id1|2019-08-12|id1-v1|id1-v2|id1-v3|
|id2|2019-08-12|id2-v1|id2-v2|id2-v3|
+---+----------+------+------+------+