Making a column a map in spark scala - scala

Question

Say I have this db:
{"customerIdMarketplace": 1234, itemId: "rocks"}
{"customerIdMarketplace": 1234, itemId: "pebbles"}
{"customerIdMarketplace": 1234, itemId: "papers"}
{"customerIdMarketplace": 2345, itemId: "socks"}
{"customerIdMarketplace": 2345, itemId: "shoes"}
I want to create this dataset:
{"customerIdMarketplace": 1234, items: [{"id": "rocks"}, {"id":"pebbles"}, {"id": "papers"}]}
{"customerIdMarketplace": 2345, items: [{"id": "socks"}, {id: "shoes"}]}
The goal is to group all the items into a list if the customerIdMarketplace is the same, and represent it as a map in a sense where each item has the key id.
What I have so far is:
db.select("customerIdMarketplace", "itemId")
.groupby("customerIdMarketplace")
.agg(collect_set("itemId").as("items"))
But this doesn't make the id key. How can I do this correctly?

Answer 1

The collect_set call is fine, but you need to make a json object for each itemId, right? so you need to create a map fist. This is the code snippet on spark shell:
// customId is short for customerIdMarketplace
scala> db.select("customerId", "itemId").groupBy("customerId").agg(collect_set(to_json(map(lit("id"), col("itemId"))))).show(false)
+----------+---------------------------------------------------+
|customerId|collect_set(to_json(map(id, itemId))) |
+----------+---------------------------------------------------+
|1234 |[{"id":"papers"}, {"id":"pebbles"}, {"id":"rocks"}]|
|2345 |[{"id":"shoes"}, {"id":"socks"}] |
+----------+---------------------------------------------------+
If you don't need json, just use map:
scala> mp.select("customerId", "itemId").groupBy("customerId").agg(collect_list(map(lit("id"), col("itemId")))).show(false)
+----------+------------------------------------------------+
|customerId|collect_list(map(id, itemId)) |
+----------+------------------------------------------------+
|1234 |[[id -> rocks], [id -> pebbles], [id -> papers]]|
|2345 |[[id -> socks], [id -> shoes]] |
+----------+------------------------------------------------+
Now if you want to store them as map instead of json, you cannot use collect_set method, since comparing maps (to check for double insertion) is not supported in spark.