Scala Spark- Peforming multiple groupby levels using Arrays as an input - scala

Question

In my Scala program, I am dealing with a problem to combine the result from multiple levels of GroupBy.
The dataset that I am using is quite big. As a small sample, I have a dataframe that looks like this:
+---+---+----+-----+-----+
| F| L| Loy|Email|State|
+---+---+----+-----+-----+
| f1| l1|loy1| null| s1|
| f1| l1|loy1| e1| s1|
| f2| l2|loy2| e2| s2|
| f2| l2|loy2| e3| null|
| f1| l1|null| e1| s3|
+---+---+----+-----+-----+
For the first level groupBy I use the following script, to obtain the result based on the same (F, L, Loy) columns:
df.groupBy("F", "L", "Loy").agg(collect_set($"Email").alias("Email"), collect_set($"State").alias("State")).show
The result is like this:
+---+---+----+--------+-----+
| F| L| Loy| Email|State|
+---+---+----+--------+-----+
| f1| l1|null| [e1]| [s3]|
| f2| l2|loy2|[e2, e3]| [s2]|
| f1| l1|loy1| [e1]| [s1]|
+---+---+----+--------+-----+
The problem that I am dealing with is that how can I perform the second level groupBy, which is based on the condition (F, L, Email) and takes as an input F and L as a String while Email column as an Array[String]. This groupBy should return a result as followed:
+---+---+----+--------+---------+
| F| L| Loy| Email| State|
+---+---+----+--------+---------+
| f1| l1|loy1| [e1]| [s3, s1]|
| f2| l2|loy2|[e2, e3]| [s2]|
+---+---+----+--------+---------+
The main goal is to reduce the number of entries as much as possible by applying groupBy in different levels. I am quite new to Scala and any help would be appreciated :)

Answer 1

Just use concat_ws() with null separator which will remove the Array of state to simple elements and then collect_set will get you again the array to states. Check this out.
scala> val df = Seq( ("f1","l1","loy1",null,"s1"),("f1","l1","loy1","e1","s1"),("f2","l2","loy2","e2","s2"),("f2","l2","loy2","e3",null),("f1","l1",null,"e1","s3")).toDF("F","L","loy","email","state")
df: org.apache.spark.sql.DataFrame = [F: string, L: string ... 3 more fields]
scala> df.show(false)
+---+---+----+-----+-----+
|F |L |loy |email|state|
+---+---+----+-----+-----+
|f1 |l1 |loy1|null |s1 |
|f1 |l1 |loy1|e1 |s1 |
|f2 |l2 |loy2|e2 |s2 |
|f2 |l2 |loy2|e3 |null |
|f1 |l1 |null|e1 |s3 |
+---+---+----+-----+-----+
scala> val df2 = df.groupBy("F", "L", "Loy").agg(collect_set($"Email").alias("Email"), collect_set($"State").alias("State"))
df2: org.apache.spark.sql.DataFrame = [F: string, L: string ... 3 more fields]
scala> df2.show(false)
+---+---+----+--------+-----+
|F |L |Loy |Email |State|
+---+---+----+--------+-----+
|f1 |l1 |null|[e1] |[s3] |
|f2 |l2 |loy2|[e2, e3]|[s2] |
|f1 |l1 |loy1|[e1] |[s1] |
+---+---+----+--------+-----+
scala> df2.groupBy("F","L","email").agg(max('loy).as("loy"),collect_set(concat_ws("",'state)).as("state")).show
+---+---+--------+----+--------+
| F| L| email| loy| state|
+---+---+--------+----+--------+
| f2| l2|[e2, e3]|loy2| [s2]|
| f1| l1| [e1]|loy1|[s3, s1]|
+---+---+--------+----+--------+
scala>