Dynamic GroupBy and count using Spark Dataframes/Datasets - scala

Question

The use case is to group by each column in a given dataset, and get the count of that column.
The resulting set is (key, value) map and then finally uinion of them all.
For eg
students = {(age, firstname, lastname)(12, "FN", "LN"), (13, "df", "gh")}
groupby age => (12, 1), (13, 1)
groupby firstname => etc
I know the brute force approach is to do a map and maintain a map for count for each column
but i wanted to see if there is something more we can do with maybe foldLeft and windows function. I tried using rollup and cube but that does groups all column together rather than indivdual

Answer 1

Assuming that you need Key, Value, Grouping Column name as three columns in the output, you would have to use the below code so that key and grouping column relationships can be understood.
Code
val df = Seq(("12", "FN", "LN"),
("13", "FN", "gh")).toDF("age", "firstname", "lastname")
df.show(false)
val initialDF = spark.createDataFrame(spark.sparkContext.emptyRDD[Row], StructType(
Seq(StructField("Key", StringType), StructField("Value", IntegerType),
StructField("GroupColumn", StringType))
))
val resultantDf = df.columns.foldLeft(initialDF)((df1, column) => df1.union(
df.groupBy(column).count().withColumn("GroupColumn", lit(column))
))
resultantDf.show(false)
resultantDf.collect().map { row =>
(row.getString(0), row.getLong(1))
}.foreach(println)
Output
INPUT DF:
+---+---------+--------+
|age|firstname|lastname|
+---+---------+--------+
|12 |FN |LN |
|13 |FN |gh |
+---+---------+--------+
OUTPUT DF:
+---+-----+-----------+
|Key|Value|GroupColumn|
+---+-----+-----------+
|12 |1 |age |
|13 |1 |age |
|FN |2 |firstname |
|gh |1 |lastname |
|LN |1 |lastname |
+---+-----+-----------+
OUTPUT LIST:
(12,1)
(13,1)
(FN,2)
(gh,1)
(LN,1)

Answer 2

Assuming that you need Union of the grouped data frames, I was able to solve it as below:
Code
val df = Seq(("12", "FN", "LN"),
("13", "FN", "gh")).toDF("age", "firstname", "lastname")
df.show(false)
val initialDF = spark.createDataFrame(spark.sparkContext.emptyRDD[Row], StructType(
Seq(StructField("column", StringType), StructField("count", IntegerType))
))
df.columns.foldLeft(initialDF)((df1, column) => df1.union(df.groupBy(column).count())).show(false)
Output
INPUT DF:
+---+---------+--------+
|age|firstname|lastname|
+---+---------+--------+
|12 |FN |LN |
|13 |FN |gh |
+---+---------+--------+
OUTPUT DF:
+------+-----+
|column|count|
+------+-----+
|12 |1 |
|13 |1 |
|FN |2 |
|gh |1 |
|LN |1 |
+------+-----+