Group by after group by spark - scala

Question

I have a dataframe with 4 columns co1, col2, col3 and col4. I need to:
Group dataframe based on key col1 and col2
Then group other columns like col3 and col4 and display counts for col3 and col4.
Input
col1 col2 col3 col4
1 1 2 4
1 1 2 4
1 1 3 5
Output
col1 col2 col_name col_value cnt
1 1 col3 2 2
1 1 col3 3 1
1 1 col4 4 2
1 1 col4 5 1
Is this possible?

Answer 1

This the case for melt like operation. You can use implementation provided by ahue as an answer to How to melt Spark DataFrame?.
val df = Seq(
(1, 1, 2, 4), (1, 1, 2, 4), (1, 1, 3, 5)
).toDF("col1", "col2", "col3", "col4")
df.melt(
Seq("col1", "col2"), Seq("col3", "col4"), "col_name", "col_value"
).groupBy("col1", "col2", "col_name", "col_value").count.show
// +----+----+--------+---------+-----+
// |col1|col2|col_name|col_value|count|
// +----+----+--------+---------+-----+
// | 1| 1| col3| 3| 1|
// | 1| 1| col4| 5| 1|
// | 1| 1| col4| 4| 2|
// | 1| 1| col3| 2| 2|
// +----+----+--------+---------+-----+

Answer 2

Here's one approach which should work for aribitrary numbers of key-columns and value-columns (Note that the sample dataset has been expanded for illustration purpose):
val df = Seq(
(1, 1, 2, 4, 6),
(1, 1, 2, 4, 7),
(1, 1, 3, 5, 7)
).toDF("col1", "col2", "col3", "col4", "col5")
import org.apache.spark.sql.functions._
val keyCols = Seq("col1", "col2")
val valCols = Seq("col3", "col4", "col5")
val dfList = valCols.map( c => {
val grpCols = keyCols :+ c
df.groupBy(grpCols.head, grpCols.tail: _*).agg(count(col(c)).as("cnt")).
select(keyCols.map(col) :+ lit(c).as("col_name") :+ col(c).as("col_value") :+ col("cnt"): _*)
} )
dfList.reduce(_ union _).show
// +----+----+--------+---------+---+
// |col1|col2|col_name|col_value|cnt|
// +----+----+--------+---------+---+
// | 1| 1| col3| 3| 1|
// | 1| 1| col3| 2| 2|
// | 1| 1| col4| 4| 2|
// | 1| 1| col4| 5| 1|
// | 1| 1| col5| 6| 1|
// | 1| 1| col5| 7| 2|
// +----+----+--------+---------+---+

Answer 3

We can use groupBy and union to achieve this.
val x = Seq((1, 1,2,4),(1, 1,2,4),(1, 1,3,5)).toDF("col1", "col2", "col3", "col4")
val y = x.groupBy("col1", "col2","col3").
agg(count(col("col3")).alias("cnt")).
withColumn("col_name", lit("col3")).
select(col("col1"), col("col2"), col("col_name"), col("col3").alias("col_value"), col("cnt"))
val z = x.groupBy("col1", "col2","col4").
agg(count(col("col4")).alias("cnt")).
withColumn("col_name", lit("col4")).
select(col("col1"), col("col2"), col("col_name"), col("col4").alias("col_value"), col("cnt"))
y.union(z).show()