Scala RDD groupbykey without using groupbykey function - scala

Question

I am trying to get an RDD[(String, Iterable[String])] without using groupbykey. These are my tuples:
(Group 1, John)
(Group 2, Sam)
(Group 1, Mary)
(Group 3, Pam)
I need to get:
(Group 1, List(John, Mary)), (Group 2, List(Sam)), (Group 3, List(Pam))
without using groupby or groupbykeys function. How can I do this?

Answer 1

So if you want to use Spark APIs, you can use a windowing function over key, and try to collect values into a list:
import org.apache.spark.sql.functions._
import org.apache.spark.sql.expressions.Window
val df = Seq(
(Group 1, John),
(Group 2, Sam),
(Group 1, Mary),
(Group 3, Pam)
).toDF("key", "value")
val keyWindow = Window.partitionBy("key")
val result = df
.withColumn(
"values",
collect_list(col("value")).over(keyWindow)
)
.select("key", "values")
.distinct
result.show()
// here is the result:
+-------+------------+
| key| values|
+-------+------------+
|Group 1|[John, Mary]|
|Group 2| [Sam]|
|Group 3| [Pam]|
+-------+------------+
You can then convert this df to rdd easily!
Update:
So if you want to use plain scala, assuming the data is given as follows:
val data = Seq(
("Group 1", "John"),
("Group 2", "Sam"),
("Group 1", "Mary"),
("Group 3", "Pam")
)
You need to foldLeft over the data, with a state of Map[String, List[String]] (which basically means a mapping from each key/group to values), and update the state of your iteration (the Map thing)
val result = data.foldLeft(Map.empty[String, List[String]]) {
case (acc, (key, value)) =>
acc.updatedWith(key) {
case Some(values) => Some(value :: values)
case None => Some(value :: Nil)
}
}
Just to make things clearer, you can use .collect on a dataframe to collect the rows as an Array. And, to make an RDD from the resulting Map[String, List[String]], you can use spark.sparkContext.parallelize(result.toSeq).