Doing left outer join on multiple data frames in spark scala - scala

Question

I am newbie in Spark. I trying to achieve below use case using scala.
-DataFrame 1
| col A | col B |
-----------------
| 1 | a |
| 2 | a |
| 3 | a |
-DataFrame 2
| col A | col B |
-----------------
| 1 | b |
| 3 | b |
-DataFrame 3
| col A | col B |
-----------------
| 2 | c |
| 3 | c |
Final Output frame should be
| col A | col B |
-----------------
| 1 | a,b |
| 2 | a,c |
| 3 | a,b,c |
Number of frames are not limited to 3 , it can be any number less than 100.So I am using for each in which I am printing each of the data frame.
Can some one please help me how I can create final data frame in which I can have output in above format with N data frames.
I appreciate your help.

Answer 1

I see this question today. I suggest that you use python to solve it. It's easier to write than scala. Here are they:
from pyspark.sql import SQLContext
from pyspark.sql.functions import concat_ws
d1=sc.parallelize([(1, "a"), (2, "a"), (3,"a")]).toDF().toDF("Col_A","Col_B")
d2=sc.parallelize([(1, "b"), (2, "b")]).toDF().toDF("Col_A", "Col_B")
d3=sc.parallelize([(2, "c"), (3, "c")]).toDF().toDF("Col_A", "Col_B")
d4=d1.join(d2,'Col_A','left').join(d3,'Col_A','left').select(d1.Col_A.alias("col A"),concat_ws(',',d1.Col_B,d2.Col_B,d3.Col_B).alias("col B"))
df4.show()
+-----+-----+
|col A|col B|
+-----+-----+
| 1
| a,b|
| 2
|a,b,c|
| 3
| a,c|
+-----+-----+
You see the result!

Answer 2

You can use foldLeft to iteratively merge data with outer join
import org.apache.spark.sql.Row
import org.apache.spark.sql.functions._
val df1 = Seq((1, "a"), (2, "a"), (3, "a")).toDF("Col A", "Col B")
val df2 = Seq((1, "b"), (2, "b")).toDF("Col A", "Col B")
val df3 = Seq((2, "c"), (3, "c")).toDF("Col A", "Col B")
val dfs = Seq(df2, df3)
val bs = (0 to dfs.size).map(i => s"Col B $i")
dfs.foldLeft(df1)(
(acc, df) => acc.join(df, Seq("Col A"), "fullouter")
).toDF("Col A" +: bs: _*).select($"Col A", array(bs map col: _*)).map {
case Row(a: Int, bs: Seq[_]) =>
// Drop nulls and concat
(a, bs.filter(_ != null).map(_.toString).mkString(","))
}.toDF("Col A", "Col B").show
// +-----+-----+
// |Col A|Col B|
// +-----+-----+
// | 1| a,b|
// | 3| a,c|
// | 2|a,b,c|
// +-----+-----+
but if you really think
it can be any number less than 100
then it is just unrealistic. join is the most expensive operation in Spark, and even with all the optimizer improvements, it is just not gonna work.