How to get this using scala - scala

Question

**DF1** **DF2** **output_DF**
120 D A 120 null A
120 E B 120 null B
125 F C 120 null C
D 120 D D
E 120 E E
F 120 null F
G 120 null G
H 120 null H
125 null A
125 null B
125 null C
125 null D
125 null E
125 F F
125 null G
125 null H
From dataframe 1 and 2 need to get the final output dataframe in spark-shell.
where A,B,C,D,E,F are in date format(yyyy-MM-dd) & 120,125 are the ticket_id's column where there are thousands of ticket_id's.
I just extracted one out of it here.

Answer 1

To get the expected result you can use df.join() and df.na.fill() (as mentioned in comments), like this:
For Spark 2.0+
val resultDF = df1.select("col1").distinct.collect.map(_.getInt(0)).map(id => df1.filter(s"col1 = $id").join(df2, df1("col2") === df2("value"), "right").na.fill(id)).reduce(_ union _)
For Spark 1.6
val resultDF = df1.select("col1").distinct.collect.map(_.getInt(0)).map(id => df1.filter(s"col1 = $id").join(df2, df1("col2") === df2("value"), "right").na.fill(id)).reduce(_ unionAll _)
It will give you the following result -
+---+----+-----+
|120|null| A|
|120|null| B|
|120|null| C|
|120| D| D|
|120| E| E|
|120|null| F|
|120|null| G|
|120|null| H|
|125|null| A|
|125|null| B|
|125|null| C|
|125|null| D|
|125|null| E|
|125| F| F|
|125|null| G|
|125|null| H|
+---+----+-----+
I hope it helps!

Answer 2

Full join of possible values, then left join with original dataframe:
import hiveContext.implicits._
val df1Data = List((120, "D"), (120, "E"), (125, "F"))
val df2Data = List("A", "B", "C", "D", "E", "F", "G", "H")
val df1 = sparkContext.parallelize(df1Data).toDF("id", "date")
val df2 = sparkContext.parallelize(df2Data).toDF("date")
// get unique ID: 120, 125
val uniqueIDDF = df1.select(col("id")).distinct()
val fullJoin = uniqueIDDF.join(df2)
val result = fullJoin.as("full").join(df1.as("df1"), col("full.id") === col("df1.id") && col("full.date") === col("df1.date"), "left_outer")
val sorted = result.select(col("full.id"), col("df1.date"), col("full.date")).sort(col("full.id"), col("full.date"))
sorted.show(false)
output:
+---+----+----+
|id |date|date|
+---+----+----+
|120|null|A |
|120|null|B |
|120|null|C |
|120|D |D |
|120|E |E |
|120|null|F |
|120|null|G |
|120|null|H |
|125|null|A |
|125|null|B |
|125|null|C |
|125|null|D |
|125|null|E |
|125|F |F |
|125|null|G |
|125|null|H |
+---+----+----+
Sorting here just for show the same result, can be skipped.