manipulating data table joining in scala data frame - scala

Question

Given two data frames in scala, T1, T2, I wanna only keep values in T1 where IDs exist in T2, but values don't exist in T2. For instance
T1
+---+---+
| ID|val|
+---+---+
| 1| x|
| 1| y|
| 1| z|
| 2| x|
| 2| y|
| 3| x|
| 3| y|
| 3| z|
| 3| k|
| 4| x|
| 4| y|
| 4| z|
| 5| x|
+---+---+
T2
+---+---+
| ID|val|
+---+---+
| 1| x|
| 1| y|
| 2| x|
| 3| x|
| 3| y|
| 4| x|
| 4| y|
| 4| z|
| 5| x|
+---+---+
For ID=1, values x, y exist in T2, and for ID=2, values x exists in T2, and for ID=3, values x, y exist in T2, and for ID=4, values x, y, z exist in T2, and for ID=5, value x exists in T3, so excluding those values, we get
+---+---+
| ID|val|
+---+---+
| 1| z|
| 2| y|
| 3| z|
| 3| k|
+---+---+
as answer.
I'm trying something like
T1.join(T2, Seq("ID", "val")).filter(T1.col("ID")===T2.col("ID") && T1.col("val")===T2.col("val"))
but not sure if this is correct/efficient approach or not.
Any help would be greatly appreciated. I'm kinda new to scala.

Answer 1

You have several ways to achive this operation.
This operation is the difference between df1 and df2, you could try the following approach with dataframe API:
val result2 = df1.except(df2)
result2.show()
+---+---+
| ID|val|
+---+---+
| 1| z|
| 3| z|
| 2| y|
| 3| k|
+---+---+
Another approach with the Dataframe API would be with anti-join
val joinCondition = col("ID") === col("ID1") and col("val") === col("val1")
val result3 = df1.join(df2, joinCondition, "left_anti")
result3.show()
+---+---+
| ID|val|
+---+---+
| 1| z|
| 3| z|
| 2| y|
| 3| k|
+---+---+
And finally this approach would be transform dataframes to a set to calculate the difference between the two sets
val setting = (df1.collect().toSet -- df2.collect().toSet).toList
val result = sc
.parallelize(setting.map(r => (r(0).toString.toInt,r(1).toString)))
.toDF("ID","val")
result.show()
+---+---+
| ID|val|
+---+---+
| 2| y|
| 3| z|
| 3| k|
| 1| z|
+---+---+