generic join between two dataframes spark/scala - scala

Question

I have two dataframes i.e. left & right. I have working solution for my question. I need a way to make it generic. My question is at the end here.
leftDF:
+------+---------+-------+-------+
|leftId|leftAltId|leftCur|leftAmt|
+------+---------+-------+-------+
|1 |100 |USD |20 |
|2 |200 |INR |100 |
|4 |500 |MXN |100 |
+------+---------+-------+-------+
rightDF:
+-------+----------+--------+--------+
|rightId|rightAltId|rightCur|rightAmt|
+-------+----------+--------+--------+
|1 |300 |USD |20 |
|3 |400 |MXN |100 |
|4 |600 |MXN |200 |
+-------+----------+--------+--------+
I want to perform a join between these two dataframes and I expect four dataframes as output
transactions that exists in leftDF & not in rightDF
transactions that exists in rightDF & not in leftDF
transactions that have alteast one of the ids common between two dataframes
3.a Strict Match : same currency, amount between two dataframes. Example: transaction with id 1.
3.b Relaxed Match : transactions that have same Id but different currency/amount combo. Example transaction with id 4.
Here's the desired output:
transactions that exists in leftDF & not in rightDF
+------+---------+-------+-------+-------+----------+--------+--------+
|leftId|leftAltId|leftCur|leftAmt|rightId|rightAltId|rightCur|rightAmt|
+------+---------+-------+-------+-------+----------+--------+--------+
|2 |200 |INR |100 |null |null |null |null |
+------+---------+-------+-------+-------+----------+--------+--------+
transactions that exists in rightDF & not in leftDF
+------+---------+-------+-------+-------+----------+--------+--------+
|leftId|leftAtId|leftCur|leftAmt|rightId|rightAltId|rightCur|rightAmt|
+------+---------+-------+-------+-------+----------+--------+--------+
|null |null |null |null |3 |400 |MXN |100 |
+------+---------+-------+-------+-------+----------+--------+--------+
transactions that have alteast one of the ids common between two dataframes
+------+---------+-------+-------+-------+----------+--------+--------+
|leftId|leftAltId|leftCur|leftAmt|rightId|rightAltId|rightCur|rightAmt|
+------+---------+-------+-------+-------+----------+--------+--------+
|1 |100 |USD |20 |1 |300 |USD |20 |
|4 |500 |MXN |100 |4 |600 |MXN |200 |
+------+---------+-------+-------+-------+----------+--------+--------+
3.a Strict Match : same currency, amount between two dataframes. Example: transaction with id 1.
+------+---------+-------+-------+-------+----------+--------+--------+
|leftId|leftAltId|leftCur|leftAmt|rightId|rightAltId|rightCur|rightAmt|
+------+---------+-------+-------+-------+----------+--------+--------+
|1 |100 |USD |20 |1 |300 |USD |20 |
+------+---------+-------+-------+-------+----------+--------+--------+
3.b Relaxed Match : transactions that have same Id but different currency/amount combo. Example transaction with id 4.
+------+---------+-------+-------+-------+----------+--------+--------+
|leftId|leftAltId|leftCur|leftAmt|rightId|rightAltId|rightCur|rightAmt|
+------+---------+-------+-------+-------+----------+--------+--------+
|4 |500 |MXN |100 |4 |600 |MXN |200 |
+------+---------+-------+-------+-------+----------+--------+--------+
Here's the working code I have for it:
import sparkSession.implicits._
val leftDF: DataFrame = Seq((1, 100, "USD", 20), (2, 200, "INR", 100), (4, 500, "MXN", 100)).toDF("leftId", "leftAltId", "leftCur", "leftAmt")
val rightDF: DataFrame = Seq((1, 300, "USD", 20), (3, 400, "MXN", 100), (4, 600, "MXN", 200)).toDF("rightId", "rightAltId", "rightCur", "rightAmt")
leftDF.show(false)
rightDF.show(false)
val idMatchQuery = leftDF("leftId") === rightDF("rightId") || leftDF("leftAltId") === rightDF("rightAltId")
val currencyMatchQuery = leftDF("leftCur") === rightDF("rightCur") && leftDF("leftAmt") === rightDF("rightAmt")
val leftOnlyQuery = (col("leftId").isNotNull && col("rightId").isNull) || (col("leftAltId").isNotNull && col("rightAltId").isNull)
val rightOnlyQuery = (col("rightId").isNotNull && col("leftId").isNull) || (col("rightAltId").isNotNull && col("leftAltId").isNull)
val matchQuery = (col("rightId").isNotNull && col("leftId").isNotNull) || (col("rightAltId").isNotNull && col("leftAltId").isNotNull)
val result = leftDF.join(rightDF, idMatchQuery, "fullouter")
val leftOnlyDF = result.filter(leftOnlyQuery)
val rightOnlyDF = result.filter(rightOnlyQuery)
val matchDF = result.filter(matchQuery)
val strictMatchDF = matchDF.filter(currencyMatchQuery.equalTo(true))
val relaxedMatchDF = matchDF.filter(currencyMatchQuery.equalTo(false))
leftOnlyDF.show(false)
rightOnlyDF.show(false)
matchDF.show(false)
strictMatchDF.show(false)
relaxedMatchDF.show(false)
What I'm looking for:
I want to be able to take the column names to join on, as a list and make the code generic.
for e.g.
val relaxedJoinList = Array(("leftId", "rightId"), ("leftAltId", "rightAltId"))
val strictJoinList = Array(("leftCur", "rightCur"), ("leftAmt", "rightAmt"))

Answer 1

I want to be able to take the column names to join on, as a list and make the code generic.
This is not a perfect suggestion but would definitely help you get generalized. The suggestion is to go with foldLeft
val relaxedJoinList = Array(("leftId", "rightId"), ("leftAltId", "rightAltId"))
val rHead = relaxedJoinList.head
val strictJoinList = Array(("leftCur", "rightCur"), ("leftAmt", "rightAmt"))
val sHead = strictJoinList.head
val idMatchQuery = relaxedJoinList.tail.foldLeft(leftDF(rHead._1) === rightDF(rHead._2)){(x, y) => x || leftDF(y._1) === rightDF(y._2)}
val currencyMatchQuery = strictJoinList.tail.foldLeft(leftDF(sHead._1) === rightDF(sHead._2)){(x, y) => x && leftDF(y._1) === rightDF(y._2)}
val leftOnlyQuery = relaxedJoinList.tail.foldLeft(col(rHead._1).isNotNull && col(rHead._2).isNull){(x, y) => x || col(y._1).isNotNull && col(y._2).isNull}
val rightOnlyQuery = relaxedJoinList.tail.foldLeft(col(rHead._1).isNull && col(rHead._2).isNotNull){(x, y) => x || col(y._1).isNull && col(y._2).isNotNull}
val matchQuery = relaxedJoinList.tail.foldLeft(col(rHead._1).isNotNull && col(rHead._2).isNotNull){(x, y) => x || col(y._1).isNotNull && col(y._2).isNotNull}
The rest of the code is as yours
I hope the answer is helpful