Create JOIN condition on variable number of columns in Scala - scala

Question

Suppose I have two data frames and I would like to join them based on certain columns. The list of these join columns can differ based on the data frames that I'm joining, but we can always count on the fact that the two data frames we will join using joinDfs will always have the same column names as joinCols.
I am trying to figure out how to form the joinCondition given the assumptions/requirements above. Currently, it is returning (((a.colName1 = b.colName1) AND (a.colName2 = b.colName2)) AND (a.colName3 = b.colName3)), which is not quite returning what I'm expecting from the INNER JOIN in the example below.
Thank you in advance for helping me, a newbie to Scala and Spark figure out how to form a proper joinCondition.
import org.apache.spark.sql.functions._
import org.apache.spark.sql.{Column, DataFrame, Row}
import org.apache.spark.sql.types._
def joinDfs(firstDf: DataFrame, secondDf: DataFrame, joinCols: Array[String]): DataFrame = {
val firstDfAlias: String = "a"
val secondDfAlias: String = "b"
// This is what I am trying to figure out and need help with
val joinCondition = joinCols
.map(c => col(f"${firstDfAlias}.${c}") === col(f"${secondDfAlias}.${c}"))
.reduce((x,y) => (x && y))
secondDf.as(secondDfAlias).join(
firstDfAlias.as(firstDfAlias),
joinCondition,
"inner"
).select(cols.map(col): _*)
}
// This is an example of data frames that I'm trying to join
// But these data frames can change in terms of number of columns in each
// and data types, etc. The only thing we know for sure is that these
// data frames will contain some or all columns with the same name and
// we will use them to join the two data frames.
val firstDfSchema = StructType(List(
StructField(name = "colName1", dataType = LongType, nullable=true),
StructField(name = "colName2", dataType = StringType, nullable=true),
StructField(name = "colName3", dataType = LongType, nullable=true),
StructField(name = "market_id", dataType = LongType, nullable=true)
))
val firstDfData = Seq(
Row(123L, "123", 123L, 123L),
Row(234L, "234", 234L, 234L),
Row(345L, "345", 345L, 345L),
Row(456L, "456", 456L, 456L)
)
val firstDf = spark.createDataFrame(spark.sparkContext.parallelize(firstDfData), firstDfSchema)
val secondDfSchema = StructType(List(
StructField(name = "colName1", dataType = LongType, nullable=true),
StructField(name = "colName2", dataType = StringType, nullable = true),
StructField(name = "colName3", dataType = LongType, nullable = true),
StructField(name = "num_orders", dataType = LongType, nullable=true)
))
val secondDfData = Seq(
Row(123L, "123", 123L, 1L),
Row(234L, "234", 234L, 2L),
Row(567L, "567", 567L, 3L)
)
val secondDf = spark.createDataFrame(spark.sparkContext.parallelize(secondDfData), secondDfSchema)
// Suppose we are going to join the two data frames above on the following columns
val joinCols: Array[String] = Array("colName1", "colName2", "colName3")
val finalDf = joinDfs(firstDf, secondDf, joinCols)