How to select the dynamic columns during a spark join? - scala

Question

I am trying to join 2 data frame, in the 1st DF i need to pass a dynamic number of column and join that with another DF. The complexity I am facing here i have a case statement with the output of 1st DF. I am able get the desired output by creating the temp view. But not able to achieve the same output through spark.
Below is the snippet, i have tried and works as expected.
// Sample DF1
val studentDF = Seq(
(1, "Peter","M",15,"Tution Received"),
(2, "Merry","F",14,null),
(3, "Sam","M",16,"Tution Received"),
(4, "Kat","O",16,null),
(5, "Keivn","M",18,null)
).toDF("Enrollment", "Name","Gender","Age","Notes")
//Sample DF2
val studentFees = Seq((1,"$500","Deposit"),(2, "$800","Deposit"),(3,"$200","Deposit"),(4,"$100","Deposit")).toDF("Enrollment","Fees","Notes")
studentDF.createOrReplaceTempView("STUDENT")
studentFees.createOrReplaceTempView("FEES")
val displayColumns = List("Enrollment","Name","Gender").map("a."+_).reduce(_+","+_)
val queryStr = spark.sql(s"select $displayColumns, case when a.Notes is null then b.Notes else a.Notes end as Notes, b.Fees from STUDENT a join FEES b on a.Enrollment=b.Enrollment")
queryStr.show()
---------+-----+------+---------------+----+
|Enrollment| Name|Gender| Notes|Fees|
+----------+-----+------+---------------+----+
| 1|Peter| M|Tution Received|$500|
| 2|Merry| F| Deposit|$800|
| 3| Sam| M|Tution Received|$200|
| 4| Kat| O| Deposit|$100|
+----------+-----+------+---------------+----+
// Below is not giving the desired output
val displayColumns = List("Enrollment","Name","Gender","Notes")
val queryStr = studentDF.select(displayColumns.head, displayColumns.tail: _*).alias("a").join(studentFees.as("b"),Seq("Enrollment"),"inner").withColumn("Notes",when($"a.Notes".isNull,$"b.Notes").otherwise($"a.Notes"))
queryStr.show()
Enrollment| Name|Gender| Notes|Fees| Notes|
+----------+-----+------+---------------+----+---------------+
| 1|Peter| M|Tution Received|$500|Tution Received|
| 2|Merry| F| Deposit|$800| Deposit|
| 3| Sam| M|Tution Received|$200|Tution Received|
| 4| Kat| O| Deposit|$100| Deposit|
+----------+-----+------+---------------+----+---------------+
// Expecting the output like below.
---------+-----+------+---------------+----+
|Enrollment| Name|Gender| Notes|Fees|
+----------+-----+------+---------------+----+
| 1|Peter| M|Tution Received|$500|
| 2|Merry| F| Deposit|$800|
| 3| Sam| M|Tution Received|$200|
| 4| Kat| O| Deposit|$100|
+----------+-----+------+---------------+----+
Is there a better way to handle such scenarios instead of crating temp table/views?

Answer 1

Thank You all whoever read my post!!
I was able to find the solution for my problem.
val displayColumns = List("Enrollment","Name","Gender","Notes")
val queryStr = studentDF.select(displayColumns.head, displayColumns.tail: _*).alias("a").join(studentFees.as("b"),Seq("Enrollment"),"inner").select($"a.*",when($"a.Notes".isNull,$"b.Notes").otherwise($"a.Notes").as("Notes"),$"b.Fees").drop($"a.Notes")