Dynamically selecting columns with parameter using Scala - Spark - scala

Question

I need to dynamically select columns from one of the two tables I am joining. The name of one of the columns to be selected is passed to a variable. Here are the details.
The table names are passed to variables. So is the join_id and the join_type.
//Creating scala variables for each table
var table_name_a = dbutils.widgets.get("table_name_a")
var table_name_b = dbutils.widgets.get("table_name_b")
//Create scala variable for Join Id
var join_id = dbutils.widgets.get("table_name_b") + "Id"
// Define join type
var join_type = dbutils.widgets.get("join_type")
Then, I join the tables. I want to select all columns from table A and only two columns from table B: one column is called "Description" no matter what table B is passed in the parameter above; the second column has the same name of the table B, e.g., if table B's name is Employee, I want to select a column named "Employee" from table B. The code below selects all columns from table A and the Description column from table B (aliased). But I still need to select another column from table B that has the same name as the table. I don't know in advance how many columns table B has in total nor column order or their names - since Table B is passed as a parameter.
// Joining Tables
var df_joined_tables = df_a
.join(df_b,
df_a(join_id)===df_b(join_id),
join_type
).select($"df_a.*",$"df_b.Description".alias(table_name_b + " Description"))
My question is: How do I pass the variable table_name_b as a column I am trying to select from table B?
I tried the code below which is obviously wrong because in "$"df_b.table_name_b" the "table_name_b" is supposed to be the content of the parameter and not the name of the columns itself.
var df_joined_tables = df_a
.join(df_b,
df_a(join_id)===df_b(join_id),
join_type
).select($"df_a.*",$"df_b.Description".alias(table_name_b + " Description"),$"df_b.table_name_b")
Then I tried the code below and it gives the error: "value table_name_b is not a member of org.apache.spark.sql.DataFrame"
var df_joined_tables = df_a
.join(df_b,
df_a(join_id)===df_b(join_id),
join_type
).select($"df_a.*",$"df_b.Description".alias(table_name_b + " Description"),df_b.table_name_b)
How do I pass the variable table_name_b as a column I need to select from table B?

Answer 1

You can build a List[org.apache.spark.sql.Column] and use it on your select function like the below example:
// sample input:
val df = Seq(
("A", 1, 6, 7),
("B", 2, 7, 6),
("C", 3, 8, 5),
("D", 4, 9, 4),
("E", 5, 8, 3)
).toDF("name", "col1", "col2", "col3")
df.printSchema()
val columnNames = List("col1", "col2") // string column names from your params
val columnsToSelect = columnNames.map(col(_)) // convert the required column names from string to column type
df.select(columnsToSelect: _*).show() // using the list of columns
// output:
+----+----+
|col1|col2|
+----+----+
| 1| 6|
| 2| 7|
| 3| 8|
| 4| 9|
| 5| 8|
+----+----+
Similarly can be applied for join's
Update
Adding another example:
val aliasTableA = "tableA"
val aliasTableB = "tableB"
val joinField = "name"
val df1 = Seq(
("A", 1, 6, 7),
("B", 2, 7, 6),
("C", 3, 8, 5),
("D", 4, 9, 4),
("E", 5, 8, 3)
).toDF("name", "col1", "col2", "col3")
val df2 = Seq(
("A", 11, 61, 71),
("B", 21, 71, 61),
("C", 31, 81, 51)
).toDF("name", "col_1", "col_2", "col_3")
df1.alias(aliasTableA)
.join(df2.alias(aliasTableB), Seq(joinField))
.selectExpr(s"${aliasTableA}.*", s"${aliasTableB}.col_1", s"${aliasTableB}.col_2").show()
// output:
+----+----+----+----+-----+-----+
|name|col1|col2|col3|col_1|col_2|
+----+----+----+----+-----+-----+
| A| 1| 6| 7| 11| 61|
| B| 2| 7| 6| 21| 71|
| C| 3| 8| 5| 31| 81|
+----+----+----+----+-----+-----+