Using a Map to rename and select columns on an Apache Spark Dataframe (Scala) [duplicate] - scala

Question

This question already has answers here:
Renaming column names of a DataFrame in Spark Scala
(6 answers)
Closed 4 years ago.
Starting with a dataframe:
val someDF = Seq(
(8, "bat", "h"),
(64, "mouse", "t"),
(-27, "horse", "x")
).toDF("number", "thing", "letter")
someDF.show()
+------+-----+------+
|number|thing|letter|
+------+-----+------+
| 8| bat| h|
| 64|mouse| t|
| -27|horse| x|
+------+-----+------+
and a Map:
val lookup = Map(
"number" -> "id",
"thing" -> "animal"
)
I'd like to select and rename the columns such that number becomes id, thing becomes animal and so on.
The renaming is covered in another Stack Overflow question: Renaming column names of a DataFrame in Spark Scala, I'm sure there is a straightforward way to do the select at the same time that I'm not seeing.
I thought something along these lines would work, but get lots of type mismatches despite the input is a string and it works with a Seq instead of map:
val renamed_selected = someDF.select(
lookup.map(m => col(m._1).as(m._2))
):_*
So the desired output is:
+------+------+
|id |animal|
+------+------+
| 8| bat |
| 64|mouse |
| -27|horse |
+------+------+
Thanks 👍🏻
Clarification on duplicate question flag: The question Renaming column names of a DataFrame in Spark Scala does not cover how to rename and select columns at the same time.

Answer 1

Here is one way; Use pattern matching to check whether the name exists in the lookup, and give the column an alias if it does otherwise use the original name:
val cols = someDF.columns.map(name => lookup.get(name) match {
case Some(newname) => col(name).as(newname)
case None => col(name)
})
someDF.select(cols: _*).show
+---+------+------+
| id|animal|letter|
+---+------+------+
| 8| bat| h|
| 64| mouse| t|
|-27| horse| x|
+---+------+------+
If you only need columns in the lookup:
val cols = someDF.columns.collect(name => lookup.get(name) match {
case Some(newname) => col(name).as(newname)
})
someDF.select(cols: _*).show
+---+------+
| id|animal|
+---+------+
| 8| bat|
| 64| mouse|
|-27| horse|
+---+------+