Remove columns from Dataframe that all have one value (more efficient) - scala

Question

Let's say I have following dataframe:
/*
+---------+--------+----------+--------+
|a |b | c |d |
+---------+--------+----------+--------+
| bob| -1| 5| -1|
| alice| -1| -1| -1|
+---------+--------+----------+--------+
*/
I want to remove columns which only have -1 in all rows (in this case b and d). I found a solution but when I run my job I found out it was very inefficient:
private def removeEmptyColumns(df: DataFrame): DataFrame = {
val types = List("IntegerType", "DoubleType", "LongType")
val dTypes: Array[(String, String)] = df.dtypes
dTypes.foldLeft(df)((d, t) => {
val colType = t._2
val colName = t._1
if (types.contains(colType)) {
if (colType.equals("IntegerType")) {
if (d.select(colName).filter(col(colName) =!= -1).take(1).length == 0) d.drop(colName)
else d
} else if (colType.equals("DoubleType")) {
if (d.select(colName).filter(col(colName) =!= -1.0).take(1).length == 0) d.drop(colName)
else d
} else {
if (d.select(colName).filter(col(colName) =!= -1).take(1).length == 0) d.drop(colName)
else d
}
} else {
d
}
})
}
Is there a better solution or way to improve my existing code?
(I think this line val count = d.select(colName).distinct.count is the bottleneck)
I am using Spark 2.2 atm.
Many thanks

Answer 1

Instead of counting number of distinct values try to check if there exist any other value that is not -1
d.select(colName).filter(_ != -1).take(1).length == 0
Another approach
Instead of going through the dataframe n times (once for each column) you can try to collect statistics all at once
val summary = d.agg(
max(col1).as(s"${col1}_max"), min(col1).as(s"${col1}_min")),
max(col2).as(s"${col1}_max"), min(col2).as(s"${col2}_min")),
...)
.first
Then compare if min and max value for the column is the same -1