Smaller or equal comparison syntax error - scala

Question

My UDF is comparing if time difference between two columns is within 5 days limit. If == operator is used, expression compiles properly, but <= (or lt) fails with type mismatch error. Code:
val isExpiration : (Column, Column, Column) =>
Column = (BCED, termEnd, agrEnd) =>
{
if(abs(datediff(if(termEnd == null) {agrEnd} else {termEnd}, BCED)) lt 6)
{lit(0)}
else
{lit(1)}
}
Error:
notebook:3: error: type mismatch;
found : org.apache.spark.sql.Column
required: Boolean
if(abs(datediff(if(termEnd == null) {agrEnd} else {termEnd}, BCED)) lt 6) {lit(0)}...
^
I must be missing something obvious - can anyone advice how to test if Column value is smaller or equal to a constant?

Answer 1

It looks like you have mixed udf and Spark functions, you need to use only one of them. When possible it's always preferable not to use and udf since those can not be optimized (and are thus generally slower). Without udf it could be done as follows:
df.withColumn("end", when($"termEnd".isNull, $"agrEnd").otherwise($"termEnd"))
.withColumn("expired", when(abs(datediff($"end", $"BCED")) lt 6, 0).otherwise(1))
I introduced a temporary column to make the code a bit more readable.
Using an udf it could, for example, be done as follows:
val isExpired = udf((a: Date, b: Date) => {
if ((math.abs(a.getTime() - b.getTime()) / (1000 * 3600 * 24)) < 6) {
0
} else {
1
}
})
df.withColumn("end", when($"termEnd".isNull, $"agrEnd").otherwise($"termEnd"))
.withColumn("expired", isExpired($"end", $"BCED"))
Here, I again made use of a temporary column but this logic could be moved into the udf if preferred.

Answer 2

That's because abs(col).lt(6) returns an object of type Column and if expects the condition to be evaluated to true or false which is a Scala Boolean type.
Plus, UDF doesn't work on Column Data Type, it works with Scala Objects (Int, String, Boolean etc)
Since all you're doing is using Spark SQL functions, you can rewrite your UDF like this:
val isExpiration = (
when(
abs(datediff(coalesce($"termEnd", $"agrEnd") , $"BCED")) <= 6, lit(0)
).otherwise(lit(1))
)
And, the usage would be:
df.show
//+----------+----------+----------+
//| BCED| termEnd| agrEnd|
//+----------+----------+----------+
//|2018-06-10|2018-06-25|2018-06-25|
//|2018-06-10| null|2018-06-15|
//+----------+----------+----------+
df.withColumn("x", isExpiration).show
//+----------+----------+----------+---+
//| BCED| termEnd| agrEnd| x|
//+----------+----------+----------+---+
//|2018-06-10|2018-06-25|2018-06-25| 1|
//|2018-06-10| null|2018-06-15| 0|
//+----------+----------+----------+---+