Casting a column in a spark DataFrame without nulls being created - scala

Question

Is there a way to cast columns in Spark and have it fail in case of a type mismatch rather than getting a null returned?
As an example I have a DF with all string columns but one of them I want to cast to date
+----------+------------+------------+
| service| eventType|process_date|
+----------+------------+------------+
| myservice| myeventtype| 2020-10-15|
| myservice| myeventtype| 2020-02-15|
|myservice2|myeventtype3| notADate |
+----------+------------+------------+
If I try to cast this with the main cast function df.withColumn("process_date", df("process_date").cast(targetType)) it will replace the bad data with a null
+----------+------------+------------+
| service| eventType|process_date|
+----------+------------+------------+
| myservice| myeventtype| 2020-10-15|
| myservice| myeventtype| 2020-02-15|
|myservice2|myeventtype3| null|
+----------+------------+------------+
Using this function in my current program could result in dangerous loss of data that I might not catch until it's too late.

Answer 1

I find two ways of doing what you want.
First, if you really want the process to fail when a Date is not parseable you can use UDF:
import java.time.LocalDate
import org.apache.spark.sql.SparkSession
import org.apache.spark.sql.functions._
import org.apache.spark.sql.types.DateType
object Data {
val tuples = List(
("myservice", "myeventtype", "2020-10-15"),
("myservice", "myeventtype", "2020-02-15"),
("myservice2", "myeventtype3", "notADate")
)
}
object BadDates {
def main(args: Array[String]) {
val spark = SparkSession.builder.master("local[2]").appName("Simple Application").getOrCreate()
import spark.implicits._
val dfBad = Data.tuples.toDF("service","eventType","process_date")
val dateConvertUdf = udf({str : String => java.sql.Date.valueOf(LocalDate.parse(str))})
dfBad
.withColumn("process_date", dateConvertUdf(col("process_date")))
.show()
}
}
This will fail with the following exception:
Exception in thread "main" org.apache.spark.SparkException: Failed to execute user defined function(BadDates$$$Lambda$1122/934288610: (string) => date)
at org.apache.spark.sql.catalyst.expressions.ScalaUDF.eval(ScalaUDF.scala:1130)
at org.apache.spark.sql.catalyst.expressions.Alias.eval(namedExpressions.scala:156)
...
Caused by: java.time.format.DateTimeParseException: Text 'notADate' could not be parsed at index 0
at java.time.format.DateTimeFormatter.parseResolved0(DateTimeFormatter.java:1949)
Alternatively you can do the conversion and check if the converted value is null but the original isn't for any line:
object BadDates2 {
def main(args: Array[String]) {
val spark = SparkSession.builder.master("local[2]").appName("Simple Application").getOrCreate()
import spark.implicits._
val dfBad = Data.tuples.toDF("service","eventType","process_date")
val df = dfBad
.withColumn("process_date_dat", col("process_date").cast(DateType))
val badLines = df
.filter(col("process_date").isNotNull && col("process_date_dat").isNull)
.count()
assert(badLines==0) //This will fail, badLines is 1
}
}