Spark UDF thread safety - scala

Question

I'm using Spark to take a dataframe containing a column of dates, and create 3 new columns containing the time in days, weeks, and months between the date in the column and today.
My concern is around the use of SimpleDateFormat, which isn't thread safe. Ordinarily without Spark this would be fine since it's a local variable, but with Spark's lazy evaluation, is sharing a single SimpleDateFormat instance over multiple UDFs likely to cause an issue?
def calcTimeDifference(...){
val sdf = new SimpleDateFormat(dateFormat)
val dayDifference = udf{(x: String) => math.abs(Days.daysBetween(new DateTime(sdf.parse(x)), presentDate).getDays)}
output = output.withColumn("days", dayDifference(myCol))
val weekDifference = udf{(x: String) => math.abs(Weeks.weeksBetween(new DateTime(sdf.parse(x)), presentDate).getWeeks)}
output = output.withColumn("weeks", weekDifference(myCol))
val monthDifference = udf{(x: String) => math.abs(Months.monthsBetween(new DateTime(sdf.parse(x)), presentDate).getMonths)}
output = output.withColumn("months", monthDifference(myCol))
}

Answer 1

I dont think its safe, as we know, SimpleDateFormat is not thread-safe.
So I prefer this method to use SimpleDateFormat in Spark if you need:
import java.text.SimpleDateFormat
import java.util.SimpleTimeZone
/**
* Thread Safe SimpleDateFormat for Spark.
*/
object ThreadSafeFormat extends ThreadLocal[SimpleDateFormat] {
override def initialValue(): SimpleDateFormat = {
val dateFormat = new SimpleDateFormat("yyyy-MM-dd:H")
// if you need get UTC time, you can set UTC timezone
val utcTimeZone = new SimpleTimeZone(SimpleTimeZone.UTC_TIME, "UTC")
dateFormat.setTimeZone(utcTimeZone)
dateFormat
}
}
Then use ThreadSafeFormat.get() to get Thread-safe SimpleDateFormat to do anything.