Date diff of ISO 8601 timestemp strings in Spark - scala

Question

I have two datetime strings in ISO 8601 format:
2017-05-30T09:15:06.050298Z
2017-05-30T09:15:06.054939Z
I want the time difference between above two strings using Scala in Spark environment.

Answer 1

As you said in the comments you're using Joda-Time, here's an answer using it.
You said that you're calling daysBetween. But both dates are in the same day, so the result will always be zero. To get the difference between the dates with millisecond precision, just subtract the millis value from both DateTime objects:
import org.joda.time.DateTime
val s1 = "2017-05-30T09:15:06.050298Z"
val s2 = "2017-05-30T09:15:06.054939Z"
val diffInMillis = DateTime.parse(s2).getMillis() - DateTime.parse(s1).getMillis()
The diffInMillis will be 4 - the first date's fraction-of-second is 050298 and the second's is 054939, but joda's DateTime has milliseconds precision, so the last 3 digits are discarded. You can check that by doing:
println(DateTime.parse(s1))
println(DateTime.parse(s2))
This will output:
2017-05-30T09:15:06.050Z
2017-05-30T09:15:06.054Z
As you can see, the difference between the dates is 4 milliseconds.
New Java Date/Time API
Joda-Time is in maintainance mode and is being replaced by the new APIs, so I don't recommend start a new project with it. Even in joda's website it says: "Note that Joda-Time is considered to be a largely “finished” project. No major enhancements are planned. If using Java SE 8, please migrate to java.time (JSR-310)."
If you have the new java.time API available (JDK >= 1.8), you can also use it. If java.time classes are not available (JDK <= 1.7), you can try the scala time, which is based on the ThreeTen Backport, a great backport for Java 8's new date/time classes.
The code below works for both.
The only difference is the package names (in Java 8 is java.time and in ThreeTen Backport (or Android's ThreeTenABP) is org.threeten.bp), but the classes and methods names are the same.
The difference is that this API has nanoseconds precision, so you can get the difference between the dates in nanoseconds.
import java.time.Instant
import java.time.temporal.ChronoUnit
val s1 = "2017-05-30T09:15:06.050298Z"
val s2 = "2017-05-30T09:15:06.054939Z"
// difference in nanoseconds
val diffInNanos = ChronoUnit.NANOS.between(Instant.parse(s1), Instant.parse(s2))
The value of diffInNanos is 4641000. If you still want this value in milliseconds, you can divide it by 1000000, or use ChronoUnit.MILLIS instead of ChronoUnit.NANOS.

Answer 2

Nanoseconds with LocalDateTime of Java 8
As Spark does not support date diff above Seconds, we need to create a UDF for Millis or Nanos.
Date time related imports
import java.time.LocalDateTime;
import java.time.format.DateTimeFormatter;
import java.time.temporal.ChronoField;
Create UDF to date diff by nanoseconds
spark.udf.register("date_diff_nano", (d1: String, d2: String) =>
val dtFormatter = DateTimeFormatter.ofPattern("yyyy-MM-dd'T'HH:mm:ss.n'Z'")
val dt1 = LocalDateTime.parse(d1, dtFormatter)
val dt2 = LocalDateTime.parse(d2, dtFormatter)
(dt1.getLong(ChronoField.NANO_OF_DAY) - dt2.getLong(ChronoField.NANO_OF_DAY))
)
Check: help in building DateTimeFormatter pattern
By modifying ChronoField.NANO_OF_DAY to ChronoField.MICRO_OF_DAY
in UDF last line, we can get the date diff in micro sec also.
Now, use the UDF on any DataFrame/DataSet object.
import spark.implicits._ //to use $-notation on columns
// create the dataframe df
val df = ...
val resultDf = df.withColumn("date_diff", date_diff_nano($"dt1", $"dt2"))
Here dt1 and dt2 are DateTime columns in df
Seconds diff with unix_timestamp of Spark SQL
Use Spark SQL predefined unix_timestamp(date, format) function to convert a date to seconds of the day (But Java SimpleDateFormat can support parsing up to Milliseconds), then you can do Date diff with Spark SQL using unix_timestamp.
import org.apache.spark.sql.functions.unix_timestamp
val resultDf = df.withColumn("date_diff_sec",
(unix_timestamp($"dt1"), unix_timestamp($"dt2")))
Days diff b/w two dates using datediff
It accepts the datetime value of following formats
java.sql.Timestamp
java.sql.Date
String format of 'YYYY-MM-dd'
String format of 'YYYY-MM-dd HH:mm:ss'
import org.apache.spark.sql.functions.datediff
val resultDf = df.withColumn("date_diff_days", datediff($"dt1", $"dt2"))

Answer 3

You can use the xml date parser, since it must adhere to ISO-8601:
val t1 = javax.xml.bind.DatatypeConverter.parseDateTime("2017-05-30T09:15:06.050298Z")
val t2 = javax.xml.bind.DatatypeConverter.parseDateTime("2017-05-30T09:15:06.054939Z")
val diff = t1.getTimeInMillis - t2.getTimeInMillis