Find the facility with the longest interval without accidents using Apache Spark SQL - scala

Question

I have the next dataset:
|facility|date |accidents|
| foo |2019-01-01|1 |
| foo |2019-01-02|null |
| foo |2019-01-03|null |
| foo |2019-01-04|2 |
| bar |2019-01-01|1 |
| bar |2019-01-02|null |
| bar |2019-01-03|3 |
And the goal is to find a facility with the longest continuous period of time without accidents:
|facility|startDate |interval|
|foo |2019-01-02|2 |
Is it possible to do this using Spark SQL? Thanks
P.S. Code sample:
case class FacilityRecord(name: String, date: java.sql.Date, accidents: Option[Int])
case class IntervalWithoutAccidents(name: String, startDate: java.sql.Date, interval: Int)
implicit val spark: SparkSession = SparkSession.builder
.appName("Test")
.master("local")
.getOrCreate()
import spark.implicits._
val facilityRecords = Seq(
FacilityRecord("foo", Date.valueOf("2019-01-01"), Some(1)),
FacilityRecord("foo", Date.valueOf("2019-01-02"), None),
FacilityRecord("foo", Date.valueOf("2019-01-03"), None),
FacilityRecord("foo", Date.valueOf("2019-01-04"), Some(2)),
FacilityRecord("bar", Date.valueOf("2019-01-01"), Some(1)),
FacilityRecord("bar", Date.valueOf("2019-01-02"), None),
FacilityRecord("bar", Date.valueOf("2019-01-03"), Some(3))
)
val facilityRecordsDataset = spark.createDataset(facilityRecords)
facilityRecordsDataset.show()
val intervalWithoutAccidents: IntervalWithoutAccidents = ??? // TODO: find the interval
val expectedInterval = IntervalWithoutAccidents("foo", startDate = Date.valueOf("2019-01-02"), interval = 2)
assert(expectedInterval == intervalWithoutAccidents)
println(intervalWithoutAccidents)

Answer 1

Here's a 2-step approach:
Create column accident_date and for each facility compute interval value in every row between the current date and the next accident date using Window function first.
Compute the max interval per facility using Window function max and filter for the rows that have the max interval value.
Example code below:
import java.sql.Date
import org.apache.spark.sql.functions._
import org.apache.spark.sql.expressions.Window
import spark.implicits._
val df = Seq(
("foo", Date.valueOf("2019-01-01"), Some(1)),
("foo", Date.valueOf("2019-01-02"), None),
("foo", Date.valueOf("2019-01-03"), None),
("foo", Date.valueOf("2019-01-04"), Some(2)),
("bar", Date.valueOf("2019-01-01"), Some(1)),
("bar", Date.valueOf("2019-01-02"), None),
("bar", Date.valueOf("2019-01-03"), Some(3))
).toDF("facility", "date", "accidents")
val win = Window.partitionBy($"facility").orderBy($"date").
rowsBetween(0, Window.unboundedFollowing)
Step #1: Compute interval
val df2 = df.
withColumn("accident_date", when($"accidents".isNotNull, $"date")).
withColumn("interval",
datediff(first($"accident_date", ignoreNulls=true).over(win), $"date")
)
df2.show
// +--------+----------+---------+-------------+--------+
// |facility| date|accidents|accident_date|interval|
// +--------+----------+---------+-------------+--------+
// | bar|2019-01-01| 1| 2019-01-01| 0|
// | bar|2019-01-02| null| null| 1|
// | bar|2019-01-03| 3| 2019-01-03| 0|
// | foo|2019-01-01| 1| 2019-01-01| 0|
// | foo|2019-01-02| null| null| 2|
// | foo|2019-01-03| null| null| 1|
// | foo|2019-01-04| 2| 2019-01-04| 0|
// +--------+----------+---------+-------------+--------+
Step #2: Compute max interval
df2.select($"facility", $"date".as("start_date"),
max($"interval").over(Window.partitionBy($"facility")).as("max_interval")
).
where($"interval" === $"max_interval").
show
// +--------+----------+------------+
// |facility|start_date|max_interval|
// +--------+----------+------------+
// | bar|2019-01-02| 1|
// | foo|2019-01-02| 2|
// +--------+----------+------------+

Answer 2

Well, we can derive this just with One-step **SQL analytics function **lag()over(Window) and first_value()over(window).
Here is the code (added one extra test where 'foo' had another accident on "2019-01-05" and also no-accident on "2019-01-06")
val accidentDf = Seq(
("foo", Date.valueOf("2019-01-01"), Some(1)),
("foo", Date.valueOf("2019-01-02"), None),
("foo", Date.valueOf("2019-01-03"), None),
("foo", Date.valueOf("2019-01-04"), Some(2)),
("bar", Date.valueOf("2019-01-01"), Some(1)),
("bar", Date.valueOf("2019-01-02"), None),
("bar", Date.valueOf("2019-01-03"), Some(3)),
("foo", Date.valueOf("2019-01-05"), Some(3)),
("foo", Date.valueOf("2019-01-06"), None)
).toDF("facility", "date", "accidents")
accidentDf.createOrReplaceTempView("accident_table")
now we try to find for a particular row in a facility partition when did the last accident happened . If no accident happened then we keep the last_accident_report_date as first_value else call the date in the row as when the accident happened
Then we see for each rows what is the datediff on date and last row's last_accident_report_date
Then we select where datediff is highest.
Here is the query
val sparkSql="""select facility,date,accidents ,
lag(CASE
WHEN accidents is NULL
then first(date) over(partition by facility order by date)
else date END ,1)
over(partition by facility order by date) as last_accident_report_date ,
datediff(date,
lag(CASE WHEN accidents is NULL then first(date)
over(partition by facility order by date) else date END ,1)
over(partition by facility order by date))
as no_accident_days_rank from accident_table order by no_accident_days_rank desc, facility"""
RESULT
scala> spark.sql(sparkSql).show(20,false)
+--------+----------+---------+-------------------------+---------------------+
|facility|date |accidents|last_accident_report_date|no_accident_days_rank|
+--------+----------+---------+-------------------------+---------------------+
|foo |2019-01-04|2 |2019-01-01 |3 |
|bar |2019-01-03|3 |2019-01-01 |2 |
|foo |2019-01-03|null |2019-01-01 |2 |
|bar |2019-01-02|null |2019-01-01 |1 |
|foo |2019-01-02|null |2019-01-01 |1 |
|foo |2019-01-06|null |2019-01-05 |1 |
|foo |2019-01-05|3 |2019-01-04 |1 |
|bar |2019-01-01|1 |null |null |
|foo |2019-01-01|1 |null |null |
+--------+----------+---------+-------------------------+---------------------+