How to optimize dataset aggregation on top of java objects dataset - scala

Question

could you please support me on stupid question:
I have some Java class:
public class ProbePoint implements Serializable, Cloneable {
private long arrivalTimeMillis = 0;
private long captureTimeMillis = 0;
//...
}
public class Trip implements Serializable, Cloneable {
private ArrayList<ProbePoint> points = new ArrayList<>();
//...
}
I have Dataset[Trip]. I need to collect some min/max values. What would be better implementation of next:
public class DataRanges implements Serializable {
private long minCaptureTs;
private long maxArrivalTs;
}
val timesDs: Dataset[DataRanges] = trips.mapPartitions(t => {
var minCaptTime = Long.MaxValue
var maxArrTime = Long.MinValue
t.foreach(f => {
if (f.points.head < minCaptTime) minCaptTime = f.points.head
if (f.points.last.getArrivalTimeMillis > maxArrTime) maxArrTime = f.points.last.getArrivalTimeMillis
})
Iterator[DataRanges](
new DataRanges(minStartTime, maxEndTime, minArrTime, maxArrTime))
})(Encoders.bean(classOf[DataRanges]))
val times = timesDs.agg(min("minCaptureTs"), max("maxArrivalTs")).head()
}
}

Answer 1

Looking at the Java classes, the schema of Dataset[Trip] should be
root
|-- points: array (nullable = true)
| |-- element: struct (containsNull = true)
| | |-- arrivalTimeMillis: long (nullable = false)
| | |-- captureTimeMillis: long (nullable = false)
It would be possible to explode the array and then take the min and max of the resulting columns, thus simplyfing the code a bit:
val df = tripsDF
.withColumn("exploded", explode($"points"))
.withColumn("arrivalTimeMillis", $"exploded.arrivalTimeMillis")
.withColumn("captureTimeMillis", $"exploded.captureTimeMillis")
val Row(minArrivaltime: Long, maxCaptureTimeMillis: Long) =
df.agg(min("arrivalTimeMillis"), max("captureTimeMillis")).head
println(minArrivaltime)
println(maxCaptureTimeMillis)
The code in the question assumes that the arrays in the Trip class are sorted: the minimal capture time is always taken from the first element of the array and the maximum arrival time is always taken from the last one. This code takes the minimum and maximum over all ProbePoints, so the logic is slightly different.