Data processing for time series data-spark - scala

Question

Given the below sample data,
t- timeseries datetime sample,
lat-latitude,
long-longitude
t lat long
0 27 28
5 27 28
10 27 28
15 29 49
20 29 49
25 27 28
30 27 28
I want to get the output similar to this, I want to process the time series data in such a way that grouping the pair of lat long I am able to get the distinct time series interval for the pair.
I am doing the processing in spark
Lat-long interval
(27,28) (0,10)
(29,49) (15,20)
(27,28) (25,30)

Answer 1

I wouldn't have suggested you this solution if your data were huge but since you commented
I am processing the day wise data which is stored in cassandara ,size of 5-6k rows of records/second
following solution proposal should be fine
Looking at your given dataframe, schema should be as
root
|-- t: integer (nullable = false)
|-- lat: integer (nullable = false)
|-- long: integer (nullable = false)
And your expected output suggests that you would need an additional column for grouping the dataframe which would require you to collect the data on one executor
val collectedRDD = df.collect()
var varianceCount, lattitude, longitude = 0
val groupedData = new ArrayBuffer[(Int, Int, Int, Int)]()
for(rdd <- collectedRDD) {
val t = rdd.getAs[Int]("t")
val lat = rdd.getAs[Int]("lat")
val long = rdd.getAs[Int]("long")
if (lat != lattitude || long != longitude) {
varianceCount = varianceCount + 1
lattitude = lat
longitude = long
groupedData.append((t, lat, long, varianceCount))
}
else {
groupedData.append((t, lat, long, varianceCount))
}
}
Then you convert the ArrayBuffer to dataframe and use groupBy and aggregation as
val finalDF = groupedData
.toDF("t", "lat", "long", "grouped")
.groupBy(struct("lat", "long").as("lat-long"), col("grouped"))
.agg(struct(min("t"), max("t")).as("interval"))
.drop("grouped")
finalDF should be
+--------+--------+
|lat-long|interval|
+--------+--------+
|[29,49] |[15,20] |
|[27,28] |[0,10] |
|[27,28] |[25,30] |
+--------+--------+
I hope the answer is helpful