Spark Structured Streaming groupByKey on a time Window not working - scala

Question

I need to batch my Kafka stream into time windows of 10 minutes each and then run some batch processing on it.
Note: records below have a timestamp field
val records = spark.readStream
.format("kafka")
.option("kafka.bootstrap.servers", brokerPool)
.option("subscribe", topic)
.option("startingOffsets", kafkaOffset)
.load()
I add a time window to each record using,
.withColumn("window", window($"timing", windowDuration))
I created some helper classes like
case class TimingWindow(
start: java.sql.Timestamp,
end: java.sql.Timestamp
)
case class RecordWithWindow(
record: MyRecord,
groupingWindow: TimingWindow
)
Now I have a DF of type [RecordWithWindow]
All this works very well.
Next,
metricsWithWindow
.groupByKey(_.groupingWindow)
//By grouping, I get several records per time window
//resulting an object of the below type which I write out to HDFS
case class WindowWithRecords(
records: Seq[MyRecord],
window: TimingWindow
)
Where I examine HDFS,
Example:
Expected :
Each WindowWithRecords object having a unique TimingWindow
WindowWithRecordsA(TimingWindowA, Seq(MyRecordA, MyRecordB, MyRecordC))
Actual :
More than one WindowWithRecords object with the same TimingWindow
WindowWithRecordsA(TimingWindowA, Seq(MyRecordA, MyRecordB))
WindowWithRecordsB(TimingWindowA, Seq(MyRecordC))
Looks like the groupByKey logic is not working well.
I hope my question is clear. Any pointers would be helpful.

Answer 1

Found the problem:
I was not using an explicit trigger when processing the window. As a result, Spark was creating micro batches as soon as it could, as opposed to doing it at the end of the window.
streamingQuery
.writeStream
.trigger(Trigger.ProcessingTime(windowDuration))
...
.start
This was a result of me misunderstanding Spark documentation.
Note: groupByKey uses the object's hashcode. It is important to make sure that the hashcode of the object is consistent.