Why memory sink writes nothing in append mode? - scala

Question

I used Spark's structured streaming to stream messages from Kafka. The data was then aggregated, and wrote to a memory sink with append mode. However, when I tried to query the memory, it returned nothing. Below are the code:
result = model
.withColumn("timeStamp", col("startTimeStamp").cast("timestamp"))
.withWatermark("timeStamp", "5 minutes")
.groupBy(window(col("timeStamp"), "5 minutes").alias("window"))
.agg(
count("*").alias("total")
);
// writing to memory
StreamingQuery query = result.writeStream()
.outputMode(OutputMode.Append())
.queryName("datatable")
.format("memory")
.start();
// query data in memory
new Timer().scheduleAtFixedRate(new TimerTask() {
#Override
public void run() {
sparkSession.sql("SELECT * FROM datatable").show();
}
}, 10000, 10000);
The result is always:
|window|total|
+------+-----+
+------+-----+
If I used outputMode = complete, then I could get the aggregated data. But that's not my choice as the requirement is to use append mode.
Is there any problem with the code?
Thanks,

Answer 1

In append mode,
The output of a windowed aggregation is delayed the late threshold specified in withWatermark()
In your case, the delay is 5 minutes, I know nothing about your input data, but I guess you probably need to wait for 5 minutes.
I suggest you read (again?) the docs for Structured Streaming: