Kafka windowed stream make grace and suppress key aware - apache-kafka

Question

I currently have a simple stream of data, for example:
|-----|--------|-------|
| Key | TS(ms) | Value |
|-----|--------|-------|
| A | 1000 | 0 |
| A | 1000 | 0 |
| A | 61000 | 0 |
| A | 61000 | 0 |
| A | 121000 | 0 |
| A | 121000 | 0 |
| A | 181000 | 10 |
| A | 181000 | 10 |
| A | 241000 | 10 |
| A | 241000 | 10 |
| B | 1000 | 0 |
| B | 1000 | 0 |
| B | 61000 | 0 |
| B | 61000 | 0 |
| B | 121000 | 0 |
| B | 121000 | 0 |
| B | 181000 | 10 |
| B | 181000 | 10 |
| B | 1000 | 10 |
| B | 241000 | 10 |
| B | 241000 | 10 |
|-----|--------|-------|
this is also the order I publish the data in a topic, the value isn't really an integer but an avro value but the key is a string.
My code is this:
KStream<Windowed<String>, Long> aggregatedStream = inputStream
.groupByKey()
.windowedBy(TimeWindows.of(Duration.ofMinutes(1)).grace(Duration.ZERO))
.count()
.toStream();
aggregatedStream.print(Printed.toSysOut());
The output of the print is:
[KTABLE-TOSTREAM-0000000003]: [A#0/60000], 1
[KTABLE-TOSTREAM-0000000003]: [A#0/60000], 2
[KTABLE-TOSTREAM-0000000003]: [A#60000/120000], 1
[KTABLE-TOSTREAM-0000000003]: [A#60000/120000], 2
[KTABLE-TOSTREAM-0000000003]: [A#120000/180000], 1
[KTABLE-TOSTREAM-0000000003]: [A#120000/180000], 2
[KTABLE-TOSTREAM-0000000003]: [A#180000/240000], 1
[KTABLE-TOSTREAM-0000000003]: [A#180000/240000], 2
[KTABLE-TOSTREAM-0000000003]: [A#240000/300000], 1
[KTABLE-TOSTREAM-0000000003]: [A#240000/300000], 2
[KTABLE-TOSTREAM-0000000003]: [B#240000/300000], 1
[KTABLE-TOSTREAM-0000000003]: [B#240000/300000], 2
It seems that the grace period applies globally independently of the key of the stream, what I expect instead (if possible) is to receive all the 10 window counts of key A and the 10 window counts of key B.
In a way that the grace only closes windows based on the key of the stream.
Is that possible?

Answer 1

It seems that grace and suppress uses a global timestamp for each partition, so it's not possible to have a different one per each key.
What's possible instead is to disable the grace period and use a custom transformer instead of the regular suppress to do be able to suppress by key.
For example this is part of our code:
KStream<String, ...> aggregatedStream = pairsStream
.groupByKey()
.windowedBy(TimeWindows.of(Duration.ofMinutes(1)))
.aggregate(...your aggregation logic...)
.toStream()
.flatTransform(new TransformerSupplier<Windowed<String>, AggregateOutput, Iterable<KeyValue<String, SuppressedOutput>>>() {
#Override
public Transformer<Windowed<String>, AggregateOutput, Iterable<KeyValue<String, SuppressedOutput>>> get() {
return new Transformer<Windowed<String>, AggregateOutput, Iterable<KeyValue<String, SuppressedOutput>>>() {
KeyValueStore<String, SuppressedOutput> store;
#SuppressWarnings("unchecked")
#Override
public void init(ProcessorContext context) {
store = (KeyValueStore<String, SuppressedOutput>) context.getStateStore("suppress-store");
}
#Override
public Iterable<KeyValue<String, SuppressedOutput>> transform(Windowed<String> window, AggregateOutput sequenceList) {
String messageKey = window.key();
long windowEndTimestamp = window.window().endTime().toEpochMilli();
SuppressedOutput currentSuppressedOutput = new SuppressedOutput(windowEndTimestamp, sequenceList);
SuppressedOutput storeValue = store.get(messageKey);
if (storeValue == null) {
// First time we receive a window for that key
}
if (windowEndTimestamp > storeValue.getTimestamp()) {
// Received a new window
}
if (windowEndTimestamp < storeValue.getTimestamp()) {
// Window older than the last window we've received
}
store.put(messageKey, currentSuppressedOutput);
return new ArrayList<>();
}
#Override
public void close() {
}
};
}
}, "suppress-store")