Kafka Streams - Filter messages that appear frequently in a time window - apache-kafka

Question

I am trying to filter for any messages whose key appears more often than a threshold N in a given (hopping) time window of length T.
For example, in the following stream:
#time, key
0, A
1, B
2, A
3, C
4, D
5, A
6, B
7, C
8, C
9, D
10, A
11, D
12, D
13, D
14, D
15, D
and N=2 and T=3, the outcome should be
0, A
2, A
7, C
8, C
9, D
11, D
12, D
13, D
14, D
15, D
Alternatively, if the above is not possible, a simplification would be only to filter for the messages after the threshold has been met:
#time, key
2, A
8, C
11, D
12, D
13, D
14, D
15, D
Is this possible with Kafka Streams?
So far I have tried to create a windowed count (instance of KTable) of the stream and join it back to the original stream. I change the key of the windowed count back to the original key using KTable#toStream((k,v) -> k.key()) and performing a dummy aggregation back to an instance of KTable. This seems to introduce a delay which causes the leftJoin to miss messages which come very close after the threshold is exceeded.
final Serde<String> stringSerde = Serdes.String();
final Serde<Long> longSerde = Serdes.Long();
KStream<String, Long> wcount = source.groupByKey()
.count(TimeWindows.of(TimeUnit.SECONDS.toMillis(5)),"Counts")
.toStream((k,v) -> k.key());
// perform dummy aggregation to get KTable
KTable<String, Long> wcountTable = wcount.groupByKey(stringSerde, longSerde)
.reduce((aggValue, newValue) -> newValue,
"dummy-aggregation-store");
// left join and filter with threshold N=1
source.leftJoin(wcountTable, (leftValue, rightValue) -> rightValue,stringSerde, stringSerde )
.filter((k,v) -> v!=null)
.filter((k,v) -> v>1)
.print("output");
I have also tried to perform a KStream-KStream join with an appropriate window (leaving out the dummy aggregation):
source.join(wcount, (leftValue, rightValue) -> rightValue, JoinWindows.of(TimeUnit.SECONDS.toMillis(5)),stringSerde, stringSerde, longSerde)
.filter((k,v) -> v!=null)
.filter((k,v) -> v>1)
.print("output");
This results in duplicate outputs since each UPSERT into wcount triggers an event.

Answer 1

This is certainly possible. You can apply a windowed aggregation that collect all raw data in a list (ie, you manually materialize the window). Afterwards, you apply a flatMap that evaluates the window. If the threshold is not met yet, you emit nothing. If the threshold is met for the first time, you emit all buffered data. For all further calls of flatMap with a larger count than the threshold, you only emit the latest one in the list (you know that you did emit all others an the call to flatMap before, ie, emit only the newly added one).
Note: you need to disable KTable cache, ie, set config parameter "cache.max.bytes.buffering" = 0. Otherwise, the algorithms won't work correctly.
Something like this:
KStream<Windowed<K>, List<V>> windows = stream.groupByKey()
.aggregate(
/*init with empty list*/,
/*add value to list in agg*/,
TimeWindows.of()...),
...)
.toStream();
KStream<K,V> thresholdMetStream = windows.flatMap(
/* if List#size < threshold
then return empty-list, ie, nothing
elseif List#size == threshold
then return whole list
else [List#size > threshold]
then return last element from list
*/);

Answer 2

AFAIK this is the perfect fit for the Count-Min-Sketch algorithm. See for example the stream-lib implementation:
https://github.com/addthis/stream-lib