What's a viable solution to deferred processing of the events? - apache-kafka

Question

Given the system, which is consuming the event stream from Kafka in order to analyze some records stored in the database.
In some cases, the event matches some condition that means, that corresponding record should be analyzed later in the future.
Perhaps, the most simple solution to implement this logic is to write timestamp of future processing to the database and periodically perform some kind of select to find required records for re-processing.
Maybe there is another more convenient and scalable way to do it? It looks like another timestamped event stream which could be processed when the current time become greater or equal than timestamp of event, what's the options to implement such behavior?

Answer 1

In my opinion depending on how long you need to store it, you can just create a stream that filter for these events and push it into a new topic that can be processed later. If it is more for historical purpose then it might be better to push it into a DBMS.

Answer 2

You can try state store in Kafka Stream. Which can be used by stream processing applications to store and query data in later.
Kafka Stream automatically creates and manages such state stores when you are calling stateful operators such as count() or aggregate(), or when you are windowing a stream. It will be store in In-memory however you can store in somewhere persistent storage e.g. portworx to handle fault scenario.
Below show how you initialize StateStore
StoreBuilder<KeyValueStore<String, String>> statStore = Stores
.keyValueStoreBuilder(Stores.persistentKeyValueStore("uniqueName"), Serdes.String(),
Serdes.String())
.withLoggingDisabled(); // disable backing up the store to a change log topic
Below show how to add state store inside Kafka Stream
Topology builder = new Topology();
builder.addSource("Source", topic)
.addProcessor("SourceProcessName", () -> new ProcessorClass(), "Source")
.addStateStore(statStore, "SourceProcessName")
.addSink("SinkProcessName", sinkTopic, "SourceProcessName");
In the Process Method, You can store Kafka topic message as key, value
KeyValueStore<String, String> dsStore = (KeyValueStore<String, String>) context.getStateStore("statStore");
KeyValueIterator<String, String> iter = this.dsStore.all();
while (iter.hasNext()) {
KeyValue<String, String> entry = iter.next();
}