Apache Kafka : Check existence of message in a Topic - apache-kafka

Question

I have a situation where i need to check whether a particular message already exists in a topic or not, i need absolutely no duplicates in the topic.
Can any one suggest any graceful way of doing this, rather than consuming all the messages and checking against them.

Answer 1

I do not consider myself an expert in Kafka, but I think what you pretend is "against" the essence of Kafka.
However I come out with a solution using the Kafka Streams library for Java. Basically, the process is the following:
Map every message into a new key-value where the key is a combination of the previous key and its value: (key1, message1) -> (key1-message1, message1)
Group the messages using the keys, as a result of this operation you obtain a KGroupedStream.
Apply a reduce function, modifying the value to some custom value such as the string "Duplicated value".
Convert the resulting KTable after the reduce into a KStream and push it into a new Kafka Topic.
There are so many assumptions in the previous explanation, I am going to provide some code in order to give some light:
KStreamBuilder builder = new KStreamBuilder();
KStream<String, String> resources = builder.stream("topic-where-the-messages-are-sent");
KeyValueMapper<String, String, KeyValue<String,String>> kvMapper = new KeyValueMapper<String, String, KeyValue<String,String>>() {
public KeyValue<String, String> apply(String key, String value) {
return new KeyValue<String, String>(key + "-" + value, value);
}
};
Reducer<String> reducer = new Reducer<String>() {
public String apply(String value1, String value2) {
return "Duplicated message";
}
};
resources.map(kvMapper)
.groupByKey()
.reduce(reducer, "test-store-name")
.toStream()
.to("unique-message-output");
KafkaStreams streams = new KafkaStreams(builder, props);
streams.start();
Have in mind that this is probably not an optimal solution and maybe you would not consider it as a "graceful" way of solving your problem.
I hope it helps.