Kafka Streams not writing expected result after countByKey - apache-kafka

Question

Using Kafka Streams (version 0.10.0.1) and Kafka Broker (0.10.0.1) I'm trying to generate counts based on message keys. I produce my messages with the following command:
./bin/kafka-console-producer.sh --broker-list localhost:9092 --topic kafka-streams-topic --property parse.key=true --property key.separator=,
When I run the above command I can send a key and value like this:
1,{"value":10}
This will send a message to kafka that has a key = 1 and a value = {"value":10}.
My goal is to then count how many messages have the key=1. Given the above commands the count would be 1.
Here is the code that I am using:
public class StreamProcessor {
public static void main(String[] args) {
KStreamBuilder builder = new KStreamBuilder();
final Serde<Long> longSerde = Serdes.Long();
final Serde<String> stringSerde = Serdes.String();
KStream<String, String> values = builder.stream(stringSerde, stringSerde, "kafka-streams-topic");
KStream<String, Long> counts = values
.countByKey(stringSerde, "valueCounts")
.toStream();
counts.print(stringSerde, longSerde);
counts.to(stringSerde, longSerde, "message-counts-topic");
KafkaStreams streams = new KafkaStreams(builder, properties());
streams.start();
Runtime.getRuntime().addShutdownHook(new Thread(streams::close));
}
private static Properties properties() {
final Properties streamsConfiguration = new Properties();
streamsConfiguration.put(StreamsConfig.APPLICATION_ID_CONFIG, "kafka-streams-poc");
streamsConfiguration.put(StreamsConfig.BOOTSTRAP_SERVERS_CONFIG, "localhost:9092");
streamsConfiguration.put(StreamsConfig.ZOOKEEPER_CONNECT_CONFIG, "localhost:2181");
streamsConfiguration.put(StreamsConfig.KEY_SERDE_CLASS_CONFIG, Serdes.String().getClass().getName());
streamsConfiguration.put(StreamsConfig.VALUE_SERDE_CLASS_CONFIG, Serdes.String().getClass().getName());
return streamsConfiguration;
}
}
When I run counts.print(stringSerde,longSerde) I get:
1 , 1
Meaning that I have a key=1 and their is 1 message that has that key. That is what I expect.
However, when the following line runs:
counts.to(stringSerde, longSerde, "message-counts-topic");
The topic called message-counts-topic gets a message sent to it but when I try to read the message using this command:
./bin/kafka-console-consumer.sh --zookeeper localhost:2181 --topic message-counts-topic --property print.key=true --property key.separator=, --from-beginning
I get the following output:
1 ,
Where the 1 is the key and nothing is displayed for the value. I expect to see the message 1 , 1. But for some reason the count value is lost, even though its displayed when calling the print method.

Answer 1

You need to specify a different value deserializer for bin/kafka-console-consumer.sh. Add the following:
--property value.deserializer=org.apache.kafka.common.serialization.LongDeserializer
Default String deserializer fails to read the long value correctly.