How to know when record is committed in Kafka? - apache-kafka

Question

In case of integration testing, I send a record into Kafka, and I would like to know when it will be processed and committed, and then do my assertions (instead of using a Thread.sleep)...
Here is my try :
public void sendRecordAndWaitUntilItsNotConsumed(ProducerRecord<String, String> record)
throws ExecutionException, InterruptedException {
RecordMetadata recordMetadata = producer.send(record).get();
TopicPartition topicPartition = new TopicPartition(recordMetadata.topic(),
recordMetadata.partition());
try (KafkaConsumer<String, String> consumer = new KafkaConsumer<>(consumerConfig)) {
consumer.assign(Collections.singletonList(topicPartition));
long recordOffset = recordMetadata.offset();
long currentOffset = getCurrentOffset(consumer, topicPartition);
while (currentOffset <= recordOffset) {
currentOffset = getCurrentOffset(consumer, topicPartition);
LOGGER.info("Waiting for message to be consumed - Current Offset = " + currentOffset
+ " - Record Offset = " + recordOffset);
}
}
}
private long getCurrentOffset(KafkaConsumer<String, String> consumer,
TopicPartition topicPartition) {
consumer.seekToEnd(Collections.emptyList());
return consumer.position(topicPartition);
}
But it doesn't work. Indeed, I disabled the commit of the consumer, and it doesn't loop on Waiting for message to be consumed...

Answer 1

It works using KafkaConsumer#committed instead of KafkaConsumer#position.
private void sendRecordAndWaitUntilItsNotConsumed(ProducerRecord<String, String> record) throws InterruptedException, ExecutionException {
RecordMetadata recordMetadata = producer.send(record).get();
TopicPartition topicPartition = new TopicPartition(recordMetadata.topic(),
recordMetadata.partition());
consumer.assign(Collections.singletonList(topicPartition));
long recordOffset = recordMetadata.offset();
long currentOffset = getCurrentOffset(topicPartition);
while (currentOffset < recordOffset) {
currentOffset = getCurrentOffset(topicPartition);
LOGGER.info("Waiting for message to be consumed - Current Offset = " + currentOffset
+ " - Record Offset = " + recordOffset);
TimeUnit.MILLISECONDS.sleep(200);
}
}
private long getCurrentOffset(TopicPartition topicPartition) {
OffsetAndMetadata offsetAndMetadata = consumer.committed(topicPartition);
return offsetAndMetadata != null ? offsetAndMetadata.offset() - 1 : -1;
}