define non retryable exception in kafka - apache-kafka

Question

hi i have a kafka consumer application that uses spring kafka. It is consuming messages batch wise. But before that it was consuming sequentially. When i consume sequentially i used below annotation
#RetryableTopic(
attempts = "3}",
backoff = #Backoff(delay = 1000, multiplier = 2.0),
autoCreateTopics = "false",
topicSuffixingStrategy = TopicSuffixingStrategy.SUFFIX_WITH_INDEX_VALUE,
exclude = {CustomNonRetryableException.class})
In my code i throw CustomNonRetryableException whenever i dont need to retry an exception scenario. For other exceptions it will retry 3 times.
But when i switched to batch processing, i removed above code and used below kafkalistenercontainerfactory.
#Bean
public ConcurrentKafkaListenerContainerFactory<String, Object> kafkaListenerContainerFactory() {
ConcurrentKafkaListenerContainerFactory<String, Object> factory =
new ConcurrentKafkaListenerContainerFactory<>();
factory.setConsumerFactory(consumerFactory());
factory.getContainerProperties().setCommitLogLevel(LogIfLevelEnabled.Level.DEBUG);
factory.getContainerProperties().setMissingTopicsFatal(false);
factory.setBatchListener(true);
factory.getContainerProperties().setAckMode(ContainerProperties.AckMode.BATCH);
factory.setCommonErrorHandler(new DefaultErrorHandler(new DeadLetterPublishingRecoverer(kafkaTemplate(),
(r, e) -> {
return new TopicPartition(r.topic() + "-dlt", r.partition());
}),new FixedBackOff(1000L, 2L)));
return factory;
}
Now what i'm trying to do is apply that CustomNonRetryableException class so that it wont retry 3 times. I want only CustomNonRetryableException throwing scenarios to be retried one time and send to the dlt topic. How can i achieve it?

Answer 1

It's a bug - see this answer Spring Boot Kafka Batch DefaultErrorHandler addNotRetryableExceptions?
It is fixed and will be available in the next release.
Also see the note in that answer about the preferred way to handle errors when using a batch listener, so that only the failed record is retried, instead of the whole batch.