error handling when consume as a batch in kafka - apache-kafka

Question

i have a kafka consumer written in java spring boot (spirng kafka). My consumer is like below.
#RetryableTopic(
attempts = "4",
backoff = #Backoff(delay = 1000, multiplier = 2.0),
autoCreateTopics = "false",
topicSuffixingStrategy = TopicSuffixingStrategy.SUFFIX_WITH_INDEX_VALUE,
include = {ResourceAccessException.class, MyCustomRetryableException.class})
#KafkaListener(topics = "myTopic", groupId = "myGroup", autoStartup = "true", concurrency = "3")
public void consume(#Header(KafkaHeaders.RECEIVED_TOPIC) String topic,
#Header("custom_header_1") String customHeader1,
#Header("custom_header_2") String customHeader2,
#Header("custom_header_3") String customHeader3,
#Header(required = false, name = KafkaHeaders.RECEIVED_MESSAGE_KEY) String key,
#Payload(required = false) String message) {
log.info("-------------------------");
log.info(key);
log.info(message);
log.info("-------------------------");
}
I have used #RetryableTopic annotation to handle errors. I have written a custom exception class and whatever method that throw my custom exception class (MyCustomRetryableException.class), it will retry according to the backoff with number of attempts defined in the retryable annotation. So in here i dont have to do anything. Kafka will simple publish failing messages to the correct dlt topic. All i have to do is create dlt related topic since i have used autoCreateTopics = "false".
Now i'm trying to consume messages in batch wise. I changed my kafka config like below in order to consume in batch wise.
#Bean
public ConsumerFactory<String, Object> consumerFactory() {
Map<String, Object> config = new HashMap<>();
// default configs like bootstrap servers, key and value deserializers are here
config.put(ConsumerConfig.MAX_POLL_RECORDS_CONFIG, "5");
return new DefaultKafkaConsumerFactory<>(config);
}
#Bean
public ConcurrentKafkaListenerContainerFactory<String, Object> kafkaListenerContainerFactory() {
ConcurrentKafkaListenerContainerFactory<String, Object> factory =
new ConcurrentKafkaListenerContainerFactory<>();
factory.setConsumerFactory(consumerFactory());
factory.getContainerProperties().setCommitLogLevel(LogIfLevelEnabled.Level.DEBUG);
factory.setBatchListener(true);
return factory;
}
Now that i have added batch listeners, #RetryableTopic is not supported with it. So how can i achieve the publishing failed messages to DLT task which was previously handled by #RetryableTopic ?
If anyone can answer with an example it would be great. Thank you in advance.

Answer 1

See the documentation.
Use a DefaultErrorHandler with a DeadLetterPublishingRecoverer.
Non blocking retries are not supported; the retries will use the configured BackOff.
Throw a BatchListenerFailedException to indicate which record in the batch failed and just that one will be sent to the DLT.
With any other exception, the whole batch will be retried (and sent to the DLT if retries are exhausted).
https://docs.spring.io/spring-kafka/docs/current/reference/html/#retrying-batch-eh