Kafka - Retries and Recovery not invoked - apache-kafka

Question

I have implemented a custom listener i.e. not using the #KafkaListener annotation due to the fact that my application need to dynamically lsiten to topics. I have seen suggestions of moving to Spring kafka 2.6.x but I can't upgrade due to the fact that I am stuck (at least for now) with Spring 5.1.X.RELEASE which means I can only use Spring-kafka 2.2.x.
My question is, how can I achieve retry, recovery and error handling with Spring-kafka 2.2.x?
ConcurrentKafkaListenerContainerFactory listenerFactory = new ConcurrentKafkaListenerContainerFactory();
listenerFactory.setConsumerFactory(consumerFactory);
configurer.configure(listenerFactory, consumerFactory);
listenerFactory.setConcurrency(listenerConcurrency);
listenerFactory.setStatefulRetry(Boolean.TRUE);
listenerFactory.setBatchListener(isBatchListener);
listenerFactory.getContainerProperties().setTransactionManager(chainedKafkaTransactionManager);
listenerFactory.getContainerProperties().setAckOnError(false);
listenerFactory.setRetryTemplate(retryTemplate(kafkaEhCacheRetryManager));
listenerFactory.setRecoveryCallback(kafkaRecoverer);
My retry template looks like:
RetryTemplate retryTemplate(EhCacheCacheManager kafkaEhCacheRetryManager) {
ExponentialBackOffPolicy exponentialBackOffPolicy = new ExponentialBackOffPolicy();
exponentialBackOffPolicy.setInitialInterval(initialIntervalForRetries);
exponentialBackOffPolicy.setSleeper(new ThreadWaitSleeper());
exponentialBackOffPolicy.setMultiplier(2.0);
exponentialBackOffPolicy.setMaxInterval(maxIntervalForRetries);
RetryTemplate retryTemplate = new RetryTemplate();
retryTemplate.setRetryContextCache(new KafkaEhRetryContextCache(kafkaEhCacheRetryManager));
retryTemplate.setBackOffPolicy(exponentialBackOffPolicy);
// KafkaTransactionalRetryPolicy extends SimpleRetryPolicy
KafkaTransactionalRetryPolicy retryPolicy = new KafkaTransactionalRetryPolicy(kafkaTemplate);
retryPolicy.setMaxAttempts(maxAttempts);
retryTemplate.setRetryPolicy(kafkaTransactionalRetryPolicy);
return retryTemplate;
}
My listener looks like:
public class MyKafkaListener implements MessageListener<String, String> {
#Override
#Transactional(value = "chainedKafkaTransactionManager")
public void onMessage(final ConsumerRecord<String, String> consumerRecord){
throw new RuntimeException("thrown out of out anger");
}
}
with this config:
spring.kafka:
bootstrap-servers: ${service.kakfa.host}
admin:
client-id: test-consumers
bootstrap-servers: ${service.kakfa.host}
consumer:
bootstrap-servers: ${service.kakfa.host}
group-id: local-consumers
client-id: local-consumers
auto-offset-reset: earliest
value-deserializer: org.apache.kafka.common.serialization.StringDeserializer
enable-auto-commit: false
isolation-level: read_committed
producer:
bootstrap-servers: ${service.kakfa.host}
client-id: local-producer
acks: all
retries: 3
transaction-id-prefix: local-producer-tx-
properties:
enable.idempotence: true
transactional.id: tran-id-1-
max.in.flight.requests.per.connection: 5
listener.concurrency: 1
I have seen several examples on StackOverflow on how to do this, but none has worked so far.

Answer 1

The retry mechanism you are trying to use only applies to #KafakListeners. It is built into a listener adapter used to call the listener POJO.
In newer versions, the SeekToCurrentErrorHandler and DefaultAfterRollbackProcessor have a back off (Since 2.3), eliminating the need for a retry template at the listener level, in favor of retry at the container level.
With your own listener you would have to use a RetryTemplate within the listener code itself.
BTW, Spring 5.1.x is no longer supported https://github.com/spring-projects/spring-framework/wiki/Spring-Framework-Versions#supported-versions