Spring Integration - Kafka Outbound Adapter Acknowledge Issue - apache-kafka

Question

Before i post my question, i would like to thank Gary and Artem for helping me in resolving my issues and bcoz of that i am able to successfuly post messages from JMS to Kafka with transaction in place.
Now, i am facing another issue and testing what will happen when my Kafka is down.
When kafka is down for first few retries kafka outbound adapter throws exception and messages are returned back to JMS and retried again and again.
However, after couple of retries , even when kafka is down, messages are dequeued from JMS and i get the following exception:
2017-07-10 23:27:51.117 ERROR 16116 --- [enerContainer-1] o.s.k.support.LoggingProducerListener : Exception thrown when sending a message with key='null' and payload='Test JPMC' to topic test:
org.apache.kafka.common.errors.TimeoutException: Failed to update metadata after 60000 ms.
My integration xml is :
<?xml version="1.0" encoding="UTF-8"?>
<beans xmlns="http://www.springframework.org/schema/beans"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:jms="http://www.springframework.org/schema/integration/jms"
xmlns:integration="http://www.springframework.org/schema/integration"
xmlns:int-kafka="http://www.springframework.org/schema/integration/kafka"
xmlns:task="http://www.springframework.org/schema/task"
xsi:schemaLocation="http://www.springframework.org/schema/beans
http://www.springframework.org/schema/beans/spring-beans.xsd
http://www.springframework.org/schema/integration/jms
http://www.springframework.org/schema/integration/jms/spring-integration-jms.xsd
http://www.springframework.org/schema/integration/kafka
http://www.springframework.org/schema/integration/kafka/spring-integration-kafka.xsd">
<jms:message-driven-channel-adapter
id="helloJMSAdapater" container="requestListenerContainer"
channel="helloChannel" extract-payload="true" error-channel="errorChannel"/>
<integration:recipient-list-router
input-channel="errorChannel">
<integration:recipient channel="errorOutputChannel" />
<integration:recipient channel="rethrowChannel" />
</integration:recipient-list-router>
<jms:outbound-channel-adapter id="errorQueueChannelAdapter"
channel="errorOutputChannel" destination="errorQueue" connection-factory="jmsConnectionfactory"
delivery-persistent="true" explicit-qos-enabled="true" />
<int-kafka:outbound-channel-adapter
id="kafkaOutboundChannelAdapter" kafka-template="kafkaTemplate"
auto-startup="true" sync="true" channel="inputToKafka" topic="test">
</int-kafka:outbound-channel-adapter>
</beans>
I dont want to acknowledge the JMS messages unless they are successfully posted into kafka.
Is it because of some default parameters that kafka is setting?
My kafka Config is below:
#Configuration
#Component
public class KafkaConfig {
#Bean
public KafkaTemplate<String, String> kafkaTemplate() {
return new KafkaTemplate<>(producerFactory());
}
#Bean
public ProducerFactory<String, String> producerFactory() {
Map<String, Object> props = new HashMap<>();
props.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, "localhost:9092");//this.brokerAddress);
props.put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, "org.apache.kafka.common.serialization.StringSerializer");
props.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, "org.apache.kafka.common.serialization.StringSerializer");
// set more properties
return new DefaultKafkaProducerFactory<>(props);
}
}

Answer 1

That isn't Kafka problem. If you say your message is "dequeued from JMS", be sure that the redelivery policy on the queue is configured to infinite.
For example ActiveMQ story is here: http://activemq.apache.org/redelivery-policy.html
maximumRedeliveries 6 Sets the maximum number of times a message will be redelivered before it is considered a poisoned pill and returned to the broker so it can go to a Dead Letter Queue.
Set to -1 for unlimited redeliveries.