kafka client getting TOPIC_AUTHORIZATION_FAILED on cluster restart - apache-kafka

Question

I am using an SSL enabled kafka cluster to connect to consumer and publish messages. Below is the tech stack.
spring-kafka : 2.6.6
spring-boot : 2.4.3
Kafka properties
kafka:
bootstrap-servers: ${BOOTSTRAP-SERVERS-HOST}
subscription-topic: TEST
properties:
security.protocol: SSL
ssl.truststore.location: ${SUBSCRIPTION_TRUSTSTORE_PATH}
ssl.truststore.password: ${SUBSCRIPTION_TRUSTSTORE_PWD}
ssl.keystore.location: ${SUBSCRIPTION_KEYSTORE_PATH}
ssl.keystore.password: ${SUBSCRIPTION_KEYSTORE_PWD}
Issue:
Kafka Client application is up connected to the kafka cluster consumer and publishing messages as expected.
Now we stop the kafka broker/cluster below error is logged.
could not be established. Broker may not be available.
This is fine and expected as broker/cluster is down.
Now We start the broker/cluster and below error start appearing and kafka consumer stops consuming messages from topic however kafka publisher is able to send message to the topic. [application restart resolves this issue]
Trying to understand the root cause any help is much appreciated.
2022-01-13 13:34:52.078 [TEST.CONSUMER-GROUP-0-C-1] ERROR--SUBSCRIPTION - -org.apache.kafka.clients.Metadata.checkUnauthorizedTopics - [Consumer clientId=consumer-TEST.CONSUMER-GROUP-1, groupId=TEST.CONSUMER-GROUP] Topic authorization failed for topics [TEST]
2022-01-13 13:34:52.078 [TEST.CONSUMER-GROUP-0-C-1] ERROR- -SUBSCRIPTION - -org.springframework.core.log.LogAccessor.error - Authorization Exception and no authorizationExceptionRetryInterval set
org.apache.kafka.common.errors.TopicAuthorizationException: Not authorized to access topics: [TEST]
2022-01-13 13:34:52.081 [TEST.CONSUMER-GROUP-0-C-1] ERROR- IRVS-SUBSCRIPTION - -org.springframework.core.log.LogAccessor.error - Fatal consumer exception; stopping container
2022-01-13 13:34:52.083 [TEST.CONSUMER-GROUP-0-C-1] INFO - IRVS-SUBSCRIPTION - -org.springframework.scheduling.concurrent.ExecutorConfigurationSupport.shutdown - Shutting down ExecutorService

Answer 1

Above issue was resolved after adding
AuthorizationException RetryInterval
Below is an example illustrating this
#Bean
ConcurrentKafkaListenerContainerFactory<Object, Object> kafkaListenerContainerFactory(
ConcurrentKafkaListenerContainerFactoryConfigurer configurer,
ConsumerFactory<Object, Object> kafkaConsumerFactory) {
ConcurrentKafkaListenerContainerFactory<Object, Object> factory =
new ConcurrentKafkaListenerContainerFactory<>();
factory.setConcurrency(2);
configurer.configure(factory, kafkaConsumerFactory);
// other setters like error handler , retry handler
// setting Authorization Exception Retry Interval
factory.getContainerProperties()
.setAuthorizationExceptionRetryInterval(Duration.ofSeconds(5l));
return factory;
}