#KafkaListener not recovering after DisconnectException - apache-kafka

Question

I have a Kafka consumer (Spring boot) configured using #KafkaListener. This was running in production and all was good until as part of the maintenance the brokers were restarted. By docs, I was expecting that the kafka listener would recover once broker is back up. However this is not what I observed from the logs. The logs stopped with following Exception:
2020-04-22 11:11:28,802|INFO|automator-consumer-app-id-0-C-1|org.apache.kafka.clients.FetchSessionHandler|[Consumer clientId=automator-consumer-app-id-0, groupId=automator-consumer-app-id] Node 10 was unable to process the fetch request with (sessionId=2138208872, epoch=348): FETCH_SESSION_ID_NOT_FOUND.
2020-04-22 11:24:23,798|INFO|automator-consumer-app-id-0-C-1|org.apache.kafka.clients.FetchSessionHandler|[Consumer clientId=automator-consumer-app-id-0, groupId=automator-consumer-app-id] Error sending fetch request (sessionId=499459047, epoch=314160) to node 7: org.apache.kafka.common.errors.DisconnectException.
2020-04-22 11:36:37,241|INFO|automator-consumer-app-id-0-C 1|org.apache.kafka.clients.FetchSessionHandler|[Consumer clientId=automator-consumer-app-id-0, groupId=automator-consumer-app-id] Error sending fetch request (sessionId=2033512553, epoch=342949) to node 4: org.apache.kafka.common.errors.DisconnectException.
Once the application was restarted, the connectivity reestablished. I was wondering if this could be related with any of the consumer configuration below.
2020-04-22 12:46:59,681|INFO|main|org.apache.kafka.clients.consumer.ConsumerConfig|ConsumerConfig values:
allow.auto.create.topics = true
auto.commit.interval.ms = 5000
auto.offset.reset = latest
bootstrap.servers = [msk-e00-br1.int.bell.ca:9092]
check.crcs = true
client.dns.lookup = default
client.id = automator-consumer-app-id-0
client.rack =
connections.max.idle.ms = 540000
default.api.timeout.ms = 60000
enable.auto.commit = false
exclude.internal.topics = true
fetch.max.bytes = 52428800
fetch.max.wait.ms = 500
fetch.min.bytes = 1
group.id = automator-consumer-app-id
group.instance.id = null
heartbeat.interval.ms = 3000
interceptor.classes = []
internal.leave.group.on.close = true
isolation.level = read_uncommitted
key.deserializer = class org.apache.kafka.common.serialization.StringDeserializer
max.partition.fetch.bytes = 1048576
max.poll.interval.ms = 300000
max.poll.records = 500
metadata.max.age.ms = 300000
metric.reporters = []
metrics.num.samples = 2
metrics.recording.level = INFO
metrics.sample.window.ms = 30000
partition.assignment.strategy = [class org.apache.kafka.clients.consumer.RangeAssignor]
receive.buffer.bytes = 65536
reconnect.backoff.max.ms = 1000
reconnect.backoff.ms = 50
request.timeout.ms = 30000
retry.backoff.ms = 100
sasl.client.callback.handler.class = null
sasl.jaas.config = null
sasl.kerberos.kinit.cmd = /usr/bin/kinit
sasl.kerberos.min.time.before.relogin = 60000
sasl.kerberos.service.name = null
sasl.kerberos.ticket.renew.jitter = 0.05
sasl.kerberos.ticket.renew.window.factor = 0.8
sasl.login.callback.handler.class = null
sasl.login.class = null
sasl.login.refresh.buffer.seconds = 300
sasl.login.refresh.min.period.seconds = 60
sasl.login.refresh.window.factor = 0.8
sasl.login.refresh.window.jitter = 0.05
sasl.mechanism = GSSAPI
security.protocol = PLAINTEXT
send.buffer.bytes = 131072
session.timeout.ms = 10000
ssl.cipher.suites = null
ssl.enabled.protocols = [TLSv1.2, TLSv1.1, TLSv1]
ssl.endpoint.identification.algorithm = https
ssl.key.password = null
ssl.keymanager.algorithm = SunX509
ssl.keystore.location = null
ssl.keystore.password = null
ssl.keystore.type = JKS
ssl.protocol = TLS
ssl.provider = null
ssl.secure.random.implementation = null
ssl.trustmanager.algorithm = PKIX
ssl.truststore.location = null
ssl.truststore.password = null
ssl.truststore.type = JKS
value.deserializer = class org.apache.kafka.common.serialization.StringDeserializer

Answer 1

Increase the value of
max.incremental.fetch.session.cache.slots
. The default value is 1K. You can refer the answer here : How to check the actual number of incremental fetch session cache slots used in Kafka cluster?

Answer 2

Basically, your application is continuously listening the messages from the topic, suppose if there is no message published in the topic you will get this type of exception.
org.apache.kafka.common.errors.DisconnectException: null
Disconnect Exception class
If we start sending messages to topic, application will start running and consume those messages.
Here you need to increase the request timeout in your properties files.
consumer.request.timeout.ms: