Separate Apache Kafka clusters unreachable at the same time - kafka_network_socketserver_networkprocessoravgidlepercent goes to zero - apache-kafka

Question

We have 4 Kafka clusters:
ENV1: 6 brokers and 3 zookeepers
ENV2: 6 brokers and 3 zookeepers
ENV3: 8 brokers (on 2 DCs, 4-4 brokers) and 9 zookeepers (on 3 DCs, 3-3-3 nodes)
ENV4: 16 brokers (on 2 DCs, 8-8 brokers) and 9 zookeepers (on 3 DCs, 3-3-3 nodes)
All of the Kafka brokers are on version 2.7.0, and all of the ZK nodes are on version 3.4.13. Every Kafka brokers and ZK nodes are VMs. All the four environments are running in a separate subnet. Swap is turned off everywhere. All the clusters are Kerberized and are using a separate high available AD for it, which contains 7 kerberos servers.
VM parameters:
ENV1:
Kafka brokers:
16 GB RAM,
8 vCPU,
1120 GB Hard Disk,
RHEL 7.9
ZK nodes:
4 GB RAM,
2 vCPU,
105 GB Hard Disk,
RHEL 8.5
ENV2:
Kafka brokers:
16 GB RAM,
8 vCPU,
1120 GB Hard Disk,
RHEL 7.9
ZK nodes:
4 GB RAM,
2 vCPU,
105 GB Hard Disk,
RHEL 8.5
ENV3:
Kafka brokers:
24 GB RAM,
8 vCPU,
2120 GB Hard Disk,
RHEL 7.9
ZK nodes:
8 GB RAM,
2 vCPU,
200 GB Hard Disk,
RHEL 8.5
ENV4:
Kafka brokers:
24 GB RAM,
8 vCPU,
7145 GB Hard Disk,
RHEL 7.9
ZK nodes:
8 GB RAM,
2 vCPU,
200 GB Hard Disk,
RHEL 7.9
We have the following issue on every environments at the same time, 3-4 times a day for a few seconds: our kafka_network_socketserver_networkprocessoravgidlepercent metrics goes down to zero on every brokers, and our cluster becames unreachable, even our brokers cannot communicate with each other when this happens. Here is a picture of it from our Grafana dashboard:
We can see the following ERRORs in the server log, but we suppose all of them are just consequences:
ERROR Error while processing notification change for path = /kafka-acl-changes (kafka.common.ZkNodeChangeNotificationListener)
kafka.zookeeper.ZooKeeperClientExpiredException: Session expired either before or while waiting for connection
at kafka.zookeeper.ZooKeeperClient.$anonfun$waitUntilConnected$3(ZooKeeperClient.scala:270)
at kafka.zookeeper.ZooKeeperClient.waitUntilConnected(ZooKeeperClient.scala:258)
at kafka.zookeeper.ZooKeeperClient.$anonfun$waitUntilConnected$1(ZooKeeperClient.scala:252)
at kafka.zookeeper.ZooKeeperClient.waitUntilConnected(ZooKeeperClient.scala:252)
at kafka.zk.KafkaZkClient.retryRequestsUntilConnected(KafkaZkClient.scala:1730)
at kafka.zk.KafkaZkClient.retryRequestsUntilConnected(KafkaZkClient.scala:1700)
at kafka.zk.KafkaZkClient.retryRequestUntilConnected(KafkaZkClient.scala:1695)
at kafka.zk.KafkaZkClient.getChildren(KafkaZkClient.scala:719)
at kafka.common.ZkNodeChangeNotificationListener.kafka$common$ZkNodeChangeNotificationListener$$processNotifications(ZkNodeChangeNotificationListener.scala:83)
at kafka.common.ZkNodeChangeNotificationListener$ChangeNotification.process(ZkNodeChangeNotificationListener.scala:120)
at kafka.common.ZkNodeChangeNotificationListener$ChangeEventProcessThread.doWork(ZkNodeChangeNotificationListener.scala:146)
at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:96)
ERROR [ReplicaManager broker=1] Error processing append operation on partition topic-4 (kafka.server.ReplicaManager)
org.apache.kafka.common.errors.OutOfOrderSequenceException: Out of order sequence number for producerId 461002 at offset 5761036 in partition topic-4: 1022 (incoming seq. number), 1014 (current end sequence number)
ERROR [KafkaApi-5] Number of alive brokers '0' does not meet the required replication factor '3' for the offsets topic (configured via 'offsets.topic.replication.factor'). This error can be ignored if the cluster is starting up and not all brokers are up yet. (kafka.server.KafkaApis)
ERROR [KafkaApi-11] Error when handling request: clientId=broker-12-fetcher-0, correlationId=8972304, api=FETCH, version=12, body={replica_id=12,max_wait_ms=500,min_bytes=1,max_bytes=10485760,isolation_level=0,session_id=683174603,session_epoch=47,topics=[{topic=__consumer_offsets,partitions=[{partition=36,current_leader_epoch=294,fetch_offset=1330675527,last_fetched_epoch=-1,log_start_offset=0,partition_max_bytes=1048576,_tagged_fields={}},{partition=25,current_leader_epoch=288,fetch_offset=3931235489,last_fetched_epoch=-1,log_start_offset=0,partition_max_bytes=1048576,_tagged_fields={}}],_tagged_fields={}}],forgotten_topics_data=[],rack_id=,_tagged_fields={}} (kafka.server.KafkaApis)
org.apache.kafka.common.errors.NotLeaderOrFollowerException: Leader not local for partition topic-42 on broker 11
ERROR [GroupCoordinator 5]: Group consumer-group_id could not complete rebalance because no members rejoined (kafka.coordinator.group.GroupCoordinator)
ERROR [Log partition=topic-1, dir=/kafka/logs] Could not find offset index file corresponding to log file /kafka/logs/topic-1/00000000000001842743.log, recovering segment and rebuilding index files... (kafka.log.Log)
[ReplicaFetcher replicaId=16, leaderId=10, fetcherId=0] Error in response for fetch request (type=FetchRequest, replicaId=16, maxWait=500, minBytes=1, maxBytes=10485760, fetchData={}, isolationLevel=READ_UNCOMMITTED, toForget=, metadata=(sessionId=1439560581, epoch=1138570), rackId=) (kafka.server.ReplicaFetcherThread) java.io.IOException: Connection to 10 was disconnected before the response was read at org.apache.kafka.clients.NetworkClientUtils.sendAndReceive(NetworkClientUtils.java:100) at kafka.server.ReplicaFetcherBlockingSend.sendRequest(ReplicaFetcherBlockingSend.scala:110) at kafka.server.ReplicaFetcherThread.fetchFromLeader(ReplicaFetcherThread.scala:211) at kafka.server.AbstractFetcherThread.processFetchRequest(AbstractFetcherThread.scala:301) at kafka.server.AbstractFetcherThread.$anonfun$maybeFetch$3(AbstractFetcherThread.scala:136) at kafka.server.AbstractFetcherThread.maybeFetch(AbstractFetcherThread.scala:135) at kafka.server.AbstractFetcherThread.doWork(AbstractFetcherThread.scala:118) at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:96)
I will update the issue if needed, eg. with relevant Kafka configs.
We know that it might be some issue with our infrastructure, but can not see problem with the network, neither on the Kerberos side, so thats why I'm asking for help from you guys. Do you have any idea what may cause this issue? Every idea may be helpful, because we run out of it.
Thanks in advance!