Troubleshooting for kafka offline partitions - apache-kafka

Question

After unexpected shutdown of brokers, some of the topic partitions remain offline even if all the brokers are back up and running. Does anyone know the solution for this issue ?
2019-05-17T10:40:32,379 [myid:] - INFO [controller-event-thread:Logging$class#70] - [Controller 3]: Starting preferred replica leader election for partitions [topic,9]
2019-05-17T10:40:32,379 [myid:] - INFO [controller-event-thread:Logging$class#70] - [Partition state machine on Controller 3]: Invoking state change to OnlinePartition for partitions [topic,9]
2019-05-17T10:40:32,380 [myid:] - INFO [controller-event-thread:Logging$class#70] - [PreferredReplicaPartitionLeaderSelector]: Current leader -1 for partition [topic,9] is not the preferred replica. Triggering preferred replica leader election
2019-05-17T10:40:32,380 [myid:] - WARN [controller-event-thread:Logging$class#85] - [Controller 3]: Partition [topic,9] failed to complete preferred replica leader election. Leader is -1

Answer 1

My colleague and I just ran into a similar problem, however, we were trying to delete a topic that had offline partitions. The key to your problem is that your leader is -1.
The way we fixed this was by manually editing the znode in Zookeeper to point the leader to a broker that was online and doing a rolling restart of the cluster. Using the Zookeeper cli get the following znode:
/brokers/topics/<my-topic>/partitions/0/state.
In our case it returned:
{"controller_epoch":52,"leader":-1,"version":1,"leader_epoch":35,"isr":[5]}
Notice that the leader is -1. You might try updating the znode, setting the leader to a broker that is up and running.