First started Kafka server & leader failure blocks consumers - apache-kafka

Question

I'm basically doing the kafka quickstart using kafka_2.11-2.0.0 which comes with zookeeper.version=3.4.13-2d71af4dbe22557fda74f9a9b4309b15a7487f03, built on 06/29/2018 00:39 GMT.
I also use Ubuntu:
lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 16.04.5 LTS
Release: 16.04
Codename: xenial
I do these with the order below:
start zookeeper
start kafka server0 with port 9092
start kafka server1 with port 9093
start kafka server2 with port 9094
create a topic: kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 3 --partitions 1 --topic my-replicated-topic
produce some messages: kafka-console-producer.sh --broker-list localhost:9092 --topic my-replicated-topic
consume from server0: kafka-console-consumer.sh --bootstrap-server localhost:9092 --from-beginning --topic my-replicated-topic
consume from server1: kafka-console-consumer.sh --bootstrap-server localhost:9093 --from-beginning --topic my-replicated-topic
check the leader for my-replicated-topic and find it to be server0 -> here's the tricky part; one should kill server1 and then maybe server2 but never server0 and then restore them just in order to get server0 to be the leader of my-replicated-topic
kill server0
check for the new leader (happens to be server2): kafka-topics.sh --describe --zookeeper localhost:2181 --topic my-replicated-topic
produce some messages to server2: kafka-console-producer.sh --broker-list localhost:9094 --topic my-replicated-topic
consume from server2 (or server1): kafka-console-consumer.sh --bootstrap-server localhost:9094 --from-beginning --topic my-replicated-topic -> this will hang till restarting server0
starting again server0
consumer from server2 outputs all messages including the one sent to server2 when was the leader
What is wrong and how one would solve the problem so not to matter which server becomes the leader?

Answer 1

I think this is due to replication factor of "__consumer_offsets" topic. This is set to one in server.properties file for testing purpose.
set offsets.topic.replication.factor=3 for high availability.
Copied from KAFKA-7526 comment