Unable to produce message when leader is not available - apache-kafka

Question

I am unable to produce message when Leader is not available. I created new topic with replication factor 2 using below command
~/kafka/bin/kafka-topics.sh --create --zookeeper localhost:2181 --topic TestTopic202 --partitions 1 --replication-factor 2
Then
~/kafka/bin/kafka-topics.sh --describe --zookeeper localhost:2181 --topic TestTopic202
Output:
OpenJDK 64-Bit Server VM warning: If the number of processors is expected to increase from one, then you should configure the number of parallel GC threads appropri
ately using -XX:ParallelGCThreads=N
Topic:TestTopic202 PartitionCount:1 ReplicationFactor:2 Configs:
Topic: TestTopic202 Partition: 0 Leader: 1 Replicas: 1,2 Isr: 1,2
After creating topic I stopped broker-1 which is leader of this topic to test fault tolerance. I expected broker-2 will be elected as leader but I received broker down error message.
~/kafka/bin/kafka-console-producer.sh --broker-list 34.93.59.30:9092 --topic TestTopic202
OpenJDK 64-Bit Server VM warning: If the number of processors is expected to increase from one, then you should configure the number of parallel GC threads appropriately using -XX:ParallelGCThreads=N
>Hi
After bringing down,
^C[kafka#hgtestsrv1 ~]$ ~/kafka/bin/kafka-console-producer.sh --broker-list localhost:9092 --topic TestTopic202
\OpenJDK 64-Bit Server VM warning: If the number of processors is expected to increase from one, then you should configure the number of parallel GC threads appropriately using -XX:ParallelGCThreads=N
>Hi
[2019-09-03 12:25:47,305] WARN [Producer clientId=console-producer] Connection to node -1 (localhost/127.0.0.1:9092) could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient)
[2019-09-03 12:25:47,407] WARN [Producer clientId=console-producer] Connection to node -1 (localhost/127.0.0.1:9092) could not be established. Broker may not be ava
ilable. (org.apache.kafka.clients.NetworkClient)
[2019-09-03 12:25:47,511] WARN [Producer clientId=console-producer] Connection to node -1 (localhost/127.0.0.1:9092) could not be established. Broker may not be ava
ilable. (org.apache.kafka.clients.NetworkClient)
[2019-09-03 12:25:47,765] WARN [Producer clientId=console-producer] Connection to node -1 (localhost/127.0.0.1:9092) could not be established. Broker may not be ava
ilable. (org.apache.kafka.clients.NetworkClient)
[2019-09-03 12:25:48,222] WARN [Producer clientId=console-producer] Connection to node -1 (localhost/127.0.0.1:9092) could not be established. Broker may not be ava
ilable. (org.apache.kafka.clients.NetworkClient)
^Corg.apache.kafka.common.KafkaException: Producer closed while send in progress
at org.apache.kafka.clients.producer.KafkaProducer.doSend(KafkaProducer.java:862)
at org.apache.kafka.clients.producer.KafkaProducer.send(KafkaProducer.java:839)
at kafka.tools.ConsoleProducer$.send(ConsoleProducer.scala:75)
at kafka.tools.ConsoleProducer$.main(ConsoleProducer.scala:57)
at kafka.tools.ConsoleProducer.main(ConsoleProducer.scala)
Caused by: org.apache.kafka.common.KafkaException: Requested metadata update after close
at org.apache.kafka.clients.Metadata.awaitUpdate(Metadata.java:200)
at org.apache.kafka.clients.producer.KafkaProducer.waitOnMetadata(KafkaProducer.java:982)
at org.apache.kafka.clients.producer.KafkaProducer.doSend(KafkaProducer.java:859)
... 4 more
Can anyone tell me how to test fault tolerance?
My server.properties
default.replication.factor=2
offsets.topic.replication.factor=2
transaction.state.log.replication.factor=2
transaction.state.log.min.isr=1

Answer 1

Before producing a message to Kafka, Kafka fetches metadata regarding how many partitions are there for that topic and which broker is the leader for which partition etc.
Approach #1. Now, it is important to give your bootstrap servers (broker list) of the producer to the broker that is reachable. You said, you have brought broker-1 down, so you cannot include it as the only one in your Kafka producer bootstrap servers.
The following configuration needs to be changed.
bootstrap.servers=<IP>:<Port>
Set this to your broker-2 and see.
For your question of testing fault tolerance...
Your approach is right, but you must include multiple (if not all) brokers in your producer/consumer bootstrap.servers property.
You gave localhost:9092 Is this the broker which you brought down? Where is the other broker running? The running broker must be given to the producer.
Check to see if the other broker is running and is registered with the same zookeeper.
Update:
You gave 2 different IPs 34.93.59.30:9092 and next localhost:9092, I suspect that they are two different brokers on two different machines.
Approach #2. If it is the case, ensure that (as said before) your localhost:9092 broker is registered with the zookeeper of 34.93.59.30:9092 broker (It should mostly be registered with 34.93.59.30:2181 if you use the default ports).
So in the server.properties of your localhost:9092 (your second broker you should give the same zookeeper as your broker-1).
zookeeper.connect=34.93.59.30:2181
and foremost, ensure that your localhost can connect to that first.
Approach #3. If you still were not able to figure this out check the output of your describe topics.
~/kafka/bin/kafka-topics.sh --describe --zookeeper localhost:2181 --topic TestTopic202
after you brought the broker-1 down. It should show you the leader.