Kafka consumer and producer throwing error - apache-kafka

Question

I am getting below error while trying to pass words in Kafka-consumer,
commands with i entered
console-1:(for producer)
export PATH=$PATH:/usr/hdp/current/kafka-broker/bin
kafka-topics.sh --create --zookeeper ip-172-31-20-58.ec2.internal:2181 --replication-factor 1 --partitions 1 --topic testuday1234
kafka-console-producer.sh --broker-list ip-172-31-20-58.ec2.internal:6667 --topic testuday1234
console-2: (for consumer)
export PATH=$PATH:/usr/hdp/current/kafka-broker/bin
kafka-console-consumer.sh --zookeeper localhost:2181 --topic testuday1234 --from-beginning
Please help me in resolving these error
Error i am getting in Producer console:
[udaychitukula6587#ip-172-31-38-183 ~]$ kafka-console-producer.sh --broker-list ip-172-31-20-58.ec2.internal:6667 --topic testuday1234
hi
[2018-05-28 15:27:36,761] WARN Error while fetching metadata [{TopicMetadata for topic testuday1234 ->
No partition metadata for topic testuday1234 due to kafka.common.LeaderNotAvailableException}] for topic [testuday1234]: class kafka.common.LeaderNotAvailableExcep
tion (kafka.producer.BrokerPartitionInfo)
[2018-05-28 15:27:36,774] WARN Error while fetching metadata [{TopicMetadata for topic testuday1234 ->
No partition metadata for topic testuday1234 due to kafka.common.LeaderNotAvailableException}] for topic [testuday1234]: class kafka.common.LeaderNotAvailableExcep
tion (kafka.producer.BrokerPartitionInfo)
Error i am getting in consumer console:
[udaychitukula6587#ip-172-31-38-183 ~]$ kafka-console-consumer.sh --zookeeper localhost:2181 --topic testuday123 --from-beginning
{metadata.broker.list=ip-172-31-20-58.ec2.internal:6667,ip-172-31-53-48.ec2.internal:6667,ip-172-31-60-179.ec2.internal:6667, request.timeout.ms=30000, client.id=c
onsole-consumer-63526, security.protocol=PLAINTEXT}
{metadata.broker.list=ip-172-31-20-58.ec2.internal:6667,ip-172-31-53-48.ec2.internal:6667,ip-172-31-60-179.ec2.internal:6667, request.timeout.ms=30000, client.id=c
onsole-consumer-63526, security.protocol=PLAINTEXT}

Answer 1

There is a couple of things that I've noted here.
First, in the newer versions (I think from 0.10.1) of Kafka for the console consumer, we need to use --bootstrap-server option as opposed to --zookeeper. Could you please confirm the version you're using? and also try to run the consumer command with --bootstrap-server option?
Second, for the producer in such a scenario, I would recommend checking 3 things to confirm where the issue might be:
The leader of a Kafka cluster is elected by the zookeeper, so it might be worth checking by running a zookeeper-client shell to see if there is an active controller in the Kafka cluster (in the znode path - /brokers/ids/[brokerId]).
Try running a Kafka-topics --describe --topic command to see if the topic has an active leader partition i.e. the Leader column in the output of the command should NOT have None. I've run into this myself before.
The last one is about the port number of the broker, could you please check and confirm if the broker is actually listening on that port. You'll find this information (listeners and advertised.listeners) in server.properties file on the broker. I found this post that you might find useful where the user had a problem with the port 6667.
I hope this helps!