kafka.tools.GetOffsetShell with enable Kerberos Security on Kafka 1.0 - apache-kafka

I enabled the Kerberos Authentication on Kafka 1.0. When I try to kafka-run-class.sh kafka.tools.GetOffsetShell --broker-list :9092 --topic --time -1, It hits error. Here are the Error,
[kafka#KLBIRKFKI1 kafka_2.11-1.0.1]$ bin/kafka-run-class.sh
kafka.tools.GetOffsetShell --broker-list klbirkfki1:9092 --topic
abdullah --time -1
[2018-06-26 12:59:00,058] WARN Fetching topic metadata with
correlation id 0 for topics [Set(abdullah)] from broker
[BrokerEndPoint(0,klbirkfki1,9092)] failed (kafka.client.ClientUtils$)
java.io.EOFException
at org.apache.kafka.common.network.NetworkReceive.readFromReadableChannel(NetworkReceive.java:124)
at kafka.network.BlockingChannel.readCompletely(BlockingChannel.scala:131)
at kafka.network.BlockingChannel.receive(BlockingChannel.scala:122)
at kafka.producer.SyncProducer.liftedTree1$1(SyncProducer.scala:82)
at kafka.producer.SyncProducer.kafka$producer$SyncProducer$$doSend(SyncProducer.scala:79)
at kafka.producer.SyncProducer.send(SyncProducer.scala:124)
at kafka.client.ClientUtils$.fetchTopicMetadata(ClientUtils.scala:63)
at kafka.client.ClientUtils$.fetchTopicMetadata(ClientUtils.scala:98)
at kafka.tools.GetOffsetShell$.main(GetOffsetShell.scala:79)
at kafka.tools.GetOffsetShell.main(GetOffsetShell.scala)
Exception in thread "main" kafka.common.KafkaException: fetching topic metadata for topics [Set(abdullah)] from broker [ArrayBuffer(BrokerEndPoint(0,klbirkfki1,9092))] failed
at kafka.client.ClientUtils$.fetchTopicMetadata(ClientUtils.scala:77)
at kafka.client.ClientUtils$.fetchTopicMetadata(ClientUtils.scala:98)
at kafka.tools.GetOffsetShell$.main(GetOffsetShell.scala:79)
at kafka.tools.GetOffsetShell.main(GetOffsetShell.scala)
Caused by: java.io.EOFException
at org.apache.kafka.common.network.NetworkReceive.readFromReadableChannel(NetworkReceive.java:124)
at kafka.network.BlockingChannel.readCompletely(BlockingChannel.scala:131)
at kafka.network.BlockingChannel.receive(BlockingChannel.scala:122)
at kafka.producer.SyncProducer.liftedTree1$1(SyncProducer.scala:82)
at kafka.producer.SyncProducer.kafka$producer$SyncProducer$$doSend(SyncProducer.scala:79)
at kafka.producer.SyncProducer.send(SyncProducer.scala:124)
at kafka.client.ClientUtils$.fetchTopicMetadata(ClientUtils.scala:63)
... 3 more
[kafka#KLBIRKFKI1 kafka_2.11-1.0.1]$ bin/kafka-run-class.sh kafka.tools.GetOffsetShell --broker-list klbirkfki1:9092 --topic abdullah --time -1
[2018-06-26 12:59:00,058] WARN Fetching topic metadata with correlation id 0 for topics [Set(abdullah)] from broker [BrokerEndPoint(0,klbirkfki1,9092)] failed (kafka.client.ClientUtils$)
java.io.EOFException
at org.apache.kafka.common.network.NetworkReceive.readFromReadableChannel(NetworkReceive.java:124)
at kafka.network.BlockingChannel.readCompletely(BlockingChannel.scala:131)
at kafka.network.BlockingChannel.receive(BlockingChannel.scala:122)
at kafka.producer.SyncProducer.liftedTree1$1(SyncProducer.scala:82)
at kafka.producer.SyncProducer.kafka$producer$SyncProducer$$doSend(SyncProducer.scala:79)
at kafka.producer.SyncProducer.send(SyncProducer.scala:124)
at kafka.client.ClientUtils$.fetchTopicMetadata(ClientUtils.scala:63)
at kafka.client.ClientUtils$.fetchTopicMetadata(ClientUtils.scala:98)
at kafka.tools.GetOffsetShell$.main(GetOffsetShell.scala:79)
at kafka.tools.GetOffsetShell.main(GetOffsetShell.scala)
Exception in thread "main" kafka.common.KafkaException: fetching topic metadata for topics [Set(abdullah)] from broker [ArrayBuffer(BrokerEndPoint(0,klbirkfki1,9092))] failed
at kafka.client.ClientUtils$.fetchTopicMetadata(ClientUtils.scala:77)
at kafka.client.ClientUtils$.fetchTopicMetadata(ClientUtils.scala:98)
at kafka.tools.GetOffsetShell$.main(GetOffsetShell.scala:79)
at kafka.tools.GetOffsetShell.main(GetOffsetShell.scala)
Caused by: java.io.EOFException
at org.apache.kafka.common.network.NetworkReceive.readFromReadableChannel(NetworkReceive.java:124)
at kafka.network.BlockingChannel.readCompletely(BlockingChannel.scala:131)
at kafka.network.BlockingChannel.receive(BlockingChannel.scala:122)
at kafka.producer.SyncProducer.liftedTree1$1(SyncProducer.scala:82)
at kafka.producer.SyncProducer.kafka$producer$SyncProducer$$doSend(SyncProducer.scala:79)
at kafka.producer.SyncProducer.send(SyncProducer.scala:124)
at kafka.client.ClientUtils$.fetchTopicMetadata(ClientUtils.scala:63)
... 3 more

Related

Kafka consumer disconnected from kafka cli

I a consuming Kafka messages from Kafka cli with following command
kafka-console-consumer.bat --bootstrap-server kafka:9092 --from-beginning --topic mytopic --consumer.config ....\config\consumer.properties
and I am getting following errors
[2022-09-23 15:59:33,175] WARN [Consumer clientId=console-consumer, groupId=group] Bootstrap broker kafka:9092 (id: -1 rack: null) disconnected (org.apache.kafka.clients.NetworkClient)
[2022-09-23 15:59:54,344] WARN [Consumer clientId=console-consumer, groupId=group] Connection to node -1 (kafka:9092) could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient)
[2022-09-23 15:59:54,345] WARN [Consumer clientId=console-consumer, groupId=group] Bootstrap broker kafka:9092 (id: -1 rack: null) disconnected (org.apache.kafka.clients.NetworkClient)
[2022-09-23 16:00:15,634] WARN [Consumer clientId=console-consumer, groupId=group] Connection to node -1 (kafka:9092) could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient)
I have updated server.properties with following configuration
port = 9092
advertised.host.name = kafka
KAFKA_CFG_LISTENERS=PLAINTEXT://kafka:9092
advertised.listeners=PLAINTEXT://kafka:9092

Bootstrap broker disconnected => KeeperErrorCode = NoNode for /brokers/topcis/xxxx/partitions/2/state + Key no found

I am trying to run kafka-console-consume.sh for a topic XXX
sh kafka-console-consumer.sh --bootstrap-server abcd:9092, bcde:9092, cdef:9092 --topic XXX
below error appear:
WARN clients.NetworkClient: Botstrapbroker abcd:9092 disconnected
WARN clients.NetworkClient: Botstrapbroker bcde:9092 disconnected
WARN clients.NetworkClient: Botstrapbroker cdef:9092 disconnected
When I check in the broker log, no error except below warning:
WARN nl.techop.kafka.dao.zookeeper.KafkaZkClient: KeeperErrorCode =
NoNode for /brokers/topics/XXX/partitions/2/state WARN
com.yammer.metrics.reporting.MetricsServlet: Error evaluating gauge
java.util.NoSuchElementException: key not found: [XXX,0]
What I have tried:
I created a testing123 topic successfully with below:
sh kafka-topics.sh --create --zookeeper defg:2181, dfde:2181, cdef:2181
--replication-factor 3 --partitions 3 --topic testing123
however when I try with kafka-console-producer.sh, same error happen:
sh kafka-console-producer.sh --broker-list abcd:9092, bcde:9092, cdef:9092
WARN clients.NetworkClient: Botstrap broker abcd:9092 disconnected
WARN clients.NetworkClient: Botstrap broker bcde:9092 disconnected
WARN clients.NetworkClient: Botstrap broker cdef:9092 disconnected

Thanks all.
The cluster is Kerberos Enabled.
I got the issue resolved by adding the below to my consumer.config properties file:
group.id = flume
security.protocol=SASL_PLAINTEXT
sasl.kerberos.service.name=kafka sasl.mechanism=GSSAPI
ssl.client.auth=none
and run below:
sh kafka-console-consumer.sh --bootstrap-server abcd:9092, bcde:9092, cdef:9092 --topic XXX --consumer.config /xxx/xxx/xxx/consumer.config.properties

org.apache.kafka.common.errors.TimeoutException: Failed to send request after 5000 ms

I am using Kafka V 0.10.2 (8 Cores, 16 GB RAM). While executing the command
./kafka-consumer-groups.sh --bootstrap-server localhost:9092
--describe --group
Sometimes I am getting this exception :
Exception in consumer group command (kafka.admin.ConsumerGroupCommand$)
java.lang.RuntimeException: Request GROUP_COORDINATOR failed on brokers List(localhost:9092 (id: -1 rack: null))
at kafka.admin.AdminClient.sendAnyNode(AdminClient.scala:66)
at kafka.admin.AdminClient.findCoordinator(AdminClient.scala:71)
at kafka.admin.AdminClient.describeConsumerGroup(AdminClient.scala:157)
at kafka.admin.ConsumerGroupCommand$KafkaConsumerGroupService.collectGroupAssignment(ConsumerGroupCommand.scala:399)
at kafka.admin.ConsumerGroupCommand$ConsumerGroupService.describeGroup(ConsumerGroupCommand.scala:148)
at kafka.admin.ConsumerGroupCommand$ConsumerGroupService.describeGroup$(ConsumerGroupCommand.scala:147)
at kafka.admin.ConsumerGroupCommand$KafkaConsumerGroupService.describeGroup(ConsumerGroupCommand.scala:387)
at kafka.admin.ConsumerGroupCommand$.main(ConsumerGroupCommand.scala:73)
at kafka.admin.ConsumerGroupCommand.main(ConsumerGroupCommand.scala)
Am I missing something? How to fix it?

Not able to produce message in kafka

As soon as i try to produce message in my multinode multibroker kafka cluster, i get the following error and not able to produce message. I am able to list and describe topics properly.
bin/kafka-console-producer.sh --broker-list localhost:9092 --topic nil_RF2_P1
[2016-06-17 06:55:08,555] WARN Property topic is not valid (kafka.utils.VerifiableProperties)
hi
[2016-06-17 06:55:10,844] WARN Fetching topic metadata with correlation id 0 for topics [Set(nil_RF2_P1)] from broker [id:0,host:localhost,port:9092] failed (kafka.client.ClientUtils$)
java.nio.channels.ClosedChannelException
at kafka.network.BlockingChannel.send(BlockingChannel.scala:100)
at kafka.producer.SyncProducer.liftedTree1$1(SyncProducer.scala:73)
at kafka.producer.SyncProducer.kafka$producer$SyncProducer$$doSend(SyncProducer.scala:72)
at kafka.producer.SyncProducer.send(SyncProducer.scala:113)
at kafka.client.ClientUtils$.fetchTopicMetadata(ClientUtils.scala:58)
at kafka.producer.BrokerPartitionInfo.updateInfo(BrokerPartitionInfo.scala:82)
at kafka.producer.async.DefaultEventHandler$$anonfun$handle$1.apply$mcV$sp(DefaultEventHandler.scala:67)
at kafka.utils.Utils$.swallow(Utils.scala:172)
at kafka.utils.Logging$class.swallowError(Logging.scala:106)
at kafka.utils.Utils$.swallowError(Utils.scala:45)
at kafka.producer.async.DefaultEventHandler.handle(DefaultEventHandler.scala:67)
at kafka.producer.async.ProducerSendThread.tryToHandle(ProducerSendThread.scala:105)
at kafka.producer.async.ProducerSendThread$$anonfun$processEvents$3.apply(ProducerSendThread.scala:88)
at kafka.producer.async.ProducerSendThread$$anonfun$processEvents$3.apply(ProducerSendThread.scala:68)
at scala.collection.immutable.Stream.foreach(Stream.scala:594)
at kafka.producer.async.ProducerSendThread.processEvents(ProducerSendThread.scala:67)
at kafka.producer.async.ProducerSendThread.run(ProducerSendThread.scala:45)
[2016-06-17 06:55:10,847] ERROR fetching topic metadata for topics [Set(nil_RF2_P1)] from broker [ArrayBuffer(id:0,host:localhost,port:9092)] failed (kafka.utils.Utils$)
kafka.common.KafkaException: fetching topic metadata for topics [Set(nil_RF2_P1)] from broker [ArrayBuffer(id:0,host:localhost,port:9092)] failed
at kafka.client.ClientUtils$.fetchTopicMetadata(ClientUtils.scala:72)
at kafka.producer.BrokerPartitionInfo.updateInfo(BrokerPartitionInfo.scala:82)
at kafka.producer.async.DefaultEventHandler$$anonfun$handle$1.apply$mcV$sp(DefaultEventHandler.scala:67)
at kafka.utils.Utils$.swallow(Utils.scala:172)
at kafka.utils.Logging$class.swallowError(Logging.scala:106)
at kafka.utils.Utils$.swallowError(Utils.scala:45)
at kafka.producer.async.DefaultEventHandler.handle(DefaultEventHandler.scala:67)
at kafka.producer.async.ProducerSendThread.tryToHandle(ProducerSendThread.scala:105)
at kafka.producer.async.ProducerSendThread$$anonfun$processEvents$3.apply(ProducerSendThread.scala:88)
at kafka.producer.async.ProducerSendThread$$anonfun$processEvents$3.apply(ProducerSendThread.scala:68)
at scala.collection.immutable.Stream.foreach(Stream.scala:594)
at kafka.producer.async.ProducerSendThread.processEvents(ProducerSendThread.scala:67)
at kafka.producer.async.ProducerSendThread.run(ProducerSendThread.scala:45)
Caused by: java.nio.channels.ClosedChannelException
at kafka.network.BlockingChannel.send(BlockingChannel.scala:100)
at kafka.producer.SyncProducer.liftedTree1$1(SyncProducer.scala:73)
at kafka.producer.SyncProducer.kafka$producer$SyncProducer$$doSend(SyncProducer.scala:72)
at kafka.producer.SyncProducer.send(SyncProducer.scala:113)
at kafka.client.ClientUtils$.fetchTopicMetadata(ClientUtils.scala:58)
... 12 more
[2016-06-17 06:55:10,851] WARN Fetching topic metadata with correlation id 1 for topics [Set(nil_RF2_P1)] from broker [id:0,host:localhost,port:9092] failed (kafka.client.ClientUtils$)
java.nio.channels.ClosedChannelException
at kafka.network.BlockingChannel.send(BlockingChannel.scala:100)
at kafka.producer.SyncProducer.liftedTree1$1(SyncProducer.scala:73)
at kafka.producer.SyncProducer.kafka$producer$SyncProducer$$doSend(SyncProducer.scala:72)
at kafka.producer.SyncProducer.send(SyncProducer.scala:113)
at kafka.client.ClientUtils$.fetchTopicMetadata(ClientUtils.scala:58)
at kafka.producer.BrokerPartitionInfo.updateInfo(BrokerPartitionInfo.scala:82)
at kafka.producer.BrokerPartitionInfo.getBrokerPartitionInfo(BrokerPartitionInfo.scala:49)
at kafka.producer.async.DefaultEventHandler.kafka$producer$async$DefaultEventHandler$$getPartitionListForTopic(DefaultEventHandler.scala:186)
at kafka.producer.async.DefaultEventHandler$$anonfun$partitionAndCollate$1.apply(DefaultEventHandler.scala:150)
at kafka.producer.async.DefaultEventHandler$$anonfun$partitionAndCollate$1.apply(DefaultEventHandler.scala:149)
at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:48)
at kafka.producer.async.DefaultEventHandler.partitionAndCollate(DefaultEventHandler.scala:149)
at kafka.producer.async.DefaultEventHandler.dispatchSerializedData(DefaultEventHandler.scala:95)
at kafka.producer.async.DefaultEventHandler.handle(DefaultEventHandler.scala:72)
at kafka.producer.async.ProducerSendThread.tryToHandle(ProducerSendThread.scala:105)
at kafka.producer.async.ProducerSendThread$$anonfun$processEvents$3.apply(ProducerSendThread.scala:88)
at kafka.producer.async.ProducerSendThread$$anonfun$processEvents$3.apply(ProducerSendThread.scala:68)
at scala.collection.immutable.Stream.foreach(Stream.scala:594)
at kafka.producer.async.ProducerSendThread.processEvents(ProducerSendThread.scala:67)
at kafka.producer.async.ProducerSendThread.run(ProducerSendThread.scala:45)
Has someone faced a similar issue??
Regards

Hi Got the solution to it.
I had to change the producer.properties file for my multi node multi broker kafka cluster , with the following :
metadata.broker.list=2:172.16.30.4:9092,4:172.16.30.4:9093,1:172.16.30.5:9092,3:172.16.30.5:9093
Explanation:
We have 2 nodes
172.16.30.4
172.16.30.5
with 2 brokers running in ports
9092
9093
The preceeding integer in each ip is the myid of the zookeeper, while successing integer is port!!
Hope it would help some one.

Kafka consumer: fetching topic metadata for topics from broker [ArrayBuffer(id:0,host:user-Desktop,port:9092)] failed

Tried to run Kafka producer in one machine and consumer in another machine.
Set the following properties:
advertised.host.name
advertised.port
But getting the following error on console consumer:
bin/kafka-console-consumer.sh --zookeeper ip:2181 --topic topic
--from-beginning
[2016-01-18 16:38:00,939] WARN Fetching topic metadata with correlation id 2112 for topics [Set(topic)] from broker [id:0,host:user-Desktop,port:9092] failed (kafka.client.ClientUtils$)
java.nio.channels.ClosedChannelException
at kafka.network.BlockingChannel.send(BlockingChannel.scala:100)
at kafka.producer.SyncProducer.liftedTree1$1(SyncProducer.scala:73)
at kafka.producer.SyncProducer.kafka$producer$SyncProducer$$doSend(SyncProducer.scala:72)
at kafka.producer.SyncProducer.send(SyncProducer.scala:113)
at kafka.client.ClientUtils$.fetchTopicMetadata(ClientUtils.scala:58)
at kafka.client.ClientUtils$.fetchTopicMetadata(ClientUtils.scala:93)
at kafka.consumer.ConsumerFetcherManager$LeaderFinderThread.doWork(ConsumerFetcherManager.scala:66)
at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:60)
[2016-01-18 16:38:00,939] WARN [console-consumer-82496_gopikrishnan-B85M-D3H-A-1453114849146-e6661d41-leader-finder-thread], Failed to find leader for Set([topic,0]) (kafka.consumer.ConsumerFetcherManager$LeaderFinderThread)
kafka.common.KafkaException: fetching topic metadata for topics [Set(topic)] from broker [ArrayBuffer(id:0,host:user-Desktop,port:9092)] failed
at kafka.client.ClientUtils$.fetchTopicMetadata(ClientUtils.scala:72)
at kafka.client.ClientUtils$.fetchTopicMetadata(ClientUtils.scala:93)
at kafka.consumer.ConsumerFetcherManager$LeaderFinderThread.doWork(ConsumerFetcherManager.scala:66)
at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:60)
Caused by: java.nio.channels.ClosedChannelException
at kafka.network.BlockingChannel.send(BlockingChannel.scala:100)
at kafka.producer.SyncProducer.liftedTree1$1(SyncProducer.scala:73)
at kafka.producer.SyncProducer.kafka$producer$SyncProducer$$doSend(SyncProducer.scala:72)
at kafka.producer.SyncProducer.send(SyncProducer.scala:113)
at kafka.client.ClientUtils$.fetchTopicMetadata(ClientUtils.scala:58)
... 3 more
What has to be done to fix the issue. Thanks in advance.

Adding host address in etc/hosts fixed the issue.