Problems to consume from kafka created by docker-compose - apache-kafka

I'm trying to get the docker compose from wurstmeister/kafka-docker to work on my mint, however, I'm getting this error when I try to consume some messages from it.
WARN [console-consumer-7198_c0d552365e3b-1455224038666-5fc060cd-leader-finder-thread], Failed to find leader for Set([teste,0]) (kafka.consumer.ConsumerFetcherManager$LeaderFinderThread)
kafka.common.KafkaException: fetching topic metadata for topics [Set(teste)] from broker [ArrayBuffer(BrokerEndPoint(1001,127.0.0.1,32774))] failed
at kafka.client.ClientUtils$.fetchTopicMetadata(ClientUtils.scala:73)
at kafka.client.ClientUtils$.fetchTopicMetadata(ClientUtils.scala:94)
at kafka.consumer.ConsumerFetcherManager$LeaderFinderThread.doWork(ConsumerFetcherManager.scala:66)
at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:63)
Caused by: java.nio.channels.ClosedChannelException
at kafka.network.BlockingChannel.send(BlockingChannel.scala:110)
at kafka.producer.SyncProducer.liftedTree1$1(SyncProducer.scala:75)
at kafka.producer.SyncProducer.kafka$producer$SyncProducer$$doSend(SyncProducer.scala:74)
at kafka.producer.SyncProducer.send(SyncProducer.scala:119)
at kafka.client.ClientUtils$.fetchTopicMetadata(ClientUtils.scala:59)
... 3 more
The log generated by zookeeper is :
zookeeper_1 | 2016-02-11 21:10:30,268 [myid:] - INFO [ProcessThread(sid:0 cport:-1)::PrepRequestProcessor#645] - Got user-level KeeperException when processing sessionid:0x152d21258da000c type:create cxid:0x2 zxid:0x51 txntype:-1 reqpath:n/a Error Path:/consumers Error:KeeperErrorCode = NodeExists for /consumers
zookeeper_1 | 2016-02-11 21:10:30,483 [myid:] - INFO [ProcessThread(sid:0 cport:-1)::PrepRequestProcessor#645] - Got user-level KeeperException when processing sessionid:0x152d21258da000c type:create cxid:0x19 zxid:0x55 txntype:-1 reqpath:n/a Error Path:/consumers/console-consumer-51477/owners/teste Error:KeeperErrorCode = NoNode for /consumers/console-consumer-51477/owners/teste
zookeeper_1 | 2016-02-11 21:10:30,485 [myid:] - INFO [ProcessThread(sid:0 cport:-1)::PrepRequestProcessor#645] - Got user-level KeeperException when processing sessionid:0x152d21258da000c type:create cxid:0x1a zxid:0x56 txntype:-1 reqpath:n/a Error Path:/consumers/console-consumer-51477/owners Error:KeeperErrorCode = NoNode for /consumers/console-consumer-51477/owners
I followed this tutorial, and I was able to create a topic and produce messages to it, the problem occured when I tried to consume. I set my KAFKA_ADVERTISED_HOST_NAME to 127.0.0.1.
Could this be the problem?
Commands that I executed:
$KAFKA_HOME/bin/kafka-topics.sh --create --topic topic --partitions 4 --zookeeper $ZK --replication-factor 2
$KAFKA_HOME/bin/kafka-topics.sh --describe --topic topic --zookeeper $ZK
$KAFKA_HOME/bin/kafka-console-producer.sh --topic=topic --broker-list=`broker-list.sh`
$KAFKA_HOME/bin/kafka-console-consumer.sh --topic=topic --zookeeper=$ZK

Related

Zookeeper + BadVersion for /brokers/topics/topic_name/partitions/6/state

from zookeeper log we can see huge lines with the same error as
BadVersion for /brokers/topics/topic_name/partitions/6/state
example from the zookeeper log:
2022-03-04 03:09:23,503 [myid:3] - INFO [ProcessThread(sid:3 cport:-1)::PrepRequestProcessor#643] - Got user-level KeeperException when processing sessionid:0x27f4d8506b21199 type:setData cxid:0x25483f7 zxid:0x280109b155 txntype:-1 reqpath:n/a Error Path:/brokers/topics/my_first_car/partitions/6/state Error:KeeperErrorCode = BadVersion for /brokers/topics/my_first_car/partitions/6/state
any idea what this errors means?
other similar posts - https://zookeeper.apache.org/doc/r3.2.2/api/org/apache/zookeeper/ZooKeeper.html
Distributed state-machine's zookeeper ensemble fails while processing parallel regions with error KeeperErrorCode = BadVersion

Error when starting Kafka server in Ubuntu linux

I have been trying to setup Kafka in Ubuntu according to the quickstart guide, but I am having an issue with the Kafka server. I can start Zookeeper without problems, the problem comes when starting the server with the following command:
bin/kafka-server-start.sh config/server.properties
In the terminal of the server I get the following error:
[2019-07-09 21:30:24,997] WARN [Controller id=0, targetBrokerId=0] Error connecting to node 0.0.0.18:9092 (id: 0 rack: null) (org.apache.kafka.clients.NetworkClient)
java.net.SocketException: Invalid argument
at sun.nio.ch.Net.connect0(Native Method)
at sun.nio.ch.Net.connect(Net.java:454)
at sun.nio.ch.Net.connect(Net.java:446)
at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:648)
at org.apache.kafka.common.network.Selector.doConnect(Selector.java:278)
at org.apache.kafka.common.network.Selector.connect(Selector.java:256)
at org.apache.kafka.clients.NetworkClient.initiateConnect(NetworkClient.java:920)
at org.apache.kafka.clients.NetworkClient.ready(NetworkClient.java:287)
at org.apache.kafka.clients.NetworkClientUtils.awaitReady(NetworkClientUtils.java:65)
at kafka.controller.RequestSendThread.brokerReady(ControllerChannelManager.scala:282)
at kafka.controller.RequestSendThread.doWork(ControllerChannelManager.scala:236)
at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:82)
[2019-07-09 21:30:25,026] INFO [ExpirationReaper-0-ElectPreferredLeader]: Stopped (kafka.server.DelayedOperationPurgatory$ExpiredOperationReaper)
[2019-07-09 21:30:25,026] INFO [ExpirationReaper-0-ElectPreferredLeader]: Shutdown completed (kafka.server.DelayedOperationPurgatory$ExpiredOperationReaper)
[2019-07-09 21:30:25,029] INFO [ReplicaManager broker=0] Shut down completely (kafka.server.ReplicaManager)
In the Zookeeper terminal I get the following:
INFO Got user-level KeeperException when processing sessionid:0x100004ba9750000 type:create cxid:0x2 zxid:0x3 txntype:-1 reqpath:n/a Error Path:/brokers Error:KeeperErrorCode = NoNode for /brokers (org.apache.zookeeper.server.PrepRequestProcessor)
[2019-07-09 20:46:41,080] INFO Got user-level KeeperException when processing sessionid:0x100004ba9750000 type:create cxid:0x6 zxid:0x7 txntype:-1 reqpath:n/a Error Path:/config Error:KeeperErrorCode = NoNode for /config (org.apache.zookeeper.server.PrepRequestProcessor)
[2019-07-09 20:46:41,087] INFO Got user-level KeeperException when processing sessionid:0x100004ba9750000 type:create cxid:0x9 zxid:0xa txntype:-1 reqpath:n/a Error Path:/admin Error:KeeperErrorCode = NoNode for /admin (org.apache.zookeeper.server.PrepRequestProcessor)
[2019-07-09 20:46:41,233] INFO Got user-level KeeperException when processing sessionid:0x100004ba9750000 type:create cxid:0x15 zxid:0x15 txntype:-1 reqpath:n/a Error Path:/cluster Error:KeeperErrorCode = NoNode for /cluster (org.apache.zookeeper.server.PrepRequestProcessor)
[2019-07-09 20:46:41,803] INFO Got user-level KeeperException when processing sessionid:0x100004ba9750000 type:multi cxid:0x32 zxid:0x1c txntype:-1 reqpath:n/a aborting remaining multi ops. Error Path:/admin/preferred_replica_election Error:KeeperErrorCode = NoNode for /admin/preferred_replica_election (org.apache.zookeeper.server.PrepRequestProcessor)
[2019-07-09 21:29:00,344] WARN Unable to read additional data from client sessionid 0x100004ba9750000, likely client has closed socket (org.apache.zookeeper.server.NIOServerCnxn)
[2019-07-09 21:29:00,348] INFO Closed socket connection for client /127.0.0.1:39738 which had sessionid 0x100004ba9750000 (org.apache.zookeeper.server.NIOServerCnxn)
What could be the issue?

Not able to read kafka topic from another Host

I have two hosts A and B. A has zookeeper and kafka(0.9.0.1) installed. B has spark installed on it.
Now when im trying to read the kafka topic from A through spark streaming and persist the data in to B.
Here is the issue, when i run the spark app, it thoroughly throws the below errors continously,
17/04/20 05:22:16 ERROR ReceiverTracker: Deregistered receiver for stream 0: Error starting receiver 0 - kafka.common.ConsumerRebalanceFailedException: OlapEventsJamba_appEvent_HOSTA-1492680125896-b52bf576 can't rebalance after 4 retries
at kafka.consumer.ZookeeperConsumerConnector$ZKRebalancerListener.syncedRebalance(ZookeeperConsumerConnector.scala:660)
at kafka.consumer.ZookeeperConsumerConnector.kafka$consumer$ZookeeperConsumerConnector$$reinitializeConsumer(ZookeeperConsumerConnector.scala:967)
at kafka.consumer.ZookeeperConsumerConnector.consume(ZookeeperConsumerConnector.scala:254)
at kafka.consumer.ZookeeperConsumerConnector.createMessageStreams(ZookeeperConsumerConnector.scala:156)
at org.apache.spark.streaming.kafka.KafkaReceiver.onStart(KafkaInputDStream.scala:111)
at org.apache.spark.streaming.receiver.ReceiverSupervisor.startReceiver(ReceiverSupervisor.scala:149)
at org.apache.spark.streaming.receiver.ReceiverSupervisor.start(ReceiverSupervisor.scala:131)
at org.apache.spark.streaming.scheduler.ReceiverTracker$ReceiverTrackerEndpoint$$anonfun$9.apply(ReceiverTracker.scala:597)
at org.apache.spark.streaming.scheduler.ReceiverTracker$ReceiverTrackerEndpoint$$anonfun$9.apply(ReceiverTracker.scala:587)
at org.apache.spark.SparkContext$$anonfun$33.apply(SparkContext.scala:1979)
at org.apache.spark.SparkContext$$anonfun$33.apply(SparkContext.scala:1979)
at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:70)
at org.apache.spark.scheduler.Task.run(Task.scala:86)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:274)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
17/04/20 05:22:16 WARN TaskSetManager: Lost task 0.0 in stage 8.0 (TID 80, prd-mapr-node1.fishbowl.com): kafka.common.ConsumerRebalanceFailedException: OlapEventsJamba_appEvent_HOSTA-1492680125896-b52bf576 can't rebalance after 4 retries
at kafka.consumer.ZookeeperConsumerConnector$ZKRebalancerListener.syncedRebalance(ZookeeperConsumerConnector.scala:660)
at kafka.consumer.ZookeeperConsumerConnector.kafka$consumer$ZookeeperConsumerConnector$$reinitializeConsumer(ZookeeperConsumerConnector.scala:967)
at kafka.consumer.ZookeeperConsumerConnector.consume(ZookeeperConsumerConnector.scala:254)
at kafka.consumer.ZookeeperConsumerConnector.createMessageStreams(ZookeeperConsumerConnector.scala:156)
at org.apache.spark.streaming.kafka.KafkaReceiver.onStart(KafkaInputDStream.scala:111)
at org.apache.spark.streaming.receiver.ReceiverSupervisor.startReceiver(ReceiverSupervisor.scala:149)
at org.apache.spark.streaming.receiver.ReceiverSupervisor.start(ReceiverSupervisor.scala:131)
at org.apache.spark.streaming.scheduler.ReceiverTracker$ReceiverTrackerEndpoint$$anonfun$9.apply(ReceiverTracker.scala:597)
at org.apache.spark.streaming.scheduler.ReceiverTracker$ReceiverTrackerEndpoint$$anonfun$9.apply(ReceiverTracker.scala:587)
at org.apache.spark.SparkContext$$anonfun$33.apply(SparkContext.scala:1979)
at org.apache.spark.SparkContext$$anonfun$33.apply(SparkContext.scala:1979)
at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:70)
at org.apache.spark.scheduler.Task.run(Task.scala:86)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:274)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Below are the steps I tried to resolve the issue
set the following values in consumer.props and zookeeper.props
zookeeper.connection.timeout.ms=1000000
zookeeper.session.timeout.ms=10000
rebalance.backoff.ms=10000
Both A and B are telnet'ing and pinging.
Then, I tried to print the schema of the topic message, but the output is only
root
Spark app was creating a _success file in the output directory though.
Here is a snap of the zookeeper logs:
[2017-04-20 09:49:23,648] INFO Got user-level KeeperException when processing sessionid:0x15b8aa804290015 type:create cxid:0x79 zxid:0x1a1f txntype:-1 reqpath:n/a
Error Path:/consumers/test-group/owners/topic_test/0 Error:KeeperErrorCode = InvalidACL for /consumers/test-group/owners/topic_test/0 (org.apache.zookeeper.server.PrepRequestProcessor)
[2017-04-20 09:49:25,978] INFO Got user-level KeeperException when processing sessionid:0x15b8aa804290015 type:create cxid:0xea zxid:0x1a20 txntype:-1 reqpath:n/a
Error Path:/consumers/test-group/owners/topic_test/0 Error:KeeperErrorCode = InvalidACL for /consumers/test-group/owners/topic_test/0 (org.apache.zookeeper.server.PrepRequestProcessor)
[2017-04-20 09:49:28,320] INFO Got user-level KeeperException when processing sessionid:0x15b8aa804290015 type:create cxid:0x15b zxid:0x1a21 txntype:-1 reqpath:n/a
Error Path:/consumers/test-group/owners/topic_test/0 Error:KeeperErrorCode = InvalidACL for /consumers/test-group/owners/topic_test/0 (org.apache.zookeeper.server.PrepRequestProcessor)
[2017-04-20 09:49:30,653] INFO Got user-level KeeperException when processing sessionid:0x15b8aa804290015 type:create cxid:0x1cc zxid:0x1a22 txntype:-1 reqpath:n/a
Error Path:/consumers/test-group/owners/topic_test/0 Error:KeeperErrorCode = InvalidACL for /consumers/test-group/owners/topic_test/0 (org.apache.zookeeper.server.PrepRequestProcessor)
What can be the issue? Any help is appreciated.

Zookeeper using high CPU

I am new to zookeeper and I have a single node zookeeper setup on a system having 2 cores and 16 GB RAM. The number of active connections on zookeeper is near about 700.And it is taking 2 GB of RAM and 100% CPU(on both the cores). I am unable to find out what is causing this high CPU Utilization
My Zookeeper Config is:
# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements. See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
# The ASF licenses this file to You under the Apache License, Version 2.0
# (the "License"); you may not use this file except in compliance with
# the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
# the directory where the snapshot is stored.
dataDir=/path
# the port at which the clients will connect
clientPort=2181
# disable the per-ip limit on the number of connections since this is a non-production config
maxClientCnxns=0
and the zookeeper logs are:
[2016-12-30 08:14:47,057] INFO Client attempting to establish new session at /10.179.63.152:43480 (org.apache.zookeeper.server.ZooKeeperServer)
[2016-12-30 08:14:47,107] INFO Established session 0x1594ae64cde0188 with negotiated timeout 10000 for client /xx.xxx.xx.xxx:43480 (org.apache.zookeeper.server.ZooKeeperServer)
[2016-12-30 08:14:48,099] INFO Got user-level KeeperException when processing sessionid:0x1594ae64cde0188 type:create cxid:0x2 zxid:0xa798c81b txntype:-1 reqpath:n/a Error Path:/consumers Error:KeeperErrorCode = NodeExists for /consumers (org.apache.zookeeper.server.PrepRequestProcessor)
[2016-12-30 08:14:48,100] INFO Got user-level KeeperException when processing sessionid:0x1594ae64cde0188 type:create cxid:0x3 zxid:0xa798c81c txntype:-1 reqpath:n/a Error Path:/consumers/prod Error:KeeperErrorCode = NodeExists for /consumers/prod (org.apache.zookeeper.server.PrepRequestProcessor)
[2016-12-30 08:14:48,101] INFO Got user-level KeeperException when processing sessionid:0x1594ae64cde0188 type:create cxid:0x4 zxid:0xa798c81d txntype:-1 reqpath:n/a Error Path:/consumers/prod/ids Error:KeeperErrorCode = NodeExists for /consumers/prod/ids (org.apache.zookeeper.server.PrepRequestProcessor)
[2016-12-30 08:15:25,273] INFO Accepted socket connection from /10.185.3.226:58898 (org.apache.zookeeper.server.NIOServerCnxnFactory)
[2016-12-30 08:15:25,284] INFO Client attempting to establish new session at /xx.xxx.xx.xxx:58898 (org.apache.zookeeper.server.ZooKeeperServer)
[2016-12-30 08:15:25,347] INFO Established session 0x1594ae64cde0189 with negotiated timeout 10000 for client /xx.xxx.xx.xxx:58898 (org.apache.zookeeper.server.ZooKeeperServer)
[2016-12-30 08:15:26,762] INFO Got user-level KeeperException when processing sessionid:0x1594ae64cde0189 type:create cxid:0x2 zxid:0xa798cb43 txntype:-1 reqpath:n/a Error Path:/consumers Error:KeeperErrorCode = NodeExists for /consumers (org.apache.zookeeper.server.PrepRequestProcessor)
[2016-12-30 08:15:26,762] INFO Got user-level KeeperException when processing sessionid:0x1594ae64cde0189 type:create cxid:0x3 zxid:0xa798cb44 txntype:-1 reqpath:n/a Error Path:/consumers/prod Error:KeeperErrorCode = NodeExists for /consumers/prod (org.apache.zookeeper.server.PrepRequestProcessor)
[2016-12-30 08:15:26,763] INFO Got user-level KeeperException when processing sessionid:0x1594ae64cde0189 type:create cxid:0x4 zxid:0xa798cb45 txntype:-1 reqpath:n/a Error Path:/consumers/prod/ids Error:KeeperErrorCode = NodeExists for /consumers/prod/ids (org.apache.zookeeper.server.PrepRequestProcessor)
[2016-12-30 08:15:56,786] INFO Accepted socket connection from /xx.xxx.xx.xxx:43574 (org.apache.zookeeper.server.NIOServerCnxnFactory)
[2016-12-30 08:15:56,790] INFO Client attempting to establish new session at /xx.xxx.xx.xxx:43574 (org.apache.zookeeper.server.ZooKeeperServer)
[2016-12-30 08:15:56,843] INFO Established session 0x1594ae64cde018a with negotiated timeout 10000 for client /10.185.178.18:43574 (org.apache.zookeeper.server.ZooKeeperServer)
[2016-12-30 08:15:57,743] INFO Got user-level KeeperException when processing sessionid:0x1594ae64cde018a type:create cxid:0x2 zxid:0xa798db9d txntype:-1 reqpath:n/a Error Path:/consumers Error:KeeperErrorCode = NodeExists for /consumers (org.apache.zookeeper.server.PrepRequestProcessor)
[2016-12-30 08:15:57,743] INFO Got user-level KeeperException when processing sessionid:0x1594ae64cde018a type:create cxid:0x3 zxid:0xa798db9e txntype:-1 reqpath:n/a Error Path:/consumers/prod Error:KeeperErrorCode = NodeExists for /consumers/prod (org.apache.zookeeper.server.PrepRequestProcessor)
[2016-12-30 08:15:57,744] INFO Got user-level KeeperException when processing sessionid:0x1594ae64cde018a type:create cxid:0x4 zxid:0xa798db9f txntype:-1 reqpath:n/a Error Path:/consumers/prod/ids Error:KeeperErrorCode = NodeExists for /consumers/prod/ids (org.apache.zookeeper.server.PrepRequestProcessor)
Can someone help me identify the problem .

KeeperErrorCode = NoNode for /admin/preferred_replica_election

when i starting kafka,zookeeper happens error.
INFO Got user-level KeeperException when processing sessionid:0x156028651c00001 type:delete cxid:0x1b zxid:0x59 txntype:-1 reqpath:n/a Error Path:/admin/preferred_replica_election Error:KeeperErrorCode = NoNode for /admin/preferred_replica_election (org.apache.zookeeper.server.PrepRequestProcessor)
Please be informed that this is not an error it is just a log warning on zookeeper .Howeaver the kafka-producer will not be able to produce messages to the topic on the first go.But will be on the subsequent trials.......while using a kafka-node library.
Are you running on single node with replication 1? Then it should not bother you. You can read more about it here https://cwiki.apache.org/confluence/display/KAFKA/Replication+tools
Exceptions can also be seen for znode path /brokers/ids or /brokers because they are not present(on zookeeper) initially when you start the kafka-server.