when i starting kafka,zookeeper happens error.
INFO Got user-level KeeperException when processing sessionid:0x156028651c00001 type:delete cxid:0x1b zxid:0x59 txntype:-1 reqpath:n/a Error Path:/admin/preferred_replica_election Error:KeeperErrorCode = NoNode for /admin/preferred_replica_election (org.apache.zookeeper.server.PrepRequestProcessor)
Please be informed that this is not an error it is just a log warning on zookeeper .Howeaver the kafka-producer will not be able to produce messages to the topic on the first go.But will be on the subsequent trials.......while using a kafka-node library.
Are you running on single node with replication 1? Then it should not bother you. You can read more about it here https://cwiki.apache.org/confluence/display/KAFKA/Replication+tools
Exceptions can also be seen for znode path /brokers/ids or /brokers because they are not present(on zookeeper) initially when you start the kafka-server.
Related
from zookeeper log we can see huge lines with the same error as
BadVersion for /brokers/topics/topic_name/partitions/6/state
example from the zookeeper log:
2022-03-04 03:09:23,503 [myid:3] - INFO [ProcessThread(sid:3 cport:-1)::PrepRequestProcessor#643] - Got user-level KeeperException when processing sessionid:0x27f4d8506b21199 type:setData cxid:0x25483f7 zxid:0x280109b155 txntype:-1 reqpath:n/a Error Path:/brokers/topics/my_first_car/partitions/6/state Error:KeeperErrorCode = BadVersion for /brokers/topics/my_first_car/partitions/6/state
any idea what this errors means?
other similar posts - https://zookeeper.apache.org/doc/r3.2.2/api/org/apache/zookeeper/ZooKeeper.html
Distributed state-machine's zookeeper ensemble fails while processing parallel regions with error KeeperErrorCode = BadVersion
I have been trying to setup Kafka in Ubuntu according to the quickstart guide, but I am having an issue with the Kafka server. I can start Zookeeper without problems, the problem comes when starting the server with the following command:
bin/kafka-server-start.sh config/server.properties
In the terminal of the server I get the following error:
[2019-07-09 21:30:24,997] WARN [Controller id=0, targetBrokerId=0] Error connecting to node 0.0.0.18:9092 (id: 0 rack: null) (org.apache.kafka.clients.NetworkClient)
java.net.SocketException: Invalid argument
at sun.nio.ch.Net.connect0(Native Method)
at sun.nio.ch.Net.connect(Net.java:454)
at sun.nio.ch.Net.connect(Net.java:446)
at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:648)
at org.apache.kafka.common.network.Selector.doConnect(Selector.java:278)
at org.apache.kafka.common.network.Selector.connect(Selector.java:256)
at org.apache.kafka.clients.NetworkClient.initiateConnect(NetworkClient.java:920)
at org.apache.kafka.clients.NetworkClient.ready(NetworkClient.java:287)
at org.apache.kafka.clients.NetworkClientUtils.awaitReady(NetworkClientUtils.java:65)
at kafka.controller.RequestSendThread.brokerReady(ControllerChannelManager.scala:282)
at kafka.controller.RequestSendThread.doWork(ControllerChannelManager.scala:236)
at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:82)
[2019-07-09 21:30:25,026] INFO [ExpirationReaper-0-ElectPreferredLeader]: Stopped (kafka.server.DelayedOperationPurgatory$ExpiredOperationReaper)
[2019-07-09 21:30:25,026] INFO [ExpirationReaper-0-ElectPreferredLeader]: Shutdown completed (kafka.server.DelayedOperationPurgatory$ExpiredOperationReaper)
[2019-07-09 21:30:25,029] INFO [ReplicaManager broker=0] Shut down completely (kafka.server.ReplicaManager)
In the Zookeeper terminal I get the following:
INFO Got user-level KeeperException when processing sessionid:0x100004ba9750000 type:create cxid:0x2 zxid:0x3 txntype:-1 reqpath:n/a Error Path:/brokers Error:KeeperErrorCode = NoNode for /brokers (org.apache.zookeeper.server.PrepRequestProcessor)
[2019-07-09 20:46:41,080] INFO Got user-level KeeperException when processing sessionid:0x100004ba9750000 type:create cxid:0x6 zxid:0x7 txntype:-1 reqpath:n/a Error Path:/config Error:KeeperErrorCode = NoNode for /config (org.apache.zookeeper.server.PrepRequestProcessor)
[2019-07-09 20:46:41,087] INFO Got user-level KeeperException when processing sessionid:0x100004ba9750000 type:create cxid:0x9 zxid:0xa txntype:-1 reqpath:n/a Error Path:/admin Error:KeeperErrorCode = NoNode for /admin (org.apache.zookeeper.server.PrepRequestProcessor)
[2019-07-09 20:46:41,233] INFO Got user-level KeeperException when processing sessionid:0x100004ba9750000 type:create cxid:0x15 zxid:0x15 txntype:-1 reqpath:n/a Error Path:/cluster Error:KeeperErrorCode = NoNode for /cluster (org.apache.zookeeper.server.PrepRequestProcessor)
[2019-07-09 20:46:41,803] INFO Got user-level KeeperException when processing sessionid:0x100004ba9750000 type:multi cxid:0x32 zxid:0x1c txntype:-1 reqpath:n/a aborting remaining multi ops. Error Path:/admin/preferred_replica_election Error:KeeperErrorCode = NoNode for /admin/preferred_replica_election (org.apache.zookeeper.server.PrepRequestProcessor)
[2019-07-09 21:29:00,344] WARN Unable to read additional data from client sessionid 0x100004ba9750000, likely client has closed socket (org.apache.zookeeper.server.NIOServerCnxn)
[2019-07-09 21:29:00,348] INFO Closed socket connection for client /127.0.0.1:39738 which had sessionid 0x100004ba9750000 (org.apache.zookeeper.server.NIOServerCnxn)
What could be the issue?
Thanks in advance. Please help me to resolve this below mentioned kafka error.
00000000000000.txnindex and rebuilding index... (kafka.log.Log)
[2018-09-25 12:48:05,462] ERROR There was an error in one of the threads during logs loading: java.lang.InternalError: a fault occurred in a recent unsafe memory access operation in compiled Java code (kafka.log.LogManager)
[2018-09-25 12:48:05,469] FATAL [KafkaServer id=0] Fatal error during KafkaServer startup. Prepare to shutdown (kafka.server.KafkaServer)
java.lang.InternalError: a fault occurred in a recent unsafe memory access operation in compiled Java code
at java.nio.HeapByteBuffer.<init>(HeapByteBuffer.java:57)
at java.nio.ByteBuffer.allocate(ByteBuffer.java:335)
at org.apache.kafka.common.record.FileLogInputStream$FileChannelRecordBatch.loadBatchWithSize(FileLogInputStream.java:209)
at org.apache.kafka.common.record.FileLogInputStream$FileChannelRecordBatch.loadFullBatch(FileLogInputStream.java:192)
at org.apache.kafka.common.record.FileLogInputStream$FileChannelRecordBatch.ensureValid(FileLogInputStream.java:164)
at kafka.log.LogSegment$$anonfun$recover$1.apply(LogSegment.scala:263)
at kafka.log.LogSegment$$anonfun$recover$1.apply(LogSegment.scala:262)
at scala.collection.Iterator$class.foreach(Iterator.scala:891)
at scala.collection.AbstractIterator.foreach(Iterator.scala:1334)
My guess is that your Kafka log directory (log.dirs in server.properties) is out of memory. When Kafka attempts to rebuild the index, there is insufficient memory and therefore Kafka broker cannot be started.
Assuming log.dirs=/var/lib/kafka
df -hT /var/lib/kafka
will display storage usage for your log directory.
I am new to zookeeper and I have a single node zookeeper setup on a system having 2 cores and 16 GB RAM. The number of active connections on zookeeper is near about 700.And it is taking 2 GB of RAM and 100% CPU(on both the cores). I am unable to find out what is causing this high CPU Utilization
My Zookeeper Config is:
# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements. See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
# The ASF licenses this file to You under the Apache License, Version 2.0
# (the "License"); you may not use this file except in compliance with
# the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
# the directory where the snapshot is stored.
dataDir=/path
# the port at which the clients will connect
clientPort=2181
# disable the per-ip limit on the number of connections since this is a non-production config
maxClientCnxns=0
and the zookeeper logs are:
[2016-12-30 08:14:47,057] INFO Client attempting to establish new session at /10.179.63.152:43480 (org.apache.zookeeper.server.ZooKeeperServer)
[2016-12-30 08:14:47,107] INFO Established session 0x1594ae64cde0188 with negotiated timeout 10000 for client /xx.xxx.xx.xxx:43480 (org.apache.zookeeper.server.ZooKeeperServer)
[2016-12-30 08:14:48,099] INFO Got user-level KeeperException when processing sessionid:0x1594ae64cde0188 type:create cxid:0x2 zxid:0xa798c81b txntype:-1 reqpath:n/a Error Path:/consumers Error:KeeperErrorCode = NodeExists for /consumers (org.apache.zookeeper.server.PrepRequestProcessor)
[2016-12-30 08:14:48,100] INFO Got user-level KeeperException when processing sessionid:0x1594ae64cde0188 type:create cxid:0x3 zxid:0xa798c81c txntype:-1 reqpath:n/a Error Path:/consumers/prod Error:KeeperErrorCode = NodeExists for /consumers/prod (org.apache.zookeeper.server.PrepRequestProcessor)
[2016-12-30 08:14:48,101] INFO Got user-level KeeperException when processing sessionid:0x1594ae64cde0188 type:create cxid:0x4 zxid:0xa798c81d txntype:-1 reqpath:n/a Error Path:/consumers/prod/ids Error:KeeperErrorCode = NodeExists for /consumers/prod/ids (org.apache.zookeeper.server.PrepRequestProcessor)
[2016-12-30 08:15:25,273] INFO Accepted socket connection from /10.185.3.226:58898 (org.apache.zookeeper.server.NIOServerCnxnFactory)
[2016-12-30 08:15:25,284] INFO Client attempting to establish new session at /xx.xxx.xx.xxx:58898 (org.apache.zookeeper.server.ZooKeeperServer)
[2016-12-30 08:15:25,347] INFO Established session 0x1594ae64cde0189 with negotiated timeout 10000 for client /xx.xxx.xx.xxx:58898 (org.apache.zookeeper.server.ZooKeeperServer)
[2016-12-30 08:15:26,762] INFO Got user-level KeeperException when processing sessionid:0x1594ae64cde0189 type:create cxid:0x2 zxid:0xa798cb43 txntype:-1 reqpath:n/a Error Path:/consumers Error:KeeperErrorCode = NodeExists for /consumers (org.apache.zookeeper.server.PrepRequestProcessor)
[2016-12-30 08:15:26,762] INFO Got user-level KeeperException when processing sessionid:0x1594ae64cde0189 type:create cxid:0x3 zxid:0xa798cb44 txntype:-1 reqpath:n/a Error Path:/consumers/prod Error:KeeperErrorCode = NodeExists for /consumers/prod (org.apache.zookeeper.server.PrepRequestProcessor)
[2016-12-30 08:15:26,763] INFO Got user-level KeeperException when processing sessionid:0x1594ae64cde0189 type:create cxid:0x4 zxid:0xa798cb45 txntype:-1 reqpath:n/a Error Path:/consumers/prod/ids Error:KeeperErrorCode = NodeExists for /consumers/prod/ids (org.apache.zookeeper.server.PrepRequestProcessor)
[2016-12-30 08:15:56,786] INFO Accepted socket connection from /xx.xxx.xx.xxx:43574 (org.apache.zookeeper.server.NIOServerCnxnFactory)
[2016-12-30 08:15:56,790] INFO Client attempting to establish new session at /xx.xxx.xx.xxx:43574 (org.apache.zookeeper.server.ZooKeeperServer)
[2016-12-30 08:15:56,843] INFO Established session 0x1594ae64cde018a with negotiated timeout 10000 for client /10.185.178.18:43574 (org.apache.zookeeper.server.ZooKeeperServer)
[2016-12-30 08:15:57,743] INFO Got user-level KeeperException when processing sessionid:0x1594ae64cde018a type:create cxid:0x2 zxid:0xa798db9d txntype:-1 reqpath:n/a Error Path:/consumers Error:KeeperErrorCode = NodeExists for /consumers (org.apache.zookeeper.server.PrepRequestProcessor)
[2016-12-30 08:15:57,743] INFO Got user-level KeeperException when processing sessionid:0x1594ae64cde018a type:create cxid:0x3 zxid:0xa798db9e txntype:-1 reqpath:n/a Error Path:/consumers/prod Error:KeeperErrorCode = NodeExists for /consumers/prod (org.apache.zookeeper.server.PrepRequestProcessor)
[2016-12-30 08:15:57,744] INFO Got user-level KeeperException when processing sessionid:0x1594ae64cde018a type:create cxid:0x4 zxid:0xa798db9f txntype:-1 reqpath:n/a Error Path:/consumers/prod/ids Error:KeeperErrorCode = NodeExists for /consumers/prod/ids (org.apache.zookeeper.server.PrepRequestProcessor)
Can someone help me identify the problem .
I'm trying to get the docker compose from wurstmeister/kafka-docker to work on my mint, however, I'm getting this error when I try to consume some messages from it.
WARN [console-consumer-7198_c0d552365e3b-1455224038666-5fc060cd-leader-finder-thread], Failed to find leader for Set([teste,0]) (kafka.consumer.ConsumerFetcherManager$LeaderFinderThread)
kafka.common.KafkaException: fetching topic metadata for topics [Set(teste)] from broker [ArrayBuffer(BrokerEndPoint(1001,127.0.0.1,32774))] failed
at kafka.client.ClientUtils$.fetchTopicMetadata(ClientUtils.scala:73)
at kafka.client.ClientUtils$.fetchTopicMetadata(ClientUtils.scala:94)
at kafka.consumer.ConsumerFetcherManager$LeaderFinderThread.doWork(ConsumerFetcherManager.scala:66)
at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:63)
Caused by: java.nio.channels.ClosedChannelException
at kafka.network.BlockingChannel.send(BlockingChannel.scala:110)
at kafka.producer.SyncProducer.liftedTree1$1(SyncProducer.scala:75)
at kafka.producer.SyncProducer.kafka$producer$SyncProducer$$doSend(SyncProducer.scala:74)
at kafka.producer.SyncProducer.send(SyncProducer.scala:119)
at kafka.client.ClientUtils$.fetchTopicMetadata(ClientUtils.scala:59)
... 3 more
The log generated by zookeeper is :
zookeeper_1 | 2016-02-11 21:10:30,268 [myid:] - INFO [ProcessThread(sid:0 cport:-1)::PrepRequestProcessor#645] - Got user-level KeeperException when processing sessionid:0x152d21258da000c type:create cxid:0x2 zxid:0x51 txntype:-1 reqpath:n/a Error Path:/consumers Error:KeeperErrorCode = NodeExists for /consumers
zookeeper_1 | 2016-02-11 21:10:30,483 [myid:] - INFO [ProcessThread(sid:0 cport:-1)::PrepRequestProcessor#645] - Got user-level KeeperException when processing sessionid:0x152d21258da000c type:create cxid:0x19 zxid:0x55 txntype:-1 reqpath:n/a Error Path:/consumers/console-consumer-51477/owners/teste Error:KeeperErrorCode = NoNode for /consumers/console-consumer-51477/owners/teste
zookeeper_1 | 2016-02-11 21:10:30,485 [myid:] - INFO [ProcessThread(sid:0 cport:-1)::PrepRequestProcessor#645] - Got user-level KeeperException when processing sessionid:0x152d21258da000c type:create cxid:0x1a zxid:0x56 txntype:-1 reqpath:n/a Error Path:/consumers/console-consumer-51477/owners Error:KeeperErrorCode = NoNode for /consumers/console-consumer-51477/owners
I followed this tutorial, and I was able to create a topic and produce messages to it, the problem occured when I tried to consume. I set my KAFKA_ADVERTISED_HOST_NAME to 127.0.0.1.
Could this be the problem?
Commands that I executed:
$KAFKA_HOME/bin/kafka-topics.sh --create --topic topic --partitions 4 --zookeeper $ZK --replication-factor 2
$KAFKA_HOME/bin/kafka-topics.sh --describe --topic topic --zookeeper $ZK
$KAFKA_HOME/bin/kafka-console-producer.sh --topic=topic --broker-list=`broker-list.sh`
$KAFKA_HOME/bin/kafka-console-consumer.sh --topic=topic --zookeeper=$ZK