from zookeeper log we can see huge lines with the same error as
BadVersion for /brokers/topics/topic_name/partitions/6/state
example from the zookeeper log:
2022-03-04 03:09:23,503 [myid:3] - INFO [ProcessThread(sid:3 cport:-1)::PrepRequestProcessor#643] - Got user-level KeeperException when processing sessionid:0x27f4d8506b21199 type:setData cxid:0x25483f7 zxid:0x280109b155 txntype:-1 reqpath:n/a Error Path:/brokers/topics/my_first_car/partitions/6/state Error:KeeperErrorCode = BadVersion for /brokers/topics/my_first_car/partitions/6/state
any idea what this errors means?
other similar posts - https://zookeeper.apache.org/doc/r3.2.2/api/org/apache/zookeeper/ZooKeeper.html
Distributed state-machine's zookeeper ensemble fails while processing parallel regions with error KeeperErrorCode = BadVersion
Related
We are running Hyperledger fabric network with Kafka and zookeeper in production using docker swarm on Azure VM (4 Kafka node, 3 zookeeper nodes) it was running fine but just 2 days back suddenly zookeeper had a restart, after that there's continuous restart on zookeeper having time interval of 6-8 hours.
logs on Kafka node
[2020-07-04 07:48:53,492] INFO [ReplicaFetcher replicaId=2, leaderId=1, fetcherId=0] Stopped (kafka.server.ReplicaFetcherThread)
[2020-07-04 07:48:53,492] INFO [ReplicaFetcher replicaId=2, leaderId=1, fetcherId=0] Shutdown completed (kafka.server.ReplicaFetcherThread)
[2020-07-04 07:48:53,499] INFO [ReplicaFetcherManager on broker 2] Removed fetcher for partitions xxxx-xxxxx-xxx-xxxxx.
zookeeper leader logs
2020-07-04 07:46:27,070 [myid:3] - INFO [ProcessThread(sid:3 cport:-1)::PrepRequestProcessor#653] - Got user-level KeeperException when processing sessionid:0x10101beb22c0000 type:create cxid:0x4 zxid:0x2e00000114 txntype:-1 reqpath:n/a Error Path:/brokers/ids Error:KeeperErrorCode = NodeExists for /brokers/ids
2020-07-04 07:48:43,084 [myid:3] - INFO [SessionTracker:ZooKeeperServer#355] - Expiring session 0x2010551ef290000, timeout of 6000ms exceeded
2020-07-04 07:48:43,085 [myid:3] - INFO [ProcessThread(sid:3 cport:-1)::PrepRequestProcessor#487] - Processed session termination for sessionid: 0x2010551ef290000
2020-07-04 07:48:43,091 [myid:3] - INFO [CommitProcessor:3:NIOServerCnxn#1056] - Closed socket connection for client /100.0.20.80:60672 which had sessionid 0x2010551ef290000
2020-07-04 07:48:55,182 [myid:3] - ERROR [LearnerHandler-/100.0.20.80:58940:LearnerHandler#648] - Unexpected exception causing shutdown while sock still open
java.net.SocketTimeoutException: Read timed out
at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
at java.net.SocketInputStream.read(SocketInputStream.java:171)
at java.net.SocketInputStream.read(SocketInputStream.java:141)
at java.io.BufferedInputStream.fill(BufferedInputStream.java:246)
at java.io.BufferedInputStream.read(BufferedInputStream.java:265)
at java.io.DataInputStream.readInt(DataInputStream.java:387)
at org.apache.jute.BinaryInputArchive.readInt(BinaryInputArchive.java:63)
at org.apache.zookeeper.server.quorum.QuorumPacket.deserialize(QuorumPacket.java:85)
at org.apache.jute.BinaryInputArchive.readRecord(BinaryInputArchive.java:99)
at org.apache.zookeeper.server.quorum.LearnerHandler.run(LearnerHandler.java:559)
2020-07-04 07:48:55,183 [myid:3] - WARN [LearnerHandler-/100.0.20.80:58940:LearnerHandler#661] - ******* GOODBYE /100.0.20.80:58940 ********
2020-07-04 07:49:57,623 [myid:3] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory#222] - Accepted socket connection from /100.0.20.80:37838
2020-07-04 07:49:57,637 [myid:3] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:ZooKeeperServer#949] - Client attempting to establish new session at /100.0.20.80:37838
2020-07-04 07:49:57,641 [myid:3] - INFO [CommitProcessor:3:ZooKeeperServer#694] - Established session 0x300ed4720900000 with negotiated timeout 12000 for client /100.0.20.80:37838
2020-07-04 07:49:57,670 [myid:3] - INFO [ProcessThread(sid:3 cport:-1)::PrepRequestProcessor#653] - Got user-level KeeperException when processing sessionid:0x300ed4720900000 type:setData cxid:0x1 zxid:0x2e000003b2 txntype:-1 reqpath:n/a Error Path:/brokers/topics/xxxxxxxxxxxx/partitions/0/state Error:KeeperErrorCode = BadVersion for /brokers/topics/xxxxxxxxxxxx/partitions/0/state
my zoo.cfg
clientPort=2181
dataDir=/data
dataLogDir=/datalog
tickTime=6000
initLimit=10
syncLimit=2
autopurge.snapRetainCount=3
autopurge.purgeInterval=1
server.1=xxx.xxx.com:2888:3888
server.2=xxx.xxx.com:2888:3888
server.3=0.0.0.0:2888:3888
I have been trying to setup Kafka in Ubuntu according to the quickstart guide, but I am having an issue with the Kafka server. I can start Zookeeper without problems, the problem comes when starting the server with the following command:
bin/kafka-server-start.sh config/server.properties
In the terminal of the server I get the following error:
[2019-07-09 21:30:24,997] WARN [Controller id=0, targetBrokerId=0] Error connecting to node 0.0.0.18:9092 (id: 0 rack: null) (org.apache.kafka.clients.NetworkClient)
java.net.SocketException: Invalid argument
at sun.nio.ch.Net.connect0(Native Method)
at sun.nio.ch.Net.connect(Net.java:454)
at sun.nio.ch.Net.connect(Net.java:446)
at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:648)
at org.apache.kafka.common.network.Selector.doConnect(Selector.java:278)
at org.apache.kafka.common.network.Selector.connect(Selector.java:256)
at org.apache.kafka.clients.NetworkClient.initiateConnect(NetworkClient.java:920)
at org.apache.kafka.clients.NetworkClient.ready(NetworkClient.java:287)
at org.apache.kafka.clients.NetworkClientUtils.awaitReady(NetworkClientUtils.java:65)
at kafka.controller.RequestSendThread.brokerReady(ControllerChannelManager.scala:282)
at kafka.controller.RequestSendThread.doWork(ControllerChannelManager.scala:236)
at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:82)
[2019-07-09 21:30:25,026] INFO [ExpirationReaper-0-ElectPreferredLeader]: Stopped (kafka.server.DelayedOperationPurgatory$ExpiredOperationReaper)
[2019-07-09 21:30:25,026] INFO [ExpirationReaper-0-ElectPreferredLeader]: Shutdown completed (kafka.server.DelayedOperationPurgatory$ExpiredOperationReaper)
[2019-07-09 21:30:25,029] INFO [ReplicaManager broker=0] Shut down completely (kafka.server.ReplicaManager)
In the Zookeeper terminal I get the following:
INFO Got user-level KeeperException when processing sessionid:0x100004ba9750000 type:create cxid:0x2 zxid:0x3 txntype:-1 reqpath:n/a Error Path:/brokers Error:KeeperErrorCode = NoNode for /brokers (org.apache.zookeeper.server.PrepRequestProcessor)
[2019-07-09 20:46:41,080] INFO Got user-level KeeperException when processing sessionid:0x100004ba9750000 type:create cxid:0x6 zxid:0x7 txntype:-1 reqpath:n/a Error Path:/config Error:KeeperErrorCode = NoNode for /config (org.apache.zookeeper.server.PrepRequestProcessor)
[2019-07-09 20:46:41,087] INFO Got user-level KeeperException when processing sessionid:0x100004ba9750000 type:create cxid:0x9 zxid:0xa txntype:-1 reqpath:n/a Error Path:/admin Error:KeeperErrorCode = NoNode for /admin (org.apache.zookeeper.server.PrepRequestProcessor)
[2019-07-09 20:46:41,233] INFO Got user-level KeeperException when processing sessionid:0x100004ba9750000 type:create cxid:0x15 zxid:0x15 txntype:-1 reqpath:n/a Error Path:/cluster Error:KeeperErrorCode = NoNode for /cluster (org.apache.zookeeper.server.PrepRequestProcessor)
[2019-07-09 20:46:41,803] INFO Got user-level KeeperException when processing sessionid:0x100004ba9750000 type:multi cxid:0x32 zxid:0x1c txntype:-1 reqpath:n/a aborting remaining multi ops. Error Path:/admin/preferred_replica_election Error:KeeperErrorCode = NoNode for /admin/preferred_replica_election (org.apache.zookeeper.server.PrepRequestProcessor)
[2019-07-09 21:29:00,344] WARN Unable to read additional data from client sessionid 0x100004ba9750000, likely client has closed socket (org.apache.zookeeper.server.NIOServerCnxn)
[2019-07-09 21:29:00,348] INFO Closed socket connection for client /127.0.0.1:39738 which had sessionid 0x100004ba9750000 (org.apache.zookeeper.server.NIOServerCnxn)
What could be the issue?
I am new to zookeeper and I have a single node zookeeper setup on a system having 2 cores and 16 GB RAM. The number of active connections on zookeeper is near about 700.And it is taking 2 GB of RAM and 100% CPU(on both the cores). I am unable to find out what is causing this high CPU Utilization
My Zookeeper Config is:
# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements. See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
# The ASF licenses this file to You under the Apache License, Version 2.0
# (the "License"); you may not use this file except in compliance with
# the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
# the directory where the snapshot is stored.
dataDir=/path
# the port at which the clients will connect
clientPort=2181
# disable the per-ip limit on the number of connections since this is a non-production config
maxClientCnxns=0
and the zookeeper logs are:
[2016-12-30 08:14:47,057] INFO Client attempting to establish new session at /10.179.63.152:43480 (org.apache.zookeeper.server.ZooKeeperServer)
[2016-12-30 08:14:47,107] INFO Established session 0x1594ae64cde0188 with negotiated timeout 10000 for client /xx.xxx.xx.xxx:43480 (org.apache.zookeeper.server.ZooKeeperServer)
[2016-12-30 08:14:48,099] INFO Got user-level KeeperException when processing sessionid:0x1594ae64cde0188 type:create cxid:0x2 zxid:0xa798c81b txntype:-1 reqpath:n/a Error Path:/consumers Error:KeeperErrorCode = NodeExists for /consumers (org.apache.zookeeper.server.PrepRequestProcessor)
[2016-12-30 08:14:48,100] INFO Got user-level KeeperException when processing sessionid:0x1594ae64cde0188 type:create cxid:0x3 zxid:0xa798c81c txntype:-1 reqpath:n/a Error Path:/consumers/prod Error:KeeperErrorCode = NodeExists for /consumers/prod (org.apache.zookeeper.server.PrepRequestProcessor)
[2016-12-30 08:14:48,101] INFO Got user-level KeeperException when processing sessionid:0x1594ae64cde0188 type:create cxid:0x4 zxid:0xa798c81d txntype:-1 reqpath:n/a Error Path:/consumers/prod/ids Error:KeeperErrorCode = NodeExists for /consumers/prod/ids (org.apache.zookeeper.server.PrepRequestProcessor)
[2016-12-30 08:15:25,273] INFO Accepted socket connection from /10.185.3.226:58898 (org.apache.zookeeper.server.NIOServerCnxnFactory)
[2016-12-30 08:15:25,284] INFO Client attempting to establish new session at /xx.xxx.xx.xxx:58898 (org.apache.zookeeper.server.ZooKeeperServer)
[2016-12-30 08:15:25,347] INFO Established session 0x1594ae64cde0189 with negotiated timeout 10000 for client /xx.xxx.xx.xxx:58898 (org.apache.zookeeper.server.ZooKeeperServer)
[2016-12-30 08:15:26,762] INFO Got user-level KeeperException when processing sessionid:0x1594ae64cde0189 type:create cxid:0x2 zxid:0xa798cb43 txntype:-1 reqpath:n/a Error Path:/consumers Error:KeeperErrorCode = NodeExists for /consumers (org.apache.zookeeper.server.PrepRequestProcessor)
[2016-12-30 08:15:26,762] INFO Got user-level KeeperException when processing sessionid:0x1594ae64cde0189 type:create cxid:0x3 zxid:0xa798cb44 txntype:-1 reqpath:n/a Error Path:/consumers/prod Error:KeeperErrorCode = NodeExists for /consumers/prod (org.apache.zookeeper.server.PrepRequestProcessor)
[2016-12-30 08:15:26,763] INFO Got user-level KeeperException when processing sessionid:0x1594ae64cde0189 type:create cxid:0x4 zxid:0xa798cb45 txntype:-1 reqpath:n/a Error Path:/consumers/prod/ids Error:KeeperErrorCode = NodeExists for /consumers/prod/ids (org.apache.zookeeper.server.PrepRequestProcessor)
[2016-12-30 08:15:56,786] INFO Accepted socket connection from /xx.xxx.xx.xxx:43574 (org.apache.zookeeper.server.NIOServerCnxnFactory)
[2016-12-30 08:15:56,790] INFO Client attempting to establish new session at /xx.xxx.xx.xxx:43574 (org.apache.zookeeper.server.ZooKeeperServer)
[2016-12-30 08:15:56,843] INFO Established session 0x1594ae64cde018a with negotiated timeout 10000 for client /10.185.178.18:43574 (org.apache.zookeeper.server.ZooKeeperServer)
[2016-12-30 08:15:57,743] INFO Got user-level KeeperException when processing sessionid:0x1594ae64cde018a type:create cxid:0x2 zxid:0xa798db9d txntype:-1 reqpath:n/a Error Path:/consumers Error:KeeperErrorCode = NodeExists for /consumers (org.apache.zookeeper.server.PrepRequestProcessor)
[2016-12-30 08:15:57,743] INFO Got user-level KeeperException when processing sessionid:0x1594ae64cde018a type:create cxid:0x3 zxid:0xa798db9e txntype:-1 reqpath:n/a Error Path:/consumers/prod Error:KeeperErrorCode = NodeExists for /consumers/prod (org.apache.zookeeper.server.PrepRequestProcessor)
[2016-12-30 08:15:57,744] INFO Got user-level KeeperException when processing sessionid:0x1594ae64cde018a type:create cxid:0x4 zxid:0xa798db9f txntype:-1 reqpath:n/a Error Path:/consumers/prod/ids Error:KeeperErrorCode = NodeExists for /consumers/prod/ids (org.apache.zookeeper.server.PrepRequestProcessor)
Can someone help me identify the problem .
when i starting kafka,zookeeper happens error.
INFO Got user-level KeeperException when processing sessionid:0x156028651c00001 type:delete cxid:0x1b zxid:0x59 txntype:-1 reqpath:n/a Error Path:/admin/preferred_replica_election Error:KeeperErrorCode = NoNode for /admin/preferred_replica_election (org.apache.zookeeper.server.PrepRequestProcessor)
Please be informed that this is not an error it is just a log warning on zookeeper .Howeaver the kafka-producer will not be able to produce messages to the topic on the first go.But will be on the subsequent trials.......while using a kafka-node library.
Are you running on single node with replication 1? Then it should not bother you. You can read more about it here https://cwiki.apache.org/confluence/display/KAFKA/Replication+tools
Exceptions can also be seen for znode path /brokers/ids or /brokers because they are not present(on zookeeper) initially when you start the kafka-server.
I'm trying to get the docker compose from wurstmeister/kafka-docker to work on my mint, however, I'm getting this error when I try to consume some messages from it.
WARN [console-consumer-7198_c0d552365e3b-1455224038666-5fc060cd-leader-finder-thread], Failed to find leader for Set([teste,0]) (kafka.consumer.ConsumerFetcherManager$LeaderFinderThread)
kafka.common.KafkaException: fetching topic metadata for topics [Set(teste)] from broker [ArrayBuffer(BrokerEndPoint(1001,127.0.0.1,32774))] failed
at kafka.client.ClientUtils$.fetchTopicMetadata(ClientUtils.scala:73)
at kafka.client.ClientUtils$.fetchTopicMetadata(ClientUtils.scala:94)
at kafka.consumer.ConsumerFetcherManager$LeaderFinderThread.doWork(ConsumerFetcherManager.scala:66)
at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:63)
Caused by: java.nio.channels.ClosedChannelException
at kafka.network.BlockingChannel.send(BlockingChannel.scala:110)
at kafka.producer.SyncProducer.liftedTree1$1(SyncProducer.scala:75)
at kafka.producer.SyncProducer.kafka$producer$SyncProducer$$doSend(SyncProducer.scala:74)
at kafka.producer.SyncProducer.send(SyncProducer.scala:119)
at kafka.client.ClientUtils$.fetchTopicMetadata(ClientUtils.scala:59)
... 3 more
The log generated by zookeeper is :
zookeeper_1 | 2016-02-11 21:10:30,268 [myid:] - INFO [ProcessThread(sid:0 cport:-1)::PrepRequestProcessor#645] - Got user-level KeeperException when processing sessionid:0x152d21258da000c type:create cxid:0x2 zxid:0x51 txntype:-1 reqpath:n/a Error Path:/consumers Error:KeeperErrorCode = NodeExists for /consumers
zookeeper_1 | 2016-02-11 21:10:30,483 [myid:] - INFO [ProcessThread(sid:0 cport:-1)::PrepRequestProcessor#645] - Got user-level KeeperException when processing sessionid:0x152d21258da000c type:create cxid:0x19 zxid:0x55 txntype:-1 reqpath:n/a Error Path:/consumers/console-consumer-51477/owners/teste Error:KeeperErrorCode = NoNode for /consumers/console-consumer-51477/owners/teste
zookeeper_1 | 2016-02-11 21:10:30,485 [myid:] - INFO [ProcessThread(sid:0 cport:-1)::PrepRequestProcessor#645] - Got user-level KeeperException when processing sessionid:0x152d21258da000c type:create cxid:0x1a zxid:0x56 txntype:-1 reqpath:n/a Error Path:/consumers/console-consumer-51477/owners Error:KeeperErrorCode = NoNode for /consumers/console-consumer-51477/owners
I followed this tutorial, and I was able to create a topic and produce messages to it, the problem occured when I tried to consume. I set my KAFKA_ADVERTISED_HOST_NAME to 127.0.0.1.
Could this be the problem?
Commands that I executed:
$KAFKA_HOME/bin/kafka-topics.sh --create --topic topic --partitions 4 --zookeeper $ZK --replication-factor 2
$KAFKA_HOME/bin/kafka-topics.sh --describe --topic topic --zookeeper $ZK
$KAFKA_HOME/bin/kafka-console-producer.sh --topic=topic --broker-list=`broker-list.sh`
$KAFKA_HOME/bin/kafka-console-consumer.sh --topic=topic --zookeeper=$ZK