I used Curator to newClient, and set retry policy, but if the connection string is wrong, retry connected will always running until out of memory, but I want the program exit once retry 3 times.
RetryPolicy retryPolicy = new ExponentialBackoffRetry(1000, 3);
CuratorFrameworkFactory.newClient(zkAddress,retryPolicy);
CuratorFrameworkFactory.builder().connectString(zkAddress)
.sessionTimeoutMs(5000).connectionTimeoutMs(12000)
.retryPolicy(retryPolicy).build();
09:03:18.810 [main-SendThread(0:0:0:0:0:0:0:1:2181)] INFO org.apache.zookeeper.ClientCnxn - Opening socket connection to server 0:0:0:0:0:0:0:1/0:0:0:0:0:0:0:1:2181. Will not attempt to authenticate using SASL (unknown error)
09:03:19.825 [main-SendThread(0:0:0:0:0:0:0:1:2181)] WARN org.apache.zookeeper.ClientCnxn - Session 0x0 for server null, unexpected error, closing socket connection and attempting reconnect
java.net.ConnectException: Connection refused: no further information
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(Unknown Source)
at o rg.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:361)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1081)
Related
I am getting lot of NoClassDefFoundError in a running Kafka server. I am not sure what is causing this? Has anyone else faced this issue?
I am using Kafka v2.1.1 and Java 11. I recently upgraded to v2.1.1
Update:
Note that I am seeing this exception in only 2 brokers out of the 5 in entire cluster.
Exception 1:
[2019-02-28 00:54:49,288] INFO Successfully authenticated client: authenticationID=rajputs/hostbased#UNIX.DESHAW.COM; authorizationID=rajputs/hostbased#UNIX.DESHAW.COM. (org.apache.kafka.common.security.authenticator.SaslServerCallbackHandler)
[2019-02-28 00:54:49,469] ERROR [KafkaApi-4] Error when handling request: clientId=consumer-2, correlationId=1686, api=METADATA, body={topics=[fps.rajputs.jpse.desim_se],allow_auto_topic_creation=true} (kafka.server.KafkaApis)
java.lang.NoClassDefFoundError: scala/collection/immutable/SortedMap$$anon$1
at scala.collection.immutable.SortedMap.filterKeys(SortedMap.scala:87)
at scala.collection.immutable.SortedMap.filterKeys$(SortedMap.scala:87)
at scala.collection.immutable.TreeMap.filterKeys(TreeMap.scala:46)
at kafka.security.auth.SimpleAclAuthorizer.$anonfun$getMatchingAcls$1(SimpleAclAuthorizer.scala:245)
at kafka.utils.CoreUtils$.inLock(CoreUtils.scala:251)
at kafka.utils.CoreUtils$.inReadLock(CoreUtils.scala:257)
at kafka.security.auth.SimpleAclAuthorizer.getMatchingAcls(SimpleAclAuthorizer.scala:233)
at kafka.security.auth.SimpleAclAuthorizer.aclsAllowAccess$1(SimpleAclAuthorizer.scala:154)
at kafka.security.auth.SimpleAclAuthorizer.authorize(SimpleAclAuthorizer.scala:159)
at kafka.server.KafkaApis.$anonfun$authorize$1(KafkaApis.scala:371)
at kafka.server.KafkaApis.$anonfun$authorize$1$adapted(KafkaApis.scala:371)
at scala.Option.forall(Option.scala:247)
at kafka.server.KafkaApis.authorize(KafkaApis.scala:371)
at kafka.server.KafkaApis.$anonfun$handleTopicMetadataRequest$1(KafkaApis.scala:970)
at kafka.server.KafkaApis.$anonfun$handleTopicMetadataRequest$1$adapted(KafkaApis.scala:970)
at scala.collection.TraversableLike.$anonfun$partition$1(TraversableLike.scala:313)
at scala.collection.immutable.Set$Set1.foreach(Set.scala:95)
at scala.collection.TraversableLike.partition(TraversableLike.scala:313)
at scala.collection.TraversableLike.partition$(TraversableLike.scala:311)
at scala.collection.AbstractTraversable.partition(Traversable.scala:104)
at kafka.server.KafkaApis.handleTopicMetadataRequest(KafkaApis.scala:970)
at kafka.server.KafkaApis.handle(KafkaApis.scala:109)
at kafka.server.KafkaRequestHandler.run(KafkaRequestHandler.scala:69)
at java.base/java.lang.Thread.run(Thread.java:834)
Exception 2:
[2019-02-28 01:05:58,920] INFO [GroupMetadataManager brokerId=4] Removed 0 expired offsets in 0 milliseconds. (kafka.coordinator.group.GroupMetadataManager)
[2019-02-28 01:06:04,296] WARN Client session timed out, have not heard from server in 4002ms for sessionid 0x468d0a6602700cc (org.apache.zookeeper.ClientCnxn)
[2019-02-28 01:06:04,296] INFO Client session timed out, have not heard from server in 4002ms for sessionid 0x468d0a6602700cc, closing socket connection and attempting reconnect (org.apache.zookeeper.ClientCnxn)
[2019-02-28 01:06:04,972] INFO Client will use GSSAPI as SASL mechanism. (org.apache.zookeeper.client.ZooKeeperSaslClient)
[2019-02-28 01:06:04,972] INFO Opening socket connection to server mwkafka-zk-test-03.dr/10.218.247.19:2181. Will attempt to SASL-authenticate using Login Context section 'Client' (org.apache.zookeeper.ClientCnxn)
[2019-02-28 01:06:04,973] INFO Socket connection established to mwkafka-zk-test-03.dr/10.218.247.19:2181, initiating session (org.apache.zookeeper.ClientCnxn)
[2019-02-28 01:06:04,973] WARN Session 0x468d0a6602700cc for server mwkafka-zk-test-03.dr/10.218.247.19:2181, unexpected error, closing socket connection and attempting reconnect (org.apache.zookeeper.ClientCnxn)
java.lang.NoClassDefFoundError: org/apache/zookeeper/proto/SetWatches
at org.apache.zookeeper.ClientCnxn$SendThread.primeConnection(ClientCnxn.java:929)
at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:363)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1145)
Caused by: java.lang.ClassNotFoundException: org.apache.zookeeper.proto.SetWatches
at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:583)
at java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:178)
at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:521)
... 3 more
Exception in thread "main" java.lang.NoSuchMethodError: io.confluent.rest.Application.parseListeners(Ljava/util/List;ILjava/util/List;Ljava/lang/String;)Ljava/util/List;
at io.confluent.kafka.schemaregistry.storage.KafkaSchemaRegistry.getPortForIdentity(KafkaSchemaRegistry.java:204)
at io.confluent.kafka.schemaregistry.storage.KafkaSchemaRegistry.(KafkaSchemaRegistry.java:133)
etc/schema-registry/schema-registry.properties
listeners=http://0.0.0.0:8081
kafkastore.connection.url=localhost:2181
kafkastore.topic=_schemas
debug=false
kafka and zookeeper are already running.
Why logs from zookeeper keep on coming like
[2017-10-17 09:57:31,352] INFO Accepted socket connection from /13.**.**.***:39572 (org.apache.zookeeper.server.NIOServerCnxnFactory)
[2017-10-17 09:57:31,352] WARN Exception causing close of session 0x0 due to java.io.EOFException (org.apache.zookeeper.server.NIOServerCnxn)
[2017-10-17 09:57:31,352] INFO Closed socket connection for client /13.58.108.150:39572 (no session established for client) (org.apache.zookeeper.server.NIOServerCnxn)
[2017-10-17 09:57:31,438] INFO Accepted socket connection from /13.**.**.***:39574 (org.apache.zookeeper.server.NIOServerCnxnFactory)
[2017-10-17 09:57:31,438] WARN Exception causing close of session 0x0 due to java.io.EOFException (org.apache.zookeeper.server.NIOServerCnxn)
[2017-10-17 09:57:31,438] INFO Closed socket connection for client /13.**.***.**:39574 (no session established for client) (org.apache.zookeeper.server.NIOServerCnxn)
I was wondering maybe this will be the cause of failure for schema-registry.
Any suggestions.
NoSuchMethodError indicates your CLASSPATH is misconfigured.
It's not clear what version you're running or what OS you're using but Windows is not officially supported, and pater versions of Confluent Platform have likely fixed this, or using the Docker images should work as well
in my situation the problem was caused by hostname, check if hostname is equal to "localhost"
Problem "Scheme registry fail on start"
Test Solution "set Hostname to "localhost""
If this solve your problem, you can config permantly yout hostname:
modify file /etc/hostname
I am running a 5 node kafka broker with a 5 node zookeeper ensemble.
On Kafka nodes I keep getting this error
[2016-11-14 19:05:11,345] INFO Opening socket connection to server 10.105.23.188/10.105.23.188:2181. Will not attempt to authenticate using SASL (unknown error) (org.apache.zookeeper.ClientCnxn)
[2016-11-14 19:05:11,345] INFO Socket connection established to 10.105.23.188/10.105.23.188:2181, initiating session (org.apache.zookeeper.ClientCnxn)
[2016-11-14 19:05:11,346] INFO Unable to read additional data from server sessionid 0x55861d8ea6f0000, likely server has closed socket, closing socket connection and attempting reconnect (org.apache.zookeeper.ClientCnxn)
[2016-11-14 19:05:11,721] INFO Opening socket connection to server 10.105.25.4/10.105.25.4:2181. Will not attempt to authenticate using SASL (unknown error) (org.apache.zookeeper.ClientCnxn)
[2016-11-14 19:05:11,721] INFO Socket connection established to 10.105.25.4/10.105.25.4:2181, initiating session (org.apache.zookeeper.ClientCnxn)
[2016-11-14 19:05:11,722] INFO Unable to read additional data from server sessionid 0x55861d8ea6f0000, likely server has closed socket, closing socket connection and attempting reconnect (org.apache.zookeeper.ClientCnxn)
[2016-11-14 19:05:11,841] INFO Opening socket connection to server 10.105.24.4/10.105.24.4:2181. Will not attempt to authenticate using SASL (unknown error) (org.apache.zookeeper.ClientCnxn)
[2016-11-14 19:05:11,841] WARN Session 0x55861d8ea6f0000 for server null, unexpected error, closing socket connection and attempting reconnect (org.apache.zookeeper.ClientCnxn)
java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:361)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1081)
[2016-11-14 19:05:12,319] INFO Opening socket connection to server 10.105.27.23/10.105.27.23:2181. Will not attempt to authenticate using SASL (unknown error) (org.apache.zookeeper.ClientCnxn)
[2016-11-14 19:05:12,319] INFO Socket connection established to 10.105.27.23/10.105.27.23:2181, initiating session (org.apache.zookeeper.ClientCnxn)
[2016-11-14 19:05:12,320] INFO Unable to read additional data from server sessionid 0x55861d8ea6f0000, likely server has closed socket, closing socket connection and attempting reconnect (org.apache.zookeeper.ClientCnxn)
[2016-11-14 19:05:12,395] WARN [ReplicaFetcherThread-0-5], Error in fetch kafka.server.ReplicaFetcherThread$FetchRequest#3fbe518f. Possible cause: org.apache.kafka.common.protocol.types.SchemaException: Error reading field 'responses': Error reading array of size 1204063, only 920 bytes available (kafka.server.ReplicaFetcherThread)
On Zookeeper nodes :
I get the following on the leader
2016-11-14 19:14:47,502 [myid:4] - ERROR [LearnerHandler-/10.105.25.4:50099:LearnerHandler#631] - Unexpected exception causing shutdown while sock still open
java.io.EOFException
at java.io.DataInputStream.readInt(DataInputStream.java:392)
at org.apache.jute.BinaryInputArchive.readInt(BinaryInputArchive.java:63)
at org.apache.zookeeper.server.quorum.QuorumPacket.deserialize(QuorumPacket.java:83)
at org.apache.jute.BinaryInputArchive.readRecord(BinaryInputArchive.java:99)
at org.apache.zookeeper.server.quorum.LearnerHandler.run(LearnerHandler.java:513)
When a consumer tries to read from this cluster it get this error repeatedly
2016-11-14 19:17:36,847 WARN {main} [kafka.clients.NetworkClient$DefaultMetadataUpdater:handleResponse:629] Error while fetching metadata with correlation id 1603 : {=UNKNOWN_TOPIC_OR_PARTITION}
Please help me debug this. The consumers have auto commit enabled. If you need me to provide more configuration please let me know.
Thanks!
I have two zookeeper/kafka servers.
say,
10.10.1.9
10.10.1.10
When i start my zokeeper with configuration:
On 10.10.1.9,
dataDir=/var/zookeeper
# the port at which the clients will connect
clientPort=2181
# disable the per-ip limit on the number of connections since this is a non-production config
maxClientCnxns=0
initLimit=5
syncLimit=2
server.1=10.10.1.9:2888:3888
server.2=10.10.1.10:2888:3888
I get the following warning on 10.10.1.9:
[2016-05-23 07:26:56,047] WARN Cannot open channel to 2 at election address /10.10.1.10:3888 (org.apache.zookeeper.server.quorum.QuorumCnxManager)
java.net.ConnectException: Connection refused
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:579)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:368)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.toSend(QuorumCnxManager.java:341)
at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.process(FastLeaderElection.java:449)
at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.run(FastLeaderElection.java:430)
at java.lang.Thread.run(Thread.java:745)
Meaning to say that zookeeper is not able to connect to 10.10.1.10 server.
ON 10.10.1.10 zookeeper logs, i get the following logs:
[2016-05-23 06:01:48,513] INFO binding to port 0.0.0.0/0.0.0.0:2181 (org.apache.zookeeper.server.NIOServerCnxnFactory)
[2016-05-23 07:23:00,400] INFO Accepted socket connection from /10.10.1.9:44808 (org.apache.zookeeper.server.NIOServerCnxnFactory)
[2016-05-23 07:23:05,537] WARN Exception causing close of session 0x0 due to java.io.IOException: Len error -720899 (org.apache.zookeeper.server.NIOServerCnxn)
[2016-05-23 07:23:05,538] INFO Closed socket connection for client /10.10.1.9:44808 (no session established for client) (org.apache.zookeeper.server.NIOServerCnxn)
[2016-05-23 07:24:43,180] INFO Accepted socket connection from /10.10.1.9:44821 (org.apache.zookeeper.server.NIOServerCnxnFactory)
[2016-05-23 07:24:45,189] WARN Exception causing close of session 0x0 due to java.io.IOException: Len error -1179651 (org.apache.zookeeper.server.NIOServerCnxn)
[2016-05-23 07:24:45,189] INFO Closed socket connection for client /10.10.1.9:44821 (no session established for client) (org.apache.zookeeper.server.NIOServerCnxn)
[2016-05-23 07:47:47,166] INFO Accepted socket connection from /10.10.1.9:45015 (org.apache.zookeeper.server.NIOServerCnxnFactory)
[2016-05-23 07:47:51,080] WARN caught end of stream exception (org.apache.zookeeper.server.NIOServerCnxn)
EndOfStreamException: Unable to read additional data from client sessionid 0x0, likely client has closed socket
at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:228)
at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
at java.lang.Thread.run(Thread.java:745)
[2016-05-23 07:47:51,091] INFO Closed socket connection for client /10.10.1.9:45015 (no session established for client) (org.apache.zookeeper.server.NIOServerCnxn)
[2016-05-23 07:56:05,693] INFO Accepted socket connection from /10.10.1.9:45085 (org.apache.zookeeper.server.NIOServerCnxnFactory)
[2016-05-23 07:56:11,359] WARN caught end of stream exception (org.apache.zookeeper.server.NIOServerCnxn)
EndOfStreamException: Unable to read additional data from client sessionid 0x0, likely client has closed socket
at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:228)
at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
at java.lang.Thread.run(Thread.java:745)
[2016-05-23 07:56:11,360] INFO Closed socket connection for client /10.10.1.9:45085 (no session established for client) (org.apache.zookeeper.server.NIOServerCnxn)
[2016-05-23 08:01:22,066] INFO Accepted socket connection from /10.10.1.9:45127 (org.apache.zookeeper.server.NIOServerCnxnFactory)
[2016-05-23 08:01:26,015] WARN caught end of stream exception (org.apache.zookeeper.server.NIOServerCnxn)
EndOfStreamException: Unable to read additional data from client sessionid 0x0, likely client has closed socket
at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:228)
at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
at java.lang.Thread.run(Thread.java:745)
Anybody encountered such issue? Any idea?
Can you list down all the port permissions or any thing else to be done for the two zookeeper servers to be able to communicate.
Thanks in advance guys!!
How to programmatically detect which server in a ZooKeeper ensemble a client is connected to?
I'm using the Apache Curator API and I am listening for state changes in connection by registering ConnectionStateListener. I would like to know which server in the ensemble a client is connected to when the client reconnects if the server it was connected to goes down.
You can see this in the logs produced by Curator. In the example output below, the CuratorFramework client has been given 4 different ZooKeeper instances in the connectionString that it can connect to. As can be seen in the log, it choses the first:
21:13:45.384 [main] INFO org.apache.curator.framework.imps.CuratorFrameworkImpl - Starting
21:13:45.386 [main] INFO org.apache.zookeeper.ZooKeeper - Initiating client connection, connectString=127.0.0.1:2181,127.0.0.1:2182,127.0.0.1:2183,127.0.0.1:2184 sessionTimeout=60000 watcher=org.apache.curator.ConnectionState#2876f0c
21:13:45.388 [main-SendThread(127.0.0.1:2181)] INFO org.apache.zookeeper.ClientCnxn - Opening socket connection to server 127.0.0.1/127.0.0.1:2181. Will not attempt to authenticate using SASL (unknown error)
21:13:45.388 [main-SendThread(127.0.0.1:2181)] INFO org.apache.zookeeper.ClientCnxn - Socket connection established to 127.0.0.1/127.0.0.1:2181, initiating session
21:13:45.392 [main-SendThread(127.0.0.1:2181)] INFO org.apache.zookeeper.ClientCnxn - Session establishment complete on server 127.0.0.1/127.0.0.1:2181, sessionid = 0x14aac461eb70004, negotiated timeout = 40000
In case the ZooKeeper server that the client has connected to crashes, you will also see the new server that the client connects to in the logs:
21:23:03.675 [main-SendThread(127.0.0.1:2182)] INFO org.apache.zookeeper.ClientCnxn - Opening socket connection to server 127.0.0.1/127.0.0.1:2182. Will not attempt to authenticate using SASL (unknown error)
21:23:03.677 [main-SendThread(127.0.0.1:2182)] INFO org.apache.zookeeper.ClientCnxn - Socket connection established to 127.0.0.1/127.0.0.1:2182, initiating session
21:23:03.697 [main-SendThread(127.0.0.1:2182)] INFO org.apache.zookeeper.ClientCnxn - Session establishment complete on server 127.0.0.1/127.0.0.1:2182, sessionid = 0x14aac461eb70004, negotiated timeout = 40000
21:23:03.697 [main-EventThread] INFO org.apache.curator.framework.state.ConnectionStateManager - State change: RECONNECTED