I deployed Strimzi topic operator on openshift using the following documentation: https://strimzi.io/docs/0.12.2/full.html#deploying-the-topic-operator-standalone-deploying. This does not work and error shown below:
2020-02-26 13:40:30 INFO ZooKeeper:100 - Client environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib`enter code here`
2020-02-26 13:40:30 INFO ZooKeeper:100 - Client environment:java.io.tmpdir=/tmp
2020-02-26 13:40:30 INFO ZooKeeper:100 - Client environment:java.compiler=<NA>
2020-02-26 13:40:30 INFO ZooKeeper:100 - Client environment:os.name=Linux
2020-02-26 13:40:30 INFO ZooKeeper:100 - Client environment:os.arch=amd64
2020-02-26 13:40:30 INFO ZooKeeper:100 - Client environment:os.version=3.10.0-1062.12.1.el7.x86_64
2020-02-26 13:40:30 INFO ZooKeeper:100 - Client environment:user.name=?
2020-02-26 13:40:30 INFO ZooKeeper:100 - Client environment:user.home=?
2020-02-26 13:40:30 INFO ZooKeeper:100 - Client environment:user.dir=/opt/strimzi
2020-02-26 13:40:30 INFO ZooKeeper:442 - Initiating client connection, connectString=172.30.93.216:2181 sessionTimeout=500000 watcher=org.I0Itec.zkclient.ZkClient#74e374fc
2020-02-26 13:40:31 INFO ZkClient:936 - Waiting for keeper state SyncConnected
2020-02-26 13:40:31 INFO ClientCnxn:1025 - Opening socket connection to server soumo-zookeeper-client.kube-system.svc.cluster.local/172.30.93.216:2181. Will not attempt to authenticate using SASL (unknown error)
2020-02-26 13:40:31 INFO ClientCnxn:879 - Socket connection established to soumo-zookeeper-client.kube-system.svc.cluster.local/172.30.93.216:2181, initiating session
2020-02-26 13:40:31 WARN ClientCnxn:1164 - Session 0x0 for server soumo-zookeeper-client.kube-system.svc.cluster.local/172.30.93.216:2181, unexpected error, closing socket connection and attempting reconnect
java.io.IOException: Connection reset by peer
at sun.nio.ch.FileDispatcherImpl.read0(Native Method) ~[?:1.8.0_242]
at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39) ~[?:1.8.0_242]
at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223) ~[?:1.8.0_242]
at sun.nio.ch.IOUtil.read(IOUtil.java:192) ~[?:1.8.0_242]
at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:377) ~[?:1.8.0_242]
at org.apache.zookeeper.ClientCnxnSocketNIO.doIO(ClientCnxnSocketNIO.java:68) ~[org.apache.zookeeper.zookeeper-3.4.14.jar:3.4.14-4c25d480e66aadd371de8bd2fd8da255ac140bcf]
at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:366) ~[org.apache.zookeeper.zookeeper-3.4.14.jar:3.4.14-4c25d480e66aadd371de8bd2fd8da255ac140bcf]
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1141) [org.apache.zookeeper.zookeeper-3.4.14.jar:3.4.14-4c25d480e66aadd371de8bd2fd8da255ac140bcf]
2020-02-26 13:40:34 INFO ClientCnxn:1025 - Opening socket connection to server soumo-zookeeper-client.kube-system.svc.cluster.local/172.30.93.216:2181. Will not attempt to authenticate using SASL (unknown error)
2020-02-26 13:40:34 INFO ClientCnxn:879 - Socket connection established to soumo-zookeeper-client.kube-system.svc.cluster.local/172.30.93.216:2181, initiating session
2020-02-26 13:40:34 WARN ClientCnxn:1164 - Session 0x0 for server soumo-zookeeper-client.kube-system.svc.cluster.local/172.30.93.216:2181, unexpected error, closing socket connection and attempting reconnect
java.io.IOException: Connection reset by peer
Related
I'm trying to setup Apache Storm (1.0.2) on my Macbook Pro but apparently running into timeout issues if I try to deploy the topology. Also the UI hangs up spitting the same exception.
3491 [main] INFO o.a.s.StormSubmitter - Generated ZooKeeper secret payload for MD5-digest: -8915636774701640550:-6510752657961785886
3580 [main] INFO o.a.s.s.a.AuthUtils - Got AutoCreds []
Exception in thread "main" java.lang.RuntimeException: org.apache.storm.thrift.transport.TTransportException: java.net.ConnectException: Operation timed out (Connection timed out)
at org.apache.storm.security.auth.TBackoffConnect.retryNext(TBackoffConnect.java:64)
at org.apache.storm.security.auth.TBackoffConnect.doConnectWithRetry(TBackoffConnect.java:56)
at org.apache.storm.security.auth.ThriftClient.reconnect(ThriftClient.java:99)
at org.apache.storm.security.auth.ThriftClient.<init>(ThriftClient.java:69)
at org.apache.storm.utils.NimbusClient.<init>(NimbusClient.java:106)
at org.apache.storm.utils.NimbusClient.getConfiguredClientAs(NimbusClient.java:78)
at org.apache.storm.StormSubmitter.topologyNameExists(StormSubmitter.java:371)
at org.apache.storm.StormSubmitter.submitTopologyAs(StormSubmitter.java:233)
at org.apache.storm.StormSubmitter.submitTopology(StormSubmitter.java:311)
at org.apache.storm.StormSubmitter.submitTopology(StormSubmitter.java:157)
Caused by: org.apache.storm.thrift.transport.TTransportException: java.net.ConnectException: Operation timed out (Connection timed out)
at org.apache.storm.thrift.transport.TSocket.open(TSocket.java:226)
at org.apache.storm.thrift.transport.TFramedTransport.open(TFramedTransport.java:81)
at org.apache.storm.security.auth.SimpleTransportPlugin.connect(SimpleTransportPlugin.java:103)
at org.apache.storm.security.auth.TBackoffConnect.doConnectWithRetry(TBackoffConnect.java:53)
... 9 more
Caused by: java.net.ConnectException: Operation timed out (Connection timed out)
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:589)
at org.apache.storm.thrift.transport.TSocket.open(TSocket.java:221)
... 12 more
I'm using the default storm.yaml configuration from the github repository; without any change and default zoo.cfg file for zookeeper as well.
# The number of milliseconds of each tick
tickTime=2000
# The number of ticks that the initial
# synchronization phase can take
initLimit=5
# The number of ticks that can pass between
# sending a request and getting an acknowledgement
syncLimit=2
# the directory where the snapshot is stored.
# do not use /tmp for storage, /tmp here is just
# example sakes.
dataDir=/tmp/zookeeper
# the port at which the clients will connect
clientPort=2181
clientPortAddress=localhost
# the maximum number of client connections.
# increase this if you need to handle more clients
#maxClientCnxns=60
#
# Be sure to read the maintenance section of the
# administrator guide before turning on autopurge.
#
# http://zookeeper.apache.org/doc/current/zookeeperAdmin.html#sc_maintenance
#
# The number of snapshots to retain in dataDir
#autopurge.snapRetainCount=3
# Purge task interval in hours
# Set to "0" to disable auto purge feature
#autopurge.purgeInterval=1
I came across similar issues which prompted me to check my hosts file; which I've posted as below
##
# Host Database
#
# localhost is used to configure the loopback interface
# when the system is booting. Do not change this entry.
##
255.255.255.255 broadcasthost
127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4
::1 ip6-localhost ip6-localhost.localdomain localhost6 localhost6.localdomain6
When I start the zookeeper server; I believe it get's started as usual.
2017-11-27 16:05:14,314 [myid:] - INFO [main:QuorumPeerConfig#103] - Reading configuration from: /Users/aniket.alhat/Tools/zookeeper/bin/../conf/zoo.cfg
2017-11-27 16:05:14,318 [myid:] - INFO [main:DatadirCleanupManager#78] - autopurge.snapRetainCount set to 3
2017-11-27 16:05:14,318 [myid:] - INFO [main:DatadirCleanupManager#79] - autopurge.purgeInterval set to 0
2017-11-27 16:05:14,318 [myid:] - INFO [main:DatadirCleanupManager#101] - Purge task is not scheduled.
2017-11-27 16:05:14,318 [myid:] - WARN [main:QuorumPeerMain#113] - Either no config or no quorum defined in config, running in standalone mode
2017-11-27 16:05:14,329 [myid:] - INFO [main:QuorumPeerConfig#103] - Reading configuration from: /Users/aniket.alhat/Tools/zookeeper/bin/../conf/zoo.cfg
2017-11-27 16:05:14,330 [myid:] - INFO [main:ZooKeeperServerMain#95] - Starting server
2017-11-27 16:05:14,335 [myid:] - INFO [main:Environment#100] - Server environment:zookeeper.version=3.4.6-1569965, built on 02/20/2014 09:09 GMT
2017-11-27 16:05:14,336 [myid:] - INFO [main:Environment#100] - Server environment:host.name=10.9.157.77
2017-11-27 16:05:14,336 [myid:] - INFO [main:Environment#100] - Server environment:java.version=1.8.0_131
2017-11-27 16:05:14,336 [myid:] - INFO [main:Environment#100] - Server environment:java.vendor=Oracle Corporation
2017-11-27 16:05:14,336 [myid:] - INFO [main:Environment#100] - Server environment:java.home=/Library/Java/JavaVirtualMachines/jdk1.8.0_131.jdk/Contents/Home/jre
2017-11-27 16:05:14,336 [myid:] - INFO [main:Environment#100] - Server environment:java.class.path=/Users/aniket.alhat/Tools/zookeeper/bin/../build/classes:/Users/aniket.alhat/Tools/zookeeper/bin/../build/lib/*.jar:/Users/aniket.alhat/Tools/zookeeper/bin/../lib/slf4j-log4j12-1.6.1.jar:/Users/aniket.alhat/Tools/zookeeper/bin/../lib/slf4j-api-1.6.1.jar:/Users/aniket.alhat/Tools/zookeeper/bin/../lib/netty-3.7.0.Final.jar:/Users/aniket.alhat/Tools/zookeeper/bin/../lib/log4j-1.2.16.jar:/Users/aniket.alhat/Tools/zookeeper/bin/../lib/jline-0.9.94.jar:/Users/aniket.alhat/Tools/zookeeper/bin/../zookeeper-3.4.6.jar:/Users/aniket.alhat/Tools/zookeeper/bin/../src/java/lib/*.jar:/Users/aniket.alhat/Tools/zookeeper/bin/../conf:
2017-11-27 16:05:14,336 [myid:] - INFO [main:Environment#100] - Server environment:java.library.path=/Users/aniket.alhat/Library/Java/Extensions:/Library/Java/Extensions:/Network/Library/Java/Extensions:/System/Library/Java/Extensions:/usr/lib/java:.
2017-11-27 16:05:14,336 [myid:] - INFO [main:Environment#100] - Server environment:java.io.tmpdir=/var/folders/9c/g5cj60_j1x344r3zpd_hr99j5jwnk4/T/
2017-11-27 16:05:14,336 [myid:] - INFO [main:Environment#100] - Server environment:java.compiler=<NA>
2017-11-27 16:05:14,337 [myid:] - INFO [main:Environment#100] - Server environment:os.name=Mac OS X
2017-11-27 16:05:14,337 [myid:] - INFO [main:Environment#100] - Server environment:os.arch=x86_64
2017-11-27 16:05:14,338 [myid:] - INFO [main:Environment#100] - Server environment:os.version=10.12.6
2017-11-27 16:05:14,338 [myid:] - INFO [main:Environment#100] - Server environment:user.name=aniket.alhat
2017-11-27 16:05:14,338 [myid:] - INFO [main:Environment#100] - Server environment:user.home=/Users/aniket.alhat
2017-11-27 16:05:14,338 [myid:] - INFO [main:Environment#100] - Server environment:user.dir=/Users/aniket.alhat/Tools/zookeeper-3.4.6
2017-11-27 16:05:14,344 [myid:] - INFO [main:ZooKeeperServer#755] - tickTime set to 2000
2017-11-27 16:05:14,344 [myid:] - INFO [main:ZooKeeperServer#764] - minSessionTimeout set to -1
2017-11-27 16:05:14,344 [myid:] - INFO [main:ZooKeeperServer#773] - maxSessionTimeout set to -1
2017-11-27 16:05:14,361 [myid:] - INFO [main:NIOServerCnxnFactory#94] - binding to port localhost/127.0.0.1:2181
And I also don't see any errors in nimbus log
2017-11-27 16:05:35.365 o.a.s.s.o.a.c.f.i.CuratorFrameworkImpl [INFO] Starting
2017-11-27 16:05:35.373 o.a.s.s.o.a.z.ZooKeeper [INFO] Client environment:zookeeper.version=3.4.6-1569965, built on 02/20/2014 09:09 GMT
2017-11-27 16:05:35.373 o.a.s.s.o.a.z.ZooKeeper [INFO] Client environment:host.name=10.49.48.134
2017-11-27 16:05:35.373 o.a.s.s.o.a.z.ZooKeeper [INFO] Client environment:java.version=1.8.0_131
2017-11-27 16:05:35.373 o.a.s.s.o.a.z.ZooKeeper [INFO] Client environment:java.vendor=Oracle Corporation
2017-11-27 16:05:35.373 o.a.s.s.o.a.z.ZooKeeper [INFO] Client environment:java.home=/Library/Java/JavaVirtualMachines/jdk1.8.0_131.jdk/Contents/Home/jre
2017-11-27 16:05:35.373 o.a.s.s.o.a.z.ZooKeeper [INFO] Client environment:java.class.path=/Users/aniket.alhat/Tools/apache-storm-1.0.2/lib/asm-5.0.3.jar:/Users/aniket.alhat/Tools/apache-storm-1.0.2/lib/clojure-1.7.0.jar:/Users/aniket.alhat/Tools/apache-storm-1.0.2/lib/disruptor-3.3.2.jar:/Users/aniket.alhat/Tools/apache-storm-1.0.2/lib/kryo-3.0.3.jar:/Users/aniket.alhat/Tools/apache-storm-1.0.2/lib/log4j-api-2.1.jar:/Users/aniket.alhat/Tools/apache-storm-1.0.2/lib/log4j-core-2.1.jar:/Users/aniket.alhat/Tools/apache-storm-1.0.2/lib/log4j-over-slf4j-1.6.6.jar:/Users/aniket.alhat/Tools/apache-storm-1.0.2/lib/log4j-slf4j-impl-2.1.jar:/Users/aniket.alhat/Tools/apache-storm-1.0.2/lib/minlog-1.3.0.jar:/Users/aniket.alhat/Tools/apache-storm-1.0.2/lib/objenesis-2.1.jar:/Users/aniket.alhat/Tools/apache-storm-1.0.2/lib/reflectasm-1.10.1.jar:/Users/aniket.alhat/Tools/apache-storm-1.0.2/lib/servlet-api-2.5.jar:/Users/aniket.alhat/Tools/apache-storm-1.0.2/lib/slf4j-api-1.7.7.jar:/Users/aniket.alhat/Tools/apache-storm-1.0.2/lib/storm-core-1.0.2.jar:/Users/aniket.alhat/Tools/apache-storm-1.0.2/lib/storm-rename-hack-1.0.2.jar:/Users/aniket.alhat/Tools/storm/conf
2017-11-27 16:05:35.373 o.a.s.s.o.a.z.ZooKeeper [INFO] Client environment:java.library.path=/usr/local/lib:/opt/local/lib:/usr/lib
2017-11-27 16:05:35.373 o.a.s.s.o.a.z.ZooKeeper [INFO] Client environment:java.io.tmpdir=/var/folders/9c/g5cj60_j1x344r3zpd_hr99j5jwnk4/T/
2017-11-27 16:05:35.373 o.a.s.s.o.a.z.ZooKeeper [INFO] Client environment:java.compiler=<NA>
2017-11-27 16:05:35.373 o.a.s.s.o.a.z.ZooKeeper [INFO] Client environment:os.name=Mac OS X
2017-11-27 16:05:35.373 o.a.s.s.o.a.z.ZooKeeper [INFO] Client environment:os.arch=x86_64
2017-11-27 16:05:35.373 o.a.s.s.o.a.z.ZooKeeper [INFO] Client environment:os.version=10.12.6
2017-11-27 16:05:35.373 o.a.s.s.o.a.z.ZooKeeper [INFO] Client environment:user.name=aniket.alhat
2017-11-27 16:05:35.373 o.a.s.s.o.a.z.ZooKeeper [INFO] Client environment:user.home=/Users/aniket.alhat
2017-11-27 16:05:35.373 o.a.s.s.o.a.z.ZooKeeper [INFO] Client environment:user.dir=/Users/aniket.alhat/Tools/apache-storm-1.0.2
2017-11-27 16:05:35.374 o.a.s.s.o.a.z.ZooKeeper [INFO] Initiating client connection, connectString=localhost:2181/storm sessionTimeout=20000 watcher=org.apache.storm.shade.org.apache.curator.ConnectionState#eac3a26
2017-11-27 16:05:35.397 o.a.s.s.o.a.z.ClientCnxn [INFO] Opening socket connection to server localhost/127.0.0.1:2181. Will not attempt to authenticate using SASL (unknown error)
2017-11-27 16:05:35.400 o.a.s.b.FileBlobStoreImpl [INFO] Creating new blob store based in storm-local/blobs
2017-11-27 16:05:35.406 o.a.s.d.nimbus [INFO] Using default scheduler
2017-11-27 16:05:35.408 o.a.s.s.o.a.c.f.i.CuratorFrameworkImpl [INFO] Starting
2017-11-27 16:05:35.409 o.a.s.s.o.a.z.ZooKeeper [INFO] Initiating client connection, connectString=localhost:2181 sessionTimeout=20000 watcher=org.apache.storm.shade.org.apache.curator.ConnectionState#68868328
2017-11-27 16:05:35.411 o.a.s.s.o.a.z.ClientCnxn [INFO] Opening socket connection to server localhost/127.0.0.1:2181. Will not attempt to authenticate using SASL (unknown error)
2017-11-27 16:05:35.438 o.a.s.s.o.a.c.f.i.CuratorFrameworkImpl [INFO] Starting
2017-11-27 16:05:35.438 o.a.s.s.o.a.z.ZooKeeper [INFO] Initiating client connection, connectString=localhost:2181 sessionTimeout=20000 watcher=org.apache.storm.shade.org.apache.curator.ConnectionState#512d6e60
2017-11-27 16:05:35.440 o.a.s.s.o.a.z.ClientCnxn [INFO] Opening socket connection to server localhost/127.0.0.1:2181. Will not attempt to authenticate using SASL (unknown error)
2017-11-27 16:05:35.478 o.a.s.s.o.a.z.ClientCnxn [INFO] Socket connection established to localhost/127.0.0.1:2181, initiating session
2017-11-27 16:05:35.478 o.a.s.s.o.a.z.ClientCnxn [INFO] Socket connection established to localhost/127.0.0.1:2181, initiating session
2017-11-27 16:05:35.479 o.a.s.s.o.a.z.ClientCnxn [INFO] Socket connection established to localhost/127.0.0.1:2181, initiating session
2017-11-27 16:05:35.513 o.a.s.s.o.a.z.ClientCnxn [INFO] Session establishment complete on server localhost/127.0.0.1:2181, sessionid = 0x15ffc4b4d950000, negotiated timeout = 20000
2017-11-27 16:05:35.513 o.a.s.s.o.a.z.ClientCnxn [INFO] Session establishment complete on server localhost/127.0.0.1:2181, sessionid = 0x15ffc4b4d950002, negotiated timeout = 20000
2017-11-27 16:05:35.513 o.a.s.s.o.a.z.ClientCnxn [INFO] Session establishment complete on server localhost/127.0.0.1:2181, sessionid = 0x15ffc4b4d950001, negotiated timeout = 20000
2017-11-27 16:05:35.517 o.a.s.s.o.a.c.f.s.ConnectionStateManager [INFO] State change: CONNECTED
2017-11-27 16:05:35.517 o.a.s.s.o.a.c.f.s.ConnectionStateManager [INFO] State change: CONNECTED
2017-11-27 16:05:35.517 o.a.s.s.o.a.c.f.s.ConnectionStateManager [INFO] State change: CONNECTED
2017-11-27 16:05:35.518 o.a.s.zookeeper [INFO] Zookeeper state update: :connected:none
2017-11-27 16:05:35.518 o.a.s.zookeeper [INFO] Zookeeper state update: :connected:none
2017-11-27 16:05:35.531 o.a.s.s.o.a.c.f.i.CuratorFrameworkImpl [INFO] backgroundOperationsLoop exiting
2017-11-27 16:05:35.534 o.a.s.s.o.a.z.ZooKeeper [INFO] Session: 0x15ffc4b4d950002 closed
2017-11-27 16:05:35.534 o.a.s.s.o.a.z.ClientCnxn [INFO] EventThread shut down
2017-11-27 16:05:35.536 o.a.s.s.o.a.c.f.i.CuratorFrameworkImpl [INFO] Starting
2017-11-27 16:05:35.536 o.a.s.s.o.a.z.ZooKeeper [INFO] Initiating client connection, connectString=localhost:2181/storm sessionTimeout=20000 watcher=org.apache.storm.shade.org.apache.curator.ConnectionState#3722c145
I would really appreciate if I could get some help to fix the timeout issue.
Check if the Nimbus is started correctly. I faced similar issue when an instance of Nimbus was not terminated correctly.
Try to kill the process and restart Nimbus.
After lot of trial-and-error I discovered that my Nimbus process gets started with a IP address 10.9.157.77 while ifconfig gives me 10.49.52.97 not sure why/how this is happening, I'll really appreciate if someone can help me figure it out.
nimbus.log
2017-11-30 16:47:00.342 o.a.s.zookeeper [INFO] 10.9.157.77 gained leadership, checking if it has all the topology code locally.
2017-11-30 16:47:00.350 o.a.s.zookeeper [INFO] active-topology-ids [] local-topology-ids [] diff-topology []
2017-11-30 16:47:00.350 o.a.s.zookeeper [INFO] Accepting leadership, all active topology found localy.
2017-11-30 16:47:00.352 o.a.s.d.m.MetricsUtils [INFO] Using statistics reporter plugin:org.apache.storm.daemon.metrics.reporters.JmxPreparableReporter
ifconfig
en0: flags=8863<UP,BROADCAST,SMART,RUNNING,SIMPLEX,MULTICAST> mtu 1500
ether xx:xx:xx:xx:xx:xx
inet6 fe80::427:8998:bb4d:b2bd%en0 prefixlen 64 secured scopeid 0x4
inet 10.49.52.97 netmask 0xfffffc00 broadcast 10.49.55.255
Continuing forward I found everytime I was starting the nimbus process IP address 10.9.157.77 was been acquired magically and was stored in zookeeper as well.
[zk: localhost:2181(CONNECTED) 15] ls /storm/nimbuses
[10.9.157.77:6627]
I cleaned /storm directory with rmr and restart nimbus creating the directory once again, but there was no change.
I also tried flushing DNS cache, command used was sudo killall -HUP mDNSResponder
I also observed that the IP magical IP wasn't same after restarts, it changed to 10.49.48.134
2017-11-30 17:28:59.630 o.a.s.zookeeper [INFO] 10.49.48.134 gained leadership, checking if it has all the topology code locally.
2017-11-30 17:28:59.646 o.a.s.zookeeper [INFO] active-topology-ids [] local-topology-ids [] diff-topology []
2017-11-30 17:28:59.646 o.a.s.zookeeper [INFO] Accepting leadership, all active topology found localy.
Later I disconnected from Wifi and started everything once again and I was able to start Storm UI run command storm list deploy topology locally.
You can add storm.local.hostname= at your storm/conf/storm.yaml, and restart. Also work with IPv4/FQDN and not IPv6. This worked for me (same Storm 1.0.2)
If there are still problems, you can also add nimbus.seeds= with the Nimbus's host.
I have installed Kerberos and Sentry on the CDH cluster successfully (cm-5.8.4), now I need to use Apache Kylin to do some data analysis, but I can't start Kylin and install the sample.sh. When I try to run sh sample.sh, I get this error message:
2017-08-02 14:44:44,805 INFO [main] zookeeper.ZooKeeper: Client
environment:java.library.path=/opt/cloudera/parcels/CDH-5.8.4-
1.cdh5.8.4.p0.5/lib/hadoop/lib/native/:/opt/cloudera/parcels/CDH-5.8.4-
1.cdh5.8.4.p0.5/lib/hadoop/lib/native/:/opt/cloudera/parcels/CDH-5.8.4-
1.cdh5.8.4.p0.5/lib/hadoop/lib/native:/opt/cloudera/parcels/CDH-5.8.4-
1.cdh5.8.4.p0.5/lib/hbase/bin/../lib/native/Linux-amd64-64 2017-08-02 14:44:44,806 INFO [main] zookeeper.ZooKeeper: Client
environment:java.io.tmpdir=/tmp 2017-08-02 14:44:44,806 INFO [main]
zookeeper.ZooKeeper: Client environment:java.compiler= 2017-08-02
14:44:44,806 INFO [main] zookeeper.ZooKeeper: Client
environment:os.name=Linux 2017-08-02 14:44:44,806 INFO [main]
zookeeper.ZooKeeper: Client environment:os.arch=amd64 2017-08-02
14:44:44,806 INFO [main] zookeeper.ZooKeeper: Client
environment:os.version=3.10.0-229.el7.x86_64 2017-08-02 14:44:44,806
INFO [main] zookeeper.ZooKeeper: Client environment:user.name=root
2017-08-02 14:44:44,806 INFO [main] zookeeper.ZooKeeper: Client
environment:user.home=/root 2017-08-02 14:44:44,806 INFO [main]
zookeeper.ZooKeeper: Client
environment:user.dir=/data1/apache-kylin-1.6.0-cdh5.7-bin 2017-08-02
14:44:44,806 INFO [main] zookeeper.ZooKeeper: Initiating client
connection, connectString=localhost:2181 sessionTimeout=90000
watcher=hconnection-0x3cc79c020x0, quorum=localhost:2181,
baseZNode=/hbase 2017-08-02 14:44:44,823 INFO
[main-SendThread(localhost:2181)] zookeeper.ClientCnxn: Opening
socket connection to server localhost/127.0.0.1:2181. Will not
attempt to authenticate using SASL (unknown error) 2017-08-02
14:44:44,829 WARN [main-SendThread(localhost:2181)]
zookeeper.ClientCnxn: Session 0x0 for server null, unexpected error,
closing socket connection and attempting reconnect
java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:350)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1081)
2017-08-02 14:44:45,934 INFO [main-SendThread(localhost:2181)]
zookeeper.ClientCnxn: Opening socket connection to server
localhost/127.0.0.1:2181. Will not attempt to authenticate using SASL
(unknown error) 2017-08-02 14:44:45,935 WARN
[main-SendThread(localhost:2181)] zookeeper.ClientCnxn: Session 0x0
for server null, unexpected error, closing socket connection and
attempting reconnect
Does anyone know how to solve this problem or show me a guide to use Kylin on a kerberized cluster? Thanks!
The version of Kylin is apache-kylin-1.6.0-cdh5.7-bin.
to install kylin on kerberized cluster is quiet simple.
step 1. ensure you install kylin on a cdh gateway.
step 2. setup the environment correctly.
such as hadoop_home;hive_conf;hbase_conf etc.
step 3. follow the install instruction on kylin website.
if you need more help, leave me a message , so i can show you more detailed documentation on the first 2 steps.
Getting below when trying to connect to zookeeper url using java
.Any pointers will be helpful
INFO 2017-04-11 21:07:34,482 [main] kafka.utils.VerifiableProperties - Verifying properties
INFO 2017-04-11 21:07:34,697 [main] kafka.utils.VerifiableProperties - Property auto.offset.reset is overridden to smallest
INFO 2017-04-11 21:07:34,698 [main] kafka.utils.VerifiableProperties - Property auto.commit.interval.ms is overridden to 1000
INFO 2017-04-11 21:07:34,698 [main] kafka.utils.VerifiableProperties - Property group.id is overridden to n_secondary
INFO 2017-04-11 21:07:34,698 [main] kafka.utils.VerifiableProperties - Property zookeeper.connect is overridden to xxx-xxx-0001.xxxx.xxxx.net:6667,xxx-xxx-0002.xxxx.xxxx.net:6667,xxx-xxx-0003.xxxx.xxxx.net:6667
INFO 2017-04-11 21:07:34,698 [main] kafka.utils.VerifiableProperties - Property zookeeper.session.timeout.ms is overridden to 6400
INFO 2017-04-11 21:07:34,698 [main] kafka.utils.VerifiableProperties - Property consumer.timeout.ms is overridden to 30
INFO 2017-04-11 21:07:34,699 [main] kafka.utils.VerifiableProperties - Property zookeeper.sync.time.ms is overridden to 2000
INFO 2017-04-11 21:07:34,728 [main] kafka.consumer.ZookeeperConsumerConnector - [n__secondary_xxx-xxx--0001.xxxx.wxxxx.net-1491944854725-7865dd0f], Connecting to zookeeper instance at xxx-xxx-0001.xxxx.xxxx.net:6667,xxx-xxx-0002.xxxx.xxxx.net:6667,xxx-xxx-0003.xxxx.xxxx.net:6667
INFO 2017-04-11 21:07:34,910 [ZkClient-EventThread-19-xxx-xxx-0001.xxxx.xxxx.net:6667,xxx-xxx-0002.xxxx.xxxx.net:6667,xxx-xxx-0003.xxxx.xxxx.net:6667] org.I0Itec.zkclient.ZkEventThread - Starting ZkClient event thread.
INFO 2017-04-11 21:07:34,923 [main] org.apache.zookeeper.ZooKeeper - Client environment:zookeeper.version=3.4.8--1, built on 02/06/2016 03:18 GMT
INFO 2017-04-11 21:07:34,923 [main] org.apache.zookeeper.ZooKeeper - Client environment:host.name=ncl-sna-0001.corp.wayport.net
INFO 2017-04-11 21:07:34,923 [main] org.apache.zookeeper.ZooKeeper - Client environment:java.version=1.8.0_121
INFO 2017-04-11 21:07:34,923 [main] org.apache.zookeeper.ZooKeeper - Client environment:java.vendor=Oracle Corporation
INFO 2017-04-11 21:07:34,923 [main] org.apache.zookeeper.ZooKeeper - Client environment:java.home=/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.121-0.b13.el6_8.x86_64/jre
INFO 2017-04-11 21:07:34,923 [main] org.apache.zookeeper.ZooKeeper - Client environment:java.class.path=/opt//Scripts/consumer/ConsumerKafka.jar
INFO 2017-04-11 21:07:34,923 [main] org.apache.zookeeper.ZooKeeper - Client environment:java.library.path=/opt//////linux2x86/lib64:/opt/////linux2x86/lib64:/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
INFO 2017-04-11 21:07:34,923 [main] org.apache.zookeeper.ZooKeeper - Client environment:java.io.tmpdir=/tmp
INFO 2017-04-11 21:07:34,923 [main] org.apache.zookeeper.ZooKeeper - Client environment:java.compiler=<NA>
INFO 2017-04-11 21:07:34,923 [main] org.apache.zookeeper.ZooKeeper - Client environment:os.name=Linux
INFO 2017-04-11 21:07:34,923 [main] org.apache.zookeeper.ZooKeeper - Client environment:os.arch=amd64
INFO 2017-04-11 21:07:34,923 [main] org.apache.zookeeper.ZooKeeper - Client environment:os.version=2.6.32-642.15.1.el6.x86_64
INFO 2017-04-11 21:07:34,923 [main] org.apache.zookeeper.ZooKeeper - Client environment:user.name=n_
INFO 2017-04-11 21:07:34,924 [main] org.apache.zookeeper.ZooKeeper - Client environment:user.home=/var/home/n_
INFO 2017-04-11 21:07:34,924 [main] org.apache.zookeeper.ZooKeeper - Client environment:user.dir=/opt////
INFO 2017-04-11 21:07:34,927 [main] org.apache.zookeeper.ZooKeeper - Initiating client connection, connectString=xxx-xxx-0001.xxxx.xxxx.net:6667,xxx-xxx-0002.xxxx.xxxx.net:6667,xxx-xxx-0003.xxxx.xxxx.net:6667 sessionTimeout=6400 watcher=org.I0Itec.zkclient.ZkClient#7e0b85f9
INFO 2017-04-11 21:07:34,964 [main] org.I0Itec.zkclient.ZkClient - Waiting for keeper state SyncConnected
INFO 2017-04-11 21:07:34,969 [main-SendThread(hxxx-xxx-0003.xxxx.xxxx.net:6667)] org.apache.zookeeper.ClientCnxn - Opening socket connection to server xxx-xxx-0003.xxxx.xxxx.net:6667/xxx.xxx.xxx.2xx:6667. Will not attempt to authenticate using SASL (unknown error)
INFO 2017-04-11 21:07:35,072 [main-SendThread(hxxx-xxx-0003.xxxx.xxxx.net:6667)] org.apache.zookeeper.ClientCnxn - Socket connection established to xxx-xxx-0003.xxxx.xxxx.net:6667/xxx.xxx.xxx.2xx:6667, initiating session
WARN 2017-04-11 21:07:36,401 [main-SendThread(hxxx-xxx-0001.xxxx.xxxx.net:6667)] org.apache.zookeeper.ClientCnxn - Client session timed out, have not heard from server in 2137ms for sessionid 0x0
INFO 2017-04-11 21:07:36,401 [main-SendThread(hxxx-xxx-0001.xxxx.xxxx.net:6667)] org.apache.zookeeper.ClientCnxn - Client session timed out, have not heard from server in 2137ms for sessionid 0x0, closing socket connection and attempting reconnect
INFO 2017-04-11 21:07:36,707 [main-SendThread(hxxx-xxx-0003.xxxx.xxxx.net:6667)] org.apache.zookeeper.ClientCnxn - Opening socket connection to xxx-xxx-0003.xxxx.xxxx.net:6667/xxx.xxx.xxx.2xx:6667. Will not attempt to authenticate using SASL (unknown error)
INFO 2017-04-11 21:07:36,717 [main-SendThread(hxxx-xxx-0003.xxxx.xxxx.net:6667)] org.apache.zookeeper.ClientCnxn - Socket connection established to xxx-xxx-0003.xxxx.xxxx.net:6667/xxx.xxx.xxx.2xx:6667, initiating session
WARN 2017-04-11 21:07:37,208 [main-SendThread(hxxx-xxx-0003.xxxx.xxxx.net:6667)] org.apache.zookeeper.ClientCnxn - Client session timed out, have not heard from server in 2136ms for sessionid 0x0
INFO 2017-04-11 21:07:37,209 [main-SendThread(hxxx-xxx-0003.xxxx.xxxx.net:6667)] org.apache.zookeeper.ClientCnxn - Client session timed out, have not heard from server in 2136ms for sessionid 0x0, closing socket connection and attempting reconnect
INFO 2017-04-11 21:07:38,227 [main-SendThread(hxxx-xxx-0001.xxxx.xxxx.net:6667)] org.apache.zookeeper.ClientCnxn - Opening socket connection to serverxxx-xxx-0001.xxxx.xxxx.net:6667/xxx.xxx.xxx.2xx:6667. Will not attempt to authenticate using SASL (unknown error)
INFO 2017-04-11 21:07:38,238 [main-SendThread(hxxx-xxx-0001.xxxx.xxxx.net:6667)] org.apache.zookeeper.ClientCnxn - Socket connection established to xxx-xxx-0001.xxxx.xxxx.net:6667/xxx.xxx.xxx.2xx:6667, initiating session
WARN 2017-04-11 21:07:38,852 [main-SendThread(hxxx-xxx-0003.xxxx.xxxx.net:6667)] org.apache.zookeeper.ClientCnxn - Client session timed out, have not heard from server in 2135ms for sessionid 0x0
INFO 2017-04-11 21:07:38,852 [main-SendThread(hxxx-xxx-0003.xxxx.xxxx.net:6667)] org.apache.zookeeper.ClientCnxn - Client session timed out, have not heard from server in 2135ms for sessionid 0x0, closing socket connection and attempting reconnect
INFO 2017-04-11 21:07:39,809 [main-SendThread(hxxx-xxx-0002.xxxx.xxxx.net:6667)] org.apache.zookeeper.ClientCnxn - Opening socket connection to server xxx-xxx-0002.xxxx.xxxx.net:6667/xxx.xxx.xxx.2xx:6667. Will not attempt to authenticate using SASL (unknown error)
INFO 2017-04-11 21:07:39,820 [main-SendThread(hxxx-xxx-0002.xxxx.xxxx.net:6667)] org.apache.zookeeper.ClientCnxn - Socket connection established to xxx-xxx-0002.xxxx.xxxx.net:6667/xxx.xxx.xxx.2xx:6667, initiating session
WARN 2017-04-11 21:07:40,374 [main-SendThread(hxxx-xxx-0001.xxxx.xxxx.net:6667)] org.apache.zookeeper.ClientCnxn - Client session timed out, have not heard from server in 2135ms for sessionid 0x0
INFO 2017-04-11 21:07:40,374 [main-SendThread(hxxx-xxx-0001.xxxx.xxxx.net:6667)] org.apache.zookeeper.ClientCnxn - Client session timed out, have not heard from server in 2135ms for sessionid 0x0, closing socket connection and attempting reconnect
INFO 2017-04-11 21:07:40,545 [ZkClient-EventThread-19-hxxx-xxx-0001.xxxx.xxxx.net:6667,hxxx-xxx-0002.xxxx.xxxx.net:6667,hxxx-xxx-0003.xxxx.xxxx.net:6667] org.I0Itec.zkclient.ZkEventThread - Terminate ZkClient event thread.
INFO 2017-04-11 21:07:41,318 [main-SendThread(hxxx-xxx-0002.xxxx.xxxx.net:6667)] org.apache.zookeeper.ClientCnxn - Opening socket connection to server hxxx-xxx-0001.xxxx.xxxx.net:6667/xxx.xxx.xxx.xxx:6667. Will not attempt to authenticate using SASL (unknown error)
INFO 2017-04-11 21:07:41,329 [main-SendThread(hxxx-xxx-0002.xxxx.xxxx.net:6667)] org.apache.zookeeper.ClientCnxn - Socket connection established to hak-dal-0002.corp.wayport.net/xxx.xxx.xxx.xxx:6667, initiating session
INFO 2017-04-11 21:07:41,365 [ZkClient-EventThread-19-hxxx-xxx-0001.xxxx.xxxx.net:6667hxxx-xxx-0002.xxxx.xxxx.net:6667,hxxx-xxx-0003.xxxx.xxxx.net:6667] org.I0Itec.zkclient.ZkEventThread - Terminate ZkClient event thread.
WARN 2017-04-11 21:07:41,954 [main-SendThread(hxxx-xxx-0002.xxxx.xxxx.net:6667)] org.apache.zookeeper.ClientCnxn - Client session timed out, have not heard from server in 2134ms for sessionid 0x0
INFO 2017-04-11 21:07:42,061 [main] org.apache.zookeeper.ZooKeeper - Session: 0x0 closed
INFO 2017-04-11 21:07:42,068 [main-EventThread] org.apache.zookeeper.ClientCnxn - EventThread shut down for session: 0x0
INFO 2017-04-11 21:07:43,319 [main] kafka.utils.VerifiableProperties - Verifying properties
WARN 2017-04-11 21:07:43,465 [main-SendThread(hxxx-xxx-0002.xxxx.xxxx.net:6667)] org.apache.zookeeper.ClientCnxn - Client session timed out, have not heard from server in 2136ms for sessionid 0x0
My Concern is what does this "Will not attempt to authenticate using SASL (unknown error) " means ?
Is this an issue .If so how this can be resolved?
Thanks and Regards
Jinesh Luke
I have solved this issue by clustering zookeeper
I'm running Kafka (0.10.0.0) in a Docker on a Mac (w/docker-machine). I derived my Dockerfile from Spotify's, which means Kafka and Zookeeper run in the same image.
My instance starts cleanly and poking around inside it appears everything is normal/OK.
Docker maps ports 2181 and 9092 to high-ports 32822 and 32820 in this case. From outside my running Kafka Docker I am able to successfully telnet 192.168.99.100 32822 (where 192.168.99.100 is the IP of my docker-machine). From there I can issue a zookeeper command and get expected output.
It all seems so encouraging, but... I then try this code:
val numPartitions = 4
val replicationFactor = 1
val topicConfig = new java.util.Properties
// zookeeper = "192.168.99.100:32822"
val zkClient = ZkUtils(zookeeper, 10000, 10000, false)
try {
AdminUtils.createTopic(zkClient, topic, numPartitions, replicationFactor, topicConfig)
} catch {
case k: kafka.common.TopicExistsException => // do nothing...topic exists
}
zkClient.close()
This produces this error output:
DEBUG ZkConnection - Creating new ZookKeeper instance to connect to 192.168.99.100:32822.
INFO ZkEventThread - Starting ZkClient event thread.
INFO ZooKeeper - Client environment:zookeeper.version=3.4.6-1569965, built on 02/20/2014 09:09 GMT
INFO ZooKeeper - Client environment:host.name=172.25.42.82
INFO ZooKeeper - Client environment:java.version=1.8.0_60
INFO ZooKeeper - Client environment:java.vendor=Oracle Corporation
INFO ZooKeeper - Client environment:java.home=/Library/Java/JavaVirtualMachines/jdk1.8.0_60.jdk/Contents/Home/jre
INFO ZooKeeper - Client environment:java.class.path=/usr/local/Cellar/sbt/0.13.11/libexec/sbt-launch.jar
INFO ZooKeeper - Client environment:java.library.path=/Users/wmy965/Library/Java/Extensions:/Library/Java/Extensions:/Network/Library/Java/Extensions:/System/Library/Java/Extensions:/usr/lib/java:.
INFO ZooKeeper - Client environment:java.io.tmpdir=/var/folders/ph/ccz4n1qs62n0bn8mqdg94gswt1jlwk/T/
INFO ZooKeeper - Client environment:java.compiler=<NA>
INFO ZooKeeper - Client environment:os.name=Mac OS X
INFO ZooKeeper - Client environment:os.arch=x86_64
INFO ZooKeeper - Client environment:os.version=10.11.5
INFO ZooKeeper - Client environment:user.name=wmy965
INFO ZooKeeper - Client environment:user.home=/Users/wmy965
INFO ZooKeeper - Client environment:user.dir=/Users/wmy965/git/LateKafka
INFO ZooKeeper - Initiating client connection, connectString=192.168.99.100:32822 sessionTimeout=10000 watcher=org.I0Itec.zkclient.ZkClient#55397e3
DEBUG ClientCnxn - zookeeper.disableAutoWatchReset is false
DEBUG ZkClient - Awaiting connection to Zookeeper server
INFO ZkClient - Waiting for keeper state SyncConnected
INFO ClientCnxn - Opening socket connection to server 192.168.99.100/192.168.99.100:32822. Will not attempt to authenticate using SASL (unknown error)
WARN ClientCnxn - Session 0x0 for server null, unexpected error, closing socket connection and attempting reconnect
java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:361)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1081)
DEBUG ClientCnxnSocketNIO - Ignoring exception during shutdown input
java.nio.channels.ClosedChannelException
at sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:780)
at sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:399)
at org.apache.zookeeper.ClientCnxnSocketNIO.cleanup(ClientCnxnSocketNIO.java:200)
at org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:1185)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1110)
DEBUG ClientCnxnSocketNIO - Ignoring exception during shutdown output
java.nio.channels.ClosedChannelException
at sun.nio.ch.SocketChannelImpl.shutdownOutput(SocketChannelImpl.java:797)
at sun.nio.ch.SocketAdaptor.shutdownOutput(SocketAdaptor.java:407)
at org.apache.zookeeper.ClientCnxnSocketNIO.cleanup(ClientCnxnSocketNIO.java:207)
at org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:1185)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1110)
INFO ClientCnxn - Opening socket connection to server 192.168.99.100/192.168.99.100:32822. Will not attempt to authenticate using SASL (unknown error)
INFO ClientCnxn - Socket connection established to 192.168.99.100/192.168.99.100:32822, initiating session
DEBUG ClientCnxn - Session establishment request sent on 192.168.99.100/192.168.99.100:32822
INFO ClientCnxn - Session establishment complete on server 192.168.99.100/192.168.99.100:32822, sessionid = 0x155225c51720000, negotiated timeout = 10000
DEBUG ZkClient - Received event: WatchedEvent state:SyncConnected type:None path:null
INFO ZkClient - zookeeper state changed (SyncConnected)
DEBUG ZkClient - Leaving process event
DEBUG ZkClient - State is SyncConnected
DEBUG ClientCnxn - Reading reply sessionid:0x155225c51720000, packet:: clientPath:null serverPath:null finished:false header:: 1,8 replyHeader:: 1,1,-101 request:: '/brokers/ids,F response:: v{}
It looks like I can't connect (presumably to zookeeper). Why not?
In new kafka streams, the ip of producer must have been knowing by kafka (docker). Kafka send their uuid (you can show this in /etc/hosts inside kafka docker) and espect response from this.
Summary:
Map uuid kafka docker to docker-machine in /etc/host of mac OS.
To help you, how to change etc/host file in mac:
https://www.tekrevue.com/tip/edit-hosts-file-mac-os-x/
Cleaner would be to set advertised.listeners=host-ip:port since advertised.host.name and advertised.port are deprecated in Kafka server.properties file.
If set host-ip to 0.0.0.0 it will listen requests from anywhere. But it's insecure.
How to programmatically detect which server in a ZooKeeper ensemble a client is connected to?
I'm using the Apache Curator API and I am listening for state changes in connection by registering ConnectionStateListener. I would like to know which server in the ensemble a client is connected to when the client reconnects if the server it was connected to goes down.
You can see this in the logs produced by Curator. In the example output below, the CuratorFramework client has been given 4 different ZooKeeper instances in the connectionString that it can connect to. As can be seen in the log, it choses the first:
21:13:45.384 [main] INFO org.apache.curator.framework.imps.CuratorFrameworkImpl - Starting
21:13:45.386 [main] INFO org.apache.zookeeper.ZooKeeper - Initiating client connection, connectString=127.0.0.1:2181,127.0.0.1:2182,127.0.0.1:2183,127.0.0.1:2184 sessionTimeout=60000 watcher=org.apache.curator.ConnectionState#2876f0c
21:13:45.388 [main-SendThread(127.0.0.1:2181)] INFO org.apache.zookeeper.ClientCnxn - Opening socket connection to server 127.0.0.1/127.0.0.1:2181. Will not attempt to authenticate using SASL (unknown error)
21:13:45.388 [main-SendThread(127.0.0.1:2181)] INFO org.apache.zookeeper.ClientCnxn - Socket connection established to 127.0.0.1/127.0.0.1:2181, initiating session
21:13:45.392 [main-SendThread(127.0.0.1:2181)] INFO org.apache.zookeeper.ClientCnxn - Session establishment complete on server 127.0.0.1/127.0.0.1:2181, sessionid = 0x14aac461eb70004, negotiated timeout = 40000
In case the ZooKeeper server that the client has connected to crashes, you will also see the new server that the client connects to in the logs:
21:23:03.675 [main-SendThread(127.0.0.1:2182)] INFO org.apache.zookeeper.ClientCnxn - Opening socket connection to server 127.0.0.1/127.0.0.1:2182. Will not attempt to authenticate using SASL (unknown error)
21:23:03.677 [main-SendThread(127.0.0.1:2182)] INFO org.apache.zookeeper.ClientCnxn - Socket connection established to 127.0.0.1/127.0.0.1:2182, initiating session
21:23:03.697 [main-SendThread(127.0.0.1:2182)] INFO org.apache.zookeeper.ClientCnxn - Session establishment complete on server 127.0.0.1/127.0.0.1:2182, sessionid = 0x14aac461eb70004, negotiated timeout = 40000
21:23:03.697 [main-EventThread] INFO org.apache.curator.framework.state.ConnectionStateManager - State change: RECONNECTED