I am trying to manually start zookeeper. I run
# source zkServer.sh start
It outputs:
JMX enabled by default
Using config: /opt/zookeeper-3.4.6/bin/../conf/zoo.cfg
Starting zookeeper ... STARTED
When I run #jps, it outputs
15360 QuorumPeerMain
15412 Jps
From what I read online, Zookeeper is the same process as QuorumPeerMain listed above. But then when I check its status using
source zkServer.sh status
It hangs at:
JMX enabled by default
Using config: /opt/zookeeper-3.4.6/bin/../conf/zoo.cfg
So I run
#source zkServer.sh status > templogs.txt
Running above makes terminal to flash below for a moment (I have to run this command many times to read what actually is output on terminal before it closes) and closes terminal
JMX enabled by default
Using config:
grep: No such file or directory
grep: No such file or directory
Writing to templogs.txt following
Error contacting service. It is probably not running.
When I open zookeeper.out, I can see this output:
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/opt/zookeeper-3.4.6/lib/slf4j-log4j12-1.6.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/opt/hive-0.12.0-cdh5.0.3/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/opt/hbase-0.96.1.1-cdh5.0.3/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/opt/spark-0.9.0-cdh/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/opt/splicemachine/lib/slf4j-log4j12-1.6.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
2015-06-29 15:49:40,831 [myid:] - INFO [main:QuorumPeerConfig#103] - Reading configuration from: /opt/zookeeper-3.4.6/bin/../conf/zoo.cfg
2015-06-29 15:49:40,836 [myid:] - INFO [main:DatadirCleanupManager#78] - autopurge.snapRetainCount set to 3
2015-06-29 15:49:40,836 [myid:] - INFO [main:DatadirCleanupManager#79] - autopurge.purgeInterval set to 0
2015-06-29 15:49:40,836 [myid:] - INFO [main:DatadirCleanupManager#101] - Purge task is not scheduled.
2015-06-29 15:49:40,837 [myid:] - WARN [main:QuorumPeerMain#113] - Either no config or no quorum defined in config, running in standalone mode
2015-06-29 15:49:40,847 [myid:] - INFO [main:QuorumPeerConfig#103] - Reading configuration from: /opt/zookeeper-3.4.6/bin/../conf/zoo.cfg
2015-06-29 15:49:40,847 [myid:] - INFO [main:ZooKeeperServerMain#95] - Starting server
2015-06-29 15:49:40,896 [myid:] - INFO [main:Environment#100] - Server environment:zookeeper.version=3.4.6-1569965, built on 02/20/2014 09:09 GMT
2015-06-29 15:49:40,896 [myid:] - INFO [main:Environment#100] - Server environment:host.name=ingester
2015-06-29 15:49:40,896 [myid:] - INFO [main:Environment#100] - Server environment:java.version=1.8.0_25
2015-06-29 15:49:40,896 [myid:] - INFO [main:Environment#100] - Server environment:java.vendor=Oracle Corporation
2015-06-29 15:49:40,900 [myid:] - INFO [main:Environment#100] - Server environment:java.home=/usr/java/jdk1.8.0_25/jre
2015-06-29 15:49:40,901 [myid:] - INFO [main:Environment#100] - Server environment:java.class.path=/opt/zookeeper-3.4.6/bin/../build/classes:/opt
I omit the huge path string that follows, and give below the remaining log:
2015-06-29 15:49:40,902 [myid:] - INFO [main:Environment#100] - Server environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
2015-06-29 15:49:40,902 [myid:] - INFO [main:Environment#100] - Server environment:java.io.tmpdir=/tmp
2015-06-29 15:49:40,902 [myid:] - INFO [main:Environment#100] - Server environment:java.compiler=<NA>
2015-06-29 15:49:40,903 [myid:] - INFO [main:Environment#100] - Server environment:os.name=Linux
2015-06-29 15:49:40,903 [myid:] - INFO [main:Environment#100] - Server environment:os.arch=amd64
2015-06-29 15:49:40,903 [myid:] - INFO [main:Environment#100] - Server environment:os.version=3.17.8-200.fc20.x86_64
2015-06-29 15:49:40,903 [myid:] - INFO [main:Environment#100] - Server environment:user.name=root
2015-06-29 15:49:40,904 [myid:] - INFO [main:Environment#100] - Server environment:user.home=/root
2015-06-29 15:49:40,904 [myid:] - INFO [main:Environment#100] - Server environment:user.dir=/root
2015-06-29 15:49:40,909 [myid:] - INFO [main:ZooKeeperServer#755] - tickTime set to 2000
2015-06-29 15:49:40,909 [myid:] - INFO [main:ZooKeeperServer#764] - minSessionTimeout set to -1
2015-06-29 15:49:40,909 [myid:] - INFO [main:ZooKeeperServer#773] - maxSessionTimeout set to -1
2015-06-29 15:49:40,918 [myid:] - INFO [main:NIOServerCnxnFactory#94] - binding to port 0.0.0.0/0.0.0.0:2181
Is it like zookeeper stuck at obtaining bindings to port 2181?
But when I run lsof -i:2181 -s, it outputs:
COMMAND PID USER FD TYPE DEVICE SIZE NODE NAME
java 15360 root 467u IPv6 18340028 TCP *:eforward (LISTEN)
The pid is that of the QuorumPeerMain.
Running source zkServer.sh status on another PC, properly gives
Mode: standalone
But on this PC, I am pretty much screwed. Can anyone help me.
Have you validated that the config file is fine?
Try to run Zookeeper with the config file full path
Example: zkServer.sh start /etc/zookeeper/conf/zoo.cfg
It seems some problem with the PORT binding
"binding to port 0.0.0.0/0.0.0.0:2181"
and the problem should be resolved by adding below the host entry to the hosts file
127.0.0.1 localhost
In my case zookeeper 3.7.0. just need shutdown tomcat server
In my case i not install JAVA
so it was not started
please download JDK and install
then try again
Related
I'm using Confluent Community 6.0.1. Three nodes Kafka cluster:
devKafka04: Kafka Broker1, Zookeeper 1
devKafka05: Kafka Broker2, Zookeeper 2
devKafka06: Kafka Broker3, Zookeeper 3
The SSL encryption is already working well on the Kafka Brokers.
I'd like to add SASL to enable mutual authentication between Kafka and Zookeeper.
I was following the Confluent document:
https://docs.confluent.io/platform/current/kafka/incremental-security-upgrade.html#adding-security-to-a-running-zk-cluster
[Updates] After I applied the changes, Zookeeper could not start on the secureclientPort. That's why the Kafka broker couldn't start. Here are the error log and docker compose configurations.
I'm wondering if there's something with the confluent zookeeper image.
Please help me out. Thanks.
$ sudo docker logs zookeeper
===> User
uid=1000(appuser) gid=1000(appuser) groups=1000(appuser)
===> Configuring ...
===> Running preflight checks ...
===> Check if /var/lib/zookeeper/data is writable ...
===> Check if /var/lib/zookeeper/log is writable ...
===> Launching ...
===> Printing /var/lib/zookeeper/data/myid
1===> Launching zookeeper ...
[2021-03-24 19:03:08,857] INFO Reading configuration from: /etc/kafka/zookeeper.properties (org.apache.zookeeper.server.quorum.QuorumPeerConfig)
[2021-03-24 19:03:08,862] INFO clientPortAddress is 0.0.0.0:2181 (org.apache.zookeeper.server.quorum.QuorumPeerConfig)
[2021-03-24 19:03:08,862] INFO secureClientPort is not set (org.apache.zookeeper.server.quorum.QuorumPeerConfig)
[2021-03-24 19:03:08,876] INFO autopurge.snapRetainCount set to 3 (org.apache.zookeeper.server.DatadirCleanupManager)
[2021-03-24 19:03:08,876] INFO autopurge.purgeInterval set to 0 (org.apache.zookeeper.server.DatadirCleanupManager)
[2021-03-24 19:03:08,876] INFO Purge task is not scheduled. (org.apache.zookeeper.server.DatadirCleanupManager)
[2021-03-24 19:03:08,880] INFO Log4j 1.2 jmx support found and enabled. (org.apache.zookeeper.jmx.ManagedUtil)
[2021-03-24 19:03:08,904] INFO Starting quorum peer (org.apache.zookeeper.server.quorum.QuorumPeerMain)
[2021-03-24 19:03:08,909] INFO Using org.apache.zookeeper.server.NIOServerCnxnFactory as server connection factory (org.apache.zookeeper.server.ServerCnxnFactory)
[2021-03-24 19:03:08,917] INFO Setting -D jdk.tls.rejectClientInitiatedRenegotiation=true to disable client-initiated TLS renegotiation (org.apache.zookeeper.common.X509Util)
[2021-03-24 19:03:08,953] INFO Server successfully logged in. (org.apache.zookeeper.Login)
[2021-03-24 19:03:08,957] INFO Configuring NIO connection handler with 10s sessionless connection timeout, 1 selector thread(s), 8 worker threads, and 64 kB direct buffers. (org.apache.zookeeper.server.NIOServerCnxnFactory)
[2021-03-24 19:03:08,961] INFO binding to port 0.0.0.0/0.0.0.0:2181 (org.apache.zookeeper.server.NIOServerCnxnFactory)
[2021-03-24 19:03:08,986] INFO Logging initialized #929ms to org.eclipse.jetty.util.log.Slf4jLog (org.eclipse.jetty.util.log)
[2021-03-24 19:03:09,081] WARN o.e.j.s.ServletContextHandler#6c2c1385{/,null,UNAVAILABLE} contextPath ends with /* (org.eclipse.jetty.server.handler.ContextHandler)
[2021-03-24 19:03:09,082] WARN Empty contextPath (org.eclipse.jetty.server.handler.ContextHandler)
[2021-03-24 19:03:09,097] INFO zookeeper.snapshot.trust.empty : false (org.apache.zookeeper.server.persistence.FileTxnSnapLog)
[2021-03-24 19:03:09,102] INFO Local sessions disabled (org.apache.zookeeper.server.quorum.QuorumPeer)
[2021-03-24 19:03:09,102] INFO Local session upgrading disabled (org.apache.zookeeper.server.quorum.QuorumPeer)
[2021-03-24 19:03:09,102] INFO tickTime set to 3000 (org.apache.zookeeper.server.quorum.QuorumPeer)
[2021-03-24 19:03:09,102] INFO minSessionTimeout set to 6000 (org.apache.zookeeper.server.quorum.QuorumPeer)
[2021-03-24 19:03:09,102] INFO maxSessionTimeout set to 60000 (org.apache.zookeeper.server.quorum.QuorumPeer)
[2021-03-24 19:03:09,102] INFO initLimit set to 10 (org.apache.zookeeper.server.quorum.QuorumPeer)
[2021-03-24 19:03:09,115] INFO zookeeper.snapshotSizeFactor = 0.33 (org.apache.zookeeper.server.ZKDatabase)
[2021-03-24 19:03:09,116] INFO Using insecure (non-TLS) quorum communication (org.apache.zookeeper.server.quorum.QuorumPeer)
[2021-03-24 19:03:09,117] INFO Port unification disabled (org.apache.zookeeper.server.quorum.QuorumPeer)
[2021-03-24 19:03:09,117] INFO QuorumPeer communication is not secured! (SASL auth disabled) (org.apache.zookeeper.server.quorum.QuorumPeer)
[2021-03-24 19:03:09,117] INFO quorum.cnxn.threads.size set to 20 (org.apache.zookeeper.server.quorum.QuorumPeer)
[2021-03-24 19:03:09,118] INFO Reading snapshot /var/lib/zookeeper/data/version-2/snapshot.a00000000 (org.apache.zookeeper.server.persistence.FileSnap)
[2021-03-24 19:03:09,213] INFO jetty-9.4.24.v20191120; built: 2019-11-20T21:37:49.771Z; git: 363d5f2df3a8a28de40604320230664b9c793c16; jvm 11.0.9.1+1-LTS (org.eclipse.jetty.server.Server)
[2021-03-24 19:03:09,261] INFO DefaultSessionIdManager workerName=node0 (org.eclipse.jetty.server.session)
[2021-03-24 19:03:09,261] INFO No SessionScavenger set, using defaults (org.eclipse.jetty.server.session)
[2021-03-24 19:03:09,263] INFO node0 Scavenging every 660000ms (org.eclipse.jetty.server.session)
[2021-03-24 19:03:09,272] INFO Started o.e.j.s.ServletContextHandler#6c2c1385{/,null,AVAILABLE} (org.eclipse.jetty.server.handler.ContextHandler)
[2021-03-24 19:03:09,281] INFO Started ServerConnector#6d07a63d{HTTP/1.1,[http/1.1]}{0.0.0.0:8080} (org.eclipse.jetty.server.AbstractConnector)
[2021-03-24 19:03:09,281] INFO Started #1224ms (org.eclipse.jetty.server.Server)
[2021-03-24 19:03:09,281] INFO Started AdminServer on address 0.0.0.0, port 8080 and command URL /commands (org.apache.zookeeper.server.admin.JettyAdminServer)
[2021-03-24 19:03:09,288] INFO Election port bind maximum retries is 3 (org.apache.zookeeper.server.quorum.QuorumCnxManager)
[2021-03-24 19:03:09,290] INFO 1 is accepting connections now, my election bind port: devkafka04/172.16.87.141:3888 (org.apache.zookeeper.server.quorum.QuorumCnxManager)
[2021-03-24 19:03:09,301] INFO LOOKING (org.apache.zookeeper.server.quorum.QuorumPeer)
[2021-03-24 19:03:09,303] INFO New election. My id = 1, proposed zxid=0x1600000030 (org.apache.zookeeper.server.quorum.FastLeaderElection)
[2021-03-24 19:03:09,308] INFO Notification: 2 (message format version), 1 (n.leader), 0x1600000030 (n.zxid), 0x1 (n.round), LOOKING (n.state), 1 (n.sid), 0x16 (n.peerEPoch), LOOKING (my state)0 (n.config version) (org.apache.zookeeper.server.quorum.FastLeaderElection)
[2021-03-24 19:03:09,310] INFO Have smaller server identifier, so dropping the connection: (myId:1 --> sid:3) (org.apache.zookeeper.server.quorum.QuorumCnxManager)
[2021-03-24 19:03:09,312] INFO Received connection request from /172.16.87.143:53340 (org.apache.zookeeper.server.quorum.QuorumCnxManager)
[2021-03-24 19:03:09,315] INFO Have smaller server identifier, so dropping the connection: (myId:1 --> sid:2) (org.apache.zookeeper.server.quorum.QuorumCnxManager)
[2021-03-24 19:03:09,316] INFO Notification: 2 (message format version), 2 (n.leader), 0x150000002b (n.zxid), 0xa (n.round), FOLLOWING (n.state), 3 (n.sid), 0x16 (n.peerEPoch), LOOKING (my state)0 (n.config version) (org.apache.zookeeper.server.quorum.FastLeaderElection)
[2021-03-24 19:03:09,317] INFO Received connection request from /172.16.87.142:51704 (org.apache.zookeeper.server.quorum.QuorumCnxManager)
[2021-03-24 19:03:09,319] INFO Notification: 2 (message format version), 2 (n.leader), 0x150000002b (n.zxid), 0xa (n.round), LEADING (n.state), 2 (n.sid), 0x16 (n.peerEPoch), LOOKING (my state)0 (n.config version) (org.apache.zookeeper.server.quorum.FastLeaderElection)
[2021-03-24 19:03:09,320] INFO Notification: 2 (message format version), 2 (n.leader), 0x150000002b (n.zxid), 0xa (n.round), FOLLOWING (n.state), 3 (n.sid), 0x16 (n.peerEPoch), LOOKING (my state)0 (n.config version) (org.apache.zookeeper.server.quorum.FastLeaderElection)
[2021-03-24 19:03:09,320] INFO FOLLOWING (org.apache.zookeeper.server.quorum.QuorumPeer)
[2021-03-24 19:03:09,323] INFO Notification: 2 (message format version), 2 (n.leader), 0x150000002b (n.zxid), 0xa (n.round), LEADING (n.state), 2 (n.sid), 0x16 (n.peerEPoch), FOLLOWING (my state)0 (n.config version) (org.apache.zookeeper.server.quorum.FastLeaderElection)
[2021-03-24 19:03:09,330] INFO TCP NoDelay set to: true (org.apache.zookeeper.server.quorum.Learner)
[2021-03-24 19:03:09,336] INFO Server environment:zookeeper.version=3.5.8-f439ca583e70862c3068a1f2a7d4d068eec33315, built on 05/04/2020 15:53 GMT (org.apache.zookeeper.server.ZooKeeperServer)
[2021-03-24 19:03:09,336] INFO Server environment:host.name=devkafka04 (org.apache.zookeeper.server.ZooKeeperServer)
[2021-03-24 19:03:09,336] INFO Server environment:java.version=11.0.9.1 (org.apache.zookeeper.server.ZooKeeperServer)
[2021-03-24 19:03:09,336] INFO Server environment:java.vendor=Azul Systems, Inc. (org.apache.zookeeper.server.ZooKeeperServer)
[2021-03-24 19:03:09,336] INFO Server environment:java.home=/usr/lib/jvm/zulu11-ca (org.apache.zookeeper.server.ZooKeeperServer)
[2021-03-24 19:03:09,336] INFO Server environment:java.class.path=/usr/bin/../share/java/kafka/activation-1.1.1.jar:/usr/bin/../share/java/kafka/aopalliance-repackaged-2.6.1.jar:/usr/bin/../share/java/kafka/argparse4j-0.7.0.jar:/usr/bin/../share/java/kafka/audience-annotations-0.5.0.jar:/usr/bin/../share/java/kafka/commons-cli-1.4.jar:/usr/bin/../share/java/kafka/commons-lang3-3.8.1.jar:/usr/bin/../share/java/kafka/confluent-log4j-1.2.17-cp2.jar:/usr/bin/../share/java/kafka/connect-api-6.0.1-ccs.jar:/usr/bin/../share/java/kafka/connect-basic-auth-extension-6.0.1-ccs.jar:/usr/bin/../share/java/kafka/connect-file-6.0.1-ccs.jar:/usr/bin/../share/java/kafka/connect-json-6.0.1-ccs.jar:/usr/bin/../share/java/kafka/connect-mirror-6.0.1-ccs.jar:/usr/bin/../share/java/kafka/connect-mirror-client-6.0.1-ccs.jar:/usr/bin/../share/java/kafka/connect-runtime-6.0.1-ccs.jar:/usr/bin/../share/java/kafka/connect-transforms-6.0.1-ccs.jar:/usr/bin/../share/java/kafka/hk2-api-2.6.1.jar:/usr/bin/../share/java/kafka/hk2-locator-2.6.1.jar:/usr/bin/../share/java/kafka/hk2-utils-2.6.1.jar:/usr/bin/../share/java/kafka/jackson-annotations-2.10.5.jar:/usr/bin/../share/java/kafka/jackson-core-2.10.5.jar:/usr/bin/../share/java/kafka/jackson-databind-2.10.5.jar:/usr/bin/../share/java/kafka/jackson-dataformat-csv-2.10.5.jar:/usr/bin/../share/java/kafka/jackson-datatype-jdk8-2.10.5.jar:/usr/bin/../share/java/kafka/jackson-jaxrs-base-2.10.5.jar:/usr/bin/../share/java/kafka/jackson-jaxrs-json-provider-2.10.5.jar:/usr/bin/../share/java/kafka/jackson-module-jaxb-annotations-2.10.5.jar:/usr/bin/../share/java/kafka/jackson-module-paranamer-2.10.5.jar:/usr/bin/../share/java/kafka/jackson-module-scala_2.13-2.10.5.jar:/usr/bin/../share/java/kafka/jakarta.activation-api-1.2.1.jar:/usr/bin/../share/java/kafka/jakarta.annotation-api-1.3.5.jar:/usr/bin/../share/java/kafka/jakarta.inject-2.6.1.jar:/usr/bin/../share/java/kafka/jakarta.validation-api-2.0.2.jar:/usr/bin/../share/java/kafka/jakarta.ws.rs-api-2.1.6.jar:/usr/bin/../share/java/kafka/jakarta.xml.bind-api-2.3.2.jar:/usr/bin/../share/java/kafka/javassist-3.25.0-GA.jar:/usr/bin/../share/java/kafka/javassist-3.26.0-GA.jar:/usr/bin/../share/java/kafka/javax.servlet-api-3.1.0.jar:/usr/bin/../share/java/kafka/javax.ws.rs-api-2.1.1.jar:/usr/bin/../share/java/kafka/jaxb-api-2.3.0.jar:/usr/bin/../share/java/kafka/jersey-client-2.30.jar:/usr/bin/../share/java/kafka/jersey-common-2.30.jar:/usr/bin/../share/java/kafka/jersey-container-servlet-2.30.jar:/usr/bin/../share/java/kafka/jersey-container-servlet-core-2.30.jar:/usr/bin/../share/java/kafka/jersey-hk2-2.30.jar:/usr/bin/../share/java/kafka/jersey-media-jaxb-2.30.jar:/usr/bin/../share/java/kafka/jersey-server-2.30.jar:/usr/bin/../share/java/kafka/jetty-client-9.4.24.v20191120.jar:/usr/bin/../share/java/kafka/jetty-continuation-9.4.24.v20191120.jar:/usr/bin/../share/java/kafka/jetty-http-9.4.24.v20191120.jar:/usr/bin/../share/java/kafka/jetty-io-9.4.24.v20191120.jar:/usr/bin/../share/java/kafka/jetty-security-9.4.24.v20191120.jar:/usr/bin/../share/java/kafka/jetty-server-9.4.24.v20191120.jar:/usr/bin/../share/java/kafka/jetty-servlet-9.4.24.v20191120.jar:/usr/bin/../share/java/kafka/jetty-servlets-9.4.24.v20191120.jar:/usr/bin/../share/java/kafka/jetty-util-9.4.24.v20191120.jar:/usr/bin/../share/java/kafka/jopt-simple-5.0.4.jar:/usr/bin/../share/java/kafka/kafka-clients-6.0.1-ccs.jar:/usr/bin/../share/java/kafka/kafka-log4j-appender-6.0.1-ccs.jar:/usr/bin/../share/java/kafka/kafka-streams-6.0.1-ccs.jar:/usr/bin/../share/java/kafka/kafka-streams-examples-6.0.1-ccs.jar:/usr/bin/../share/java/kafka/kafka-streams-scala_2.13-6.0.1-ccs.jar:/usr/bin/../share/java/kafka/kafka-streams-test-utils-6.0.1-ccs.jar:/usr/bin/../share/java/kafka/kafka-tools-6.0.1-ccs.jar:/usr/bin/../share/java/kafka/kafka.jar:/usr/bin/../share/java/kafka/kafka_2.13-6.0.1-ccs-javadoc.jar:/usr/bin/../share/java/kafka/kafka_2.13-6.0.1-ccs-scaladoc.jar:/usr/bin/../share/java/kafka/kafka_2.13-6.0.1-ccs-sources.jar:/usr/bin/../share/java/kafka/kafka_2.13-6.0.1-ccs-test-sources.jar:/usr/bin/../share/java/kafka/kafka_2.13-6.0.1-ccs-test.jar:/usr/bin/../share/java/kafka/kafka_2.13-6.0.1-ccs.jar:/usr/bin/../share/java/kafka/lz4-java-1.7.1.jar:/usr/bin/../share/java/kafka/maven-artifact-3.6.3.jar:/usr/bin/../share/java/kafka/metrics-core-2.2.0.jar:/usr/bin/../share/java/kafka/netty-buffer-4.1.50.Final.jar:/usr/bin/../share/java/kafka/netty-codec-4.1.50.Final.jar:/usr/bin/../share/java/kafka/netty-common-4.1.50.Final.jar:/usr/bin/../share/java/kafka/netty-handler-4.1.50.Final.jar:/usr/bin/../share/java/kafka/netty-resolver-4.1.50.Final.jar:/usr/bin/../share/java/kafka/netty-transport-4.1.50.Final.jar:/usr/bin/../share/java/kafka/netty-transport-native-epoll-4.1.50.Final.jar:/usr/bin/../share/java/kafka/netty-transport-native-unix-common-4.1.50.Final.jar:/usr/bin/../share/java/kafka/osgi-resource-locator-1.0.3.jar:/usr/bin/../share/java/kafka/paranamer-2.8.jar:/usr/bin/../share/java/kafka/plexus-utils-3.2.1.jar:/usr/bin/../share/java/kafka/reflections-0.9.12.jar:/usr/bin/../share/java/kafka/rocksdbjni-5.18.4.jar:/usr/bin/../share/java/kafka/scala-collection-compat_2.13-2.1.6.jar:/usr/bin/../share/java/kafka/scala-java8-compat_2.13-0.9.1.jar:/usr/bin/../share/java/kafka/scala-library-2.13.2.jar:/usr/bin/../share/java/kafka/slf4j-api-1.7.30.jar:/usr/bin/../share/java/kafka/scala-logging_2.13-3.9.2.jar:/usr/bin/../share/java/kafka/scala-reflect-2.13.2.jar:/usr/bin/../share/java/kafka/slf4j-log4j12-1.7.30.jar:/usr/bin/../share/java/kafka/snappy-java-1.1.7.3.jar:/usr/bin/../share/java/kafka/zookeeper-3.5.8.jar:/usr/bin/../share/java/kafka/zookeeper-jute-3.5.8.jar:/usr/bin/../share/java/kafka/zstd-jni-1.4.4-7.jar:/usr/bin/../share/java/confluent-telemetry/* (org.apache.zookeeper.server.ZooKeeperServer)
[2021-03-24 19:03:09,336] INFO Server environment:java.library.path=/usr/java/packages/lib:/usr/lib64:/lib64:/lib:/usr/lib (org.apache.zookeeper.server.ZooKeeperServer)
[2021-03-24 19:03:09,336] INFO Server environment:java.io.tmpdir=/tmp (org.apache.zookeeper.server.ZooKeeperServer)
[2021-03-24 19:03:09,336] INFO Server environment:java.compiler=<NA> (org.apache.zookeeper.server.ZooKeeperServer)
[2021-03-24 19:03:09,337] INFO Server environment:os.name=Linux (org.apache.zookeeper.server.ZooKeeperServer)
[2021-03-24 19:03:09,337] INFO Server environment:os.arch=amd64 (org.apache.zookeeper.server.ZooKeeperServer)
[2021-03-24 19:03:09,337] INFO Server environment:os.version=3.10.0-1160.21.1.el7.x86_64 (org.apache.zookeeper.server.ZooKeeperServer)
[2021-03-24 19:03:09,337] INFO Server environment:user.name=appuser (org.apache.zookeeper.server.ZooKeeperServer)
[2021-03-24 19:03:09,337] INFO Server environment:user.home=/home/appuser (org.apache.zookeeper.server.ZooKeeperServer)
[2021-03-24 19:03:09,337] INFO Server environment:user.dir=/home/appuser (org.apache.zookeeper.server.ZooKeeperServer)
[2021-03-24 19:03:09,337] INFO Server environment:os.memory.free=498MB (org.apache.zookeeper.server.ZooKeeperServer)
[2021-03-24 19:03:09,337] INFO Server environment:os.memory.max=512MB (org.apache.zookeeper.server.ZooKeeperServer)
[2021-03-24 19:03:09,337] INFO Server environment:os.memory.total=512MB (org.apache.zookeeper.server.ZooKeeperServer)
[2021-03-24 19:03:09,338] INFO minSessionTimeout set to 6000 (org.apache.zookeeper.server.ZooKeeperServer)
[2021-03-24 19:03:09,339] INFO maxSessionTimeout set to 60000 (org.apache.zookeeper.server.ZooKeeperServer)
[2021-03-24 19:03:09,339] INFO Created server with tickTime 3000 minSessionTimeout 6000 maxSessionTimeout 60000 datadir /var/lib/zookeeper/log/version-2 snapdir /var/lib/zookeeper/data/version-2 (org.apache.zookeeper.server.ZooKeeperServer)
[2021-03-24 19:03:09,339] INFO FOLLOWING - LEADER ELECTION TOOK - 18 MS (org.apache.zookeeper.server.quorum.Learner)
[2021-03-24 19:03:09,345] INFO Getting a diff from the leader 0x1600000030 (org.apache.zookeeper.server.quorum.Learner)
[2021-03-24 19:03:09,350] INFO Learner received NEWLEADER message (org.apache.zookeeper.server.quorum.Learner)
[2021-03-24 19:03:09,363] INFO Learner received UPTODATE message (org.apache.zookeeper.server.quorum.Learner)
[2021-03-24 19:03:09,367] INFO Configuring CommitProcessor with 4 worker threads. (org.apache.zookeeper.server.quorum.CommitProcessor)
$ sudo docker logs kafka
===> User
uid=1000(appuser) gid=1000(appuser) groups=1000(appuser)
===> Configuring ...
SSL is enabled.
SASL is enabled.
===> Running preflight checks ...
===> Check if /var/lib/kafka/data is writable ...
===> Skipping Zookeeper health check for SSL connections...
===> Launching ...
===> Launching kafka ...
[2021-03-23 21:43:43,453] INFO Registered kafka:type=kafka.Log4jController MBean (kafka.utils.Log4jControllerRegistration$)
[2021-03-23 21:43:43,838] INFO Setting -D jdk.tls.rejectClientInitiatedRenegotiation=true to disable client-initiated TLS renegotiation (org.apache.zookeeper.common.X509Util)
[2021-03-23 21:43:43,900] INFO Registered signal handlers for TERM, INT, HUP (org.apache.kafka.common.utils.LoggingSignalHandler)
[2021-03-23 21:43:43,904] INFO starting (kafka.server.KafkaServer)
[2021-03-23 21:43:43,905] INFO Connecting to zookeeper on devkafka04:2182,devkafka05:2182,devkafka06:2182 (kafka.server.KafkaServer)
[2021-03-23 21:43:43,927] INFO [ZooKeeperClient Kafka server] Initializing a new session to devkafka04:2182,devkafka05:2182,devkafka06:2182. (kafka.zookeeper.ZooKeeperClient)
[2021-03-23 21:43:43,934] INFO Client environment:zookeeper.version=3.5.8-f439ca583e70862c3068a1f2a7d4d068eec33315, built on 05/04/2020 15:53 GMT (org.apache.zookeeper.ZooKeeper)
[2021-03-23 21:43:43,934] INFO Client environment:host.name=devkafka04 (org.apache.zookeeper.ZooKeeper)
[2021-03-23 21:43:43,934] INFO Client environment:java.version=11.0.9.1 (org.apache.zookeeper.ZooKeeper)
[2021-03-23 21:43:43,934] INFO Client environment:java.vendor=Azul Systems, Inc. (org.apache.zookeeper.ZooKeeper)
------ Repeating lines removed ---------
'Client' (org.apache.zookeeper.ClientCnxn)
[2021-03-23 21:43:59,947] INFO Socket error occurred: devkafka05/172.16.87.142:2182: Connection refused (org.apache.zookeeper.ClientCnxn)
[2021-03-23 21:44:01,048] INFO Client successfully logged in. (org.apache.zookeeper.Login)
[2021-03-23 21:44:01,048] INFO Client will use DIGEST-MD5 as SASL mechanism. (org.apache.zookeeper.client.ZooKeeperSaslClient)
[2021-03-23 21:44:01,048] INFO Opening socket connection to server devkafka04/172.16.87.141:2182. Will attempt to SASL-authenticate using Login Context section 'Client' (org.apache.zookeeper.ClientCnxn)
[2021-03-23 21:44:01,049] INFO Socket error occurred: devkafka04/172.16.87.141:2182: Connection refused (org.apache.zookeeper.ClientCnxn)
[2021-03-23 21:44:01,150] INFO Client successfully logged in. (org.apache.zookeeper.Login)
[2021-03-23 21:44:01,150] INFO Client will use DIGEST-MD5 as SASL mechanism. (org.apache.zookeeper.client.ZooKeeperSaslClient)
[2021-03-23 21:44:01,150] INFO Opening socket connection to server devkafka06/172.16.87.143:2182. Will attempt to SASL-authenticate using Login Context section 'Client' (org.apache.zookeeper.ClientCnxn)
[2021-03-23 21:44:01,153] INFO Socket error occurred: devkafka06/172.16.87.143:2182: Connection refused (org.apache.zookeeper.ClientCnxn)
[2021-03-23 21:44:01,254] INFO Client successfully logged in. (org.apache.zookeeper.Login)
[2021-03-23 21:44:01,254] INFO Client will use DIGEST-MD5 as SASL mechanism. (org.apache.zookeeper.client.ZooKeeperSaslClient)
[2021-03-23 21:44:01,254] INFO Opening socket connection to server devkafka05/172.16.87.142:2182. Will attempt to SASL-authenticate using Login Context section 'Client' (org.apache.zookeeper.ClientCnxn)
[2021-03-23 21:44:01,255] INFO Socket error occurred: devkafka05/172.16.87.142:2182: Connection refused (org.apache.zookeeper.ClientCnxn)
[2021-03-23 21:44:01,952] INFO [ZooKeeperClient Kafka server] Closing. (kafka.zookeeper.ZooKeeperClient)
[2021-03-23 21:44:02,356] INFO Client successfully logged in. (org.apache.zookeeper.Login)
[2021-03-23 21:44:02,357] INFO Client will use DIGEST-MD5 as SASL mechanism. (org.apache.zookeeper.client.ZooKeeperSaslClient)
[2021-03-23 21:44:02,357] INFO Opening socket connection to server devkafka04/172.16.87.141:2182. Will attempt to SASL-authenticate using Login Context section 'Client' (org.apache.zookeeper.ClientCnxn)
[2021-03-23 21:44:02,462] INFO Session: 0x0 closed (org.apache.zookeeper.ZooKeeper)
[2021-03-23 21:44:02,463] INFO EventThread shut down for session: 0x0 (org.apache.zookeeper.ClientCnxn)
[2021-03-23 21:44:02,465] INFO [ZooKeeperClient Kafka server] Closed. (kafka.zookeeper.ZooKeeperClient)
[2021-03-23 21:44:02,469] ERROR Fatal error during KafkaServer startup. Prepare to shutdown (kafka.server.KafkaServer)
kafka.zookeeper.ZooKeeperClientTimeoutException: Timed out waiting for connection while in state: CONNECTING
at kafka.zookeeper.ZooKeeperClient.waitUntilConnected(ZooKeeperClient.scala:262)
at kafka.zookeeper.ZooKeeperClient.<init>(ZooKeeperClient.scala:119)
at kafka.zk.KafkaZkClient$.apply(KafkaZkClient.scala:1865)
at kafka.server.KafkaServer.createZkClient$1(KafkaServer.scala:419)
at kafka.server.KafkaServer.initZkClient(KafkaServer.scala:444)
at kafka.server.KafkaServer.startup(KafkaServer.scala:222)
at kafka.server.KafkaServerStartable.startup(KafkaServerStartable.scala:44)
at kafka.Kafka$.main(Kafka.scala:82)
at kafka.Kafka.main(Kafka.scala)
[2021-03-23 21:44:02,471] INFO shutting down (kafka.server.KafkaServer)
[2021-03-23 21:44:02,478] INFO shut down completed (kafka.server.KafkaServer)
[2021-03-23 21:44:02,478] ERROR Exiting Kafka. (kafka.server.KafkaServerStartable)
[2021-03-23 21:44:02,479] INFO shutting down (kafka.server.KafkaServer)
$ sudo cat kafka-docker-compose.yml
version: '3'
services:
kafka:
image: confluentinc/cp-kafka:6.0.1
container_name: kafka
network_mode: host
restart: always
ports:
- "9092:9092"
- "9093:9093"
- "9094:9094"
- "49998:49998"
- "49999:49999"
environment:
KAFKA_BROKER_ID: 1
KAFKA_ZOOKEEPER_CONNECT: 'devkafka04:2182,devkafka05:2182,devkafka06:2182'
KAFKA_ZOOKEEPER_SSL_CLIENT_ENABLE: 'true'
KAFKA_ZOOKEEPER_CLIENTCNXNSOCKET: org.apache.zookeeper.ClientCnxnSocketNetty
KAFKA_ZOOKEEPER_SSL_TRUSTSTORE_FILENAME: kafka.server.truststore.jks
KAFKA_ZOOKEEPER_SSL_TRUSTSTORE_CREDENTIALS: creds
KAFKA_ZOOKEEPER_SET_ACL: 'true'
KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://devkafka04:9092,SSL://devkafka04:9093,SASL_SSL://devkafka04:9094
KAFKA_LISTENERS: PLAINTEXT://devkafka04:9092,SSL://devkafka04:9093,SASL_SSL://devkafka04:9094
KAFKA_SASL_ENABLED_MECHANISMS: DIGEST-MD5
KAFKA_SECURITY_INTER_BROKER_PROTOCOL: SSL
KAFKA_SSL_CLIENT_AUTH: requested
KAFKA_SSL_TRUSTSTORE_FILENAME: kafka.server.truststore.jks
KAFKA_SSL_TRUSTSTORE_CREDENTIALS: creds
KAFKA_SSL_KEYSTORE_FILENAME: devkafka04.server.keystore.jks
KAFKA_SSL_KEYSTORE_CREDENTIALS: creds
KAFKA_SSL_KEY_CREDENTIALS: creds
KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 3
KAFKA_GROUP_INITIAL_REBALANCE_DELAY_MS: 0
KAFKA_AUTO_CREATE_TOPICS_ENABLE: "false"
KAFKA_OPTS: -Djava.security.auth.login.config=/etc/kafka/jmx/kafka_server_jaas.conf -Djava.rmi.server.hostname=devkafka04 -Dcom.sun.management.jmxremote.local.only=false -Dcom.sun.management.jmxremote.rmi.port=49998 -Dcom.sun.management.jmxremote.port=49998 -Dcom.sun.management.jmxremote=true -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -javaagent:/etc/kafka/jmx/jmx_prometheus_javaagent-0.14.0.jar=49999:/etc/kafka/jmx/kafka-2_0_0.yml
CONFLUENT_SUPPORT_METRICS_ENABLE: "false"
volumes:
- /media/kafka/data:/var/lib/kafka/data
- /media/kafka/secrets:/etc/kafka/secrets
- /usr/local/src/kafka/jmx:/etc/kafka/jmx
$ sudo cat jmx/kafka_server_jaas.conf
KafkaServer {
org.apache.kafka.common.security.plain.PlainLoginModule required
username="kafkabroker"
password="kafkabroker-secret"
user_kafkabroker="kafkabroker-secret"
user_kafka-broker-metric-reporter="kafkabroker-metric-reporter-secret"
user_client="client-secret";
};
Client {
org.apache.zookeeper.server.auth.DigestLoginModule required
username="kafka"
password="kafka-secret";
};
$ sudo cat zookeeper-docker-compose.yml
version: '3'
services:
zookeeper:
image: confluentinc/cp-zookeeper:6.0.1
container_name: zookeeper
network_mode: host
restart: always
ports:
- "2181:2181"
- "2182:2182"
- "2888:2888"
- "3888:3888"
- "39998:39998"
- "39999:39999"
environment:
ZOOKEEPER_SERVER_ID: 1
ZOOKEEPER_CLIENT_PORT: 2181
ZOOKEEPER_SERVERS: devkafka04:2888:3888;devkafka05:2888:3888;devkafka06:2888:3888
ZOOKEEPER_AUTHPROVIDER_SASL: org.apache.zookeeper.server.auth.SASLAuthenticationProvider
ZOOKEEPER_AUTHPROVIDER_x509: org.apache.zookeeper.server.auth.X509AuthenticationProvider
ZOOKEEPER_SECURECLIENTPORT: 2182
ZOOKEEPER_SERVERCNXNFACTORY: org.apache.zookeeper.server.NettyServerCnxnFactory
ZOOKEEPER_SSL_TRUSTSTORE_FILENAME: kafka.server.truststore.jks
ZOOKEEPER_SSL_TRUSTSTORE_CREDENTIALS: creds
ZOOKEEPER_SSL_KEYSTORE_FILENAME: devkafka05.server.keystore.jks
ZOOKEEPER_SSL_KEYSTORE_CREDENTIALS: creds
ZOOKEEPER_SSL_KEY_CREDENTIALS: creds
ZOOKEEPER_SSL_CLIENTAUTH: none
KAFKA_OPTS: -Djava.security.auth.login.config=/etc/zookeeper/jmx/zookeeper_jaas.conf -Dzookeeper.4lw.commands.whitelist=* -Djava.rmi.server.hostname=devkafka04 -Dcom.sun.management.jmxremote.local.only=false -Dcom.sun.management.jmxremote.rmi.port=39998 -Dcom.sun.management.jmxremote.port=39998 -Dcom.sun.management.jmxremote=true -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -javaagent:/etc/zookeeper/jmx/jmx_prometheus_javaagent-0.14.0.jar=39999:/etc/zookeeper/jmx/jmx-zookeeper-prometheus.yaml
volumes:
- /media/zookeeper/data:/var/lib/zookeeper/data
- /media/zookeeper/log:/var/lib/zookeeper/log
- /media/zookeeper/secrets:/etc/zookeeper/secrets
- /usr/local/src/zookeeper/jmx:/etc/zookeeper/jmx
$ sudo cat jmx/zookeeper_jaas.conf
Server {
org.apache.zookeeper.server.auth.DigestLoginModule required
user_kafka="kafka-secret";
};
Try using KAFKA_ZOOKEEPER_CLIENT_CNXN_SOCKET instead of KAFKA_ZOOKEEPER_CLIENTCNXNSOCKET.
So I have secured a kafka cluster through below security mechanism in server.properties and added respective kafka and zookeeper jaas.config files
security.inter.broker.protocol=SASL_PLAINTEXT
sasl.mechanism.inter.broker.protocol=PLAIN
and using below command to start the kafka tool
kafkatool -J -Djava.security.auth.login.config=client_jaas.config
where client_jaas.config consists of below content
sasl.mechanism=PLAIN
# Configure SASL_SSL if SSL encryption is enabled, otherwise configure SASL_PLAINTEXT
security.protocol=SASL_PLAINTEXT
sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required \
username="admin" \
password="password";
Kafkatool gets launched successfully cluster is being connected but when click on topic to see list of all the topic it gives me below error
Configured to use PLAINTTEXT but broker 0 does not support it.
Below is the content of the server_jaas.config file
KafkaServer {
org.apache.kafka.common.security.plain.PlainLoginModule required
username="admin"
password="password"
user_admin="password";
};
Client {
org.apache.kafka.common.security.plain.PlainLoginModule required
username="admin"
password="password";
};
Below are the listeners details in the server.properties file
listeners=SASL_PLAINTEXT://0.0.0.0:9092
advertised.listeners=SASL_PLAINTEXT://:9092
If I see the full error log of kafaktool, of the output of command kafkatool -J-Djava.security.auth.login.config=client_jaas.config, below is what I see
01/Jun/2019 13:40:30.436 INFO com.kafkatool.ui.MainApp - Starting application : Kafka Tool
01/Jun/2019 13:40:30.437 INFO com.kafkatool.ui.MainApp - Version : 2.0.4
01/Jun/2019 13:40:30.437 INFO com.kafkatool.ui.MainApp - Built : Feb 21, 2019
01/Jun/2019 13:40:30.438 INFO com.kafkatool.ui.MainApp - user.home : /home/vivek
01/Jun/2019 13:40:30.438 INFO com.kafkatool.ui.MainApp - user.dir : /opt/kafkatool2
01/Jun/2019 13:40:30.438 INFO com.kafkatool.ui.MainApp - os.name : Linux
01/Jun/2019 13:40:30.438 INFO com.kafkatool.ui.MainApp - java.runtime.version : 1.8.0_191-8u191-b12-2ubuntu0.18.04.1-b12
01/Jun/2019 13:40:30.439 INFO com.kafkatool.ui.MainApp - max memory=1305 MB
01/Jun/2019 13:40:30.439 INFO com.kafkatool.ui.MainApp - available processors=4
01/Jun/2019 13:40:30.440 INFO com.kafkatool.ui.MainApp - java.security.auth.login.config=client_jaas.config
01/Jun/2019 13:40:30.443 INFO com.kafkatool.ui.MainApp - java.security.auth.login.config exists = false
01/Jun/2019 13:40:30.449 INFO com.kafkatool.common.ExternalDecoderManager - Finding plugins in directory /opt/kafkatool2/plugins
01/Jun/2019 13:40:30.449 INFO com.kafkatool.common.ExternalDecoderManager - Found files in plugin directory, count=1
01/Jun/2019 13:40:30.450 INFO com.kafkatool.ui.MainApp - Loading user settings
01/Jun/2019 13:40:30.497 INFO com.kafkatool.ui.MainApp - Loading server group settings
01/Jun/2019 13:40:30.500 INFO com.kafkatool.ui.MainApp - Loading server connection settings
01/Jun/2019 13:40:39.854 INFO org.I0Itec.zkclient.ZkEventThread - Starting ZkClient event thread.
01/Jun/2019 13:40:39.862 INFO org.apache.zookeeper.ZooKeeper - Client environment:zookeeper.version=3.3.3-1203054, built on 11/17/2011 05:47 GMT
01/Jun/2019 13:40:39.862 INFO org.apache.zookeeper.ZooKeeper - Client environment:host.name=anakata
01/Jun/2019 13:40:39.863 INFO org.apache.zookeeper.ZooKeeper - Client environment:java.version=1.8.0_191
01/Jun/2019 13:40:39.863 INFO org.apache.zookeeper.ZooKeeper - Client environment:java.vendor=Oracle Corporation
01/Jun/2019 13:40:39.863 INFO org.apache.zookeeper.ZooKeeper - Client environment:java.home=/usr/lib/jvm/java-8-openjdk-amd64/jre
01/Jun/2019 13:40:39.863 INFO org.apache.zookeeper.ZooKeeper - Client environment:java.class.path=/opt/kafkatool2/.install4j/i4jruntime.jar:/opt/kafkatool2/lib/glazedlists.jar:/opt/kafkatool2/lib/gson-2.3.1.jar:/opt/kafkatool2/lib/jgoodies-common-1.6.0.jar:/opt/kafkatool2/lib/jgoodies-looks-2.5.3.jar:/opt/kafkatool2/lib/jide-oss.jar:/opt/kafkatool2/lib/jline.jar:/opt/kafkatool2/lib/jopt-simple.jar:/opt/kafkatool2/lib/kafka-clients.jar:/opt/kafkatool2/lib/kafka.jar:/opt/kafkatool2/lib/log4j.jar:/opt/kafkatool2/lib/lz4.jar:/opt/kafkatool2/lib/metrics-core.jar:/opt/kafkatool2/lib/netty.jar:/opt/kafkatool2/lib/ofjar.jar:/opt/kafkatool2/lib/quaqua.jar:/opt/kafkatool2/lib/scala-library.jar:/opt/kafkatool2/lib/scala-parser-combinators.jar:/opt/kafkatool2/lib/scala-xml.jar:/opt/kafkatool2/lib/slf4j-api.jar:/opt/kafkatool2/lib/slf4j-log4j12.jar:/opt/kafkatool2/lib/snappy.jar:/opt/kafkatool2/lib/zkclient.jar:/opt/kafkatool2/lib/zookeeper.jar
01/Jun/2019 13:40:39.864 INFO org.apache.zookeeper.ZooKeeper - Client environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib/x86_64-linux-gnu/jni:/lib/x86_64-linux-gnu:/usr/lib/x86_64-linux-gnu:/usr/lib/jni:/lib:/usr/lib
01/Jun/2019 13:40:39.864 INFO org.apache.zookeeper.ZooKeeper - Client environment:java.io.tmpdir=/tmp
01/Jun/2019 13:40:39.864 INFO org.apache.zookeeper.ZooKeeper - Client environment:java.compiler=<NA>
01/Jun/2019 13:40:39.864 INFO org.apache.zookeeper.ZooKeeper - Client environment:os.name=Linux
01/Jun/2019 13:40:39.865 INFO org.apache.zookeeper.ZooKeeper - Client environment:os.arch=amd64
01/Jun/2019 13:40:39.865 INFO org.apache.zookeeper.ZooKeeper - Client environment:os.version=4.15.0-48-generic
01/Jun/2019 13:40:39.865 INFO org.apache.zookeeper.ZooKeeper - Client environment:user.name=vivek
01/Jun/2019 13:40:39.866 INFO org.apache.zookeeper.ZooKeeper - Client environment:user.home=/home/vivek
01/Jun/2019 13:40:39.882 INFO org.apache.zookeeper.ZooKeeper - Client environment:user.dir=/opt/kafkatool2
01/Jun/2019 13:40:39.884 INFO org.apache.zookeeper.ZooKeeper - Initiating client connection, connectString=localhost:2181 sessionTimeout=30000 watcher=org.I0Itec.zkclient.ZkClient#60b5eeff
01/Jun/2019 13:40:39.917 INFO org.apache.zookeeper.ClientCnxn - Opening socket connection to server localhost/127.0.0.1:2181
01/Jun/2019 13:40:39.927 INFO org.apache.zookeeper.ClientCnxn - Socket connection established to localhost/127.0.0.1:2181, initiating session
01/Jun/2019 13:40:39.947 INFO org.apache.zookeeper.ClientCnxn - Session establishment complete on server localhost/127.0.0.1:2181, sessionid = 0x100097569c60010, negotiated timeout = 30000
01/Jun/2019 13:40:39.949 INFO org.I0Itec.zkclient.ZkClient - zookeeper state changed (SyncConnected)
01/Jun/2019 13:40:40.024 INFO com.kafkatool.model.Broker - Parsing endpoint SASL_PLAINTEXT://anakata:9092
01/Jun/2019 13:40:40.025 INFO com.kafkatool.model.KafkaMapper - Getting brokers from zookeeper, found 1
01/Jun/2019 13:40:40.025 INFO org.I0Itec.zkclient.ZkEventThread - Terminate ZkClient event thread.
01/Jun/2019 13:40:40.044 INFO org.apache.zookeeper.ZooKeeper - Session: 0x100097569c60010 closed
01/Jun/2019 13:40:40.044 INFO org.apache.zookeeper.ClientCnxn - EventThread shut down
01/Jun/2019 13:40:40.046 ERROR com.kafkatool.model.Broker - Configured to use PLAINTEXT but broker 0 does not support it. Configured types {SASL_PLAINTEXT=BrokerEndpoint{protocol='SASL_PLAINTEXT', host='anakata', port=9092}}
01/Jun/2019 13:40:40.048 ERROR com.kafkatool.model.KafkaMapper - Error getting topics
java.lang.Exception: Configured to use PLAINTEXT but broker 0 does not support it.
at com.kafkatool.model.Broker.getSecurityTypeConnectString(Broker.java:169)
at com.kafkatool.model.KafkaMapper.getProperties(KafkaMapper.java:258)
at com.kafkatool.model.KafkaMapper.getNewConsumer(KafkaMapper.java:1100)
at com.kafkatool.model.KafkaMapper.getNewConsumer(KafkaMapper.java:1095)
at com.kafkatool.model.KafkaMapper.getNewConsumer(KafkaMapper.java:1085)
at com.kafkatool.model.KafkaMapper.getTopics(KafkaMapper.java:420)
at com.kafkatool.model.ServerConnection.getTopics(ServerConnection.java:221)
at com.kafkatool.model.ServerConnection.hasTopic(ServerConnection.java:203)
at com.kafkatool.model.KafkaMapper.startPoller(KafkaMapper.java:105)
at com.kafkatool.model.ServerConnection.connectInt(ServerConnection.java:336)
at com.kafkatool.model.ServerConnection.connect(ServerConnection.java:315)
at com.kafkatool.common.AsyncServerConnector.run(AsyncServerConnector.java:43)
at java.lang.Thread.run(Thread.java:748)
01/Jun/2019 13:40:40.052 ERROR com.kafkatool.model.KafkaMapper - Error starting consumer offset poller
java.lang.Exception: Configured to use PLAINTEXT but broker 0 does not support it.
at com.kafkatool.model.Broker.getSecurityTypeConnectString(Broker.java:169)
at com.kafkatool.model.KafkaMapper.getProperties(KafkaMapper.java:258)
at com.kafkatool.model.KafkaMapper.getNewConsumer(KafkaMapper.java:1100)
at com.kafkatool.model.KafkaMapper.getNewConsumer(KafkaMapper.java:1095)
at com.kafkatool.model.KafkaMapper.getNewConsumer(KafkaMapper.java:1085)
at com.kafkatool.model.KafkaMapper.getTopics(KafkaMapper.java:420)
at com.kafkatool.model.ServerConnection.getTopics(ServerConnection.java:221)
at com.kafkatool.model.ServerConnection.hasTopic(ServerConnection.java:203)
at com.kafkatool.model.KafkaMapper.startPoller(KafkaMapper.java:105)
at com.kafkatool.model.ServerConnection.connectInt(ServerConnection.java:336)
at com.kafkatool.model.ServerConnection.connect(ServerConnection.java:315)
at com.kafkatool.common.AsyncServerConnector.run(AsyncServerConnector.java:43)
The comment line in your client_jaas.config does actually contain a clue on this:
Configure SASL_SSL if SSL encryption is enabled, otherwise configure SASL_PLAINTEXT
There seems to be a conflict in the 2 config settings you've provided.
In server.properties, you've mentioned:
security.inter.broker.protocol=SASL_PLAINTEXT
whereas in your client_jaas.config, you've mentioned:
security.protocol=SASL_SSL
Try changing this to SASL_PLAINTEXT.
Hope this helps!
I am trying to insttall zookeeper in my Windows. I am getting the error bellow no matter which suggestion I followed in zookeeper + Kafka - Unable to create data directory.
I am running it as Administrator and I have tried all these options:
#dataDir=/tmp/zookeeper
#dataDir=:\zookeeper-3.4.14\
#dataDir=C:\\_d\\WSs\\kafka\\zookeeper-3.4.14\\data
#dataDir=:\\\\zookeeper\\\\data
dataDir=C:\\_d\\WSs\\kafka\\zookeeper-3.4.14
I donĀ“t think it is relevant but let me add here: I have Java 11.
Any idea why it is happening will be appreciated.
Full logs
C:\Windows\system32>zkserver
C:\Windows\system32>call "C:\Program Files\Java\jdk-11.0.2"\bin\java "-Dzookeeper.log.dir=C:\_d\WSs\kafka\zookeeper-3.4.14\bin\.." "-Dzookeeper.root.logger=INFO,CONSOLE" -cp "C:\_d\WSs\kafka\zookeeper-3.4.14\bin\..\build\classes;C:\_d\WSs\kafka\zookeeper-3.4.14\bin\..\build\lib\*;C:\_d\WSs\kafka\zookeeper-3.4.14\bin\..\*;C:\_d\WSs\kafka\zookeeper-3.4.14\bin\..\lib\*;C:\_d\WSs\kafka\zookeeper-3.4.14\bin\..\conf" org.apache.zookeeper.server.quorum.QuorumPeerMain "C:\_d\WSs\kafka\zookeeper-3.4.14\bin\..\conf\zoo.cfg"
2019-04-18 15:17:42,629 [myid:] - INFO [main:QuorumPeerConfig#136] - Reading configuration from: C:\_d\WSs\kafka\zookeeper-3.4.14\bin\..\conf\zoo.cfg
2019-04-18 15:17:42,644 [myid:] - INFO [main:DatadirCleanupManager#78] - autopurge.snapRetainCount set to 3
2019-04-18 15:17:42,644 [myid:] - INFO [main:DatadirCleanupManager#79] - autopurge.purgeInterval set to 0
2019-04-18 15:17:42,644 [myid:] - INFO [main:DatadirCleanupManager#101] - Purge task is not scheduled.
2019-04-18 15:17:42,644 [myid:] - WARN [main:QuorumPeerMain#116] - Either no config or no quorum defined in config, running in standalone mode
2019-04-18 15:17:42,769 [myid:] - INFO [main:QuorumPeerConfig#136] - Reading configuration from: C:\_d\WSs\kafka\zookeeper-3.4.14\bin\..\conf\zoo.cfg
2019-04-18 15:17:42,769 [myid:] - INFO [main:ZooKeeperServerMain#98] - Starting server
2019-04-18 15:17:47,344 [myid:] - INFO [main:Environment#100] - Server environment:zookeeper.version=3.4.14-4c25d480e66aadd371de8bd2fd8da255ac140bcf, built on 03/06/2019 16:18 GMT
2019-04-18 15:17:47,344 [myid:] - INFO [main:Environment#100] - Server environment:host.name=DESKTOP-AKCNE7F
2019-04-18 15:17:47,344 [myid:] - INFO [main:Environment#100] - Server environment:java.version=11.0.2
2019-04-18 15:17:47,344 [myid:] - INFO [main:Environment#100] - Server environment:java.vendor=Oracle Corporation
2019-04-18 15:17:47,344 [myid:] - INFO [main:Environment#100] - Server environment:java.home=C:\Program Files\Java\jdk-11.0.2
2019-04-18 15:17:47,344 [myid:] - INFO [main:Environment#100] - Server environment:java.class.path=C:\_d\WSs\kafka\zookeeper-3.4.14\bin\..\build\classes;C:\_d\WSs\kafka\zookeeper-3.4.14\bin\..\build\lib\*;C:\_d\WSs\kafka\zookeeper-3.4.14\bin\..\zookeeper-3.4.14.jar;C:\_d\WSs\kafka\zookeeper-3.4.14\bin\..\lib\audience-annotations-0.5.0.jar;C:\_d\WSs\kafka\zookeeper-3.4.14\bin\..\lib\jline-0.9.94.jar;C:\_d\WSs\kafka\zookeeper-3.4.14\bin\..\lib\log4j-1.2.17.jar;C:\_d\WSs\kafka\zookeeper-3.4.14\bin\..\lib\netty-3.10.6.Final.jar;C:\_d\WSs\kafka\zookeeper-3.4.14\bin\..\lib\slf4j-api-1.7.25.jar;C:\_d\WSs\kafka\zookeeper-3.4.14\bin\..\lib\slf4j-log4j12-1.7.25.jar;C:\_d\WSs\kafka\zookeeper-3.4.14\bin\..\conf
2019-04-18 15:17:47,344 [myid:] - INFO [main:Environment#100] - Server environment:java.library.path=C:\Program Files\Java\jdk-11.0.2\bin;C:\Windows\Sun\Java\bin;C:\Windows\system32;C:\Windows;C:\Program Files (x86)\Intel\Intel(R) Management Engine Components\iCLS\;C:\Program Files\Intel\Intel(R) Management Engine Components\iCLS\;C:\Windows\system32;C:\Windows;C:\Windows\System32\Wbem;C:\Windows\System32\WindowsPowerShell\v1.0\;C:\Windows\System32\OpenSSH\;C:\Program Files (x86)\Intel\Intel(R) Management Engine Components\DAL;C:\Program Files\Intel\Intel(R) Management Engine Components\DAL;C:\Program Files (x86)\Intel\Intel(R) Management Engine Components\IPT;C:\Program Files\Intel\Intel(R) Management Engine Components\IPT;C:\Program Files\Java\jdk-11.0.2\bin;C:\Program Files\Git\cmd;C:\ProgramData\chocolatey\bin;C:\_d\tools\apache-maven-3.6.0\bin;C:\_d\WSs\kafka\zookeeper-3.4.14\bin\;C:\Users\jimis\AppData\Local\Programs\Python\Python37-32\Scripts\;C:\Users\jimis\AppData\Local\Programs\Python\Python37-32\;C:\Users\jimis\AppData\Local\Microsoft\WindowsApps;.
2019-04-18 15:17:47,360 [myid:] - INFO [main:Environment#100] - Server environment:java.io.tmpdir=C:\Users\jimis\AppData\Local\Temp\
2019-04-18 15:17:47,360 [myid:] - INFO [main:Environment#100] - Server environment:java.compiler=<NA>
2019-04-18 15:17:47,360 [myid:] - INFO [main:Environment#100] - Server environment:os.name=Windows 10
2019-04-18 15:17:47,360 [myid:] - INFO [main:Environment#100] - Server environment:os.arch=amd64
2019-04-18 15:17:47,376 [myid:] - INFO [main:Environment#100] - Server environment:os.version=10.0
2019-04-18 15:17:47,376 [myid:] - INFO [main:Environment#100] - Server environment:user.name=jimis
2019-04-18 15:17:47,376 [myid:] - INFO [main:Environment#100] - Server environment:user.home=C:\Users\jimis
2019-04-18 15:17:47,376 [myid:] - INFO [main:Environment#100] - Server environment:user.dir=C:\Windows\system32
2019-04-18 15:17:47,391 [myid:] - INFO [main:ZooKeeperServer#836] - tickTime set to 2000
2019-04-18 15:17:47,391 [myid:] - INFO [main:ZooKeeperServer#845] - minSessionTimeout set to -1
2019-04-18 15:17:47,391 [myid:] - INFO [main:ZooKeeperServer#854] - maxSessionTimeout set to -1
2019-04-18 15:17:47,782 [myid:] - INFO [main:ServerCnxnFactory#117] - Using org.apache.zookeeper.server.NIOServerCnxnFactory as server connection factory
2019-04-18 15:17:47,797 [myid:] - INFO [main:NIOServerCnxnFactory#89] - binding to port 0.0.0.0/0.0.0.0:2181
2019-04-18 15:18:00,365 [myid:] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory#222] - Accepted socket connection from /127.0.0.1:54057
2019-04-18 15:18:00,375 [myid:] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory#222] - Accepted socket connection from /127.0.0.1:54058
2019-04-18 15:18:00,378 [myid:] - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn#383] - Exception causing close of session 0x0: Len error 1195725856
2019-04-18 15:18:00,379 [myid:] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn#1056] - Closed socket connection for client /127.0.0.1:54057 (no session established for client)
*** Edited
*** edited
*** The answer to my question is "you can ignore the fact that I get an error while curl 127.0.0.1:port. Kafka is working anyway.
Are you trying to do a "HTTP GET" against the zookeeper client port?
So the error comes from NIOServerCnxn.java:readLength which is expecting either a 4-letter command or buffer where the first 4 bytes represent size.
The number 1195725856 in hex is 0x47455420 which is "GET " in ASCII.
So the error message is caused when you try to do a HTTP GET" against the 2181 port.
$ curl http://0.0.0.0:2181/
curl: (52) Empty reply from server
$ sudo tail /var/log/zookeeper/zookeeper.out
...
2019-04-19 12:56:25,303 [myid:3] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory#215] - Accepted
2019-04-19 12:56:25,304 [myid:3] - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn#383] - Exception causing close of session 0x0: Len error 1195725856
2019-04-19 12:56:25,304 [myid:3] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn#1040] - Closed socket connection for client /127.0.0.1:33011 (no session established for client)
This WARN message is safe to ignore since ZooKeeper will just close that client session which is implied by the curl response.
I have Kafka setup on my machine and was running fine.
I then installed NetBeans and I think that messed things up. Now my zookeeper is not starting.
After netBeans installation I faced an error while running it
The JDK is missing and is required to run some netbeans modules
I solved the issue using this post
Now when I start zookeeper using:
sudo bin/zookeeper-server-start.sh config/zookeeper.properties
I get the following traceback:
[2018-01-31 17:30:09,953] INFO Reading configuration from: config/zookeeper.properties (org.apache.zookeeper.server.quorum.QuorumPeerConfig)
[2018-01-31 17:30:09,959] INFO autopurge.snapRetainCount set to 3 (org.apache.zookeeper.server.DatadirCleanupManager)
[2018-01-31 17:30:09,959] INFO autopurge.purgeInterval set to 0 (org.apache.zookeeper.server.DatadirCleanupManager)
[2018-01-31 17:30:09,959] INFO Purge task is not scheduled. (org.apache.zookeeper.server.DatadirCleanupManager)
[2018-01-31 17:30:09,959] WARN Either no config or no quorum defined in config, running in standalone mode (org.apache.zookeeper.server.quorum.QuorumPeerMain)
[2018-01-31 17:30:09,979] INFO Reading configuration from: config/zookeeper.properties (org.apache.zookeeper.server.quorum.QuorumPeerConfig)
[2018-01-31 17:30:09,980] INFO Starting server (org.apache.zookeeper.server.ZooKeeperServerMain)
[2018-01-31 17:30:09,991] INFO Server environment:zookeeper.version=3.4.10- 39d3a4f269333c922ed3db283be479f9deacaa0f, built on 03/23/2017 10:13 GMT (org.apache.zookeeper.server.ZooKeeperServer)
[2018-01-31 17:30:09,991] INFO Server environment:host.name=jojo (org.apache.zookeeper.server.ZooKeeperServer)
[2018-01-31 17:30:09,991] INFO Server environment:java.version=1.8.0_161 (org.apache.zookeeper.server.ZooKeeperServer)
[2018-01-31 17:30:09,992] INFO Server environment:java.vendor=Oracle Corporation (org.apache.zookeeper.server.ZooKeeperServer)
[2018-01-31 17:30:09,992] INFO Server environment:java.home=/usr/lib/jvm/java-8-oracle/jre (org.apache.zookeeper.server.ZooKeeperServer)
[2018-01-31 17:30:09,992] INFO Server environment:java.class.path=:/opt/Kafka/kafka_2.12- 0.11.0.0/bin/../libs/aopalliance-repackaged-2.5.0- b05.jar:/opt/Kafka/kafka_2.12-0.11.0.0/bin/../libs/argparse4j- 0.7.0.jar:/opt/Kafka/kafka_2.12-0.11.0.0/bin/../libs/commons-lang3- 3.5.jar:/opt/Kafka/kafka_2.12-0.11.0.0/bin/../libs/connect-api-0.11.0.0.jar:/opt/Kafka/kafka_2.12-0.11.0.0/bin/../libs/connect-file-0.11.0.0.jar:/opt/Kafka/kafka_2.12-0.11.0.0/bin/../libs/connect-json-0.11.0.0.jar:/opt/Kafka/kafka_2.12-0.11.0.0/bin/../libs/connect-runtime-0.11.0.0.jar:/opt/Kafka/kafka_2.12-0.11.0.0/bin/../libs/connect- transforms-0.11.0.0.jar:/opt/Kafka/kafka_2.12-0.11.0.0/bin/../libs/guava-20.0.jar:/opt/Kafka/kafka_2.12-0.11.0.0/bin/../libs/hk2-api-2.5.0-b05.jar:/opt/Kafka/kafka_2.12-0.11.0.0/bin/../libs/hk2-locator-2.5.0-b05.jar:/opt/Kafka/kafka_2.12-0.11.0.0/bin/../libs/hk2-utils-2.5.0-b05.jar:/opt/Kafka/kafka_2.12-0.11.0.0/bin/../libs/jackson-annotations-2.8.5.jar:/opt/Kafka/kafka_2.12-0.11.0.0/bin/../libs/jackson-core-2.8.5.jar:/opt/Kafka/kafka_2.12-0.11.0.0/bin/../libs/jackson-databind-2.8.5.jar:/opt/Kafka/kafka_2.12-0.11.0.0/bin/../libs/jackson-jaxrs-base-2.8.5.jar:/opt/Kafka/kafka_2.12-0.11.0.0/bin/../libs/jackson-jaxrs-json-provider-2.8.5.jar:/opt/Kafka/kafka_2.12-0.11.0.0/bin/../libs/jackson- module-jaxb-annotations-2.8.5.jar:/opt/Kafka/kafka_2.12-0.11.0.0/bin/../libs/javassist-3.21.0-GA.jar:/opt/Kafka/kafka_2.12-0.11.0.0/bin/../libs/javax.annotation-api-1.2.jar:/opt/Kafka/kafka_2.12-0.11.0.0/bin/../libs/javax.inject-1.jar:/opt/Kafka/kafka_2.12-0.11.0.0/bin/../libs/javax.inject-2.5.0-b05.jar:/opt/Kafka/kafka_2.12-0.11.0.0/bin/../libs/javax.servlet-api-3.1.0.jar:/opt/Kafka/kafka_2.12-0.11.0.0/bin/../libs/javax.ws.rs-api-2.0.1.jar:/opt/Kafka/kafka_2.12-0.11.0.0/bin/../libs/jersey-client-2.24.jar:/opt/Kafka/kafka_2.12-0.11.0.0/bin/../libs/jersey-common-2.24.jar:/opt/Kafka/kafka_2.12-0.11.0.0/bin/../libs/jersey-container-servlet-2.24.jar:/opt/Kafka/kafka_2.12-0.11.0.0/bin/../libs/jersey-container-servlet-core-2.24.jar:/opt/Kafka/kafka_2.12-0.11.0.0/bin/../libs/jersey-guava-2.24.jar:/opt/Kafka/kafka_2.12-0.11.0.0/bin/../libs/jersey-media-jaxb-2.24.jar:/opt/Kafka/kafka_2.12-0.11.0.0/bin/../libs/jersey-server-2.24.jar:/opt/Kafka/kafka_2.12-0.11.0.0/bin/../libs/jetty-continuation-9.2.15.v20160210.jar:/opt/Kafka/kafka_2.12-0.11.0.0/bin/../libs/jetty-http-9.2.15.v20160210.jar:/opt/Kafka/kafka_2.12-0.11.0.0/bin/../libs/jetty-io-9.2.15.v20160210.jar:/opt/Kafka/kafka_2.12-0.11.0.0/bin/../libs/jetty-security-9.2.15.v20160210.jar:/opt/Kafka/kafka_2.12-0.11.0.0/bin/../libs/jetty-server-9.2.15.v20160210.jar:/opt/Kafka/kafka_2.12-0.11.0.0/bin/../libs/jetty-servlet-9.2.15.v20160210.jar:/opt/Kafka/kafka_2.12-0.11.0.0/bin/../libs/jetty- servlets-9.2.15.v20160210.jar:/opt/Kafka/kafka_2.12-0.11.0.0/bin/../libs/jetty-util-9.2.15.v20160210.jar:/opt/Kafka/kafka_2.12-0.11.0.0/bin/../libs/jopt-simple-5.0.3.jar:/opt/Kafka/kafka_2.12-0.11.0.0/bin/../libs/kafka_2.12-0.11.0.0.jar:/opt/Kafka/kafka_2.12-0.11.0.0/bin/../libs/kafka_2.12-0.11.0.0-sources.jar:/opt/Kafka/kafka_2.12-0.11.0.0/bin/../libs/kafka_2.12-0.11.0.0-test-sources.jar:/opt/Kafka/kafka_2.12-0.11.0.0/bin/../libs/kafka-clients- 0.11.0.0.jar:/opt/Kafka/kafka_2.12-0.11.0.0/bin/../libs/kafka-log4j-appender-0.11.0.0.jar:/opt/Kafka/kafka_2.12-0.11.0.0/bin/../libs/kafka-streams-0.11.0.0.jar:/opt/Kafka/kafka_2.12-0.11.0.0/bin/../libs/kafka-streams-examples-0.11.0.0.jar:/opt/Kafka/kafka_2.12-0.11.0.0/bin/../libs/kafka-tools-0.11.0.0.jar:/opt/Kafka/kafka_2.12-0.11.0.0/bin/../libs/log4j-1.2.17.jar:/opt/Kafka/kafka_2.12-0.11.0.0/bin/../libs/lz4-1.3.0.jar:/opt/Kafka/kafka_2.12-0.11.0.0/bin/../libs/maven-artifact-3.5.0.jar:/opt/Kafka/kafka_2.12-0.11.0.0/bin/../libs/metrics-core-2.2.0.jar:/opt/Kafka/kafka_2.12-0.11.0.0/bin/../libs/osgi-resource-locator-1.0.1.jar:/opt/Kafka/kafka_2.12-0.11.0.0/bin/../libs/plexus-utils-3.0.24.jar:/opt/Kafka/kafka_2.12-0.11.0.0/bin/../libs/reflections-0.9.11.jar:/opt/Kafka/kafka_2.12-0.11.0.0/bin/../libs/rocksdbjni-5.0.1.jar:/opt/Kafka/kafka_2.12-0.11.0.0/bin/../libs/scala-library-2.12.2.jar:/opt/Kafka/kafka_2.12-0.11.0.0/bin/../libs/scala-parser-combinators_2.12-1.0.4.jar:/opt/Kafka/kafka_2.12-0.11.0.0/bin/../libs/slf4j-api-1.7.25.jar:/opt/Kafka/kafka_2.12-0.11.0.0/bin/../libs/slf4j-log4j12-1.7.25.jar:/opt/Kafka/kafka_2.12-0.11.0.0/bin/../libs/snappy-java-1.1.2.6.jar:/opt/Kafka/kafka_2.12-0.11.0.0/bin/../libs/validation-api-1.1.0.Final.jar:/opt/Kafka/kafka_2.12-0.11.0.0/bin/../libs/zkclient-0.10.jar:/opt/Kafka/kafka_2.12-0.11.0.0/bin/../libs/zookeeper-3.4.10.jar (org.apache.zookeeper.server.ZooKeeperServer)
[2018-01-31 17:30:09,992] INFO Server environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib (org.apache.zookeeper.server.ZooKeeperServer)
[2018-01-31 17:30:09,992] INFO Server environment:java.io.tmpdir=/tmp (org.apache.zookeeper.server.ZooKeeperServer)
[2018-01-31 17:30:09,992] INFO Server environment:java.compiler=<NA> (org.apache.zookeeper.server.ZooKeeperServer)
[2018-01-31 17:30:09,992] INFO Server environment:os.name=Linux (org.apache.zookeeper.server.ZooKeeperServer)
[2018-01-31 17:30:09,992] INFO Server environment:os.arch=amd64 (org.apache.zookeeper.server.ZooKeeperServer)
[2018-01-31 17:30:09,992] INFO Server environment:os.version=4.4.0-112-generic (org.apache.zookeeper.server.ZooKeeperServer)
[2018-01-31 17:30:09,992] INFO Server environment:user.name=root (org.apache.zookeeper.server.ZooKeeperServer)
[2018-01-31 17:30:09,992] INFO Server environment:user.home=/root (org.apache.zookeeper.server.ZooKeeperServer)
[2018-01-31 17:30:09,993] INFO Server environment:user.dir=/opt/Kafka/kafka_2.12-0.11.0.0 (org.apache.zookeeper.server.ZooKeeperServer)
[2018-01-31 17:30:09,999] INFO tickTime set to 3000 (org.apache.zookeeper.server.ZooKeeperServer)
[2018-01-31 17:30:09,999] INFO minSessionTimeout set to -1 (org.apache.zookeeper.server.ZooKeeperServer)
[2018-01-31 17:30:09,999] INFO maxSessionTimeout set to -1 (org.apache.zookeeper.server.ZooKeeperServer)
[2018-01-31 17:30:10,007] INFO binding to port 0.0.0.0/0.0.0.0:2181 (org.apache.zookeeper.server.NIOServerCnxnFactory)
[2018-01-31 17:30:10,008] ERROR Unexpected exception, exiting abnormally (org.apache.zookeeper.server.ZooKeeperServerMain)
java.net.BindException: Address already in use
at sun.nio.ch.Net.bind0(Native Method)
at sun.nio.ch.Net.bind(Net.java:433)
at sun.nio.ch.Net.bind(Net.java:425)
at sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:223)
at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:74)
at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:67)
at org.apache.zookeeper.server.NIOServerCnxnFactory.configure(NIOServerCnxnFa ctory.java:90)
at org.apache.zookeeper.server.ZooKeeperServerMain.runFromConfig(ZooKeeperSer verMain.java:117)
at org.apache.zookeeper.server.ZooKeeperServerMain.initializeAndRun(ZooKeeper ServerMain.java:87)
at org.apache.zookeeper.server.ZooKeeperServerMain.main(ZooKeeperServerMain.j ava:53)
at org.apache.zookeeper.server.quorum.QuorumPeerMain.initializeAndRun(QuorumP eerMain.java:116)
at org.apache.zookeeper.server.quorum.QuorumPeerMain.main(QuorumPeerMain.java:78)
I have tried this:
netstat -ant | grep :2181
Gives me the following output:
tcp6 0 0 :::2181 :::* LISTEN
tcp6 0 0 127.0.0.1:43566 127.0.0.1:2181 ESTABLISHED
tcp6 0 0 127.0.0.1:2181 127.0.0.1:43566 ESTABLISHED
Before installing NetBeans the output was:
tcp6 0 0 :::2181 :::* LISTEN
I think this is the issue, but I don't know how to resolve this.
I have tried sudo lsof -i :2181
Output:
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
java 1005 zookeeper 33u IPv6 17209 0t0 TCP *:2181 (LISTEN)
java 1005 zookeeper 34u IPv6 327225 0t0 TCP localhost:2181->localhost:43566 (ESTABLISHED)
java 22585 root 88u IPv6 324552 0t0 TCP localhost:43566->localhost:2181 (ESTABLISHED)
Go to Kafka folder where you installed and type following command :
sudo bin/zookeeper-server-stop.sh
Then try to execute the following command :
bin/zookeeper-server-start.sh config/zookeeper.properties
Hope you will get rid off from your Error.
You should go to the config folder. Then there is a file zookeeper.properties open the file and set clientPort=5181(if it is unused port) and set maxClientCnxns=5 (initially here was 0 instead of 5, you can set it as your requirement).
I'm trying to setup Apache Storm (1.0.2) on my Macbook Pro but apparently running into timeout issues if I try to deploy the topology. Also the UI hangs up spitting the same exception.
3491 [main] INFO o.a.s.StormSubmitter - Generated ZooKeeper secret payload for MD5-digest: -8915636774701640550:-6510752657961785886
3580 [main] INFO o.a.s.s.a.AuthUtils - Got AutoCreds []
Exception in thread "main" java.lang.RuntimeException: org.apache.storm.thrift.transport.TTransportException: java.net.ConnectException: Operation timed out (Connection timed out)
at org.apache.storm.security.auth.TBackoffConnect.retryNext(TBackoffConnect.java:64)
at org.apache.storm.security.auth.TBackoffConnect.doConnectWithRetry(TBackoffConnect.java:56)
at org.apache.storm.security.auth.ThriftClient.reconnect(ThriftClient.java:99)
at org.apache.storm.security.auth.ThriftClient.<init>(ThriftClient.java:69)
at org.apache.storm.utils.NimbusClient.<init>(NimbusClient.java:106)
at org.apache.storm.utils.NimbusClient.getConfiguredClientAs(NimbusClient.java:78)
at org.apache.storm.StormSubmitter.topologyNameExists(StormSubmitter.java:371)
at org.apache.storm.StormSubmitter.submitTopologyAs(StormSubmitter.java:233)
at org.apache.storm.StormSubmitter.submitTopology(StormSubmitter.java:311)
at org.apache.storm.StormSubmitter.submitTopology(StormSubmitter.java:157)
Caused by: org.apache.storm.thrift.transport.TTransportException: java.net.ConnectException: Operation timed out (Connection timed out)
at org.apache.storm.thrift.transport.TSocket.open(TSocket.java:226)
at org.apache.storm.thrift.transport.TFramedTransport.open(TFramedTransport.java:81)
at org.apache.storm.security.auth.SimpleTransportPlugin.connect(SimpleTransportPlugin.java:103)
at org.apache.storm.security.auth.TBackoffConnect.doConnectWithRetry(TBackoffConnect.java:53)
... 9 more
Caused by: java.net.ConnectException: Operation timed out (Connection timed out)
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:589)
at org.apache.storm.thrift.transport.TSocket.open(TSocket.java:221)
... 12 more
I'm using the default storm.yaml configuration from the github repository; without any change and default zoo.cfg file for zookeeper as well.
# The number of milliseconds of each tick
tickTime=2000
# The number of ticks that the initial
# synchronization phase can take
initLimit=5
# The number of ticks that can pass between
# sending a request and getting an acknowledgement
syncLimit=2
# the directory where the snapshot is stored.
# do not use /tmp for storage, /tmp here is just
# example sakes.
dataDir=/tmp/zookeeper
# the port at which the clients will connect
clientPort=2181
clientPortAddress=localhost
# the maximum number of client connections.
# increase this if you need to handle more clients
#maxClientCnxns=60
#
# Be sure to read the maintenance section of the
# administrator guide before turning on autopurge.
#
# http://zookeeper.apache.org/doc/current/zookeeperAdmin.html#sc_maintenance
#
# The number of snapshots to retain in dataDir
#autopurge.snapRetainCount=3
# Purge task interval in hours
# Set to "0" to disable auto purge feature
#autopurge.purgeInterval=1
I came across similar issues which prompted me to check my hosts file; which I've posted as below
##
# Host Database
#
# localhost is used to configure the loopback interface
# when the system is booting. Do not change this entry.
##
255.255.255.255 broadcasthost
127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4
::1 ip6-localhost ip6-localhost.localdomain localhost6 localhost6.localdomain6
When I start the zookeeper server; I believe it get's started as usual.
2017-11-27 16:05:14,314 [myid:] - INFO [main:QuorumPeerConfig#103] - Reading configuration from: /Users/aniket.alhat/Tools/zookeeper/bin/../conf/zoo.cfg
2017-11-27 16:05:14,318 [myid:] - INFO [main:DatadirCleanupManager#78] - autopurge.snapRetainCount set to 3
2017-11-27 16:05:14,318 [myid:] - INFO [main:DatadirCleanupManager#79] - autopurge.purgeInterval set to 0
2017-11-27 16:05:14,318 [myid:] - INFO [main:DatadirCleanupManager#101] - Purge task is not scheduled.
2017-11-27 16:05:14,318 [myid:] - WARN [main:QuorumPeerMain#113] - Either no config or no quorum defined in config, running in standalone mode
2017-11-27 16:05:14,329 [myid:] - INFO [main:QuorumPeerConfig#103] - Reading configuration from: /Users/aniket.alhat/Tools/zookeeper/bin/../conf/zoo.cfg
2017-11-27 16:05:14,330 [myid:] - INFO [main:ZooKeeperServerMain#95] - Starting server
2017-11-27 16:05:14,335 [myid:] - INFO [main:Environment#100] - Server environment:zookeeper.version=3.4.6-1569965, built on 02/20/2014 09:09 GMT
2017-11-27 16:05:14,336 [myid:] - INFO [main:Environment#100] - Server environment:host.name=10.9.157.77
2017-11-27 16:05:14,336 [myid:] - INFO [main:Environment#100] - Server environment:java.version=1.8.0_131
2017-11-27 16:05:14,336 [myid:] - INFO [main:Environment#100] - Server environment:java.vendor=Oracle Corporation
2017-11-27 16:05:14,336 [myid:] - INFO [main:Environment#100] - Server environment:java.home=/Library/Java/JavaVirtualMachines/jdk1.8.0_131.jdk/Contents/Home/jre
2017-11-27 16:05:14,336 [myid:] - INFO [main:Environment#100] - Server environment:java.class.path=/Users/aniket.alhat/Tools/zookeeper/bin/../build/classes:/Users/aniket.alhat/Tools/zookeeper/bin/../build/lib/*.jar:/Users/aniket.alhat/Tools/zookeeper/bin/../lib/slf4j-log4j12-1.6.1.jar:/Users/aniket.alhat/Tools/zookeeper/bin/../lib/slf4j-api-1.6.1.jar:/Users/aniket.alhat/Tools/zookeeper/bin/../lib/netty-3.7.0.Final.jar:/Users/aniket.alhat/Tools/zookeeper/bin/../lib/log4j-1.2.16.jar:/Users/aniket.alhat/Tools/zookeeper/bin/../lib/jline-0.9.94.jar:/Users/aniket.alhat/Tools/zookeeper/bin/../zookeeper-3.4.6.jar:/Users/aniket.alhat/Tools/zookeeper/bin/../src/java/lib/*.jar:/Users/aniket.alhat/Tools/zookeeper/bin/../conf:
2017-11-27 16:05:14,336 [myid:] - INFO [main:Environment#100] - Server environment:java.library.path=/Users/aniket.alhat/Library/Java/Extensions:/Library/Java/Extensions:/Network/Library/Java/Extensions:/System/Library/Java/Extensions:/usr/lib/java:.
2017-11-27 16:05:14,336 [myid:] - INFO [main:Environment#100] - Server environment:java.io.tmpdir=/var/folders/9c/g5cj60_j1x344r3zpd_hr99j5jwnk4/T/
2017-11-27 16:05:14,336 [myid:] - INFO [main:Environment#100] - Server environment:java.compiler=<NA>
2017-11-27 16:05:14,337 [myid:] - INFO [main:Environment#100] - Server environment:os.name=Mac OS X
2017-11-27 16:05:14,337 [myid:] - INFO [main:Environment#100] - Server environment:os.arch=x86_64
2017-11-27 16:05:14,338 [myid:] - INFO [main:Environment#100] - Server environment:os.version=10.12.6
2017-11-27 16:05:14,338 [myid:] - INFO [main:Environment#100] - Server environment:user.name=aniket.alhat
2017-11-27 16:05:14,338 [myid:] - INFO [main:Environment#100] - Server environment:user.home=/Users/aniket.alhat
2017-11-27 16:05:14,338 [myid:] - INFO [main:Environment#100] - Server environment:user.dir=/Users/aniket.alhat/Tools/zookeeper-3.4.6
2017-11-27 16:05:14,344 [myid:] - INFO [main:ZooKeeperServer#755] - tickTime set to 2000
2017-11-27 16:05:14,344 [myid:] - INFO [main:ZooKeeperServer#764] - minSessionTimeout set to -1
2017-11-27 16:05:14,344 [myid:] - INFO [main:ZooKeeperServer#773] - maxSessionTimeout set to -1
2017-11-27 16:05:14,361 [myid:] - INFO [main:NIOServerCnxnFactory#94] - binding to port localhost/127.0.0.1:2181
And I also don't see any errors in nimbus log
2017-11-27 16:05:35.365 o.a.s.s.o.a.c.f.i.CuratorFrameworkImpl [INFO] Starting
2017-11-27 16:05:35.373 o.a.s.s.o.a.z.ZooKeeper [INFO] Client environment:zookeeper.version=3.4.6-1569965, built on 02/20/2014 09:09 GMT
2017-11-27 16:05:35.373 o.a.s.s.o.a.z.ZooKeeper [INFO] Client environment:host.name=10.49.48.134
2017-11-27 16:05:35.373 o.a.s.s.o.a.z.ZooKeeper [INFO] Client environment:java.version=1.8.0_131
2017-11-27 16:05:35.373 o.a.s.s.o.a.z.ZooKeeper [INFO] Client environment:java.vendor=Oracle Corporation
2017-11-27 16:05:35.373 o.a.s.s.o.a.z.ZooKeeper [INFO] Client environment:java.home=/Library/Java/JavaVirtualMachines/jdk1.8.0_131.jdk/Contents/Home/jre
2017-11-27 16:05:35.373 o.a.s.s.o.a.z.ZooKeeper [INFO] Client environment:java.class.path=/Users/aniket.alhat/Tools/apache-storm-1.0.2/lib/asm-5.0.3.jar:/Users/aniket.alhat/Tools/apache-storm-1.0.2/lib/clojure-1.7.0.jar:/Users/aniket.alhat/Tools/apache-storm-1.0.2/lib/disruptor-3.3.2.jar:/Users/aniket.alhat/Tools/apache-storm-1.0.2/lib/kryo-3.0.3.jar:/Users/aniket.alhat/Tools/apache-storm-1.0.2/lib/log4j-api-2.1.jar:/Users/aniket.alhat/Tools/apache-storm-1.0.2/lib/log4j-core-2.1.jar:/Users/aniket.alhat/Tools/apache-storm-1.0.2/lib/log4j-over-slf4j-1.6.6.jar:/Users/aniket.alhat/Tools/apache-storm-1.0.2/lib/log4j-slf4j-impl-2.1.jar:/Users/aniket.alhat/Tools/apache-storm-1.0.2/lib/minlog-1.3.0.jar:/Users/aniket.alhat/Tools/apache-storm-1.0.2/lib/objenesis-2.1.jar:/Users/aniket.alhat/Tools/apache-storm-1.0.2/lib/reflectasm-1.10.1.jar:/Users/aniket.alhat/Tools/apache-storm-1.0.2/lib/servlet-api-2.5.jar:/Users/aniket.alhat/Tools/apache-storm-1.0.2/lib/slf4j-api-1.7.7.jar:/Users/aniket.alhat/Tools/apache-storm-1.0.2/lib/storm-core-1.0.2.jar:/Users/aniket.alhat/Tools/apache-storm-1.0.2/lib/storm-rename-hack-1.0.2.jar:/Users/aniket.alhat/Tools/storm/conf
2017-11-27 16:05:35.373 o.a.s.s.o.a.z.ZooKeeper [INFO] Client environment:java.library.path=/usr/local/lib:/opt/local/lib:/usr/lib
2017-11-27 16:05:35.373 o.a.s.s.o.a.z.ZooKeeper [INFO] Client environment:java.io.tmpdir=/var/folders/9c/g5cj60_j1x344r3zpd_hr99j5jwnk4/T/
2017-11-27 16:05:35.373 o.a.s.s.o.a.z.ZooKeeper [INFO] Client environment:java.compiler=<NA>
2017-11-27 16:05:35.373 o.a.s.s.o.a.z.ZooKeeper [INFO] Client environment:os.name=Mac OS X
2017-11-27 16:05:35.373 o.a.s.s.o.a.z.ZooKeeper [INFO] Client environment:os.arch=x86_64
2017-11-27 16:05:35.373 o.a.s.s.o.a.z.ZooKeeper [INFO] Client environment:os.version=10.12.6
2017-11-27 16:05:35.373 o.a.s.s.o.a.z.ZooKeeper [INFO] Client environment:user.name=aniket.alhat
2017-11-27 16:05:35.373 o.a.s.s.o.a.z.ZooKeeper [INFO] Client environment:user.home=/Users/aniket.alhat
2017-11-27 16:05:35.373 o.a.s.s.o.a.z.ZooKeeper [INFO] Client environment:user.dir=/Users/aniket.alhat/Tools/apache-storm-1.0.2
2017-11-27 16:05:35.374 o.a.s.s.o.a.z.ZooKeeper [INFO] Initiating client connection, connectString=localhost:2181/storm sessionTimeout=20000 watcher=org.apache.storm.shade.org.apache.curator.ConnectionState#eac3a26
2017-11-27 16:05:35.397 o.a.s.s.o.a.z.ClientCnxn [INFO] Opening socket connection to server localhost/127.0.0.1:2181. Will not attempt to authenticate using SASL (unknown error)
2017-11-27 16:05:35.400 o.a.s.b.FileBlobStoreImpl [INFO] Creating new blob store based in storm-local/blobs
2017-11-27 16:05:35.406 o.a.s.d.nimbus [INFO] Using default scheduler
2017-11-27 16:05:35.408 o.a.s.s.o.a.c.f.i.CuratorFrameworkImpl [INFO] Starting
2017-11-27 16:05:35.409 o.a.s.s.o.a.z.ZooKeeper [INFO] Initiating client connection, connectString=localhost:2181 sessionTimeout=20000 watcher=org.apache.storm.shade.org.apache.curator.ConnectionState#68868328
2017-11-27 16:05:35.411 o.a.s.s.o.a.z.ClientCnxn [INFO] Opening socket connection to server localhost/127.0.0.1:2181. Will not attempt to authenticate using SASL (unknown error)
2017-11-27 16:05:35.438 o.a.s.s.o.a.c.f.i.CuratorFrameworkImpl [INFO] Starting
2017-11-27 16:05:35.438 o.a.s.s.o.a.z.ZooKeeper [INFO] Initiating client connection, connectString=localhost:2181 sessionTimeout=20000 watcher=org.apache.storm.shade.org.apache.curator.ConnectionState#512d6e60
2017-11-27 16:05:35.440 o.a.s.s.o.a.z.ClientCnxn [INFO] Opening socket connection to server localhost/127.0.0.1:2181. Will not attempt to authenticate using SASL (unknown error)
2017-11-27 16:05:35.478 o.a.s.s.o.a.z.ClientCnxn [INFO] Socket connection established to localhost/127.0.0.1:2181, initiating session
2017-11-27 16:05:35.478 o.a.s.s.o.a.z.ClientCnxn [INFO] Socket connection established to localhost/127.0.0.1:2181, initiating session
2017-11-27 16:05:35.479 o.a.s.s.o.a.z.ClientCnxn [INFO] Socket connection established to localhost/127.0.0.1:2181, initiating session
2017-11-27 16:05:35.513 o.a.s.s.o.a.z.ClientCnxn [INFO] Session establishment complete on server localhost/127.0.0.1:2181, sessionid = 0x15ffc4b4d950000, negotiated timeout = 20000
2017-11-27 16:05:35.513 o.a.s.s.o.a.z.ClientCnxn [INFO] Session establishment complete on server localhost/127.0.0.1:2181, sessionid = 0x15ffc4b4d950002, negotiated timeout = 20000
2017-11-27 16:05:35.513 o.a.s.s.o.a.z.ClientCnxn [INFO] Session establishment complete on server localhost/127.0.0.1:2181, sessionid = 0x15ffc4b4d950001, negotiated timeout = 20000
2017-11-27 16:05:35.517 o.a.s.s.o.a.c.f.s.ConnectionStateManager [INFO] State change: CONNECTED
2017-11-27 16:05:35.517 o.a.s.s.o.a.c.f.s.ConnectionStateManager [INFO] State change: CONNECTED
2017-11-27 16:05:35.517 o.a.s.s.o.a.c.f.s.ConnectionStateManager [INFO] State change: CONNECTED
2017-11-27 16:05:35.518 o.a.s.zookeeper [INFO] Zookeeper state update: :connected:none
2017-11-27 16:05:35.518 o.a.s.zookeeper [INFO] Zookeeper state update: :connected:none
2017-11-27 16:05:35.531 o.a.s.s.o.a.c.f.i.CuratorFrameworkImpl [INFO] backgroundOperationsLoop exiting
2017-11-27 16:05:35.534 o.a.s.s.o.a.z.ZooKeeper [INFO] Session: 0x15ffc4b4d950002 closed
2017-11-27 16:05:35.534 o.a.s.s.o.a.z.ClientCnxn [INFO] EventThread shut down
2017-11-27 16:05:35.536 o.a.s.s.o.a.c.f.i.CuratorFrameworkImpl [INFO] Starting
2017-11-27 16:05:35.536 o.a.s.s.o.a.z.ZooKeeper [INFO] Initiating client connection, connectString=localhost:2181/storm sessionTimeout=20000 watcher=org.apache.storm.shade.org.apache.curator.ConnectionState#3722c145
I would really appreciate if I could get some help to fix the timeout issue.
Check if the Nimbus is started correctly. I faced similar issue when an instance of Nimbus was not terminated correctly.
Try to kill the process and restart Nimbus.
After lot of trial-and-error I discovered that my Nimbus process gets started with a IP address 10.9.157.77 while ifconfig gives me 10.49.52.97 not sure why/how this is happening, I'll really appreciate if someone can help me figure it out.
nimbus.log
2017-11-30 16:47:00.342 o.a.s.zookeeper [INFO] 10.9.157.77 gained leadership, checking if it has all the topology code locally.
2017-11-30 16:47:00.350 o.a.s.zookeeper [INFO] active-topology-ids [] local-topology-ids [] diff-topology []
2017-11-30 16:47:00.350 o.a.s.zookeeper [INFO] Accepting leadership, all active topology found localy.
2017-11-30 16:47:00.352 o.a.s.d.m.MetricsUtils [INFO] Using statistics reporter plugin:org.apache.storm.daemon.metrics.reporters.JmxPreparableReporter
ifconfig
en0: flags=8863<UP,BROADCAST,SMART,RUNNING,SIMPLEX,MULTICAST> mtu 1500
ether xx:xx:xx:xx:xx:xx
inet6 fe80::427:8998:bb4d:b2bd%en0 prefixlen 64 secured scopeid 0x4
inet 10.49.52.97 netmask 0xfffffc00 broadcast 10.49.55.255
Continuing forward I found everytime I was starting the nimbus process IP address 10.9.157.77 was been acquired magically and was stored in zookeeper as well.
[zk: localhost:2181(CONNECTED) 15] ls /storm/nimbuses
[10.9.157.77:6627]
I cleaned /storm directory with rmr and restart nimbus creating the directory once again, but there was no change.
I also tried flushing DNS cache, command used was sudo killall -HUP mDNSResponder
I also observed that the IP magical IP wasn't same after restarts, it changed to 10.49.48.134
2017-11-30 17:28:59.630 o.a.s.zookeeper [INFO] 10.49.48.134 gained leadership, checking if it has all the topology code locally.
2017-11-30 17:28:59.646 o.a.s.zookeeper [INFO] active-topology-ids [] local-topology-ids [] diff-topology []
2017-11-30 17:28:59.646 o.a.s.zookeeper [INFO] Accepting leadership, all active topology found localy.
Later I disconnected from Wifi and started everything once again and I was able to start Storm UI run command storm list deploy topology locally.
You can add storm.local.hostname= at your storm/conf/storm.yaml, and restart. Also work with IPv4/FQDN and not IPv6. This worked for me (same Storm 1.0.2)
If there are still problems, you can also add nimbus.seeds= with the Nimbus's host.