I have just started learning Kafka/Zookeper. Issue I am facing is, when I try to run the console consumer to verify the message for a topic, I get the below error-
vagrant#debian-70rc1-x64-vbox4210:~$ $KAFKA_HOME/bin/kafka-console-consumer.sh --zookeeper 192.168.33.10:2181 --topic topic-3
[2015-02-22 22:56:36,666] INFO Verifying properties (kafka.utils.VerifiableProperties)
[2015-02-22 22:56:36,698] INFO Property group.id is overridden to console-consumer-92151 (kafka.utils.VerifiableProperties)
[2015-02-22 22:56:36,699] INFO Property fetch.message.max.bytes is overridden to 1048576 (kafka.utils.VerifiableProperties)
[2015-02-22 22:56:36,700] INFO Property consumer.timeout.ms is overridden to -1 (kafka.utils.VerifiableProperties)
[2015-02-22 22:56:36,700] INFO Property socket.timeout.ms is overridden to 30000 (kafka.utils.VerifiableProperties)
[2015-02-22 22:56:36,700] INFO Property auto.offset.reset is overridden to largest (kafka.utils.VerifiableProperties)
[2015-02-22 22:56:36,701] INFO Property zookeeper.connect is overridden to 192.168.33.10:2181 (kafka.utils.VerifiableProperties)
[2015-02-22 22:56:36,701] INFO Property socket.receive.buffer.bytes is overridden to 2097152 (kafka.utils.VerifiableProperties)
[2015-02-22 22:56:36,701] INFO Property fetch.wait.max.ms is overridden to 100 (kafka.utils.VerifiableProperties)
[2015-02-22 22:56:36,702] INFO Property auto.commit.enable is overridden to true (kafka.utils.VerifiableProperties)
[2015-02-22 22:56:36,702] INFO Property refresh.leader.backoff.ms is overridden to 200 (kafka.utils.VerifiableProperties)
[2015-02-22 22:56:36,702] INFO Property auto.commit.interval.ms is overridden to 60000 (kafka.utils.VerifiableProperties)
[2015-02-22 22:56:36,703] INFO Property fetch.min.bytes is overridden to 1 (kafka.utils.VerifiableProperties)
[2015-02-22 22:56:36,734] INFO [console-consumer-92151_debian-70rc1-x64-vbox4210-1424645796729-11446a89], Connecting to zookeeper instance at 192.168.33.10:2181 (kafka.consumer.ZookeeperConsumerConnector)
[2015-02-22 22:56:36,754] INFO Starting ZkClient event thread. (org.I0Itec.zkclient.ZkEventThread)
[2015-02-22 22:56:36,755] INFO Client environment:zookeeper.version=3.3.3-1203054, built on 11/17/2011 05:47 GMT (org.apache.zookeeper.ZooKeeper)
[2015-02-22 22:56:36,756] INFO Client environment:host.name=debian-70rc1-x64-vbox4210.vagrantup.com (org.apache.zookeeper.ZooKeeper)
[2015-02-22 22:56:36,756] INFO Client environment:java.version=1.7.0_75 (org.apache.zookeeper.ZooKeeper)
[2015-02-22 22:56:36,757] INFO Client environment:java.vendor=Oracle Corporation (org.apache.zookeeper.ZooKeeper)
[2015-02-22 22:56:36,757] INFO Client environment:java.home=/usr/lib/jvm/java-7-openjdk-amd64/jre (org.apache.zookeeper.ZooKeeper)
[2015-02-22 22:56:36,757] INFO Client environment:java.class.path=:/usr/local/kafka/kafka-0.8.0-beta1-src//bin/../core/target/scala-2.8.0/kafka_2.8.0-0.8.0-beta1.jar:/usr/local/kafka/kafka-0.8.0-beta1-src//bin/../core/target/scala-2.8.0/kafka-assembly-0.8.0-beta1-deps.jar:/usr/local/kafka/kafka-0.8.0-beta1-src//bin/../perf/target/scala-2.8.0/kafka-perf_2.8.0-0.8.0-beta1.jar:/usr/local/kafka/kafka-0.8.0-beta1-src//bin/../libs/*.jar:/usr/local/kafka/kafka-0.8.0-beta1-src//bin/../kafka*.jar (org.apache.zookeeper.ZooKeeper)
[2015-02-22 22:56:36,758] INFO Client environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib/x86_64-linux-gnu/jni:/lib/x86_64-linux-gnu:/usr/lib/x86_64-linux-gnu:/usr/lib/jni:/lib:/usr/lib (org.apache.zookeeper.ZooKeeper)
[2015-02-22 22:56:36,758] INFO Client environment:java.io.tmpdir=/tmp (org.apache.zookeeper.ZooKeeper)
[2015-02-22 22:56:36,758] INFO Client environment:java.compiler=<NA> (org.apache.zookeeper.ZooKeeper)
[2015-02-22 22:56:36,759] INFO Client environment:os.name=Linux (org.apache.zookeeper.ZooKeeper)
[2015-02-22 22:56:36,759] INFO Client environment:os.arch=amd64 (org.apache.zookeeper.ZooKeeper)
[2015-02-22 22:56:36,759] INFO Client environment:os.version=3.2.0-4-amd64 (org.apache.zookeeper.ZooKeeper)
[2015-02-22 22:56:36,759] INFO Client environment:user.name=vagrant (org.apache.zookeeper.ZooKeeper)
[2015-02-22 22:56:36,760] INFO Client environment:user.home=/home/vagrant (org.apache.zookeeper.ZooKeeper)
[2015-02-22 22:56:36,760] INFO Client environment:user.dir=/home/vagrant (org.apache.zookeeper.ZooKeeper)
[2015-02-22 22:56:36,761] INFO Initiating client connection, connectString=192.168.33.10:2181 sessionTimeout=6000 watcher=org.I0Itec.zkclient.ZkClient#234a98fa (org.apache.zookeeper.ZooKeeper)
[2015-02-22 22:56:36,773] INFO Opening socket connection to server /192.168.33.10:2181 (org.apache.zookeeper.ClientCnxn)
[2015-02-22 22:56:42,778] INFO Terminate ZkClient event thread. (org.I0Itec.zkclient.ZkEventThread)
[2015-02-22 22:56:46,791] INFO Socket connection established to 192.168.33.10/192.168.33.10:2181, initiating session (org.apache.zookeeper.ClientCnxn)
[2015-02-22 22:56:46,801] INFO Session establishment complete on server 192.168.33.10/192.168.33.10:2181, sessionid = 0x14bb3752996000d, negotiated timeout = 6000 (org.apache.zookeeper.ClientCnxn)
[2015-02-22 22:56:46,803] INFO Session: 0x14bb3752996000d closed (org.apache.zookeeper.ZooKeeper)
Exception in thread "main" org.I0Itec.zkclient.exception.ZkTimeoutException: Unable to connect to zookeeper server within timeout: 6000
at org.I0Itec.zkclient.ZkClient.connect(ZkClient.java:880)
at org.I0Itec.zkclient.ZkClient.<init>(ZkClient.java:98)
at org.I0Itec.zkclient.ZkClient.<init>(ZkClient.java:84)
at kafka.consumer.ZookeeperConsumerConnector.connectZk(ZookeeperConsumerConnector.scala:151)
at kafka.consumer.ZookeeperConsumerConnector.<init>(ZookeeperConsumerConnector.scala:112)
at kafka.consumer.ZookeeperConsumerConnector.<init>(ZookeeperConsumerConnector.scala:123)
at kafka.consumer.Consumer$.create(ConsumerConnector.scala:89)
at kafka.consumer.ConsoleConsumer$.main(ConsoleConsumer.scala:178)
at kafka.consumer.ConsoleConsumer.main(ConsoleConsumer.scala)
[2015-02-22 22:56:46,804] INFO EventThread shut down (org.apache.zookeeper.ClientCnxn)
It says-
Exception in thread "main" org.I0Itec.zkclient.exception.ZkTimeoutException: Unable to connect to zookeeper server within timeout: 6000
I verified the topic and it is present. What is going wrong. Please help.
I had the same problem and here is how I fixed it:
1) Stop all Kafka and Zookeeper processes
ps -aux | grep zoo
ps -aux | grep kafka
(then proceed to kill all process ids from above jobs)
2) Run zookeeper
<kafka_dir>/bin/zookeeper-server-start.sh ../config/zookeeper.properties
3) Run kafka server
<kafka_dir>/bin/kafka-server-start.sh ../config/server.properties
Try -> telnet 192.168.33.10 2181
If the telnet fails, then the issue is due to below reason.
Zookeeper Server NRPE iptables chain does not appear to a have a default accept rule.
Unless you just want your own iptables chain for NRPE, you could use:
iptables -I INPUT -s 0/0 -p tcp --dport 2181 -j ACCEPT
This solves the issue.
You can try the follwoing to see if it helps ..
1) Make sure the zookeeper is up and running, try to connect to it using the zkCli client .
2) append --from-beginning at the end of your command to see if you get any diifference in the output , so the command should be changed to :
$KAFKA_HOME/bin/kafka-console-consumer.sh --zookeeper xxx.xx.xx.xxx:2181 --topic topic-3 --from-beginning
In my case it was already connected to the host from which I was trying, so it was not able to make any new connection to the host. I had to change the config: maxClientCnxns so that it was able to make more connection to that host. More details here.
Related
Here's the log:
kafka 16:54:47.56
kafka 16:54:47.57 Welcome to the Bitnami kafka container
kafka 16:54:47.57 Subscribe to project updates by watching https://github.com/bitnami/bitnami-docker-kafka
kafka 16:54:47.57 Submit issues and feature requests at https://github.com/bitnami/bitnami-docker-kafka/issues
kafka 16:54:47.57
kafka 16:54:47.57 INFO ==> ** Starting Kafka setup **
kafka 16:54:47.62 DEBUG ==> Validating settings in KAFKA_* env vars...
kafka 16:54:47.64 WARN ==> You set the environment variable ALLOW_PLAINTEXT_LISTENER=yes. For safety reasons, do not use this flag in a production environment.
kafka 16:54:47.64 INFO ==> Initializing Kafka...
kafka 16:54:47.65 INFO ==> No injected configuration files found, creating default config files
kafka 16:54:47.89 INFO ==> Configuring Kafka for inter-broker communications with SASL_PLAINTEXT authentication.
kafka 16:54:47.89 INFO ==> Configuring Kafka for client communications with SASL_PLAINTEXT authentication.
kafka 16:54:47.91 INFO ==> Generating JAAS authentication file
kafka 16:54:47.93 INFO ==> ** Kafka setup finished! **
kafka 16:54:47.95 INFO ==> ** Starting Kafka **
[2022-05-29 16:54:49,343] INFO Registered kafka:type=kafka.Log4jController MBean (kafka.utils.Log4jControllerRegistration$)
[2022-05-29 16:54:49,988] INFO Setting -D jdk.tls.rejectClientInitiatedRenegotiation=true to disable client-initiated TLS renegotiation (org.apache.zookeeper.common.X509Util)
[2022-05-29 16:54:50,157] INFO Registered signal handlers for TERM, INT, HUP (org.apache.kafka.common.utils.LoggingSignalHandler)
[2022-05-29 16:54:50,163] INFO starting (kafka.server.KafkaServer)
[2022-05-29 16:54:50,164] INFO Connecting to zookeeper on sharif-zookeeper (kafka.server.KafkaServer)
[2022-05-29 16:54:50,188] INFO [ZooKeeperClient Kafka server] Initializing a new session to sharif-zookeeper. (kafka.zookeeper.ZooKeeperClient)
[2022-05-29 16:54:50,194] INFO Client environment:zookeeper.version=3.5.9-83df9301aa5c2a5d284a9940177808c01bc35cef, built on 01/06/2021 20:03 GMT (org.apache.zookeeper.ZooKeeper)
[2022-05-29 16:54:50,196] INFO Client environment:host.name=sharif-kafka-0.sharif-kafka-headless.default.svc.cluster.local (org.apache.zookeeper.ZooKeeper)
[2022-05-29 16:54:50,196] INFO Client environment:java.version=11.0.12 (org.apache.zookeeper.ZooKeeper)
[2022-05-29 16:54:50,196] INFO Client environment:java.vendor=BellSoft (org.apache.zookeeper.ZooKeeper)
[2022-05-29 16:54:50,196] INFO Client environment:java.home=/opt/bitnami/java (org.apache.zookeeper.ZooKeeper)
[2022-05-29 16:54:50,197] INFO Client environment:java.class.path=/opt/bitnami/kafka/bin/../libs/activation-1.1.1.jar:/opt/bitnami/kafka/bin/../libs/aopalliance-repackaged-2.6.1.jar:/opt/bitnami/kafka/bin/../libs/argparse4j-0.7.0.jar:/opt/bitnami/kafka/bin/../libs/audience-annotations-0.5.0.jar:/opt/bitnami/kafka/bin/../libs/commons-cli-1.4.jar:/opt/bitnami/kafka/bin/../libs/commons-lang3-3.8.1.jar:/opt/bitnami/kafka/bin/../libs/connect-api-2.8.0.jar:/opt/bitnami/kafka/bin/../libs/connect-basic-auth-extension-2.8.0.jar:/opt/bitnami/kafka/bin/../libs/connect-file-2.8.0.jar:/opt/bitnami/kafka/bin/../libs/connect-json-2.8.0.jar:/opt/bitnami/kafka/bin/../libs/connect-mirror-2.8.0.jar:/opt/bitnami/kafka/bin/../libs/connect-mirror-client-2.8.0.jar:/opt/bitnami/kafka/bin/../libs/connect-runtime-2.8.0.jar:/opt/bitnami/kafka/bin/../libs/connect-transforms-2.8.0.jar:/opt/bitnami/kafka/bin/../libs/hk2-api-2.6.1.jar:/opt/bitnami/kafka/bin/../libs/hk2-locator-2.6.1.jar:/opt/bitnami/kafka/bin/../libs/hk2-utils-2.6.1.jar:/opt/bitnami/kafka/bin/../libs/jackson-annotations-2.10.5.jar:/opt/bitnami/kafka/bin/../libs/jackson-core-2.10.5.jar:/opt/bitnami/kafka/bin/../libs/jackson-databind-2.10.5.1.jar:/opt/bitnami/kafka/bin/../libs/jackson-dataformat-csv-2.10.5.jar:/opt/bitnami/kafka/bin/../libs/jackson-datatype-jdk8-2.10.5.jar:/opt/bitnami/kafka/bin/../libs/jackson-jaxrs-base-2.10.5.jar:/opt/bitnami/kafka/bin/../libs/jackson-jaxrs-json-provider-2.10.5.jar:/opt/bitnami/kafka/bin/../libs/jackson-module-jaxb-annotations-2.10.5.jar:/opt/bitnami/kafka/bin/../libs/jackson-module-paranamer-2.10.5.jar:/opt/bitnami/kafka/bin/../libs/jackson-module-scala_2.12-2.10.5.jar:/opt/bitnami/kafka/bin/../libs/jakarta.activation-api-1.2.1.jar:/opt/bitnami/kafka/bin/../libs/jakarta.annotation-api-1.3.5.jar:/opt/bitnami/kafka/bin/../libs/jakarta.inject-2.6.1.jar:/opt/bitnami/kafka/bin/../libs/jakarta.validation-api-2.0.2.jar:/opt/bitnami/kafka/bin/../libs/jakarta.ws.rs-api-2.1.6.jar:/opt/bitnami/kafka/bin/../libs/jakarta.xml.bind-api-2.3.2.jar:/opt/bitnami/kafka/bin/../libs/javassist-3.27.0-GA.jar:/opt/bitnami/kafka/bin/../libs/javax.servlet-api-3.1.0.jar:/opt/bitnami/kafka/bin/../libs/javax.ws.rs-api-2.1.1.jar:/opt/bitnami/kafka/bin/../libs/jaxb-api-2.3.0.jar:/opt/bitnami/kafka/bin/../libs/jersey-client-2.31.jar:/opt/bitnami/kafka/bin/../libs/jersey-common-2.31.jar:/opt/bitnami/kafka/bin/../libs/jersey-container-servlet-2.31.jar:/opt/bitnami/kafka/bin/../libs/jersey-container-servlet-core-2.31.jar:/opt/bitnami/kafka/bin/../libs/jersey-hk2-2.31.jar:/opt/bitnami/kafka/bin/../libs/jersey-media-jaxb-2.31.jar:/opt/bitnami/kafka/bin/../libs/jersey-server-2.31.jar:/opt/bitnami/kafka/bin/../libs/jetty-client-9.4.39.v20210325.jar:/opt/bitnami/kafka/bin/../libs/jetty-continuation-9.4.39.v20210325.jar:/opt/bitnami/kafka/bin/../libs/jetty-http-9.4.39.v20210325.jar:/opt/bitnami/kafka/bin/../libs/jetty-io-9.4.39.v20210325.jar:/opt/bitnami/kafka/bin/../libs/jetty-security-9.4.39.v20210325.jar:/opt/bitnami/kafka/bin/../libs/jetty-server-9.4.39.v20210325.jar:/opt/bitnami/kafka/bin/../libs/jetty-servlet-9.4.39.v20210325.jar:/opt/bitnami/kafka/bin/../libs/jetty-servlets-9.4.39.v20210325.jar:/opt/bitnami/kafka/bin/../libs/jetty-util-9.4.39.v20210325.jar:/opt/bitnami/kafka/bin/../libs/jetty-util-ajax-9.4.39.v20210325.jar:/opt/bitnami/kafka/bin/../libs/jline-3.12.1.jar:/opt/bitnami/kafka/bin/../libs/jopt-simple-5.0.4.jar:/opt/bitnami/kafka/bin/../libs/kafka-clients-2.8.0.jar:/opt/bitnami/kafka/bin/../libs/kafka-log4j-appender-2.8.0.jar:/opt/bitnami/kafka/bin/../libs/kafka-metadata-2.8.0.jar:/opt/bitnami/kafka/bin/../libs/kafka-raft-2.8.0.jar:/opt/bitnami/kafka/bin/../libs/kafka-shell-2.8.0.jar:/opt/bitnami/kafka/bin/../libs/kafka-streams-2.8.0.jar:/opt/bitnami/kafka/bin/../libs/kafka-streams-examples-2.8.0.jar:/opt/bitnami/kafka/bin/../libs/kafka-streams-scala_2.12-2.8.0.jar:/opt/bitnami/kafka/bin/../libs/kafka-streams-test-utils-2.8.0.jar:/opt/bitnami/kafka/bin/../libs/kafka-tools-2.8.0.jar:/opt/bitnami/kafka/bin/../libs/kafka_2.12-2.8.0-sources.jar:/opt/bitnami/kafka/bin/../libs/kafka_2.12-2.8.0.jar:/opt/bitnami/kafka/bin/../libs/log4j-1.2.17.jar:/opt/bitnami/kafka/bin/../libs/lz4-java-1.7.1.jar:/opt/bitnami/kafka/bin/../libs/maven-artifact-3.6.3.jar:/opt/bitnami/kafka/bin/../libs/metrics-core-2.2.0.jar:/opt/bitnami/kafka/bin/../libs/netty-buffer-4.1.62.Final.jar:/opt/bitnami/kafka/bin/../libs/netty-codec-4.1.62.Final.jar:/opt/bitnami/kafka/bin/../libs/netty-common-4.1.62.Final.jar:/opt/bitnami/kafka/bin/../libs/netty-handler-4.1.62.Final.jar:/opt/bitnami/kafka/bin/../libs/netty-resolver-4.1.62.Final.jar:/opt/bitnami/kafka/bin/../libs/netty-transport-4.1.62.Final.jar:/opt/bitnami/kafka/bin/../libs/netty-transport-native-epoll-4.1.62.Final.jar:/opt/bitnami/kafka/bin/../libs/netty-transport-native-unix-common-4.1.62.Final.jar:/opt/bitnami/kafka/bin/../libs/osgi-resource-locator-1.0.3.jar:/opt/bitnami/kafka/bin/../libs/paranamer-2.8.jar:/opt/bitnami/kafka/bin/../libs/plexus-utils-3.2.1.jar:/opt/bitnami/kafka/bin/../libs/reflections-0.9.12.jar:/opt/bitnami/kafka/bin/../libs/rocksdbjni-5.18.4.jar:/opt/bitnami/kafka/bin/../libs/scala-collection-compat_2.12-2.3.0.jar:/opt/bitnami/kafka/bin/../libs/scala-java8-compat_2.12-0.9.1.jar:/opt/bitnami/kafka/bin/../libs/scala-library-2.12.13.jar:/opt/bitnami/kafka/bin/../libs/scala-logging_2.12-3.9.2.jar:/opt/bitnami/kafka/bin/../libs/scala-reflect-2.12.13.jar:/opt/bitnami/kafka/bin/../libs/slf4j-api-1.7.30.jar:/opt/bitnami/kafka/bin/../libs/slf4j-log4j12-1.7.30.jar:/opt/bitnami/kafka/bin/../libs/snappy-java-1.1.8.1.jar:/opt/bitnami/kafka/bin/../libs/zookeeper-3.5.9.jar:/opt/bitnami/kafka/bin/../libs/zookeeper-jute-3.5.9.jar:/opt/bitnami/kafka/bin/../libs/zstd-jni-1.4.9-1.jar (org.apache.zookeeper.ZooKeeper)
[2022-05-29 16:54:50,197] INFO Client environment:java.library.path=/usr/java/packages/lib:/usr/lib64:/lib64:/lib:/usr/lib (org.apache.zookeeper.ZooKeeper)
[2022-05-29 16:54:50,198] INFO Client environment:java.io.tmpdir=/tmp (org.apache.zookeeper.ZooKeeper)
[2022-05-29 16:54:50,198] INFO Client environment:java.compiler=<NA> (org.apache.zookeeper.ZooKeeper)
[2022-05-29 16:54:50,198] INFO Client environment:os.name=Linux (org.apache.zookeeper.ZooKeeper)
[2022-05-29 16:54:50,198] INFO Client environment:os.arch=amd64 (org.apache.zookeeper.ZooKeeper)
[2022-05-29 16:54:50,198] INFO Client environment:os.version=5.4.190-107.353.amzn2.x86_64 (org.apache.zookeeper.ZooKeeper)
[2022-05-29 16:54:50,199] INFO Client environment:user.name=? (org.apache.zookeeper.ZooKeeper)
[2022-05-29 16:54:50,199] INFO Client environment:user.home=? (org.apache.zookeeper.ZooKeeper)
[2022-05-29 16:54:50,199] INFO Client environment:user.dir=/ (org.apache.zookeeper.ZooKeeper)
[2022-05-29 16:54:50,199] INFO Client environment:os.memory.free=1011MB (org.apache.zookeeper.ZooKeeper)
[2022-05-29 16:54:50,200] INFO Client environment:os.memory.max=1024MB (org.apache.zookeeper.ZooKeeper)
[2022-05-29 16:54:50,200] INFO Client environment:os.memory.total=1024MB (org.apache.zookeeper.ZooKeeper)
[2022-05-29 16:54:50,203] INFO Initiating client connection, connectString=sharif-zookeeper sessionTimeout=18000 watcher=kafka.zookeeper.ZooKeeperClient$ZooKeeperClientWatcher$#447a020 (org.apache.zookeeper.ZooKeeper)
[2022-05-29 16:54:50,210] INFO jute.maxbuffer value is 4194304 Bytes (org.apache.zookeeper.ClientCnxnSocket)
[2022-05-29 16:54:50,216] INFO zookeeper.request.timeout value is 0. feature enabled= (org.apache.zookeeper.ClientCnxn)
[2022-05-29 16:54:50,218] INFO [ZooKeeperClient Kafka server] Waiting until connected. (kafka.zookeeper.ZooKeeperClient)
[2022-05-29 16:54:50,367] INFO Client successfully logged in. (org.apache.zookeeper.Login)
[2022-05-29 16:54:50,371] INFO Client will use DIGEST-MD5 as SASL mechanism. (org.apache.zookeeper.client.ZooKeeperSaslClient)
[2022-05-29 16:54:50,381] INFO Opening socket connection to server sharif-zookeeper/10.100.190.137:2181. Will attempt to SASL-authenticate using Login Context section 'Client' (org.apache.zookeeper.ClientCnxn)
[2022-05-29 16:54:50,389] INFO Socket connection established, initiating session, client: /192.168.34.166:57652, server: sharif-zookeeper/10.100.190.137:2181 (org.apache.zookeeper.ClientCnxn)
[2022-05-29 16:54:50,398] INFO Session establishment complete on server sharif-zookeeper/10.100.190.137:2181, sessionid = 0x100003ea9fd0008, negotiated timeout = 18000 (org.apache.zookeeper.ClientCnxn)
[2022-05-29 16:54:50,406] INFO [ZooKeeperClient Kafka server] Connected. (kafka.zookeeper.ZooKeeperClient)
[2022-05-29 16:54:50,420] ERROR SASL authentication failed using login context 'Client' with exception: {} (org.apache.zookeeper.client.ZooKeeperSaslClient)
javax.security.sasl.SaslException: Error in authenticating with a Zookeeper Quorum member: the quorum member's saslToken is null.
at org.apache.zookeeper.client.ZooKeeperSaslClient.createSaslToken(ZooKeeperSaslClient.java:312)
at org.apache.zookeeper.client.ZooKeeperSaslClient.respondToServer(ZooKeeperSaslClient.java:275)
at org.apache.zookeeper.ClientCnxn$SendThread.readResponse(ClientCnxn.java:882)
at org.apache.zookeeper.ClientCnxnSocketNIO.doIO(ClientCnxnSocketNIO.java:103)
at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:365)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1223)
[2022-05-29 16:54:50,431] INFO Unable to read additional data from server sessionid 0x100003ea9fd0008, likely server has closed socket, closing socket connection and attempting reconnect (org.apache.zookeeper.ClientCnxn)
[2022-05-29 16:54:50,430] ERROR [ZooKeeperClient Kafka server] Auth failed. (kafka.zookeeper.ZooKeeperClient)
[2022-05-29 16:54:50,452] INFO EventThread shut down for session: 0x100003ea9fd0008 (org.apache.zookeeper.ClientCnxn)
My kafka_jaas file in kafka server:
KafkaClient {
org.apache.kafka.common.security.plain.PlainLoginModule required
username="user_kafka"
password="secret";
};
KafkaServer {
org.apache.kafka.common.security.plain.PlainLoginModule required
username="kafka"
password="CMwvKfeVJociGkSToMZQ"
user_kafka="CMwvKfeVJociGkSToMZQ"
user_user_kafka="secret";
org.apache.kafka.common.security.scram.ScramLoginModule required;
};
Client {
org.apache.kafka.common.security.plain.PlainLoginModule required
username="kafka"
password="2q0T4HFZwa21DCRlfqxX";
};
My zoo_jaas file in ZooKeeper server:
Client {
org.apache.zookeeper.server.auth.DigestLoginModule required
username="kafka"
password="secret";
};
Server {
org.apache.zookeeper.server.auth.DigestLoginModule required
user_kafka="secret";
};
Any help/suggestion would be really beneficial. Thanks.
The following setup worked for me:
Kafka jaas file:
KafkaServer {
org.apache.kafka.common.security.plain.PlainLoginModule required
username="admin"
password="admin-secret"
user_admin="admin-secret";
};
Client {
org.apache.kafka.common.security.plain.PlainLoginModule required
username="admin"
password="admin-secret";
};
Zookeeper jaas file:
Server {
org.apache.zookeeper.server.auth.DigestLoginModule required
user_admin="admin-secret";
};
Kafka producer/consumer client properties:
security.protocol=SASL_PLAINTEXT
sasl.mechanism=PLAIN
sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required username="admin" password="admin-secret";
I noticed that you was running Kafka with Bitnami Kafka container. It can be running successfully using Docker Compose.
First, create a docker-compose.yml file like this:
version: '3'
services:
zookeeper:
image: 'bitnami/zookeeper:3.6'
ports:
- '2181:2181'
environment:
- ZOO_ENABLE_AUTH=yes
- ZOO_SERVER_USERS=kafka
- ZOO_SERVER_PASSWORDS=secret
- ZOO_CLIENT_USER=kafka
- ZOO_CLIENT_PASSWORD=secret
kafka:
image: 'bitnami/kafka:2.8.1'
ports:
- '9093:9093'
environment:
- ALLOW_PLAINTEXT_LISTENER=no
- KAFKA_CFG_ZOOKEEPER_CONNECT=zookeeper:2181
- KAFKA_CFG_LISTENERS=INTERNAL://:9092,CLIENT://:9093,
- KAFKA_CFG_ADVERTISED_LISTENERS=INTERNAL://kafka:9092,CLIENT://localhost:9093
- KAFKA_INTER_BROKER_LISTENER_NAME=INTERNAL
- KAFKA_CFG_LISTENER_SECURITY_PROTOCOL_MAP=INTERNAL:SASL_PLAINTEXT,CLIENT:SASL_PLAINTEXT
- KAFKA_CFG_SASL_MECHANISM_INTER_BROKER_PROTOCOL=PLAIN
#Client credentials
- KAFKA_CLIENT_USERS=user_kafka
- KAFKA_CLIENT_PASSWORDS=secret
#Interbroker credentials
- KAFKA_INTER_BROKER_USER=kafka
- KAFKA_INTER_BROKER_PASSWORD=CMwvKfeVJociGkSToMZQ
#Zookeeper credentials
- KAFKA_ZOOKEEPER_PROTOCOL=SASL
- KAFKA_ZOOKEEPER_USER=kafka
- KAFKA_ZOOKEEPER_PASSWORD=secret
depends_on:
- zookeeper
And then launch the containers using:
$ docker-compose up -d
# list the containers
$ docker-compose ps
Finally you will find kafka and zookeeper containers running.
This question already has answers here:
How to install Kafka on Windows?
(9 answers)
Closed 1 year ago.
I downloaded the kafka_2.13-3.0.0 in Window 10. I opened up CMD as admin. I was able to run zookeeper using the command bin\windows\zookeeper-server-start.bat config\zookeeper.properties. However, when i tried to start the kafka server with the following command
bin\windows\kafka-server-start.bat config\server.properties
, it gave me the following error" below. I am confused here because I can see the kafka-logs being created (see image below). Can someone help me out please ?
Output from running Zookeeper command
[2021-12-01 20:16:17,208] INFO Server environment:java.io.tmpdir=C:\Users\user123456~1\AppData\Local\Temp\ (org.apache.zookeeper.server.ZooKeeperServer)
[2021-12-01 20:16:17,208] INFO Server environment:java.compiler=<NA> (org.apache.zookeeper.server.ZooKeeperServer)
[2021-12-01 20:16:17,208] INFO Server environment:os.name=Windows 10 (org.apache.zookeeper.server.ZooKeeperServer)
[2021-12-01 20:16:17,209] INFO Server environment:os.arch=amd64 (org.apache.zookeeper.server.ZooKeeperServer)
[2021-12-01 20:16:17,209] INFO Server environment:os.version=10.0 (org.apache.zookeeper.server.ZooKeeperServer)
[2021-12-01 20:16:17,209] INFO Server environment:user.name=user123456 (org.apache.zookeeper.server.ZooKeeperServer)
[2021-12-01 20:16:17,210] INFO Server environment:user.home=C:\Users\user123456 (org.apache.zookeeper.server.ZooKeeperServer)
[2021-12-01 20:16:17,210] INFO Server environment:user.dir=C:\kafka_2.13-3.0.0 (org.apache.zookeeper.server.ZooKeeperServer)
[2021-12-01 20:16:17,211] INFO Server environment:os.memory.free=493MB (org.apache.zookeeper.server.ZooKeeperServer)
[2021-12-01 20:16:17,211] INFO Server environment:os.memory.max=512MB (org.apache.zookeeper.server.ZooKeeperServer)
[2021-12-01 20:16:17,211] INFO Server environment:os.memory.total=512MB (org.apache.zookeeper.server.ZooKeeperServer)
[2021-12-01 20:16:17,212] INFO zookeeper.enableEagerACLCheck = false (org.apache.zookeeper.server.ZooKeeperServer)
[2021-12-01 20:16:17,212] INFO zookeeper.digest.enabled = true (org.apache.zookeeper.server.ZooKeeperServer)
[2021-12-01 20:16:17,212] INFO zookeeper.closeSessionTxn.enabled = true (org.apache.zookeeper.server.ZooKeeperServer)
[2021-12-01 20:16:17,213] INFO zookeeper.flushDelay=0 (org.apache.zookeeper.server.ZooKeeperServer)
[2021-12-01 20:16:17,213] INFO zookeeper.maxWriteQueuePollTime=0 (org.apache.zookeeper.server.ZooKeeperServer)
[2021-12-01 20:16:17,214] INFO zookeeper.maxBatchSize=1000 (org.apache.zookeeper.server.ZooKeeperServer)
[2021-12-01 20:16:17,214] INFO zookeeper.intBufferStartingSizeBytes = 1024 (org.apache.zookeeper.server.ZooKeeperServer)
[2021-12-01 20:16:17,215] INFO Weighed connection throttling is disabled (org.apache.zookeeper.server.BlueThrottle)
[2021-12-01 20:16:17,216] INFO minSessionTimeout set to 6000 (org.apache.zookeeper.server.ZooKeeperServer)
[2021-12-01 20:16:17,216] INFO maxSessionTimeout set to 60000 (org.apache.zookeeper.server.ZooKeeperServer)
[2021-12-01 20:16:17,217] INFO Response cache size is initialized with value 400. (org.apache.zookeeper.server.ResponseCache)
[2021-12-01 20:16:17,217] INFO Response cache size is initialized with value 400. (org.apache.zookeeper.server.ResponseCache)
[2021-12-01 20:16:17,218] INFO zookeeper.pathStats.slotCapacity = 60 (org.apache.zookeeper.server.util.RequestPathMetricsCollector)
[2021-12-01 20:16:17,220] INFO zookeeper.pathStats.slotDuration = 15 (org.apache.zookeeper.server.util.RequestPathMetricsCollector)
[2021-12-01 20:16:17,221] INFO zookeeper.pathStats.maxDepth = 6 (org.apache.zookeeper.server.util.RequestPathMetricsCollector)
[2021-12-01 20:16:17,221] INFO zookeeper.pathStats.initialDelay = 5 (org.apache.zookeeper.server.util.RequestPathMetricsCollector)
[2021-12-01 20:16:17,221] INFO zookeeper.pathStats.delay = 5 (org.apache.zookeeper.server.util.RequestPathMetricsCollector)
[2021-12-01 20:16:17,222] INFO zookeeper.pathStats.enabled = false (org.apache.zookeeper.server.util.RequestPathMetricsCollector)
[2021-12-01 20:16:17,224] INFO The max bytes for all large requests are set to 104857600 (org.apache.zookeeper.server.ZooKeeperServer)
[2021-12-01 20:16:17,224] INFO The large request threshold is set to -1 (org.apache.zookeeper.server.ZooKeeperServer)
[2021-12-01 20:16:17,224] INFO Created server with tickTime 3000 minSessionTimeout 6000 maxSessionTimeout 60000 clientPortListenBacklog -1 datadir \tmp\zookeeper\version-2 snapdir \tmp\zookeeper\version-2 (org.apache.zookeeper.server.ZooKeeperServer)
[2021-12-01 20:16:17,234] INFO Using org.apache.zookeeper.server.NIOServerCnxnFactory as server connection factory (org.apache.zookeeper.server.ServerCnxnFactory)
[2021-12-01 20:16:17,235] WARN maxCnxns is not configured, using default value 0. (org.apache.zookeeper.server.ServerCnxnFactory)
[2021-12-01 20:16:17,236] INFO Configuring NIO connection handler with 10s sessionless connection timeout, 2 selector thread(s), 32 worker threads, and 64 kB direct buffers. (org.apache.zookeeper.server.NIOServerCnxnFactory)
[2021-12-01 20:16:17,238] INFO binding to port 0.0.0.0/0.0.0.0:2181 (org.apache.zookeeper.server.NIOServerCnxnFactory)
[2021-12-01 20:16:17,250] INFO Using org.apache.zookeeper.server.watch.WatchManager as watch manager (org.apache.zookeeper.server.watch.WatchManagerFactory)
[2021-12-01 20:16:17,250] INFO Using org.apache.zookeeper.server.watch.WatchManager as watch manager (org.apache.zookeeper.server.watch.WatchManagerFactory)
[2021-12-01 20:16:17,251] INFO zookeeper.snapshotSizeFactor = 0.33 (org.apache.zookeeper.server.ZKDatabase)
[2021-12-01 20:16:17,251] INFO zookeeper.commitLogCount=500 (org.apache.zookeeper.server.ZKDatabase)
[2021-12-01 20:16:17,256] INFO zookeeper.snapshot.compression.method = CHECKED (org.apache.zookeeper.server.persistence.SnapStream)
[2021-12-01 20:16:17,256] INFO Snapshotting: 0x0 to \tmp\zookeeper\version-2\snapshot.0 (org.apache.zookeeper.server.persistence.FileTxnSnapLog)
[2021-12-01 20:16:17,260] INFO Snapshot loaded in 8 ms, highest zxid is 0x0, digest is 1371985504 (org.apache.zookeeper.server.ZKDatabase)
[2021-12-01 20:16:17,260] INFO Snapshotting: 0x0 to \tmp\zookeeper\version-2\snapshot.0 (org.apache.zookeeper.server.persistence.FileTxnSnapLog)
[2021-12-01 20:16:17,261] INFO Snapshot taken in 1 ms (org.apache.zookeeper.server.ZooKeeperServer)
[2021-12-01 20:16:17,274] INFO PrepRequestProcessor (sid:0) started, reconfigEnabled=false (org.apache.zookeeper.server.PrepRequestProcessor)
[2021-12-01 20:16:17,274] INFO zookeeper.request_throttler.shutdownTimeout = 10000 (org.apache.zookeeper.server.RequestThrottler)
[2021-12-01 20:16:17,286] INFO Using checkIntervalMs=60000 maxPerMinute=10000 maxNeverUsedIntervalMs=0 (org.apache.zookeeper.server.ContainerManager)
[2021-12-01 20:16:17,287] INFO ZooKeeper audit is disabled. (org.apache.zookeeper.audit.ZKAuditProvider)
Output from running kafka command
[2021-12-01 20:18:22,577] INFO Client environment:java.io.tmpdir=C:\Users\user123456~1\AppData\Local\Temp\ (org.apache.zookeeper.ZooKeeper)
[2021-12-01 20:18:22,577] INFO Client environment:java.compiler=<NA> (org.apache.zookeeper.ZooKeeper)
[2021-12-01 20:18:22,578] INFO Client environment:os.name=Windows 10 (org.apache.zookeeper.ZooKeeper)
[2021-12-01 20:18:22,579] INFO Client environment:os.arch=amd64 (org.apache.zookeeper.ZooKeeper)
[2021-12-01 20:18:22,579] INFO Client environment:os.version=10.0 (org.apache.zookeeper.ZooKeeper)
[2021-12-01 20:18:22,580] INFO Client environment:user.name=user123456 (org.apache.zookeeper.ZooKeeper)
[2021-12-01 20:18:22,581] INFO Client environment:user.home=C:\Users\user123456 (org.apache.zookeeper.ZooKeeper)
[2021-12-01 20:18:22,581] INFO Client environment:user.dir=C:\kafka_2.13-3.0.0 (org.apache.zookeeper.ZooKeeper)
[2021-12-01 20:18:22,582] INFO Client environment:os.memory.free=1011MB (org.apache.zookeeper.ZooKeeper)
[2021-12-01 20:18:22,583] INFO Client environment:os.memory.max=1024MB (org.apache.zookeeper.ZooKeeper)
[2021-12-01 20:18:22,583] INFO Client environment:os.memory.total=1024MB (org.apache.zookeeper.ZooKeeper)
[2021-12-01 20:18:22,585] INFO Initiating client connection, connectString=localhost:2181 sessionTimeout=18000 watcher=kafka.zookeeper.ZooKeeperClient$ZooKeeperClientWatcher$#74f0ea28 (org.apache.zookeeper.ZooKeeper)
[2021-12-01 20:18:22,593] INFO jute.maxbuffer value is 4194304 Bytes (org.apache.zookeeper.ClientCnxnSocket)
[2021-12-01 20:18:22,597] INFO zookeeper.request.timeout value is 0. feature enabled=false (org.apache.zookeeper.ClientCnxn)
[2021-12-01 20:18:22,598] INFO [ZooKeeperClient Kafka server] Waiting until connected. (kafka.zookeeper.ZooKeeperClient)
[2021-12-01 20:18:22,601] INFO Opening socket connection to server localhost/127.0.0.1:2181. (org.apache.zookeeper.ClientCnxn)
[2021-12-01 20:18:22,601] INFO SASL config status: Will not attempt to authenticate using SASL (unknown error) (org.apache.zookeeper.ClientCnxn)
[2021-12-01 20:18:22,602] INFO Socket connection established, initiating session, client: /127.0.0.1:57191, server: localhost/127.0.0.1:2181 (org.apache.zookeeper.ClientCnxn)
[2021-12-01 20:18:22,617] INFO Session establishment complete on server localhost/127.0.0.1:2181, session id = 0x1000008c9c20000, negotiated timeout = 18000 (org.apache.zookeeper.ClientCnxn)
[2021-12-01 20:18:22,619] INFO [ZooKeeperClient Kafka server] Connected. (kafka.zookeeper.ZooKeeperClient)
[2021-12-01 20:18:22,720] INFO [feature-zk-node-event-process-thread]: Starting (kafka.server.FinalizedFeatureChangeListener$ChangeNotificationProcessorThread)
[2021-12-01 20:18:22,729] INFO Feature ZK node at path: /feature does not exist (kafka.server.FinalizedFeatureChangeListener)
[2021-12-01 20:18:22,729] INFO Cleared cache (kafka.server.FinalizedFeatureCache)
[2021-12-01 20:18:22,844] INFO Cluster ID = j7KyANF-S2OuLlr_IYyqDw (kafka.server.KafkaServer)
[2021-12-01 20:18:22,846] WARN No meta.properties file under dir C:\tmp\kafka-logs\meta.properties (kafka.server.BrokerMetadataCheckpoint)
[2021-12-01 20:18:22,876] INFO KafkaConfig values:
...
[2021-12-01 20:18:22,918] INFO [ThrottledChannelReaper-Fetch]: Starting (kafka.server.ClientQuotaManager$ThrottledChannelReaper)
[2021-12-01 20:18:22,919] INFO [ThrottledChannelReaper-Produce]: Starting (kafka.server.ClientQuotaManager$ThrottledChannelReaper)
[2021-12-01 20:18:22,919] INFO [ThrottledChannelReaper-Request]: Starting (kafka.server.ClientQuotaManager$ThrottledChannelReaper)
[2021-12-01 20:18:22,920] INFO [ThrottledChannelReaper-ControllerMutation]: Starting (kafka.server.ClientQuotaManager$ThrottledChannelReaper)
[2021-12-01 20:18:22,931] INFO Log directory C:\tmp\kafka-logs not found, creating it. (kafka.log.LogManager)
[2021-12-01 20:18:22,933] ERROR Failed to create or validate data directory C:\tmp\kafka-logs (kafka.server.LogDirFailureChannel)
java.nio.file.AccessDeniedException: C:\tmp
at sun.nio.fs.WindowsException.translateToIOException(WindowsException.java:83)
at sun.nio.fs.WindowsException.rethrowAsIOException(WindowsException.java:97)
at sun.nio.fs.WindowsException.rethrowAsIOException(WindowsException.java:102)
at sun.nio.fs.WindowsFileSystemProvider.newFileChannel(WindowsFileSystemProvider.java:115)
at java.nio.channels.FileChannel.open(FileChannel.java:287)
at java.nio.channels.FileChannel.open(FileChannel.java:335)
at org.apache.kafka.common.utils.Utils.flushDir(Utils.java:953)
at kafka.log.LogManager.$anonfun$createAndValidateLogDirs$1(LogManager.scala:159)
at scala.collection.IterableOnceOps.foreach(IterableOnce.scala:563)
at scala.collection.IterableOnceOps.foreach$(IterableOnce.scala:561)
at scala.collection.AbstractIterable.foreach(Iterable.scala:919)
at kafka.log.LogManager.createAndValidateLogDirs(LogManager.scala:149)
at kafka.log.LogManager.<init>(LogManager.scala:88)
at kafka.log.LogManager$.apply(LogManager.scala:1283)
at kafka.server.KafkaServer.startup(KafkaServer.scala:254)
at kafka.Kafka$.main(Kafka.scala:109)
at kafka.Kafka.main(Kafka.scala)
[2021-12-01 20:18:22,936] ERROR Shutdown broker because none of the specified log dirs from C:\tmp\kafka-logs can be created or validated (kafka.log.LogManager)
I also tried running icacls "C:\tmp\" /q /c /t /grant Users:F. I am still getting the same error.
It turns out that 3.0.0 has issues with Windows. I downgraded to 2.8.1 and things worked fine. I hate Windows.
I am trying to run Kafka cluster on AWS EKS cluster (v1.16). I am using bitnami helm charts.
https://github.com/bitnami/charts/tree/master/bitnami/kafka
https://github.com/bitnami/charts/tree/master/bitnami/zookeeper
I have deployed zookeeper ensemble successfully using below command:
helm install zookeeper bitnami/zookeeper --set replicaCount=3 --set auth.enabled=false --set allowAnonymousLogin=true --set persistence.storageClass=ebs --set persistence.accessModes={ReadWriteOnce} --set persistence.size=1Gi --set podLabels."app\.kubernetes\.io/version"="1.0"
It outputs:
ZooKeeper can be accessed via port 2181 on the following DNS name from within your cluster:
zookeeper.pulse.svc.cluster.local
Now, I am trying to deploy Kafka cluster with below command:
helm install kafka bitnami/kafka --set replicaCount=3 --set zookeeper.enabled=false --set externalZookeeper.servers=zookeeper.pulse.svc.cluster.local --set autoCreateTopicsEnable=true --set persistence.storageClass=ebs --set persistence.accessModes={ReadWriteOnce} --set persistence.size=1Gi --set podLabels."app\.kubernetes\.io/version"="1.0"
It outputs:
Kafka can be accessed by consumers via port 9092 on the following DNS name from within your cluster:
kafka.pulse.svc.cluster.local
Each Kafka broker can be accessed by producers via port 9092 on the following DNS name(s) from within your cluster:
kafka-0.kafka-headless.pulse.svc.cluster.local
kafka-1.kafka-headless.pulse.svc.cluster.local
kafka-2.kafka-headless.pulse.svc.cluster.local
It creates 3 pods but none of the pod is able to connect to zookeeper. I am not getting what is the issue here.
Kafka pod logs:
2020-07-06T11:22:40.506134648Z 11:22:40.50 Welcome to the Bitnami kafka container
2020-07-06T11:22:40.507301179Z 11:22:40.50 Subscribe to project updates by watching https://github.com/bitnami/bitnami-docker-kafka
2020-07-06T11:22:40.508519907Z 11:22:40.50 Submit issues and feature requests at https://github.com/bitnami/bitnami-docker-kafka/issues
2020-07-06T11:22:40.51039472Z 11:22:40.50
2020-07-06T11:22:40.511630347Z 11:22:40.51 INFO ==> ** Starting Kafka setup **
2020-07-06T11:22:40.55379314Z 11:22:40.55 WARN ==> You set the environment variable ALLOW_PLAINTEXT_LISTENER=yes. For safety reasons, do not use this flag in a production environment.
2020-07-06T11:22:40.561203295Z 11:22:40.56 INFO ==> Initializing Kafka...
2020-07-06T11:22:40.565054949Z 11:22:40.56 INFO ==> No injected configuration files found, creating default config files
2020-07-06T11:22:40.723721499Z 11:22:40.72 INFO ==> Configuring Kafka for inter-broker communications with PLAINTEXT authentication.
2020-07-06T11:22:40.726161543Z 11:22:40.72 WARN ==> Inter-broker communications are configured as PLAINTEXT. This is not safe for production environments.
2020-07-06T11:22:40.727497832Z 11:22:40.72 INFO ==> Configuring Kafka for client communications with PLAINTEXT authentication.
2020-07-06T11:22:40.731790674Z 11:22:40.73 WARN ==> Client communications are configured using PLAINTEXT listeners. For safety reasons, do not use this in a production environment.
2020-07-06T11:22:40.73699684Z 11:22:40.73 INFO ==> ** Kafka setup finished! **
2020-07-06T11:22:40.737001986Z
2020-07-06T11:22:40.746297253Z 11:22:40.74 INFO ==> ** Starting Kafka **
2020-07-06T11:22:41.512303802Z [2020-07-06 11:22:41,511] INFO Registered kafka:type=kafka.Log4jController MBean (kafka.utils.Log4jControllerRegistration$)
2020-07-06T11:22:42.008231959Z [2020-07-06 11:22:42,007] INFO starting (kafka.server.KafkaServer)
2020-07-06T11:22:42.009112085Z [2020-07-06 11:22:42,008] INFO Connecting to zookeeper on zookeeper.pulse.svc.cluster.local (kafka.server.KafkaServer)
2020-07-06T11:22:42.028233655Z [2020-07-06 11:22:42,028] INFO [ZooKeeperClient Kafka server] Initializing a new session to zookeeper.pulse.svc.cluster.local. (kafka.zookeeper.ZooKeeperClient)
2020-07-06T11:22:42.032763227Z [2020-07-06 11:22:42,032] INFO Client environment:zookeeper.version=3.5.7-f0fdd52973d373ffd9c86b81d99842dc2c7f660e, built on 02/10/2020 11:30 GMT (org.apache.zookeeper.ZooKeeper)
2020-07-06T11:22:42.032776511Z [2020-07-06 11:22:42,032] INFO Client environment:host.name=kafka-0.kafka-headless.pulse.svc.cluster.local (org.apache.zookeeper.ZooKeeper)
2020-07-06T11:22:42.03283528Z [2020-07-06 11:22:42,032] INFO Client environment:java.version=11.0.7 (org.apache.zookeeper.ZooKeeper)
2020-07-06T11:22:42.032984414Z [2020-07-06 11:22:42,032] INFO Client environment:java.vendor=BellSoft (org.apache.zookeeper.ZooKeeper)
2020-07-06T11:22:42.033005091Z [2020-07-06 11:22:42,032] INFO Client environment:java.home=/opt/bitnami/java (org.apache.zookeeper.ZooKeeper)
2020-07-06T11:22:42.03312054Z [2020-07-06 11:22:42,032] INFO Client environment:java.class.path=/opt/bitnami/kafka/bin/../libs/activation-1.1.1.jar:/opt/bitnami/kafka/bin/../libs/aopalliance-repackaged-2.5.0.jar:/opt/bitnami/kafka/bin/../libs/argparse4j-0.7.0.jar:/opt/bitnami/kafka/bin/../libs/audience-annotations-0.5.0.jar:/opt/bitnami/kafka/bin/../libs/commons-cli-1.4.jar:/opt/bitnami/kafka/bin/../libs/commons-lang3-3.8.1.jar:/opt/bitnami/kafka/bin/../libs/connect-api-2.5.0.jar:/opt/bitnami/kafka/bin/../libs/connect-basic-auth-extension-2.5.0.jar:/opt/bitnami/kafka/bin/../libs/connect-file-2.5.0.jar:/opt/bitnami/kafka/bin/../libs/connect-json-2.5.0.jar:/opt/bitnami/kafka/bin/../libs/connect-mirror-2.5.0.jar:/opt/bitnami/kafka/bin/../libs/connect-mirror-client-2.5.0.jar:/opt/bitnami/kafka/bin/../libs/connect-runtime-2.5.0.jar:/opt/bitnami/kafka/bin/../libs/connect-transforms-2.5.0.jar:/opt/bitnami/kafka/bin/../libs/hk2-api-2.5.0.jar:/opt/bitnami/kafka/bin/../libs/hk2-locator-2.5.0.jar:/opt/bitnami/kafka/bin/../libs/hk2-utils-2.5.0.jar:/opt/bitnami/kafka/bin/../libs/jackson-annotations-2.10.2.jar:/opt/bitnami/kafka/bin/../libs/jackson-core-2.10.2.jar:/opt/bitnami/kafka/bin/../libs/jackson-databind-2.10.2.jar:/opt/bitnami/kafka/bin/../libs/jackson-dataformat-csv-2.10.2.jar:/opt/bitnami/kafka/bin/../libs/jackson-datatype-jdk8-2.10.2.jar:/opt/bitnami/kafka/bin/../libs/jackson-jaxrs-base-2.10.2.jar:/opt/bitnami/kafka/bin/../libs/jackson-jaxrs-json-provider-2.10.2.jar:/opt/bitnami/kafka/bin/../libs/jackson-module-jaxb-annotations-2.10.2.jar:/opt/bitnami/kafka/bin/../libs/jackson-module-paranamer-2.10.2.jar:/opt/bitnami/kafka/bin/../libs/jackson-module-scala_2.12-2.10.2.jar:/opt/bitnami/kafka/bin/../libs/jakarta.activation-api-1.2.1.jar:/opt/bitnami/kafka/bin/../libs/jakarta.annotation-api-1.3.4.jar:/opt/bitnami/kafka/bin/../libs/jakarta.inject-2.5.0.jar:/opt/bitnami/kafka/bin/../libs/jakarta.ws.rs-api-2.1.5.jar:/opt/bitnami/kafka/bin/../libs/jakarta.xml.bind-api-2.3.2.jar:/opt/bitnami/kafka/bin/../libs/javassist-3.22.0-CR2.jar:/opt/bitnami/kafka/bin/../libs/javassist-3.26.0-GA.jar:/opt/bitnami/kafka/bin/../libs/javax.servlet-api-3.1.0.jar:/opt/bitnami/kafka/bin/../libs/javax.ws.rs-api-2.1.1.jar:/opt/bitnami/kafka/bin/../libs/jaxb-api-2.3.0.jar:/opt/bitnami/kafka/bin/../libs/jersey-client-2.28.jar:/opt/bitnami/kafka/bin/../libs/jersey-common-2.28.jar:/opt/bitnami/kafka/bin/../libs/jersey-container-servlet-2.28.jar:/opt/bitnami/kafka/bin/../libs/jersey-container-servlet-core-2.28.jar:/opt/bitnami/kafka/bin/../libs/jersey-hk2-2.28.jar:/opt/bitnami/kafka/bin/../libs/jersey-media-jaxb-2.28.jar:/opt/bitnami/kafka/bin/../libs/jersey-server-2.28.jar:/opt/bitnami/kafka/bin/../libs/jetty-client-9.4.24.v20191120.jar:/opt/bitnami/kafka/bin/../libs/jetty-continuation-9.4.24.v20191120.jar:/opt/bitnami/kafka/bin/../libs/jetty-http-9.4.24.v20191120.jar:/opt/bitnami/kafka/bin/../libs/jetty-io-9.4.24.v20191120.jar:/opt/bitnami/kafka/bin/../libs/jetty-security-9.4.24.v20191120.jar:/opt/bitnami/kafka/bin/../libs/jetty-server-9.4.24.v20191120.jar:/opt/bitnami/kafka/bin/../libs/jetty-servlet-9.4.24.v20191120.jar:/opt/bitnami/kafka/bin/../libs/jetty-servlets-9.4.24.v20191120.jar:/opt/bitnami/kafka/bin/../libs/jetty-util-9.4.24.v20191120.jar:/opt/bitnami/kafka/bin/../libs/jopt-simple-5.0.4.jar:/opt/bitnami/kafka/bin/../libs/kafka-clients-2.5.0.jar:/opt/bitnami/kafka/bin/../libs/kafka-log4j-appender-2.5.0.jar:/opt/bitnami/kafka/bin/../libs/kafka-streams-2.5.0.jar:/opt/bitnami/kafka/bin/../libs/kafka-streams-examples-2.5.0.jar:/opt/bitnami/kafka/bin/../libs/kafka-streams-scala_2.12-2.5.0.jar:/opt/bitnami/kafka/bin/../libs/kafka-streams-test-utils-2.5.0.jar:/opt/bitnami/kafka/bin/../libs/kafka-tools-2.5.0.jar:/opt/bitnami/kafka/bin/../libs/kafka_2.12-2.5.0-sources.jar:/opt/bitnami/kafka/bin/../libs/kafka_2.12-2.5.0.jar:/opt/bitnami/kafka/bin/../libs/log4j-1.2.17.jar:/opt/bitnami/kafka/bin/../libs/lz4-java-1.7.1.jar:/opt/bitnami/kafka/bin/../libs/maven-artifact-3.6.3.jar:/opt/bitnami/kafka/bin/../libs/metrics-core-2.2.0.jar:/opt/bitnami/kafka/bin/../libs/netty-buffer-4.1.45.Final.jar:/opt/bitnami/kafka/bin/../libs/netty-codec-4.1.45.Final.jar:/opt/bitnami/kafka/bin/../libs/netty-common-4.1.45.Final.jar:/opt/bitnami/kafka/bin/../libs/netty-handler-4.1.45.Final.jar:/opt/bitnami/kafka/bin/../libs/netty-resolver-4.1.45.Final.jar:/opt/bitnami/kafka/bin/../libs/netty-transport-4.1.45.Final.jar:/opt/bitnami/kafka/bin/../libs/netty-transport-native-epoll-4.1.45.Final.jar:/opt/bitnami/kafka/bin/../libs/netty-transport-native-unix-common-4.1.45.Final.jar:/opt/bitnami/kafka/bin/../libs/osgi-resource-locator-1.0.1.jar:/opt/bitnami/kafka/bin/../libs/paranamer-2.8.jar:/opt/bitnami/kafka/bin/../libs/plexus-utils-3.2.1.jar:/opt/bitnami/kafka/bin/../libs/reflections-0.9.12.jar:/opt/bitnami/kafka/bin/../libs/rocksdbjni-5.18.3.jar:/opt/bitnami/kafka/bin/../libs/scala-collection-compat_2.12-2.1.3.jar:/opt/bitnami/kafka/bin/../libs/scala-java8-compat_2.12-0.9.0.jar:/opt/bitnami/kafka/bin/../libs/scala-library-2.12.10.jar:/opt/bitnami/kafka/bin/../libs/scala-logging_2.12-3.9.2.jar:/opt/bitnami/kafka/bin/../libs/scala-reflect-2.12.10.jar:/opt/bitnami/kafka/bin/../libs/slf4j-api-1.7.30.jar:/opt/bitnami/kafka/bin/../libs/slf4j-log4j12-1.7.30.jar:/opt/bitnami/kafka/bin/../libs/snappy-java-1.1.7.3.jar:/opt/bitnami/kafka/bin/../libs/validation-api-2.0.1.Final.jar:/opt/bitnami/kafka/bin/../libs/zookeeper-3.5.7.jar:/opt/bitnami/kafka/bin/../libs/zookeeper-jute-3.5.7.jar:/opt/bitnami/kafka/bin/../libs/zstd-jni-1.4.4-7.jar (org.apache.zookeeper.ZooKeeper)
2020-07-06T11:22:42.033182063Z [2020-07-06 11:22:42,033] INFO Client environment:java.library.path=/usr/java/packages/lib:/usr/lib64:/lib64:/lib:/usr/lib (org.apache.zookeeper.ZooKeeper)
2020-07-06T11:22:42.033188827Z [2020-07-06 11:22:42,033] INFO Client environment:java.io.tmpdir=/tmp (org.apache.zookeeper.ZooKeeper)
2020-07-06T11:22:42.03322714Z [2020-07-06 11:22:42,033] INFO Client environment:java.compiler=<NA> (org.apache.zookeeper.ZooKeeper)
2020-07-06T11:22:42.033253354Z [2020-07-06 11:22:42,033] INFO Client environment:os.name=Linux (org.apache.zookeeper.ZooKeeper)
2020-07-06T11:22:42.033280159Z [2020-07-06 11:22:42,033] INFO Client environment:os.arch=amd64 (org.apache.zookeeper.ZooKeeper)
2020-07-06T11:22:42.033324405Z [2020-07-06 11:22:42,033] INFO Client environment:os.version=4.14.181-140.257.amzn2.x86_64 (org.apache.zookeeper.ZooKeeper)
2020-07-06T11:22:42.033355581Z [2020-07-06 11:22:42,033] INFO Client environment:user.name=? (org.apache.zookeeper.ZooKeeper)
2020-07-06T11:22:42.033399449Z [2020-07-06 11:22:42,033] INFO Client environment:user.home=? (org.apache.zookeeper.ZooKeeper)
2020-07-06T11:22:42.03340494Z [2020-07-06 11:22:42,033] INFO Client environment:user.dir=/ (org.apache.zookeeper.ZooKeeper)
2020-07-06T11:22:42.033468947Z [2020-07-06 11:22:42,033] INFO Client environment:os.memory.free=1015MB (org.apache.zookeeper.ZooKeeper)
2020-07-06T11:22:42.033509114Z [2020-07-06 11:22:42,033] INFO Client environment:os.memory.max=1024MB (org.apache.zookeeper.ZooKeeper)
2020-07-06T11:22:42.033536891Z [2020-07-06 11:22:42,033] INFO Client environment:os.memory.total=1024MB (org.apache.zookeeper.ZooKeeper)
2020-07-06T11:22:42.035251257Z [2020-07-06 11:22:42,035] INFO Initiating client connection, connectString=zookeeper.pulse.svc.cluster.local sessionTimeout=18000 watcher=kafka.zookeeper.ZooKeeperClient$ZooKeeperClientWatcher$#6ee6f53 (org.apache.zookeeper.ZooKeeper)
2020-07-06T11:22:42.038953719Z [2020-07-06 11:22:42,038] INFO jute.maxbuffer value is 4194304 Bytes (org.apache.zookeeper.ClientCnxnSocket)
2020-07-06T11:22:42.043407452Z [2020-07-06 11:22:42,043] INFO zookeeper.request.timeout value is 0. feature enabled= (org.apache.zookeeper.ClientCnxn)
2020-07-06T11:22:42.045196444Z [2020-07-06 11:22:42,045] INFO [ZooKeeperClient Kafka server] Waiting until connected. (kafka.zookeeper.ZooKeeperClient)
2020-07-06T11:22:42.053941415Z [2020-07-06 11:22:42,053] INFO Opening socket connection to server zookeeper.pulse.svc.cluster.local/172.20.162.36:2181. Will not attempt to authenticate using SASL (unknown error) (org.apache.zookeeper.ClientCnxn)
2020-07-06T11:22:42.057906383Z [2020-07-06 11:22:42,057] INFO Socket connection established, initiating session, client: /100.64.5.213:52738, server: zookeeper.pulse.svc.cluster.local/172.20.162.36:2181 (org.apache.zookeeper.ClientCnxn)
2020-07-06T11:22:42.061035524Z [2020-07-06 11:22:42,060] INFO Unable to read additional data from server sessionid 0x0, likely server has closed socket, closing socket connection and attempting reconnect (org.apache.zookeeper.ClientCnxn)
2020-07-06T11:22:43.632054003Z [2020-07-06 11:22:43,631] INFO Opening socket connection to server zookeeper.pulse.svc.cluster.local/172.20.162.36:2181. Will not attempt to authenticate using SASL (unknown error) (org.apache.zookeeper.ClientCnxn)
2020-07-06T11:22:43.632596098Z [2020-07-06 11:22:43,632] INFO Socket connection established, initiating session, client: /100.64.5.213:52756, server: zookeeper.pulse.svc.cluster.local/172.20.162.36:2181 (org.apache.zookeeper.ClientCnxn)
2020-07-06T11:22:43.634993004Z [2020-07-06 11:22:43,634] INFO Unable to read additional data from server sessionid 0x0, likely server has closed socket, closing socket connection and attempting reconnect (org.apache.zookeeper.ClientCnxn)
2020-07-06T11:22:44.760870715Z [2020-07-06 11:22:44,760] INFO Opening socket connection to server zookeeper.pulse.svc.cluster.local/172.20.162.36:2181. Will not attempt to authenticate using SASL (unknown error) (org.apache.zookeeper.ClientCnxn)
2020-07-06T11:22:44.761283232Z [2020-07-06 11:22:44,761] INFO Socket connection established, initiating session, client: /100.64.5.213:52772, server: zookeeper.pulse.svc.cluster.local/172.20.162.36:2181 (org.apache.zookeeper.ClientCnxn)
2020-07-06T11:22:44.763353195Z [2020-07-06 11:22:44,763] INFO Unable to read additional data from server sessionid 0x0, likely server has closed socket, closing socket connection and attempting reconnect (org.apache.zookeeper.ClientCnxn)
2
2020-07-06T11:23:12.738834004Z [2020-07-06 11:23:12,738] INFO Session: 0x0 closed (org.apache.zookeeper.ZooKeeper)
2020-07-06T11:23:12.738918322Z [2020-07-06 11:23:12,738] INFO EventThread shut down for session: 0x0 (org.apache.zookeeper.ClientCnxn)
2020-07-06T11:23:12.740751654Z [2020-07-06 11:23:12,740] INFO [ZooKeeperClient Kafka server] Closed. (kafka.zookeeper.ZooKeeperClient)
2020-07-06T11:23:12.745313347Z [2020-07-06 11:23:12,743] ERROR Fatal error during KafkaServer startup. Prepare to shutdown (kafka.server.KafkaServer)
2020-07-06T11:23:12.745331011Z kafka.zookeeper.ZooKeeperClientTimeoutException: Timed out waiting for connection while in state: CONNECTING
2020-07-06T11:23:12.745335139Z at kafka.zookeeper.ZooKeeperClient.$anonfun$waitUntilConnected$3(ZooKeeperClient.scala:262)
2020-07-06T11:23:12.745338245Z at kafka.zookeeper.ZooKeeperClient.waitUntilConnected(ZooKeeperClient.scala:258)
2020-07-06T11:23:12.745340837Z at kafka.zookeeper.ZooKeeperClient.<init>(ZooKeeperClient.scala:119)
2020-07-06T11:23:12.745343374Z at kafka.zk.KafkaZkClient$.apply(KafkaZkClient.scala:1863)
2020-07-06T11:23:12.745345577Z at kafka.server.KafkaServer.createZkClient$1(KafkaServer.scala:378)
2020-07-06T11:23:12.745347726Z at kafka.server.KafkaServer.initZkClient(KafkaServer.scala:403)
2020-07-06T11:23:12.745349947Z at kafka.server.KafkaServer.startup(KafkaServer.scala:210)
2020-07-06T11:23:12.745352077Z at kafka.server.KafkaServerStartable.startup(KafkaServerStartable.scala:44)
2020-07-06T11:23:12.745354263Z at kafka.Kafka$.main(Kafka.scala:82)
2020-07-06T11:23:12.745368643Z at kafka.Kafka.main(Kafka.scala)
2020-07-06T11:23:12.745806818Z [2020-07-06 11:23:12,745] INFO shutting down (kafka.server.KafkaServer)
2020-07-06T11:23:12.752833659Z [2020-07-06 11:23:12,752] INFO shut down completed (kafka.server.KafkaServer)
2020-07-06T11:23:12.753305908Z [2020-07-06 11:23:12,753] ERROR Exiting Kafka. (kafka.server.KafkaServerStartable)
2020-07-06T11:23:12.757961524Z [2020-07-06 11:23:12,757] INFO shutting down (kafka.server.KafkaServer)
Close
Also from Kafka pod machine, curl gives below o/p:
istio-proxy#kafka-0:/$ curl zookeeper.pulse.svc.cluster.local:2181
curl: (52) Empty reply from server
istio-proxy#kafka-0:/$
Note: I am having istio sidecars with mTLS disabled.
Please help.
UPDATE
HI,
It comes out to be the Istio porxy issue. I uninstalled Istio and it worked out.
https://github.com/bitnami/bitnami-docker-kafka/issues/38#issuecomment-451381003
This works fine for me on my local cluster. Since you are using EKS, you are most likely using AWS CNI(?). CNI allocates IP addresses in your VPC and if you are not allowing your security groups access to the VPC range it will not be able to access the pods. (172.20.162.36:2181 looks like VPC address for example).
Another thing you can check if is if you have some NetworkPolicy preventing access:
kubectl get netpol
It's kind of odd that you get the expected response from Zookeeper:
curl zookeeper.pulse.svc.cluster.local:2181
curl: (52) Empty reply from server
So it could be possible that zookeeper.pulse.svc.cluster.local is resolving to an 'accessible' :2181. In any case, it looks like a firewall/network policy issue.
I am very new to using Microservices and having trouble running Kafka after I have started zookeeper.
Zookeeper starts fine but when I try to start my Kafka server it throws an error.
I have searched on google to try and solve my problem but its quite overwhelming, as I am not sure what all these different config files mean/do.
I have tried by enabling listeners=PLAINTEXT://:9092 in server settings but it doesn't work.
I have also tried to un and reinstalled Kafka and ZooKeeper but I still get the same error.
Antonias-MacBook-Pro:kafka antonia$ kafka-server-start /usr/local/etc/kafka/server.properties
[2020-02-25 10:26:55,944] INFO Registered kafka:type=kafka.Log4jController MBean (kafka.utils.Log4jControllerRegistration$)
[2020-02-25 10:26:56,514] INFO Registered signal handlers for TERM, INT, HUP (org.apache.kafka.common.utils.LoggingSignalHandler)
[2020-02-25 10:26:56,514] INFO starting (kafka.server.KafkaServer)
[2020-02-25 10:26:56,515] INFO Connecting to zookeeper on localhost:2181 (kafka.server.KafkaServer)
[2020-02-25 10:26:56,541] INFO [ZooKeeperClient Kafka server] Initializing a new session to localhost:2181. (kafka.zookeeper.ZooKeeperClient)
[2020-02-25 10:26:56,571] INFO Client environment:zookeeper.version=3.5.6-c11b7e26bc554b8523dc929761dd28808913f091, built on 10/08/2019 20:18 GMT (org.apache.zookeeper.ZooKeeper)
[2020-02-25 10:26:56,571] INFO Client environment:host.name=10.111.180.36 (org.apache.zookeeper.ZooKeeper)
[2020-02-25 10:26:56,571] INFO Client environment:java.version=1.8.0_242 (org.apache.zookeeper.ZooKeeper)
[2020-02-25 10:26:56,571] INFO Client environment:java.vendor=AdoptOpenJDK (org.apache.zookeeper.ZooKeeper)
[2020-02-25 10:26:56,571] INFO Client environment:java.home=/Library/Java/JavaVirtualMachines/adoptopenjdk-8.jdk/Contents/Home/jre (org.apache.zookeeper.ZooKeeper)
[2020-02-25 10:26:56,571] INFO Client environment:java.class.path=/usr/local/Cellar/kafka/2.4.0/libexec/bin/../libs/activation-1.1.1.jar:/usr/local/Cellar/kafka/2.4.0/libexec/bin/../libs/aopalliance-repackaged-2.5.0.jar:/usr/local/Cellar/kafka/2.4.0/libexec/bin/../libs/argparse4j-0.7.0.jar:/usr/local/Cellar/kafka/2.4.0/libexec/bin/../libs/audience-annotations-0.5.0.jar:/usr/local/Cellar/kafka/2.4.0/libexec/bin/../libs/commons-cli-1.4.jar:/usr/local/Cellar/kafka/2.4.0/libexec/bin/../libs/commons-lang3-3.8.1.jar:/usr/local/Cellar/kafka/2.4.0/libexec/bin/../libs/connect-api-2.4.0.jar:/usr/local/Cellar/kafka/2.4.0/libexec/bin/../libs/connect-basic-auth-extension-2.4.0.jar:/usr/local/Cellar/kafka/2.4.0/libexec/bin/../libs/connect-file-2.4.0.jar:/usr/local/Cellar/kafka/2.4.0/libexec/bin/../libs/connect-json-2.4.0.jar:/usr/local/Cellar/kafka/2.4.0/libexec/bin/../libs/connect-mirror-2.4.0.jar:/usr/local/Cellar/kafka/2.4.0/libexec/bin/../libs/connect-mirror-client-2.4.0.jar:/usr/local/Cellar/kafka/2.4.0/libexec/bin/../libs/connect-runtime-2.4.0.jar:/usr/local/Cellar/kafka/2.4.0/libexec/bin/../libs/connect-transforms-2.4.0.jar:/usr/local/Cellar/kafka/2.4.0/libexec/bin/../libs/guava-20.0.jar:/usr/local/Cellar/kafka/2.4.0/libexec/bin/../libs/hk2-api-2.5.0.jar:/usr/local/Cellar/kafka/2.4.0/libexec/bin/../libs/hk2-locator-2.5.0.jar:/usr/local/Cellar/kafka/2.4.0/libexec/bin/../libs/hk2-utils-2.5.0.jar:/usr/local/Cellar/kafka/2.4.0/libexec/bin/../libs/jackson-annotations-2.10.0.jar:/usr/local/Cellar/kafka/2.4.0/libexec/bin/../libs/jackson-core-2.10.0.jar:/usr/local/Cellar/kafka/2.4.0/libexec/bin/../libs/jackson-databind-2.10.0.jar:/usr/local/Cellar/kafka/2.4.0/libexec/bin/../libs/jackson-dataformat-csv-2.10.0.jar:/usr/local/Cellar/kafka/2.4.0/libexec/bin/../libs/jackson-datatype-jdk8-2.10.0.jar:/usr/local/Cellar/kafka/2.4.0/libexec/bin/../libs/jackson-jaxrs-base-2.10.0.jar:/usr/local/Cellar/kafka/2.4.0/libexec/bin/../libs/jackson-jaxrs-json-provider-2.10.0.jar:/usr/local/Cellar/kafka/2.4.0/libexec/bin/../libs/jackson-module-jaxb-annotations-2.10.0.jar:/usr/local/Cellar/kafka/2.4.0/libexec/bin/../libs/jackson-module-paranamer-2.10.0.jar:/usr/local/Cellar/kafka/2.4.0/libexec/bin/../libs/jackson-module-scala_2.12-2.10.0.jar:/usr/local/Cellar/kafka/2.4.0/libexec/bin/../libs/jakarta.activation-api-1.2.1.jar:/usr/local/Cellar/kafka/2.4.0/libexec/bin/../libs/jakarta.annotation-api-1.3.4.jar:/usr/local/Cellar/kafka/2.4.0/libexec/bin/../libs/jakarta.inject-2.5.0.jar:/usr/local/Cellar/kafka/2.4.0/libexec/bin/../libs/jakarta.ws.rs-api-2.1.5.jar:/usr/local/Cellar/kafka/2.4.0/libexec/bin/../libs/jakarta.xml.bind-api-2.3.2.jar:/usr/local/Cellar/kafka/2.4.0/libexec/bin/../libs/javassist-3.22.0-CR2.jar:/usr/local/Cellar/kafka/2.4.0/libexec/bin/../libs/javax.servlet-api-3.1.0.jar:/usr/local/Cellar/kafka/2.4.0/libexec/bin/../libs/javax.ws.rs-api-2.1.1.jar:/usr/local/Cellar/kafka/2.4.0/libexec/bin/../libs/jaxb-api-2.3.0.jar:/usr/local/Cellar/kafka/2.4.0/libexec/bin/../libs/jersey-client-2.28.jar:/usr/local/Cellar/kafka/2.4.0/libexec/bin/../libs/jersey-common-2.28.jar:/usr/local/Cellar/kafka/2.4.0/libexec/bin/../libs/jersey-container-servlet-2.28.jar:/usr/local/Cellar/kafka/2.4.0/libexec/bin/../libs/jersey-container-servlet-core-2.28.jar:/usr/local/Cellar/kafka/2.4.0/libexec/bin/../libs/jersey-hk2-2.28.jar:/usr/local/Cellar/kafka/2.4.0/libexec/bin/../libs/jersey-media-jaxb-2.28.jar:/usr/local/Cellar/kafka/2.4.0/libexec/bin/../libs/jersey-server-2.28.jar:/usr/local/Cellar/kafka/2.4.0/libexec/bin/../libs/jetty-client-9.4.20.v20190813.jar:/usr/local/Cellar/kafka/2.4.0/libexec/bin/../libs/jetty-continuation-9.4.20.v20190813.jar:/usr/local/Cellar/kafka/2.4.0/libexec/bin/../libs/jetty-http-9.4.20.v20190813.jar:/usr/local/Cellar/kafka/2.4.0/libexec/bin/../libs/jetty-io-9.4.20.v20190813.jar:/usr/local/Cellar/kafka/2.4.0/libexec/bin/../libs/jetty-security-9.4.20.v20190813.jar:/usr/local/Cellar/kafka/2.4.0/libexec/bin/../libs/jetty-server-9.4.20.v20190813.jar:/usr/local/Cellar/kafka/2.4.0/libexec/bin/../libs/jetty-servlet-9.4.20.v20190813.jar:/usr/local/Cellar/kafka/2.4.0/libexec/bin/../libs/jetty-servlets-9.4.20.v20190813.jar:/usr/local/Cellar/kafka/2.4.0/libexec/bin/../libs/jetty-util-9.4.20.v20190813.jar:/usr/local/Cellar/kafka/2.4.0/libexec/bin/../libs/jopt-simple-5.0.4.jar:/usr/local/Cellar/kafka/2.4.0/libexec/bin/../libs/kafka-clients-2.4.0.jar:/usr/local/Cellar/kafka/2.4.0/libexec/bin/../libs/kafka-log4j-appender-2.4.0.jar:/usr/local/Cellar/kafka/2.4.0/libexec/bin/../libs/kafka-streams-2.4.0.jar:/usr/local/Cellar/kafka/2.4.0/libexec/bin/../libs/kafka-streams-examples-2.4.0.jar:/usr/local/Cellar/kafka/2.4.0/libexec/bin/../libs/kafka-streams-scala_2.12-2.4.0.jar:/usr/local/Cellar/kafka/2.4.0/libexec/bin/../libs/kafka-streams-test-utils-2.4.0.jar:/usr/local/Cellar/kafka/2.4.0/libexec/bin/../libs/kafka-tools-2.4.0.jar:/usr/local/Cellar/kafka/2.4.0/libexec/bin/../libs/kafka_2.12-2.4.0-sources.jar:/usr/local/Cellar/kafka/2.4.0/libexec/bin/../libs/kafka_2.12-2.4.0.jar:/usr/local/Cellar/kafka/2.4.0/libexec/bin/../libs/log4j-1.2.17.jar:/usr/local/Cellar/kafka/2.4.0/libexec/bin/../libs/lz4-java-1.6.0.jar:/usr/local/Cellar/kafka/2.4.0/libexec/bin/../libs/maven-artifact-3.6.1.jar:/usr/local/Cellar/kafka/2.4.0/libexec/bin/../libs/metrics-core-2.2.0.jar:/usr/local/Cellar/kafka/2.4.0/libexec/bin/../libs/netty-buffer-4.1.42.Final.jar:/usr/local/Cellar/kafka/2.4.0/libexec/bin/../libs/netty-codec-4.1.42.Final.jar:/usr/local/Cellar/kafka/2.4.0/libexec/bin/../libs/netty-common-4.1.42.Final.jar:/usr/local/Cellar/kafka/2.4.0/libexec/bin/../libs/netty-handler-4.1.42.Final.jar:/usr/local/Cellar/kafka/2.4.0/libexec/bin/../libs/netty-resolver-4.1.42.Final.jar:/usr/local/Cellar/kafka/2.4.0/libexec/bin/../libs/netty-transport-4.1.42.Final.jar:/usr/local/Cellar/kafka/2.4.0/libexec/bin/../libs/netty-transport-native-epoll-4.1.42.Final.jar:/usr/local/Cellar/kafka/2.4.0/libexec/bin/../libs/netty-transport-native-unix-common-4.1.42.Final.jar:/usr/local/Cellar/kafka/2.4.0/libexec/bin/../libs/osgi-resource-locator-1.0.1.jar:/usr/local/Cellar/kafka/2.4.0/libexec/bin/../libs/paranamer-2.8.jar:/usr/local/Cellar/kafka/2.4.0/libexec/bin/../libs/plexus-utils-3.2.0.jar:/usr/local/Cellar/kafka/2.4.0/libexec/bin/../libs/reflections-0.9.11.jar:/usr/local/Cellar/kafka/2.4.0/libexec/bin/../libs/rocksdbjni-5.18.3.jar:/usr/local/Cellar/kafka/2.4.0/libexec/bin/../libs/scala-collection-compat_2.12-2.1.2.jar:/usr/local/Cellar/kafka/2.4.0/libexec/bin/../libs/scala-java8-compat_2.12-0.9.0.jar:/usr/local/Cellar/kafka/2.4.0/libexec/bin/../libs/scala-library-2.12.10.jar:/usr/local/Cellar/kafka/2.4.0/libexec/bin/../libs/scala-logging_2.12-3.9.2.jar:/usr/local/Cellar/kafka/2.4.0/libexec/bin/../libs/scala-reflect-2.12.10.jar:/usr/local/Cellar/kafka/2.4.0/libexec/bin/../libs/slf4j-api-1.7.28.jar:/usr/local/Cellar/kafka/2.4.0/libexec/bin/../libs/slf4j-log4j12-1.7.28.jar:/usr/local/Cellar/kafka/2.4.0/libexec/bin/../libs/snappy-java-1.1.7.3.jar:/usr/local/Cellar/kafka/2.4.0/libexec/bin/../libs/validation-api-2.0.1.Final.jar:/usr/local/Cellar/kafka/2.4.0/libexec/bin/../libs/zookeeper-3.5.6.jar:/usr/local/Cellar/kafka/2.4.0/libexec/bin/../libs/zookeeper-jute-3.5.6.jar:/usr/local/Cellar/kafka/2.4.0/libexec/bin/../libs/zstd-jni-1.4.3-1.jar (org.apache.zookeeper.ZooKeeper)
[2020-02-25 10:26:56,572] INFO Client environment:java.library.path=/Users/antonia/Library/Java/Extensions:/Library/Java/Extensions:/Network/Library/Java/Extensions:/System/Library/Java/Extensions:/usr/lib/java:. (org.apache.zookeeper.ZooKeeper)
[2020-02-25 10:26:56,572] INFO Client environment:java.io.tmpdir=/var/folders/13/gvhn5dqj4179ynmpm6pkp9kh0000gn/T/ (org.apache.zookeeper.ZooKeeper)
[2020-02-25 10:26:56,572] INFO Client environment:java.compiler=<NA> (org.apache.zookeeper.ZooKeeper)
[2020-02-25 10:26:56,572] INFO Client environment:os.name=Mac OS X (org.apache.zookeeper.ZooKeeper)
[2020-02-25 10:26:56,572] INFO Client environment:os.arch=x86_64 (org.apache.zookeeper.ZooKeeper)
[2020-02-25 10:26:56,573] INFO Client environment:os.version=10.14.5 (org.apache.zookeeper.ZooKeeper)
[2020-02-25 10:26:56,573] INFO Client environment:user.name=antonia (org.apache.zookeeper.ZooKeeper)
[2020-02-25 10:26:56,573] INFO Client environment:user.home=/Users/antonia (org.apache.zookeeper.ZooKeeper)
[2020-02-25 10:26:56,573] INFO Client environment:user.dir=/usr/local/etc/kafka (org.apache.zookeeper.ZooKeeper)
[2020-02-25 10:26:56,573] INFO Client environment:os.memory.free=977MB (org.apache.zookeeper.ZooKeeper)
[2020-02-25 10:26:56,573] INFO Client environment:os.memory.max=1024MB (org.apache.zookeeper.ZooKeeper)
[2020-02-25 10:26:56,573] INFO Client environment:os.memory.total=1024MB (org.apache.zookeeper.ZooKeeper)
[2020-02-25 10:26:56,575] INFO Initiating client connection, connectString=localhost:2181 sessionTimeout=6000 watcher=kafka.zookeeper.ZooKeeperClient$ZooKeeperClientWatcher$#28eaa59a (org.apache.zookeeper.ZooKeeper)
[2020-02-25 10:26:56,582] INFO Setting -D jdk.tls.rejectClientInitiatedRenegotiation=true to disable client-initiated TLS renegotiation (org.apache.zookeeper.common.X509Util)
[2020-02-25 10:26:56,592] INFO jute.maxbuffer value is 4194304 Bytes (org.apache.zookeeper.ClientCnxnSocket)
[2020-02-25 10:26:56,599] INFO zookeeper.request.timeout value is 0. feature enabled= (org.apache.zookeeper.ClientCnxn)
[2020-02-25 10:26:56,601] INFO [ZooKeeperClient Kafka server] Waiting until connected. (kafka.zookeeper.ZooKeeperClient)
[2020-02-25 10:26:56,605] INFO Opening socket connection to server localhost/0:0:0:0:0:0:0:1:2181. Will not attempt to authenticate using SASL (unknown error) (org.apache.zookeeper.ClientCnxn)
[2020-02-25 10:26:56,625] INFO Socket connection established, initiating session, client: /0:0:0:0:0:0:0:1:63152, server: localhost/0:0:0:0:0:0:0:1:2181 (org.apache.zookeeper.ClientCnxn)
[2020-02-25 10:26:56,641] INFO Session establishment complete on server localhost/0:0:0:0:0:0:0:1:2181, sessionid = 0x10004969600000a, negotiated timeout = 6000 (org.apache.zookeeper.ClientCnxn)
[2020-02-25 10:26:56,645] INFO [ZooKeeperClient Kafka server] Connected. (kafka.zookeeper.ZooKeeperClient)
[2020-02-25 10:26:56,979] INFO Cluster ID = S4SZ31nVRTCQ4uwRJ9_7mg (kafka.server.KafkaServer)
[2020-02-25 10:26:56,992] ERROR Fatal error during KafkaServer startup. Prepare to shutdown (kafka.server.KafkaServer)
kafka.common.InconsistentClusterIdException: The Cluster ID S4SZ31nVRTCQ4uwRJ9_7mg doesn't match stored clusterId Some(Y_mQi4q4TSuhlWdx4DHiaQ) in meta.properties. The broker is trying to join the wrong cluster. Configured zookeeper.connect may be wrong.
at kafka.server.KafkaServer.startup(KafkaServer.scala:220)
at kafka.server.KafkaServerStartable.startup(KafkaServerStartable.scala:44)
at kafka.Kafka$.main(Kafka.scala:84)
at kafka.Kafka.main(Kafka.scala)
[2020-02-25 10:26:56,995] INFO shutting down (kafka.server.KafkaServer)
[2020-02-25 10:26:56,997] INFO [ZooKeeperClient Kafka server] Closing. (kafka.zookeeper.ZooKeeperClient)
[2020-02-25 10:26:57,109] INFO Session: 0x10004969600000a closed (org.apache.zookeeper.ZooKeeper)
[2020-02-25 10:26:57,109] INFO EventThread shut down for session: 0x10004969600000a (org.apache.zookeeper.ClientCnxn)
[2020-02-25 10:26:57,111] INFO [ZooKeeperClient Kafka server] Closed. (kafka.zookeeper.ZooKeeperClient)
[2020-02-25 10:26:57,118] INFO shut down completed (kafka.server.KafkaServer)
[2020-02-25 10:26:57,119] ERROR Exiting Kafka. (kafka.server.KafkaServerStartable)
[2020-02-25 10:26:57,120] INFO shutting down (kafka.server.KafkaServer)
The cause of the problem is shown in this message:
kafka.common.InconsistentClusterIdException:
The Cluster ID S4SZ31nVRTCQ4uwRJ9_7mg
doesn't match stored clusterId Some(Y_mQi4q4TSuhlWdx4DHiaQ)
in meta.properties.
The broker is trying to join the wrong cluster.
Configured zookeeper.connect may be wrong.
The above problem occurs when a new instance of Kafka is being started up on data storage created by another kafka server. Kafka stores its messages in 'log' files.
How to fix the problem?
The problem can be fixed in these steps:
Shutdown both Kafka and Zookeeper
If required, take backup of the existing logs of Kafka and Zookeeper
Delete the log directories of both Kafka and Zookeeper
Restart Zookeeper and Kafka
Kafka shuts down without error after a certain time interval.
kafka server log:
[2016-01-11 12:15:06,435] INFO Verifying properties (kafka.utils.VerifiableProperties)
[2016-01-11 12:15:06,489] INFO Property broker.id is overridden to 0 (kafka.utils.VerifiableProperties)
[2016-01-11 12:15:06,489] INFO Property host.name is overridden to 172.16.1.22 (kafka.utils.VerifiableProperties)
[2016-01-11 12:15:06,490] INFO Property log.cleaner.enable is overridden to false (kafka.utils.VerifiableProperties)
[2016-01-11 12:15:06,490] INFO Property log.dirs is overridden to /data/www/wifiin/logs/kafka (kafka.utils.VerifiableProperties)
[2016-01-11 12:15:06,490] INFO Property log.retention.check.interval.ms is overridden to 300000 (kafka.utils.VerifiableProperties)
[2016-01-11 12:15:06,491] INFO Property log.retention.hours is overridden to 168 (kafka.utils.VerifiableProperties)
[2016-01-11 12:15:06,491] INFO Property log.segment.bytes is overridden to 1073741824 (kafka.utils.VerifiableProperties)
[2016-01-11 12:15:06,491] INFO Property num.io.threads is overridden to 8 (kafka.utils.VerifiableProperties)
[2016-01-11 12:15:06,491] INFO Property num.network.threads is overridden to 3 (kafka.utils.VerifiableProperties)
[2016-01-11 12:15:06,492] INFO Property num.partitions is overridden to 1 (kafka.utils.VerifiableProperties)
[2016-01-11 12:15:06,492] INFO Property num.recovery.threads.per.data.dir is overridden to 1 (kafka.utils.VerifiableProperties)
[2016-01-11 12:15:06,492] INFO Property port is overridden to 9092 (kafka.utils.VerifiableProperties)
[2016-01-11 12:15:06,493] INFO Property socket.receive.buffer.bytes is overridden to 102400 (kafka.utils.VerifiableProperties)
[2016-01-11 12:15:06,493] INFO Property socket.request.max.bytes is overridden to 104857600 (kafka.utils.VerifiableProperties)
[2016-01-11 12:15:06,493] INFO Property socket.send.buffer.bytes is overridden to 102400 (kafka.utils.VerifiableProperties)
[2016-01-11 12:15:06,493] INFO Property zookeeper.connect is overridden to 127.0.0.1:2181 (kafka.utils.VerifiableProperties)
[2016-01-11 12:15:06,494] INFO Property zookeeper.connection.timeout.ms is overridden to 6000 (kafka.utils.VerifiableProperties)
[2016-01-11 12:15:06,550] INFO [Kafka Server 0], starting (kafka.server.KafkaServer)
[2016-01-11 12:15:06,553] INFO [Kafka Server 0], Connecting to zookeeper on 127.0.0.1:2181 (kafka.server.KafkaServer)
[2016-01-11 12:15:06,568] INFO Starting ZkClient event thread. (org.I0Itec.zkclient.ZkEventThread)
[2016-01-11 12:15:06,579] INFO Client environment:zookeeper.version=3.4.6-1569965, built on 02/20/2014 09:09 GMT (org.apache.zookeeper.ZooKeeper)
[2016-01-11 12:15:06,579] INFO Client environment:host.name=wifiin-analysis-22 (org.apache.zookeeper.ZooKeeper)
[2016-01-11 12:15:06,579] INFO Client environment:java.version=1.7.0_40 (org.apache.zookeeper.ZooKeeper)
[2016-01-11 12:15:06,579] INFO Client environment:java.vendor=Oracle Corporation (org.apache.zookeeper.ZooKeeper)
[2016-01-11 12:15:06,579] INFO Client environment:java.home=/usr/local/java/jdk1.7.0_40/jre
(org.apache.zookeeper.ZooKeeper)
...
[2016-01-11 12:15:06,580] INFO Client environment:java.io.tmpdir=/tmp (org.apache.zookeeper.ZooKeeper)
[2016-01-11 12:15:06,580] INFO Client environment:java.compiler=<NA(org.apache.zookeeper.ZooKeeper)
[2016-01-11 12:15:06,580] INFO Client environment:os.name=Linux (org.apache.zookeeper.ZooKeeper)
[2016-01-11 12:15:06,580] INFO Client environment:os.arch=amd64 (org.apache.zookeeper.ZooKeeper)
[2016-01-11 12:15:06,580] INFO Client environment:os.version=2.6.32-573.3.1.el6.x86_64 (org.apache.zookeeper.ZooKeeper)
[2016-01-11 12:15:06,580] INFO Client environment:user.name=root (org.apache.zookeeper.ZooKeeper)
[2016-01-11 12:15:06,580] INFO Client environment:user.home=/root (org.apache.zookeeper.ZooKeeper)
[2016-01-11 12:15:06,580] INFO Client environment:user.dir=/usr/local/kafka (org.apache.zookeeper.ZooKeeper)
...[2016-01-11 12:15:17,515] INFO [Kafka Server 0], started (kafka.server.KafkaServer)
[2016-01-11 12:15:17,795] INFO [ReplicaFetcherManager on broker 0] Removed fetcher for partitions [sdkLog,0] (kafka.server.ReplicaFetcherManager)
[2016-01-11 12:15:17,856] INFO [ReplicaFetcherManager on broker 0] Removed fetcher for partitions [sdkLog,0] (kafka.server.ReplicaFetcherManager)
[2016-01-11 14:47:41,816] INFO [Kafka Server 0], shutting down (kafka.server.KafkaServer)
[2016-01-11 14:47:41,818] INFO [Kafka Server 0], Starting controlled shutdown (kafka.server.KafkaServer)
[2016-01-11 14:47:41,981] INFO [Kafka Server 0], Controlled shutdown succeeded (kafka.server.KafkaServer)
[2016-01-11 14:47:41,983] INFO Closing socket connection to /172.16.1.22. (kafka.network.Processor)
[2016-01-11 14:47:41,984] INFO [Socket Server on Broker 0], Shutting down (kafka.network.SocketServer)
[2016-01-11 14:47:41,989] INFO [Socket Server on Broker 0], Shutdown completed (kafka.network.SocketServer)
[2016-01-11 14:47:41,991] INFO [Kafka Request Handler on Broker 0], shutting down (kafka.server.KafkaRequestHandlerPool)
[2016-01-11 14:47:41,994] INFO [Kafka Request Handler on Broker 0], shut down completely (kafka.server.KafkaRequestHandlerPool)
[2016-01-11 14:47:42,267] INFO [Replica Manager on Broker 0]: Shut down (kafka.server.ReplicaManager)
[2016-01-11 14:47:42,267] INFO [ReplicaFetcherManager on broker 0] shutting down (kafka.server.ReplicaFetcherManager)
[2016-01-11 14:47:42,269] INFO [ReplicaFetcherManager on broker 0] shutdown completed (kafka.server.ReplicaFetcherManager)
[2016-01-11 14:47:42,314] INFO [Replica Manager on Broker 0]: Shut down
completely (kafka.server.ReplicaManager)
[2016-01-11 14:47:42,315] INFO Shutting down. (kafka.log.LogManager)
[2016-01-11 14:47:42,376] INFO Shutdown complete. (kafka.log.LogManager)
[2016-01-11 14:47:42,384] INFO Terminate ZkClient event thread.
(org.I0Itec.zkclient.ZkEventThread)
[2016-01-11 14:47:42,396] INFO Session: 0x15210b1e3051357 closed (org.apache.zookeeper.ZooKeeper)
[2016-01-11 14:47:42,396] INFO EventThread shut down (org.apache.zookeeper.ClientCnxn)
[2016-01-11 14:47:42,396] INFO [Kafka Server 0], shut down completed (kafka.server.KafkaServer)
At first, I believed OutOfMemory led to the shutdown, so I edited kafka's oom_score wihtout luck.What should I do to keep the Kafka broker from shutting down?
./kafka-server-start.sh -daemon ../config/server.properties
This may help you！