Kafka service start fail - apache-kafka

I'm trying to install kafka on my Ubuntu 16.04 machine.
I followed instruction of below link
https://www.digitalocean.com/community/tutorials/how-to-install-apache-kafka-on-ubuntu-18-04
As I end up create /etc/systemd/system/kafka.service
and try to start service with 'sudo systemctl start kafka'
It doesn't print any message, I could know that starting service is failed by checking output of
sudo journalctl -u
It says
sudo journalctl -u kafka
<pre><code>
kafka.service: Main process exited, code=exited, status=1/FAIL
kafka.service: Control process exited, code=exited status=1
kafka.service: Unit entered failed state.
kafka.service: Failed with result 'exit-code'.
</code></pre>
cat kafka.log
<pre><code>
java
[2020-02-08 00:01:08,241] INFO Registered kafka:type=kafka.Log4jController MBean (kafka.utils.Log4jControllerRegistration$)
[2020-02-08 00:01:08,449] INFO starting (kafka.server.KafkaServer)
[2020-02-08 00:01:08,449] INFO Connecting to zookeeper on localhost:2181 (kafka.server.KafkaServer)
[2020-02-08 00:01:08,459] INFO [ZooKeeperClient] Initializing a new session to localhost:2181. (kafka.zookeeper.ZooKeeperClient)
[2020-02-08 00:01:08,462] INFO Client environment:zookeeper.version=3.4.13-2d71af4dbe22557fda74f9a9b4309b15a7487f03, built on 06/29/2018 00:39 GMT (org.apache.zookeeper.ZooKeeper)
[2020-02-08 00:01:08,462] INFO Client environment:host.name=4whomtbts (org.apache.zookeeper.ZooKeeper)
[2020-02-08 00:01:08,462] INFO Client environment:java.version=1.8.0_242 (org.apache.zookeeper.ZooKeeper)
[2020-02-08 00:01:08,462] INFO Client environment:java.vendor=Private Build (org.apache.zookeeper.ZooKeeper)
[2020-02-08 00:01:08,462] INFO Client environment:java.home=/usr/lib/jvm/java-8-openjdk-amd64/jre (org.apache.zookeeper.ZooKeeper)
... skipped ...
[2020-02-08 00:01:15,208] INFO [ZooKeeperClient] Closed. (kafka.zookeeper.ZooKeeperClient)
[2020-02-08 00:01:15,211]<b> ERROR Fatal error during KafkaServer startup. Prepare to shutdown (kafka.server.KafkaServer)
kafka.zookeeper.ZooKeeperClientTimeoutException: Timed out waiting for connection while in state: CONNECTING
at kafka.zookeeper.ZooKeeperClient$$anonfun$kafka$zookeeper$ZooKeeperClient$$waitUntilConnected$1.apply$mcV$sp(ZooKeeperClient.scala:268)
at kafka.zookeeper.ZooKeeperClient$$anonfun$kafka$zookeeper$ZooKeeperClient$$waitUntilConnected$1.apply(ZooKeeperClient.scala:264)
at kafka.zookeeper.ZooKeeperClient$$anonfun$kafka$zookeeper$ZooKeeperClient$$waitUntilConnected$1.apply(ZooKeeperClient.scala:264)
at kafka.utils.CoreUtils$.inLock(CoreUtils.scala:251)
at kafka.zookeeper.ZooKeeperClient.kafka$zookeeper$ZooKeeperClient$$waitUntilConnected(ZooKeeperClient.scala:264)
at kafka.zookeeper.ZooKeeperClient.<init>(ZooKeeperClient.scala:97)
at kafka.zk.KafkaZkClient$.apply(KafkaZkClient.scala:1693)
at kafka.server.KafkaServer.kafka$server$KafkaServer$$createZkClient$1(KafkaServer.scala:348)
at kafka.server.KafkaServer.initZkClient(KafkaServer.scala:372)
at kafka.server.KafkaServer.startup(KafkaServer.scala:202)
at kafka.server.KafkaServerStartable.startup(KafkaServerStartable.scala:38)
at kafka.Kafka$.main(Kafka.scala:75)
at kafka.Kafka.main(Kafka.scala) </b>
[2020-02-08 00:01:15,214] INFO shutting down (kafka.server.KafkaServer)
[2020-02-08 00:01:15,218] INFO shut down completed (kafka.server.KafkaServer)
[2020-02-08 00:01:15,218] <b>ERROR Exiting Kafka. </b>(kafka.server.KafkaServerStartable)
[2020-02-08 00:01:15,220] INFO shutting down (kafka.server.KafkaServer)
</code></pre>
+more Zookeeper information
/var/log/zookeeper contains no file.
zookeeper.service
[Unit]
Requires=network.target remote-fs.target
After=network.target remote-fs.target
[Service]
Type=simple
User=kafka
ExecStart=/home/kafka/kafka/bin/zookeeper-server-start.sh
/home/kafka/kafka/config/zookeeper.properties
ExecStop=/home/kafka/kafka/bin/zookeeper-server-stop.sh
Restart=on-abnormal
[Install]
WantedBy=multi-user.target
service --status-all
... skipped ...
[ - ] umountnfs.sh
[ - ] umountroot
[ + ] unattended-upgrades
[ + ] urandom
[ - ] uuidd
[ + ] whoopsie
[ - ] x11-common
[ + ] zookeeper
sudo journalctl -u zookeeper
systemd[1]: Started zookeeper.service.
systemd[1]: Started zookeeper.service.
I couldn't find out any clue to solve this problem by googling :(
My environment information is..
Machine : LG gram 2019 15.4
OS : Ubuntu 16.04, kernel version 5
java : openJDK 1.8.0
Thank you in advance!!!

Your zookeeper server may be gone to idle state. You can just to go to the terminal of zookeeper and hit enter. After that you can try again.

Related

Kafka Authentication with SASL_PLAINTEXT fails

Here's the log:
kafka 16:54:47.56
kafka 16:54:47.57 Welcome to the Bitnami kafka container
kafka 16:54:47.57 Subscribe to project updates by watching https://github.com/bitnami/bitnami-docker-kafka
kafka 16:54:47.57 Submit issues and feature requests at https://github.com/bitnami/bitnami-docker-kafka/issues
kafka 16:54:47.57
kafka 16:54:47.57 INFO ==> ** Starting Kafka setup **
kafka 16:54:47.62 DEBUG ==> Validating settings in KAFKA_* env vars...
kafka 16:54:47.64 WARN ==> You set the environment variable ALLOW_PLAINTEXT_LISTENER=yes. For safety reasons, do not use this flag in a production environment.
kafka 16:54:47.64 INFO ==> Initializing Kafka...
kafka 16:54:47.65 INFO ==> No injected configuration files found, creating default config files
kafka 16:54:47.89 INFO ==> Configuring Kafka for inter-broker communications with SASL_PLAINTEXT authentication.
kafka 16:54:47.89 INFO ==> Configuring Kafka for client communications with SASL_PLAINTEXT authentication.
kafka 16:54:47.91 INFO ==> Generating JAAS authentication file
kafka 16:54:47.93 INFO ==> ** Kafka setup finished! **
kafka 16:54:47.95 INFO ==> ** Starting Kafka **
[2022-05-29 16:54:49,343] INFO Registered kafka:type=kafka.Log4jController MBean (kafka.utils.Log4jControllerRegistration$)
[2022-05-29 16:54:49,988] INFO Setting -D jdk.tls.rejectClientInitiatedRenegotiation=true to disable client-initiated TLS renegotiation (org.apache.zookeeper.common.X509Util)
[2022-05-29 16:54:50,157] INFO Registered signal handlers for TERM, INT, HUP (org.apache.kafka.common.utils.LoggingSignalHandler)
[2022-05-29 16:54:50,163] INFO starting (kafka.server.KafkaServer)
[2022-05-29 16:54:50,164] INFO Connecting to zookeeper on sharif-zookeeper (kafka.server.KafkaServer)
[2022-05-29 16:54:50,188] INFO [ZooKeeperClient Kafka server] Initializing a new session to sharif-zookeeper. (kafka.zookeeper.ZooKeeperClient)
[2022-05-29 16:54:50,194] INFO Client environment:zookeeper.version=3.5.9-83df9301aa5c2a5d284a9940177808c01bc35cef, built on 01/06/2021 20:03 GMT (org.apache.zookeeper.ZooKeeper)
[2022-05-29 16:54:50,196] INFO Client environment:host.name=sharif-kafka-0.sharif-kafka-headless.default.svc.cluster.local (org.apache.zookeeper.ZooKeeper)
[2022-05-29 16:54:50,196] INFO Client environment:java.version=11.0.12 (org.apache.zookeeper.ZooKeeper)
[2022-05-29 16:54:50,196] INFO Client environment:java.vendor=BellSoft (org.apache.zookeeper.ZooKeeper)
[2022-05-29 16:54:50,196] INFO Client environment:java.home=/opt/bitnami/java (org.apache.zookeeper.ZooKeeper)
[2022-05-29 16:54:50,197] INFO Client environment:java.class.path=/opt/bitnami/kafka/bin/../libs/activation-1.1.1.jar:/opt/bitnami/kafka/bin/../libs/aopalliance-repackaged-2.6.1.jar:/opt/bitnami/kafka/bin/../libs/argparse4j-0.7.0.jar:/opt/bitnami/kafka/bin/../libs/audience-annotations-0.5.0.jar:/opt/bitnami/kafka/bin/../libs/commons-cli-1.4.jar:/opt/bitnami/kafka/bin/../libs/commons-lang3-3.8.1.jar:/opt/bitnami/kafka/bin/../libs/connect-api-2.8.0.jar:/opt/bitnami/kafka/bin/../libs/connect-basic-auth-extension-2.8.0.jar:/opt/bitnami/kafka/bin/../libs/connect-file-2.8.0.jar:/opt/bitnami/kafka/bin/../libs/connect-json-2.8.0.jar:/opt/bitnami/kafka/bin/../libs/connect-mirror-2.8.0.jar:/opt/bitnami/kafka/bin/../libs/connect-mirror-client-2.8.0.jar:/opt/bitnami/kafka/bin/../libs/connect-runtime-2.8.0.jar:/opt/bitnami/kafka/bin/../libs/connect-transforms-2.8.0.jar:/opt/bitnami/kafka/bin/../libs/hk2-api-2.6.1.jar:/opt/bitnami/kafka/bin/../libs/hk2-locator-2.6.1.jar:/opt/bitnami/kafka/bin/../libs/hk2-utils-2.6.1.jar:/opt/bitnami/kafka/bin/../libs/jackson-annotations-2.10.5.jar:/opt/bitnami/kafka/bin/../libs/jackson-core-2.10.5.jar:/opt/bitnami/kafka/bin/../libs/jackson-databind-2.10.5.1.jar:/opt/bitnami/kafka/bin/../libs/jackson-dataformat-csv-2.10.5.jar:/opt/bitnami/kafka/bin/../libs/jackson-datatype-jdk8-2.10.5.jar:/opt/bitnami/kafka/bin/../libs/jackson-jaxrs-base-2.10.5.jar:/opt/bitnami/kafka/bin/../libs/jackson-jaxrs-json-provider-2.10.5.jar:/opt/bitnami/kafka/bin/../libs/jackson-module-jaxb-annotations-2.10.5.jar:/opt/bitnami/kafka/bin/../libs/jackson-module-paranamer-2.10.5.jar:/opt/bitnami/kafka/bin/../libs/jackson-module-scala_2.12-2.10.5.jar:/opt/bitnami/kafka/bin/../libs/jakarta.activation-api-1.2.1.jar:/opt/bitnami/kafka/bin/../libs/jakarta.annotation-api-1.3.5.jar:/opt/bitnami/kafka/bin/../libs/jakarta.inject-2.6.1.jar:/opt/bitnami/kafka/bin/../libs/jakarta.validation-api-2.0.2.jar:/opt/bitnami/kafka/bin/../libs/jakarta.ws.rs-api-2.1.6.jar:/opt/bitnami/kafka/bin/../libs/jakarta.xml.bind-api-2.3.2.jar:/opt/bitnami/kafka/bin/../libs/javassist-3.27.0-GA.jar:/opt/bitnami/kafka/bin/../libs/javax.servlet-api-3.1.0.jar:/opt/bitnami/kafka/bin/../libs/javax.ws.rs-api-2.1.1.jar:/opt/bitnami/kafka/bin/../libs/jaxb-api-2.3.0.jar:/opt/bitnami/kafka/bin/../libs/jersey-client-2.31.jar:/opt/bitnami/kafka/bin/../libs/jersey-common-2.31.jar:/opt/bitnami/kafka/bin/../libs/jersey-container-servlet-2.31.jar:/opt/bitnami/kafka/bin/../libs/jersey-container-servlet-core-2.31.jar:/opt/bitnami/kafka/bin/../libs/jersey-hk2-2.31.jar:/opt/bitnami/kafka/bin/../libs/jersey-media-jaxb-2.31.jar:/opt/bitnami/kafka/bin/../libs/jersey-server-2.31.jar:/opt/bitnami/kafka/bin/../libs/jetty-client-9.4.39.v20210325.jar:/opt/bitnami/kafka/bin/../libs/jetty-continuation-9.4.39.v20210325.jar:/opt/bitnami/kafka/bin/../libs/jetty-http-9.4.39.v20210325.jar:/opt/bitnami/kafka/bin/../libs/jetty-io-9.4.39.v20210325.jar:/opt/bitnami/kafka/bin/../libs/jetty-security-9.4.39.v20210325.jar:/opt/bitnami/kafka/bin/../libs/jetty-server-9.4.39.v20210325.jar:/opt/bitnami/kafka/bin/../libs/jetty-servlet-9.4.39.v20210325.jar:/opt/bitnami/kafka/bin/../libs/jetty-servlets-9.4.39.v20210325.jar:/opt/bitnami/kafka/bin/../libs/jetty-util-9.4.39.v20210325.jar:/opt/bitnami/kafka/bin/../libs/jetty-util-ajax-9.4.39.v20210325.jar:/opt/bitnami/kafka/bin/../libs/jline-3.12.1.jar:/opt/bitnami/kafka/bin/../libs/jopt-simple-5.0.4.jar:/opt/bitnami/kafka/bin/../libs/kafka-clients-2.8.0.jar:/opt/bitnami/kafka/bin/../libs/kafka-log4j-appender-2.8.0.jar:/opt/bitnami/kafka/bin/../libs/kafka-metadata-2.8.0.jar:/opt/bitnami/kafka/bin/../libs/kafka-raft-2.8.0.jar:/opt/bitnami/kafka/bin/../libs/kafka-shell-2.8.0.jar:/opt/bitnami/kafka/bin/../libs/kafka-streams-2.8.0.jar:/opt/bitnami/kafka/bin/../libs/kafka-streams-examples-2.8.0.jar:/opt/bitnami/kafka/bin/../libs/kafka-streams-scala_2.12-2.8.0.jar:/opt/bitnami/kafka/bin/../libs/kafka-streams-test-utils-2.8.0.jar:/opt/bitnami/kafka/bin/../libs/kafka-tools-2.8.0.jar:/opt/bitnami/kafka/bin/../libs/kafka_2.12-2.8.0-sources.jar:/opt/bitnami/kafka/bin/../libs/kafka_2.12-2.8.0.jar:/opt/bitnami/kafka/bin/../libs/log4j-1.2.17.jar:/opt/bitnami/kafka/bin/../libs/lz4-java-1.7.1.jar:/opt/bitnami/kafka/bin/../libs/maven-artifact-3.6.3.jar:/opt/bitnami/kafka/bin/../libs/metrics-core-2.2.0.jar:/opt/bitnami/kafka/bin/../libs/netty-buffer-4.1.62.Final.jar:/opt/bitnami/kafka/bin/../libs/netty-codec-4.1.62.Final.jar:/opt/bitnami/kafka/bin/../libs/netty-common-4.1.62.Final.jar:/opt/bitnami/kafka/bin/../libs/netty-handler-4.1.62.Final.jar:/opt/bitnami/kafka/bin/../libs/netty-resolver-4.1.62.Final.jar:/opt/bitnami/kafka/bin/../libs/netty-transport-4.1.62.Final.jar:/opt/bitnami/kafka/bin/../libs/netty-transport-native-epoll-4.1.62.Final.jar:/opt/bitnami/kafka/bin/../libs/netty-transport-native-unix-common-4.1.62.Final.jar:/opt/bitnami/kafka/bin/../libs/osgi-resource-locator-1.0.3.jar:/opt/bitnami/kafka/bin/../libs/paranamer-2.8.jar:/opt/bitnami/kafka/bin/../libs/plexus-utils-3.2.1.jar:/opt/bitnami/kafka/bin/../libs/reflections-0.9.12.jar:/opt/bitnami/kafka/bin/../libs/rocksdbjni-5.18.4.jar:/opt/bitnami/kafka/bin/../libs/scala-collection-compat_2.12-2.3.0.jar:/opt/bitnami/kafka/bin/../libs/scala-java8-compat_2.12-0.9.1.jar:/opt/bitnami/kafka/bin/../libs/scala-library-2.12.13.jar:/opt/bitnami/kafka/bin/../libs/scala-logging_2.12-3.9.2.jar:/opt/bitnami/kafka/bin/../libs/scala-reflect-2.12.13.jar:/opt/bitnami/kafka/bin/../libs/slf4j-api-1.7.30.jar:/opt/bitnami/kafka/bin/../libs/slf4j-log4j12-1.7.30.jar:/opt/bitnami/kafka/bin/../libs/snappy-java-1.1.8.1.jar:/opt/bitnami/kafka/bin/../libs/zookeeper-3.5.9.jar:/opt/bitnami/kafka/bin/../libs/zookeeper-jute-3.5.9.jar:/opt/bitnami/kafka/bin/../libs/zstd-jni-1.4.9-1.jar (org.apache.zookeeper.ZooKeeper)
[2022-05-29 16:54:50,197] INFO Client environment:java.library.path=/usr/java/packages/lib:/usr/lib64:/lib64:/lib:/usr/lib (org.apache.zookeeper.ZooKeeper)
[2022-05-29 16:54:50,198] INFO Client environment:java.io.tmpdir=/tmp (org.apache.zookeeper.ZooKeeper)
[2022-05-29 16:54:50,198] INFO Client environment:java.compiler=<NA> (org.apache.zookeeper.ZooKeeper)
[2022-05-29 16:54:50,198] INFO Client environment:os.name=Linux (org.apache.zookeeper.ZooKeeper)
[2022-05-29 16:54:50,198] INFO Client environment:os.arch=amd64 (org.apache.zookeeper.ZooKeeper)
[2022-05-29 16:54:50,198] INFO Client environment:os.version=5.4.190-107.353.amzn2.x86_64 (org.apache.zookeeper.ZooKeeper)
[2022-05-29 16:54:50,199] INFO Client environment:user.name=? (org.apache.zookeeper.ZooKeeper)
[2022-05-29 16:54:50,199] INFO Client environment:user.home=? (org.apache.zookeeper.ZooKeeper)
[2022-05-29 16:54:50,199] INFO Client environment:user.dir=/ (org.apache.zookeeper.ZooKeeper)
[2022-05-29 16:54:50,199] INFO Client environment:os.memory.free=1011MB (org.apache.zookeeper.ZooKeeper)
[2022-05-29 16:54:50,200] INFO Client environment:os.memory.max=1024MB (org.apache.zookeeper.ZooKeeper)
[2022-05-29 16:54:50,200] INFO Client environment:os.memory.total=1024MB (org.apache.zookeeper.ZooKeeper)
[2022-05-29 16:54:50,203] INFO Initiating client connection, connectString=sharif-zookeeper sessionTimeout=18000 watcher=kafka.zookeeper.ZooKeeperClient$ZooKeeperClientWatcher$#447a020 (org.apache.zookeeper.ZooKeeper)
[2022-05-29 16:54:50,210] INFO jute.maxbuffer value is 4194304 Bytes (org.apache.zookeeper.ClientCnxnSocket)
[2022-05-29 16:54:50,216] INFO zookeeper.request.timeout value is 0. feature enabled= (org.apache.zookeeper.ClientCnxn)
[2022-05-29 16:54:50,218] INFO [ZooKeeperClient Kafka server] Waiting until connected. (kafka.zookeeper.ZooKeeperClient)
[2022-05-29 16:54:50,367] INFO Client successfully logged in. (org.apache.zookeeper.Login)
[2022-05-29 16:54:50,371] INFO Client will use DIGEST-MD5 as SASL mechanism. (org.apache.zookeeper.client.ZooKeeperSaslClient)
[2022-05-29 16:54:50,381] INFO Opening socket connection to server sharif-zookeeper/10.100.190.137:2181. Will attempt to SASL-authenticate using Login Context section 'Client' (org.apache.zookeeper.ClientCnxn)
[2022-05-29 16:54:50,389] INFO Socket connection established, initiating session, client: /192.168.34.166:57652, server: sharif-zookeeper/10.100.190.137:2181 (org.apache.zookeeper.ClientCnxn)
[2022-05-29 16:54:50,398] INFO Session establishment complete on server sharif-zookeeper/10.100.190.137:2181, sessionid = 0x100003ea9fd0008, negotiated timeout = 18000 (org.apache.zookeeper.ClientCnxn)
[2022-05-29 16:54:50,406] INFO [ZooKeeperClient Kafka server] Connected. (kafka.zookeeper.ZooKeeperClient)
[2022-05-29 16:54:50,420] ERROR SASL authentication failed using login context 'Client' with exception: {} (org.apache.zookeeper.client.ZooKeeperSaslClient)
javax.security.sasl.SaslException: Error in authenticating with a Zookeeper Quorum member: the quorum member's saslToken is null.
at org.apache.zookeeper.client.ZooKeeperSaslClient.createSaslToken(ZooKeeperSaslClient.java:312)
at org.apache.zookeeper.client.ZooKeeperSaslClient.respondToServer(ZooKeeperSaslClient.java:275)
at org.apache.zookeeper.ClientCnxn$SendThread.readResponse(ClientCnxn.java:882)
at org.apache.zookeeper.ClientCnxnSocketNIO.doIO(ClientCnxnSocketNIO.java:103)
at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:365)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1223)
[2022-05-29 16:54:50,431] INFO Unable to read additional data from server sessionid 0x100003ea9fd0008, likely server has closed socket, closing socket connection and attempting reconnect (org.apache.zookeeper.ClientCnxn)
[2022-05-29 16:54:50,430] ERROR [ZooKeeperClient Kafka server] Auth failed. (kafka.zookeeper.ZooKeeperClient)
[2022-05-29 16:54:50,452] INFO EventThread shut down for session: 0x100003ea9fd0008 (org.apache.zookeeper.ClientCnxn)
My kafka_jaas file in kafka server:
KafkaClient {
org.apache.kafka.common.security.plain.PlainLoginModule required
username="user_kafka"
password="secret";
};
KafkaServer {
org.apache.kafka.common.security.plain.PlainLoginModule required
username="kafka"
password="CMwvKfeVJociGkSToMZQ"
user_kafka="CMwvKfeVJociGkSToMZQ"
user_user_kafka="secret";
org.apache.kafka.common.security.scram.ScramLoginModule required;
};
Client {
org.apache.kafka.common.security.plain.PlainLoginModule required
username="kafka"
password="2q0T4HFZwa21DCRlfqxX";
};
My zoo_jaas file in ZooKeeper server:
Client {
org.apache.zookeeper.server.auth.DigestLoginModule required
username="kafka"
password="secret";
};
Server {
org.apache.zookeeper.server.auth.DigestLoginModule required
user_kafka="secret";
};
Any help/suggestion would be really beneficial. Thanks.
The following setup worked for me:
Kafka jaas file:
KafkaServer {
org.apache.kafka.common.security.plain.PlainLoginModule required
username="admin"
password="admin-secret"
user_admin="admin-secret";
};
Client {
org.apache.kafka.common.security.plain.PlainLoginModule required
username="admin"
password="admin-secret";
};
Zookeeper jaas file:
Server {
org.apache.zookeeper.server.auth.DigestLoginModule required
user_admin="admin-secret";
};
Kafka producer/consumer client properties:
security.protocol=SASL_PLAINTEXT
sasl.mechanism=PLAIN
sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required username="admin" password="admin-secret";
I noticed that you was running Kafka with Bitnami Kafka container. It can be running successfully using Docker Compose.
First, create a docker-compose.yml file like this:
version: '3'
services:
zookeeper:
image: 'bitnami/zookeeper:3.6'
ports:
- '2181:2181'
environment:
- ZOO_ENABLE_AUTH=yes
- ZOO_SERVER_USERS=kafka
- ZOO_SERVER_PASSWORDS=secret
- ZOO_CLIENT_USER=kafka
- ZOO_CLIENT_PASSWORD=secret
kafka:
image: 'bitnami/kafka:2.8.1'
ports:
- '9093:9093'
environment:
- ALLOW_PLAINTEXT_LISTENER=no
- KAFKA_CFG_ZOOKEEPER_CONNECT=zookeeper:2181
- KAFKA_CFG_LISTENERS=INTERNAL://:9092,CLIENT://:9093,
- KAFKA_CFG_ADVERTISED_LISTENERS=INTERNAL://kafka:9092,CLIENT://localhost:9093
- KAFKA_INTER_BROKER_LISTENER_NAME=INTERNAL
- KAFKA_CFG_LISTENER_SECURITY_PROTOCOL_MAP=INTERNAL:SASL_PLAINTEXT,CLIENT:SASL_PLAINTEXT
- KAFKA_CFG_SASL_MECHANISM_INTER_BROKER_PROTOCOL=PLAIN
#Client credentials
- KAFKA_CLIENT_USERS=user_kafka
- KAFKA_CLIENT_PASSWORDS=secret
#Interbroker credentials
- KAFKA_INTER_BROKER_USER=kafka
- KAFKA_INTER_BROKER_PASSWORD=CMwvKfeVJociGkSToMZQ
#Zookeeper credentials
- KAFKA_ZOOKEEPER_PROTOCOL=SASL
- KAFKA_ZOOKEEPER_USER=kafka
- KAFKA_ZOOKEEPER_PASSWORD=secret
depends_on:
- zookeeper
And then launch the containers using:
$ docker-compose up -d
# list the containers
$ docker-compose ps
Finally you will find kafka and zookeeper containers running.

Getting fatal error showing time out exception when trying to run kafka server

[2021-04-08 02:53:18,713] INFO [ZooKeeperClient Kafka server] Closing. (kafka.zookeeper.ZooKeeperClient)
[2021-04-08 02:53:33,182] WARN Client session timed out, have not heard from server in 18000ms for sessionid 0x0 (org.apache.zookeeper.ClientCnxn)
[2021-04-08 02:53:33,288] INFO Session: 0x0 closed (org.apache.zookeeper.ZooKeeper)
[2021-04-08 02:53:33,288] INFO EventThread shut down for session: 0x0 (org.apache.zookeeper.ClientCnxn)
[2021-04-08 02:53:33,290] INFO [ZooKeeperClient Kafka server] Closed. (kafka.zookeeper.ZooKeeperClient)
[2021-04-08 02:53:33,295] ****ERROR Fatal error during KafkaServer startup. Prepare to shutdown (kafka.server.KafkaServer)
kafka.zookeeper.ZooKeeperClientTimeoutException: Timed out waiting for connection while in state: CONNECTING**
at kafka.zookeeper.ZooKeeperClient.waitUntilConnected(ZooKeeperClient.scala:262)
at kafka.zookeeper.ZooKeeperClient.<init>(ZooKeeperClient.scala:119)
at kafka.zk.KafkaZkClient$.apply(KafkaZkClient.scala:1881)
at kafka.server.KafkaServer.createZkClient$1(KafkaServer.scala:441)
at kafka.server.KafkaServer.initZkClient(KafkaServer.scala:466)
at kafka.server.KafkaServer.startup(KafkaServer.scala:233)
at kafka.server.KafkaServerStartable.startup(KafkaServerStartable.scala:44)
at kafka.Kafka$.main(Kafka.scala:82)
at kafka.Kafka.main(Kafka.scala)
[2021-04-08 02:53:33,300] INFO shutting down (kafka.server.KafkaServer)
[2021-04-08 02:53:33,308] INFO App info kafka.server for 0 unregistered (org.apache.kafka.common.utils.AppInfoParser)
[2021-04-08 02:53:33,309] INFO shut down completed (kafka.server.KafkaServer)
[2021-04-08 02:53:33,310] ERROR Exiting Kafka. (kafka.server.KafkaServerStartable)
[2021-04-08 02:53:33,311] INFO shutting down (kafka.server.KafkaServer)
It seems that the zk server is not started. Kafka needs zookeeper, you need to download and start zookeeper before start kafka server.

Kafka unable to connect to zookeeper ensemble on EKS

I am trying to run Kafka cluster on AWS EKS cluster (v1.16). I am using bitnami helm charts.
https://github.com/bitnami/charts/tree/master/bitnami/kafka
https://github.com/bitnami/charts/tree/master/bitnami/zookeeper
I have deployed zookeeper ensemble successfully using below command:
helm install zookeeper bitnami/zookeeper --set replicaCount=3 --set auth.enabled=false --set allowAnonymousLogin=true --set persistence.storageClass=ebs --set persistence.accessModes={ReadWriteOnce} --set persistence.size=1Gi --set podLabels."app\.kubernetes\.io/version"="1.0"
It outputs:
ZooKeeper can be accessed via port 2181 on the following DNS name from within your cluster:
zookeeper.pulse.svc.cluster.local
Now, I am trying to deploy Kafka cluster with below command:
helm install kafka bitnami/kafka --set replicaCount=3 --set zookeeper.enabled=false --set externalZookeeper.servers=zookeeper.pulse.svc.cluster.local --set autoCreateTopicsEnable=true --set persistence.storageClass=ebs --set persistence.accessModes={ReadWriteOnce} --set persistence.size=1Gi --set podLabels."app\.kubernetes\.io/version"="1.0"
It outputs:
Kafka can be accessed by consumers via port 9092 on the following DNS name from within your cluster:
kafka.pulse.svc.cluster.local
Each Kafka broker can be accessed by producers via port 9092 on the following DNS name(s) from within your cluster:
kafka-0.kafka-headless.pulse.svc.cluster.local
kafka-1.kafka-headless.pulse.svc.cluster.local
kafka-2.kafka-headless.pulse.svc.cluster.local
It creates 3 pods but none of the pod is able to connect to zookeeper. I am not getting what is the issue here.
Kafka pod logs:
2020-07-06T11:22:40.506134648Z 11:22:40.50 Welcome to the Bitnami kafka container
2020-07-06T11:22:40.507301179Z 11:22:40.50 Subscribe to project updates by watching https://github.com/bitnami/bitnami-docker-kafka
2020-07-06T11:22:40.508519907Z 11:22:40.50 Submit issues and feature requests at https://github.com/bitnami/bitnami-docker-kafka/issues
2020-07-06T11:22:40.51039472Z 11:22:40.50
2020-07-06T11:22:40.511630347Z 11:22:40.51 INFO ==> ** Starting Kafka setup **
2020-07-06T11:22:40.55379314Z 11:22:40.55 WARN ==> You set the environment variable ALLOW_PLAINTEXT_LISTENER=yes. For safety reasons, do not use this flag in a production environment.
2020-07-06T11:22:40.561203295Z 11:22:40.56 INFO ==> Initializing Kafka...
2020-07-06T11:22:40.565054949Z 11:22:40.56 INFO ==> No injected configuration files found, creating default config files
2020-07-06T11:22:40.723721499Z 11:22:40.72 INFO ==> Configuring Kafka for inter-broker communications with PLAINTEXT authentication.
2020-07-06T11:22:40.726161543Z 11:22:40.72 WARN ==> Inter-broker communications are configured as PLAINTEXT. This is not safe for production environments.
2020-07-06T11:22:40.727497832Z 11:22:40.72 INFO ==> Configuring Kafka for client communications with PLAINTEXT authentication.
2020-07-06T11:22:40.731790674Z 11:22:40.73 WARN ==> Client communications are configured using PLAINTEXT listeners. For safety reasons, do not use this in a production environment.
2020-07-06T11:22:40.73699684Z 11:22:40.73 INFO ==> ** Kafka setup finished! **
2020-07-06T11:22:40.737001986Z
2020-07-06T11:22:40.746297253Z 11:22:40.74 INFO ==> ** Starting Kafka **
2020-07-06T11:22:41.512303802Z [2020-07-06 11:22:41,511] INFO Registered kafka:type=kafka.Log4jController MBean (kafka.utils.Log4jControllerRegistration$)
2020-07-06T11:22:42.008231959Z [2020-07-06 11:22:42,007] INFO starting (kafka.server.KafkaServer)
2020-07-06T11:22:42.009112085Z [2020-07-06 11:22:42,008] INFO Connecting to zookeeper on zookeeper.pulse.svc.cluster.local (kafka.server.KafkaServer)
2020-07-06T11:22:42.028233655Z [2020-07-06 11:22:42,028] INFO [ZooKeeperClient Kafka server] Initializing a new session to zookeeper.pulse.svc.cluster.local. (kafka.zookeeper.ZooKeeperClient)
2020-07-06T11:22:42.032763227Z [2020-07-06 11:22:42,032] INFO Client environment:zookeeper.version=3.5.7-f0fdd52973d373ffd9c86b81d99842dc2c7f660e, built on 02/10/2020 11:30 GMT (org.apache.zookeeper.ZooKeeper)
2020-07-06T11:22:42.032776511Z [2020-07-06 11:22:42,032] INFO Client environment:host.name=kafka-0.kafka-headless.pulse.svc.cluster.local (org.apache.zookeeper.ZooKeeper)
2020-07-06T11:22:42.03283528Z [2020-07-06 11:22:42,032] INFO Client environment:java.version=11.0.7 (org.apache.zookeeper.ZooKeeper)
2020-07-06T11:22:42.032984414Z [2020-07-06 11:22:42,032] INFO Client environment:java.vendor=BellSoft (org.apache.zookeeper.ZooKeeper)
2020-07-06T11:22:42.033005091Z [2020-07-06 11:22:42,032] INFO Client environment:java.home=/opt/bitnami/java (org.apache.zookeeper.ZooKeeper)
2020-07-06T11:22:42.03312054Z [2020-07-06 11:22:42,032] INFO Client environment:java.class.path=/opt/bitnami/kafka/bin/../libs/activation-1.1.1.jar:/opt/bitnami/kafka/bin/../libs/aopalliance-repackaged-2.5.0.jar:/opt/bitnami/kafka/bin/../libs/argparse4j-0.7.0.jar:/opt/bitnami/kafka/bin/../libs/audience-annotations-0.5.0.jar:/opt/bitnami/kafka/bin/../libs/commons-cli-1.4.jar:/opt/bitnami/kafka/bin/../libs/commons-lang3-3.8.1.jar:/opt/bitnami/kafka/bin/../libs/connect-api-2.5.0.jar:/opt/bitnami/kafka/bin/../libs/connect-basic-auth-extension-2.5.0.jar:/opt/bitnami/kafka/bin/../libs/connect-file-2.5.0.jar:/opt/bitnami/kafka/bin/../libs/connect-json-2.5.0.jar:/opt/bitnami/kafka/bin/../libs/connect-mirror-2.5.0.jar:/opt/bitnami/kafka/bin/../libs/connect-mirror-client-2.5.0.jar:/opt/bitnami/kafka/bin/../libs/connect-runtime-2.5.0.jar:/opt/bitnami/kafka/bin/../libs/connect-transforms-2.5.0.jar:/opt/bitnami/kafka/bin/../libs/hk2-api-2.5.0.jar:/opt/bitnami/kafka/bin/../libs/hk2-locator-2.5.0.jar:/opt/bitnami/kafka/bin/../libs/hk2-utils-2.5.0.jar:/opt/bitnami/kafka/bin/../libs/jackson-annotations-2.10.2.jar:/opt/bitnami/kafka/bin/../libs/jackson-core-2.10.2.jar:/opt/bitnami/kafka/bin/../libs/jackson-databind-2.10.2.jar:/opt/bitnami/kafka/bin/../libs/jackson-dataformat-csv-2.10.2.jar:/opt/bitnami/kafka/bin/../libs/jackson-datatype-jdk8-2.10.2.jar:/opt/bitnami/kafka/bin/../libs/jackson-jaxrs-base-2.10.2.jar:/opt/bitnami/kafka/bin/../libs/jackson-jaxrs-json-provider-2.10.2.jar:/opt/bitnami/kafka/bin/../libs/jackson-module-jaxb-annotations-2.10.2.jar:/opt/bitnami/kafka/bin/../libs/jackson-module-paranamer-2.10.2.jar:/opt/bitnami/kafka/bin/../libs/jackson-module-scala_2.12-2.10.2.jar:/opt/bitnami/kafka/bin/../libs/jakarta.activation-api-1.2.1.jar:/opt/bitnami/kafka/bin/../libs/jakarta.annotation-api-1.3.4.jar:/opt/bitnami/kafka/bin/../libs/jakarta.inject-2.5.0.jar:/opt/bitnami/kafka/bin/../libs/jakarta.ws.rs-api-2.1.5.jar:/opt/bitnami/kafka/bin/../libs/jakarta.xml.bind-api-2.3.2.jar:/opt/bitnami/kafka/bin/../libs/javassist-3.22.0-CR2.jar:/opt/bitnami/kafka/bin/../libs/javassist-3.26.0-GA.jar:/opt/bitnami/kafka/bin/../libs/javax.servlet-api-3.1.0.jar:/opt/bitnami/kafka/bin/../libs/javax.ws.rs-api-2.1.1.jar:/opt/bitnami/kafka/bin/../libs/jaxb-api-2.3.0.jar:/opt/bitnami/kafka/bin/../libs/jersey-client-2.28.jar:/opt/bitnami/kafka/bin/../libs/jersey-common-2.28.jar:/opt/bitnami/kafka/bin/../libs/jersey-container-servlet-2.28.jar:/opt/bitnami/kafka/bin/../libs/jersey-container-servlet-core-2.28.jar:/opt/bitnami/kafka/bin/../libs/jersey-hk2-2.28.jar:/opt/bitnami/kafka/bin/../libs/jersey-media-jaxb-2.28.jar:/opt/bitnami/kafka/bin/../libs/jersey-server-2.28.jar:/opt/bitnami/kafka/bin/../libs/jetty-client-9.4.24.v20191120.jar:/opt/bitnami/kafka/bin/../libs/jetty-continuation-9.4.24.v20191120.jar:/opt/bitnami/kafka/bin/../libs/jetty-http-9.4.24.v20191120.jar:/opt/bitnami/kafka/bin/../libs/jetty-io-9.4.24.v20191120.jar:/opt/bitnami/kafka/bin/../libs/jetty-security-9.4.24.v20191120.jar:/opt/bitnami/kafka/bin/../libs/jetty-server-9.4.24.v20191120.jar:/opt/bitnami/kafka/bin/../libs/jetty-servlet-9.4.24.v20191120.jar:/opt/bitnami/kafka/bin/../libs/jetty-servlets-9.4.24.v20191120.jar:/opt/bitnami/kafka/bin/../libs/jetty-util-9.4.24.v20191120.jar:/opt/bitnami/kafka/bin/../libs/jopt-simple-5.0.4.jar:/opt/bitnami/kafka/bin/../libs/kafka-clients-2.5.0.jar:/opt/bitnami/kafka/bin/../libs/kafka-log4j-appender-2.5.0.jar:/opt/bitnami/kafka/bin/../libs/kafka-streams-2.5.0.jar:/opt/bitnami/kafka/bin/../libs/kafka-streams-examples-2.5.0.jar:/opt/bitnami/kafka/bin/../libs/kafka-streams-scala_2.12-2.5.0.jar:/opt/bitnami/kafka/bin/../libs/kafka-streams-test-utils-2.5.0.jar:/opt/bitnami/kafka/bin/../libs/kafka-tools-2.5.0.jar:/opt/bitnami/kafka/bin/../libs/kafka_2.12-2.5.0-sources.jar:/opt/bitnami/kafka/bin/../libs/kafka_2.12-2.5.0.jar:/opt/bitnami/kafka/bin/../libs/log4j-1.2.17.jar:/opt/bitnami/kafka/bin/../libs/lz4-java-1.7.1.jar:/opt/bitnami/kafka/bin/../libs/maven-artifact-3.6.3.jar:/opt/bitnami/kafka/bin/../libs/metrics-core-2.2.0.jar:/opt/bitnami/kafka/bin/../libs/netty-buffer-4.1.45.Final.jar:/opt/bitnami/kafka/bin/../libs/netty-codec-4.1.45.Final.jar:/opt/bitnami/kafka/bin/../libs/netty-common-4.1.45.Final.jar:/opt/bitnami/kafka/bin/../libs/netty-handler-4.1.45.Final.jar:/opt/bitnami/kafka/bin/../libs/netty-resolver-4.1.45.Final.jar:/opt/bitnami/kafka/bin/../libs/netty-transport-4.1.45.Final.jar:/opt/bitnami/kafka/bin/../libs/netty-transport-native-epoll-4.1.45.Final.jar:/opt/bitnami/kafka/bin/../libs/netty-transport-native-unix-common-4.1.45.Final.jar:/opt/bitnami/kafka/bin/../libs/osgi-resource-locator-1.0.1.jar:/opt/bitnami/kafka/bin/../libs/paranamer-2.8.jar:/opt/bitnami/kafka/bin/../libs/plexus-utils-3.2.1.jar:/opt/bitnami/kafka/bin/../libs/reflections-0.9.12.jar:/opt/bitnami/kafka/bin/../libs/rocksdbjni-5.18.3.jar:/opt/bitnami/kafka/bin/../libs/scala-collection-compat_2.12-2.1.3.jar:/opt/bitnami/kafka/bin/../libs/scala-java8-compat_2.12-0.9.0.jar:/opt/bitnami/kafka/bin/../libs/scala-library-2.12.10.jar:/opt/bitnami/kafka/bin/../libs/scala-logging_2.12-3.9.2.jar:/opt/bitnami/kafka/bin/../libs/scala-reflect-2.12.10.jar:/opt/bitnami/kafka/bin/../libs/slf4j-api-1.7.30.jar:/opt/bitnami/kafka/bin/../libs/slf4j-log4j12-1.7.30.jar:/opt/bitnami/kafka/bin/../libs/snappy-java-1.1.7.3.jar:/opt/bitnami/kafka/bin/../libs/validation-api-2.0.1.Final.jar:/opt/bitnami/kafka/bin/../libs/zookeeper-3.5.7.jar:/opt/bitnami/kafka/bin/../libs/zookeeper-jute-3.5.7.jar:/opt/bitnami/kafka/bin/../libs/zstd-jni-1.4.4-7.jar (org.apache.zookeeper.ZooKeeper)
2020-07-06T11:22:42.033182063Z [2020-07-06 11:22:42,033] INFO Client environment:java.library.path=/usr/java/packages/lib:/usr/lib64:/lib64:/lib:/usr/lib (org.apache.zookeeper.ZooKeeper)
2020-07-06T11:22:42.033188827Z [2020-07-06 11:22:42,033] INFO Client environment:java.io.tmpdir=/tmp (org.apache.zookeeper.ZooKeeper)
2020-07-06T11:22:42.03322714Z [2020-07-06 11:22:42,033] INFO Client environment:java.compiler=<NA> (org.apache.zookeeper.ZooKeeper)
2020-07-06T11:22:42.033253354Z [2020-07-06 11:22:42,033] INFO Client environment:os.name=Linux (org.apache.zookeeper.ZooKeeper)
2020-07-06T11:22:42.033280159Z [2020-07-06 11:22:42,033] INFO Client environment:os.arch=amd64 (org.apache.zookeeper.ZooKeeper)
2020-07-06T11:22:42.033324405Z [2020-07-06 11:22:42,033] INFO Client environment:os.version=4.14.181-140.257.amzn2.x86_64 (org.apache.zookeeper.ZooKeeper)
2020-07-06T11:22:42.033355581Z [2020-07-06 11:22:42,033] INFO Client environment:user.name=? (org.apache.zookeeper.ZooKeeper)
2020-07-06T11:22:42.033399449Z [2020-07-06 11:22:42,033] INFO Client environment:user.home=? (org.apache.zookeeper.ZooKeeper)
2020-07-06T11:22:42.03340494Z [2020-07-06 11:22:42,033] INFO Client environment:user.dir=/ (org.apache.zookeeper.ZooKeeper)
2020-07-06T11:22:42.033468947Z [2020-07-06 11:22:42,033] INFO Client environment:os.memory.free=1015MB (org.apache.zookeeper.ZooKeeper)
2020-07-06T11:22:42.033509114Z [2020-07-06 11:22:42,033] INFO Client environment:os.memory.max=1024MB (org.apache.zookeeper.ZooKeeper)
2020-07-06T11:22:42.033536891Z [2020-07-06 11:22:42,033] INFO Client environment:os.memory.total=1024MB (org.apache.zookeeper.ZooKeeper)
2020-07-06T11:22:42.035251257Z [2020-07-06 11:22:42,035] INFO Initiating client connection, connectString=zookeeper.pulse.svc.cluster.local sessionTimeout=18000 watcher=kafka.zookeeper.ZooKeeperClient$ZooKeeperClientWatcher$#6ee6f53 (org.apache.zookeeper.ZooKeeper)
2020-07-06T11:22:42.038953719Z [2020-07-06 11:22:42,038] INFO jute.maxbuffer value is 4194304 Bytes (org.apache.zookeeper.ClientCnxnSocket)
2020-07-06T11:22:42.043407452Z [2020-07-06 11:22:42,043] INFO zookeeper.request.timeout value is 0. feature enabled= (org.apache.zookeeper.ClientCnxn)
2020-07-06T11:22:42.045196444Z [2020-07-06 11:22:42,045] INFO [ZooKeeperClient Kafka server] Waiting until connected. (kafka.zookeeper.ZooKeeperClient)
2020-07-06T11:22:42.053941415Z [2020-07-06 11:22:42,053] INFO Opening socket connection to server zookeeper.pulse.svc.cluster.local/172.20.162.36:2181. Will not attempt to authenticate using SASL (unknown error) (org.apache.zookeeper.ClientCnxn)
2020-07-06T11:22:42.057906383Z [2020-07-06 11:22:42,057] INFO Socket connection established, initiating session, client: /100.64.5.213:52738, server: zookeeper.pulse.svc.cluster.local/172.20.162.36:2181 (org.apache.zookeeper.ClientCnxn)
2020-07-06T11:22:42.061035524Z [2020-07-06 11:22:42,060] INFO Unable to read additional data from server sessionid 0x0, likely server has closed socket, closing socket connection and attempting reconnect (org.apache.zookeeper.ClientCnxn)
2020-07-06T11:22:43.632054003Z [2020-07-06 11:22:43,631] INFO Opening socket connection to server zookeeper.pulse.svc.cluster.local/172.20.162.36:2181. Will not attempt to authenticate using SASL (unknown error) (org.apache.zookeeper.ClientCnxn)
2020-07-06T11:22:43.632596098Z [2020-07-06 11:22:43,632] INFO Socket connection established, initiating session, client: /100.64.5.213:52756, server: zookeeper.pulse.svc.cluster.local/172.20.162.36:2181 (org.apache.zookeeper.ClientCnxn)
2020-07-06T11:22:43.634993004Z [2020-07-06 11:22:43,634] INFO Unable to read additional data from server sessionid 0x0, likely server has closed socket, closing socket connection and attempting reconnect (org.apache.zookeeper.ClientCnxn)
2020-07-06T11:22:44.760870715Z [2020-07-06 11:22:44,760] INFO Opening socket connection to server zookeeper.pulse.svc.cluster.local/172.20.162.36:2181. Will not attempt to authenticate using SASL (unknown error) (org.apache.zookeeper.ClientCnxn)
2020-07-06T11:22:44.761283232Z [2020-07-06 11:22:44,761] INFO Socket connection established, initiating session, client: /100.64.5.213:52772, server: zookeeper.pulse.svc.cluster.local/172.20.162.36:2181 (org.apache.zookeeper.ClientCnxn)
2020-07-06T11:22:44.763353195Z [2020-07-06 11:22:44,763] INFO Unable to read additional data from server sessionid 0x0, likely server has closed socket, closing socket connection and attempting reconnect (org.apache.zookeeper.ClientCnxn)
2
2020-07-06T11:23:12.738834004Z [2020-07-06 11:23:12,738] INFO Session: 0x0 closed (org.apache.zookeeper.ZooKeeper)
2020-07-06T11:23:12.738918322Z [2020-07-06 11:23:12,738] INFO EventThread shut down for session: 0x0 (org.apache.zookeeper.ClientCnxn)
2020-07-06T11:23:12.740751654Z [2020-07-06 11:23:12,740] INFO [ZooKeeperClient Kafka server] Closed. (kafka.zookeeper.ZooKeeperClient)
2020-07-06T11:23:12.745313347Z [2020-07-06 11:23:12,743] ERROR Fatal error during KafkaServer startup. Prepare to shutdown (kafka.server.KafkaServer)
2020-07-06T11:23:12.745331011Z kafka.zookeeper.ZooKeeperClientTimeoutException: Timed out waiting for connection while in state: CONNECTING
2020-07-06T11:23:12.745335139Z at kafka.zookeeper.ZooKeeperClient.$anonfun$waitUntilConnected$3(ZooKeeperClient.scala:262)
2020-07-06T11:23:12.745338245Z at kafka.zookeeper.ZooKeeperClient.waitUntilConnected(ZooKeeperClient.scala:258)
2020-07-06T11:23:12.745340837Z at kafka.zookeeper.ZooKeeperClient.<init>(ZooKeeperClient.scala:119)
2020-07-06T11:23:12.745343374Z at kafka.zk.KafkaZkClient$.apply(KafkaZkClient.scala:1863)
2020-07-06T11:23:12.745345577Z at kafka.server.KafkaServer.createZkClient$1(KafkaServer.scala:378)
2020-07-06T11:23:12.745347726Z at kafka.server.KafkaServer.initZkClient(KafkaServer.scala:403)
2020-07-06T11:23:12.745349947Z at kafka.server.KafkaServer.startup(KafkaServer.scala:210)
2020-07-06T11:23:12.745352077Z at kafka.server.KafkaServerStartable.startup(KafkaServerStartable.scala:44)
2020-07-06T11:23:12.745354263Z at kafka.Kafka$.main(Kafka.scala:82)
2020-07-06T11:23:12.745368643Z at kafka.Kafka.main(Kafka.scala)
2020-07-06T11:23:12.745806818Z [2020-07-06 11:23:12,745] INFO shutting down (kafka.server.KafkaServer)
2020-07-06T11:23:12.752833659Z [2020-07-06 11:23:12,752] INFO shut down completed (kafka.server.KafkaServer)
2020-07-06T11:23:12.753305908Z [2020-07-06 11:23:12,753] ERROR Exiting Kafka. (kafka.server.KafkaServerStartable)
2020-07-06T11:23:12.757961524Z [2020-07-06 11:23:12,757] INFO shutting down (kafka.server.KafkaServer)
Close
Also from Kafka pod machine, curl gives below o/p:
istio-proxy#kafka-0:/$ curl zookeeper.pulse.svc.cluster.local:2181
curl: (52) Empty reply from server
istio-proxy#kafka-0:/$
Note: I am having istio sidecars with mTLS disabled.
Please help.
UPDATE
HI,
It comes out to be the Istio porxy issue. I uninstalled Istio and it worked out.
https://github.com/bitnami/bitnami-docker-kafka/issues/38#issuecomment-451381003
This works fine for me on my local cluster. Since you are using EKS, you are most likely using AWS CNI(?). CNI allocates IP addresses in your VPC and if you are not allowing your security groups access to the VPC range it will not be able to access the pods. (172.20.162.36:2181 looks like VPC address for example).
Another thing you can check if is if you have some NetworkPolicy preventing access:
kubectl get netpol
It's kind of odd that you get the expected response from Zookeeper:
curl zookeeper.pulse.svc.cluster.local:2181
curl: (52) Empty reply from server
So it could be possible that zookeeper.pulse.svc.cluster.local is resolving to an 'accessible' :2181. In any case, it looks like a firewall/network policy issue.

Zookeeper for Apache-Kafka: problems with port 2181 in Ubuntu 18.04.01 Server

I'm trying to figure out how to successfully reach the end of this tutorial, for the installation and first running of Apache-Kafka : How To Install Apache Kafka on Ubuntu 18.04 LTS
After killing the previously running zookeeper instance, I tried, again!, to start the zookeeper server again, but when the process reaches the phase of port binding, it stops forever..: KafkaActivationFailed
marco#PC:/usr/share/zookeeper$ sudo lsof -i :2181
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
java 4545 zookeeper 32u IPv6 43042 0t0 TCP *:2181 (LISTEN)
marco#PC:/usr/share/zookeeper$ sudo kill 4545
marco#PC:/usr/local/kafka$ bin/zookeeper-server-start.sh
config/zookeeper.properties
[2018-09-04 12:18:14,608] INFO Reading configuration from:
config/zookeeper.properties
(org.apache.zookeeper.server.quorum.QuorumPeerConfig)
[2018-09-04 12:18:14,609] INFO autopurge.snapRetainCount set to 3
(org.apache.zookeeper.server.DatadirCleanupManager)
[2018-09-04 12:18:14,609] INFO autopurge.purgeInterval set to 0
(org.apache.zookeeper.server.DatadirCleanupManager)
[2018-09-04 12:18:14,609] INFO Purge task is not scheduled.
(org.apache.zookeeper.server.DatadirCleanupManager)
[2018-09-04 12:18:14,610] WARN Either no config or no quorum defined in
config, running in standalone mode
(org.apache.zookeeper.server.quorum.QuorumPeerMain)
[2018-09-04 12:18:14,620] INFO Reading configuration from:
config/zookeeper.properties
(org.apache.zookeeper.server.quorum.QuorumPeerConfig)
[2018-09-04 12:18:14,620] INFO Starting server
(org.apache.zookeeper.server.ZooKeeperServerMain)
[2018-09-04 12:18:14,624] INFO Server
environment:zookeeper.version=3.4.13-2d71af4dbe22557fda74f9a9b4309b15a7487f03
built on 06/29/2018 00:39 GMT (org.apache.zookeeper.server.ZooKeeperServer)
[2018-09-04 12:18:14,624] INFO Server environment:host.name=PC.station
(org.apache.zookeeper.server.ZooKeeperServer)
[2018-09-04 12:18:14,625] INFO Server environment:java.version=1.8.0_181
(org.apache.zookeeper.server.ZooKeeperServer)
[2018-09-04 12:18:14,625] INFO Server environment:java.vendor=Oracle
Corporation (org.apache.zookeeper.server.ZooKeeperServer)
[2018-09-04 12:18:14,625] INFO Server environment:java.home=/usr/lib/jvm
/java-8-oracle/jre (org.apache.zookeeper.server.ZooKeeperServer)
[2018-09-04 12:18:14,625] INFO Server environment:java.class.path=/usr/local
/kafka/bin/../libs/activation-1.1.1.jar:/usr/local/kafka/bin/..
/libs/aopalliance-repackaged-2.5.0-b42.jar:/usr/local/kafka/bin/..
/libs/argparse4j-0.7.0.jar:/usr/local/kafka/bin/../libs/audience-annotations-
0.5.0.jar:/usr/local/kafka/bin/../libs/commons-lang3-3.5.jar:/usr/local/kafka
/bin/../libs/connect-api-2.0.0.jar:/usr/local/kafka/bin/../libs/connect-
basic-auth-extension-2.0.0.jar:/usr/local/kafka/bin/../libs/connect-
file-2.0.0.jar:/usr/local/kafka/bin/../libs/connect-json-2.0.0.jar:/usr/local
/kafka/bin/../libs/connect-runtime-2.0.0.jar:/usr/local/kafka/bin/..
/libs/connect-transforms-2.0.0.jar:/usr/local/kafka/bin/../libs/guava-
20.0.jar:/usr/local/kafka/bin/../libs/hk2-api-2.5.0-b42.jar:/usr/local/kafka
/bin/../libs/hk2-locator-2.5.0-b42.jar:/usr/local/kafka/bin/../libs/hk2-
utils-2.5.0-b42.jar:/usr/local/kafka/bin/../libs/jackson-annotations-
2.9.6.jar:/usr/local/kafka/bin/../libs/jackson-core-2.9.6.jar:/usr/local
/kafka/bin/../libs/jackson-databind-2.9.6.jar:/usr/local/kafka/bin/..
/libs/jackson-jaxrs-base-2.9.6.jar:/usr/local/kafka/bin/../libs/jackson-
jaxrs-json-provider-2.9.6.jar:/usr/local/kafka/bin/../libs/jackson-module-
jaxb-annotations-2.9.6.jar:/usr/local/kafka/bin/../libs/javassist-3.22.0-
CR2.jar:/usr/local/kafka/bin/../libs/javax.annotation-api-1.2.jar:/usr/local
/kafka/bin/../libs/javax.inject-1.jar:/usr/local/kafka/bin/..
/libs/javax.inject-2.5.0-b42.jar:/usr/local/kafka/bin/../libs/javax.servlet-
api-3.1.0.jar:/usr/local/kafka/bin/../libs/javax.ws.rs-api-2.1.jar:/usr/local
/kafka/bin/../libs/jaxb-api-2.3.0.jar:/usr/local/kafka/bin/../libs/jersey-
client-2.27.jar:/usr/local/kafka/bin/../libs/jersey-common-2.27.jar:
/usr/local/kafka/bin/../libs/jersey-container-servlet-2.27.jar:/usr/local
/kafka/bin/../libs/jersey-container-servlet-core-2.27.jar:/usr/local/kafka
/bin/../libs/jersey-hk2-2.27.jar:/usr/local/kafka/bin/../libs/jersey-media-
jaxb-2.27.jar:/usr/local/kafka/bin/../libs/jersey-server-2.27.jar:/usr/local
/kafka/bin/../libs/jetty-client-9.4.11.v20180605.jar:/usr/local/kafka/bin/..
/libs/jetty-continuation-9.4.11.v20180605.jar:/usr/local/kafka/bin/..
/libs/jetty-http-9.4.11.v20180605.jar:/usr/local/kafka/bin/../libs/jetty-
io-9.4.11.v20180605.jar:/usr/local/kafka/bin/../libs/jetty-security-
9.4.11.v20180605.jar:/usr/local/kafka/bin/../libs/jetty-server-
9.4.11.v20180605.jar:/usr/local/kafka/bin/../libs/jetty-servlet-
9.4.11.v20180605.jar:/usr/local/kafka/bin/../libs/jetty-servlets-
9.4.11.v20180605.jar:/usr/local/kafka/bin/../libs/jetty-
util-9.4.11.v20180605.jar:/usr/local/kafka/bin/../libs/jopt-simple-5.0.4.jar:
/usr/local/kafka/bin/../libs/kafka_2.11-2.0.0.jar:/usr/local/kafka/bin/..
/libs/kafka_2.11-2.0.0-sources.jar:/usr/local/kafka/bin/../libs/kafka-
clients-2.0.0.jar:/usr/local/kafka/bin/../libs/kafka-log4j-appender-
2.0.0.jar:/usr/local/kafka/bin/../libs/kafka-streams-2.0.0.jar:/usr/local
/kafka/bin/../libs/kafka-streams-examples-2.0.0.jar:/usr/local/kafka/bin/..
/libs/kafka-streams-scala_2.11-2.0.0.jar:/usr/local/kafka/bin/../libs/kafka-
streams-test-utils-2.0.0.jar:/usr/local/kafka/bin/../libs/kafka-tools-
2.0.0.jar:/usr/local/kafka/bin/../libs/log4j-1.2.17.jar:/usr/local/kafka
/bin/../libs/lz4-java-1.4.1.jar:/usr/local/kafka/bin/../libs/maven-artifact-
3.5.3.jar:/usr/local/kafka/bin/../libs/metrics-core-2.2.0.jar:/usr/local
/kafka/bin/../libs/osgi-resource-locator-1.0.1.jar:/usr/local/kafka/bin/..
/libs/plexus-utils-3.1.0.jar:/usr/local/kafka/bin/../libs/reflections-
0.9.11.jar:/usr/local/kafka/bin/../libs/rocksdbjni-5.7.3.jar:/usr/local/kafka
/bin/../libs/scala-library-2.11.12.jar:/usr/local/kafka/bin/../libs/scala-
logging_2.11-3.9.0.jar:/usr/local/kafka/bin/../libs/scala-reflect-
2.11.12.jar:/usr/local/kafka/bin/../libs/slf4j-api-1.7.25.jar:/usr/local
/kafka/bin/../libs/slf4j-log4j12-1.7.25.jar:/usr/local/kafka/bin/..
/libs/snappy-java-1.1.7.1.jar:/usr/local/kafka/bin/../libs/validation-
api-1.1.0.Final.jar:/usr/local/kafka/bin/../libs/zkclient-0.10.jar:/usr/local
/kafka/bin/../libs/zookeeper-3.4.13.jar
(org.apache.zookeeper.server.ZooKeeperServer)
[2018-09-04 12:18:14,625] INFO Server environment:java.library.path=/usr
/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
(org.apache.zookeeper.server.ZooKeeperServer)
[2018-09-04 12:18:14,625] INFO Server environment:java.io.tmpdir=/tmp
(org.apache.zookeeper.server.ZooKeeperServer)
[2018-09-04 12:18:14,625] INFO Server environment:java.compiler=<NA>
(org.apache.zookeeper.server.ZooKeeperServer)
[2018-09-04 12:18:14,625] INFO Server environment:os.name=Linux
(org.apache.zookeeper.server.ZooKeeperServer)
[2018-09-04 12:18:14,625] INFO Server environment:os.arch=amd64
(org.apache.zookeeper.server.ZooKeeperServer)
[2018-09-04 12:18:14,625] INFO Server environment:os.version=4.15.0-33-
generic (org.apache.zookeeper.server.ZooKeeperServer)
[2018-09-04 12:18:14,625] INFO Server environment:user.name=marco
(org.apache.zookeeper.server.ZooKeeperServer)
[2018-09-04 12:18:14,625] INFO Server environment:user.home=/home/marco
(org.apache.zookeeper.server.ZooKeeperServer)
[2018-09-04 12:18:14,625] INFO Server environment:user.dir=/usr/local/kafka
(org.apache.zookeeper.server.ZooKeeperServer)
[2018-09-04 12:18:14,630] INFO tickTime set to 3000
(org.apache.zookeeper.server.ZooKeeperServer)
[2018-09-04 12:18:14,630] INFO minSessionTimeout set to -1
(org.apache.zookeeper.server.ZooKeeperServer)
[2018-09-04 12:18:14,630] INFO maxSessionTimeout set to -1
(org.apache.zookeeper.server.ZooKeeperServer)
[2018-09-04 12:18:14,634] INFO Using
org.apache.zookeeper.server.NIOServerCnxn Factory as server connection
factory (org.apache.zookeeper.server.ServerCnxnFactory)
[2018-09-04 12:18:14,636] INFO binding to port 0.0.0.0/0.0.0.0:2181
(org.apache.zookeeper.server.NIOServerCnxnFactory)
The problem is that rebooting zookeeper seems active:
marco#PC:~$ netstat -ant | grep :2181
tcp6 0 0 :::2181 :::*
LISTEN
marco#PC:~$ systemctl status zookeeper
● zookeeper.service - LSB: centralized coordination service
Loaded: loaded (/etc/init.d/zookeeper; generated)
Active: active (running) since Tue 2018-09-04 15:51:24 CEST; 2min 19s
ago
Docs: man:systemd-sysv-generator(8)
Process: 1919 ExecStart=/etc/init.d/zookeeper start (code=exited,
status=0/SUCCESS)
Tasks: 25 (limit: 4915)
CGroup: /system.slice/zookeeper.service
└─1997 /usr/bin/java -cp /etc/zookeeper/conf:/usr/share
/java/jline.jar:/usr/share/java/log4j-1.2.jar:/usr/share
/java/xercesImpl.jar:/usr/share/java/xmlParserAPIs.jar:/usr/share
/java/netty.jar:/usr/share/java/slf4j-api.jar:/usr
Sep 04 15:51:24 PC systemd[1]: Starting LSB: centralized coordination
service...
Sep 04 15:51:24 PC systemd[1]: Started LSB: centralized coordination
service.
But when trying to run kafka... :
marco#PC:~$ cd /usr/local/kafka
marco#PC:/usr/local/kafka$ bin/zookeeper-server-start.sh
config/zookeeper.properties
[2018-09-04 15:54:29,981] INFO Reading configuration from:
config/zookeeper.properties
(org.apache.zookeeper.server.quorum.QuorumPeerConfig)
[2018-09-04 15:54:29,982] INFO autopurge.snapRetainCount set to 3
(org.apache.zookeeper.server.DatadirCleanupManager)
[2018-09-04 15:54:29,983] INFO autopurge.purgeInterval set to 0
(org.apache.zookeeper.server.DatadirCleanupManager)
...
[2018-09-04 15:54:30,012] INFO binding to port 0.0.0.0/0.0.0.0:2181
(org.apache.zookeeper.server.NIOServerCnxnFactory)
[2018-09-04 15:54:30,013] ERROR Unexpected exception, exiting abnormally
(org.apache.zookeeper.server.ZooKeeperServerMain)
java.net.BindException: Address already in use
at sun.nio.ch.Net.bind0(Native Method)
at sun.nio.ch.Net.bind(Net.java:433)
at sun.nio.ch.Net.bind(Net.java:425)
at
sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:223)
at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:74)
at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:67)
$ bin/zookeeper-server-start.sh config/zookeeper.properties
This started okay if you look at the logs
[2018-09-04 12:18:14,636] INFO binding to port 0.0.0.0/0.0.0.0:2181
(org.apache.zookeeper.server.NIOServerCnxnFactory)
And this message will sit there until a Zookeeper client such as Kafka connects to Zookeeper.
When you say you're "trying to run kafka", you're running zookeeper-server-start again, not kafka-server-start, so Zookeeper is already bound to port 2181 and it won't start a second one.
Maybe I "solved" the problem:
I left opened the "idle" window ( First Window ), I opened a new window and started the kafka server : Step 2.b of ApachaKafka-QuickStart : /usr/local/ kafka/ bin/kafka-server-start.sh config/server.properties ( Second Window ). Then I opened another window to create a topic and send some messages:
marco#PC:/usr/local/kafka$ bin/kafka-topics.sh --create --zookeeper localhost:2181
--replication-factor 1 --partitions 1 --topic test
Created topic "test".
marco#PC:/usr/local/kafka$ bin/kafka-topics.sh --list --zookeeper localhost:2181
test
marco#PC:/usr/local/kafka$ bin/kafka-console-producer.sh --broker-list localhost:9092
--topic test
>This is a message
>This is another message
>
( ThirdWindow ) . Finally, in another new window I started a consumer :
marco#PC:/usr/local/kafka$ bin/kafka-console-consumer.sh --bootstrap-server
localhost:9092 --topic test --from-beginning
This is a message
This is another message
: ( FourthWindow )
What I actually do not understand now is why I should open all these four sessions.
Marco

Not able to run Kafka server

I am trying to run kafka server on Ubuntu. When I am running this command:
> bin/zookeeper-server-start.sh config/zookeeper.properties
I am not getting anything and when I am trying to run Kafka server by this command:
> bin/kafka-server-start.sh config/server.properties
I am getting this error.
WARN Session 0x0 for server null, unexpected error, closing socket connection and attempting reconnect (org.apache.zookeeper.ClientCnxn)
java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:361)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1081)
[2017-05-23 13:22:45,136] INFO Terminate ZkClient event thread. (org.I0Itec.zkclient.ZkEventThread)
[2017-05-23 13:22:45,754] INFO Opening socket connection to server localhost/127.0.0.1:2181. Will not attempt to authenticate using SASL (unknown error) (org.apache.zookeeper.ClientCnxn)
[2017-05-23 13:22:45,857] INFO Session: 0x0 closed (org.apache.zookeeper.ZooKeeper)
[2017-05-23 13:22:45,859] FATAL Fatal error during KafkaServer startup. Prepare to shutdown (kafka.server.KafkaServer)
org.I0Itec.zkclient.exception.ZkTimeoutException: Unable to connect to zookeeper server within timeout: 6000
at org.I0Itec.zkclient.ZkClient.connect(ZkClient.java:1223)
at org.I0Itec.zkclient.ZkClient.<init>(ZkClient.java:155)
at org.I0Itec.zkclient.ZkClient.<init>(ZkClient.java:129)
at kafka.utils.ZkUtils$.createZkClientAndConnection(ZkUtils.scala:89)
at kafka.utils.ZkUtils$.apply(ZkUtils.scala:71)
at kafka.server.KafkaServer.initZk(KafkaServer.scala:278)
at kafka.server.KafkaServer.startup(KafkaServer.scala:168)
at kafka.server.KafkaServerStartable.startup(KafkaServerStartable.scala:37)
at kafka.Kafka$.main(Kafka.scala:67)
at kafka.Kafka.main(Kafka.scala)
[2017-05-23 13:22:45,859] INFO EventThread shut down (org.apache.zookeeper.ClientCnxn)
[2017-05-23 13:22:45,860] INFO shutting down (kafka.server.KafkaServer)
[2017-05-23 13:22:45,867] INFO shut down completed (kafka.server.KafkaServer)
[2017-05-23 13:22:45,868] FATAL Fatal error during KafkaServerStartable startup. Prepare to shutdown (kafka.server.KafkaServerStartable)
org.I0Itec.zkclient.exception.ZkTimeoutException: Unable to connect to zookeeper server within timeout: 6000
at org.I0Itec.zkclient.ZkClient.connect(ZkClient.java:1223)
at org.I0Itec.zkclient.ZkClient.<init>(ZkClient.java:155)
at org.I0Itec.zkclient.ZkClient.<init>(ZkClient.java:129)
at kafka.utils.ZkUtils$.createZkClientAndConnection(ZkUtils.scala:89)
at kafka.utils.ZkUtils$.apply(ZkUtils.scala:71)
at kafka.server.KafkaServer.initZk(KafkaServer.scala:278)
at kafka.server.KafkaServer.startup(KafkaServer.scala:168)
at kafka.server.KafkaServerStartable.startup(KafkaServerStartable.scala:37)
at kafka.Kafka$.main(Kafka.scala:67)
at kafka.Kafka.main(Kafka.scala)
[2017-05-23 13:22:45,871] INFO shutting down (kafka.server.KafkaServer)
Kafka output suggests zookeeper is not starting.
If zookeeper-server-start.sh produces no output, please check zookeeper.log. If that doesn't help, try finding the zookeeper java process using ps, e.g.:
ps -elf | grep zookeeper
If it's running you can use the PID of the process to obtain a thread dump or attach a debugger to see where it's stuck at.
Hope that helps.