Related
This question already has answers here:
How to install Kafka on Windows?
(9 answers)
Closed 1 year ago.
I downloaded the kafka_2.13-3.0.0 in Window 10. I opened up CMD as admin. I was able to run zookeeper using the command bin\windows\zookeeper-server-start.bat config\zookeeper.properties. However, when i tried to start the kafka server with the following command
bin\windows\kafka-server-start.bat config\server.properties
, it gave me the following error" below. I am confused here because I can see the kafka-logs being created (see image below). Can someone help me out please ?
Output from running Zookeeper command
[2021-12-01 20:16:17,208] INFO Server environment:java.io.tmpdir=C:\Users\user123456~1\AppData\Local\Temp\ (org.apache.zookeeper.server.ZooKeeperServer)
[2021-12-01 20:16:17,208] INFO Server environment:java.compiler=<NA> (org.apache.zookeeper.server.ZooKeeperServer)
[2021-12-01 20:16:17,208] INFO Server environment:os.name=Windows 10 (org.apache.zookeeper.server.ZooKeeperServer)
[2021-12-01 20:16:17,209] INFO Server environment:os.arch=amd64 (org.apache.zookeeper.server.ZooKeeperServer)
[2021-12-01 20:16:17,209] INFO Server environment:os.version=10.0 (org.apache.zookeeper.server.ZooKeeperServer)
[2021-12-01 20:16:17,209] INFO Server environment:user.name=user123456 (org.apache.zookeeper.server.ZooKeeperServer)
[2021-12-01 20:16:17,210] INFO Server environment:user.home=C:\Users\user123456 (org.apache.zookeeper.server.ZooKeeperServer)
[2021-12-01 20:16:17,210] INFO Server environment:user.dir=C:\kafka_2.13-3.0.0 (org.apache.zookeeper.server.ZooKeeperServer)
[2021-12-01 20:16:17,211] INFO Server environment:os.memory.free=493MB (org.apache.zookeeper.server.ZooKeeperServer)
[2021-12-01 20:16:17,211] INFO Server environment:os.memory.max=512MB (org.apache.zookeeper.server.ZooKeeperServer)
[2021-12-01 20:16:17,211] INFO Server environment:os.memory.total=512MB (org.apache.zookeeper.server.ZooKeeperServer)
[2021-12-01 20:16:17,212] INFO zookeeper.enableEagerACLCheck = false (org.apache.zookeeper.server.ZooKeeperServer)
[2021-12-01 20:16:17,212] INFO zookeeper.digest.enabled = true (org.apache.zookeeper.server.ZooKeeperServer)
[2021-12-01 20:16:17,212] INFO zookeeper.closeSessionTxn.enabled = true (org.apache.zookeeper.server.ZooKeeperServer)
[2021-12-01 20:16:17,213] INFO zookeeper.flushDelay=0 (org.apache.zookeeper.server.ZooKeeperServer)
[2021-12-01 20:16:17,213] INFO zookeeper.maxWriteQueuePollTime=0 (org.apache.zookeeper.server.ZooKeeperServer)
[2021-12-01 20:16:17,214] INFO zookeeper.maxBatchSize=1000 (org.apache.zookeeper.server.ZooKeeperServer)
[2021-12-01 20:16:17,214] INFO zookeeper.intBufferStartingSizeBytes = 1024 (org.apache.zookeeper.server.ZooKeeperServer)
[2021-12-01 20:16:17,215] INFO Weighed connection throttling is disabled (org.apache.zookeeper.server.BlueThrottle)
[2021-12-01 20:16:17,216] INFO minSessionTimeout set to 6000 (org.apache.zookeeper.server.ZooKeeperServer)
[2021-12-01 20:16:17,216] INFO maxSessionTimeout set to 60000 (org.apache.zookeeper.server.ZooKeeperServer)
[2021-12-01 20:16:17,217] INFO Response cache size is initialized with value 400. (org.apache.zookeeper.server.ResponseCache)
[2021-12-01 20:16:17,217] INFO Response cache size is initialized with value 400. (org.apache.zookeeper.server.ResponseCache)
[2021-12-01 20:16:17,218] INFO zookeeper.pathStats.slotCapacity = 60 (org.apache.zookeeper.server.util.RequestPathMetricsCollector)
[2021-12-01 20:16:17,220] INFO zookeeper.pathStats.slotDuration = 15 (org.apache.zookeeper.server.util.RequestPathMetricsCollector)
[2021-12-01 20:16:17,221] INFO zookeeper.pathStats.maxDepth = 6 (org.apache.zookeeper.server.util.RequestPathMetricsCollector)
[2021-12-01 20:16:17,221] INFO zookeeper.pathStats.initialDelay = 5 (org.apache.zookeeper.server.util.RequestPathMetricsCollector)
[2021-12-01 20:16:17,221] INFO zookeeper.pathStats.delay = 5 (org.apache.zookeeper.server.util.RequestPathMetricsCollector)
[2021-12-01 20:16:17,222] INFO zookeeper.pathStats.enabled = false (org.apache.zookeeper.server.util.RequestPathMetricsCollector)
[2021-12-01 20:16:17,224] INFO The max bytes for all large requests are set to 104857600 (org.apache.zookeeper.server.ZooKeeperServer)
[2021-12-01 20:16:17,224] INFO The large request threshold is set to -1 (org.apache.zookeeper.server.ZooKeeperServer)
[2021-12-01 20:16:17,224] INFO Created server with tickTime 3000 minSessionTimeout 6000 maxSessionTimeout 60000 clientPortListenBacklog -1 datadir \tmp\zookeeper\version-2 snapdir \tmp\zookeeper\version-2 (org.apache.zookeeper.server.ZooKeeperServer)
[2021-12-01 20:16:17,234] INFO Using org.apache.zookeeper.server.NIOServerCnxnFactory as server connection factory (org.apache.zookeeper.server.ServerCnxnFactory)
[2021-12-01 20:16:17,235] WARN maxCnxns is not configured, using default value 0. (org.apache.zookeeper.server.ServerCnxnFactory)
[2021-12-01 20:16:17,236] INFO Configuring NIO connection handler with 10s sessionless connection timeout, 2 selector thread(s), 32 worker threads, and 64 kB direct buffers. (org.apache.zookeeper.server.NIOServerCnxnFactory)
[2021-12-01 20:16:17,238] INFO binding to port 0.0.0.0/0.0.0.0:2181 (org.apache.zookeeper.server.NIOServerCnxnFactory)
[2021-12-01 20:16:17,250] INFO Using org.apache.zookeeper.server.watch.WatchManager as watch manager (org.apache.zookeeper.server.watch.WatchManagerFactory)
[2021-12-01 20:16:17,250] INFO Using org.apache.zookeeper.server.watch.WatchManager as watch manager (org.apache.zookeeper.server.watch.WatchManagerFactory)
[2021-12-01 20:16:17,251] INFO zookeeper.snapshotSizeFactor = 0.33 (org.apache.zookeeper.server.ZKDatabase)
[2021-12-01 20:16:17,251] INFO zookeeper.commitLogCount=500 (org.apache.zookeeper.server.ZKDatabase)
[2021-12-01 20:16:17,256] INFO zookeeper.snapshot.compression.method = CHECKED (org.apache.zookeeper.server.persistence.SnapStream)
[2021-12-01 20:16:17,256] INFO Snapshotting: 0x0 to \tmp\zookeeper\version-2\snapshot.0 (org.apache.zookeeper.server.persistence.FileTxnSnapLog)
[2021-12-01 20:16:17,260] INFO Snapshot loaded in 8 ms, highest zxid is 0x0, digest is 1371985504 (org.apache.zookeeper.server.ZKDatabase)
[2021-12-01 20:16:17,260] INFO Snapshotting: 0x0 to \tmp\zookeeper\version-2\snapshot.0 (org.apache.zookeeper.server.persistence.FileTxnSnapLog)
[2021-12-01 20:16:17,261] INFO Snapshot taken in 1 ms (org.apache.zookeeper.server.ZooKeeperServer)
[2021-12-01 20:16:17,274] INFO PrepRequestProcessor (sid:0) started, reconfigEnabled=false (org.apache.zookeeper.server.PrepRequestProcessor)
[2021-12-01 20:16:17,274] INFO zookeeper.request_throttler.shutdownTimeout = 10000 (org.apache.zookeeper.server.RequestThrottler)
[2021-12-01 20:16:17,286] INFO Using checkIntervalMs=60000 maxPerMinute=10000 maxNeverUsedIntervalMs=0 (org.apache.zookeeper.server.ContainerManager)
[2021-12-01 20:16:17,287] INFO ZooKeeper audit is disabled. (org.apache.zookeeper.audit.ZKAuditProvider)
Output from running kafka command
[2021-12-01 20:18:22,577] INFO Client environment:java.io.tmpdir=C:\Users\user123456~1\AppData\Local\Temp\ (org.apache.zookeeper.ZooKeeper)
[2021-12-01 20:18:22,577] INFO Client environment:java.compiler=<NA> (org.apache.zookeeper.ZooKeeper)
[2021-12-01 20:18:22,578] INFO Client environment:os.name=Windows 10 (org.apache.zookeeper.ZooKeeper)
[2021-12-01 20:18:22,579] INFO Client environment:os.arch=amd64 (org.apache.zookeeper.ZooKeeper)
[2021-12-01 20:18:22,579] INFO Client environment:os.version=10.0 (org.apache.zookeeper.ZooKeeper)
[2021-12-01 20:18:22,580] INFO Client environment:user.name=user123456 (org.apache.zookeeper.ZooKeeper)
[2021-12-01 20:18:22,581] INFO Client environment:user.home=C:\Users\user123456 (org.apache.zookeeper.ZooKeeper)
[2021-12-01 20:18:22,581] INFO Client environment:user.dir=C:\kafka_2.13-3.0.0 (org.apache.zookeeper.ZooKeeper)
[2021-12-01 20:18:22,582] INFO Client environment:os.memory.free=1011MB (org.apache.zookeeper.ZooKeeper)
[2021-12-01 20:18:22,583] INFO Client environment:os.memory.max=1024MB (org.apache.zookeeper.ZooKeeper)
[2021-12-01 20:18:22,583] INFO Client environment:os.memory.total=1024MB (org.apache.zookeeper.ZooKeeper)
[2021-12-01 20:18:22,585] INFO Initiating client connection, connectString=localhost:2181 sessionTimeout=18000 watcher=kafka.zookeeper.ZooKeeperClient$ZooKeeperClientWatcher$#74f0ea28 (org.apache.zookeeper.ZooKeeper)
[2021-12-01 20:18:22,593] INFO jute.maxbuffer value is 4194304 Bytes (org.apache.zookeeper.ClientCnxnSocket)
[2021-12-01 20:18:22,597] INFO zookeeper.request.timeout value is 0. feature enabled=false (org.apache.zookeeper.ClientCnxn)
[2021-12-01 20:18:22,598] INFO [ZooKeeperClient Kafka server] Waiting until connected. (kafka.zookeeper.ZooKeeperClient)
[2021-12-01 20:18:22,601] INFO Opening socket connection to server localhost/127.0.0.1:2181. (org.apache.zookeeper.ClientCnxn)
[2021-12-01 20:18:22,601] INFO SASL config status: Will not attempt to authenticate using SASL (unknown error) (org.apache.zookeeper.ClientCnxn)
[2021-12-01 20:18:22,602] INFO Socket connection established, initiating session, client: /127.0.0.1:57191, server: localhost/127.0.0.1:2181 (org.apache.zookeeper.ClientCnxn)
[2021-12-01 20:18:22,617] INFO Session establishment complete on server localhost/127.0.0.1:2181, session id = 0x1000008c9c20000, negotiated timeout = 18000 (org.apache.zookeeper.ClientCnxn)
[2021-12-01 20:18:22,619] INFO [ZooKeeperClient Kafka server] Connected. (kafka.zookeeper.ZooKeeperClient)
[2021-12-01 20:18:22,720] INFO [feature-zk-node-event-process-thread]: Starting (kafka.server.FinalizedFeatureChangeListener$ChangeNotificationProcessorThread)
[2021-12-01 20:18:22,729] INFO Feature ZK node at path: /feature does not exist (kafka.server.FinalizedFeatureChangeListener)
[2021-12-01 20:18:22,729] INFO Cleared cache (kafka.server.FinalizedFeatureCache)
[2021-12-01 20:18:22,844] INFO Cluster ID = j7KyANF-S2OuLlr_IYyqDw (kafka.server.KafkaServer)
[2021-12-01 20:18:22,846] WARN No meta.properties file under dir C:\tmp\kafka-logs\meta.properties (kafka.server.BrokerMetadataCheckpoint)
[2021-12-01 20:18:22,876] INFO KafkaConfig values:
...
[2021-12-01 20:18:22,918] INFO [ThrottledChannelReaper-Fetch]: Starting (kafka.server.ClientQuotaManager$ThrottledChannelReaper)
[2021-12-01 20:18:22,919] INFO [ThrottledChannelReaper-Produce]: Starting (kafka.server.ClientQuotaManager$ThrottledChannelReaper)
[2021-12-01 20:18:22,919] INFO [ThrottledChannelReaper-Request]: Starting (kafka.server.ClientQuotaManager$ThrottledChannelReaper)
[2021-12-01 20:18:22,920] INFO [ThrottledChannelReaper-ControllerMutation]: Starting (kafka.server.ClientQuotaManager$ThrottledChannelReaper)
[2021-12-01 20:18:22,931] INFO Log directory C:\tmp\kafka-logs not found, creating it. (kafka.log.LogManager)
[2021-12-01 20:18:22,933] ERROR Failed to create or validate data directory C:\tmp\kafka-logs (kafka.server.LogDirFailureChannel)
java.nio.file.AccessDeniedException: C:\tmp
at sun.nio.fs.WindowsException.translateToIOException(WindowsException.java:83)
at sun.nio.fs.WindowsException.rethrowAsIOException(WindowsException.java:97)
at sun.nio.fs.WindowsException.rethrowAsIOException(WindowsException.java:102)
at sun.nio.fs.WindowsFileSystemProvider.newFileChannel(WindowsFileSystemProvider.java:115)
at java.nio.channels.FileChannel.open(FileChannel.java:287)
at java.nio.channels.FileChannel.open(FileChannel.java:335)
at org.apache.kafka.common.utils.Utils.flushDir(Utils.java:953)
at kafka.log.LogManager.$anonfun$createAndValidateLogDirs$1(LogManager.scala:159)
at scala.collection.IterableOnceOps.foreach(IterableOnce.scala:563)
at scala.collection.IterableOnceOps.foreach$(IterableOnce.scala:561)
at scala.collection.AbstractIterable.foreach(Iterable.scala:919)
at kafka.log.LogManager.createAndValidateLogDirs(LogManager.scala:149)
at kafka.log.LogManager.<init>(LogManager.scala:88)
at kafka.log.LogManager$.apply(LogManager.scala:1283)
at kafka.server.KafkaServer.startup(KafkaServer.scala:254)
at kafka.Kafka$.main(Kafka.scala:109)
at kafka.Kafka.main(Kafka.scala)
[2021-12-01 20:18:22,936] ERROR Shutdown broker because none of the specified log dirs from C:\tmp\kafka-logs can be created or validated (kafka.log.LogManager)
I also tried running icacls "C:\tmp\" /q /c /t /grant Users:F. I am still getting the same error.
It turns out that 3.0.0 has issues with Windows. I downgraded to 2.8.1 and things worked fine. I hate Windows.
I'm using Confluent Community 6.0.1. Three nodes Kafka cluster:
devKafka04: Kafka Broker1, Zookeeper 1
devKafka05: Kafka Broker2, Zookeeper 2
devKafka06: Kafka Broker3, Zookeeper 3
The SSL encryption is already working well on the Kafka Brokers.
I'd like to add SASL to enable mutual authentication between Kafka and Zookeeper.
I was following the Confluent document:
https://docs.confluent.io/platform/current/kafka/incremental-security-upgrade.html#adding-security-to-a-running-zk-cluster
[Updates] After I applied the changes, Zookeeper could not start on the secureclientPort. That's why the Kafka broker couldn't start. Here are the error log and docker compose configurations.
I'm wondering if there's something with the confluent zookeeper image.
Please help me out. Thanks.
$ sudo docker logs zookeeper
===> User
uid=1000(appuser) gid=1000(appuser) groups=1000(appuser)
===> Configuring ...
===> Running preflight checks ...
===> Check if /var/lib/zookeeper/data is writable ...
===> Check if /var/lib/zookeeper/log is writable ...
===> Launching ...
===> Printing /var/lib/zookeeper/data/myid
1===> Launching zookeeper ...
[2021-03-24 19:03:08,857] INFO Reading configuration from: /etc/kafka/zookeeper.properties (org.apache.zookeeper.server.quorum.QuorumPeerConfig)
[2021-03-24 19:03:08,862] INFO clientPortAddress is 0.0.0.0:2181 (org.apache.zookeeper.server.quorum.QuorumPeerConfig)
[2021-03-24 19:03:08,862] INFO secureClientPort is not set (org.apache.zookeeper.server.quorum.QuorumPeerConfig)
[2021-03-24 19:03:08,876] INFO autopurge.snapRetainCount set to 3 (org.apache.zookeeper.server.DatadirCleanupManager)
[2021-03-24 19:03:08,876] INFO autopurge.purgeInterval set to 0 (org.apache.zookeeper.server.DatadirCleanupManager)
[2021-03-24 19:03:08,876] INFO Purge task is not scheduled. (org.apache.zookeeper.server.DatadirCleanupManager)
[2021-03-24 19:03:08,880] INFO Log4j 1.2 jmx support found and enabled. (org.apache.zookeeper.jmx.ManagedUtil)
[2021-03-24 19:03:08,904] INFO Starting quorum peer (org.apache.zookeeper.server.quorum.QuorumPeerMain)
[2021-03-24 19:03:08,909] INFO Using org.apache.zookeeper.server.NIOServerCnxnFactory as server connection factory (org.apache.zookeeper.server.ServerCnxnFactory)
[2021-03-24 19:03:08,917] INFO Setting -D jdk.tls.rejectClientInitiatedRenegotiation=true to disable client-initiated TLS renegotiation (org.apache.zookeeper.common.X509Util)
[2021-03-24 19:03:08,953] INFO Server successfully logged in. (org.apache.zookeeper.Login)
[2021-03-24 19:03:08,957] INFO Configuring NIO connection handler with 10s sessionless connection timeout, 1 selector thread(s), 8 worker threads, and 64 kB direct buffers. (org.apache.zookeeper.server.NIOServerCnxnFactory)
[2021-03-24 19:03:08,961] INFO binding to port 0.0.0.0/0.0.0.0:2181 (org.apache.zookeeper.server.NIOServerCnxnFactory)
[2021-03-24 19:03:08,986] INFO Logging initialized #929ms to org.eclipse.jetty.util.log.Slf4jLog (org.eclipse.jetty.util.log)
[2021-03-24 19:03:09,081] WARN o.e.j.s.ServletContextHandler#6c2c1385{/,null,UNAVAILABLE} contextPath ends with /* (org.eclipse.jetty.server.handler.ContextHandler)
[2021-03-24 19:03:09,082] WARN Empty contextPath (org.eclipse.jetty.server.handler.ContextHandler)
[2021-03-24 19:03:09,097] INFO zookeeper.snapshot.trust.empty : false (org.apache.zookeeper.server.persistence.FileTxnSnapLog)
[2021-03-24 19:03:09,102] INFO Local sessions disabled (org.apache.zookeeper.server.quorum.QuorumPeer)
[2021-03-24 19:03:09,102] INFO Local session upgrading disabled (org.apache.zookeeper.server.quorum.QuorumPeer)
[2021-03-24 19:03:09,102] INFO tickTime set to 3000 (org.apache.zookeeper.server.quorum.QuorumPeer)
[2021-03-24 19:03:09,102] INFO minSessionTimeout set to 6000 (org.apache.zookeeper.server.quorum.QuorumPeer)
[2021-03-24 19:03:09,102] INFO maxSessionTimeout set to 60000 (org.apache.zookeeper.server.quorum.QuorumPeer)
[2021-03-24 19:03:09,102] INFO initLimit set to 10 (org.apache.zookeeper.server.quorum.QuorumPeer)
[2021-03-24 19:03:09,115] INFO zookeeper.snapshotSizeFactor = 0.33 (org.apache.zookeeper.server.ZKDatabase)
[2021-03-24 19:03:09,116] INFO Using insecure (non-TLS) quorum communication (org.apache.zookeeper.server.quorum.QuorumPeer)
[2021-03-24 19:03:09,117] INFO Port unification disabled (org.apache.zookeeper.server.quorum.QuorumPeer)
[2021-03-24 19:03:09,117] INFO QuorumPeer communication is not secured! (SASL auth disabled) (org.apache.zookeeper.server.quorum.QuorumPeer)
[2021-03-24 19:03:09,117] INFO quorum.cnxn.threads.size set to 20 (org.apache.zookeeper.server.quorum.QuorumPeer)
[2021-03-24 19:03:09,118] INFO Reading snapshot /var/lib/zookeeper/data/version-2/snapshot.a00000000 (org.apache.zookeeper.server.persistence.FileSnap)
[2021-03-24 19:03:09,213] INFO jetty-9.4.24.v20191120; built: 2019-11-20T21:37:49.771Z; git: 363d5f2df3a8a28de40604320230664b9c793c16; jvm 11.0.9.1+1-LTS (org.eclipse.jetty.server.Server)
[2021-03-24 19:03:09,261] INFO DefaultSessionIdManager workerName=node0 (org.eclipse.jetty.server.session)
[2021-03-24 19:03:09,261] INFO No SessionScavenger set, using defaults (org.eclipse.jetty.server.session)
[2021-03-24 19:03:09,263] INFO node0 Scavenging every 660000ms (org.eclipse.jetty.server.session)
[2021-03-24 19:03:09,272] INFO Started o.e.j.s.ServletContextHandler#6c2c1385{/,null,AVAILABLE} (org.eclipse.jetty.server.handler.ContextHandler)
[2021-03-24 19:03:09,281] INFO Started ServerConnector#6d07a63d{HTTP/1.1,[http/1.1]}{0.0.0.0:8080} (org.eclipse.jetty.server.AbstractConnector)
[2021-03-24 19:03:09,281] INFO Started #1224ms (org.eclipse.jetty.server.Server)
[2021-03-24 19:03:09,281] INFO Started AdminServer on address 0.0.0.0, port 8080 and command URL /commands (org.apache.zookeeper.server.admin.JettyAdminServer)
[2021-03-24 19:03:09,288] INFO Election port bind maximum retries is 3 (org.apache.zookeeper.server.quorum.QuorumCnxManager)
[2021-03-24 19:03:09,290] INFO 1 is accepting connections now, my election bind port: devkafka04/172.16.87.141:3888 (org.apache.zookeeper.server.quorum.QuorumCnxManager)
[2021-03-24 19:03:09,301] INFO LOOKING (org.apache.zookeeper.server.quorum.QuorumPeer)
[2021-03-24 19:03:09,303] INFO New election. My id = 1, proposed zxid=0x1600000030 (org.apache.zookeeper.server.quorum.FastLeaderElection)
[2021-03-24 19:03:09,308] INFO Notification: 2 (message format version), 1 (n.leader), 0x1600000030 (n.zxid), 0x1 (n.round), LOOKING (n.state), 1 (n.sid), 0x16 (n.peerEPoch), LOOKING (my state)0 (n.config version) (org.apache.zookeeper.server.quorum.FastLeaderElection)
[2021-03-24 19:03:09,310] INFO Have smaller server identifier, so dropping the connection: (myId:1 --> sid:3) (org.apache.zookeeper.server.quorum.QuorumCnxManager)
[2021-03-24 19:03:09,312] INFO Received connection request from /172.16.87.143:53340 (org.apache.zookeeper.server.quorum.QuorumCnxManager)
[2021-03-24 19:03:09,315] INFO Have smaller server identifier, so dropping the connection: (myId:1 --> sid:2) (org.apache.zookeeper.server.quorum.QuorumCnxManager)
[2021-03-24 19:03:09,316] INFO Notification: 2 (message format version), 2 (n.leader), 0x150000002b (n.zxid), 0xa (n.round), FOLLOWING (n.state), 3 (n.sid), 0x16 (n.peerEPoch), LOOKING (my state)0 (n.config version) (org.apache.zookeeper.server.quorum.FastLeaderElection)
[2021-03-24 19:03:09,317] INFO Received connection request from /172.16.87.142:51704 (org.apache.zookeeper.server.quorum.QuorumCnxManager)
[2021-03-24 19:03:09,319] INFO Notification: 2 (message format version), 2 (n.leader), 0x150000002b (n.zxid), 0xa (n.round), LEADING (n.state), 2 (n.sid), 0x16 (n.peerEPoch), LOOKING (my state)0 (n.config version) (org.apache.zookeeper.server.quorum.FastLeaderElection)
[2021-03-24 19:03:09,320] INFO Notification: 2 (message format version), 2 (n.leader), 0x150000002b (n.zxid), 0xa (n.round), FOLLOWING (n.state), 3 (n.sid), 0x16 (n.peerEPoch), LOOKING (my state)0 (n.config version) (org.apache.zookeeper.server.quorum.FastLeaderElection)
[2021-03-24 19:03:09,320] INFO FOLLOWING (org.apache.zookeeper.server.quorum.QuorumPeer)
[2021-03-24 19:03:09,323] INFO Notification: 2 (message format version), 2 (n.leader), 0x150000002b (n.zxid), 0xa (n.round), LEADING (n.state), 2 (n.sid), 0x16 (n.peerEPoch), FOLLOWING (my state)0 (n.config version) (org.apache.zookeeper.server.quorum.FastLeaderElection)
[2021-03-24 19:03:09,330] INFO TCP NoDelay set to: true (org.apache.zookeeper.server.quorum.Learner)
[2021-03-24 19:03:09,336] INFO Server environment:zookeeper.version=3.5.8-f439ca583e70862c3068a1f2a7d4d068eec33315, built on 05/04/2020 15:53 GMT (org.apache.zookeeper.server.ZooKeeperServer)
[2021-03-24 19:03:09,336] INFO Server environment:host.name=devkafka04 (org.apache.zookeeper.server.ZooKeeperServer)
[2021-03-24 19:03:09,336] INFO Server environment:java.version=11.0.9.1 (org.apache.zookeeper.server.ZooKeeperServer)
[2021-03-24 19:03:09,336] INFO Server environment:java.vendor=Azul Systems, Inc. (org.apache.zookeeper.server.ZooKeeperServer)
[2021-03-24 19:03:09,336] INFO Server environment:java.home=/usr/lib/jvm/zulu11-ca (org.apache.zookeeper.server.ZooKeeperServer)
[2021-03-24 19:03:09,336] INFO Server environment:java.class.path=/usr/bin/../share/java/kafka/activation-1.1.1.jar:/usr/bin/../share/java/kafka/aopalliance-repackaged-2.6.1.jar:/usr/bin/../share/java/kafka/argparse4j-0.7.0.jar:/usr/bin/../share/java/kafka/audience-annotations-0.5.0.jar:/usr/bin/../share/java/kafka/commons-cli-1.4.jar:/usr/bin/../share/java/kafka/commons-lang3-3.8.1.jar:/usr/bin/../share/java/kafka/confluent-log4j-1.2.17-cp2.jar:/usr/bin/../share/java/kafka/connect-api-6.0.1-ccs.jar:/usr/bin/../share/java/kafka/connect-basic-auth-extension-6.0.1-ccs.jar:/usr/bin/../share/java/kafka/connect-file-6.0.1-ccs.jar:/usr/bin/../share/java/kafka/connect-json-6.0.1-ccs.jar:/usr/bin/../share/java/kafka/connect-mirror-6.0.1-ccs.jar:/usr/bin/../share/java/kafka/connect-mirror-client-6.0.1-ccs.jar:/usr/bin/../share/java/kafka/connect-runtime-6.0.1-ccs.jar:/usr/bin/../share/java/kafka/connect-transforms-6.0.1-ccs.jar:/usr/bin/../share/java/kafka/hk2-api-2.6.1.jar:/usr/bin/../share/java/kafka/hk2-locator-2.6.1.jar:/usr/bin/../share/java/kafka/hk2-utils-2.6.1.jar:/usr/bin/../share/java/kafka/jackson-annotations-2.10.5.jar:/usr/bin/../share/java/kafka/jackson-core-2.10.5.jar:/usr/bin/../share/java/kafka/jackson-databind-2.10.5.jar:/usr/bin/../share/java/kafka/jackson-dataformat-csv-2.10.5.jar:/usr/bin/../share/java/kafka/jackson-datatype-jdk8-2.10.5.jar:/usr/bin/../share/java/kafka/jackson-jaxrs-base-2.10.5.jar:/usr/bin/../share/java/kafka/jackson-jaxrs-json-provider-2.10.5.jar:/usr/bin/../share/java/kafka/jackson-module-jaxb-annotations-2.10.5.jar:/usr/bin/../share/java/kafka/jackson-module-paranamer-2.10.5.jar:/usr/bin/../share/java/kafka/jackson-module-scala_2.13-2.10.5.jar:/usr/bin/../share/java/kafka/jakarta.activation-api-1.2.1.jar:/usr/bin/../share/java/kafka/jakarta.annotation-api-1.3.5.jar:/usr/bin/../share/java/kafka/jakarta.inject-2.6.1.jar:/usr/bin/../share/java/kafka/jakarta.validation-api-2.0.2.jar:/usr/bin/../share/java/kafka/jakarta.ws.rs-api-2.1.6.jar:/usr/bin/../share/java/kafka/jakarta.xml.bind-api-2.3.2.jar:/usr/bin/../share/java/kafka/javassist-3.25.0-GA.jar:/usr/bin/../share/java/kafka/javassist-3.26.0-GA.jar:/usr/bin/../share/java/kafka/javax.servlet-api-3.1.0.jar:/usr/bin/../share/java/kafka/javax.ws.rs-api-2.1.1.jar:/usr/bin/../share/java/kafka/jaxb-api-2.3.0.jar:/usr/bin/../share/java/kafka/jersey-client-2.30.jar:/usr/bin/../share/java/kafka/jersey-common-2.30.jar:/usr/bin/../share/java/kafka/jersey-container-servlet-2.30.jar:/usr/bin/../share/java/kafka/jersey-container-servlet-core-2.30.jar:/usr/bin/../share/java/kafka/jersey-hk2-2.30.jar:/usr/bin/../share/java/kafka/jersey-media-jaxb-2.30.jar:/usr/bin/../share/java/kafka/jersey-server-2.30.jar:/usr/bin/../share/java/kafka/jetty-client-9.4.24.v20191120.jar:/usr/bin/../share/java/kafka/jetty-continuation-9.4.24.v20191120.jar:/usr/bin/../share/java/kafka/jetty-http-9.4.24.v20191120.jar:/usr/bin/../share/java/kafka/jetty-io-9.4.24.v20191120.jar:/usr/bin/../share/java/kafka/jetty-security-9.4.24.v20191120.jar:/usr/bin/../share/java/kafka/jetty-server-9.4.24.v20191120.jar:/usr/bin/../share/java/kafka/jetty-servlet-9.4.24.v20191120.jar:/usr/bin/../share/java/kafka/jetty-servlets-9.4.24.v20191120.jar:/usr/bin/../share/java/kafka/jetty-util-9.4.24.v20191120.jar:/usr/bin/../share/java/kafka/jopt-simple-5.0.4.jar:/usr/bin/../share/java/kafka/kafka-clients-6.0.1-ccs.jar:/usr/bin/../share/java/kafka/kafka-log4j-appender-6.0.1-ccs.jar:/usr/bin/../share/java/kafka/kafka-streams-6.0.1-ccs.jar:/usr/bin/../share/java/kafka/kafka-streams-examples-6.0.1-ccs.jar:/usr/bin/../share/java/kafka/kafka-streams-scala_2.13-6.0.1-ccs.jar:/usr/bin/../share/java/kafka/kafka-streams-test-utils-6.0.1-ccs.jar:/usr/bin/../share/java/kafka/kafka-tools-6.0.1-ccs.jar:/usr/bin/../share/java/kafka/kafka.jar:/usr/bin/../share/java/kafka/kafka_2.13-6.0.1-ccs-javadoc.jar:/usr/bin/../share/java/kafka/kafka_2.13-6.0.1-ccs-scaladoc.jar:/usr/bin/../share/java/kafka/kafka_2.13-6.0.1-ccs-sources.jar:/usr/bin/../share/java/kafka/kafka_2.13-6.0.1-ccs-test-sources.jar:/usr/bin/../share/java/kafka/kafka_2.13-6.0.1-ccs-test.jar:/usr/bin/../share/java/kafka/kafka_2.13-6.0.1-ccs.jar:/usr/bin/../share/java/kafka/lz4-java-1.7.1.jar:/usr/bin/../share/java/kafka/maven-artifact-3.6.3.jar:/usr/bin/../share/java/kafka/metrics-core-2.2.0.jar:/usr/bin/../share/java/kafka/netty-buffer-4.1.50.Final.jar:/usr/bin/../share/java/kafka/netty-codec-4.1.50.Final.jar:/usr/bin/../share/java/kafka/netty-common-4.1.50.Final.jar:/usr/bin/../share/java/kafka/netty-handler-4.1.50.Final.jar:/usr/bin/../share/java/kafka/netty-resolver-4.1.50.Final.jar:/usr/bin/../share/java/kafka/netty-transport-4.1.50.Final.jar:/usr/bin/../share/java/kafka/netty-transport-native-epoll-4.1.50.Final.jar:/usr/bin/../share/java/kafka/netty-transport-native-unix-common-4.1.50.Final.jar:/usr/bin/../share/java/kafka/osgi-resource-locator-1.0.3.jar:/usr/bin/../share/java/kafka/paranamer-2.8.jar:/usr/bin/../share/java/kafka/plexus-utils-3.2.1.jar:/usr/bin/../share/java/kafka/reflections-0.9.12.jar:/usr/bin/../share/java/kafka/rocksdbjni-5.18.4.jar:/usr/bin/../share/java/kafka/scala-collection-compat_2.13-2.1.6.jar:/usr/bin/../share/java/kafka/scala-java8-compat_2.13-0.9.1.jar:/usr/bin/../share/java/kafka/scala-library-2.13.2.jar:/usr/bin/../share/java/kafka/slf4j-api-1.7.30.jar:/usr/bin/../share/java/kafka/scala-logging_2.13-3.9.2.jar:/usr/bin/../share/java/kafka/scala-reflect-2.13.2.jar:/usr/bin/../share/java/kafka/slf4j-log4j12-1.7.30.jar:/usr/bin/../share/java/kafka/snappy-java-1.1.7.3.jar:/usr/bin/../share/java/kafka/zookeeper-3.5.8.jar:/usr/bin/../share/java/kafka/zookeeper-jute-3.5.8.jar:/usr/bin/../share/java/kafka/zstd-jni-1.4.4-7.jar:/usr/bin/../share/java/confluent-telemetry/* (org.apache.zookeeper.server.ZooKeeperServer)
[2021-03-24 19:03:09,336] INFO Server environment:java.library.path=/usr/java/packages/lib:/usr/lib64:/lib64:/lib:/usr/lib (org.apache.zookeeper.server.ZooKeeperServer)
[2021-03-24 19:03:09,336] INFO Server environment:java.io.tmpdir=/tmp (org.apache.zookeeper.server.ZooKeeperServer)
[2021-03-24 19:03:09,336] INFO Server environment:java.compiler=<NA> (org.apache.zookeeper.server.ZooKeeperServer)
[2021-03-24 19:03:09,337] INFO Server environment:os.name=Linux (org.apache.zookeeper.server.ZooKeeperServer)
[2021-03-24 19:03:09,337] INFO Server environment:os.arch=amd64 (org.apache.zookeeper.server.ZooKeeperServer)
[2021-03-24 19:03:09,337] INFO Server environment:os.version=3.10.0-1160.21.1.el7.x86_64 (org.apache.zookeeper.server.ZooKeeperServer)
[2021-03-24 19:03:09,337] INFO Server environment:user.name=appuser (org.apache.zookeeper.server.ZooKeeperServer)
[2021-03-24 19:03:09,337] INFO Server environment:user.home=/home/appuser (org.apache.zookeeper.server.ZooKeeperServer)
[2021-03-24 19:03:09,337] INFO Server environment:user.dir=/home/appuser (org.apache.zookeeper.server.ZooKeeperServer)
[2021-03-24 19:03:09,337] INFO Server environment:os.memory.free=498MB (org.apache.zookeeper.server.ZooKeeperServer)
[2021-03-24 19:03:09,337] INFO Server environment:os.memory.max=512MB (org.apache.zookeeper.server.ZooKeeperServer)
[2021-03-24 19:03:09,337] INFO Server environment:os.memory.total=512MB (org.apache.zookeeper.server.ZooKeeperServer)
[2021-03-24 19:03:09,338] INFO minSessionTimeout set to 6000 (org.apache.zookeeper.server.ZooKeeperServer)
[2021-03-24 19:03:09,339] INFO maxSessionTimeout set to 60000 (org.apache.zookeeper.server.ZooKeeperServer)
[2021-03-24 19:03:09,339] INFO Created server with tickTime 3000 minSessionTimeout 6000 maxSessionTimeout 60000 datadir /var/lib/zookeeper/log/version-2 snapdir /var/lib/zookeeper/data/version-2 (org.apache.zookeeper.server.ZooKeeperServer)
[2021-03-24 19:03:09,339] INFO FOLLOWING - LEADER ELECTION TOOK - 18 MS (org.apache.zookeeper.server.quorum.Learner)
[2021-03-24 19:03:09,345] INFO Getting a diff from the leader 0x1600000030 (org.apache.zookeeper.server.quorum.Learner)
[2021-03-24 19:03:09,350] INFO Learner received NEWLEADER message (org.apache.zookeeper.server.quorum.Learner)
[2021-03-24 19:03:09,363] INFO Learner received UPTODATE message (org.apache.zookeeper.server.quorum.Learner)
[2021-03-24 19:03:09,367] INFO Configuring CommitProcessor with 4 worker threads. (org.apache.zookeeper.server.quorum.CommitProcessor)
$ sudo docker logs kafka
===> User
uid=1000(appuser) gid=1000(appuser) groups=1000(appuser)
===> Configuring ...
SSL is enabled.
SASL is enabled.
===> Running preflight checks ...
===> Check if /var/lib/kafka/data is writable ...
===> Skipping Zookeeper health check for SSL connections...
===> Launching ...
===> Launching kafka ...
[2021-03-23 21:43:43,453] INFO Registered kafka:type=kafka.Log4jController MBean (kafka.utils.Log4jControllerRegistration$)
[2021-03-23 21:43:43,838] INFO Setting -D jdk.tls.rejectClientInitiatedRenegotiation=true to disable client-initiated TLS renegotiation (org.apache.zookeeper.common.X509Util)
[2021-03-23 21:43:43,900] INFO Registered signal handlers for TERM, INT, HUP (org.apache.kafka.common.utils.LoggingSignalHandler)
[2021-03-23 21:43:43,904] INFO starting (kafka.server.KafkaServer)
[2021-03-23 21:43:43,905] INFO Connecting to zookeeper on devkafka04:2182,devkafka05:2182,devkafka06:2182 (kafka.server.KafkaServer)
[2021-03-23 21:43:43,927] INFO [ZooKeeperClient Kafka server] Initializing a new session to devkafka04:2182,devkafka05:2182,devkafka06:2182. (kafka.zookeeper.ZooKeeperClient)
[2021-03-23 21:43:43,934] INFO Client environment:zookeeper.version=3.5.8-f439ca583e70862c3068a1f2a7d4d068eec33315, built on 05/04/2020 15:53 GMT (org.apache.zookeeper.ZooKeeper)
[2021-03-23 21:43:43,934] INFO Client environment:host.name=devkafka04 (org.apache.zookeeper.ZooKeeper)
[2021-03-23 21:43:43,934] INFO Client environment:java.version=11.0.9.1 (org.apache.zookeeper.ZooKeeper)
[2021-03-23 21:43:43,934] INFO Client environment:java.vendor=Azul Systems, Inc. (org.apache.zookeeper.ZooKeeper)
------ Repeating lines removed ---------
'Client' (org.apache.zookeeper.ClientCnxn)
[2021-03-23 21:43:59,947] INFO Socket error occurred: devkafka05/172.16.87.142:2182: Connection refused (org.apache.zookeeper.ClientCnxn)
[2021-03-23 21:44:01,048] INFO Client successfully logged in. (org.apache.zookeeper.Login)
[2021-03-23 21:44:01,048] INFO Client will use DIGEST-MD5 as SASL mechanism. (org.apache.zookeeper.client.ZooKeeperSaslClient)
[2021-03-23 21:44:01,048] INFO Opening socket connection to server devkafka04/172.16.87.141:2182. Will attempt to SASL-authenticate using Login Context section 'Client' (org.apache.zookeeper.ClientCnxn)
[2021-03-23 21:44:01,049] INFO Socket error occurred: devkafka04/172.16.87.141:2182: Connection refused (org.apache.zookeeper.ClientCnxn)
[2021-03-23 21:44:01,150] INFO Client successfully logged in. (org.apache.zookeeper.Login)
[2021-03-23 21:44:01,150] INFO Client will use DIGEST-MD5 as SASL mechanism. (org.apache.zookeeper.client.ZooKeeperSaslClient)
[2021-03-23 21:44:01,150] INFO Opening socket connection to server devkafka06/172.16.87.143:2182. Will attempt to SASL-authenticate using Login Context section 'Client' (org.apache.zookeeper.ClientCnxn)
[2021-03-23 21:44:01,153] INFO Socket error occurred: devkafka06/172.16.87.143:2182: Connection refused (org.apache.zookeeper.ClientCnxn)
[2021-03-23 21:44:01,254] INFO Client successfully logged in. (org.apache.zookeeper.Login)
[2021-03-23 21:44:01,254] INFO Client will use DIGEST-MD5 as SASL mechanism. (org.apache.zookeeper.client.ZooKeeperSaslClient)
[2021-03-23 21:44:01,254] INFO Opening socket connection to server devkafka05/172.16.87.142:2182. Will attempt to SASL-authenticate using Login Context section 'Client' (org.apache.zookeeper.ClientCnxn)
[2021-03-23 21:44:01,255] INFO Socket error occurred: devkafka05/172.16.87.142:2182: Connection refused (org.apache.zookeeper.ClientCnxn)
[2021-03-23 21:44:01,952] INFO [ZooKeeperClient Kafka server] Closing. (kafka.zookeeper.ZooKeeperClient)
[2021-03-23 21:44:02,356] INFO Client successfully logged in. (org.apache.zookeeper.Login)
[2021-03-23 21:44:02,357] INFO Client will use DIGEST-MD5 as SASL mechanism. (org.apache.zookeeper.client.ZooKeeperSaslClient)
[2021-03-23 21:44:02,357] INFO Opening socket connection to server devkafka04/172.16.87.141:2182. Will attempt to SASL-authenticate using Login Context section 'Client' (org.apache.zookeeper.ClientCnxn)
[2021-03-23 21:44:02,462] INFO Session: 0x0 closed (org.apache.zookeeper.ZooKeeper)
[2021-03-23 21:44:02,463] INFO EventThread shut down for session: 0x0 (org.apache.zookeeper.ClientCnxn)
[2021-03-23 21:44:02,465] INFO [ZooKeeperClient Kafka server] Closed. (kafka.zookeeper.ZooKeeperClient)
[2021-03-23 21:44:02,469] ERROR Fatal error during KafkaServer startup. Prepare to shutdown (kafka.server.KafkaServer)
kafka.zookeeper.ZooKeeperClientTimeoutException: Timed out waiting for connection while in state: CONNECTING
at kafka.zookeeper.ZooKeeperClient.waitUntilConnected(ZooKeeperClient.scala:262)
at kafka.zookeeper.ZooKeeperClient.<init>(ZooKeeperClient.scala:119)
at kafka.zk.KafkaZkClient$.apply(KafkaZkClient.scala:1865)
at kafka.server.KafkaServer.createZkClient$1(KafkaServer.scala:419)
at kafka.server.KafkaServer.initZkClient(KafkaServer.scala:444)
at kafka.server.KafkaServer.startup(KafkaServer.scala:222)
at kafka.server.KafkaServerStartable.startup(KafkaServerStartable.scala:44)
at kafka.Kafka$.main(Kafka.scala:82)
at kafka.Kafka.main(Kafka.scala)
[2021-03-23 21:44:02,471] INFO shutting down (kafka.server.KafkaServer)
[2021-03-23 21:44:02,478] INFO shut down completed (kafka.server.KafkaServer)
[2021-03-23 21:44:02,478] ERROR Exiting Kafka. (kafka.server.KafkaServerStartable)
[2021-03-23 21:44:02,479] INFO shutting down (kafka.server.KafkaServer)
$ sudo cat kafka-docker-compose.yml
version: '3'
services:
kafka:
image: confluentinc/cp-kafka:6.0.1
container_name: kafka
network_mode: host
restart: always
ports:
- "9092:9092"
- "9093:9093"
- "9094:9094"
- "49998:49998"
- "49999:49999"
environment:
KAFKA_BROKER_ID: 1
KAFKA_ZOOKEEPER_CONNECT: 'devkafka04:2182,devkafka05:2182,devkafka06:2182'
KAFKA_ZOOKEEPER_SSL_CLIENT_ENABLE: 'true'
KAFKA_ZOOKEEPER_CLIENTCNXNSOCKET: org.apache.zookeeper.ClientCnxnSocketNetty
KAFKA_ZOOKEEPER_SSL_TRUSTSTORE_FILENAME: kafka.server.truststore.jks
KAFKA_ZOOKEEPER_SSL_TRUSTSTORE_CREDENTIALS: creds
KAFKA_ZOOKEEPER_SET_ACL: 'true'
KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://devkafka04:9092,SSL://devkafka04:9093,SASL_SSL://devkafka04:9094
KAFKA_LISTENERS: PLAINTEXT://devkafka04:9092,SSL://devkafka04:9093,SASL_SSL://devkafka04:9094
KAFKA_SASL_ENABLED_MECHANISMS: DIGEST-MD5
KAFKA_SECURITY_INTER_BROKER_PROTOCOL: SSL
KAFKA_SSL_CLIENT_AUTH: requested
KAFKA_SSL_TRUSTSTORE_FILENAME: kafka.server.truststore.jks
KAFKA_SSL_TRUSTSTORE_CREDENTIALS: creds
KAFKA_SSL_KEYSTORE_FILENAME: devkafka04.server.keystore.jks
KAFKA_SSL_KEYSTORE_CREDENTIALS: creds
KAFKA_SSL_KEY_CREDENTIALS: creds
KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 3
KAFKA_GROUP_INITIAL_REBALANCE_DELAY_MS: 0
KAFKA_AUTO_CREATE_TOPICS_ENABLE: "false"
KAFKA_OPTS: -Djava.security.auth.login.config=/etc/kafka/jmx/kafka_server_jaas.conf -Djava.rmi.server.hostname=devkafka04 -Dcom.sun.management.jmxremote.local.only=false -Dcom.sun.management.jmxremote.rmi.port=49998 -Dcom.sun.management.jmxremote.port=49998 -Dcom.sun.management.jmxremote=true -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -javaagent:/etc/kafka/jmx/jmx_prometheus_javaagent-0.14.0.jar=49999:/etc/kafka/jmx/kafka-2_0_0.yml
CONFLUENT_SUPPORT_METRICS_ENABLE: "false"
volumes:
- /media/kafka/data:/var/lib/kafka/data
- /media/kafka/secrets:/etc/kafka/secrets
- /usr/local/src/kafka/jmx:/etc/kafka/jmx
$ sudo cat jmx/kafka_server_jaas.conf
KafkaServer {
org.apache.kafka.common.security.plain.PlainLoginModule required
username="kafkabroker"
password="kafkabroker-secret"
user_kafkabroker="kafkabroker-secret"
user_kafka-broker-metric-reporter="kafkabroker-metric-reporter-secret"
user_client="client-secret";
};
Client {
org.apache.zookeeper.server.auth.DigestLoginModule required
username="kafka"
password="kafka-secret";
};
$ sudo cat zookeeper-docker-compose.yml
version: '3'
services:
zookeeper:
image: confluentinc/cp-zookeeper:6.0.1
container_name: zookeeper
network_mode: host
restart: always
ports:
- "2181:2181"
- "2182:2182"
- "2888:2888"
- "3888:3888"
- "39998:39998"
- "39999:39999"
environment:
ZOOKEEPER_SERVER_ID: 1
ZOOKEEPER_CLIENT_PORT: 2181
ZOOKEEPER_SERVERS: devkafka04:2888:3888;devkafka05:2888:3888;devkafka06:2888:3888
ZOOKEEPER_AUTHPROVIDER_SASL: org.apache.zookeeper.server.auth.SASLAuthenticationProvider
ZOOKEEPER_AUTHPROVIDER_x509: org.apache.zookeeper.server.auth.X509AuthenticationProvider
ZOOKEEPER_SECURECLIENTPORT: 2182
ZOOKEEPER_SERVERCNXNFACTORY: org.apache.zookeeper.server.NettyServerCnxnFactory
ZOOKEEPER_SSL_TRUSTSTORE_FILENAME: kafka.server.truststore.jks
ZOOKEEPER_SSL_TRUSTSTORE_CREDENTIALS: creds
ZOOKEEPER_SSL_KEYSTORE_FILENAME: devkafka05.server.keystore.jks
ZOOKEEPER_SSL_KEYSTORE_CREDENTIALS: creds
ZOOKEEPER_SSL_KEY_CREDENTIALS: creds
ZOOKEEPER_SSL_CLIENTAUTH: none
KAFKA_OPTS: -Djava.security.auth.login.config=/etc/zookeeper/jmx/zookeeper_jaas.conf -Dzookeeper.4lw.commands.whitelist=* -Djava.rmi.server.hostname=devkafka04 -Dcom.sun.management.jmxremote.local.only=false -Dcom.sun.management.jmxremote.rmi.port=39998 -Dcom.sun.management.jmxremote.port=39998 -Dcom.sun.management.jmxremote=true -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -javaagent:/etc/zookeeper/jmx/jmx_prometheus_javaagent-0.14.0.jar=39999:/etc/zookeeper/jmx/jmx-zookeeper-prometheus.yaml
volumes:
- /media/zookeeper/data:/var/lib/zookeeper/data
- /media/zookeeper/log:/var/lib/zookeeper/log
- /media/zookeeper/secrets:/etc/zookeeper/secrets
- /usr/local/src/zookeeper/jmx:/etc/zookeeper/jmx
$ sudo cat jmx/zookeeper_jaas.conf
Server {
org.apache.zookeeper.server.auth.DigestLoginModule required
user_kafka="kafka-secret";
};
Try using KAFKA_ZOOKEEPER_CLIENT_CNXN_SOCKET instead of KAFKA_ZOOKEEPER_CLIENTCNXNSOCKET.
I'm trying to figure out how to successfully reach the end of this tutorial, for the installation and first running of Apache-Kafka : How To Install Apache Kafka on Ubuntu 18.04 LTS
After killing the previously running zookeeper instance, I tried, again!, to start the zookeeper server again, but when the process reaches the phase of port binding, it stops forever..: KafkaActivationFailed
marco#PC:/usr/share/zookeeper$ sudo lsof -i :2181
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
java 4545 zookeeper 32u IPv6 43042 0t0 TCP *:2181 (LISTEN)
marco#PC:/usr/share/zookeeper$ sudo kill 4545
marco#PC:/usr/local/kafka$ bin/zookeeper-server-start.sh
config/zookeeper.properties
[2018-09-04 12:18:14,608] INFO Reading configuration from:
config/zookeeper.properties
(org.apache.zookeeper.server.quorum.QuorumPeerConfig)
[2018-09-04 12:18:14,609] INFO autopurge.snapRetainCount set to 3
(org.apache.zookeeper.server.DatadirCleanupManager)
[2018-09-04 12:18:14,609] INFO autopurge.purgeInterval set to 0
(org.apache.zookeeper.server.DatadirCleanupManager)
[2018-09-04 12:18:14,609] INFO Purge task is not scheduled.
(org.apache.zookeeper.server.DatadirCleanupManager)
[2018-09-04 12:18:14,610] WARN Either no config or no quorum defined in
config, running in standalone mode
(org.apache.zookeeper.server.quorum.QuorumPeerMain)
[2018-09-04 12:18:14,620] INFO Reading configuration from:
config/zookeeper.properties
(org.apache.zookeeper.server.quorum.QuorumPeerConfig)
[2018-09-04 12:18:14,620] INFO Starting server
(org.apache.zookeeper.server.ZooKeeperServerMain)
[2018-09-04 12:18:14,624] INFO Server
environment:zookeeper.version=3.4.13-2d71af4dbe22557fda74f9a9b4309b15a7487f03
built on 06/29/2018 00:39 GMT (org.apache.zookeeper.server.ZooKeeperServer)
[2018-09-04 12:18:14,624] INFO Server environment:host.name=PC.station
(org.apache.zookeeper.server.ZooKeeperServer)
[2018-09-04 12:18:14,625] INFO Server environment:java.version=1.8.0_181
(org.apache.zookeeper.server.ZooKeeperServer)
[2018-09-04 12:18:14,625] INFO Server environment:java.vendor=Oracle
Corporation (org.apache.zookeeper.server.ZooKeeperServer)
[2018-09-04 12:18:14,625] INFO Server environment:java.home=/usr/lib/jvm
/java-8-oracle/jre (org.apache.zookeeper.server.ZooKeeperServer)
[2018-09-04 12:18:14,625] INFO Server environment:java.class.path=/usr/local
/kafka/bin/../libs/activation-1.1.1.jar:/usr/local/kafka/bin/..
/libs/aopalliance-repackaged-2.5.0-b42.jar:/usr/local/kafka/bin/..
/libs/argparse4j-0.7.0.jar:/usr/local/kafka/bin/../libs/audience-annotations-
0.5.0.jar:/usr/local/kafka/bin/../libs/commons-lang3-3.5.jar:/usr/local/kafka
/bin/../libs/connect-api-2.0.0.jar:/usr/local/kafka/bin/../libs/connect-
basic-auth-extension-2.0.0.jar:/usr/local/kafka/bin/../libs/connect-
file-2.0.0.jar:/usr/local/kafka/bin/../libs/connect-json-2.0.0.jar:/usr/local
/kafka/bin/../libs/connect-runtime-2.0.0.jar:/usr/local/kafka/bin/..
/libs/connect-transforms-2.0.0.jar:/usr/local/kafka/bin/../libs/guava-
20.0.jar:/usr/local/kafka/bin/../libs/hk2-api-2.5.0-b42.jar:/usr/local/kafka
/bin/../libs/hk2-locator-2.5.0-b42.jar:/usr/local/kafka/bin/../libs/hk2-
utils-2.5.0-b42.jar:/usr/local/kafka/bin/../libs/jackson-annotations-
2.9.6.jar:/usr/local/kafka/bin/../libs/jackson-core-2.9.6.jar:/usr/local
/kafka/bin/../libs/jackson-databind-2.9.6.jar:/usr/local/kafka/bin/..
/libs/jackson-jaxrs-base-2.9.6.jar:/usr/local/kafka/bin/../libs/jackson-
jaxrs-json-provider-2.9.6.jar:/usr/local/kafka/bin/../libs/jackson-module-
jaxb-annotations-2.9.6.jar:/usr/local/kafka/bin/../libs/javassist-3.22.0-
CR2.jar:/usr/local/kafka/bin/../libs/javax.annotation-api-1.2.jar:/usr/local
/kafka/bin/../libs/javax.inject-1.jar:/usr/local/kafka/bin/..
/libs/javax.inject-2.5.0-b42.jar:/usr/local/kafka/bin/../libs/javax.servlet-
api-3.1.0.jar:/usr/local/kafka/bin/../libs/javax.ws.rs-api-2.1.jar:/usr/local
/kafka/bin/../libs/jaxb-api-2.3.0.jar:/usr/local/kafka/bin/../libs/jersey-
client-2.27.jar:/usr/local/kafka/bin/../libs/jersey-common-2.27.jar:
/usr/local/kafka/bin/../libs/jersey-container-servlet-2.27.jar:/usr/local
/kafka/bin/../libs/jersey-container-servlet-core-2.27.jar:/usr/local/kafka
/bin/../libs/jersey-hk2-2.27.jar:/usr/local/kafka/bin/../libs/jersey-media-
jaxb-2.27.jar:/usr/local/kafka/bin/../libs/jersey-server-2.27.jar:/usr/local
/kafka/bin/../libs/jetty-client-9.4.11.v20180605.jar:/usr/local/kafka/bin/..
/libs/jetty-continuation-9.4.11.v20180605.jar:/usr/local/kafka/bin/..
/libs/jetty-http-9.4.11.v20180605.jar:/usr/local/kafka/bin/../libs/jetty-
io-9.4.11.v20180605.jar:/usr/local/kafka/bin/../libs/jetty-security-
9.4.11.v20180605.jar:/usr/local/kafka/bin/../libs/jetty-server-
9.4.11.v20180605.jar:/usr/local/kafka/bin/../libs/jetty-servlet-
9.4.11.v20180605.jar:/usr/local/kafka/bin/../libs/jetty-servlets-
9.4.11.v20180605.jar:/usr/local/kafka/bin/../libs/jetty-
util-9.4.11.v20180605.jar:/usr/local/kafka/bin/../libs/jopt-simple-5.0.4.jar:
/usr/local/kafka/bin/../libs/kafka_2.11-2.0.0.jar:/usr/local/kafka/bin/..
/libs/kafka_2.11-2.0.0-sources.jar:/usr/local/kafka/bin/../libs/kafka-
clients-2.0.0.jar:/usr/local/kafka/bin/../libs/kafka-log4j-appender-
2.0.0.jar:/usr/local/kafka/bin/../libs/kafka-streams-2.0.0.jar:/usr/local
/kafka/bin/../libs/kafka-streams-examples-2.0.0.jar:/usr/local/kafka/bin/..
/libs/kafka-streams-scala_2.11-2.0.0.jar:/usr/local/kafka/bin/../libs/kafka-
streams-test-utils-2.0.0.jar:/usr/local/kafka/bin/../libs/kafka-tools-
2.0.0.jar:/usr/local/kafka/bin/../libs/log4j-1.2.17.jar:/usr/local/kafka
/bin/../libs/lz4-java-1.4.1.jar:/usr/local/kafka/bin/../libs/maven-artifact-
3.5.3.jar:/usr/local/kafka/bin/../libs/metrics-core-2.2.0.jar:/usr/local
/kafka/bin/../libs/osgi-resource-locator-1.0.1.jar:/usr/local/kafka/bin/..
/libs/plexus-utils-3.1.0.jar:/usr/local/kafka/bin/../libs/reflections-
0.9.11.jar:/usr/local/kafka/bin/../libs/rocksdbjni-5.7.3.jar:/usr/local/kafka
/bin/../libs/scala-library-2.11.12.jar:/usr/local/kafka/bin/../libs/scala-
logging_2.11-3.9.0.jar:/usr/local/kafka/bin/../libs/scala-reflect-
2.11.12.jar:/usr/local/kafka/bin/../libs/slf4j-api-1.7.25.jar:/usr/local
/kafka/bin/../libs/slf4j-log4j12-1.7.25.jar:/usr/local/kafka/bin/..
/libs/snappy-java-1.1.7.1.jar:/usr/local/kafka/bin/../libs/validation-
api-1.1.0.Final.jar:/usr/local/kafka/bin/../libs/zkclient-0.10.jar:/usr/local
/kafka/bin/../libs/zookeeper-3.4.13.jar
(org.apache.zookeeper.server.ZooKeeperServer)
[2018-09-04 12:18:14,625] INFO Server environment:java.library.path=/usr
/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
(org.apache.zookeeper.server.ZooKeeperServer)
[2018-09-04 12:18:14,625] INFO Server environment:java.io.tmpdir=/tmp
(org.apache.zookeeper.server.ZooKeeperServer)
[2018-09-04 12:18:14,625] INFO Server environment:java.compiler=<NA>
(org.apache.zookeeper.server.ZooKeeperServer)
[2018-09-04 12:18:14,625] INFO Server environment:os.name=Linux
(org.apache.zookeeper.server.ZooKeeperServer)
[2018-09-04 12:18:14,625] INFO Server environment:os.arch=amd64
(org.apache.zookeeper.server.ZooKeeperServer)
[2018-09-04 12:18:14,625] INFO Server environment:os.version=4.15.0-33-
generic (org.apache.zookeeper.server.ZooKeeperServer)
[2018-09-04 12:18:14,625] INFO Server environment:user.name=marco
(org.apache.zookeeper.server.ZooKeeperServer)
[2018-09-04 12:18:14,625] INFO Server environment:user.home=/home/marco
(org.apache.zookeeper.server.ZooKeeperServer)
[2018-09-04 12:18:14,625] INFO Server environment:user.dir=/usr/local/kafka
(org.apache.zookeeper.server.ZooKeeperServer)
[2018-09-04 12:18:14,630] INFO tickTime set to 3000
(org.apache.zookeeper.server.ZooKeeperServer)
[2018-09-04 12:18:14,630] INFO minSessionTimeout set to -1
(org.apache.zookeeper.server.ZooKeeperServer)
[2018-09-04 12:18:14,630] INFO maxSessionTimeout set to -1
(org.apache.zookeeper.server.ZooKeeperServer)
[2018-09-04 12:18:14,634] INFO Using
org.apache.zookeeper.server.NIOServerCnxn Factory as server connection
factory (org.apache.zookeeper.server.ServerCnxnFactory)
[2018-09-04 12:18:14,636] INFO binding to port 0.0.0.0/0.0.0.0:2181
(org.apache.zookeeper.server.NIOServerCnxnFactory)
The problem is that rebooting zookeeper seems active:
marco#PC:~$ netstat -ant | grep :2181
tcp6 0 0 :::2181 :::*
LISTEN
marco#PC:~$ systemctl status zookeeper
● zookeeper.service - LSB: centralized coordination service
Loaded: loaded (/etc/init.d/zookeeper; generated)
Active: active (running) since Tue 2018-09-04 15:51:24 CEST; 2min 19s
ago
Docs: man:systemd-sysv-generator(8)
Process: 1919 ExecStart=/etc/init.d/zookeeper start (code=exited,
status=0/SUCCESS)
Tasks: 25 (limit: 4915)
CGroup: /system.slice/zookeeper.service
└─1997 /usr/bin/java -cp /etc/zookeeper/conf:/usr/share
/java/jline.jar:/usr/share/java/log4j-1.2.jar:/usr/share
/java/xercesImpl.jar:/usr/share/java/xmlParserAPIs.jar:/usr/share
/java/netty.jar:/usr/share/java/slf4j-api.jar:/usr
Sep 04 15:51:24 PC systemd[1]: Starting LSB: centralized coordination
service...
Sep 04 15:51:24 PC systemd[1]: Started LSB: centralized coordination
service.
But when trying to run kafka... :
marco#PC:~$ cd /usr/local/kafka
marco#PC:/usr/local/kafka$ bin/zookeeper-server-start.sh
config/zookeeper.properties
[2018-09-04 15:54:29,981] INFO Reading configuration from:
config/zookeeper.properties
(org.apache.zookeeper.server.quorum.QuorumPeerConfig)
[2018-09-04 15:54:29,982] INFO autopurge.snapRetainCount set to 3
(org.apache.zookeeper.server.DatadirCleanupManager)
[2018-09-04 15:54:29,983] INFO autopurge.purgeInterval set to 0
(org.apache.zookeeper.server.DatadirCleanupManager)
...
[2018-09-04 15:54:30,012] INFO binding to port 0.0.0.0/0.0.0.0:2181
(org.apache.zookeeper.server.NIOServerCnxnFactory)
[2018-09-04 15:54:30,013] ERROR Unexpected exception, exiting abnormally
(org.apache.zookeeper.server.ZooKeeperServerMain)
java.net.BindException: Address already in use
at sun.nio.ch.Net.bind0(Native Method)
at sun.nio.ch.Net.bind(Net.java:433)
at sun.nio.ch.Net.bind(Net.java:425)
at
sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:223)
at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:74)
at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:67)
$ bin/zookeeper-server-start.sh config/zookeeper.properties
This started okay if you look at the logs
[2018-09-04 12:18:14,636] INFO binding to port 0.0.0.0/0.0.0.0:2181
(org.apache.zookeeper.server.NIOServerCnxnFactory)
And this message will sit there until a Zookeeper client such as Kafka connects to Zookeeper.
When you say you're "trying to run kafka", you're running zookeeper-server-start again, not kafka-server-start, so Zookeeper is already bound to port 2181 and it won't start a second one.
Maybe I "solved" the problem:
I left opened the "idle" window ( First Window ), I opened a new window and started the kafka server : Step 2.b of ApachaKafka-QuickStart : /usr/local/ kafka/ bin/kafka-server-start.sh config/server.properties ( Second Window ). Then I opened another window to create a topic and send some messages:
marco#PC:/usr/local/kafka$ bin/kafka-topics.sh --create --zookeeper localhost:2181
--replication-factor 1 --partitions 1 --topic test
Created topic "test".
marco#PC:/usr/local/kafka$ bin/kafka-topics.sh --list --zookeeper localhost:2181
test
marco#PC:/usr/local/kafka$ bin/kafka-console-producer.sh --broker-list localhost:9092
--topic test
>This is a message
>This is another message
>
( ThirdWindow ) . Finally, in another new window I started a consumer :
marco#PC:/usr/local/kafka$ bin/kafka-console-consumer.sh --bootstrap-server
localhost:9092 --topic test --from-beginning
This is a message
This is another message
: ( FourthWindow )
What I actually do not understand now is why I should open all these four sessions.
Marco
Zookeeper is accepting a socket connection from 50167(in my case) and then closed it.
[2017-09-04 14:44:23,926] INFO Server environment:user.dir=C:\kafka_2.11-0.11.0.0 (org.apache.zookeeper.server.ZooKeeperServer)
[2017-09-04 14:44:24,013] INFO tickTime set to 3000 (org.apache.zookeeper.server.ZooKeeperServer)
[2017-09-04 14:44:24,029] INFO minSessionTimeout set to -1 (org.apache.zookeeper.server.ZooKeeperServer)
[2017-09-04 14:44:24,045] INFO maxSessionTimeout set to -1 (org.apache.zookeeper.server.ZooKeeperServer)
[2017-09-04 14:44:24,245] INFO binding to port 0.0.0.0/0.0.0.0:2181 (org.apache.zookeeper.server.NIOServerCnxnFactory)
[2017-09-04 14:45:16,525] INFO Accepted socket connection from /0:0:0:0:0:0:0:1:50167 (org.apache.zookeeper.server.NIOServerCnxnFactory)
[2017-09-04 14:45:16,557] INFO Client attempting to establish new session at /0:0:0:0:0:0:0:1:50167 (org.apache.zookeeper.server.ZooKeeperServer)
[2017-09-04 14:45:16,572] INFO Creating new log file: log.b1 (org.apache.zookeeper.server.persistence.FileTxnLog)
[2017-09-04 14:45:16,613] INFO Established session 0x15e4c2b5f7f0000 with negotiated timeout 6000 for client /0:0:0:0:0:0:0:1:50167 (org.apache.zookeeper.server.ZooKeeperServer)
[2017-09-04 14:45:17,939] INFO Processed session termination for sessionid: 0x15e4c2b5f7f0000 (org.apache.zookeeper.server.PrepRequestProcessor)
[2017-09-04 14:45:17,970] INFO Closed socket connection for client /0:0:0:0:0:0:0:1:50167 which had sessionid 0x15e4c2b5f7f0000 (org.apache.zookeeper.server.NIOServerCnxn)
Due to this, Kafka server is also failing to run. Any solution will be helpful.
After, I deleted the old logs at location:
i.e. folders kafka_2.11-0.11.0.0kafka-logs & kafka_2.11-0.11.0.0zookeeper-data, the server started fine.
i am new to the kafka world,
i want to start zookeeper then when i type this
bin/zookeeper-server-start.sh config/zookeeper.properties
I got the following error
ERROR Unexpected exception, exiting abnormally (org.apache.zookeeper.server.ZooKeeperServerMain)
java.net.BindException: Address already in use
ERROR Unexpected exception, exiting abnormally (org.apache.zookeeper.server.ZooKeeperServerMain)
java.net.BindException: Address already in use
Then i tried netstat -nlp|grep 2181
but there is no process running
tcp 0 0 0.0.0.0:2181 0.0.0.0:* LISTEN -
Some light please
For this case,
You need to see if zookeeper is running or not
use below command
sudo lsof -i :2181
You will get
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
java 1005 zookeeper 33u IPv6 17209 0t0 TCP *:2181 (LISTEN)
java 1005 zookeeper 34u IPv6 327225 0t0 TCP localhost:2181->localhost:43566 (ESTABLISHED)
java 22585 root 88u IPv6 324552 0t0 TCP localhost:43566->localhost:2181 (ESTABLISHED)
like statement. Now kill the zookeeper to start again.
sudo kill -9 1005
Then use below to start zookeeper
bin/zookeeper-server-start.sh config/zookeeper.properties
Sounds like zookeeper server is running.
try:
bin/zkServer.sh stop from the zookeeper directory to shut it down and then:
bin/zookeeper-server-start.sh config/zookeeper.properties
from the kafka directory
That fixed my issue
There must be some stale process using the port 2181.I had the same issue.First I checked the status of the server:
/usr/share/zookeeper$ bin/zkServer.sh status
or
/usr/share/zookeeper$ echo status | nc 127.0.0.1 2181
Then, I tried to start kafka and it failed with the same error. I changed permission and run as sudo..it didn't work either.
Since i couldn't see any process using the port. I restarted my computer and it worked!!.
Check if zookeeper is already running or not by using this command.
bin/kafka-topics.sh --list --zookeeper localhost:2181
Check if you get number of topics, if you get any that means Zookeeper was already running.
So verify whether Zookeeper is already running or not.
If you are running command bin/zookeeper-server-start.sh config/zookeeper.properties
and getting error :
ERROR Unexpected exception, exiting abnormally (org.apache.zookeeper.server.ZooKeeperServerMain)
java.net.BindException: Address already in use
at sun.nio.ch.Net.bind0(Native Method)
In that case in your virtual machine port 2181 is already using by zookeeper.
so in kafka zookeeper.properties change the clientPort value to the port which is not in use like 5181
Again run the command and Zookeeper will start working.
First, stop the Zookeeper with the command below:
$ bin/zookeeper-server-stop.sh config/zookeeper.properties
Then, start it again and you should be good to go:
$ bin/zookeeper-server-start.sh config/zookeeper.properties
May be another user is running the process. check using jps if any process with Quorum is running kill it and then try
I have solved the problem by doing following commands.
Go to kafka folder where you installed and type sudo bin/zookeeper-server-stop.sh
bin/zookeeper-server-start.sh config/zookeeper.properties
I hope this helps. Best of luck!
Maybe you can stop your hbase first.
just like this follow..
[root#master kafka_2.11-0.10.1.0]# stop-hbase.sh
stopping hbase................
localhost: stopping zookeeper.
[root#master kafka_2.11-0.10.1.0]# jps
2903 ResourceManager
60745 Worker
2586 NameNode
2762 SecondaryNameNode
93996 Jps
60653 Master
[root#master kafka_2.11-0.10.1.0]# bin/zookeeper-server-start.sh config/zookeeper.properties
[2019-12-05 01:09:43,959] INFO Reading configuration from: config/zookeeper.properties (org.apache.zookeeper.server.quorum.QuorumPeerConfig)
[2019-12-05 01:09:43,965] INFO autopurge.snapRetainCount set to 3 (org.apache.zookeeper.server.DatadirCleanupManager)
[2019-12-05 01:09:43,965] INFO autopurge.purgeInterval set to 0 (org.apache.zookeeper.server.DatadirCleanupManager)
[2019-12-05 01:09:43,965] INFO Purge task is not scheduled. (org.apache.zookeeper.server.DatadirCleanupManager)
[2019-12-05 01:09:43,965] WARN Either no config or no quorum defined in config, running in standalone mode (org.apache.zookeeper.server.quorum.QuorumPeerMain)
[2019-12-05 01:09:44,013] INFO Reading configuration from: config/zookeeper.properties (org.apache.zookeeper.server.quorum.QuorumPeerConfig)
[2019-12-05 01:09:44,013] INFO Starting server (org.apache.zookeeper.server.ZooKeeperServerMain)
[2019-12-05 01:09:44,023] INFO Server environment:zookeeper.version=3.4.8--1, built on 02/06/2016 03:18 GMT (org.apache.zookeeper.server.ZooKeeperServer)
[2019-12-05 01:09:44,023] INFO Server environment:host.name=master (org.apache.zookeeper.server.ZooKeeperServer)
[2019-12-05 01:09:44,023] INFO Server environment:java.version=1.8.0_171 (org.apache.zookeeper.server.ZooKeeperServer)
[2019-12-05 01:09:44,023] INFO Server environment:java.vendor=Oracle Corporation (org.apache.zookeeper.server.ZooKeeperServer)
[2019-12-05 01:09:44,023] INFO Server environment:java.home=/usr/local/soft/jdk1.8.0_171/jre (org.apache.zookeeper.server.ZooKeeperServer)
[2019-12-05 01:09:44,023] INFO Server environment:java.class.path=:/usr/local/soft/kafka_2.11-0.10.1.0/bin/../libs/aopalliance-repackaged-2.4.0-b34.jar:/usr/local/soft/kafka_2.11-0.10.1.0/bin/../libs/argparse4j-0.5.0.jar:/usr/local/soft/kafka_2.11-0.10.1.0/bin/../libs/connect-api-0.10.1.0.jar:/usr/local/soft/kafka_2.11-0.10.1.0/bin/../libs/connect-file-0.10.1.0.jar:/usr/local/soft/kafka_2.11-0.10.1.0/bin/../libs/connect-json-0.10.1.0.jar:/usr/local/soft/kafka_2.11-0.10.1.0/bin/../libs/connect-runtime-0.10.1.0.jar:/usr/local/soft/kafka_2.11-0.10.1.0/bin/../libs/guava-18.0.jar:/usr/local/soft/kafka_2.11-0.10.1.0/bin/../libs/hk2-api-2.4.0-b34.jar:/usr/local/soft/kafka_2.11-0.10.1.0/bin/../libs/hk2-locator-2.4.0-b34.jar:/usr/local/soft/kafka_2.11-0.10.1.0/bin/../libs/hk2-utils-2.4.0-b34.jar:/usr/local/soft/kafka_2.11-0.10.1.0/bin/../libs/jackson-annotations-2.6.0.jar:/usr/local/soft/kafka_2.11-0.10.1.0/bin/../libs/jackson-core-2.6.3.jar:/usr/local/soft/kafka_2.11-0.10.1.0/bin/../libs/jackson-databind-2.6.3.jar:/usr/local/soft/kafka_2.11-0.10.1.0/bin/../libs/jackson-jaxrs-base-2.6.3.jar:/usr/local/soft/kafka_2.11-0.10.1.0/bin/../libs/jackson-jaxrs-json-provider-2.6.3.jar:/usr/local/soft/kafka_2.11-0.10.1.0/bin/../libs/jackson-module-jaxb-annotations-2.6.3.jar:/usr/local/soft/kafka_2.11-0.10.1.0/bin/../libs/javassist-3.18.2-GA.jar:/usr/local/soft/kafka_2.11-0.10.1.0/bin/../libs/javax.annotation-api-1.2.jar:/usr/local/soft/kafka_2.11-0.10.1.0/bin/../libs/javax.inject-1.jar:/usr/local/soft/kafka_2.11-0.10.1.0/bin/../libs/javax.inject-2.4.0-b34.jar:/usr/local/soft/kafka_2.11-0.10.1.0/bin/../libs/javax.servlet-api-3.1.0.jar:/usr/local/soft/kafka_2.11-0.10.1.0/bin/../libs/javax.ws.rs-api-2.0.1.jar:/usr/local/soft/kafka_2.11-0.10.1.0/bin/../libs/jersey-client-2.22.2.jar:/usr/local/soft/kafka_2.11-0.10.1.0/bin/../libs/jersey-common-2.22.2.jar:/usr/local/soft/kafka_2.11-0.10.1.0/bin/../libs/jersey-container-servlet-2.22.2.jar:/usr/local/soft/kafka_2.11-0.10.1.0/bin/../libs/jersey-container-servlet-core-2.22.2.jar:/usr/local/soft/kafka_2.11-0.10.1.0/bin/../libs/jersey-guava-2.22.2.jar:/usr/local/soft/kafka_2.11-0.10.1.0/bin/../libs/jersey-media-jaxb-2.22.2.jar:/usr/local/soft/kafka_2.11-0.10.1.0/bin/../libs/jersey-server-2.22.2.jar:/usr/local/soft/kafka_2.11-0.10.1.0/bin/../libs/jetty-continuation-9.2.15.v20160210.jar:/usr/local/soft/kafka_2.11-0.10.1.0/bin/../libs/jetty-http-9.2.15.v20160210.jar:/usr/local/soft/kafka_2.11-0.10.1.0/bin/../libs/jetty-io-9.2.15.v20160210.jar:/usr/local/soft/kafka_2.11-0.10.1.0/bin/../libs/jetty-security-9.2.15.v20160210.jar:/usr/local/soft/kafka_2.11-0.10.1.0/bin/../libs/jetty-server-9.2.15.v20160210.jar:/usr/local/soft/kafka_2.11-0.10.1.0/bin/../libs/jetty-servlet-9.2.15.v20160210.jar:/usr/local/soft/kafka_2.11-0.10.1.0/bin/../libs/jetty-servlets-9.2.15.v20160210.jar:/usr/local/soft/kafka_2.11-0.10.1.0/bin/../libs/jetty-util-9.2.15.v20160210.jar:/usr/local/soft/kafka_2.11-0.10.1.0/bin/../libs/jopt-simple-4.9.jar:/usr/local/soft/kafka_2.11-0.10.1.0/bin/../libs/kafka_2.11-0.10.1.0.jar:/usr/local/soft/kafka_2.11-0.10.1.0/bin/../libs/kafka_2.11-0.10.1.0-sources.jar:/usr/local/soft/kafka_2.11-0.10.1.0/bin/../libs/kafka_2.11-0.10.1.0-test-sources.jar:/usr/local/soft/kafka_2.11-0.10.1.0/bin/../libs/kafka-clients-0.10.1.0.jar:/usr/local/soft/kafka_2.11-0.10.1.0/bin/../libs/kafka-log4j-appender-0.10.1.0.jar:/usr/local/soft/kafka_2.11-0.10.1.0/bin/../libs/kafka-streams-0.10.1.0.jar:/usr/local/soft/kafka_2.11-0.10.1.0/bin/../libs/kafka-streams-examples-0.10.1.0.jar:/usr/local/soft/kafka_2.11-0.10.1.0/bin/../libs/kafka-tools-0.10.1.0.jar:/usr/local/soft/kafka_2.11-0.10.1.0/bin/../libs/log4j-1.2.17.jar:/usr/local/soft/kafka_2.11-0.10.1.0/bin/../libs/lz4-1.3.0.jar:/usr/local/soft/kafka_2.11-0.10.1.0/bin/../libs/metrics-core-2.2.0.jar:/usr/local/soft/kafka_2.11-0.10.1.0/bin/../libs/osgi-resource-locator-1.0.1.jar:/usr/local/soft/kafka_2.11-0.10.1.0/bin/../libs/reflections-0.9.10.jar:/usr/local/soft/kafka_2.11-0.10.1.0/bin/../libs/rocksdbjni-4.9.0.jar:/usr/local/soft/kafka_2.11-0.10.1.0/bin/../libs/scala-library-2.11.8.jar:/usr/local/soft/kafka_2.11-0.10.1.0/bin/../libs/scala-parser-combinators_2.11-1.0.4.jar:/usr/local/soft/kafka_2.11-0.10.1.0/bin/../libs/slf4j-api-1.7.21.jar:/usr/local/soft/kafka_2.11-0.10.1.0/bin/../libs/slf4j-log4j12-1.7.21.jar:/usr/local/soft/kafka_2.11-0.10.1.0/bin/../libs/snappy-java-1.1.2.6.jar:/usr/local/soft/kafka_2.11-0.10.1.0/bin/../libs/validation-api-1.1.0.Final.jar:/usr/local/soft/kafka_2.11-0.10.1.0/bin/../libs/zkclient-0.9.jar:/usr/local/soft/kafka_2.11-0.10.1.0/bin/../libs/zookeeper-3.4.8.jar (org.apache.zookeeper.server.ZooKeeperServer)
[2019-12-05 01:09:44,024] INFO Server environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib (org.apache.zookeeper.server.ZooKeeperServer)
[2019-12-05 01:09:44,024] INFO Server environment:java.io.tmpdir=/tmp (org.apache.zookeeper.server.ZooKeeperServer)
[2019-12-05 01:09:44,024] INFO Server environment:java.compiler=<NA> (org.apache.zookeeper.server.ZooKeeperServer)
[2019-12-05 01:09:44,024] INFO Server environment:os.name=Linux (org.apache.zookeeper.server.ZooKeeperServer)
[2019-12-05 01:09:44,024] INFO Server environment:os.arch=amd64 (org.apache.zookeeper.server.ZooKeeperServer)
[2019-12-05 01:09:44,024] INFO Server environment:os.version=2.6.32-431.el6.x86_64 (org.apache.zookeeper.server.ZooKeeperServer)
[2019-12-05 01:09:44,024] INFO Server environment:user.name=root (org.apache.zookeeper.server.ZooKeeperServer)
[2019-12-05 01:09:44,024] INFO Server environment:user.home=/root (org.apache.zookeeper.server.ZooKeeperServer)
[2019-12-05 01:09:44,025] INFO Server environment:user.dir=/usr/local/soft/kafka_2.11-0.10.1.0 (org.apache.zookeeper.server.ZooKeeperServer)
[2019-12-05 01:09:44,035] INFO tickTime set to 3000 (org.apache.zookeeper.server.ZooKeeperServer)
[2019-12-05 01:09:44,035] INFO minSessionTimeout set to -1 (org.apache.zookeeper.server.ZooKeeperServer)
[2019-12-05 01:09:44,035] INFO maxSessionTimeout set to -1 (org.apache.zookeeper.server.ZooKeeperServer)
[2019-12-05 01:09:44,050] INFO binding to port 0.0.0.0/0.0.0.0:2181 (org.apache.zookeeper.server.NIOServerCnxnFactory)
I got the same problem. I got it because zookeeper server is crashed by my laptop crash.
I got solved by the help from the below link.
How to recover Zookeeper from java.io.EOFException after a server crash?
I found the offending log file by opening one by one [Zookeper-data-dir]/zookeeper_0/version-2. And found a log file without head or any other thing. When I deleted it, I got my problem solved and my zookeeper server started running normally.
Seems to be the ZooKeeper port 2181 is still in use, Please follow the below steps to address this issue:
Use the netstat command to find the process that is holding onto port 2181. Kill the process that is using the ZooKeeper port 2181:
$ netstat -antp | grep 2181
tcp 0 0 0.0.0.0:2181 0.0.0.0:*
LISTEN 28016/java <defunct>
$ kill -9 28016
Restarting the kafka process solves the issue.
There is existing zookeeper running, how should I remove this. And than stop the zookeeper
bin/kafka-topics.sh --list --zookeeper localhost:2181
test
Many mentioned the process for linux versions.
In case if we want to identify the same for windows, we can identify with below
From PowerShell, finding the port listening to 2181
PS C:\Users\<username>> netstat -ano | findstr 2181
TCP [::1]:2181 [::]:0 LISTENING 4564
Option #1: kill that process id using task kill
PS C:\Users\vishnus> taskkill /F /PID 4564
In case this option doesn't work and failing
ERROR: The process with PID 4564 could not be terminated.
Reason: Access is denied.
Option #2
Go to "Task Manager" --> Services tab, check for PID column and go to that process and right click and mark it as stop. then it will get stopped.
By this time, the earlier instance is killed, so you can start the zookeeper
I have the same problem and then I debug step by step and find the solution as follows:
stop the zookeeper:
/opt/kafka_2.13-2.7.0/bin/zookeeper-server-stop.sh
Then I check whether the zookeeper process has indeed been stopped:
ps aux | grep zookeeper
If the zookeeper has been stopped, then no zookeeper process should be displayed. You can also check the zookeeper process via port number or java process. If your port number for zookeeper is the default 2181, you can check which process is listening to this port by running sudo lsof -i :2181. You can also check the all the java processs and find the corresponding zookeeper process by running sudo ps -fC java
If the zookeeper has not been stopped, run sudo kill -9 zookeeper_process_id to kill it.
After this, we are sure that zookeeper have indeed been stopped. If we run jps, the QuorumPeerMain will not be shown.
After this, we can restart the zookeeper,
/opt/kafka_2.13-2.7.0/bin/zookeeper-server-start.sh /opt/kafka_2.13-2.7.0/config/zookeeper.properties
For windows user: below step will stop the zookeeper
step1 : netstat -ano | findstr :
NOTE: port number can be replaced by the port where zookeeper is runing
Step2: Now from image , the circled one is PID so copy that and put it in below command
taskkill /PID <PIDNO/F