Kafka shuts down without error after a certain time interval.
kafka server log:
[2016-01-11 12:15:06,435] INFO Verifying properties (kafka.utils.VerifiableProperties)
[2016-01-11 12:15:06,489] INFO Property broker.id is overridden to 0 (kafka.utils.VerifiableProperties)
[2016-01-11 12:15:06,489] INFO Property host.name is overridden to 172.16.1.22 (kafka.utils.VerifiableProperties)
[2016-01-11 12:15:06,490] INFO Property log.cleaner.enable is overridden to false (kafka.utils.VerifiableProperties)
[2016-01-11 12:15:06,490] INFO Property log.dirs is overridden to /data/www/wifiin/logs/kafka (kafka.utils.VerifiableProperties)
[2016-01-11 12:15:06,490] INFO Property log.retention.check.interval.ms is overridden to 300000 (kafka.utils.VerifiableProperties)
[2016-01-11 12:15:06,491] INFO Property log.retention.hours is overridden to 168 (kafka.utils.VerifiableProperties)
[2016-01-11 12:15:06,491] INFO Property log.segment.bytes is overridden to 1073741824 (kafka.utils.VerifiableProperties)
[2016-01-11 12:15:06,491] INFO Property num.io.threads is overridden to 8 (kafka.utils.VerifiableProperties)
[2016-01-11 12:15:06,491] INFO Property num.network.threads is overridden to 3 (kafka.utils.VerifiableProperties)
[2016-01-11 12:15:06,492] INFO Property num.partitions is overridden to 1 (kafka.utils.VerifiableProperties)
[2016-01-11 12:15:06,492] INFO Property num.recovery.threads.per.data.dir is overridden to 1 (kafka.utils.VerifiableProperties)
[2016-01-11 12:15:06,492] INFO Property port is overridden to 9092 (kafka.utils.VerifiableProperties)
[2016-01-11 12:15:06,493] INFO Property socket.receive.buffer.bytes is overridden to 102400 (kafka.utils.VerifiableProperties)
[2016-01-11 12:15:06,493] INFO Property socket.request.max.bytes is overridden to 104857600 (kafka.utils.VerifiableProperties)
[2016-01-11 12:15:06,493] INFO Property socket.send.buffer.bytes is overridden to 102400 (kafka.utils.VerifiableProperties)
[2016-01-11 12:15:06,493] INFO Property zookeeper.connect is overridden to 127.0.0.1:2181 (kafka.utils.VerifiableProperties)
[2016-01-11 12:15:06,494] INFO Property zookeeper.connection.timeout.ms is overridden to 6000 (kafka.utils.VerifiableProperties)
[2016-01-11 12:15:06,550] INFO [Kafka Server 0], starting (kafka.server.KafkaServer)
[2016-01-11 12:15:06,553] INFO [Kafka Server 0], Connecting to zookeeper on 127.0.0.1:2181 (kafka.server.KafkaServer)
[2016-01-11 12:15:06,568] INFO Starting ZkClient event thread. (org.I0Itec.zkclient.ZkEventThread)
[2016-01-11 12:15:06,579] INFO Client environment:zookeeper.version=3.4.6-1569965, built on 02/20/2014 09:09 GMT (org.apache.zookeeper.ZooKeeper)
[2016-01-11 12:15:06,579] INFO Client environment:host.name=wifiin-analysis-22 (org.apache.zookeeper.ZooKeeper)
[2016-01-11 12:15:06,579] INFO Client environment:java.version=1.7.0_40 (org.apache.zookeeper.ZooKeeper)
[2016-01-11 12:15:06,579] INFO Client environment:java.vendor=Oracle Corporation (org.apache.zookeeper.ZooKeeper)
[2016-01-11 12:15:06,579] INFO Client environment:java.home=/usr/local/java/jdk1.7.0_40/jre
(org.apache.zookeeper.ZooKeeper)
...
[2016-01-11 12:15:06,580] INFO Client environment:java.io.tmpdir=/tmp (org.apache.zookeeper.ZooKeeper)
[2016-01-11 12:15:06,580] INFO Client environment:java.compiler=<NA(org.apache.zookeeper.ZooKeeper)
[2016-01-11 12:15:06,580] INFO Client environment:os.name=Linux (org.apache.zookeeper.ZooKeeper)
[2016-01-11 12:15:06,580] INFO Client environment:os.arch=amd64 (org.apache.zookeeper.ZooKeeper)
[2016-01-11 12:15:06,580] INFO Client environment:os.version=2.6.32-573.3.1.el6.x86_64 (org.apache.zookeeper.ZooKeeper)
[2016-01-11 12:15:06,580] INFO Client environment:user.name=root (org.apache.zookeeper.ZooKeeper)
[2016-01-11 12:15:06,580] INFO Client environment:user.home=/root (org.apache.zookeeper.ZooKeeper)
[2016-01-11 12:15:06,580] INFO Client environment:user.dir=/usr/local/kafka (org.apache.zookeeper.ZooKeeper)
...[2016-01-11 12:15:17,515] INFO [Kafka Server 0], started (kafka.server.KafkaServer)
[2016-01-11 12:15:17,795] INFO [ReplicaFetcherManager on broker 0] Removed fetcher for partitions [sdkLog,0] (kafka.server.ReplicaFetcherManager)
[2016-01-11 12:15:17,856] INFO [ReplicaFetcherManager on broker 0] Removed fetcher for partitions [sdkLog,0] (kafka.server.ReplicaFetcherManager)
[2016-01-11 14:47:41,816] INFO [Kafka Server 0], shutting down (kafka.server.KafkaServer)
[2016-01-11 14:47:41,818] INFO [Kafka Server 0], Starting controlled shutdown (kafka.server.KafkaServer)
[2016-01-11 14:47:41,981] INFO [Kafka Server 0], Controlled shutdown succeeded (kafka.server.KafkaServer)
[2016-01-11 14:47:41,983] INFO Closing socket connection to /172.16.1.22. (kafka.network.Processor)
[2016-01-11 14:47:41,984] INFO [Socket Server on Broker 0], Shutting down (kafka.network.SocketServer)
[2016-01-11 14:47:41,989] INFO [Socket Server on Broker 0], Shutdown completed (kafka.network.SocketServer)
[2016-01-11 14:47:41,991] INFO [Kafka Request Handler on Broker 0], shutting down (kafka.server.KafkaRequestHandlerPool)
[2016-01-11 14:47:41,994] INFO [Kafka Request Handler on Broker 0], shut down completely (kafka.server.KafkaRequestHandlerPool)
[2016-01-11 14:47:42,267] INFO [Replica Manager on Broker 0]: Shut down (kafka.server.ReplicaManager)
[2016-01-11 14:47:42,267] INFO [ReplicaFetcherManager on broker 0] shutting down (kafka.server.ReplicaFetcherManager)
[2016-01-11 14:47:42,269] INFO [ReplicaFetcherManager on broker 0] shutdown completed (kafka.server.ReplicaFetcherManager)
[2016-01-11 14:47:42,314] INFO [Replica Manager on Broker 0]: Shut down
completely (kafka.server.ReplicaManager)
[2016-01-11 14:47:42,315] INFO Shutting down. (kafka.log.LogManager)
[2016-01-11 14:47:42,376] INFO Shutdown complete. (kafka.log.LogManager)
[2016-01-11 14:47:42,384] INFO Terminate ZkClient event thread.
(org.I0Itec.zkclient.ZkEventThread)
[2016-01-11 14:47:42,396] INFO Session: 0x15210b1e3051357 closed (org.apache.zookeeper.ZooKeeper)
[2016-01-11 14:47:42,396] INFO EventThread shut down (org.apache.zookeeper.ClientCnxn)
[2016-01-11 14:47:42,396] INFO [Kafka Server 0], shut down completed (kafka.server.KafkaServer)
At first, I believed OutOfMemory led to the shutdown, so I edited kafka's oom_score wihtout luck.What should I do to keep the Kafka broker from shutting down?
./kafka-server-start.sh -daemon ../config/server.properties
This may help you！
Related
This question already has answers here:
How to install Kafka on Windows?
(9 answers)
Closed 1 year ago.
I downloaded the kafka_2.13-3.0.0 in Window 10. I opened up CMD as admin. I was able to run zookeeper using the command bin\windows\zookeeper-server-start.bat config\zookeeper.properties. However, when i tried to start the kafka server with the following command
bin\windows\kafka-server-start.bat config\server.properties
, it gave me the following error" below. I am confused here because I can see the kafka-logs being created (see image below). Can someone help me out please ?
Output from running Zookeeper command
[2021-12-01 20:16:17,208] INFO Server environment:java.io.tmpdir=C:\Users\user123456~1\AppData\Local\Temp\ (org.apache.zookeeper.server.ZooKeeperServer)
[2021-12-01 20:16:17,208] INFO Server environment:java.compiler=<NA> (org.apache.zookeeper.server.ZooKeeperServer)
[2021-12-01 20:16:17,208] INFO Server environment:os.name=Windows 10 (org.apache.zookeeper.server.ZooKeeperServer)
[2021-12-01 20:16:17,209] INFO Server environment:os.arch=amd64 (org.apache.zookeeper.server.ZooKeeperServer)
[2021-12-01 20:16:17,209] INFO Server environment:os.version=10.0 (org.apache.zookeeper.server.ZooKeeperServer)
[2021-12-01 20:16:17,209] INFO Server environment:user.name=user123456 (org.apache.zookeeper.server.ZooKeeperServer)
[2021-12-01 20:16:17,210] INFO Server environment:user.home=C:\Users\user123456 (org.apache.zookeeper.server.ZooKeeperServer)
[2021-12-01 20:16:17,210] INFO Server environment:user.dir=C:\kafka_2.13-3.0.0 (org.apache.zookeeper.server.ZooKeeperServer)
[2021-12-01 20:16:17,211] INFO Server environment:os.memory.free=493MB (org.apache.zookeeper.server.ZooKeeperServer)
[2021-12-01 20:16:17,211] INFO Server environment:os.memory.max=512MB (org.apache.zookeeper.server.ZooKeeperServer)
[2021-12-01 20:16:17,211] INFO Server environment:os.memory.total=512MB (org.apache.zookeeper.server.ZooKeeperServer)
[2021-12-01 20:16:17,212] INFO zookeeper.enableEagerACLCheck = false (org.apache.zookeeper.server.ZooKeeperServer)
[2021-12-01 20:16:17,212] INFO zookeeper.digest.enabled = true (org.apache.zookeeper.server.ZooKeeperServer)
[2021-12-01 20:16:17,212] INFO zookeeper.closeSessionTxn.enabled = true (org.apache.zookeeper.server.ZooKeeperServer)
[2021-12-01 20:16:17,213] INFO zookeeper.flushDelay=0 (org.apache.zookeeper.server.ZooKeeperServer)
[2021-12-01 20:16:17,213] INFO zookeeper.maxWriteQueuePollTime=0 (org.apache.zookeeper.server.ZooKeeperServer)
[2021-12-01 20:16:17,214] INFO zookeeper.maxBatchSize=1000 (org.apache.zookeeper.server.ZooKeeperServer)
[2021-12-01 20:16:17,214] INFO zookeeper.intBufferStartingSizeBytes = 1024 (org.apache.zookeeper.server.ZooKeeperServer)
[2021-12-01 20:16:17,215] INFO Weighed connection throttling is disabled (org.apache.zookeeper.server.BlueThrottle)
[2021-12-01 20:16:17,216] INFO minSessionTimeout set to 6000 (org.apache.zookeeper.server.ZooKeeperServer)
[2021-12-01 20:16:17,216] INFO maxSessionTimeout set to 60000 (org.apache.zookeeper.server.ZooKeeperServer)
[2021-12-01 20:16:17,217] INFO Response cache size is initialized with value 400. (org.apache.zookeeper.server.ResponseCache)
[2021-12-01 20:16:17,217] INFO Response cache size is initialized with value 400. (org.apache.zookeeper.server.ResponseCache)
[2021-12-01 20:16:17,218] INFO zookeeper.pathStats.slotCapacity = 60 (org.apache.zookeeper.server.util.RequestPathMetricsCollector)
[2021-12-01 20:16:17,220] INFO zookeeper.pathStats.slotDuration = 15 (org.apache.zookeeper.server.util.RequestPathMetricsCollector)
[2021-12-01 20:16:17,221] INFO zookeeper.pathStats.maxDepth = 6 (org.apache.zookeeper.server.util.RequestPathMetricsCollector)
[2021-12-01 20:16:17,221] INFO zookeeper.pathStats.initialDelay = 5 (org.apache.zookeeper.server.util.RequestPathMetricsCollector)
[2021-12-01 20:16:17,221] INFO zookeeper.pathStats.delay = 5 (org.apache.zookeeper.server.util.RequestPathMetricsCollector)
[2021-12-01 20:16:17,222] INFO zookeeper.pathStats.enabled = false (org.apache.zookeeper.server.util.RequestPathMetricsCollector)
[2021-12-01 20:16:17,224] INFO The max bytes for all large requests are set to 104857600 (org.apache.zookeeper.server.ZooKeeperServer)
[2021-12-01 20:16:17,224] INFO The large request threshold is set to -1 (org.apache.zookeeper.server.ZooKeeperServer)
[2021-12-01 20:16:17,224] INFO Created server with tickTime 3000 minSessionTimeout 6000 maxSessionTimeout 60000 clientPortListenBacklog -1 datadir \tmp\zookeeper\version-2 snapdir \tmp\zookeeper\version-2 (org.apache.zookeeper.server.ZooKeeperServer)
[2021-12-01 20:16:17,234] INFO Using org.apache.zookeeper.server.NIOServerCnxnFactory as server connection factory (org.apache.zookeeper.server.ServerCnxnFactory)
[2021-12-01 20:16:17,235] WARN maxCnxns is not configured, using default value 0. (org.apache.zookeeper.server.ServerCnxnFactory)
[2021-12-01 20:16:17,236] INFO Configuring NIO connection handler with 10s sessionless connection timeout, 2 selector thread(s), 32 worker threads, and 64 kB direct buffers. (org.apache.zookeeper.server.NIOServerCnxnFactory)
[2021-12-01 20:16:17,238] INFO binding to port 0.0.0.0/0.0.0.0:2181 (org.apache.zookeeper.server.NIOServerCnxnFactory)
[2021-12-01 20:16:17,250] INFO Using org.apache.zookeeper.server.watch.WatchManager as watch manager (org.apache.zookeeper.server.watch.WatchManagerFactory)
[2021-12-01 20:16:17,250] INFO Using org.apache.zookeeper.server.watch.WatchManager as watch manager (org.apache.zookeeper.server.watch.WatchManagerFactory)
[2021-12-01 20:16:17,251] INFO zookeeper.snapshotSizeFactor = 0.33 (org.apache.zookeeper.server.ZKDatabase)
[2021-12-01 20:16:17,251] INFO zookeeper.commitLogCount=500 (org.apache.zookeeper.server.ZKDatabase)
[2021-12-01 20:16:17,256] INFO zookeeper.snapshot.compression.method = CHECKED (org.apache.zookeeper.server.persistence.SnapStream)
[2021-12-01 20:16:17,256] INFO Snapshotting: 0x0 to \tmp\zookeeper\version-2\snapshot.0 (org.apache.zookeeper.server.persistence.FileTxnSnapLog)
[2021-12-01 20:16:17,260] INFO Snapshot loaded in 8 ms, highest zxid is 0x0, digest is 1371985504 (org.apache.zookeeper.server.ZKDatabase)
[2021-12-01 20:16:17,260] INFO Snapshotting: 0x0 to \tmp\zookeeper\version-2\snapshot.0 (org.apache.zookeeper.server.persistence.FileTxnSnapLog)
[2021-12-01 20:16:17,261] INFO Snapshot taken in 1 ms (org.apache.zookeeper.server.ZooKeeperServer)
[2021-12-01 20:16:17,274] INFO PrepRequestProcessor (sid:0) started, reconfigEnabled=false (org.apache.zookeeper.server.PrepRequestProcessor)
[2021-12-01 20:16:17,274] INFO zookeeper.request_throttler.shutdownTimeout = 10000 (org.apache.zookeeper.server.RequestThrottler)
[2021-12-01 20:16:17,286] INFO Using checkIntervalMs=60000 maxPerMinute=10000 maxNeverUsedIntervalMs=0 (org.apache.zookeeper.server.ContainerManager)
[2021-12-01 20:16:17,287] INFO ZooKeeper audit is disabled. (org.apache.zookeeper.audit.ZKAuditProvider)
Output from running kafka command
[2021-12-01 20:18:22,577] INFO Client environment:java.io.tmpdir=C:\Users\user123456~1\AppData\Local\Temp\ (org.apache.zookeeper.ZooKeeper)
[2021-12-01 20:18:22,577] INFO Client environment:java.compiler=<NA> (org.apache.zookeeper.ZooKeeper)
[2021-12-01 20:18:22,578] INFO Client environment:os.name=Windows 10 (org.apache.zookeeper.ZooKeeper)
[2021-12-01 20:18:22,579] INFO Client environment:os.arch=amd64 (org.apache.zookeeper.ZooKeeper)
[2021-12-01 20:18:22,579] INFO Client environment:os.version=10.0 (org.apache.zookeeper.ZooKeeper)
[2021-12-01 20:18:22,580] INFO Client environment:user.name=user123456 (org.apache.zookeeper.ZooKeeper)
[2021-12-01 20:18:22,581] INFO Client environment:user.home=C:\Users\user123456 (org.apache.zookeeper.ZooKeeper)
[2021-12-01 20:18:22,581] INFO Client environment:user.dir=C:\kafka_2.13-3.0.0 (org.apache.zookeeper.ZooKeeper)
[2021-12-01 20:18:22,582] INFO Client environment:os.memory.free=1011MB (org.apache.zookeeper.ZooKeeper)
[2021-12-01 20:18:22,583] INFO Client environment:os.memory.max=1024MB (org.apache.zookeeper.ZooKeeper)
[2021-12-01 20:18:22,583] INFO Client environment:os.memory.total=1024MB (org.apache.zookeeper.ZooKeeper)
[2021-12-01 20:18:22,585] INFO Initiating client connection, connectString=localhost:2181 sessionTimeout=18000 watcher=kafka.zookeeper.ZooKeeperClient$ZooKeeperClientWatcher$#74f0ea28 (org.apache.zookeeper.ZooKeeper)
[2021-12-01 20:18:22,593] INFO jute.maxbuffer value is 4194304 Bytes (org.apache.zookeeper.ClientCnxnSocket)
[2021-12-01 20:18:22,597] INFO zookeeper.request.timeout value is 0. feature enabled=false (org.apache.zookeeper.ClientCnxn)
[2021-12-01 20:18:22,598] INFO [ZooKeeperClient Kafka server] Waiting until connected. (kafka.zookeeper.ZooKeeperClient)
[2021-12-01 20:18:22,601] INFO Opening socket connection to server localhost/127.0.0.1:2181. (org.apache.zookeeper.ClientCnxn)
[2021-12-01 20:18:22,601] INFO SASL config status: Will not attempt to authenticate using SASL (unknown error) (org.apache.zookeeper.ClientCnxn)
[2021-12-01 20:18:22,602] INFO Socket connection established, initiating session, client: /127.0.0.1:57191, server: localhost/127.0.0.1:2181 (org.apache.zookeeper.ClientCnxn)
[2021-12-01 20:18:22,617] INFO Session establishment complete on server localhost/127.0.0.1:2181, session id = 0x1000008c9c20000, negotiated timeout = 18000 (org.apache.zookeeper.ClientCnxn)
[2021-12-01 20:18:22,619] INFO [ZooKeeperClient Kafka server] Connected. (kafka.zookeeper.ZooKeeperClient)
[2021-12-01 20:18:22,720] INFO [feature-zk-node-event-process-thread]: Starting (kafka.server.FinalizedFeatureChangeListener$ChangeNotificationProcessorThread)
[2021-12-01 20:18:22,729] INFO Feature ZK node at path: /feature does not exist (kafka.server.FinalizedFeatureChangeListener)
[2021-12-01 20:18:22,729] INFO Cleared cache (kafka.server.FinalizedFeatureCache)
[2021-12-01 20:18:22,844] INFO Cluster ID = j7KyANF-S2OuLlr_IYyqDw (kafka.server.KafkaServer)
[2021-12-01 20:18:22,846] WARN No meta.properties file under dir C:\tmp\kafka-logs\meta.properties (kafka.server.BrokerMetadataCheckpoint)
[2021-12-01 20:18:22,876] INFO KafkaConfig values:
...
[2021-12-01 20:18:22,918] INFO [ThrottledChannelReaper-Fetch]: Starting (kafka.server.ClientQuotaManager$ThrottledChannelReaper)
[2021-12-01 20:18:22,919] INFO [ThrottledChannelReaper-Produce]: Starting (kafka.server.ClientQuotaManager$ThrottledChannelReaper)
[2021-12-01 20:18:22,919] INFO [ThrottledChannelReaper-Request]: Starting (kafka.server.ClientQuotaManager$ThrottledChannelReaper)
[2021-12-01 20:18:22,920] INFO [ThrottledChannelReaper-ControllerMutation]: Starting (kafka.server.ClientQuotaManager$ThrottledChannelReaper)
[2021-12-01 20:18:22,931] INFO Log directory C:\tmp\kafka-logs not found, creating it. (kafka.log.LogManager)
[2021-12-01 20:18:22,933] ERROR Failed to create or validate data directory C:\tmp\kafka-logs (kafka.server.LogDirFailureChannel)
java.nio.file.AccessDeniedException: C:\tmp
at sun.nio.fs.WindowsException.translateToIOException(WindowsException.java:83)
at sun.nio.fs.WindowsException.rethrowAsIOException(WindowsException.java:97)
at sun.nio.fs.WindowsException.rethrowAsIOException(WindowsException.java:102)
at sun.nio.fs.WindowsFileSystemProvider.newFileChannel(WindowsFileSystemProvider.java:115)
at java.nio.channels.FileChannel.open(FileChannel.java:287)
at java.nio.channels.FileChannel.open(FileChannel.java:335)
at org.apache.kafka.common.utils.Utils.flushDir(Utils.java:953)
at kafka.log.LogManager.$anonfun$createAndValidateLogDirs$1(LogManager.scala:159)
at scala.collection.IterableOnceOps.foreach(IterableOnce.scala:563)
at scala.collection.IterableOnceOps.foreach$(IterableOnce.scala:561)
at scala.collection.AbstractIterable.foreach(Iterable.scala:919)
at kafka.log.LogManager.createAndValidateLogDirs(LogManager.scala:149)
at kafka.log.LogManager.<init>(LogManager.scala:88)
at kafka.log.LogManager$.apply(LogManager.scala:1283)
at kafka.server.KafkaServer.startup(KafkaServer.scala:254)
at kafka.Kafka$.main(Kafka.scala:109)
at kafka.Kafka.main(Kafka.scala)
[2021-12-01 20:18:22,936] ERROR Shutdown broker because none of the specified log dirs from C:\tmp\kafka-logs can be created or validated (kafka.log.LogManager)
I also tried running icacls "C:\tmp\" /q /c /t /grant Users:F. I am still getting the same error.
It turns out that 3.0.0 has issues with Windows. I downgraded to 2.8.1 and things worked fine. I hate Windows.
I'm using Confluent Community 6.0.1. Three nodes Kafka cluster:
devKafka04: Kafka Broker1, Zookeeper 1
devKafka05: Kafka Broker2, Zookeeper 2
devKafka06: Kafka Broker3, Zookeeper 3
The SSL encryption is already working well on the Kafka Brokers.
I'd like to add SASL to enable mutual authentication between Kafka and Zookeeper.
I was following the Confluent document:
https://docs.confluent.io/platform/current/kafka/incremental-security-upgrade.html#adding-security-to-a-running-zk-cluster
[Updates] After I applied the changes, Zookeeper could not start on the secureclientPort. That's why the Kafka broker couldn't start. Here are the error log and docker compose configurations.
I'm wondering if there's something with the confluent zookeeper image.
Please help me out. Thanks.
$ sudo docker logs zookeeper
===> User
uid=1000(appuser) gid=1000(appuser) groups=1000(appuser)
===> Configuring ...
===> Running preflight checks ...
===> Check if /var/lib/zookeeper/data is writable ...
===> Check if /var/lib/zookeeper/log is writable ...
===> Launching ...
===> Printing /var/lib/zookeeper/data/myid
1===> Launching zookeeper ...
[2021-03-24 19:03:08,857] INFO Reading configuration from: /etc/kafka/zookeeper.properties (org.apache.zookeeper.server.quorum.QuorumPeerConfig)
[2021-03-24 19:03:08,862] INFO clientPortAddress is 0.0.0.0:2181 (org.apache.zookeeper.server.quorum.QuorumPeerConfig)
[2021-03-24 19:03:08,862] INFO secureClientPort is not set (org.apache.zookeeper.server.quorum.QuorumPeerConfig)
[2021-03-24 19:03:08,876] INFO autopurge.snapRetainCount set to 3 (org.apache.zookeeper.server.DatadirCleanupManager)
[2021-03-24 19:03:08,876] INFO autopurge.purgeInterval set to 0 (org.apache.zookeeper.server.DatadirCleanupManager)
[2021-03-24 19:03:08,876] INFO Purge task is not scheduled. (org.apache.zookeeper.server.DatadirCleanupManager)
[2021-03-24 19:03:08,880] INFO Log4j 1.2 jmx support found and enabled. (org.apache.zookeeper.jmx.ManagedUtil)
[2021-03-24 19:03:08,904] INFO Starting quorum peer (org.apache.zookeeper.server.quorum.QuorumPeerMain)
[2021-03-24 19:03:08,909] INFO Using org.apache.zookeeper.server.NIOServerCnxnFactory as server connection factory (org.apache.zookeeper.server.ServerCnxnFactory)
[2021-03-24 19:03:08,917] INFO Setting -D jdk.tls.rejectClientInitiatedRenegotiation=true to disable client-initiated TLS renegotiation (org.apache.zookeeper.common.X509Util)
[2021-03-24 19:03:08,953] INFO Server successfully logged in. (org.apache.zookeeper.Login)
[2021-03-24 19:03:08,957] INFO Configuring NIO connection handler with 10s sessionless connection timeout, 1 selector thread(s), 8 worker threads, and 64 kB direct buffers. (org.apache.zookeeper.server.NIOServerCnxnFactory)
[2021-03-24 19:03:08,961] INFO binding to port 0.0.0.0/0.0.0.0:2181 (org.apache.zookeeper.server.NIOServerCnxnFactory)
[2021-03-24 19:03:08,986] INFO Logging initialized #929ms to org.eclipse.jetty.util.log.Slf4jLog (org.eclipse.jetty.util.log)
[2021-03-24 19:03:09,081] WARN o.e.j.s.ServletContextHandler#6c2c1385{/,null,UNAVAILABLE} contextPath ends with /* (org.eclipse.jetty.server.handler.ContextHandler)
[2021-03-24 19:03:09,082] WARN Empty contextPath (org.eclipse.jetty.server.handler.ContextHandler)
[2021-03-24 19:03:09,097] INFO zookeeper.snapshot.trust.empty : false (org.apache.zookeeper.server.persistence.FileTxnSnapLog)
[2021-03-24 19:03:09,102] INFO Local sessions disabled (org.apache.zookeeper.server.quorum.QuorumPeer)
[2021-03-24 19:03:09,102] INFO Local session upgrading disabled (org.apache.zookeeper.server.quorum.QuorumPeer)
[2021-03-24 19:03:09,102] INFO tickTime set to 3000 (org.apache.zookeeper.server.quorum.QuorumPeer)
[2021-03-24 19:03:09,102] INFO minSessionTimeout set to 6000 (org.apache.zookeeper.server.quorum.QuorumPeer)
[2021-03-24 19:03:09,102] INFO maxSessionTimeout set to 60000 (org.apache.zookeeper.server.quorum.QuorumPeer)
[2021-03-24 19:03:09,102] INFO initLimit set to 10 (org.apache.zookeeper.server.quorum.QuorumPeer)
[2021-03-24 19:03:09,115] INFO zookeeper.snapshotSizeFactor = 0.33 (org.apache.zookeeper.server.ZKDatabase)
[2021-03-24 19:03:09,116] INFO Using insecure (non-TLS) quorum communication (org.apache.zookeeper.server.quorum.QuorumPeer)
[2021-03-24 19:03:09,117] INFO Port unification disabled (org.apache.zookeeper.server.quorum.QuorumPeer)
[2021-03-24 19:03:09,117] INFO QuorumPeer communication is not secured! (SASL auth disabled) (org.apache.zookeeper.server.quorum.QuorumPeer)
[2021-03-24 19:03:09,117] INFO quorum.cnxn.threads.size set to 20 (org.apache.zookeeper.server.quorum.QuorumPeer)
[2021-03-24 19:03:09,118] INFO Reading snapshot /var/lib/zookeeper/data/version-2/snapshot.a00000000 (org.apache.zookeeper.server.persistence.FileSnap)
[2021-03-24 19:03:09,213] INFO jetty-9.4.24.v20191120; built: 2019-11-20T21:37:49.771Z; git: 363d5f2df3a8a28de40604320230664b9c793c16; jvm 11.0.9.1+1-LTS (org.eclipse.jetty.server.Server)
[2021-03-24 19:03:09,261] INFO DefaultSessionIdManager workerName=node0 (org.eclipse.jetty.server.session)
[2021-03-24 19:03:09,261] INFO No SessionScavenger set, using defaults (org.eclipse.jetty.server.session)
[2021-03-24 19:03:09,263] INFO node0 Scavenging every 660000ms (org.eclipse.jetty.server.session)
[2021-03-24 19:03:09,272] INFO Started o.e.j.s.ServletContextHandler#6c2c1385{/,null,AVAILABLE} (org.eclipse.jetty.server.handler.ContextHandler)
[2021-03-24 19:03:09,281] INFO Started ServerConnector#6d07a63d{HTTP/1.1,[http/1.1]}{0.0.0.0:8080} (org.eclipse.jetty.server.AbstractConnector)
[2021-03-24 19:03:09,281] INFO Started #1224ms (org.eclipse.jetty.server.Server)
[2021-03-24 19:03:09,281] INFO Started AdminServer on address 0.0.0.0, port 8080 and command URL /commands (org.apache.zookeeper.server.admin.JettyAdminServer)
[2021-03-24 19:03:09,288] INFO Election port bind maximum retries is 3 (org.apache.zookeeper.server.quorum.QuorumCnxManager)
[2021-03-24 19:03:09,290] INFO 1 is accepting connections now, my election bind port: devkafka04/172.16.87.141:3888 (org.apache.zookeeper.server.quorum.QuorumCnxManager)
[2021-03-24 19:03:09,301] INFO LOOKING (org.apache.zookeeper.server.quorum.QuorumPeer)
[2021-03-24 19:03:09,303] INFO New election. My id = 1, proposed zxid=0x1600000030 (org.apache.zookeeper.server.quorum.FastLeaderElection)
[2021-03-24 19:03:09,308] INFO Notification: 2 (message format version), 1 (n.leader), 0x1600000030 (n.zxid), 0x1 (n.round), LOOKING (n.state), 1 (n.sid), 0x16 (n.peerEPoch), LOOKING (my state)0 (n.config version) (org.apache.zookeeper.server.quorum.FastLeaderElection)
[2021-03-24 19:03:09,310] INFO Have smaller server identifier, so dropping the connection: (myId:1 --> sid:3) (org.apache.zookeeper.server.quorum.QuorumCnxManager)
[2021-03-24 19:03:09,312] INFO Received connection request from /172.16.87.143:53340 (org.apache.zookeeper.server.quorum.QuorumCnxManager)
[2021-03-24 19:03:09,315] INFO Have smaller server identifier, so dropping the connection: (myId:1 --> sid:2) (org.apache.zookeeper.server.quorum.QuorumCnxManager)
[2021-03-24 19:03:09,316] INFO Notification: 2 (message format version), 2 (n.leader), 0x150000002b (n.zxid), 0xa (n.round), FOLLOWING (n.state), 3 (n.sid), 0x16 (n.peerEPoch), LOOKING (my state)0 (n.config version) (org.apache.zookeeper.server.quorum.FastLeaderElection)
[2021-03-24 19:03:09,317] INFO Received connection request from /172.16.87.142:51704 (org.apache.zookeeper.server.quorum.QuorumCnxManager)
[2021-03-24 19:03:09,319] INFO Notification: 2 (message format version), 2 (n.leader), 0x150000002b (n.zxid), 0xa (n.round), LEADING (n.state), 2 (n.sid), 0x16 (n.peerEPoch), LOOKING (my state)0 (n.config version) (org.apache.zookeeper.server.quorum.FastLeaderElection)
[2021-03-24 19:03:09,320] INFO Notification: 2 (message format version), 2 (n.leader), 0x150000002b (n.zxid), 0xa (n.round), FOLLOWING (n.state), 3 (n.sid), 0x16 (n.peerEPoch), LOOKING (my state)0 (n.config version) (org.apache.zookeeper.server.quorum.FastLeaderElection)
[2021-03-24 19:03:09,320] INFO FOLLOWING (org.apache.zookeeper.server.quorum.QuorumPeer)
[2021-03-24 19:03:09,323] INFO Notification: 2 (message format version), 2 (n.leader), 0x150000002b (n.zxid), 0xa (n.round), LEADING (n.state), 2 (n.sid), 0x16 (n.peerEPoch), FOLLOWING (my state)0 (n.config version) (org.apache.zookeeper.server.quorum.FastLeaderElection)
[2021-03-24 19:03:09,330] INFO TCP NoDelay set to: true (org.apache.zookeeper.server.quorum.Learner)
[2021-03-24 19:03:09,336] INFO Server environment:zookeeper.version=3.5.8-f439ca583e70862c3068a1f2a7d4d068eec33315, built on 05/04/2020 15:53 GMT (org.apache.zookeeper.server.ZooKeeperServer)
[2021-03-24 19:03:09,336] INFO Server environment:host.name=devkafka04 (org.apache.zookeeper.server.ZooKeeperServer)
[2021-03-24 19:03:09,336] INFO Server environment:java.version=11.0.9.1 (org.apache.zookeeper.server.ZooKeeperServer)
[2021-03-24 19:03:09,336] INFO Server environment:java.vendor=Azul Systems, Inc. (org.apache.zookeeper.server.ZooKeeperServer)
[2021-03-24 19:03:09,336] INFO Server environment:java.home=/usr/lib/jvm/zulu11-ca (org.apache.zookeeper.server.ZooKeeperServer)
[2021-03-24 19:03:09,336] INFO Server environment:java.class.path=/usr/bin/../share/java/kafka/activation-1.1.1.jar:/usr/bin/../share/java/kafka/aopalliance-repackaged-2.6.1.jar:/usr/bin/../share/java/kafka/argparse4j-0.7.0.jar:/usr/bin/../share/java/kafka/audience-annotations-0.5.0.jar:/usr/bin/../share/java/kafka/commons-cli-1.4.jar:/usr/bin/../share/java/kafka/commons-lang3-3.8.1.jar:/usr/bin/../share/java/kafka/confluent-log4j-1.2.17-cp2.jar:/usr/bin/../share/java/kafka/connect-api-6.0.1-ccs.jar:/usr/bin/../share/java/kafka/connect-basic-auth-extension-6.0.1-ccs.jar:/usr/bin/../share/java/kafka/connect-file-6.0.1-ccs.jar:/usr/bin/../share/java/kafka/connect-json-6.0.1-ccs.jar:/usr/bin/../share/java/kafka/connect-mirror-6.0.1-ccs.jar:/usr/bin/../share/java/kafka/connect-mirror-client-6.0.1-ccs.jar:/usr/bin/../share/java/kafka/connect-runtime-6.0.1-ccs.jar:/usr/bin/../share/java/kafka/connect-transforms-6.0.1-ccs.jar:/usr/bin/../share/java/kafka/hk2-api-2.6.1.jar:/usr/bin/../share/java/kafka/hk2-locator-2.6.1.jar:/usr/bin/../share/java/kafka/hk2-utils-2.6.1.jar:/usr/bin/../share/java/kafka/jackson-annotations-2.10.5.jar:/usr/bin/../share/java/kafka/jackson-core-2.10.5.jar:/usr/bin/../share/java/kafka/jackson-databind-2.10.5.jar:/usr/bin/../share/java/kafka/jackson-dataformat-csv-2.10.5.jar:/usr/bin/../share/java/kafka/jackson-datatype-jdk8-2.10.5.jar:/usr/bin/../share/java/kafka/jackson-jaxrs-base-2.10.5.jar:/usr/bin/../share/java/kafka/jackson-jaxrs-json-provider-2.10.5.jar:/usr/bin/../share/java/kafka/jackson-module-jaxb-annotations-2.10.5.jar:/usr/bin/../share/java/kafka/jackson-module-paranamer-2.10.5.jar:/usr/bin/../share/java/kafka/jackson-module-scala_2.13-2.10.5.jar:/usr/bin/../share/java/kafka/jakarta.activation-api-1.2.1.jar:/usr/bin/../share/java/kafka/jakarta.annotation-api-1.3.5.jar:/usr/bin/../share/java/kafka/jakarta.inject-2.6.1.jar:/usr/bin/../share/java/kafka/jakarta.validation-api-2.0.2.jar:/usr/bin/../share/java/kafka/jakarta.ws.rs-api-2.1.6.jar:/usr/bin/../share/java/kafka/jakarta.xml.bind-api-2.3.2.jar:/usr/bin/../share/java/kafka/javassist-3.25.0-GA.jar:/usr/bin/../share/java/kafka/javassist-3.26.0-GA.jar:/usr/bin/../share/java/kafka/javax.servlet-api-3.1.0.jar:/usr/bin/../share/java/kafka/javax.ws.rs-api-2.1.1.jar:/usr/bin/../share/java/kafka/jaxb-api-2.3.0.jar:/usr/bin/../share/java/kafka/jersey-client-2.30.jar:/usr/bin/../share/java/kafka/jersey-common-2.30.jar:/usr/bin/../share/java/kafka/jersey-container-servlet-2.30.jar:/usr/bin/../share/java/kafka/jersey-container-servlet-core-2.30.jar:/usr/bin/../share/java/kafka/jersey-hk2-2.30.jar:/usr/bin/../share/java/kafka/jersey-media-jaxb-2.30.jar:/usr/bin/../share/java/kafka/jersey-server-2.30.jar:/usr/bin/../share/java/kafka/jetty-client-9.4.24.v20191120.jar:/usr/bin/../share/java/kafka/jetty-continuation-9.4.24.v20191120.jar:/usr/bin/../share/java/kafka/jetty-http-9.4.24.v20191120.jar:/usr/bin/../share/java/kafka/jetty-io-9.4.24.v20191120.jar:/usr/bin/../share/java/kafka/jetty-security-9.4.24.v20191120.jar:/usr/bin/../share/java/kafka/jetty-server-9.4.24.v20191120.jar:/usr/bin/../share/java/kafka/jetty-servlet-9.4.24.v20191120.jar:/usr/bin/../share/java/kafka/jetty-servlets-9.4.24.v20191120.jar:/usr/bin/../share/java/kafka/jetty-util-9.4.24.v20191120.jar:/usr/bin/../share/java/kafka/jopt-simple-5.0.4.jar:/usr/bin/../share/java/kafka/kafka-clients-6.0.1-ccs.jar:/usr/bin/../share/java/kafka/kafka-log4j-appender-6.0.1-ccs.jar:/usr/bin/../share/java/kafka/kafka-streams-6.0.1-ccs.jar:/usr/bin/../share/java/kafka/kafka-streams-examples-6.0.1-ccs.jar:/usr/bin/../share/java/kafka/kafka-streams-scala_2.13-6.0.1-ccs.jar:/usr/bin/../share/java/kafka/kafka-streams-test-utils-6.0.1-ccs.jar:/usr/bin/../share/java/kafka/kafka-tools-6.0.1-ccs.jar:/usr/bin/../share/java/kafka/kafka.jar:/usr/bin/../share/java/kafka/kafka_2.13-6.0.1-ccs-javadoc.jar:/usr/bin/../share/java/kafka/kafka_2.13-6.0.1-ccs-scaladoc.jar:/usr/bin/../share/java/kafka/kafka_2.13-6.0.1-ccs-sources.jar:/usr/bin/../share/java/kafka/kafka_2.13-6.0.1-ccs-test-sources.jar:/usr/bin/../share/java/kafka/kafka_2.13-6.0.1-ccs-test.jar:/usr/bin/../share/java/kafka/kafka_2.13-6.0.1-ccs.jar:/usr/bin/../share/java/kafka/lz4-java-1.7.1.jar:/usr/bin/../share/java/kafka/maven-artifact-3.6.3.jar:/usr/bin/../share/java/kafka/metrics-core-2.2.0.jar:/usr/bin/../share/java/kafka/netty-buffer-4.1.50.Final.jar:/usr/bin/../share/java/kafka/netty-codec-4.1.50.Final.jar:/usr/bin/../share/java/kafka/netty-common-4.1.50.Final.jar:/usr/bin/../share/java/kafka/netty-handler-4.1.50.Final.jar:/usr/bin/../share/java/kafka/netty-resolver-4.1.50.Final.jar:/usr/bin/../share/java/kafka/netty-transport-4.1.50.Final.jar:/usr/bin/../share/java/kafka/netty-transport-native-epoll-4.1.50.Final.jar:/usr/bin/../share/java/kafka/netty-transport-native-unix-common-4.1.50.Final.jar:/usr/bin/../share/java/kafka/osgi-resource-locator-1.0.3.jar:/usr/bin/../share/java/kafka/paranamer-2.8.jar:/usr/bin/../share/java/kafka/plexus-utils-3.2.1.jar:/usr/bin/../share/java/kafka/reflections-0.9.12.jar:/usr/bin/../share/java/kafka/rocksdbjni-5.18.4.jar:/usr/bin/../share/java/kafka/scala-collection-compat_2.13-2.1.6.jar:/usr/bin/../share/java/kafka/scala-java8-compat_2.13-0.9.1.jar:/usr/bin/../share/java/kafka/scala-library-2.13.2.jar:/usr/bin/../share/java/kafka/slf4j-api-1.7.30.jar:/usr/bin/../share/java/kafka/scala-logging_2.13-3.9.2.jar:/usr/bin/../share/java/kafka/scala-reflect-2.13.2.jar:/usr/bin/../share/java/kafka/slf4j-log4j12-1.7.30.jar:/usr/bin/../share/java/kafka/snappy-java-1.1.7.3.jar:/usr/bin/../share/java/kafka/zookeeper-3.5.8.jar:/usr/bin/../share/java/kafka/zookeeper-jute-3.5.8.jar:/usr/bin/../share/java/kafka/zstd-jni-1.4.4-7.jar:/usr/bin/../share/java/confluent-telemetry/* (org.apache.zookeeper.server.ZooKeeperServer)
[2021-03-24 19:03:09,336] INFO Server environment:java.library.path=/usr/java/packages/lib:/usr/lib64:/lib64:/lib:/usr/lib (org.apache.zookeeper.server.ZooKeeperServer)
[2021-03-24 19:03:09,336] INFO Server environment:java.io.tmpdir=/tmp (org.apache.zookeeper.server.ZooKeeperServer)
[2021-03-24 19:03:09,336] INFO Server environment:java.compiler=<NA> (org.apache.zookeeper.server.ZooKeeperServer)
[2021-03-24 19:03:09,337] INFO Server environment:os.name=Linux (org.apache.zookeeper.server.ZooKeeperServer)
[2021-03-24 19:03:09,337] INFO Server environment:os.arch=amd64 (org.apache.zookeeper.server.ZooKeeperServer)
[2021-03-24 19:03:09,337] INFO Server environment:os.version=3.10.0-1160.21.1.el7.x86_64 (org.apache.zookeeper.server.ZooKeeperServer)
[2021-03-24 19:03:09,337] INFO Server environment:user.name=appuser (org.apache.zookeeper.server.ZooKeeperServer)
[2021-03-24 19:03:09,337] INFO Server environment:user.home=/home/appuser (org.apache.zookeeper.server.ZooKeeperServer)
[2021-03-24 19:03:09,337] INFO Server environment:user.dir=/home/appuser (org.apache.zookeeper.server.ZooKeeperServer)
[2021-03-24 19:03:09,337] INFO Server environment:os.memory.free=498MB (org.apache.zookeeper.server.ZooKeeperServer)
[2021-03-24 19:03:09,337] INFO Server environment:os.memory.max=512MB (org.apache.zookeeper.server.ZooKeeperServer)
[2021-03-24 19:03:09,337] INFO Server environment:os.memory.total=512MB (org.apache.zookeeper.server.ZooKeeperServer)
[2021-03-24 19:03:09,338] INFO minSessionTimeout set to 6000 (org.apache.zookeeper.server.ZooKeeperServer)
[2021-03-24 19:03:09,339] INFO maxSessionTimeout set to 60000 (org.apache.zookeeper.server.ZooKeeperServer)
[2021-03-24 19:03:09,339] INFO Created server with tickTime 3000 minSessionTimeout 6000 maxSessionTimeout 60000 datadir /var/lib/zookeeper/log/version-2 snapdir /var/lib/zookeeper/data/version-2 (org.apache.zookeeper.server.ZooKeeperServer)
[2021-03-24 19:03:09,339] INFO FOLLOWING - LEADER ELECTION TOOK - 18 MS (org.apache.zookeeper.server.quorum.Learner)
[2021-03-24 19:03:09,345] INFO Getting a diff from the leader 0x1600000030 (org.apache.zookeeper.server.quorum.Learner)
[2021-03-24 19:03:09,350] INFO Learner received NEWLEADER message (org.apache.zookeeper.server.quorum.Learner)
[2021-03-24 19:03:09,363] INFO Learner received UPTODATE message (org.apache.zookeeper.server.quorum.Learner)
[2021-03-24 19:03:09,367] INFO Configuring CommitProcessor with 4 worker threads. (org.apache.zookeeper.server.quorum.CommitProcessor)
$ sudo docker logs kafka
===> User
uid=1000(appuser) gid=1000(appuser) groups=1000(appuser)
===> Configuring ...
SSL is enabled.
SASL is enabled.
===> Running preflight checks ...
===> Check if /var/lib/kafka/data is writable ...
===> Skipping Zookeeper health check for SSL connections...
===> Launching ...
===> Launching kafka ...
[2021-03-23 21:43:43,453] INFO Registered kafka:type=kafka.Log4jController MBean (kafka.utils.Log4jControllerRegistration$)
[2021-03-23 21:43:43,838] INFO Setting -D jdk.tls.rejectClientInitiatedRenegotiation=true to disable client-initiated TLS renegotiation (org.apache.zookeeper.common.X509Util)
[2021-03-23 21:43:43,900] INFO Registered signal handlers for TERM, INT, HUP (org.apache.kafka.common.utils.LoggingSignalHandler)
[2021-03-23 21:43:43,904] INFO starting (kafka.server.KafkaServer)
[2021-03-23 21:43:43,905] INFO Connecting to zookeeper on devkafka04:2182,devkafka05:2182,devkafka06:2182 (kafka.server.KafkaServer)
[2021-03-23 21:43:43,927] INFO [ZooKeeperClient Kafka server] Initializing a new session to devkafka04:2182,devkafka05:2182,devkafka06:2182. (kafka.zookeeper.ZooKeeperClient)
[2021-03-23 21:43:43,934] INFO Client environment:zookeeper.version=3.5.8-f439ca583e70862c3068a1f2a7d4d068eec33315, built on 05/04/2020 15:53 GMT (org.apache.zookeeper.ZooKeeper)
[2021-03-23 21:43:43,934] INFO Client environment:host.name=devkafka04 (org.apache.zookeeper.ZooKeeper)
[2021-03-23 21:43:43,934] INFO Client environment:java.version=11.0.9.1 (org.apache.zookeeper.ZooKeeper)
[2021-03-23 21:43:43,934] INFO Client environment:java.vendor=Azul Systems, Inc. (org.apache.zookeeper.ZooKeeper)
------ Repeating lines removed ---------
'Client' (org.apache.zookeeper.ClientCnxn)
[2021-03-23 21:43:59,947] INFO Socket error occurred: devkafka05/172.16.87.142:2182: Connection refused (org.apache.zookeeper.ClientCnxn)
[2021-03-23 21:44:01,048] INFO Client successfully logged in. (org.apache.zookeeper.Login)
[2021-03-23 21:44:01,048] INFO Client will use DIGEST-MD5 as SASL mechanism. (org.apache.zookeeper.client.ZooKeeperSaslClient)
[2021-03-23 21:44:01,048] INFO Opening socket connection to server devkafka04/172.16.87.141:2182. Will attempt to SASL-authenticate using Login Context section 'Client' (org.apache.zookeeper.ClientCnxn)
[2021-03-23 21:44:01,049] INFO Socket error occurred: devkafka04/172.16.87.141:2182: Connection refused (org.apache.zookeeper.ClientCnxn)
[2021-03-23 21:44:01,150] INFO Client successfully logged in. (org.apache.zookeeper.Login)
[2021-03-23 21:44:01,150] INFO Client will use DIGEST-MD5 as SASL mechanism. (org.apache.zookeeper.client.ZooKeeperSaslClient)
[2021-03-23 21:44:01,150] INFO Opening socket connection to server devkafka06/172.16.87.143:2182. Will attempt to SASL-authenticate using Login Context section 'Client' (org.apache.zookeeper.ClientCnxn)
[2021-03-23 21:44:01,153] INFO Socket error occurred: devkafka06/172.16.87.143:2182: Connection refused (org.apache.zookeeper.ClientCnxn)
[2021-03-23 21:44:01,254] INFO Client successfully logged in. (org.apache.zookeeper.Login)
[2021-03-23 21:44:01,254] INFO Client will use DIGEST-MD5 as SASL mechanism. (org.apache.zookeeper.client.ZooKeeperSaslClient)
[2021-03-23 21:44:01,254] INFO Opening socket connection to server devkafka05/172.16.87.142:2182. Will attempt to SASL-authenticate using Login Context section 'Client' (org.apache.zookeeper.ClientCnxn)
[2021-03-23 21:44:01,255] INFO Socket error occurred: devkafka05/172.16.87.142:2182: Connection refused (org.apache.zookeeper.ClientCnxn)
[2021-03-23 21:44:01,952] INFO [ZooKeeperClient Kafka server] Closing. (kafka.zookeeper.ZooKeeperClient)
[2021-03-23 21:44:02,356] INFO Client successfully logged in. (org.apache.zookeeper.Login)
[2021-03-23 21:44:02,357] INFO Client will use DIGEST-MD5 as SASL mechanism. (org.apache.zookeeper.client.ZooKeeperSaslClient)
[2021-03-23 21:44:02,357] INFO Opening socket connection to server devkafka04/172.16.87.141:2182. Will attempt to SASL-authenticate using Login Context section 'Client' (org.apache.zookeeper.ClientCnxn)
[2021-03-23 21:44:02,462] INFO Session: 0x0 closed (org.apache.zookeeper.ZooKeeper)
[2021-03-23 21:44:02,463] INFO EventThread shut down for session: 0x0 (org.apache.zookeeper.ClientCnxn)
[2021-03-23 21:44:02,465] INFO [ZooKeeperClient Kafka server] Closed. (kafka.zookeeper.ZooKeeperClient)
[2021-03-23 21:44:02,469] ERROR Fatal error during KafkaServer startup. Prepare to shutdown (kafka.server.KafkaServer)
kafka.zookeeper.ZooKeeperClientTimeoutException: Timed out waiting for connection while in state: CONNECTING
at kafka.zookeeper.ZooKeeperClient.waitUntilConnected(ZooKeeperClient.scala:262)
at kafka.zookeeper.ZooKeeperClient.<init>(ZooKeeperClient.scala:119)
at kafka.zk.KafkaZkClient$.apply(KafkaZkClient.scala:1865)
at kafka.server.KafkaServer.createZkClient$1(KafkaServer.scala:419)
at kafka.server.KafkaServer.initZkClient(KafkaServer.scala:444)
at kafka.server.KafkaServer.startup(KafkaServer.scala:222)
at kafka.server.KafkaServerStartable.startup(KafkaServerStartable.scala:44)
at kafka.Kafka$.main(Kafka.scala:82)
at kafka.Kafka.main(Kafka.scala)
[2021-03-23 21:44:02,471] INFO shutting down (kafka.server.KafkaServer)
[2021-03-23 21:44:02,478] INFO shut down completed (kafka.server.KafkaServer)
[2021-03-23 21:44:02,478] ERROR Exiting Kafka. (kafka.server.KafkaServerStartable)
[2021-03-23 21:44:02,479] INFO shutting down (kafka.server.KafkaServer)
$ sudo cat kafka-docker-compose.yml
version: '3'
services:
kafka:
image: confluentinc/cp-kafka:6.0.1
container_name: kafka
network_mode: host
restart: always
ports:
- "9092:9092"
- "9093:9093"
- "9094:9094"
- "49998:49998"
- "49999:49999"
environment:
KAFKA_BROKER_ID: 1
KAFKA_ZOOKEEPER_CONNECT: 'devkafka04:2182,devkafka05:2182,devkafka06:2182'
KAFKA_ZOOKEEPER_SSL_CLIENT_ENABLE: 'true'
KAFKA_ZOOKEEPER_CLIENTCNXNSOCKET: org.apache.zookeeper.ClientCnxnSocketNetty
KAFKA_ZOOKEEPER_SSL_TRUSTSTORE_FILENAME: kafka.server.truststore.jks
KAFKA_ZOOKEEPER_SSL_TRUSTSTORE_CREDENTIALS: creds
KAFKA_ZOOKEEPER_SET_ACL: 'true'
KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://devkafka04:9092,SSL://devkafka04:9093,SASL_SSL://devkafka04:9094
KAFKA_LISTENERS: PLAINTEXT://devkafka04:9092,SSL://devkafka04:9093,SASL_SSL://devkafka04:9094
KAFKA_SASL_ENABLED_MECHANISMS: DIGEST-MD5
KAFKA_SECURITY_INTER_BROKER_PROTOCOL: SSL
KAFKA_SSL_CLIENT_AUTH: requested
KAFKA_SSL_TRUSTSTORE_FILENAME: kafka.server.truststore.jks
KAFKA_SSL_TRUSTSTORE_CREDENTIALS: creds
KAFKA_SSL_KEYSTORE_FILENAME: devkafka04.server.keystore.jks
KAFKA_SSL_KEYSTORE_CREDENTIALS: creds
KAFKA_SSL_KEY_CREDENTIALS: creds
KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 3
KAFKA_GROUP_INITIAL_REBALANCE_DELAY_MS: 0
KAFKA_AUTO_CREATE_TOPICS_ENABLE: "false"
KAFKA_OPTS: -Djava.security.auth.login.config=/etc/kafka/jmx/kafka_server_jaas.conf -Djava.rmi.server.hostname=devkafka04 -Dcom.sun.management.jmxremote.local.only=false -Dcom.sun.management.jmxremote.rmi.port=49998 -Dcom.sun.management.jmxremote.port=49998 -Dcom.sun.management.jmxremote=true -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -javaagent:/etc/kafka/jmx/jmx_prometheus_javaagent-0.14.0.jar=49999:/etc/kafka/jmx/kafka-2_0_0.yml
CONFLUENT_SUPPORT_METRICS_ENABLE: "false"
volumes:
- /media/kafka/data:/var/lib/kafka/data
- /media/kafka/secrets:/etc/kafka/secrets
- /usr/local/src/kafka/jmx:/etc/kafka/jmx
$ sudo cat jmx/kafka_server_jaas.conf
KafkaServer {
org.apache.kafka.common.security.plain.PlainLoginModule required
username="kafkabroker"
password="kafkabroker-secret"
user_kafkabroker="kafkabroker-secret"
user_kafka-broker-metric-reporter="kafkabroker-metric-reporter-secret"
user_client="client-secret";
};
Client {
org.apache.zookeeper.server.auth.DigestLoginModule required
username="kafka"
password="kafka-secret";
};
$ sudo cat zookeeper-docker-compose.yml
version: '3'
services:
zookeeper:
image: confluentinc/cp-zookeeper:6.0.1
container_name: zookeeper
network_mode: host
restart: always
ports:
- "2181:2181"
- "2182:2182"
- "2888:2888"
- "3888:3888"
- "39998:39998"
- "39999:39999"
environment:
ZOOKEEPER_SERVER_ID: 1
ZOOKEEPER_CLIENT_PORT: 2181
ZOOKEEPER_SERVERS: devkafka04:2888:3888;devkafka05:2888:3888;devkafka06:2888:3888
ZOOKEEPER_AUTHPROVIDER_SASL: org.apache.zookeeper.server.auth.SASLAuthenticationProvider
ZOOKEEPER_AUTHPROVIDER_x509: org.apache.zookeeper.server.auth.X509AuthenticationProvider
ZOOKEEPER_SECURECLIENTPORT: 2182
ZOOKEEPER_SERVERCNXNFACTORY: org.apache.zookeeper.server.NettyServerCnxnFactory
ZOOKEEPER_SSL_TRUSTSTORE_FILENAME: kafka.server.truststore.jks
ZOOKEEPER_SSL_TRUSTSTORE_CREDENTIALS: creds
ZOOKEEPER_SSL_KEYSTORE_FILENAME: devkafka05.server.keystore.jks
ZOOKEEPER_SSL_KEYSTORE_CREDENTIALS: creds
ZOOKEEPER_SSL_KEY_CREDENTIALS: creds
ZOOKEEPER_SSL_CLIENTAUTH: none
KAFKA_OPTS: -Djava.security.auth.login.config=/etc/zookeeper/jmx/zookeeper_jaas.conf -Dzookeeper.4lw.commands.whitelist=* -Djava.rmi.server.hostname=devkafka04 -Dcom.sun.management.jmxremote.local.only=false -Dcom.sun.management.jmxremote.rmi.port=39998 -Dcom.sun.management.jmxremote.port=39998 -Dcom.sun.management.jmxremote=true -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -javaagent:/etc/zookeeper/jmx/jmx_prometheus_javaagent-0.14.0.jar=39999:/etc/zookeeper/jmx/jmx-zookeeper-prometheus.yaml
volumes:
- /media/zookeeper/data:/var/lib/zookeeper/data
- /media/zookeeper/log:/var/lib/zookeeper/log
- /media/zookeeper/secrets:/etc/zookeeper/secrets
- /usr/local/src/zookeeper/jmx:/etc/zookeeper/jmx
$ sudo cat jmx/zookeeper_jaas.conf
Server {
org.apache.zookeeper.server.auth.DigestLoginModule required
user_kafka="kafka-secret";
};
Try using KAFKA_ZOOKEEPER_CLIENT_CNXN_SOCKET instead of KAFKA_ZOOKEEPER_CLIENTCNXNSOCKET.
I am very new to using Microservices and having trouble running Kafka after I have started zookeeper.
Zookeeper starts fine but when I try to start my Kafka server it throws an error.
I have searched on google to try and solve my problem but its quite overwhelming, as I am not sure what all these different config files mean/do.
I have tried by enabling listeners=PLAINTEXT://:9092 in server settings but it doesn't work.
I have also tried to un and reinstalled Kafka and ZooKeeper but I still get the same error.
Antonias-MacBook-Pro:kafka antonia$ kafka-server-start /usr/local/etc/kafka/server.properties
[2020-02-25 10:26:55,944] INFO Registered kafka:type=kafka.Log4jController MBean (kafka.utils.Log4jControllerRegistration$)
[2020-02-25 10:26:56,514] INFO Registered signal handlers for TERM, INT, HUP (org.apache.kafka.common.utils.LoggingSignalHandler)
[2020-02-25 10:26:56,514] INFO starting (kafka.server.KafkaServer)
[2020-02-25 10:26:56,515] INFO Connecting to zookeeper on localhost:2181 (kafka.server.KafkaServer)
[2020-02-25 10:26:56,541] INFO [ZooKeeperClient Kafka server] Initializing a new session to localhost:2181. (kafka.zookeeper.ZooKeeperClient)
[2020-02-25 10:26:56,571] INFO Client environment:zookeeper.version=3.5.6-c11b7e26bc554b8523dc929761dd28808913f091, built on 10/08/2019 20:18 GMT (org.apache.zookeeper.ZooKeeper)
[2020-02-25 10:26:56,571] INFO Client environment:host.name=10.111.180.36 (org.apache.zookeeper.ZooKeeper)
[2020-02-25 10:26:56,571] INFO Client environment:java.version=1.8.0_242 (org.apache.zookeeper.ZooKeeper)
[2020-02-25 10:26:56,571] INFO Client environment:java.vendor=AdoptOpenJDK (org.apache.zookeeper.ZooKeeper)
[2020-02-25 10:26:56,571] INFO Client environment:java.home=/Library/Java/JavaVirtualMachines/adoptopenjdk-8.jdk/Contents/Home/jre (org.apache.zookeeper.ZooKeeper)
[2020-02-25 10:26:56,571] INFO Client environment:java.class.path=/usr/local/Cellar/kafka/2.4.0/libexec/bin/../libs/activation-1.1.1.jar:/usr/local/Cellar/kafka/2.4.0/libexec/bin/../libs/aopalliance-repackaged-2.5.0.jar:/usr/local/Cellar/kafka/2.4.0/libexec/bin/../libs/argparse4j-0.7.0.jar:/usr/local/Cellar/kafka/2.4.0/libexec/bin/../libs/audience-annotations-0.5.0.jar:/usr/local/Cellar/kafka/2.4.0/libexec/bin/../libs/commons-cli-1.4.jar:/usr/local/Cellar/kafka/2.4.0/libexec/bin/../libs/commons-lang3-3.8.1.jar:/usr/local/Cellar/kafka/2.4.0/libexec/bin/../libs/connect-api-2.4.0.jar:/usr/local/Cellar/kafka/2.4.0/libexec/bin/../libs/connect-basic-auth-extension-2.4.0.jar:/usr/local/Cellar/kafka/2.4.0/libexec/bin/../libs/connect-file-2.4.0.jar:/usr/local/Cellar/kafka/2.4.0/libexec/bin/../libs/connect-json-2.4.0.jar:/usr/local/Cellar/kafka/2.4.0/libexec/bin/../libs/connect-mirror-2.4.0.jar:/usr/local/Cellar/kafka/2.4.0/libexec/bin/../libs/connect-mirror-client-2.4.0.jar:/usr/local/Cellar/kafka/2.4.0/libexec/bin/../libs/connect-runtime-2.4.0.jar:/usr/local/Cellar/kafka/2.4.0/libexec/bin/../libs/connect-transforms-2.4.0.jar:/usr/local/Cellar/kafka/2.4.0/libexec/bin/../libs/guava-20.0.jar:/usr/local/Cellar/kafka/2.4.0/libexec/bin/../libs/hk2-api-2.5.0.jar:/usr/local/Cellar/kafka/2.4.0/libexec/bin/../libs/hk2-locator-2.5.0.jar:/usr/local/Cellar/kafka/2.4.0/libexec/bin/../libs/hk2-utils-2.5.0.jar:/usr/local/Cellar/kafka/2.4.0/libexec/bin/../libs/jackson-annotations-2.10.0.jar:/usr/local/Cellar/kafka/2.4.0/libexec/bin/../libs/jackson-core-2.10.0.jar:/usr/local/Cellar/kafka/2.4.0/libexec/bin/../libs/jackson-databind-2.10.0.jar:/usr/local/Cellar/kafka/2.4.0/libexec/bin/../libs/jackson-dataformat-csv-2.10.0.jar:/usr/local/Cellar/kafka/2.4.0/libexec/bin/../libs/jackson-datatype-jdk8-2.10.0.jar:/usr/local/Cellar/kafka/2.4.0/libexec/bin/../libs/jackson-jaxrs-base-2.10.0.jar:/usr/local/Cellar/kafka/2.4.0/libexec/bin/../libs/jackson-jaxrs-json-provider-2.10.0.jar:/usr/local/Cellar/kafka/2.4.0/libexec/bin/../libs/jackson-module-jaxb-annotations-2.10.0.jar:/usr/local/Cellar/kafka/2.4.0/libexec/bin/../libs/jackson-module-paranamer-2.10.0.jar:/usr/local/Cellar/kafka/2.4.0/libexec/bin/../libs/jackson-module-scala_2.12-2.10.0.jar:/usr/local/Cellar/kafka/2.4.0/libexec/bin/../libs/jakarta.activation-api-1.2.1.jar:/usr/local/Cellar/kafka/2.4.0/libexec/bin/../libs/jakarta.annotation-api-1.3.4.jar:/usr/local/Cellar/kafka/2.4.0/libexec/bin/../libs/jakarta.inject-2.5.0.jar:/usr/local/Cellar/kafka/2.4.0/libexec/bin/../libs/jakarta.ws.rs-api-2.1.5.jar:/usr/local/Cellar/kafka/2.4.0/libexec/bin/../libs/jakarta.xml.bind-api-2.3.2.jar:/usr/local/Cellar/kafka/2.4.0/libexec/bin/../libs/javassist-3.22.0-CR2.jar:/usr/local/Cellar/kafka/2.4.0/libexec/bin/../libs/javax.servlet-api-3.1.0.jar:/usr/local/Cellar/kafka/2.4.0/libexec/bin/../libs/javax.ws.rs-api-2.1.1.jar:/usr/local/Cellar/kafka/2.4.0/libexec/bin/../libs/jaxb-api-2.3.0.jar:/usr/local/Cellar/kafka/2.4.0/libexec/bin/../libs/jersey-client-2.28.jar:/usr/local/Cellar/kafka/2.4.0/libexec/bin/../libs/jersey-common-2.28.jar:/usr/local/Cellar/kafka/2.4.0/libexec/bin/../libs/jersey-container-servlet-2.28.jar:/usr/local/Cellar/kafka/2.4.0/libexec/bin/../libs/jersey-container-servlet-core-2.28.jar:/usr/local/Cellar/kafka/2.4.0/libexec/bin/../libs/jersey-hk2-2.28.jar:/usr/local/Cellar/kafka/2.4.0/libexec/bin/../libs/jersey-media-jaxb-2.28.jar:/usr/local/Cellar/kafka/2.4.0/libexec/bin/../libs/jersey-server-2.28.jar:/usr/local/Cellar/kafka/2.4.0/libexec/bin/../libs/jetty-client-9.4.20.v20190813.jar:/usr/local/Cellar/kafka/2.4.0/libexec/bin/../libs/jetty-continuation-9.4.20.v20190813.jar:/usr/local/Cellar/kafka/2.4.0/libexec/bin/../libs/jetty-http-9.4.20.v20190813.jar:/usr/local/Cellar/kafka/2.4.0/libexec/bin/../libs/jetty-io-9.4.20.v20190813.jar:/usr/local/Cellar/kafka/2.4.0/libexec/bin/../libs/jetty-security-9.4.20.v20190813.jar:/usr/local/Cellar/kafka/2.4.0/libexec/bin/../libs/jetty-server-9.4.20.v20190813.jar:/usr/local/Cellar/kafka/2.4.0/libexec/bin/../libs/jetty-servlet-9.4.20.v20190813.jar:/usr/local/Cellar/kafka/2.4.0/libexec/bin/../libs/jetty-servlets-9.4.20.v20190813.jar:/usr/local/Cellar/kafka/2.4.0/libexec/bin/../libs/jetty-util-9.4.20.v20190813.jar:/usr/local/Cellar/kafka/2.4.0/libexec/bin/../libs/jopt-simple-5.0.4.jar:/usr/local/Cellar/kafka/2.4.0/libexec/bin/../libs/kafka-clients-2.4.0.jar:/usr/local/Cellar/kafka/2.4.0/libexec/bin/../libs/kafka-log4j-appender-2.4.0.jar:/usr/local/Cellar/kafka/2.4.0/libexec/bin/../libs/kafka-streams-2.4.0.jar:/usr/local/Cellar/kafka/2.4.0/libexec/bin/../libs/kafka-streams-examples-2.4.0.jar:/usr/local/Cellar/kafka/2.4.0/libexec/bin/../libs/kafka-streams-scala_2.12-2.4.0.jar:/usr/local/Cellar/kafka/2.4.0/libexec/bin/../libs/kafka-streams-test-utils-2.4.0.jar:/usr/local/Cellar/kafka/2.4.0/libexec/bin/../libs/kafka-tools-2.4.0.jar:/usr/local/Cellar/kafka/2.4.0/libexec/bin/../libs/kafka_2.12-2.4.0-sources.jar:/usr/local/Cellar/kafka/2.4.0/libexec/bin/../libs/kafka_2.12-2.4.0.jar:/usr/local/Cellar/kafka/2.4.0/libexec/bin/../libs/log4j-1.2.17.jar:/usr/local/Cellar/kafka/2.4.0/libexec/bin/../libs/lz4-java-1.6.0.jar:/usr/local/Cellar/kafka/2.4.0/libexec/bin/../libs/maven-artifact-3.6.1.jar:/usr/local/Cellar/kafka/2.4.0/libexec/bin/../libs/metrics-core-2.2.0.jar:/usr/local/Cellar/kafka/2.4.0/libexec/bin/../libs/netty-buffer-4.1.42.Final.jar:/usr/local/Cellar/kafka/2.4.0/libexec/bin/../libs/netty-codec-4.1.42.Final.jar:/usr/local/Cellar/kafka/2.4.0/libexec/bin/../libs/netty-common-4.1.42.Final.jar:/usr/local/Cellar/kafka/2.4.0/libexec/bin/../libs/netty-handler-4.1.42.Final.jar:/usr/local/Cellar/kafka/2.4.0/libexec/bin/../libs/netty-resolver-4.1.42.Final.jar:/usr/local/Cellar/kafka/2.4.0/libexec/bin/../libs/netty-transport-4.1.42.Final.jar:/usr/local/Cellar/kafka/2.4.0/libexec/bin/../libs/netty-transport-native-epoll-4.1.42.Final.jar:/usr/local/Cellar/kafka/2.4.0/libexec/bin/../libs/netty-transport-native-unix-common-4.1.42.Final.jar:/usr/local/Cellar/kafka/2.4.0/libexec/bin/../libs/osgi-resource-locator-1.0.1.jar:/usr/local/Cellar/kafka/2.4.0/libexec/bin/../libs/paranamer-2.8.jar:/usr/local/Cellar/kafka/2.4.0/libexec/bin/../libs/plexus-utils-3.2.0.jar:/usr/local/Cellar/kafka/2.4.0/libexec/bin/../libs/reflections-0.9.11.jar:/usr/local/Cellar/kafka/2.4.0/libexec/bin/../libs/rocksdbjni-5.18.3.jar:/usr/local/Cellar/kafka/2.4.0/libexec/bin/../libs/scala-collection-compat_2.12-2.1.2.jar:/usr/local/Cellar/kafka/2.4.0/libexec/bin/../libs/scala-java8-compat_2.12-0.9.0.jar:/usr/local/Cellar/kafka/2.4.0/libexec/bin/../libs/scala-library-2.12.10.jar:/usr/local/Cellar/kafka/2.4.0/libexec/bin/../libs/scala-logging_2.12-3.9.2.jar:/usr/local/Cellar/kafka/2.4.0/libexec/bin/../libs/scala-reflect-2.12.10.jar:/usr/local/Cellar/kafka/2.4.0/libexec/bin/../libs/slf4j-api-1.7.28.jar:/usr/local/Cellar/kafka/2.4.0/libexec/bin/../libs/slf4j-log4j12-1.7.28.jar:/usr/local/Cellar/kafka/2.4.0/libexec/bin/../libs/snappy-java-1.1.7.3.jar:/usr/local/Cellar/kafka/2.4.0/libexec/bin/../libs/validation-api-2.0.1.Final.jar:/usr/local/Cellar/kafka/2.4.0/libexec/bin/../libs/zookeeper-3.5.6.jar:/usr/local/Cellar/kafka/2.4.0/libexec/bin/../libs/zookeeper-jute-3.5.6.jar:/usr/local/Cellar/kafka/2.4.0/libexec/bin/../libs/zstd-jni-1.4.3-1.jar (org.apache.zookeeper.ZooKeeper)
[2020-02-25 10:26:56,572] INFO Client environment:java.library.path=/Users/antonia/Library/Java/Extensions:/Library/Java/Extensions:/Network/Library/Java/Extensions:/System/Library/Java/Extensions:/usr/lib/java:. (org.apache.zookeeper.ZooKeeper)
[2020-02-25 10:26:56,572] INFO Client environment:java.io.tmpdir=/var/folders/13/gvhn5dqj4179ynmpm6pkp9kh0000gn/T/ (org.apache.zookeeper.ZooKeeper)
[2020-02-25 10:26:56,572] INFO Client environment:java.compiler=<NA> (org.apache.zookeeper.ZooKeeper)
[2020-02-25 10:26:56,572] INFO Client environment:os.name=Mac OS X (org.apache.zookeeper.ZooKeeper)
[2020-02-25 10:26:56,572] INFO Client environment:os.arch=x86_64 (org.apache.zookeeper.ZooKeeper)
[2020-02-25 10:26:56,573] INFO Client environment:os.version=10.14.5 (org.apache.zookeeper.ZooKeeper)
[2020-02-25 10:26:56,573] INFO Client environment:user.name=antonia (org.apache.zookeeper.ZooKeeper)
[2020-02-25 10:26:56,573] INFO Client environment:user.home=/Users/antonia (org.apache.zookeeper.ZooKeeper)
[2020-02-25 10:26:56,573] INFO Client environment:user.dir=/usr/local/etc/kafka (org.apache.zookeeper.ZooKeeper)
[2020-02-25 10:26:56,573] INFO Client environment:os.memory.free=977MB (org.apache.zookeeper.ZooKeeper)
[2020-02-25 10:26:56,573] INFO Client environment:os.memory.max=1024MB (org.apache.zookeeper.ZooKeeper)
[2020-02-25 10:26:56,573] INFO Client environment:os.memory.total=1024MB (org.apache.zookeeper.ZooKeeper)
[2020-02-25 10:26:56,575] INFO Initiating client connection, connectString=localhost:2181 sessionTimeout=6000 watcher=kafka.zookeeper.ZooKeeperClient$ZooKeeperClientWatcher$#28eaa59a (org.apache.zookeeper.ZooKeeper)
[2020-02-25 10:26:56,582] INFO Setting -D jdk.tls.rejectClientInitiatedRenegotiation=true to disable client-initiated TLS renegotiation (org.apache.zookeeper.common.X509Util)
[2020-02-25 10:26:56,592] INFO jute.maxbuffer value is 4194304 Bytes (org.apache.zookeeper.ClientCnxnSocket)
[2020-02-25 10:26:56,599] INFO zookeeper.request.timeout value is 0. feature enabled= (org.apache.zookeeper.ClientCnxn)
[2020-02-25 10:26:56,601] INFO [ZooKeeperClient Kafka server] Waiting until connected. (kafka.zookeeper.ZooKeeperClient)
[2020-02-25 10:26:56,605] INFO Opening socket connection to server localhost/0:0:0:0:0:0:0:1:2181. Will not attempt to authenticate using SASL (unknown error) (org.apache.zookeeper.ClientCnxn)
[2020-02-25 10:26:56,625] INFO Socket connection established, initiating session, client: /0:0:0:0:0:0:0:1:63152, server: localhost/0:0:0:0:0:0:0:1:2181 (org.apache.zookeeper.ClientCnxn)
[2020-02-25 10:26:56,641] INFO Session establishment complete on server localhost/0:0:0:0:0:0:0:1:2181, sessionid = 0x10004969600000a, negotiated timeout = 6000 (org.apache.zookeeper.ClientCnxn)
[2020-02-25 10:26:56,645] INFO [ZooKeeperClient Kafka server] Connected. (kafka.zookeeper.ZooKeeperClient)
[2020-02-25 10:26:56,979] INFO Cluster ID = S4SZ31nVRTCQ4uwRJ9_7mg (kafka.server.KafkaServer)
[2020-02-25 10:26:56,992] ERROR Fatal error during KafkaServer startup. Prepare to shutdown (kafka.server.KafkaServer)
kafka.common.InconsistentClusterIdException: The Cluster ID S4SZ31nVRTCQ4uwRJ9_7mg doesn't match stored clusterId Some(Y_mQi4q4TSuhlWdx4DHiaQ) in meta.properties. The broker is trying to join the wrong cluster. Configured zookeeper.connect may be wrong.
at kafka.server.KafkaServer.startup(KafkaServer.scala:220)
at kafka.server.KafkaServerStartable.startup(KafkaServerStartable.scala:44)
at kafka.Kafka$.main(Kafka.scala:84)
at kafka.Kafka.main(Kafka.scala)
[2020-02-25 10:26:56,995] INFO shutting down (kafka.server.KafkaServer)
[2020-02-25 10:26:56,997] INFO [ZooKeeperClient Kafka server] Closing. (kafka.zookeeper.ZooKeeperClient)
[2020-02-25 10:26:57,109] INFO Session: 0x10004969600000a closed (org.apache.zookeeper.ZooKeeper)
[2020-02-25 10:26:57,109] INFO EventThread shut down for session: 0x10004969600000a (org.apache.zookeeper.ClientCnxn)
[2020-02-25 10:26:57,111] INFO [ZooKeeperClient Kafka server] Closed. (kafka.zookeeper.ZooKeeperClient)
[2020-02-25 10:26:57,118] INFO shut down completed (kafka.server.KafkaServer)
[2020-02-25 10:26:57,119] ERROR Exiting Kafka. (kafka.server.KafkaServerStartable)
[2020-02-25 10:26:57,120] INFO shutting down (kafka.server.KafkaServer)
The cause of the problem is shown in this message:
kafka.common.InconsistentClusterIdException:
The Cluster ID S4SZ31nVRTCQ4uwRJ9_7mg
doesn't match stored clusterId Some(Y_mQi4q4TSuhlWdx4DHiaQ)
in meta.properties.
The broker is trying to join the wrong cluster.
Configured zookeeper.connect may be wrong.
The above problem occurs when a new instance of Kafka is being started up on data storage created by another kafka server. Kafka stores its messages in 'log' files.
How to fix the problem?
The problem can be fixed in these steps:
Shutdown both Kafka and Zookeeper
If required, take backup of the existing logs of Kafka and Zookeeper
Delete the log directories of both Kafka and Zookeeper
Restart Zookeeper and Kafka
I have just started learning Kafka/Zookeper. Issue I am facing is, when I try to run the console consumer to verify the message for a topic, I get the below error-
vagrant#debian-70rc1-x64-vbox4210:~$ $KAFKA_HOME/bin/kafka-console-consumer.sh --zookeeper 192.168.33.10:2181 --topic topic-3
[2015-02-22 22:56:36,666] INFO Verifying properties (kafka.utils.VerifiableProperties)
[2015-02-22 22:56:36,698] INFO Property group.id is overridden to console-consumer-92151 (kafka.utils.VerifiableProperties)
[2015-02-22 22:56:36,699] INFO Property fetch.message.max.bytes is overridden to 1048576 (kafka.utils.VerifiableProperties)
[2015-02-22 22:56:36,700] INFO Property consumer.timeout.ms is overridden to -1 (kafka.utils.VerifiableProperties)
[2015-02-22 22:56:36,700] INFO Property socket.timeout.ms is overridden to 30000 (kafka.utils.VerifiableProperties)
[2015-02-22 22:56:36,700] INFO Property auto.offset.reset is overridden to largest (kafka.utils.VerifiableProperties)
[2015-02-22 22:56:36,701] INFO Property zookeeper.connect is overridden to 192.168.33.10:2181 (kafka.utils.VerifiableProperties)
[2015-02-22 22:56:36,701] INFO Property socket.receive.buffer.bytes is overridden to 2097152 (kafka.utils.VerifiableProperties)
[2015-02-22 22:56:36,701] INFO Property fetch.wait.max.ms is overridden to 100 (kafka.utils.VerifiableProperties)
[2015-02-22 22:56:36,702] INFO Property auto.commit.enable is overridden to true (kafka.utils.VerifiableProperties)
[2015-02-22 22:56:36,702] INFO Property refresh.leader.backoff.ms is overridden to 200 (kafka.utils.VerifiableProperties)
[2015-02-22 22:56:36,702] INFO Property auto.commit.interval.ms is overridden to 60000 (kafka.utils.VerifiableProperties)
[2015-02-22 22:56:36,703] INFO Property fetch.min.bytes is overridden to 1 (kafka.utils.VerifiableProperties)
[2015-02-22 22:56:36,734] INFO [console-consumer-92151_debian-70rc1-x64-vbox4210-1424645796729-11446a89], Connecting to zookeeper instance at 192.168.33.10:2181 (kafka.consumer.ZookeeperConsumerConnector)
[2015-02-22 22:56:36,754] INFO Starting ZkClient event thread. (org.I0Itec.zkclient.ZkEventThread)
[2015-02-22 22:56:36,755] INFO Client environment:zookeeper.version=3.3.3-1203054, built on 11/17/2011 05:47 GMT (org.apache.zookeeper.ZooKeeper)
[2015-02-22 22:56:36,756] INFO Client environment:host.name=debian-70rc1-x64-vbox4210.vagrantup.com (org.apache.zookeeper.ZooKeeper)
[2015-02-22 22:56:36,756] INFO Client environment:java.version=1.7.0_75 (org.apache.zookeeper.ZooKeeper)
[2015-02-22 22:56:36,757] INFO Client environment:java.vendor=Oracle Corporation (org.apache.zookeeper.ZooKeeper)
[2015-02-22 22:56:36,757] INFO Client environment:java.home=/usr/lib/jvm/java-7-openjdk-amd64/jre (org.apache.zookeeper.ZooKeeper)
[2015-02-22 22:56:36,757] INFO Client environment:java.class.path=:/usr/local/kafka/kafka-0.8.0-beta1-src//bin/../core/target/scala-2.8.0/kafka_2.8.0-0.8.0-beta1.jar:/usr/local/kafka/kafka-0.8.0-beta1-src//bin/../core/target/scala-2.8.0/kafka-assembly-0.8.0-beta1-deps.jar:/usr/local/kafka/kafka-0.8.0-beta1-src//bin/../perf/target/scala-2.8.0/kafka-perf_2.8.0-0.8.0-beta1.jar:/usr/local/kafka/kafka-0.8.0-beta1-src//bin/../libs/*.jar:/usr/local/kafka/kafka-0.8.0-beta1-src//bin/../kafka*.jar (org.apache.zookeeper.ZooKeeper)
[2015-02-22 22:56:36,758] INFO Client environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib/x86_64-linux-gnu/jni:/lib/x86_64-linux-gnu:/usr/lib/x86_64-linux-gnu:/usr/lib/jni:/lib:/usr/lib (org.apache.zookeeper.ZooKeeper)
[2015-02-22 22:56:36,758] INFO Client environment:java.io.tmpdir=/tmp (org.apache.zookeeper.ZooKeeper)
[2015-02-22 22:56:36,758] INFO Client environment:java.compiler=<NA> (org.apache.zookeeper.ZooKeeper)
[2015-02-22 22:56:36,759] INFO Client environment:os.name=Linux (org.apache.zookeeper.ZooKeeper)
[2015-02-22 22:56:36,759] INFO Client environment:os.arch=amd64 (org.apache.zookeeper.ZooKeeper)
[2015-02-22 22:56:36,759] INFO Client environment:os.version=3.2.0-4-amd64 (org.apache.zookeeper.ZooKeeper)
[2015-02-22 22:56:36,759] INFO Client environment:user.name=vagrant (org.apache.zookeeper.ZooKeeper)
[2015-02-22 22:56:36,760] INFO Client environment:user.home=/home/vagrant (org.apache.zookeeper.ZooKeeper)
[2015-02-22 22:56:36,760] INFO Client environment:user.dir=/home/vagrant (org.apache.zookeeper.ZooKeeper)
[2015-02-22 22:56:36,761] INFO Initiating client connection, connectString=192.168.33.10:2181 sessionTimeout=6000 watcher=org.I0Itec.zkclient.ZkClient#234a98fa (org.apache.zookeeper.ZooKeeper)
[2015-02-22 22:56:36,773] INFO Opening socket connection to server /192.168.33.10:2181 (org.apache.zookeeper.ClientCnxn)
[2015-02-22 22:56:42,778] INFO Terminate ZkClient event thread. (org.I0Itec.zkclient.ZkEventThread)
[2015-02-22 22:56:46,791] INFO Socket connection established to 192.168.33.10/192.168.33.10:2181, initiating session (org.apache.zookeeper.ClientCnxn)
[2015-02-22 22:56:46,801] INFO Session establishment complete on server 192.168.33.10/192.168.33.10:2181, sessionid = 0x14bb3752996000d, negotiated timeout = 6000 (org.apache.zookeeper.ClientCnxn)
[2015-02-22 22:56:46,803] INFO Session: 0x14bb3752996000d closed (org.apache.zookeeper.ZooKeeper)
Exception in thread "main" org.I0Itec.zkclient.exception.ZkTimeoutException: Unable to connect to zookeeper server within timeout: 6000
at org.I0Itec.zkclient.ZkClient.connect(ZkClient.java:880)
at org.I0Itec.zkclient.ZkClient.<init>(ZkClient.java:98)
at org.I0Itec.zkclient.ZkClient.<init>(ZkClient.java:84)
at kafka.consumer.ZookeeperConsumerConnector.connectZk(ZookeeperConsumerConnector.scala:151)
at kafka.consumer.ZookeeperConsumerConnector.<init>(ZookeeperConsumerConnector.scala:112)
at kafka.consumer.ZookeeperConsumerConnector.<init>(ZookeeperConsumerConnector.scala:123)
at kafka.consumer.Consumer$.create(ConsumerConnector.scala:89)
at kafka.consumer.ConsoleConsumer$.main(ConsoleConsumer.scala:178)
at kafka.consumer.ConsoleConsumer.main(ConsoleConsumer.scala)
[2015-02-22 22:56:46,804] INFO EventThread shut down (org.apache.zookeeper.ClientCnxn)
It says-
Exception in thread "main" org.I0Itec.zkclient.exception.ZkTimeoutException: Unable to connect to zookeeper server within timeout: 6000
I verified the topic and it is present. What is going wrong. Please help.
I had the same problem and here is how I fixed it:
1) Stop all Kafka and Zookeeper processes
ps -aux | grep zoo
ps -aux | grep kafka
(then proceed to kill all process ids from above jobs)
2) Run zookeeper
<kafka_dir>/bin/zookeeper-server-start.sh ../config/zookeeper.properties
3) Run kafka server
<kafka_dir>/bin/kafka-server-start.sh ../config/server.properties
Try -> telnet 192.168.33.10 2181
If the telnet fails, then the issue is due to below reason.
Zookeeper Server NRPE iptables chain does not appear to a have a default accept rule.
Unless you just want your own iptables chain for NRPE, you could use:
iptables -I INPUT -s 0/0 -p tcp --dport 2181 -j ACCEPT
This solves the issue.
You can try the follwoing to see if it helps ..
1) Make sure the zookeeper is up and running, try to connect to it using the zkCli client .
2) append --from-beginning at the end of your command to see if you get any diifference in the output , so the command should be changed to :
$KAFKA_HOME/bin/kafka-console-consumer.sh --zookeeper xxx.xx.xx.xxx:2181 --topic topic-3 --from-beginning
In my case it was already connected to the host from which I was trying, so it was not able to make any new connection to the host. I had to change the config: maxClientCnxns so that it was able to make more connection to that host. More details here.
I am fairly new to kafka and would like input as to whether the following exception indicates an unhealthy kafka execution environment, or instead may be ignored:
java.util.NoSuchElementException: key not found: \tmp\kafka-logs
Here are the contents of that directory. Note each of the "-0" directories correspond to one of the five topics on this kafka server. e.g. calls-0 refers to the "calls" topic:
C:\apps\kafka_2.9.2-0.8.1\bin>dir \tmp\kafka-logs
Volume in drive C is Windows7_OS
Volume Serial Number is AC58-16CE
Directory of C:\tmp\kafka-logs
05/26/2014 02:11 PM <DIR> .
05/26/2014 02:11 PM <DIR> ..
05/24/2014 11:03 PM 0 .lock
05/24/2014 11:15 PM <DIR> calls-0
05/24/2014 11:31 PM <DIR> calls_no_phone-0
05/24/2014 11:31 PM <DIR> calls_online-0
05/24/2014 11:31 PM <DIR> dropped_calls-0
05/26/2014 02:11 PM 94 recovery-point-offset-checkpoint
05/24/2014 11:03 PM 0 replication-offset-checkpoint
05/24/2014 11:31 PM <DIR> vip_ips_alerts-0
The system is running Kafak version 0.8.1 for scala 2.9.2 on windows 7 using jdk 7
Here is the full startup console output (yes it is long .. but I chose to retain it since am unsure where an important clue may lie for the more experienced of you out there):
Picked up _JAVA_OPTIONS: -Djava.net.preferIPv4Stack=true
log4j:ERROR Failed to rename [logs/server.log] to [logs/server.log.2014-05-25-05].
[2014-05-26 14:05:22,767] INFO Verifying properties (kafka.utils.VerifiableProperties)
[2014-05-26 14:05:22,971] INFO Property broker.id is overridden to 0 (kafka.utils.VerifiableProperties)
[2014-05-26 14:05:22,972] INFO Property log.cleaner.enable is overridden to false (kafka.utils.VerifiableProperties)
[2014-05-26 14:05:22,972] INFO Property log.dirs is overridden to /tmp/kafka-logs (kafka.utils.VerifiableProperties)
[2014-05-26 14:05:22,973] INFO Property log.retention.check.interval.ms is overridden to 60000 (kafka.utils.VerifiableProperties)
[2014-05-26 14:05:22,973] INFO Property log.retention.hours is overridden to 168 (kafka.utils.VerifiableProperties)
[2014-05-26 14:05:22,974] INFO Property log.segment.bytes is overridden to 536870912 (kafka.utils.VerifiableProperties)
[2014-05-26 14:05:22,974] INFO Property num.io.threads is overridden to 8 (kafka.utils.VerifiableProperties)
[2014-05-26 14:05:22,975] INFO Property num.network.threads is overridden to 2 (kafka.utils.VerifiableProperties)
[2014-05-26 14:05:22,975] INFO Property num.partitions is overridden to 2 (kafka.utils.VerifiableProperties)
[2014-05-26 14:05:22,976] INFO Property port is overridden to 9092 (kafka.utils.VerifiableProperties)
[2014-05-26 14:05:22,976] INFO Property socket.receive.buffer.bytes is overridden to 1048576 (kafka.utils.VerifiableProperties)
[2014-05-26 14:05:22,977] INFO Property socket.request.max.bytes is overridden to 104857600 (kafka.utils.VerifiableProperties)
[2014-05-26 14:05:22,977] INFO Property socket.send.buffer.bytes is overridden to 1048576 (kafka.utils.VerifiableProperties)
[2014-05-26 14:05:22,978] INFO Property zookeeper.connect is overridden to localhost:2181 (kafka.utils.VerifiableProperties)
[2014-05-26 14:05:22,978] INFO Property zookeeper.connection.timeout.ms is overridden to 1000000 (kafka.utils.VerifiableProperties)
[2014-05-26 14:05:23,024] INFO [Kafka Server 0], starting (kafka.server.KafkaServer)
[2014-05-26 14:05:23,045] INFO [Kafka Server 0], Connecting to zookeeper on localhost:2181 (kafka.server.KafkaServer)
[2014-05-26 14:05:23,073] INFO Starting ZkClient event thread. (org.I0Itec.zkclient.ZkEventThread)
[2014-05-26 14:05:23,092] INFO Client environment:zookeeper.version=3.3.3-1203054, built on 11/17/2011 05:47 GMT (org.apache.zookeeper.ZooKeeper)
[2014-05-26 14:05:23,092] INFO Client environment:host.name=S80035683-SC01.china.huawei.com (org.apache.zookeeper.ZooKeeper)
[2014-05-26 14:05:23,093] INFO Client environment:java.version=1.7.0_51 (org.apache.zookeeper.ZooKeeper)
[2014-05-26 14:05:23,095] INFO Client environment:java.vendor=Oracle Corporation (org.apache.zookeeper.ZooKeeper)
[2014-05-26 14:05:23,096] INFO Client environment:java.home=c:\java7\jre (org.apache.zookeeper.ZooKeeper)
[2014-05-26 14:05:23,097] INFO Client environment:java.class.path=C:\apps\kafka_2.9.2-0.8.1\libs\jopt-simple-3.2.jar;C:\apps\kafka_2.9.2-0.8.1\libs\ka
fka_2.9.2-0.8.1.jar;C:\apps\kafka_2.9.2-0.8.1\libs\log4j-1.2.15.jar;C:\apps\kafka_2.9.2-0.8.1\libs\metrics-annotation-2.2.0.jar;C:\apps\kafka_2.9.2-0.
8.1\libs\metrics-core-2.2.0.jar;C:\apps\kafka_2.9.2-0.8.1\libs\scala-library-2.9.2.jar;C:\apps\kafka_2.9.2-0.8.1\libs\slf4j-api-1.7.2.jar;C:\apps\kafk
a_2.9.2-0.8.1\libs\snappy-java-1.0.5.jar;C:\apps\kafka_2.9.2-0.8.1\libs\zkclient-0.3.jar;C:\apps\kafka_2.9.2-0.8.1\libs\zookeeper-3.3.4.jar;;C:\Users\
S80035683\.ivy2\cache/org.xerial.snappy/snappy-java/bundles/snappy-java-1.0.5.jar;C:\Users\S80035683\.ivy2\cache/org.scala-lang/scala-library/jars/sca
la-library-2.8.0.jar;C:\Users\S80035683\.ivy2\cache/org.scala-lang/scala-compiler/jars/scala-compiler-2.8.0.jar;C:\Users\S80035683\.ivy2\cache/log4j/l
og4j/jars/log4j-1.2.15.jar;C:\Users\S80035683\.ivy2\cache/org.slf4j/slf4j-api/jars/slf4j-api-1.6.4.jar;C:\Users\S80035683\.ivy2\cache/org.apache.zooke
eper/zookeeper/jars/zookeeper-3.3.4.jar;C:\Users\S80035683\.ivy2\cache/net.sf.jopt-simple/jopt-simple/jars/jopt-simple-3.2.jar (org.apache.zookeeper.Z
ooKeeper)
[2014-05-26 14:05:23,099] INFO Client environment:java.library.path=c:\java7\bin;C:\Windows\Sun\Java\bin;C:\Windows\system32;C:\Windows;C:\shared\ConE
mu;c:\python27;c:\java7\bin;C:\Program Files\Git\cmd;C:\Program Files\Git\bin;c:\apps\sbt\bin;C:\apps\apache-maven-3.2.1\bin;c:\cygwin\bin;c:\cygwin\l
ib\awk;c:\java\bin;.;c:\shared;c:\scripts;c:\shared;C:\Program Files\eSpace-ecs;C:\Windows\system32;C:\Windows;C:\Windows\System32\Wbem;C:\Windows\Sys
tem32\WindowsPowerShell\v1.0\;C:\Program Files\Common Files\Lenovo;C:\Program Files\Common Files\Ulead Systems\MPEG;C:\SWTOOLS\ReadyApps;C:\Program Fi
les\Riverbed\Steelhead Mobile\;C:\scala\bin;C:\apps\sbt\\bin;;C:\apps\OpenSSH\bin;C:\apps\sbt13\\bin;C:\apps\nodejs\;C:\Program Files\IDM Computer Sol
utions\UltraEdit\;C:\Users\S80035683\AppData\Roaming\npm;. (org.apache.zookeeper.ZooKeeper)
[2014-05-26 14:05:23,100] INFO Client environment:java.io.tmpdir=C:\Users\S80035~1\AppData\Local\Temp\ (org.apache.zookeeper.ZooKeeper)
[2014-05-26 14:05:23,100] INFO Client environment:java.compiler=<NA> (org.apache.zookeeper.ZooKeeper)
[2014-05-26 14:05:23,100] INFO Client environment:os.name=Windows 7 (org.apache.zookeeper.ZooKeeper)
[2014-05-26 14:05:23,100] INFO Client environment:os.arch=x86 (org.apache.zookeeper.ZooKeeper)
[2014-05-26 14:05:23,100] INFO Client environment:os.version=6.1 (org.apache.zookeeper.ZooKeeper)
[2014-05-26 14:05:23,101] INFO Client environment:user.name=s80035683 (org.apache.zookeeper.ZooKeeper)
[2014-05-26 14:05:23,101] INFO Client environment:user.home=C:\Users\S80035683 (org.apache.zookeeper.ZooKeeper)
[2014-05-26 14:05:23,101] INFO Client environment:user.dir=C:\apps\kafka_2.9.2-0.8.1\bin (org.apache.zookeeper.ZooKeeper)
[2014-05-26 14:05:23,102] INFO Initiating client connection, connectString=localhost:2181 sessionTimeout=6000 watcher=org.I0Itec.zkclient.ZkClient#5ea
8af (org.apache.zookeeper.ZooKeeper)
[2014-05-26 14:05:23,125] INFO Opening socket connection to server localhost/127.0.0.1:2181 (org.apache.zookeeper.ClientCnxn)
[2014-05-26 14:05:23,127] INFO Socket connection established to localhost/127.0.0.1:2181, initiating session (org.apache.zookeeper.ClientCnxn)
[2014-05-26 14:05:23,172] INFO Session establishment complete on server localhost/127.0.0.1:2181, sessionid = 0x1463a5a1b370000, negotiated timeout =
6000 (org.apache.zookeeper.ClientCnxn)
[2014-05-26 14:05:23,174] INFO zookeeper state changed (SyncConnected) (org.I0Itec.zkclient.ZkClient)
[2014-05-26 14:05:23,620] INFO Found clean shutdown file. Skipping recovery for all logs in data directory 'C:\tmp\kafka-logs' (kafka.log.LogManager)
[2014-05-26 14:05:23,637] INFO Loading log 'calls-0' (kafka.log.LogManager)
[2014-05-26 14:05:23,778] WARN Error when freeing index buffer (kafka.log.OffsetIndex)
java.lang.NullPointerException
at kafka.log.OffsetIndex.kafka$log$OffsetIndex$$forceUnmap(OffsetIndex.scala:301)
at kafka.log.OffsetIndex$$anonfun$resize$1.apply(OffsetIndex.scala:283)
at kafka.log.OffsetIndex$$anonfun$resize$1.apply(OffsetIndex.scala:276)
at kafka.utils.Utils$.inLock(Utils.scala:538)
at kafka.log.OffsetIndex.resize(OffsetIndex.scala:276)
at kafka.log.Log.loadSegments(Log.scala:154)
at kafka.log.Log.<init>(Log.scala:64)
at kafka.log.LogManager$$anonfun$loadLogs$1$$anonfun$apply$4.apply(LogManager.scala:118)
at kafka.log.LogManager$$anonfun$loadLogs$1$$anonfun$apply$4.apply(LogManager.scala:113)
at scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:34)
at scala.collection.mutable.ArrayOps.foreach(ArrayOps.scala:38)
at kafka.log.LogManager$$anonfun$loadLogs$1.apply(LogManager.scala:113)
at kafka.log.LogManager$$anonfun$loadLogs$1.apply(LogManager.scala:105)
at scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:34)
at scala.collection.mutable.WrappedArray.foreach(WrappedArray.scala:33)
at kafka.log.LogManager.loadLogs(LogManager.scala:105)
at kafka.log.LogManager.<init>(LogManager.scala:57)
at kafka.server.KafkaServer.createLogManager(KafkaServer.scala:275)
at kafka.server.KafkaServer.startup(KafkaServer.scala:72)
at kafka.server.KafkaServerStartable.startup(KafkaServerStartable.scala:34)
at kafka.Kafka$.main(Kafka.scala:46)
at kafka.Kafka.main(Kafka.scala)
[2014-05-26 14:05:23,807] INFO Completed load of log calls-0 with log end offset 0 (kafka.log.Log)
SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
SLF4J: Defaulting to no-operation (NOP) logger implementation
SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.
[2014-05-26 14:05:23,828] INFO Loading log 'calls_no_phone-0' (kafka.log.LogManager)
[2014-05-26 14:05:23,831] WARN Error when freeing index buffer (kafka.log.OffsetIndex)
java.lang.NullPointerException
at kafka.log.OffsetIndex.kafka$log$OffsetIndex$$forceUnmap(OffsetIndex.scala:301)
at kafka.log.OffsetIndex$$anonfun$resize$1.apply(OffsetIndex.scala:283)
at kafka.log.OffsetIndex$$anonfun$resize$1.apply(OffsetIndex.scala:276)
at kafka.utils.Utils$.inLock(Utils.scala:538)
at kafka.log.OffsetIndex.resize(OffsetIndex.scala:276)
at kafka.log.Log.loadSegments(Log.scala:154)
at kafka.log.Log.<init>(Log.scala:64)
at kafka.log.LogManager$$anonfun$loadLogs$1$$anonfun$apply$4.apply(LogManager.scala:118)
at kafka.log.LogManager$$anonfun$loadLogs$1$$anonfun$apply$4.apply(LogManager.scala:113)
at scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:34)
at scala.collection.mutable.ArrayOps.foreach(ArrayOps.scala:38)
at kafka.log.LogManager$$anonfun$loadLogs$1.apply(LogManager.scala:113)
at kafka.log.LogManager$$anonfun$loadLogs$1.apply(LogManager.scala:105)
at scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:34)
at scala.collection.mutable.WrappedArray.foreach(WrappedArray.scala:33)
at kafka.log.LogManager.loadLogs(LogManager.scala:105)
at kafka.log.LogManager.<init>(LogManager.scala:57)
at kafka.server.KafkaServer.createLogManager(KafkaServer.scala:275)
at kafka.server.KafkaServer.startup(KafkaServer.scala:72)
at kafka.server.KafkaServerStartable.startup(KafkaServerStartable.scala:34)
at kafka.Kafka$.main(Kafka.scala:46)
at kafka.Kafka.main(Kafka.scala)
[2014-05-26 14:05:23,836] INFO Completed load of log calls_no_phone-0 with log end offset 0 (kafka.log.Log)
[2014-05-26 14:05:23,838] INFO Loading log 'calls_online-0' (kafka.log.LogManager)
[2014-05-26 14:05:23,845] WARN Error when freeing index buffer (kafka.log.OffsetIndex)
java.lang.NullPointerException
at kafka.log.OffsetIndex.kafka$log$OffsetIndex$$forceUnmap(OffsetIndex.scala:301)
at kafka.log.OffsetIndex$$anonfun$resize$1.apply(OffsetIndex.scala:283)
at kafka.log.OffsetIndex$$anonfun$resize$1.apply(OffsetIndex.scala:276)
at kafka.utils.Utils$.inLock(Utils.scala:538)
at kafka.log.OffsetIndex.resize(OffsetIndex.scala:276)
at kafka.log.Log.loadSegments(Log.scala:154)
at kafka.log.Log.<init>(Log.scala:64)
at kafka.log.LogManager$$anonfun$loadLogs$1$$anonfun$apply$4.apply(LogManager.scala:118)
at kafka.log.LogManager$$anonfun$loadLogs$1$$anonfun$apply$4.apply(LogManager.scala:113)
at scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:34)
at scala.collection.mutable.ArrayOps.foreach(ArrayOps.scala:38)
at kafka.log.LogManager$$anonfun$loadLogs$1.apply(LogManager.scala:113)
at kafka.log.LogManager$$anonfun$loadLogs$1.apply(LogManager.scala:105)
at scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:34)
at scala.collection.mutable.WrappedArray.foreach(WrappedArray.scala:33)
at kafka.log.LogManager.loadLogs(LogManager.scala:105)
at kafka.log.LogManager.<init>(LogManager.scala:57)
at kafka.server.KafkaServer.createLogManager(KafkaServer.scala:275)
at kafka.server.KafkaServer.startup(KafkaServer.scala:72)
at kafka.server.KafkaServerStartable.startup(KafkaServerStartable.scala:34)
at kafka.Kafka$.main(Kafka.scala:46)
at kafka.Kafka.main(Kafka.scala)
[2014-05-26 14:05:23,862] INFO Completed load of log calls_online-0 with log end offset 0 (kafka.log.Log)
[2014-05-26 14:05:23,863] INFO Loading log 'dropped_calls-0' (kafka.log.LogManager)
[2014-05-26 14:05:23,866] WARN Error when freeing index buffer (kafka.log.OffsetIndex)
java.lang.NullPointerException
at kafka.log.OffsetIndex.kafka$log$OffsetIndex$$forceUnmap(OffsetIndex.scala:301)
at kafka.log.OffsetIndex$$anonfun$resize$1.apply(OffsetIndex.scala:283)
at kafka.log.OffsetIndex$$anonfun$resize$1.apply(OffsetIndex.scala:276)
at kafka.utils.Utils$.inLock(Utils.scala:538)
at kafka.log.OffsetIndex.resize(OffsetIndex.scala:276)
at kafka.log.Log.loadSegments(Log.scala:154)
at kafka.log.Log.<init>(Log.scala:64)
at kafka.log.LogManager$$anonfun$loadLogs$1$$anonfun$apply$4.apply(LogManager.scala:118)
at kafka.log.LogManager$$anonfun$loadLogs$1$$anonfun$apply$4.apply(LogManager.scala:113)
at scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:34)
at scala.collection.mutable.ArrayOps.foreach(ArrayOps.scala:38)
at kafka.log.LogManager$$anonfun$loadLogs$1.apply(LogManager.scala:113)
at kafka.log.LogManager$$anonfun$loadLogs$1.apply(LogManager.scala:105)
at scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:34)
at scala.collection.mutable.WrappedArray.foreach(WrappedArray.scala:33)
at kafka.log.LogManager.loadLogs(LogManager.scala:105)
at kafka.log.LogManager.<init>(LogManager.scala:57)
at kafka.server.KafkaServer.createLogManager(KafkaServer.scala:275)
at kafka.server.KafkaServer.startup(KafkaServer.scala:72)
at kafka.server.KafkaServerStartable.startup(KafkaServerStartable.scala:34)
at kafka.Kafka$.main(Kafka.scala:46)
at kafka.Kafka.main(Kafka.scala)
[2014-05-26 14:05:23,869] INFO Completed load of log dropped_calls-0 with log end offset 0 (kafka.log.Log)
[2014-05-26 14:05:23,871] INFO Loading log 'vip_ips_alerts-0' (kafka.log.LogManager)
[2014-05-26 14:05:23,879] WARN Error when freeing index buffer (kafka.log.OffsetIndex)
java.lang.NullPointerException
at kafka.log.OffsetIndex.kafka$log$OffsetIndex$$forceUnmap(OffsetIndex.scala:301)
at kafka.log.OffsetIndex$$anonfun$resize$1.apply(OffsetIndex.scala:283)
at kafka.log.OffsetIndex$$anonfun$resize$1.apply(OffsetIndex.scala:276)
at kafka.utils.Utils$.inLock(Utils.scala:538)
at kafka.log.OffsetIndex.resize(OffsetIndex.scala:276)
at kafka.log.Log.loadSegments(Log.scala:154)
at kafka.log.Log.<init>(Log.scala:64)
at kafka.log.LogManager$$anonfun$loadLogs$1$$anonfun$apply$4.apply(LogManager.scala:118)
at kafka.log.LogManager$$anonfun$loadLogs$1$$anonfun$apply$4.apply(LogManager.scala:113)
at scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:34)
at scala.collection.mutable.ArrayOps.foreach(ArrayOps.scala:38)
at kafka.log.LogManager$$anonfun$loadLogs$1.apply(LogManager.scala:113)
at kafka.log.LogManager$$anonfun$loadLogs$1.apply(LogManager.scala:105)
at scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:34)
at scala.collection.mutable.WrappedArray.foreach(WrappedArray.scala:33)
at kafka.log.LogManager.loadLogs(LogManager.scala:105)
at kafka.log.LogManager.<init>(LogManager.scala:57)
at kafka.server.KafkaServer.createLogManager(KafkaServer.scala:275)
at kafka.server.KafkaServer.startup(KafkaServer.scala:72)
at kafka.server.KafkaServerStartable.startup(KafkaServerStartable.scala:34)
at kafka.Kafka$.main(Kafka.scala:46)
at kafka.Kafka.main(Kafka.scala)
[2014-05-26 14:05:23,885] INFO Completed load of log vip_ips_alerts-0 with log end offset 0 (kafka.log.Log)
[2014-05-26 14:05:23,888] INFO Starting log cleanup with a period of 60000 ms. (kafka.log.LogManager)
[2014-05-26 14:05:23,895] INFO Starting log flusher with a default period of 9223372036854775807 ms. (kafka.log.LogManager)
[2014-05-26 14:05:23,959] INFO Awaiting socket connections on 0.0.0.0:9092. (kafka.network.Acceptor)
[2014-05-26 14:05:23,961] INFO [Socket Server on Broker 0], Started (kafka.network.SocketServer)
[2014-05-26 14:05:24,155] INFO Will not load MX4J, mx4j-tools.jar is not in the classpath (kafka.utils.Mx4jLoader$)
[2014-05-26 14:05:24,219] INFO 0 successfully elected as leader (kafka.server.ZookeeperLeaderElector)
[2014-05-26 14:05:24,532] INFO New leader is 0 (kafka.server.ZookeeperLeaderElector$LeaderChangeListener)
[2014-05-26 14:05:24,539] INFO Registered broker 0 at path /brokers/ids/0 with address S80035683-SC01.china.huawei.com:9092. (kafka.utils.ZkUtils$)
[2014-05-26 14:05:24,551] INFO [Kafka Server 0], started (kafka.server.KafkaServer)
[2014-05-26 14:05:24,755] INFO [ReplicaFetcherManager on broker 0] Removed fetcher for partitions [vip_ips_alerts,0],[calls,0],[dropped_calls,0],[call
s_online,0],[calls_no_phone,0] (kafka.server.ReplicaFetcherManager)
[2014-05-26 14:05:24,771] ERROR [KafkaApi-0] error when handling request Name:LeaderAndIsrRequest;Version:0;Controller:0;ControllerEpoch:2;Correlation
Id:4;ClientId:id_0-host_null-port_9092;Leaders:id:0,host:S80035683-SC01.china.huawei.com,port:9092;PartitionState:(vip_ips_alerts,0) -> (LeaderAndIsrI
nfo:(Leader:0,ISR:0,LeaderEpoch:0,ControllerEpoch:1),ReplicationFactor:1),AllReplicas:0),(calls,0) -> (LeaderAndIsrInfo:(Leader:0,ISR:0,LeaderEpoch:0,
ControllerEpoch:1),ReplicationFactor:1),AllReplicas:0),(dropped_calls,0) -> (LeaderAndIsrInfo:(Leader:0,ISR:0,LeaderEpoch:0,ControllerEpoch:1),Replica
tionFactor:1),AllReplicas:0),(calls_online,0) -> (LeaderAndIsrInfo:(Leader:0,ISR:0,LeaderEpoch:0,ControllerEpoch:1),ReplicationFactor:1),AllReplicas:0
),(calls_no_phone,0) -> (LeaderAndIsrInfo:(Leader:0,ISR:0,LeaderEpoch:0,ControllerEpoch:1),ReplicationFactor:1),AllReplicas:0) (kafka.server.KafkaApis
)
java.util.NoSuchElementException: key not found: \tmp\kafka-logs
at scala.collection.MapLike$class.default(MapLike.scala:225)
at scala.collection.immutable.Map$Map1.default(Map.scala:107)
at scala.collection.MapLike$class.apply(MapLike.scala:135)
at scala.collection.immutable.Map$Map1.apply(Map.scala:107)
at kafka.cluster.Partition.getOrCreateReplica(Partition.scala:91)
at kafka.cluster.Partition$$anonfun$makeLeader$2.apply(Partition.scala:175)
at kafka.cluster.Partition$$anonfun$makeLeader$2.apply(Partition.scala:175)
at scala.collection.immutable.Set$Set1.foreach(Set.scala:86)
at kafka.cluster.Partition.makeLeader(Partition.scala:175)
at kafka.server.ReplicaManager$$anonfun$makeLeaders$5.apply(ReplicaManager.scala:305)
at kafka.server.ReplicaManager$$anonfun$makeLeaders$5.apply(ReplicaManager.scala:304)
at scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:95)
at scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:95)
at scala.collection.Iterator$class.foreach(Iterator.scala:772)
at scala.collection.mutable.HashTable$$anon$1.foreach(HashTable.scala:157)
at scala.collection.mutable.HashTable$class.foreachEntry(HashTable.scala:190)
at scala.collection.mutable.HashMap.foreachEntry(HashMap.scala:45)
at scala.collection.mutable.HashMap.foreach(HashMap.scala:95)
at kafka.server.ReplicaManager.makeLeaders(ReplicaManager.scala:304)
at kafka.server.ReplicaManager.becomeLeaderOrFollower(ReplicaManager.scala:258)
at kafka.server.KafkaApis.handleLeaderAndIsrRequest(KafkaApis.scala:100)
at kafka.server.KafkaApis.handle(KafkaApis.scala:72)
at kafka.server.KafkaRequestHandler.run(KafkaRequestHandler.scala:42)
at java.lang.Thread.run(Thread.java:744)
[2014-05-26 14:05:24,779] INFO [ReplicaFetcherManager on broker 0] Removed fetcher for partitions [vip_ips_alerts,0],[calls,0],[dropped_calls,0],[call
s_online,0],[calls_no_phone,0] (kafka.server.ReplicaFetcherManager)
[2014-05-26 14:05:24,787] ERROR [KafkaApi-0] error when handling request Name:LeaderAndIsrRequest;Version:0;Controller:0;ControllerEpoch:2;Correlation
Id:5;ClientId:id_0-host_null-port_9092;Leaders:id:0,host:S80035683-SC01.china.huawei.com,port:9092;PartitionState:(vip_ips_alerts,0) -> (LeaderAndIsrI
nfo:(Leader:0,ISR:0,LeaderEpoch:1,ControllerEpoch:2),ReplicationFactor:1),AllReplicas:0),(calls,0) -> (LeaderAndIsrInfo:(Leader:0,ISR:0,LeaderEpoch:1,
ControllerEpoch:2),ReplicationFactor:1),AllReplicas:0),(dropped_calls,0) -> (LeaderAndIsrInfo:(Leader:0,ISR:0,LeaderEpoch:1,ControllerEpoch:2),Replica
tionFactor:1),AllReplicas:0),(calls_online,0) -> (LeaderAndIsrInfo:(Leader:0,ISR:0,LeaderEpoch:1,ControllerEpoch:2),ReplicationFactor:1),AllReplicas:0
),(calls_no_phone,0) -> (LeaderAndIsrInfo:(Leader:0,ISR:0,LeaderEpoch:1,ControllerEpoch:2),ReplicationFactor:1),AllReplicas:0) (kafka.server.KafkaApis
)
java.util.NoSuchElementException: key not found: \tmp\kafka-logs
at scala.collection.MapLike$class.default(MapLike.scala:225)
at scala.collection.immutable.Map$Map1.default(Map.scala:107)
at scala.collection.MapLike$class.apply(MapLike.scala:135)
at scala.collection.immutable.Map$Map1.apply(Map.scala:107)
at kafka.cluster.Partition.getOrCreateReplica(Partition.scala:91)
at kafka.cluster.Partition$$anonfun$makeLeader$2.apply(Partition.scala:175)
at kafka.cluster.Partition$$anonfun$makeLeader$2.apply(Partition.scala:175)
at scala.collection.immutable.Set$Set1.foreach(Set.scala:86)
at kafka.cluster.Partition.makeLeader(Partition.scala:175)
at kafka.server.ReplicaManager$$anonfun$makeLeaders$5.apply(ReplicaManager.scala:305)
at kafka.server.ReplicaManager$$anonfun$makeLeaders$5.apply(ReplicaManager.scala:304)
at scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:95)
at scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:95)
at scala.collection.Iterator$class.foreach(Iterator.scala:772)
at scala.collection.mutable.HashTable$$anon$1.foreach(HashTable.scala:157)
at scala.collection.mutable.HashTable$class.foreachEntry(HashTable.scala:190)
at scala.collection.mutable.HashMap.foreachEntry(HashMap.scala:45)
at scala.collection.mutable.HashMap.foreach(HashMap.scala:95)
at kafka.server.ReplicaManager.makeLeaders(ReplicaManager.scala:304)
at kafka.server.ReplicaManager.becomeLeaderOrFollower(ReplicaManager.scala:258)
at kafka.server.KafkaApis.handleLeaderAndIsrRequest(KafkaApis.scala:100)
at kafka.server.KafkaApis.handle(KafkaApis.scala:72)
at kafka.server.KafkaRequestHandler.run(KafkaRequestHandler.scala:42)
at java.lang.Thread.run(Thread.java:744)
You need to change the log dir to a windows path, something like C:\tmp\kafka-logs
The bat files for kafka 0.8.1 are full of errors. Here is a very helpful blog I followed to get it up and running on windows. He gives the link to a git repo with fixed bat files.