I am fairly new to kafka and would like input as to whether the following exception indicates an unhealthy kafka execution environment, or instead may be ignored:
java.util.NoSuchElementException: key not found: \tmp\kafka-logs
Here are the contents of that directory. Note each of the "-0" directories correspond to one of the five topics on this kafka server. e.g. calls-0 refers to the "calls" topic:
C:\apps\kafka_2.9.2-0.8.1\bin>dir \tmp\kafka-logs
Volume in drive C is Windows7_OS
Volume Serial Number is AC58-16CE
Directory of C:\tmp\kafka-logs
05/26/2014 02:11 PM <DIR> .
05/26/2014 02:11 PM <DIR> ..
05/24/2014 11:03 PM 0 .lock
05/24/2014 11:15 PM <DIR> calls-0
05/24/2014 11:31 PM <DIR> calls_no_phone-0
05/24/2014 11:31 PM <DIR> calls_online-0
05/24/2014 11:31 PM <DIR> dropped_calls-0
05/26/2014 02:11 PM 94 recovery-point-offset-checkpoint
05/24/2014 11:03 PM 0 replication-offset-checkpoint
05/24/2014 11:31 PM <DIR> vip_ips_alerts-0
The system is running Kafak version 0.8.1 for scala 2.9.2 on windows 7 using jdk 7
Here is the full startup console output (yes it is long .. but I chose to retain it since am unsure where an important clue may lie for the more experienced of you out there):
Picked up _JAVA_OPTIONS: -Djava.net.preferIPv4Stack=true
log4j:ERROR Failed to rename [logs/server.log] to [logs/server.log.2014-05-25-05].
[2014-05-26 14:05:22,767] INFO Verifying properties (kafka.utils.VerifiableProperties)
[2014-05-26 14:05:22,971] INFO Property broker.id is overridden to 0 (kafka.utils.VerifiableProperties)
[2014-05-26 14:05:22,972] INFO Property log.cleaner.enable is overridden to false (kafka.utils.VerifiableProperties)
[2014-05-26 14:05:22,972] INFO Property log.dirs is overridden to /tmp/kafka-logs (kafka.utils.VerifiableProperties)
[2014-05-26 14:05:22,973] INFO Property log.retention.check.interval.ms is overridden to 60000 (kafka.utils.VerifiableProperties)
[2014-05-26 14:05:22,973] INFO Property log.retention.hours is overridden to 168 (kafka.utils.VerifiableProperties)
[2014-05-26 14:05:22,974] INFO Property log.segment.bytes is overridden to 536870912 (kafka.utils.VerifiableProperties)
[2014-05-26 14:05:22,974] INFO Property num.io.threads is overridden to 8 (kafka.utils.VerifiableProperties)
[2014-05-26 14:05:22,975] INFO Property num.network.threads is overridden to 2 (kafka.utils.VerifiableProperties)
[2014-05-26 14:05:22,975] INFO Property num.partitions is overridden to 2 (kafka.utils.VerifiableProperties)
[2014-05-26 14:05:22,976] INFO Property port is overridden to 9092 (kafka.utils.VerifiableProperties)
[2014-05-26 14:05:22,976] INFO Property socket.receive.buffer.bytes is overridden to 1048576 (kafka.utils.VerifiableProperties)
[2014-05-26 14:05:22,977] INFO Property socket.request.max.bytes is overridden to 104857600 (kafka.utils.VerifiableProperties)
[2014-05-26 14:05:22,977] INFO Property socket.send.buffer.bytes is overridden to 1048576 (kafka.utils.VerifiableProperties)
[2014-05-26 14:05:22,978] INFO Property zookeeper.connect is overridden to localhost:2181 (kafka.utils.VerifiableProperties)
[2014-05-26 14:05:22,978] INFO Property zookeeper.connection.timeout.ms is overridden to 1000000 (kafka.utils.VerifiableProperties)
[2014-05-26 14:05:23,024] INFO [Kafka Server 0], starting (kafka.server.KafkaServer)
[2014-05-26 14:05:23,045] INFO [Kafka Server 0], Connecting to zookeeper on localhost:2181 (kafka.server.KafkaServer)
[2014-05-26 14:05:23,073] INFO Starting ZkClient event thread. (org.I0Itec.zkclient.ZkEventThread)
[2014-05-26 14:05:23,092] INFO Client environment:zookeeper.version=3.3.3-1203054, built on 11/17/2011 05:47 GMT (org.apache.zookeeper.ZooKeeper)
[2014-05-26 14:05:23,092] INFO Client environment:host.name=S80035683-SC01.china.huawei.com (org.apache.zookeeper.ZooKeeper)
[2014-05-26 14:05:23,093] INFO Client environment:java.version=1.7.0_51 (org.apache.zookeeper.ZooKeeper)
[2014-05-26 14:05:23,095] INFO Client environment:java.vendor=Oracle Corporation (org.apache.zookeeper.ZooKeeper)
[2014-05-26 14:05:23,096] INFO Client environment:java.home=c:\java7\jre (org.apache.zookeeper.ZooKeeper)
[2014-05-26 14:05:23,097] INFO Client environment:java.class.path=C:\apps\kafka_2.9.2-0.8.1\libs\jopt-simple-3.2.jar;C:\apps\kafka_2.9.2-0.8.1\libs\ka
fka_2.9.2-0.8.1.jar;C:\apps\kafka_2.9.2-0.8.1\libs\log4j-1.2.15.jar;C:\apps\kafka_2.9.2-0.8.1\libs\metrics-annotation-2.2.0.jar;C:\apps\kafka_2.9.2-0.
8.1\libs\metrics-core-2.2.0.jar;C:\apps\kafka_2.9.2-0.8.1\libs\scala-library-2.9.2.jar;C:\apps\kafka_2.9.2-0.8.1\libs\slf4j-api-1.7.2.jar;C:\apps\kafk
a_2.9.2-0.8.1\libs\snappy-java-1.0.5.jar;C:\apps\kafka_2.9.2-0.8.1\libs\zkclient-0.3.jar;C:\apps\kafka_2.9.2-0.8.1\libs\zookeeper-3.3.4.jar;;C:\Users\
S80035683\.ivy2\cache/org.xerial.snappy/snappy-java/bundles/snappy-java-1.0.5.jar;C:\Users\S80035683\.ivy2\cache/org.scala-lang/scala-library/jars/sca
la-library-2.8.0.jar;C:\Users\S80035683\.ivy2\cache/org.scala-lang/scala-compiler/jars/scala-compiler-2.8.0.jar;C:\Users\S80035683\.ivy2\cache/log4j/l
og4j/jars/log4j-1.2.15.jar;C:\Users\S80035683\.ivy2\cache/org.slf4j/slf4j-api/jars/slf4j-api-1.6.4.jar;C:\Users\S80035683\.ivy2\cache/org.apache.zooke
eper/zookeeper/jars/zookeeper-3.3.4.jar;C:\Users\S80035683\.ivy2\cache/net.sf.jopt-simple/jopt-simple/jars/jopt-simple-3.2.jar (org.apache.zookeeper.Z
ooKeeper)
[2014-05-26 14:05:23,099] INFO Client environment:java.library.path=c:\java7\bin;C:\Windows\Sun\Java\bin;C:\Windows\system32;C:\Windows;C:\shared\ConE
mu;c:\python27;c:\java7\bin;C:\Program Files\Git\cmd;C:\Program Files\Git\bin;c:\apps\sbt\bin;C:\apps\apache-maven-3.2.1\bin;c:\cygwin\bin;c:\cygwin\l
ib\awk;c:\java\bin;.;c:\shared;c:\scripts;c:\shared;C:\Program Files\eSpace-ecs;C:\Windows\system32;C:\Windows;C:\Windows\System32\Wbem;C:\Windows\Sys
tem32\WindowsPowerShell\v1.0\;C:\Program Files\Common Files\Lenovo;C:\Program Files\Common Files\Ulead Systems\MPEG;C:\SWTOOLS\ReadyApps;C:\Program Fi
les\Riverbed\Steelhead Mobile\;C:\scala\bin;C:\apps\sbt\\bin;;C:\apps\OpenSSH\bin;C:\apps\sbt13\\bin;C:\apps\nodejs\;C:\Program Files\IDM Computer Sol
utions\UltraEdit\;C:\Users\S80035683\AppData\Roaming\npm;. (org.apache.zookeeper.ZooKeeper)
[2014-05-26 14:05:23,100] INFO Client environment:java.io.tmpdir=C:\Users\S80035~1\AppData\Local\Temp\ (org.apache.zookeeper.ZooKeeper)
[2014-05-26 14:05:23,100] INFO Client environment:java.compiler=<NA> (org.apache.zookeeper.ZooKeeper)
[2014-05-26 14:05:23,100] INFO Client environment:os.name=Windows 7 (org.apache.zookeeper.ZooKeeper)
[2014-05-26 14:05:23,100] INFO Client environment:os.arch=x86 (org.apache.zookeeper.ZooKeeper)
[2014-05-26 14:05:23,100] INFO Client environment:os.version=6.1 (org.apache.zookeeper.ZooKeeper)
[2014-05-26 14:05:23,101] INFO Client environment:user.name=s80035683 (org.apache.zookeeper.ZooKeeper)
[2014-05-26 14:05:23,101] INFO Client environment:user.home=C:\Users\S80035683 (org.apache.zookeeper.ZooKeeper)
[2014-05-26 14:05:23,101] INFO Client environment:user.dir=C:\apps\kafka_2.9.2-0.8.1\bin (org.apache.zookeeper.ZooKeeper)
[2014-05-26 14:05:23,102] INFO Initiating client connection, connectString=localhost:2181 sessionTimeout=6000 watcher=org.I0Itec.zkclient.ZkClient#5ea
8af (org.apache.zookeeper.ZooKeeper)
[2014-05-26 14:05:23,125] INFO Opening socket connection to server localhost/127.0.0.1:2181 (org.apache.zookeeper.ClientCnxn)
[2014-05-26 14:05:23,127] INFO Socket connection established to localhost/127.0.0.1:2181, initiating session (org.apache.zookeeper.ClientCnxn)
[2014-05-26 14:05:23,172] INFO Session establishment complete on server localhost/127.0.0.1:2181, sessionid = 0x1463a5a1b370000, negotiated timeout =
6000 (org.apache.zookeeper.ClientCnxn)
[2014-05-26 14:05:23,174] INFO zookeeper state changed (SyncConnected) (org.I0Itec.zkclient.ZkClient)
[2014-05-26 14:05:23,620] INFO Found clean shutdown file. Skipping recovery for all logs in data directory 'C:\tmp\kafka-logs' (kafka.log.LogManager)
[2014-05-26 14:05:23,637] INFO Loading log 'calls-0' (kafka.log.LogManager)
[2014-05-26 14:05:23,778] WARN Error when freeing index buffer (kafka.log.OffsetIndex)
java.lang.NullPointerException
at kafka.log.OffsetIndex.kafka$log$OffsetIndex$$forceUnmap(OffsetIndex.scala:301)
at kafka.log.OffsetIndex$$anonfun$resize$1.apply(OffsetIndex.scala:283)
at kafka.log.OffsetIndex$$anonfun$resize$1.apply(OffsetIndex.scala:276)
at kafka.utils.Utils$.inLock(Utils.scala:538)
at kafka.log.OffsetIndex.resize(OffsetIndex.scala:276)
at kafka.log.Log.loadSegments(Log.scala:154)
at kafka.log.Log.<init>(Log.scala:64)
at kafka.log.LogManager$$anonfun$loadLogs$1$$anonfun$apply$4.apply(LogManager.scala:118)
at kafka.log.LogManager$$anonfun$loadLogs$1$$anonfun$apply$4.apply(LogManager.scala:113)
at scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:34)
at scala.collection.mutable.ArrayOps.foreach(ArrayOps.scala:38)
at kafka.log.LogManager$$anonfun$loadLogs$1.apply(LogManager.scala:113)
at kafka.log.LogManager$$anonfun$loadLogs$1.apply(LogManager.scala:105)
at scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:34)
at scala.collection.mutable.WrappedArray.foreach(WrappedArray.scala:33)
at kafka.log.LogManager.loadLogs(LogManager.scala:105)
at kafka.log.LogManager.<init>(LogManager.scala:57)
at kafka.server.KafkaServer.createLogManager(KafkaServer.scala:275)
at kafka.server.KafkaServer.startup(KafkaServer.scala:72)
at kafka.server.KafkaServerStartable.startup(KafkaServerStartable.scala:34)
at kafka.Kafka$.main(Kafka.scala:46)
at kafka.Kafka.main(Kafka.scala)
[2014-05-26 14:05:23,807] INFO Completed load of log calls-0 with log end offset 0 (kafka.log.Log)
SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
SLF4J: Defaulting to no-operation (NOP) logger implementation
SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.
[2014-05-26 14:05:23,828] INFO Loading log 'calls_no_phone-0' (kafka.log.LogManager)
[2014-05-26 14:05:23,831] WARN Error when freeing index buffer (kafka.log.OffsetIndex)
java.lang.NullPointerException
at kafka.log.OffsetIndex.kafka$log$OffsetIndex$$forceUnmap(OffsetIndex.scala:301)
at kafka.log.OffsetIndex$$anonfun$resize$1.apply(OffsetIndex.scala:283)
at kafka.log.OffsetIndex$$anonfun$resize$1.apply(OffsetIndex.scala:276)
at kafka.utils.Utils$.inLock(Utils.scala:538)
at kafka.log.OffsetIndex.resize(OffsetIndex.scala:276)
at kafka.log.Log.loadSegments(Log.scala:154)
at kafka.log.Log.<init>(Log.scala:64)
at kafka.log.LogManager$$anonfun$loadLogs$1$$anonfun$apply$4.apply(LogManager.scala:118)
at kafka.log.LogManager$$anonfun$loadLogs$1$$anonfun$apply$4.apply(LogManager.scala:113)
at scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:34)
at scala.collection.mutable.ArrayOps.foreach(ArrayOps.scala:38)
at kafka.log.LogManager$$anonfun$loadLogs$1.apply(LogManager.scala:113)
at kafka.log.LogManager$$anonfun$loadLogs$1.apply(LogManager.scala:105)
at scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:34)
at scala.collection.mutable.WrappedArray.foreach(WrappedArray.scala:33)
at kafka.log.LogManager.loadLogs(LogManager.scala:105)
at kafka.log.LogManager.<init>(LogManager.scala:57)
at kafka.server.KafkaServer.createLogManager(KafkaServer.scala:275)
at kafka.server.KafkaServer.startup(KafkaServer.scala:72)
at kafka.server.KafkaServerStartable.startup(KafkaServerStartable.scala:34)
at kafka.Kafka$.main(Kafka.scala:46)
at kafka.Kafka.main(Kafka.scala)
[2014-05-26 14:05:23,836] INFO Completed load of log calls_no_phone-0 with log end offset 0 (kafka.log.Log)
[2014-05-26 14:05:23,838] INFO Loading log 'calls_online-0' (kafka.log.LogManager)
[2014-05-26 14:05:23,845] WARN Error when freeing index buffer (kafka.log.OffsetIndex)
java.lang.NullPointerException
at kafka.log.OffsetIndex.kafka$log$OffsetIndex$$forceUnmap(OffsetIndex.scala:301)
at kafka.log.OffsetIndex$$anonfun$resize$1.apply(OffsetIndex.scala:283)
at kafka.log.OffsetIndex$$anonfun$resize$1.apply(OffsetIndex.scala:276)
at kafka.utils.Utils$.inLock(Utils.scala:538)
at kafka.log.OffsetIndex.resize(OffsetIndex.scala:276)
at kafka.log.Log.loadSegments(Log.scala:154)
at kafka.log.Log.<init>(Log.scala:64)
at kafka.log.LogManager$$anonfun$loadLogs$1$$anonfun$apply$4.apply(LogManager.scala:118)
at kafka.log.LogManager$$anonfun$loadLogs$1$$anonfun$apply$4.apply(LogManager.scala:113)
at scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:34)
at scala.collection.mutable.ArrayOps.foreach(ArrayOps.scala:38)
at kafka.log.LogManager$$anonfun$loadLogs$1.apply(LogManager.scala:113)
at kafka.log.LogManager$$anonfun$loadLogs$1.apply(LogManager.scala:105)
at scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:34)
at scala.collection.mutable.WrappedArray.foreach(WrappedArray.scala:33)
at kafka.log.LogManager.loadLogs(LogManager.scala:105)
at kafka.log.LogManager.<init>(LogManager.scala:57)
at kafka.server.KafkaServer.createLogManager(KafkaServer.scala:275)
at kafka.server.KafkaServer.startup(KafkaServer.scala:72)
at kafka.server.KafkaServerStartable.startup(KafkaServerStartable.scala:34)
at kafka.Kafka$.main(Kafka.scala:46)
at kafka.Kafka.main(Kafka.scala)
[2014-05-26 14:05:23,862] INFO Completed load of log calls_online-0 with log end offset 0 (kafka.log.Log)
[2014-05-26 14:05:23,863] INFO Loading log 'dropped_calls-0' (kafka.log.LogManager)
[2014-05-26 14:05:23,866] WARN Error when freeing index buffer (kafka.log.OffsetIndex)
java.lang.NullPointerException
at kafka.log.OffsetIndex.kafka$log$OffsetIndex$$forceUnmap(OffsetIndex.scala:301)
at kafka.log.OffsetIndex$$anonfun$resize$1.apply(OffsetIndex.scala:283)
at kafka.log.OffsetIndex$$anonfun$resize$1.apply(OffsetIndex.scala:276)
at kafka.utils.Utils$.inLock(Utils.scala:538)
at kafka.log.OffsetIndex.resize(OffsetIndex.scala:276)
at kafka.log.Log.loadSegments(Log.scala:154)
at kafka.log.Log.<init>(Log.scala:64)
at kafka.log.LogManager$$anonfun$loadLogs$1$$anonfun$apply$4.apply(LogManager.scala:118)
at kafka.log.LogManager$$anonfun$loadLogs$1$$anonfun$apply$4.apply(LogManager.scala:113)
at scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:34)
at scala.collection.mutable.ArrayOps.foreach(ArrayOps.scala:38)
at kafka.log.LogManager$$anonfun$loadLogs$1.apply(LogManager.scala:113)
at kafka.log.LogManager$$anonfun$loadLogs$1.apply(LogManager.scala:105)
at scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:34)
at scala.collection.mutable.WrappedArray.foreach(WrappedArray.scala:33)
at kafka.log.LogManager.loadLogs(LogManager.scala:105)
at kafka.log.LogManager.<init>(LogManager.scala:57)
at kafka.server.KafkaServer.createLogManager(KafkaServer.scala:275)
at kafka.server.KafkaServer.startup(KafkaServer.scala:72)
at kafka.server.KafkaServerStartable.startup(KafkaServerStartable.scala:34)
at kafka.Kafka$.main(Kafka.scala:46)
at kafka.Kafka.main(Kafka.scala)
[2014-05-26 14:05:23,869] INFO Completed load of log dropped_calls-0 with log end offset 0 (kafka.log.Log)
[2014-05-26 14:05:23,871] INFO Loading log 'vip_ips_alerts-0' (kafka.log.LogManager)
[2014-05-26 14:05:23,879] WARN Error when freeing index buffer (kafka.log.OffsetIndex)
java.lang.NullPointerException
at kafka.log.OffsetIndex.kafka$log$OffsetIndex$$forceUnmap(OffsetIndex.scala:301)
at kafka.log.OffsetIndex$$anonfun$resize$1.apply(OffsetIndex.scala:283)
at kafka.log.OffsetIndex$$anonfun$resize$1.apply(OffsetIndex.scala:276)
at kafka.utils.Utils$.inLock(Utils.scala:538)
at kafka.log.OffsetIndex.resize(OffsetIndex.scala:276)
at kafka.log.Log.loadSegments(Log.scala:154)
at kafka.log.Log.<init>(Log.scala:64)
at kafka.log.LogManager$$anonfun$loadLogs$1$$anonfun$apply$4.apply(LogManager.scala:118)
at kafka.log.LogManager$$anonfun$loadLogs$1$$anonfun$apply$4.apply(LogManager.scala:113)
at scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:34)
at scala.collection.mutable.ArrayOps.foreach(ArrayOps.scala:38)
at kafka.log.LogManager$$anonfun$loadLogs$1.apply(LogManager.scala:113)
at kafka.log.LogManager$$anonfun$loadLogs$1.apply(LogManager.scala:105)
at scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:34)
at scala.collection.mutable.WrappedArray.foreach(WrappedArray.scala:33)
at kafka.log.LogManager.loadLogs(LogManager.scala:105)
at kafka.log.LogManager.<init>(LogManager.scala:57)
at kafka.server.KafkaServer.createLogManager(KafkaServer.scala:275)
at kafka.server.KafkaServer.startup(KafkaServer.scala:72)
at kafka.server.KafkaServerStartable.startup(KafkaServerStartable.scala:34)
at kafka.Kafka$.main(Kafka.scala:46)
at kafka.Kafka.main(Kafka.scala)
[2014-05-26 14:05:23,885] INFO Completed load of log vip_ips_alerts-0 with log end offset 0 (kafka.log.Log)
[2014-05-26 14:05:23,888] INFO Starting log cleanup with a period of 60000 ms. (kafka.log.LogManager)
[2014-05-26 14:05:23,895] INFO Starting log flusher with a default period of 9223372036854775807 ms. (kafka.log.LogManager)
[2014-05-26 14:05:23,959] INFO Awaiting socket connections on 0.0.0.0:9092. (kafka.network.Acceptor)
[2014-05-26 14:05:23,961] INFO [Socket Server on Broker 0], Started (kafka.network.SocketServer)
[2014-05-26 14:05:24,155] INFO Will not load MX4J, mx4j-tools.jar is not in the classpath (kafka.utils.Mx4jLoader$)
[2014-05-26 14:05:24,219] INFO 0 successfully elected as leader (kafka.server.ZookeeperLeaderElector)
[2014-05-26 14:05:24,532] INFO New leader is 0 (kafka.server.ZookeeperLeaderElector$LeaderChangeListener)
[2014-05-26 14:05:24,539] INFO Registered broker 0 at path /brokers/ids/0 with address S80035683-SC01.china.huawei.com:9092. (kafka.utils.ZkUtils$)
[2014-05-26 14:05:24,551] INFO [Kafka Server 0], started (kafka.server.KafkaServer)
[2014-05-26 14:05:24,755] INFO [ReplicaFetcherManager on broker 0] Removed fetcher for partitions [vip_ips_alerts,0],[calls,0],[dropped_calls,0],[call
s_online,0],[calls_no_phone,0] (kafka.server.ReplicaFetcherManager)
[2014-05-26 14:05:24,771] ERROR [KafkaApi-0] error when handling request Name:LeaderAndIsrRequest;Version:0;Controller:0;ControllerEpoch:2;Correlation
Id:4;ClientId:id_0-host_null-port_9092;Leaders:id:0,host:S80035683-SC01.china.huawei.com,port:9092;PartitionState:(vip_ips_alerts,0) -> (LeaderAndIsrI
nfo:(Leader:0,ISR:0,LeaderEpoch:0,ControllerEpoch:1),ReplicationFactor:1),AllReplicas:0),(calls,0) -> (LeaderAndIsrInfo:(Leader:0,ISR:0,LeaderEpoch:0,
ControllerEpoch:1),ReplicationFactor:1),AllReplicas:0),(dropped_calls,0) -> (LeaderAndIsrInfo:(Leader:0,ISR:0,LeaderEpoch:0,ControllerEpoch:1),Replica
tionFactor:1),AllReplicas:0),(calls_online,0) -> (LeaderAndIsrInfo:(Leader:0,ISR:0,LeaderEpoch:0,ControllerEpoch:1),ReplicationFactor:1),AllReplicas:0
),(calls_no_phone,0) -> (LeaderAndIsrInfo:(Leader:0,ISR:0,LeaderEpoch:0,ControllerEpoch:1),ReplicationFactor:1),AllReplicas:0) (kafka.server.KafkaApis
)
java.util.NoSuchElementException: key not found: \tmp\kafka-logs
at scala.collection.MapLike$class.default(MapLike.scala:225)
at scala.collection.immutable.Map$Map1.default(Map.scala:107)
at scala.collection.MapLike$class.apply(MapLike.scala:135)
at scala.collection.immutable.Map$Map1.apply(Map.scala:107)
at kafka.cluster.Partition.getOrCreateReplica(Partition.scala:91)
at kafka.cluster.Partition$$anonfun$makeLeader$2.apply(Partition.scala:175)
at kafka.cluster.Partition$$anonfun$makeLeader$2.apply(Partition.scala:175)
at scala.collection.immutable.Set$Set1.foreach(Set.scala:86)
at kafka.cluster.Partition.makeLeader(Partition.scala:175)
at kafka.server.ReplicaManager$$anonfun$makeLeaders$5.apply(ReplicaManager.scala:305)
at kafka.server.ReplicaManager$$anonfun$makeLeaders$5.apply(ReplicaManager.scala:304)
at scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:95)
at scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:95)
at scala.collection.Iterator$class.foreach(Iterator.scala:772)
at scala.collection.mutable.HashTable$$anon$1.foreach(HashTable.scala:157)
at scala.collection.mutable.HashTable$class.foreachEntry(HashTable.scala:190)
at scala.collection.mutable.HashMap.foreachEntry(HashMap.scala:45)
at scala.collection.mutable.HashMap.foreach(HashMap.scala:95)
at kafka.server.ReplicaManager.makeLeaders(ReplicaManager.scala:304)
at kafka.server.ReplicaManager.becomeLeaderOrFollower(ReplicaManager.scala:258)
at kafka.server.KafkaApis.handleLeaderAndIsrRequest(KafkaApis.scala:100)
at kafka.server.KafkaApis.handle(KafkaApis.scala:72)
at kafka.server.KafkaRequestHandler.run(KafkaRequestHandler.scala:42)
at java.lang.Thread.run(Thread.java:744)
[2014-05-26 14:05:24,779] INFO [ReplicaFetcherManager on broker 0] Removed fetcher for partitions [vip_ips_alerts,0],[calls,0],[dropped_calls,0],[call
s_online,0],[calls_no_phone,0] (kafka.server.ReplicaFetcherManager)
[2014-05-26 14:05:24,787] ERROR [KafkaApi-0] error when handling request Name:LeaderAndIsrRequest;Version:0;Controller:0;ControllerEpoch:2;Correlation
Id:5;ClientId:id_0-host_null-port_9092;Leaders:id:0,host:S80035683-SC01.china.huawei.com,port:9092;PartitionState:(vip_ips_alerts,0) -> (LeaderAndIsrI
nfo:(Leader:0,ISR:0,LeaderEpoch:1,ControllerEpoch:2),ReplicationFactor:1),AllReplicas:0),(calls,0) -> (LeaderAndIsrInfo:(Leader:0,ISR:0,LeaderEpoch:1,
ControllerEpoch:2),ReplicationFactor:1),AllReplicas:0),(dropped_calls,0) -> (LeaderAndIsrInfo:(Leader:0,ISR:0,LeaderEpoch:1,ControllerEpoch:2),Replica
tionFactor:1),AllReplicas:0),(calls_online,0) -> (LeaderAndIsrInfo:(Leader:0,ISR:0,LeaderEpoch:1,ControllerEpoch:2),ReplicationFactor:1),AllReplicas:0
),(calls_no_phone,0) -> (LeaderAndIsrInfo:(Leader:0,ISR:0,LeaderEpoch:1,ControllerEpoch:2),ReplicationFactor:1),AllReplicas:0) (kafka.server.KafkaApis
)
java.util.NoSuchElementException: key not found: \tmp\kafka-logs
at scala.collection.MapLike$class.default(MapLike.scala:225)
at scala.collection.immutable.Map$Map1.default(Map.scala:107)
at scala.collection.MapLike$class.apply(MapLike.scala:135)
at scala.collection.immutable.Map$Map1.apply(Map.scala:107)
at kafka.cluster.Partition.getOrCreateReplica(Partition.scala:91)
at kafka.cluster.Partition$$anonfun$makeLeader$2.apply(Partition.scala:175)
at kafka.cluster.Partition$$anonfun$makeLeader$2.apply(Partition.scala:175)
at scala.collection.immutable.Set$Set1.foreach(Set.scala:86)
at kafka.cluster.Partition.makeLeader(Partition.scala:175)
at kafka.server.ReplicaManager$$anonfun$makeLeaders$5.apply(ReplicaManager.scala:305)
at kafka.server.ReplicaManager$$anonfun$makeLeaders$5.apply(ReplicaManager.scala:304)
at scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:95)
at scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:95)
at scala.collection.Iterator$class.foreach(Iterator.scala:772)
at scala.collection.mutable.HashTable$$anon$1.foreach(HashTable.scala:157)
at scala.collection.mutable.HashTable$class.foreachEntry(HashTable.scala:190)
at scala.collection.mutable.HashMap.foreachEntry(HashMap.scala:45)
at scala.collection.mutable.HashMap.foreach(HashMap.scala:95)
at kafka.server.ReplicaManager.makeLeaders(ReplicaManager.scala:304)
at kafka.server.ReplicaManager.becomeLeaderOrFollower(ReplicaManager.scala:258)
at kafka.server.KafkaApis.handleLeaderAndIsrRequest(KafkaApis.scala:100)
at kafka.server.KafkaApis.handle(KafkaApis.scala:72)
at kafka.server.KafkaRequestHandler.run(KafkaRequestHandler.scala:42)
at java.lang.Thread.run(Thread.java:744)
You need to change the log dir to a windows path, something like C:\tmp\kafka-logs
The bat files for kafka 0.8.1 are full of errors. Here is a very helpful blog I followed to get it up and running on windows. He gives the link to a git repo with fixed bat files.
Related
This question already has answers here:
How to install Kafka on Windows?
(9 answers)
Closed 1 year ago.
I downloaded the kafka_2.13-3.0.0 in Window 10. I opened up CMD as admin. I was able to run zookeeper using the command bin\windows\zookeeper-server-start.bat config\zookeeper.properties. However, when i tried to start the kafka server with the following command
bin\windows\kafka-server-start.bat config\server.properties
, it gave me the following error" below. I am confused here because I can see the kafka-logs being created (see image below). Can someone help me out please ?
Output from running Zookeeper command
[2021-12-01 20:16:17,208] INFO Server environment:java.io.tmpdir=C:\Users\user123456~1\AppData\Local\Temp\ (org.apache.zookeeper.server.ZooKeeperServer)
[2021-12-01 20:16:17,208] INFO Server environment:java.compiler=<NA> (org.apache.zookeeper.server.ZooKeeperServer)
[2021-12-01 20:16:17,208] INFO Server environment:os.name=Windows 10 (org.apache.zookeeper.server.ZooKeeperServer)
[2021-12-01 20:16:17,209] INFO Server environment:os.arch=amd64 (org.apache.zookeeper.server.ZooKeeperServer)
[2021-12-01 20:16:17,209] INFO Server environment:os.version=10.0 (org.apache.zookeeper.server.ZooKeeperServer)
[2021-12-01 20:16:17,209] INFO Server environment:user.name=user123456 (org.apache.zookeeper.server.ZooKeeperServer)
[2021-12-01 20:16:17,210] INFO Server environment:user.home=C:\Users\user123456 (org.apache.zookeeper.server.ZooKeeperServer)
[2021-12-01 20:16:17,210] INFO Server environment:user.dir=C:\kafka_2.13-3.0.0 (org.apache.zookeeper.server.ZooKeeperServer)
[2021-12-01 20:16:17,211] INFO Server environment:os.memory.free=493MB (org.apache.zookeeper.server.ZooKeeperServer)
[2021-12-01 20:16:17,211] INFO Server environment:os.memory.max=512MB (org.apache.zookeeper.server.ZooKeeperServer)
[2021-12-01 20:16:17,211] INFO Server environment:os.memory.total=512MB (org.apache.zookeeper.server.ZooKeeperServer)
[2021-12-01 20:16:17,212] INFO zookeeper.enableEagerACLCheck = false (org.apache.zookeeper.server.ZooKeeperServer)
[2021-12-01 20:16:17,212] INFO zookeeper.digest.enabled = true (org.apache.zookeeper.server.ZooKeeperServer)
[2021-12-01 20:16:17,212] INFO zookeeper.closeSessionTxn.enabled = true (org.apache.zookeeper.server.ZooKeeperServer)
[2021-12-01 20:16:17,213] INFO zookeeper.flushDelay=0 (org.apache.zookeeper.server.ZooKeeperServer)
[2021-12-01 20:16:17,213] INFO zookeeper.maxWriteQueuePollTime=0 (org.apache.zookeeper.server.ZooKeeperServer)
[2021-12-01 20:16:17,214] INFO zookeeper.maxBatchSize=1000 (org.apache.zookeeper.server.ZooKeeperServer)
[2021-12-01 20:16:17,214] INFO zookeeper.intBufferStartingSizeBytes = 1024 (org.apache.zookeeper.server.ZooKeeperServer)
[2021-12-01 20:16:17,215] INFO Weighed connection throttling is disabled (org.apache.zookeeper.server.BlueThrottle)
[2021-12-01 20:16:17,216] INFO minSessionTimeout set to 6000 (org.apache.zookeeper.server.ZooKeeperServer)
[2021-12-01 20:16:17,216] INFO maxSessionTimeout set to 60000 (org.apache.zookeeper.server.ZooKeeperServer)
[2021-12-01 20:16:17,217] INFO Response cache size is initialized with value 400. (org.apache.zookeeper.server.ResponseCache)
[2021-12-01 20:16:17,217] INFO Response cache size is initialized with value 400. (org.apache.zookeeper.server.ResponseCache)
[2021-12-01 20:16:17,218] INFO zookeeper.pathStats.slotCapacity = 60 (org.apache.zookeeper.server.util.RequestPathMetricsCollector)
[2021-12-01 20:16:17,220] INFO zookeeper.pathStats.slotDuration = 15 (org.apache.zookeeper.server.util.RequestPathMetricsCollector)
[2021-12-01 20:16:17,221] INFO zookeeper.pathStats.maxDepth = 6 (org.apache.zookeeper.server.util.RequestPathMetricsCollector)
[2021-12-01 20:16:17,221] INFO zookeeper.pathStats.initialDelay = 5 (org.apache.zookeeper.server.util.RequestPathMetricsCollector)
[2021-12-01 20:16:17,221] INFO zookeeper.pathStats.delay = 5 (org.apache.zookeeper.server.util.RequestPathMetricsCollector)
[2021-12-01 20:16:17,222] INFO zookeeper.pathStats.enabled = false (org.apache.zookeeper.server.util.RequestPathMetricsCollector)
[2021-12-01 20:16:17,224] INFO The max bytes for all large requests are set to 104857600 (org.apache.zookeeper.server.ZooKeeperServer)
[2021-12-01 20:16:17,224] INFO The large request threshold is set to -1 (org.apache.zookeeper.server.ZooKeeperServer)
[2021-12-01 20:16:17,224] INFO Created server with tickTime 3000 minSessionTimeout 6000 maxSessionTimeout 60000 clientPortListenBacklog -1 datadir \tmp\zookeeper\version-2 snapdir \tmp\zookeeper\version-2 (org.apache.zookeeper.server.ZooKeeperServer)
[2021-12-01 20:16:17,234] INFO Using org.apache.zookeeper.server.NIOServerCnxnFactory as server connection factory (org.apache.zookeeper.server.ServerCnxnFactory)
[2021-12-01 20:16:17,235] WARN maxCnxns is not configured, using default value 0. (org.apache.zookeeper.server.ServerCnxnFactory)
[2021-12-01 20:16:17,236] INFO Configuring NIO connection handler with 10s sessionless connection timeout, 2 selector thread(s), 32 worker threads, and 64 kB direct buffers. (org.apache.zookeeper.server.NIOServerCnxnFactory)
[2021-12-01 20:16:17,238] INFO binding to port 0.0.0.0/0.0.0.0:2181 (org.apache.zookeeper.server.NIOServerCnxnFactory)
[2021-12-01 20:16:17,250] INFO Using org.apache.zookeeper.server.watch.WatchManager as watch manager (org.apache.zookeeper.server.watch.WatchManagerFactory)
[2021-12-01 20:16:17,250] INFO Using org.apache.zookeeper.server.watch.WatchManager as watch manager (org.apache.zookeeper.server.watch.WatchManagerFactory)
[2021-12-01 20:16:17,251] INFO zookeeper.snapshotSizeFactor = 0.33 (org.apache.zookeeper.server.ZKDatabase)
[2021-12-01 20:16:17,251] INFO zookeeper.commitLogCount=500 (org.apache.zookeeper.server.ZKDatabase)
[2021-12-01 20:16:17,256] INFO zookeeper.snapshot.compression.method = CHECKED (org.apache.zookeeper.server.persistence.SnapStream)
[2021-12-01 20:16:17,256] INFO Snapshotting: 0x0 to \tmp\zookeeper\version-2\snapshot.0 (org.apache.zookeeper.server.persistence.FileTxnSnapLog)
[2021-12-01 20:16:17,260] INFO Snapshot loaded in 8 ms, highest zxid is 0x0, digest is 1371985504 (org.apache.zookeeper.server.ZKDatabase)
[2021-12-01 20:16:17,260] INFO Snapshotting: 0x0 to \tmp\zookeeper\version-2\snapshot.0 (org.apache.zookeeper.server.persistence.FileTxnSnapLog)
[2021-12-01 20:16:17,261] INFO Snapshot taken in 1 ms (org.apache.zookeeper.server.ZooKeeperServer)
[2021-12-01 20:16:17,274] INFO PrepRequestProcessor (sid:0) started, reconfigEnabled=false (org.apache.zookeeper.server.PrepRequestProcessor)
[2021-12-01 20:16:17,274] INFO zookeeper.request_throttler.shutdownTimeout = 10000 (org.apache.zookeeper.server.RequestThrottler)
[2021-12-01 20:16:17,286] INFO Using checkIntervalMs=60000 maxPerMinute=10000 maxNeverUsedIntervalMs=0 (org.apache.zookeeper.server.ContainerManager)
[2021-12-01 20:16:17,287] INFO ZooKeeper audit is disabled. (org.apache.zookeeper.audit.ZKAuditProvider)
Output from running kafka command
[2021-12-01 20:18:22,577] INFO Client environment:java.io.tmpdir=C:\Users\user123456~1\AppData\Local\Temp\ (org.apache.zookeeper.ZooKeeper)
[2021-12-01 20:18:22,577] INFO Client environment:java.compiler=<NA> (org.apache.zookeeper.ZooKeeper)
[2021-12-01 20:18:22,578] INFO Client environment:os.name=Windows 10 (org.apache.zookeeper.ZooKeeper)
[2021-12-01 20:18:22,579] INFO Client environment:os.arch=amd64 (org.apache.zookeeper.ZooKeeper)
[2021-12-01 20:18:22,579] INFO Client environment:os.version=10.0 (org.apache.zookeeper.ZooKeeper)
[2021-12-01 20:18:22,580] INFO Client environment:user.name=user123456 (org.apache.zookeeper.ZooKeeper)
[2021-12-01 20:18:22,581] INFO Client environment:user.home=C:\Users\user123456 (org.apache.zookeeper.ZooKeeper)
[2021-12-01 20:18:22,581] INFO Client environment:user.dir=C:\kafka_2.13-3.0.0 (org.apache.zookeeper.ZooKeeper)
[2021-12-01 20:18:22,582] INFO Client environment:os.memory.free=1011MB (org.apache.zookeeper.ZooKeeper)
[2021-12-01 20:18:22,583] INFO Client environment:os.memory.max=1024MB (org.apache.zookeeper.ZooKeeper)
[2021-12-01 20:18:22,583] INFO Client environment:os.memory.total=1024MB (org.apache.zookeeper.ZooKeeper)
[2021-12-01 20:18:22,585] INFO Initiating client connection, connectString=localhost:2181 sessionTimeout=18000 watcher=kafka.zookeeper.ZooKeeperClient$ZooKeeperClientWatcher$#74f0ea28 (org.apache.zookeeper.ZooKeeper)
[2021-12-01 20:18:22,593] INFO jute.maxbuffer value is 4194304 Bytes (org.apache.zookeeper.ClientCnxnSocket)
[2021-12-01 20:18:22,597] INFO zookeeper.request.timeout value is 0. feature enabled=false (org.apache.zookeeper.ClientCnxn)
[2021-12-01 20:18:22,598] INFO [ZooKeeperClient Kafka server] Waiting until connected. (kafka.zookeeper.ZooKeeperClient)
[2021-12-01 20:18:22,601] INFO Opening socket connection to server localhost/127.0.0.1:2181. (org.apache.zookeeper.ClientCnxn)
[2021-12-01 20:18:22,601] INFO SASL config status: Will not attempt to authenticate using SASL (unknown error) (org.apache.zookeeper.ClientCnxn)
[2021-12-01 20:18:22,602] INFO Socket connection established, initiating session, client: /127.0.0.1:57191, server: localhost/127.0.0.1:2181 (org.apache.zookeeper.ClientCnxn)
[2021-12-01 20:18:22,617] INFO Session establishment complete on server localhost/127.0.0.1:2181, session id = 0x1000008c9c20000, negotiated timeout = 18000 (org.apache.zookeeper.ClientCnxn)
[2021-12-01 20:18:22,619] INFO [ZooKeeperClient Kafka server] Connected. (kafka.zookeeper.ZooKeeperClient)
[2021-12-01 20:18:22,720] INFO [feature-zk-node-event-process-thread]: Starting (kafka.server.FinalizedFeatureChangeListener$ChangeNotificationProcessorThread)
[2021-12-01 20:18:22,729] INFO Feature ZK node at path: /feature does not exist (kafka.server.FinalizedFeatureChangeListener)
[2021-12-01 20:18:22,729] INFO Cleared cache (kafka.server.FinalizedFeatureCache)
[2021-12-01 20:18:22,844] INFO Cluster ID = j7KyANF-S2OuLlr_IYyqDw (kafka.server.KafkaServer)
[2021-12-01 20:18:22,846] WARN No meta.properties file under dir C:\tmp\kafka-logs\meta.properties (kafka.server.BrokerMetadataCheckpoint)
[2021-12-01 20:18:22,876] INFO KafkaConfig values:
...
[2021-12-01 20:18:22,918] INFO [ThrottledChannelReaper-Fetch]: Starting (kafka.server.ClientQuotaManager$ThrottledChannelReaper)
[2021-12-01 20:18:22,919] INFO [ThrottledChannelReaper-Produce]: Starting (kafka.server.ClientQuotaManager$ThrottledChannelReaper)
[2021-12-01 20:18:22,919] INFO [ThrottledChannelReaper-Request]: Starting (kafka.server.ClientQuotaManager$ThrottledChannelReaper)
[2021-12-01 20:18:22,920] INFO [ThrottledChannelReaper-ControllerMutation]: Starting (kafka.server.ClientQuotaManager$ThrottledChannelReaper)
[2021-12-01 20:18:22,931] INFO Log directory C:\tmp\kafka-logs not found, creating it. (kafka.log.LogManager)
[2021-12-01 20:18:22,933] ERROR Failed to create or validate data directory C:\tmp\kafka-logs (kafka.server.LogDirFailureChannel)
java.nio.file.AccessDeniedException: C:\tmp
at sun.nio.fs.WindowsException.translateToIOException(WindowsException.java:83)
at sun.nio.fs.WindowsException.rethrowAsIOException(WindowsException.java:97)
at sun.nio.fs.WindowsException.rethrowAsIOException(WindowsException.java:102)
at sun.nio.fs.WindowsFileSystemProvider.newFileChannel(WindowsFileSystemProvider.java:115)
at java.nio.channels.FileChannel.open(FileChannel.java:287)
at java.nio.channels.FileChannel.open(FileChannel.java:335)
at org.apache.kafka.common.utils.Utils.flushDir(Utils.java:953)
at kafka.log.LogManager.$anonfun$createAndValidateLogDirs$1(LogManager.scala:159)
at scala.collection.IterableOnceOps.foreach(IterableOnce.scala:563)
at scala.collection.IterableOnceOps.foreach$(IterableOnce.scala:561)
at scala.collection.AbstractIterable.foreach(Iterable.scala:919)
at kafka.log.LogManager.createAndValidateLogDirs(LogManager.scala:149)
at kafka.log.LogManager.<init>(LogManager.scala:88)
at kafka.log.LogManager$.apply(LogManager.scala:1283)
at kafka.server.KafkaServer.startup(KafkaServer.scala:254)
at kafka.Kafka$.main(Kafka.scala:109)
at kafka.Kafka.main(Kafka.scala)
[2021-12-01 20:18:22,936] ERROR Shutdown broker because none of the specified log dirs from C:\tmp\kafka-logs can be created or validated (kafka.log.LogManager)
I also tried running icacls "C:\tmp\" /q /c /t /grant Users:F. I am still getting the same error.
It turns out that 3.0.0 has issues with Windows. I downgraded to 2.8.1 and things worked fine. I hate Windows.
I can't understand what was happening, I set up sasl / scram tls IN KAFKA and do everything according to the instructions. but when running, I get an error and there is nothing useful in the logs.
enter image description here
enter image description here
enter image description here
enter image description here
enter image description here
broker.id=0
num.network.threads=3
num.io.threads=8
socket.send.buffer.bytes=102400
socket.receive.buffer.bytes=102400
socket.request.max.bytes=104857600
listeners=SASL_SSL://localhost:9092
advertised.listeners=SASL_SSL://localhost:9092
sasl.enabled.mechanisms=SCRAM-SHA-256
sasl.mechanism.inter.broker.protocol=SCRAM-SHA-256
security.inter.broker.protocol=SASL_SSL
security.protocol=SASL_SSL
ssl.keystore.location=/opt/kafka/comfig/server.keystore.jks
ssl.keystore.password=admin-secret
ssl.key.password=admin-secret
ssl.truststore.location=/opt/kafka/comfig/server.truststore.jks
ssl.truststore.password=admin-secret
ssl.endpoint.identification.algorithm=
ssl.client.auth=required
ssl.enabled.protocols=TLSv1.2,TLSv1.1,TLSv1
ssl.keystore.type=JKS
ssl.truststore.type=JKS
ssl.secure.random.implementation=SHA1PRNG
log.dirs=/tmp/kafka-logs
num.partitions=1
num.recovery.threads.per.data.dir=1
offsets.topic.replication.factor=1
transaction.state.log.replication.factor=1
transaction.state.log.min.isr=1
log.retention.hours=168
log.segment.bytes=1073741824
log.retention.check.interval.ms=300000
zookeeper.connect=localhost:2181
zookeeper.connection.timeout.ms=18000
group.initial.rebalance.delay.ms=0
[2021-10-11 19:05:17,955] INFO Registered kafka:type=kafka.Log4jController MBean (kafka.utils.Log4jControllerRegistration$)
[2021-10-11 19:05:18,276] INFO Setting -D jdk.tls.rejectClientInitiatedRenegotiation=true to disable client-initiated TLS renegotiation (org.apache.zookeeper.common.X509Util)
[2021-10-11 19:05:18,328] INFO Registered signal handlers for TERM, INT, HUP (org.apache.kafka.common.utils.LoggingSignalHandler)
[2021-10-11 19:05:18,331] INFO starting (kafka.server.KafkaServer)
[2021-10-11 19:05:18,332] INFO Connecting to zookeeper on localhost:2181 (kafka.server.KafkaServer)
[2021-10-11 19:05:18,343] ERROR Fatal error during KafkaServer startup. Prepare to shutdown (kafka.server.KafkaServer)
org.apache.kafka.common.KafkaException: Exception while loading Zookeeper JAAS login context [java.security.auth.login.config=/opt/kafka/config/sasl.jaas.config, zookeeper.sasl.client=default:true, zookeeper.sasl.clientconfig=default:Client]
at org.apache.kafka.common.security.JaasUtils.isZkSaslEnabled(JaasUtils.java:67)
at kafka.server.KafkaServer.initZkClient(KafkaServer.scala:451)
at kafka.server.KafkaServer.startup(KafkaServer.scala:233)
at kafka.server.KafkaServerStartable.startup(KafkaServerStartable.scala:44)
at kafka.Kafka$.main(Kafka.scala:82)
at kafka.Kafka.main(Kafka.scala)
Caused by: java.lang.SecurityException: java.io.IOException: Configuration Error:
expected [;], read [end of file]
at java.base/sun.security.provider.ConfigFile$Spi.<init>(ConfigFile.java:137)
at java.base/sun.security.provider.ConfigFile.<init>(ConfigFile.java:102)
at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:490)
at java.base/java.lang.Class.newInstance(Class.java:584)
at java.base/javax.security.auth.login.Configuration$2.run(Configuration.java:255)
at java.base/javax.security.auth.login.Configuration$2.run(Configuration.java:246)
at java.base/java.security.AccessController.doPrivileged(Native Method)
at java.base/javax.security.auth.login.Configuration.getConfiguration(Configuration.java:245)
at org.apache.kafka.common.security.JaasUtils.isZkSaslEnabled(JaasUtils.java:63)
... 5 more
Caused by: java.io.IOException: Configuration Error:
expected [;], read [end of file]
at java.base/sun.security.provider.ConfigFile$Spi.ioException(ConfigFile.java:665)
at java.base/sun.security.provider.ConfigFile$Spi.match(ConfigFile.java:518)
at java.base/sun.security.provider.ConfigFile$Spi.parseLoginEntry(ConfigFile.java:501)
at java.base/sun.security.provider.ConfigFile$Spi.readConfig(ConfigFile.java:426)
at java.base/sun.security.provider.ConfigFile$Spi.init(ConfigFile.java:329)
at java.base/sun.security.provider.ConfigFile$Spi.init(ConfigFile.java:271)
at java.base/sun.security.provider.ConfigFile$Spi.<init>(ConfigFile.java:135)
... 16 more
[2021-10-11 19:05:18,346] INFO shutting down (kafka.server.KafkaServer)
[2021-10-11 19:05:18,352] INFO App info kafka.server for 0 unregistered (org.apache.kafka.common.utils.AppInfoParser)
[2021-10-11 19:05:18,352] INFO shut down completed (kafka.server.KafkaServer)
[2021-10-11 19:05:18,353] ERROR Exiting Kafka. (kafka.server.KafkaServerStartable)
[2021-10-11 19:05:18,357] INFO shutting down (kafka.server.KafkaServer)
[2021-10-11 19:09:29,307] INFO Registered kafka:type=kafka.Log4jController MBean (kafka.utils.Log4jControllerRegistration$)
[2021-10-11 19:09:29,596] INFO Setting -D jdk.tls.rejectClientInitiatedRenegotiation=true to disable client-initiated TLS renegotiation (org.apache.zookeeper.common.X509Util)
[2021-10-11 19:09:29,646] INFO Registered signal handlers for TERM, INT, HUP (org.apache.kafka.common.utils.LoggingSignalHandler)
[2021-10-11 19:09:29,649] INFO starting (kafka.server.KafkaServer)
[2021-10-11 19:09:29,650] INFO Connecting to zookeeper on localhost:2181 (kafka.server.KafkaServer)
[2021-10-11 19:09:29,662] ERROR Fatal error during KafkaServer startup. Prepare to shutdown (kafka.server.KafkaServer)
org.apache.kafka.common.KafkaException: Exception while loading Zookeeper JAAS login context [java.security.auth.login.config=/opt/kafka/config/sasl.jaas.config, zookeeper.sasl.client=default:true, zookeeper.sasl.clientconfig=default:Client]
at org.apache.kafka.common.security.JaasUtils.isZkSaslEnabled(JaasUtils.java:67)
at kafka.server.KafkaServer.initZkClient(KafkaServer.scala:451)
at kafka.server.KafkaServer.startup(KafkaServer.scala:233)
at kafka.server.KafkaServerStartable.startup(KafkaServerStartable.scala:44)
at kafka.Kafka$.main(Kafka.scala:82)
at kafka.Kafka.main(Kafka.scala)
Caused by: java.lang.SecurityException: java.io.IOException: Configuration Error:
expected [;], read [end of file]
at java.base/sun.security.provider.ConfigFile$Spi.<init>(ConfigFile.java:137)
at java.base/sun.security.provider.ConfigFile.<init>(ConfigFile.java:102)
at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:490)
at java.base/java.lang.Class.newInstance(Class.java:584)
at java.base/javax.security.auth.login.Configuration$2.run(Configuration.java:255)
at java.base/javax.security.auth.login.Configuration$2.run(Configuration.java:246)
at java.base/java.security.AccessController.doPrivileged(Native Method)
at java.base/javax.security.auth.login.Configuration.getConfiguration(Configuration.java:245)
at org.apache.kafka.common.security.JaasUtils.isZkSaslEnabled(JaasUtils.java:63)
... 5 more
Caused by: java.io.IOException: Configuration Error:
expected [;], read [end of file]
at java.base/sun.security.provider.ConfigFile$Spi.ioException(ConfigFile.java:665)
at java.base/sun.security.provider.ConfigFile$Spi.match(ConfigFile.java:518)
at java.base/sun.security.provider.ConfigFile$Spi.parseLoginEntry(ConfigFile.java:501)
at java.base/sun.security.provider.ConfigFile$Spi.readConfig(ConfigFile.java:426)
at java.base/sun.security.provider.ConfigFile$Spi.init(ConfigFile.java:329)
at java.base/sun.security.provider.ConfigFile$Spi.init(ConfigFile.java:271)
at java.base/sun.security.provider.ConfigFile$Spi.<init>(ConfigFile.java:135)
... 16 more
[2021-10-11 19:09:29,665] INFO shutting down (kafka.server.KafkaServer)
[2021-10-11 19:09:29,671] INFO App info kafka.server for 0 unregistered (org.apache.kafka.common.utils.AppInfoParser)
[2021-10-11 19:09:29,671] INFO shut down completed (kafka.server.KafkaServer)
[2021-10-11 19:09:29,672] ERROR Exiting Kafka. (kafka.server.KafkaServerStartable)
[2021-10-11 19:09:29,672] INFO shutting down (kafka.server.KafkaServer)
[2021-10-11 19:12:35,911] INFO Registered kafka:type=kafka.Log4jController MBean (kafka.utils.Log4jControllerRegistration$)
[2021-10-11 19:12:36,335] INFO Setting -D jdk.tls.rejectClientInitiatedRenegotiation=true to disable client-initiated TLS renegotiation (org.apache.zookeeper.common.X509Util)
[2021-10-11 19:12:36,401] INFO Registered signal handlers for TERM, INT, HUP (org.apache.kafka.common.utils.LoggingSignalHandler)
[2021-10-11 19:12:36,405] INFO starting (kafka.server.KafkaServer)
[2021-10-11 19:12:36,406] INFO Connecting to zookeeper on localhost:2181 (kafka.server.KafkaServer)
[2021-10-11 19:12:36,419] ERROR Fatal error during KafkaServer startup. Prepare to shutdown (kafka.server.KafkaServer)
org.apache.kafka.common.KafkaException: Exception while loading Zookeeper JAAS login context [java.security.auth.login.config=/opt/kafka/config/sasl.jaas.config, zookeeper.sasl.client=default:true, zookeeper.sasl.clientconfig=default:Client]
at org.apache.kafka.common.security.JaasUtils.isZkSaslEnabled(JaasUtils.java:67)
at kafka.server.KafkaServer.initZkClient(KafkaServer.scala:451)
at kafka.server.KafkaServer.startup(KafkaServer.scala:233)
at kafka.server.KafkaServerStartable.startup(KafkaServerStartable.scala:44)
at kafka.Kafka$.main(Kafka.scala:82)
at kafka.Kafka.main(Kafka.scala)
Caused by: java.lang.SecurityException: java.io.IOException: Configuration Error:
expected [;], read [end of file]
at java.base/sun.security.provider.ConfigFile$Spi.<init>(ConfigFile.java:137)
at java.base/sun.security.provider.ConfigFile.<init>(ConfigFile.java:102)
at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:490)
at java.base/java.lang.Class.newInstance(Class.java:584)
at java.base/javax.security.auth.login.Configuration$2.run(Configuration.java:255)
at java.base/javax.security.auth.login.Configuration$2.run(Configuration.java:246)
at java.base/java.security.AccessController.doPrivileged(Native Method)
at java.base/javax.security.auth.login.Configuration.getConfiguration(Configuration.java:245)
at org.apache.kafka.common.security.JaasUtils.isZkSaslEnabled(JaasUtils.java:63)
... 5 more
Caused by: java.io.IOException: Configuration Error:
expected [;], read [end of file]
at java.base/sun.security.provider.ConfigFile$Spi.ioException(ConfigFile.java:665)
at java.base/sun.security.provider.ConfigFile$Spi.match(ConfigFile.java:518)
at java.base/sun.security.provider.ConfigFile$Spi.parseLoginEntry(ConfigFile.java:501)
at java.base/sun.security.provider.ConfigFile$Spi.readConfig(ConfigFile.java:426)
at java.base/sun.security.provider.ConfigFile$Spi.init(ConfigFile.java:329)
at java.base/sun.security.provider.ConfigFile$Spi.init(ConfigFile.java:271)
at java.base/sun.security.provider.ConfigFile$Spi.<init>(ConfigFile.java:135)
... 16 more
[2021-10-11 19:12:36,423] INFO shutting down (kafka.server.KafkaServer)
[2021-10-11 19:12:36,430] INFO App info kafka.server for 0 unregistered (org.apache.kafka.common.utils.AppInfoParser)
[2021-10-11 19:12:36,430] INFO shut down completed (kafka.server.KafkaServer)
[2021-10-11 19:12:36,431] ERROR Exiting Kafka. (kafka.server.KafkaServerStartable)
[2021-10-11 19:12:36,432] INFO shutting down (kafka.server.KafkaServer)
[2021-10-11 19:12:42,438] INFO Registered kafka:type=kafka.Log4jController MBean (kafka.utils.Log4jControllerRegistration$)
[2021-10-11 19:12:42,734] INFO Setting -D jdk.tls.rejectClientInitiatedRenegotiation=true to disable client-initiated TLS renegotiation (org.apache.zookeeper.common.X509Util)
[2021-10-11 19:12:42,784] INFO Registered signal handlers for TERM, INT, HUP (org.apache.kafka.common.utils.LoggingSignalHandler)
[2021-10-11 19:12:42,788] INFO starting (kafka.server.KafkaServer)
[2021-10-11 19:12:42,788] INFO Connecting to zookeeper on localhost:2181 (kafka.server.KafkaServer)
[2021-10-11 19:12:42,802] ERROR Fatal error during KafkaServer startup. Prepare to shutdown (kafka.server.KafkaServer)
org.apache.kafka.common.KafkaException: Exception while loading Zookeeper JAAS login context [java.security.auth.login.config=/opt/kafka/config/kafka_server_jaas.conf, zookeeper.sasl.client=default:true, zookeeper.sasl.clientconfig=default:Client]
at org.apache.kafka.common.security.JaasUtils.isZkSaslEnabled(JaasUtils.java:67)
at kafka.server.KafkaServer.initZkClient(KafkaServer.scala:451)
at kafka.server.KafkaServer.startup(KafkaServer.scala:233)
at kafka.server.KafkaServerStartable.startup(KafkaServerStartable.scala:44)
at kafka.Kafka$.main(Kafka.scala:82)
at kafka.Kafka.main(Kafka.scala)
Caused by: java.lang.SecurityException: java.io.IOException: Configuration Error:
Invalid control flag, ТРЕБУЕТСЯ
at java.base/sun.security.provider.ConfigFile$Spi.<init>(ConfigFile.java:137)
at java.base/sun.security.provider.ConfigFile.<init>(ConfigFile.java:102)
at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:490)
at java.base/java.lang.Class.newInstance(Class.java:584)
at java.base/javax.security.auth.login.Configuration$2.run(Configuration.java:255)
at java.base/javax.security.auth.login.Configuration$2.run(Configuration.java:246)
at java.base/java.security.AccessController.doPrivileged(Native Method)
at java.base/javax.security.auth.login.Configuration.getConfiguration(Configuration.java:245)
at org.apache.kafka.common.security.JaasUtils.isZkSaslEnabled(JaasUtils.java:63)
... 5 more
Caused by: java.io.IOException: Configuration Error:
Invalid control flag, ТРЕБУЕТСЯ
at java.base/sun.security.provider.ConfigFile$Spi.ioException(ConfigFile.java:665)
at java.base/sun.security.provider.ConfigFile$Spi.parseLoginEntry(ConfigFile.java:468)
at java.base/sun.security.provider.ConfigFile$Spi.readConfig(ConfigFile.java:426)
at java.base/sun.security.provider.ConfigFile$Spi.init(ConfigFile.java:329)
at java.base/sun.security.provider.ConfigFile$Spi.init(ConfigFile.java:271)
at java.base/sun.security.provider.ConfigFile$Spi.<init>(ConfigFile.java:135)
... 16 more
[2021-10-11 19:12:42,805] INFO shutting down (kafka.server.KafkaServer)
[2021-10-11 19:12:42,812] INFO App info kafka.server for 0 unregistered (org.apache.kafka.common.utils.AppInfoParser)
[2021-10-11 19:12:42,812] INFO shut down completed (kafka.server.KafkaServer)
[2021-10-11 19:12:42,813] ERROR Exiting Kafka. (kafka.server.KafkaServerStartable)
[2021-10-11 19:12:42,825] INFO shutting down (kafka.server.KafkaServer)
[2021-10-11 19:17:07,832] INFO Registered kafka:type=kafka.Log4jController MBean (kafka.utils.Log4jControllerRegistration$)
[2021-10-11 19:17:08,135] INFO Setting -D jdk.tls.rejectClientInitiatedRenegotiation=true to disable client-initiated TLS renegotiation (org.apache.zookeeper.common.X509Util)
[2021-10-11 19:17:08,162] ERROR Exiting Kafka due to fatal exception (kafka.Kafka$)
java.lang.IllegalArgumentException: requirement failed: inter.broker.listener.name must be a listener name defined in advertised.listeners. The valid options based on currently configured listeners are SASL_SSL
at kafka.server.KafkaConfig.validateValues(KafkaConfig.scala:1781)
at kafka.server.KafkaConfig.<init>(KafkaConfig.scala:1756)
at kafka.server.KafkaConfig.<init>(KafkaConfig.scala:1312)
at kafka.server.KafkaServerStartable$.fromProps(KafkaServerStartable.scala:34)
at kafka.Kafka$.main(Kafka.scala:68)
at kafka.Kafka.main(Kafka.scala)
[2021-10-11 19:17:41,978] INFO Registered kafka:type=kafka.Log4jController MBean (kafka.utils.Log4jControllerRegistration$)
[2021-10-11 19:17:42,288] INFO Setting -D jdk.tls.rejectClientInitiatedRenegotiation=true to disable client-initiated TLS renegotiation (org.apache.zookeeper.common.X509Util)
[2021-10-11 19:17:42,338] INFO Registered signal handlers for TERM, INT, HUP (org.apache.kafka.common.utils.LoggingSignalHandler)
[2021-10-11 19:17:42,341] INFO starting (kafka.server.KafkaServer)
[2021-10-11 19:17:42,341] INFO Connecting to zookeeper on localhost:2181 (kafka.server.KafkaServer)
[2021-10-11 19:17:42,356] ERROR Fatal error during KafkaServer startup. Prepare to shutdown (kafka.server.KafkaServer)
org.apache.kafka.common.KafkaException: Exception while loading Zookeeper JAAS login context [java.security.auth.login.config=/opt/kafka/config/kafka_server_jaas.conf, zookeeper.sasl.client=default:true, zookeeper.sasl.clientconfig=default:Client]
at org.apache.kafka.common.security.JaasUtils.isZkSaslEnabled(JaasUtils.java:67)
at kafka.server.KafkaServer.initZkClient(KafkaServer.scala:451)
at kafka.server.KafkaServer.startup(KafkaServer.scala:233)
at kafka.server.KafkaServerStartable.startup(KafkaServerStartable.scala:44)
at kafka.Kafka$.main(Kafka.scala:82)
at kafka.Kafka.main(Kafka.scala)
Caused by: java.lang.SecurityException: java.io.IOException: Configuration Error:
Invalid control flag, ТРЕБУЕТСЯ
at java.base/sun.security.provider.ConfigFile$Spi.<init>(ConfigFile.java:137)
at java.base/sun.security.provider.ConfigFile.<init>(ConfigFile.java:102)
at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:490)
at java.base/java.lang.Class.newInstance(Class.java:584)
at java.base/javax.security.auth.login.Configuration$2.run(Configuration.java:255)
at java.base/javax.security.auth.login.Configuration$2.run(Configuration.java:246)
at java.base/java.security.AccessController.doPrivileged(Native Method)
at java.base/javax.security.auth.login.Configuration.getConfiguration(Configuration.java:245)
at org.apache.kafka.common.security.JaasUtils.isZkSaslEnabled(JaasUtils.java:63)
... 5 more
Caused by: java.io.IOException: Configuration Error:
Invalid control flag, ТРЕБУЕТСЯ
at java.base/sun.security.provider.ConfigFile$Spi.ioException(ConfigFile.java:665)
at java.base/sun.security.provider.ConfigFile$Spi.parseLoginEntry(ConfigFile.java:468)
at java.base/sun.security.provider.ConfigFile$Spi.readConfig(ConfigFile.java:426)
at java.base/sun.security.provider.ConfigFile$Spi.init(ConfigFile.java:329)
at java.base/sun.security.provider.ConfigFile$Spi.init(ConfigFile.java:271)
at java.base/sun.security.provider.ConfigFile$Spi.<init>(ConfigFile.java:135)
... 16 more
[2021-10-11 19:17:42,360] INFO shutting down (kafka.server.KafkaServer)
[2021-10-11 19:17:42,365] INFO App info kafka.server for 0 unregistered (org.apache.kafka.common.utils.AppInfoParser)
[2021-10-11 19:17:42,366] INFO shut down completed (kafka.server.KafkaServer)
[2021-10-11 19:17:42,367] ERROR Exiting Kafka. (kafka.server.KafkaServerStartable)
[2021-10-11 19:17:42,374] INFO shutting down (kafka.server.KafkaServer)
At least I think you should add a semicolon at the end of Kafka Server entry in the sasl jaas config file.
when starting the kafka broker i am getting some error:
I am giving last few lines of the error log:
INFO zookeeper state changed (SyncConnected) (org.I0Itec.zkclient.ZkClient)
[2016-05-12 01:07:01,759] INFO Log directory '/var/logs/service-bridge-logs' not found, creating it. (kafka.log.LogManager)
[2016-05-12 01:07:01,778] INFO Loading logs. (kafka.log.LogManager)
[2016-05-12 01:07:01,796] INFO Logs loading complete. (kafka.log.LogManager)
[2016-05-12 01:07:01,797] INFO Starting log cleanup with a period of 300000 ms. (kafka.log.LogManager)
[2016-05-12 01:07:01,806] INFO Starting log flusher with a default period of 9223372036854775807 ms. (kafka.log.LogManager)
[2016-05-12 01:07:01,874] INFO Awaiting socket connections on gns3-d.cloudapp.net:9092. (kafka.network.Acceptor)
[2016-05-12 01:07:01,875] INFO [Socket Server on Broker 2], Started (kafka.network.SocketServer)
[2016-05-12 01:07:02,042] INFO Will not load MX4J, mx4j-tools.jar is not in the classpath (kafka.utils.Mx4jLoader$)
[2016-05-12 01:07:02,168] INFO 2 successfully elected as leader (kafka.server.ZookeeperLeaderElector)
[2016-05-12 01:07:02,386] INFO Registered broker 2 at path /brokers/ids/2 with address 10.1.0.4:9092. (kafka.utils.ZkUtils$)
[2016-05-12 01:07:02,416] INFO [Kafka Server 2], started (kafka.server.KafkaServer)
[2016-05-12 01:07:02,529] INFO New leader is 2 (kafka.server.ZookeeperLeaderElector$LeaderChangeListener)
[2016-05-12 01:07:25,798] ERROR Closing socket for /40.122.64.23 because of error (kafka.network.Processor)
java.io.IOException: Connection reset by peer
at sun.nio.ch.FileDispatcherImpl.read0(Native Method)
at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39)
at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223)
at sun.nio.ch.IOUtil.read(IOUtil.java:197)
at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:380)
at kafka.utils.Utils$.read(Utils.scala:380)
at kafka.network.BoundedByteBufferReceive.readFrom(BoundedByteBufferReceive.scala:54)
at kafka.network.Processor.read(SocketServer.scala:444)
at kafka.network.Processor.run(SocketServer.scala:340)
at java.lang.Thread.run(Thread.java:745)
while in zookeeper sidealso am getting few error:
INFO Established session 0x154a35b64f40000 with negotiated timeout 6000 for client /10.1.0.4:36673 (org.apache.zookeeper.server.ZooKeeperServer)
[2016-05-12 01:07:02,313] INFO Got user-level KeeperException when processing sessionid:0x154a35b64f40000 type:delete cxid:0x1d zxid:0x52 txntype:-1 reqpath:n/a Error Path:/admin/preferred_replica_election Error:KeeperErrorCode = NoNode for /admin/preferred_replica_election (org.apache.zookeeper.server.PrepRequestProcessor)
[2016-05-12 01:08:33,001] INFO Expiring session 0x154a35b64f40000, timeout of 6000ms exceeded (org.apache.zookeeper.server.ZooKeeperServer)
[2016-05-12 01:08:33,001] INFO Processed session termination for sessionid: 0x154a35b64f40000 (org.apache.zookeeper.server.PrepRequestProcessor)
[2016-05-12 01:08:33,017] INFO Closed socket connection for client /10.1.0.4:36673 which had sessionid 0x154a35b64f40000 (org.apache.zookeeper.server.NIOServerCnxn)
Any idea guys??
Thanks in advance..
As user avr pointed out, this is a known bug in Kafka 0.8.2.x
These are routine informational messages misclassified as ERROR.
As of Kafka v0.9.0.0, the loglevel has been corrected to WARN.
Kafka shuts down without error after a certain time interval.
kafka server log:
[2016-01-11 12:15:06,435] INFO Verifying properties (kafka.utils.VerifiableProperties)
[2016-01-11 12:15:06,489] INFO Property broker.id is overridden to 0 (kafka.utils.VerifiableProperties)
[2016-01-11 12:15:06,489] INFO Property host.name is overridden to 172.16.1.22 (kafka.utils.VerifiableProperties)
[2016-01-11 12:15:06,490] INFO Property log.cleaner.enable is overridden to false (kafka.utils.VerifiableProperties)
[2016-01-11 12:15:06,490] INFO Property log.dirs is overridden to /data/www/wifiin/logs/kafka (kafka.utils.VerifiableProperties)
[2016-01-11 12:15:06,490] INFO Property log.retention.check.interval.ms is overridden to 300000 (kafka.utils.VerifiableProperties)
[2016-01-11 12:15:06,491] INFO Property log.retention.hours is overridden to 168 (kafka.utils.VerifiableProperties)
[2016-01-11 12:15:06,491] INFO Property log.segment.bytes is overridden to 1073741824 (kafka.utils.VerifiableProperties)
[2016-01-11 12:15:06,491] INFO Property num.io.threads is overridden to 8 (kafka.utils.VerifiableProperties)
[2016-01-11 12:15:06,491] INFO Property num.network.threads is overridden to 3 (kafka.utils.VerifiableProperties)
[2016-01-11 12:15:06,492] INFO Property num.partitions is overridden to 1 (kafka.utils.VerifiableProperties)
[2016-01-11 12:15:06,492] INFO Property num.recovery.threads.per.data.dir is overridden to 1 (kafka.utils.VerifiableProperties)
[2016-01-11 12:15:06,492] INFO Property port is overridden to 9092 (kafka.utils.VerifiableProperties)
[2016-01-11 12:15:06,493] INFO Property socket.receive.buffer.bytes is overridden to 102400 (kafka.utils.VerifiableProperties)
[2016-01-11 12:15:06,493] INFO Property socket.request.max.bytes is overridden to 104857600 (kafka.utils.VerifiableProperties)
[2016-01-11 12:15:06,493] INFO Property socket.send.buffer.bytes is overridden to 102400 (kafka.utils.VerifiableProperties)
[2016-01-11 12:15:06,493] INFO Property zookeeper.connect is overridden to 127.0.0.1:2181 (kafka.utils.VerifiableProperties)
[2016-01-11 12:15:06,494] INFO Property zookeeper.connection.timeout.ms is overridden to 6000 (kafka.utils.VerifiableProperties)
[2016-01-11 12:15:06,550] INFO [Kafka Server 0], starting (kafka.server.KafkaServer)
[2016-01-11 12:15:06,553] INFO [Kafka Server 0], Connecting to zookeeper on 127.0.0.1:2181 (kafka.server.KafkaServer)
[2016-01-11 12:15:06,568] INFO Starting ZkClient event thread. (org.I0Itec.zkclient.ZkEventThread)
[2016-01-11 12:15:06,579] INFO Client environment:zookeeper.version=3.4.6-1569965, built on 02/20/2014 09:09 GMT (org.apache.zookeeper.ZooKeeper)
[2016-01-11 12:15:06,579] INFO Client environment:host.name=wifiin-analysis-22 (org.apache.zookeeper.ZooKeeper)
[2016-01-11 12:15:06,579] INFO Client environment:java.version=1.7.0_40 (org.apache.zookeeper.ZooKeeper)
[2016-01-11 12:15:06,579] INFO Client environment:java.vendor=Oracle Corporation (org.apache.zookeeper.ZooKeeper)
[2016-01-11 12:15:06,579] INFO Client environment:java.home=/usr/local/java/jdk1.7.0_40/jre
(org.apache.zookeeper.ZooKeeper)
...
[2016-01-11 12:15:06,580] INFO Client environment:java.io.tmpdir=/tmp (org.apache.zookeeper.ZooKeeper)
[2016-01-11 12:15:06,580] INFO Client environment:java.compiler=<NA(org.apache.zookeeper.ZooKeeper)
[2016-01-11 12:15:06,580] INFO Client environment:os.name=Linux (org.apache.zookeeper.ZooKeeper)
[2016-01-11 12:15:06,580] INFO Client environment:os.arch=amd64 (org.apache.zookeeper.ZooKeeper)
[2016-01-11 12:15:06,580] INFO Client environment:os.version=2.6.32-573.3.1.el6.x86_64 (org.apache.zookeeper.ZooKeeper)
[2016-01-11 12:15:06,580] INFO Client environment:user.name=root (org.apache.zookeeper.ZooKeeper)
[2016-01-11 12:15:06,580] INFO Client environment:user.home=/root (org.apache.zookeeper.ZooKeeper)
[2016-01-11 12:15:06,580] INFO Client environment:user.dir=/usr/local/kafka (org.apache.zookeeper.ZooKeeper)
...[2016-01-11 12:15:17,515] INFO [Kafka Server 0], started (kafka.server.KafkaServer)
[2016-01-11 12:15:17,795] INFO [ReplicaFetcherManager on broker 0] Removed fetcher for partitions [sdkLog,0] (kafka.server.ReplicaFetcherManager)
[2016-01-11 12:15:17,856] INFO [ReplicaFetcherManager on broker 0] Removed fetcher for partitions [sdkLog,0] (kafka.server.ReplicaFetcherManager)
[2016-01-11 14:47:41,816] INFO [Kafka Server 0], shutting down (kafka.server.KafkaServer)
[2016-01-11 14:47:41,818] INFO [Kafka Server 0], Starting controlled shutdown (kafka.server.KafkaServer)
[2016-01-11 14:47:41,981] INFO [Kafka Server 0], Controlled shutdown succeeded (kafka.server.KafkaServer)
[2016-01-11 14:47:41,983] INFO Closing socket connection to /172.16.1.22. (kafka.network.Processor)
[2016-01-11 14:47:41,984] INFO [Socket Server on Broker 0], Shutting down (kafka.network.SocketServer)
[2016-01-11 14:47:41,989] INFO [Socket Server on Broker 0], Shutdown completed (kafka.network.SocketServer)
[2016-01-11 14:47:41,991] INFO [Kafka Request Handler on Broker 0], shutting down (kafka.server.KafkaRequestHandlerPool)
[2016-01-11 14:47:41,994] INFO [Kafka Request Handler on Broker 0], shut down completely (kafka.server.KafkaRequestHandlerPool)
[2016-01-11 14:47:42,267] INFO [Replica Manager on Broker 0]: Shut down (kafka.server.ReplicaManager)
[2016-01-11 14:47:42,267] INFO [ReplicaFetcherManager on broker 0] shutting down (kafka.server.ReplicaFetcherManager)
[2016-01-11 14:47:42,269] INFO [ReplicaFetcherManager on broker 0] shutdown completed (kafka.server.ReplicaFetcherManager)
[2016-01-11 14:47:42,314] INFO [Replica Manager on Broker 0]: Shut down
completely (kafka.server.ReplicaManager)
[2016-01-11 14:47:42,315] INFO Shutting down. (kafka.log.LogManager)
[2016-01-11 14:47:42,376] INFO Shutdown complete. (kafka.log.LogManager)
[2016-01-11 14:47:42,384] INFO Terminate ZkClient event thread.
(org.I0Itec.zkclient.ZkEventThread)
[2016-01-11 14:47:42,396] INFO Session: 0x15210b1e3051357 closed (org.apache.zookeeper.ZooKeeper)
[2016-01-11 14:47:42,396] INFO EventThread shut down (org.apache.zookeeper.ClientCnxn)
[2016-01-11 14:47:42,396] INFO [Kafka Server 0], shut down completed (kafka.server.KafkaServer)
At first, I believed OutOfMemory led to the shutdown, so I edited kafka's oom_score wihtout luck.What should I do to keep the Kafka broker from shutting down?
./kafka-server-start.sh -daemon ../config/server.properties
This may help you！
I have just started learning Kafka/Zookeper. Issue I am facing is, when I try to run the console consumer to verify the message for a topic, I get the below error-
vagrant#debian-70rc1-x64-vbox4210:~$ $KAFKA_HOME/bin/kafka-console-consumer.sh --zookeeper 192.168.33.10:2181 --topic topic-3
[2015-02-22 22:56:36,666] INFO Verifying properties (kafka.utils.VerifiableProperties)
[2015-02-22 22:56:36,698] INFO Property group.id is overridden to console-consumer-92151 (kafka.utils.VerifiableProperties)
[2015-02-22 22:56:36,699] INFO Property fetch.message.max.bytes is overridden to 1048576 (kafka.utils.VerifiableProperties)
[2015-02-22 22:56:36,700] INFO Property consumer.timeout.ms is overridden to -1 (kafka.utils.VerifiableProperties)
[2015-02-22 22:56:36,700] INFO Property socket.timeout.ms is overridden to 30000 (kafka.utils.VerifiableProperties)
[2015-02-22 22:56:36,700] INFO Property auto.offset.reset is overridden to largest (kafka.utils.VerifiableProperties)
[2015-02-22 22:56:36,701] INFO Property zookeeper.connect is overridden to 192.168.33.10:2181 (kafka.utils.VerifiableProperties)
[2015-02-22 22:56:36,701] INFO Property socket.receive.buffer.bytes is overridden to 2097152 (kafka.utils.VerifiableProperties)
[2015-02-22 22:56:36,701] INFO Property fetch.wait.max.ms is overridden to 100 (kafka.utils.VerifiableProperties)
[2015-02-22 22:56:36,702] INFO Property auto.commit.enable is overridden to true (kafka.utils.VerifiableProperties)
[2015-02-22 22:56:36,702] INFO Property refresh.leader.backoff.ms is overridden to 200 (kafka.utils.VerifiableProperties)
[2015-02-22 22:56:36,702] INFO Property auto.commit.interval.ms is overridden to 60000 (kafka.utils.VerifiableProperties)
[2015-02-22 22:56:36,703] INFO Property fetch.min.bytes is overridden to 1 (kafka.utils.VerifiableProperties)
[2015-02-22 22:56:36,734] INFO [console-consumer-92151_debian-70rc1-x64-vbox4210-1424645796729-11446a89], Connecting to zookeeper instance at 192.168.33.10:2181 (kafka.consumer.ZookeeperConsumerConnector)
[2015-02-22 22:56:36,754] INFO Starting ZkClient event thread. (org.I0Itec.zkclient.ZkEventThread)
[2015-02-22 22:56:36,755] INFO Client environment:zookeeper.version=3.3.3-1203054, built on 11/17/2011 05:47 GMT (org.apache.zookeeper.ZooKeeper)
[2015-02-22 22:56:36,756] INFO Client environment:host.name=debian-70rc1-x64-vbox4210.vagrantup.com (org.apache.zookeeper.ZooKeeper)
[2015-02-22 22:56:36,756] INFO Client environment:java.version=1.7.0_75 (org.apache.zookeeper.ZooKeeper)
[2015-02-22 22:56:36,757] INFO Client environment:java.vendor=Oracle Corporation (org.apache.zookeeper.ZooKeeper)
[2015-02-22 22:56:36,757] INFO Client environment:java.home=/usr/lib/jvm/java-7-openjdk-amd64/jre (org.apache.zookeeper.ZooKeeper)
[2015-02-22 22:56:36,757] INFO Client environment:java.class.path=:/usr/local/kafka/kafka-0.8.0-beta1-src//bin/../core/target/scala-2.8.0/kafka_2.8.0-0.8.0-beta1.jar:/usr/local/kafka/kafka-0.8.0-beta1-src//bin/../core/target/scala-2.8.0/kafka-assembly-0.8.0-beta1-deps.jar:/usr/local/kafka/kafka-0.8.0-beta1-src//bin/../perf/target/scala-2.8.0/kafka-perf_2.8.0-0.8.0-beta1.jar:/usr/local/kafka/kafka-0.8.0-beta1-src//bin/../libs/*.jar:/usr/local/kafka/kafka-0.8.0-beta1-src//bin/../kafka*.jar (org.apache.zookeeper.ZooKeeper)
[2015-02-22 22:56:36,758] INFO Client environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib/x86_64-linux-gnu/jni:/lib/x86_64-linux-gnu:/usr/lib/x86_64-linux-gnu:/usr/lib/jni:/lib:/usr/lib (org.apache.zookeeper.ZooKeeper)
[2015-02-22 22:56:36,758] INFO Client environment:java.io.tmpdir=/tmp (org.apache.zookeeper.ZooKeeper)
[2015-02-22 22:56:36,758] INFO Client environment:java.compiler=<NA> (org.apache.zookeeper.ZooKeeper)
[2015-02-22 22:56:36,759] INFO Client environment:os.name=Linux (org.apache.zookeeper.ZooKeeper)
[2015-02-22 22:56:36,759] INFO Client environment:os.arch=amd64 (org.apache.zookeeper.ZooKeeper)
[2015-02-22 22:56:36,759] INFO Client environment:os.version=3.2.0-4-amd64 (org.apache.zookeeper.ZooKeeper)
[2015-02-22 22:56:36,759] INFO Client environment:user.name=vagrant (org.apache.zookeeper.ZooKeeper)
[2015-02-22 22:56:36,760] INFO Client environment:user.home=/home/vagrant (org.apache.zookeeper.ZooKeeper)
[2015-02-22 22:56:36,760] INFO Client environment:user.dir=/home/vagrant (org.apache.zookeeper.ZooKeeper)
[2015-02-22 22:56:36,761] INFO Initiating client connection, connectString=192.168.33.10:2181 sessionTimeout=6000 watcher=org.I0Itec.zkclient.ZkClient#234a98fa (org.apache.zookeeper.ZooKeeper)
[2015-02-22 22:56:36,773] INFO Opening socket connection to server /192.168.33.10:2181 (org.apache.zookeeper.ClientCnxn)
[2015-02-22 22:56:42,778] INFO Terminate ZkClient event thread. (org.I0Itec.zkclient.ZkEventThread)
[2015-02-22 22:56:46,791] INFO Socket connection established to 192.168.33.10/192.168.33.10:2181, initiating session (org.apache.zookeeper.ClientCnxn)
[2015-02-22 22:56:46,801] INFO Session establishment complete on server 192.168.33.10/192.168.33.10:2181, sessionid = 0x14bb3752996000d, negotiated timeout = 6000 (org.apache.zookeeper.ClientCnxn)
[2015-02-22 22:56:46,803] INFO Session: 0x14bb3752996000d closed (org.apache.zookeeper.ZooKeeper)
Exception in thread "main" org.I0Itec.zkclient.exception.ZkTimeoutException: Unable to connect to zookeeper server within timeout: 6000
at org.I0Itec.zkclient.ZkClient.connect(ZkClient.java:880)
at org.I0Itec.zkclient.ZkClient.<init>(ZkClient.java:98)
at org.I0Itec.zkclient.ZkClient.<init>(ZkClient.java:84)
at kafka.consumer.ZookeeperConsumerConnector.connectZk(ZookeeperConsumerConnector.scala:151)
at kafka.consumer.ZookeeperConsumerConnector.<init>(ZookeeperConsumerConnector.scala:112)
at kafka.consumer.ZookeeperConsumerConnector.<init>(ZookeeperConsumerConnector.scala:123)
at kafka.consumer.Consumer$.create(ConsumerConnector.scala:89)
at kafka.consumer.ConsoleConsumer$.main(ConsoleConsumer.scala:178)
at kafka.consumer.ConsoleConsumer.main(ConsoleConsumer.scala)
[2015-02-22 22:56:46,804] INFO EventThread shut down (org.apache.zookeeper.ClientCnxn)
It says-
Exception in thread "main" org.I0Itec.zkclient.exception.ZkTimeoutException: Unable to connect to zookeeper server within timeout: 6000
I verified the topic and it is present. What is going wrong. Please help.
I had the same problem and here is how I fixed it:
1) Stop all Kafka and Zookeeper processes
ps -aux | grep zoo
ps -aux | grep kafka
(then proceed to kill all process ids from above jobs)
2) Run zookeeper
<kafka_dir>/bin/zookeeper-server-start.sh ../config/zookeeper.properties
3) Run kafka server
<kafka_dir>/bin/kafka-server-start.sh ../config/server.properties
Try -> telnet 192.168.33.10 2181
If the telnet fails, then the issue is due to below reason.
Zookeeper Server NRPE iptables chain does not appear to a have a default accept rule.
Unless you just want your own iptables chain for NRPE, you could use:
iptables -I INPUT -s 0/0 -p tcp --dport 2181 -j ACCEPT
This solves the issue.
You can try the follwoing to see if it helps ..
1) Make sure the zookeeper is up and running, try to connect to it using the zkCli client .
2) append --from-beginning at the end of your command to see if you get any diifference in the output , so the command should be changed to :
$KAFKA_HOME/bin/kafka-console-consumer.sh --zookeeper xxx.xx.xx.xxx:2181 --topic topic-3 --from-beginning
In my case it was already connected to the host from which I was trying, so it was not able to make any new connection to the host. I had to change the config: maxClientCnxns so that it was able to make more connection to that host. More details here.