Working on setting up a monitoring system for a Kafka cluster using statsd. I'm using the statsd library here. Currently my broker server won't start up.
I am positive my issue lies in this configuration line inside my server.properties file: metric.reporters=com.airbnb.kafka.kafka08.StatsdMetricsReporter.
When I comment that line out, the server starts up. Hell, I even get the statsd confirmation like this:
[2017-06-06 15:19:35,669] INFO Reporter is enabled and starting... (com.airbnb.metrics.StatsDReporter)
[2017-06-06 15:19:35,679] INFO Started Reporter with host=localhost, port=8125, polling_period_secs=10, prefix= (com.airbnb.metrics.StatsDReporter)
However, stats aren't reported (I believe) because this is also true: metric.reporters = []. Therefore, that line that's causing the issue must exist in the properties file, right?
When I try to start the server, it fails with this message:
[2017-06-06 15:21:34,712] FATAL [Kafka Server 0], Fatal error during KafkaServer startup. Prepare to shutdown (kafka.server.KafkaServer)
java.lang.ClassCastException: class com.airbnb.kafka.kafka08.StatsdMetricsReporter
at java.lang.Class.asSubclass(Class.java:3404)
at org.apache.kafka.common.utils.Utils.newInstance(Utils.java:356)
at org.apache.kafka.common.config.AbstractConfig.getConfiguredInstances(AbstractConfig.java:243)
at kafka.server.KafkaServer.startup(KafkaServer.scala:198)
at kafka.server.KafkaServerStartable.startup(KafkaServerStartable.scala:39)
at kafka.Kafka$.main(Kafka.scala:67)
at kafka.Kafka.main(Kafka.scala)
[2017-06-06 15:21:34,713] INFO [Kafka Server 0], shutting down (kafka.server.KafkaServer)
[2017-06-06 15:21:34,714] INFO Terminate ZkClient event thread. (org.I0Itec.zkclient.ZkEventThread)
[2017-06-06 15:21:34,718] INFO Session: 0x15c7ed717e30004 closed (org.apache.zookeeper.ZooKeeper)
[2017-06-06 15:21:34,720] INFO EventThread shut down for session: 0x15c7ed717e30004 (org.apache.zookeeper.ClientCnxn)
[2017-06-06 15:21:34,720] INFO [Kafka Server 0], shut down completed (kafka.server.KafkaServer)
[2017-06-06 15:21:34,720] FATAL Fatal error during KafkaServerStartable startup. Prepare to shutdown (kafka.server.KafkaServerStartable)
java.lang.ClassCastException: class com.airbnb.kafka.kafka08.StatsdMetricsReporter
at java.lang.Class.asSubclass(Class.java:3404)
at org.apache.kafka.common.utils.Utils.newInstance(Utils.java:356)
at org.apache.kafka.common.config.AbstractConfig.getConfiguredInstances(AbstractConfig.java:243)
at kafka.server.KafkaServer.startup(KafkaServer.scala:198)
at kafka.server.KafkaServerStartable.startup(KafkaServerStartable.scala:39)
at kafka.Kafka$.main(Kafka.scala:67)
at kafka.Kafka.main(Kafka.scala)
[2017-06-06 15:21:34,721] INFO [Kafka Server 0], shutting down (kafka.server.KafkaServer)
For the record, here is my config:
kafka:type=com.airbnb.kafka.kafka08.StatsdMetricsReporter
metric.reporters=com.airbnb.kafka.kafka08.StatsdMetricsReporter
kafka.metric.reporters=com.airbnb.kafka.kafka08.StatsdMetricsReporter
external.kafka.statsd.reporter.enabled=true
external.kafka.statsd.host=localhost
external.kafka.statsd.port=8125
external.kafka.statsd.metrics.prefix=
external.kafka.statsd.tag.enabled=true
I found the answer. I had to use kafka09 instead of kafka08.
Related
I'm learning kafka at the moment but unfortunately I'm stuck because my kafka can't start when I run the following command in cmd
kafka-server-start.bat .\config\server.properties
my zookeeper works very well
here is the error
[2021-09-23 17:18:03,849] INFO Opening socket connection to server localhost/<unresolved>:2181. Will not attempt to authenticate using SASL (unknown error) (org.apache.zookeeper.ClientCnxn)
[2021-09-23 17:18:03,850] WARN Session 0x0 for server localhost/<unresolved>:2181, unexpected error, closing socket connection and attempting reconnect (org.apache.zookeeper.ClientCnxn)
java.nio.channels.UnresolvedAddressException
at java.base/sun.nio.ch.Net.checkAddress(Net.java:149)
at java.base/sun.nio.ch.Net.checkAddress(Net.java:157)
at java.base/sun.nio.ch.SocketChannelImpl.checkRemote(SocketChannelImpl.java:816)
at java.base/sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:839)
at org.apache.zookeeper.ClientCnxnSocketNIO.registerAndConnect(ClientCnxnSocketNIO.java:277)
at org.apache.zookeeper.ClientCnxnSocketNIO.connect(ClientCnxnSocketNIO.java:287)
at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:1021)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1064)
[2021-09-23 17:18:04,318] INFO [ZooKeeperClient] Closing. (kafka.zookeeper.ZooKeeperClient)
[2021-09-23 17:18:04,956] INFO Opening socket connection to server localhost/<unresolved>:2181. Will not attempt to authenticate using SASL (unknown error) (org.apache.zookeeper.ClientCnxn)
[2021-09-23 17:18:05,061] INFO Session: 0x0 closed (org.apache.zookeeper.ZooKeeper)
[2021-09-23 17:18:05,061] INFO EventThread shut down for session: 0x0 (org.apache.zookeeper.ClientCnxn)
[2021-09-23 17:18:05,069] INFO [ZooKeeperClient] Closed. (kafka.zookeeper.ZooKeeperClient)
[2021-09-23 17:18:05,082] ERROR Fatal error during KafkaServer startup. Prepare to shutdown (kafka.server.KafkaServer)
kafka.zookeeper.ZooKeeperClientTimeoutException: Timed out waiting for connection while in state: CONNECTING
at kafka.zookeeper.ZooKeeperClient.$anonfun$waitUntilConnected$3(ZooKeeperClient.scala:230)
at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:12)
at kafka.utils.CoreUtils$.inLock(CoreUtils.scala:251)
at kafka.zookeeper.ZooKeeperClient.waitUntilConnected(ZooKeeperClient.scala:226)
at kafka.zookeeper.ZooKeeperClient.<init>(ZooKeeperClient.scala:95)
at kafka.zk.KafkaZkClient$.apply(KafkaZkClient.scala:1581)
at kafka.server.KafkaServer.createZkClient$1(KafkaServer.scala:348)
at kafka.server.KafkaServer.initZkClient(KafkaServer.scala:372)
at kafka.server.KafkaServer.startup(KafkaServer.scala:202)
at kafka.server.KafkaServerStartable.startup(KafkaServerStartable.scala:38)
at kafka.Kafka$.main(Kafka.scala:75)
at kafka.Kafka.main(Kafka.scala)
[2021-09-23 17:18:05,087] INFO shutting down (kafka.server.KafkaServer)
[2021-09-23 17:18:05,097] WARN Cannot invoke "kafka.utils.KafkaScheduler.shutdown()" because the return value of "kafka.server.KafkaServer.kafkaScheduler()" is null (kafka.utils.CoreUtils$)
java.lang.NullPointerException: Cannot invoke "kafka.utils.KafkaScheduler.shutdown()" because the return value of "kafka.server.KafkaServer.kafkaScheduler()" is null
at kafka.server.KafkaServer.$anonfun$shutdown$6(KafkaServer.scala:579)
at kafka.utils.CoreUtils$.swallow(CoreUtils.scala:86)
at kafka.server.KafkaServer.shutdown(KafkaServer.scala:579)
at kafka.server.KafkaServer.startup(KafkaServer.scala:329)
at kafka.server.KafkaServerStartable.startup(KafkaServerStartable.scala:38)
at kafka.Kafka$.main(Kafka.scala:75)
at kafka.Kafka.main(Kafka.scala)
[2021-09-23 17:18:05,111] INFO shut down completed (kafka.server.KafkaServer)
[2021-09-23 17:18:05,112] ERROR Exiting Kafka. (kafka.server.KafkaServerStartable)
[2021-09-23 17:18:05,116] INFO shutting down (kafka.server.KafkaServer)
i use kafka 2.12-2
java 17
I had the same problem (I was using Java 17). Following the advice of #OneCricketeer I switched to java 11 and everything worked as expected.
[2021-04-08 02:53:18,713] INFO [ZooKeeperClient Kafka server] Closing. (kafka.zookeeper.ZooKeeperClient)
[2021-04-08 02:53:33,182] WARN Client session timed out, have not heard from server in 18000ms for sessionid 0x0 (org.apache.zookeeper.ClientCnxn)
[2021-04-08 02:53:33,288] INFO Session: 0x0 closed (org.apache.zookeeper.ZooKeeper)
[2021-04-08 02:53:33,288] INFO EventThread shut down for session: 0x0 (org.apache.zookeeper.ClientCnxn)
[2021-04-08 02:53:33,290] INFO [ZooKeeperClient Kafka server] Closed. (kafka.zookeeper.ZooKeeperClient)
[2021-04-08 02:53:33,295] ****ERROR Fatal error during KafkaServer startup. Prepare to shutdown (kafka.server.KafkaServer)
kafka.zookeeper.ZooKeeperClientTimeoutException: Timed out waiting for connection while in state: CONNECTING**
at kafka.zookeeper.ZooKeeperClient.waitUntilConnected(ZooKeeperClient.scala:262)
at kafka.zookeeper.ZooKeeperClient.<init>(ZooKeeperClient.scala:119)
at kafka.zk.KafkaZkClient$.apply(KafkaZkClient.scala:1881)
at kafka.server.KafkaServer.createZkClient$1(KafkaServer.scala:441)
at kafka.server.KafkaServer.initZkClient(KafkaServer.scala:466)
at kafka.server.KafkaServer.startup(KafkaServer.scala:233)
at kafka.server.KafkaServerStartable.startup(KafkaServerStartable.scala:44)
at kafka.Kafka$.main(Kafka.scala:82)
at kafka.Kafka.main(Kafka.scala)
[2021-04-08 02:53:33,300] INFO shutting down (kafka.server.KafkaServer)
[2021-04-08 02:53:33,308] INFO App info kafka.server for 0 unregistered (org.apache.kafka.common.utils.AppInfoParser)
[2021-04-08 02:53:33,309] INFO shut down completed (kafka.server.KafkaServer)
[2021-04-08 02:53:33,310] ERROR Exiting Kafka. (kafka.server.KafkaServerStartable)
[2021-04-08 02:53:33,311] INFO shutting down (kafka.server.KafkaServer)
It seems that the zk server is not started. Kafka needs zookeeper, you need to download and start zookeeper before start kafka server.
Apache Kafka is failing to start. It shows in its logs "Failed to get cluster id from Zookeeper. This can happen if /cluster/id is deleted from Zookeeper."
How can I check the "/cluster/id"?
previously, when kafka is failing to start it is because I updated java on my server. So, I need to re-direct $JAVA_HOME to the new path & restart kafka and it will work again just fine. But, this case is different, I checked $JAVA_HOME & it's correct. So, I wanted to know more details of this issue. I read one of the logs file in /opt/kafka/logs and I got this log:
[2019-08-21 10:18:40,472] ERROR Fatal error during KafkaServer startup. Prepare to shutdown (kafka.server.KafkaServer)
org.apache.kafka.common.KafkaException: Failed to get cluster id from Zookeeper. This can happen if /cluster/id is deleted from Zookeeper.
at kafka.zk.KafkaZkClient.$anonfun$createOrGetClusterId$1(KafkaZkClient.scala:1498)
at scala.Option.getOrElse(Option.scala:138)
at kafka.zk.KafkaZkClient.createOrGetClusterId(KafkaZkClient.scala:1498)
at kafka.server.KafkaServer.$anonfun$getOrGenerateClusterId$1(KafkaServer.scala:390)
at scala.Option.getOrElse(Option.scala:138)
at kafka.server.KafkaServer.getOrGenerateClusterId(KafkaServer.scala:390)
at kafka.server.KafkaServer.startup(KafkaServer.scala:208)
at kafka.server.KafkaServerStartable.startup(KafkaServerStartable.scala:38)
at kafka.Kafka$.main(Kafka.scala:75)
at kafka.Kafka.main(Kafka.scala)
[2019-08-21 10:18:40,476] INFO shutting down (kafka.server.KafkaServer)
[2019-08-21 10:18:40,488] INFO [ZooKeeperClient] Closing. (kafka.zookeeper.ZooKeeperClient)
[2019-08-21 10:18:40,500] INFO Session: 0x100494a8714000a closed (org.apache.zookeeper.ZooKeeper)
[2019-08-21 10:18:40,505] INFO EventThread shut down for session: 0x100494a8714000a (org.apache.zookeeper.ClientCnxn)
[2019-08-21 10:18:40,507] INFO [ZooKeeperClient] Closed. (kafka.zookeeper.ZooKeeperClient)
[2019-08-21 10:18:40,517] INFO shut down completed (kafka.server.KafkaServer)
[2019-08-21 10:18:40,517] ERROR Exiting Kafka. (kafka.server.KafkaServerStartable)
[2019-08-21 10:18:40,554] INFO shutting down (kafka.server.KafkaServer)
when I read the previous Error
Failed to get cluster id from Zookeeper. This can happen if /cluster/id is deleted from Zookeeper.
I though that "zookeeper id" in this path was deleted by mistake /tmp/zookeeper/myid but the file is still there with the corresponding server number written in it & zk is working fine. my kafka version is 2.2.0 .
I searched online for this ERROR "Failed to get cluster id from Zookeeper. This can happen if /cluster/id is deleted from Zookeeper."
but honestly, I did not find any thing helpful.
I am trying to run kafka server on Ubuntu. When I am running this command:
> bin/zookeeper-server-start.sh config/zookeeper.properties
I am not getting anything and when I am trying to run Kafka server by this command:
> bin/kafka-server-start.sh config/server.properties
I am getting this error.
WARN Session 0x0 for server null, unexpected error, closing socket connection and attempting reconnect (org.apache.zookeeper.ClientCnxn)
java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:361)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1081)
[2017-05-23 13:22:45,136] INFO Terminate ZkClient event thread. (org.I0Itec.zkclient.ZkEventThread)
[2017-05-23 13:22:45,754] INFO Opening socket connection to server localhost/127.0.0.1:2181. Will not attempt to authenticate using SASL (unknown error) (org.apache.zookeeper.ClientCnxn)
[2017-05-23 13:22:45,857] INFO Session: 0x0 closed (org.apache.zookeeper.ZooKeeper)
[2017-05-23 13:22:45,859] FATAL Fatal error during KafkaServer startup. Prepare to shutdown (kafka.server.KafkaServer)
org.I0Itec.zkclient.exception.ZkTimeoutException: Unable to connect to zookeeper server within timeout: 6000
at org.I0Itec.zkclient.ZkClient.connect(ZkClient.java:1223)
at org.I0Itec.zkclient.ZkClient.<init>(ZkClient.java:155)
at org.I0Itec.zkclient.ZkClient.<init>(ZkClient.java:129)
at kafka.utils.ZkUtils$.createZkClientAndConnection(ZkUtils.scala:89)
at kafka.utils.ZkUtils$.apply(ZkUtils.scala:71)
at kafka.server.KafkaServer.initZk(KafkaServer.scala:278)
at kafka.server.KafkaServer.startup(KafkaServer.scala:168)
at kafka.server.KafkaServerStartable.startup(KafkaServerStartable.scala:37)
at kafka.Kafka$.main(Kafka.scala:67)
at kafka.Kafka.main(Kafka.scala)
[2017-05-23 13:22:45,859] INFO EventThread shut down (org.apache.zookeeper.ClientCnxn)
[2017-05-23 13:22:45,860] INFO shutting down (kafka.server.KafkaServer)
[2017-05-23 13:22:45,867] INFO shut down completed (kafka.server.KafkaServer)
[2017-05-23 13:22:45,868] FATAL Fatal error during KafkaServerStartable startup. Prepare to shutdown (kafka.server.KafkaServerStartable)
org.I0Itec.zkclient.exception.ZkTimeoutException: Unable to connect to zookeeper server within timeout: 6000
at org.I0Itec.zkclient.ZkClient.connect(ZkClient.java:1223)
at org.I0Itec.zkclient.ZkClient.<init>(ZkClient.java:155)
at org.I0Itec.zkclient.ZkClient.<init>(ZkClient.java:129)
at kafka.utils.ZkUtils$.createZkClientAndConnection(ZkUtils.scala:89)
at kafka.utils.ZkUtils$.apply(ZkUtils.scala:71)
at kafka.server.KafkaServer.initZk(KafkaServer.scala:278)
at kafka.server.KafkaServer.startup(KafkaServer.scala:168)
at kafka.server.KafkaServerStartable.startup(KafkaServerStartable.scala:37)
at kafka.Kafka$.main(Kafka.scala:67)
at kafka.Kafka.main(Kafka.scala)
[2017-05-23 13:22:45,871] INFO shutting down (kafka.server.KafkaServer)
Kafka output suggests zookeeper is not starting.
If zookeeper-server-start.sh produces no output, please check zookeeper.log. If that doesn't help, try finding the zookeeper java process using ps, e.g.:
ps -elf | grep zookeeper
If it's running you can use the PID of the process to obtain a thread dump or attach a debugger to see where it's stuck at.
Hope that helps.
when starting the kafka broker i am getting some error:
I am giving last few lines of the error log:
INFO zookeeper state changed (SyncConnected) (org.I0Itec.zkclient.ZkClient)
[2016-05-12 01:07:01,759] INFO Log directory '/var/logs/service-bridge-logs' not found, creating it. (kafka.log.LogManager)
[2016-05-12 01:07:01,778] INFO Loading logs. (kafka.log.LogManager)
[2016-05-12 01:07:01,796] INFO Logs loading complete. (kafka.log.LogManager)
[2016-05-12 01:07:01,797] INFO Starting log cleanup with a period of 300000 ms. (kafka.log.LogManager)
[2016-05-12 01:07:01,806] INFO Starting log flusher with a default period of 9223372036854775807 ms. (kafka.log.LogManager)
[2016-05-12 01:07:01,874] INFO Awaiting socket connections on gns3-d.cloudapp.net:9092. (kafka.network.Acceptor)
[2016-05-12 01:07:01,875] INFO [Socket Server on Broker 2], Started (kafka.network.SocketServer)
[2016-05-12 01:07:02,042] INFO Will not load MX4J, mx4j-tools.jar is not in the classpath (kafka.utils.Mx4jLoader$)
[2016-05-12 01:07:02,168] INFO 2 successfully elected as leader (kafka.server.ZookeeperLeaderElector)
[2016-05-12 01:07:02,386] INFO Registered broker 2 at path /brokers/ids/2 with address 10.1.0.4:9092. (kafka.utils.ZkUtils$)
[2016-05-12 01:07:02,416] INFO [Kafka Server 2], started (kafka.server.KafkaServer)
[2016-05-12 01:07:02,529] INFO New leader is 2 (kafka.server.ZookeeperLeaderElector$LeaderChangeListener)
[2016-05-12 01:07:25,798] ERROR Closing socket for /40.122.64.23 because of error (kafka.network.Processor)
java.io.IOException: Connection reset by peer
at sun.nio.ch.FileDispatcherImpl.read0(Native Method)
at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39)
at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223)
at sun.nio.ch.IOUtil.read(IOUtil.java:197)
at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:380)
at kafka.utils.Utils$.read(Utils.scala:380)
at kafka.network.BoundedByteBufferReceive.readFrom(BoundedByteBufferReceive.scala:54)
at kafka.network.Processor.read(SocketServer.scala:444)
at kafka.network.Processor.run(SocketServer.scala:340)
at java.lang.Thread.run(Thread.java:745)
while in zookeeper sidealso am getting few error:
INFO Established session 0x154a35b64f40000 with negotiated timeout 6000 for client /10.1.0.4:36673 (org.apache.zookeeper.server.ZooKeeperServer)
[2016-05-12 01:07:02,313] INFO Got user-level KeeperException when processing sessionid:0x154a35b64f40000 type:delete cxid:0x1d zxid:0x52 txntype:-1 reqpath:n/a Error Path:/admin/preferred_replica_election Error:KeeperErrorCode = NoNode for /admin/preferred_replica_election (org.apache.zookeeper.server.PrepRequestProcessor)
[2016-05-12 01:08:33,001] INFO Expiring session 0x154a35b64f40000, timeout of 6000ms exceeded (org.apache.zookeeper.server.ZooKeeperServer)
[2016-05-12 01:08:33,001] INFO Processed session termination for sessionid: 0x154a35b64f40000 (org.apache.zookeeper.server.PrepRequestProcessor)
[2016-05-12 01:08:33,017] INFO Closed socket connection for client /10.1.0.4:36673 which had sessionid 0x154a35b64f40000 (org.apache.zookeeper.server.NIOServerCnxn)
Any idea guys??
Thanks in advance..
As user avr pointed out, this is a known bug in Kafka 0.8.2.x
These are routine informational messages misclassified as ERROR.
As of Kafka v0.9.0.0, the loglevel has been corrected to WARN.