My Kafka consumer can not read messages from producer.
$ jps
31700 Kafka
11243 Jps
31517 QuorumPeerMain
Feeding topic to producer :
Before edit :
$ bin/kafka-console-producer.sh --broker-list localhost:9092 --topic Hello_Kafka
this is my 1st message
this is my 2nd message
After edit :
$ bin/kafka-console-producer.sh --broker-list localhost:9092 --topic Hello-Kafka
this is my 1st message
this is my 2nd message
Open another terminal and trying to read data from consumer:
$ bin/kafka-console-consumer.sh --zookeeper localhost:2181 --topic Hello-Kafka --from-beginning
After this, I do not see any message on the terminal.
These are some logs which might help to debug :
$ [2017-02-20 08:23:15,000] INFO Expiring session 0x15a49f1f2530020, timeout of 30000ms exceeded (org.apache.zookeeper.server.ZooKeeperServer)
[2017-02-20 08:23:15,000] INFO Expiring session 0x15a49f1f2530022, timeout of 30000ms exceeded (org.apache.zookeeper.server.ZooKeeperServer)
[2017-02-20 08:23:15,000] INFO Expiring session 0x15a49f1f2530021, timeout of 30000ms exceeded (org.apache.zookeeper.server.ZooKeeperServer)
[2017-02-20 08:23:15,000] INFO Processed session termination for sessionid: 0x15a49f1f2530020 (org.apache.zookeeper.server.PrepRequestProcessor)
[2017-02-20 08:23:15,000] INFO Processed session termination for sessionid: 0x15a49f1f2530022 (org.apache.zookeeper.server.PrepRequestProcessor)
[2017-02-20 08:23:15,001] INFO Processed session termination for sessionid: 0x15a49f1f2530021 (org.apache.zookeeper.server.PrepRequestProcessor)
[2017-02-20 08:26:48,764] INFO Accepted socket connection from /127.0.0.1:45806 (org.apache.zookeeper.server.NIOServerCnxnFactory)
[2017-02-20 08:26:48,766] INFO Client attempting to establish new session at /127.0.0.1:45806 (org.apache.zookeeper.server.ZooKeeperServer)
[2017-02-20 08:26:48,768] INFO Established session 0x15a49f1f2530024 with negotiated timeout 30000 for client /127.0.0.1:45806 (org.apache.zookeeper.server.ZooKeeperServer)
[2017-02-20 08:26:48,778] INFO Accepted socket connection from /127.0.0.1:45808 (org.apache.zookeeper.server.NIOServerCnxnFactory)
[2017-02-20 08:26:48,778] INFO Client attempting to establish new session at /127.0.0.1:45808 (org.apache.zookeeper.server.ZooKeeperServer)
[2017-02-20 08:26:48,779] INFO Established session 0x15a49f1f2530025 with negotiated timeout 30000 for client /127.0.0.1:45808 (org.apache.zookeeper.server.ZooKeeperServer)
[2017-02-20 08:26:48,783] INFO Accepted socket connection from /127.0.0.1:45810 (org.apache.zookeeper.server.NIOServerCnxnFactory)
[2017-02-20 08:26:48,783] INFO Client attempting to establish new session at /127.0.0.1:45810 (org.apache.zookeeper.server.ZooKeeperServer)
[2017-02-20 08:26:48,784] INFO Established session 0x15a49f1f2530026 with negotiated timeout 30000 for client /127.0.0.1:45810 (org.apache.zookeeper.server.ZooKeeperServer)
[2017-02-20 08:26:48,865] INFO Accepted socket connection from /127.0.0.1:45812 (org.apache.zookeeper.server.NIOServerCnxnFactory)
[2017-02-20 08:26:48,865] INFO Client attempting to establish new session at /127.0.0.1:45812 (org.apache.zookeeper.server.ZooKeeperServer)
[2017-02-20 08:26:48,867] INFO Established session 0x15a49f1f2530027 with negotiated timeout 6000 for client /127.0.0.1:45812 (org.apache.zookeeper.server.ZooKeeperServer)
[2017-02-20 08:26:48,930] INFO Got user-level KeeperException when processing sessionid:0x15a49f1f2530027 type:create cxid:0x2 zxid:0xa9 txntype:-1 reqpath:n/a Error Path:/consumers Error:KeeperErrorCode = NodeExists for /consumers (org.apache.zookeeper.server.PrepRequestProcessor)
[2017-02-20 08:27:50,978] INFO [Group Metadata Manager on Broker 0]: Removed 0 expired offsets in 0 milliseconds. (kafka.coordinator.GroupMetadataManager)
Related
Zookeeper is accepting a socket connection from 50167(in my case) and then closed it.
[2017-09-04 14:44:23,926] INFO Server environment:user.dir=C:\kafka_2.11-0.11.0.0 (org.apache.zookeeper.server.ZooKeeperServer)
[2017-09-04 14:44:24,013] INFO tickTime set to 3000 (org.apache.zookeeper.server.ZooKeeperServer)
[2017-09-04 14:44:24,029] INFO minSessionTimeout set to -1 (org.apache.zookeeper.server.ZooKeeperServer)
[2017-09-04 14:44:24,045] INFO maxSessionTimeout set to -1 (org.apache.zookeeper.server.ZooKeeperServer)
[2017-09-04 14:44:24,245] INFO binding to port 0.0.0.0/0.0.0.0:2181 (org.apache.zookeeper.server.NIOServerCnxnFactory)
[2017-09-04 14:45:16,525] INFO Accepted socket connection from /0:0:0:0:0:0:0:1:50167 (org.apache.zookeeper.server.NIOServerCnxnFactory)
[2017-09-04 14:45:16,557] INFO Client attempting to establish new session at /0:0:0:0:0:0:0:1:50167 (org.apache.zookeeper.server.ZooKeeperServer)
[2017-09-04 14:45:16,572] INFO Creating new log file: log.b1 (org.apache.zookeeper.server.persistence.FileTxnLog)
[2017-09-04 14:45:16,613] INFO Established session 0x15e4c2b5f7f0000 with negotiated timeout 6000 for client /0:0:0:0:0:0:0:1:50167 (org.apache.zookeeper.server.ZooKeeperServer)
[2017-09-04 14:45:17,939] INFO Processed session termination for sessionid: 0x15e4c2b5f7f0000 (org.apache.zookeeper.server.PrepRequestProcessor)
[2017-09-04 14:45:17,970] INFO Closed socket connection for client /0:0:0:0:0:0:0:1:50167 which had sessionid 0x15e4c2b5f7f0000 (org.apache.zookeeper.server.NIOServerCnxn)
Due to this, Kafka server is also failing to run. Any solution will be helpful.
After, I deleted the old logs at location:
i.e. folders kafka_2.11-0.11.0.0kafka-logs & kafka_2.11-0.11.0.0zookeeper-data, the server started fine.
We use kafka 0.10.1.1 and it is running fine for few hours and sometimes few days. All of sudden it starts giving the below exception and broker loses connection between them. The zookeeper and kafka server processes are running, but not accepting any connection.
We are running kafka and zookeeper on the same node and its a 2 nodes cluster setup. This is just our dev environment.
The below exception was observed in the server log.
2017-03-23 14:05:52,729] WARN [ReplicaFetcherThread-0-38], Error in fetch kafka.server.ReplicaFetcherThread$FetchRequest#1893e027 (kafka.server.ReplicaFetcherThread)
java.io.IOException: Connection to 2 was disconnected before the response was read
at kafka.utils.NetworkClientBlockingOps$$anonfun$blockingSendAndReceive$extension$1$$anonfun$apply$1.apply(NetworkClientBlockingOps.scala:115)
at kafka.utils.NetworkClientBlockingOps$$anonfun$blockingSendAndReceive$extension$1$$anonfun$apply$1.apply(NetworkClientBlockingOps.scala:112)
at scala.Option.foreach(Option.scala:257)
at kafka.utils.NetworkClientBlockingOps$$anonfun$blockingSendAndReceive$extension$1.apply(NetworkClientBlockingOps.scala:112)
at kafka.utils.NetworkClientBlockingOps$$anonfun$blockingSendAndReceive$extension$1.apply(NetworkClientBlockingOps.scala:108)
at kafka.utils.NetworkClientBlockingOps$.recursivePoll$1(NetworkClientBlockingOps.scala:137)
at kafka.utils.NetworkClientBlockingOps$.kafka$utils$NetworkClientBlockingOps$$pollContinuously$extension(NetworkClientBlockingOps.scala:143)
at kafka.utils.NetworkClientBlockingOps$.blockingSendAndReceive$extension(NetworkClientBlockingOps.scala:108)
at kafka.server.ReplicaFetcherThread.sendRequest(ReplicaFetcherThread.scala:253)
at kafka.server.ReplicaFetcherThread.fetch(ReplicaFetcherThread.scala:238)
at kafka.server.ReplicaFetcherThread.fetch(ReplicaFetcherThread.scala:42)
at kafka.server.AbstractFetcherThread.processFetchRequest(AbstractFetcherThread.scala:118)
at kafka.server.AbstractFetcherThread.doWork(AbstractFetcherThread.scala:103)
From Zookeeper.log,
[2017-03-22 22:42:15,600] INFO Client attempting to establish new session at /10.141.202.141:59930 (org.apache.zookeeper.server.ZooKeeperServer)
[2017-03-22 22:42:15,603] INFO Established session 0x25af82f142c0000 with negotiated timeout 6000 for client /11.121.102.441:59930 (org.apache.zookeeper.server.ZooKeeperServer)
[2017-03-22 22:42:29,322] INFO Got user-level KeeperException when processing sessionid:0x25af82f142c0000 type:create cxid:0x3a8 zxid:0x1e0000001a txntype:-1 reqpath:n/a Error Path:/brokers Error:KeeperErrorCode = NodeExists for /brokers (org.apache.zookeeper.server.PrepRequestProcessor)
[2017-03-22 22:42:29,329] INFO Got user-level KeeperException when processing sessionid:0x25af82f142c0000 type:create cxid:0x3a9 zxid:0x1e0000001b txntype:-1 reqpath:n/a Error Path:/brokers/ids Error:KeeperErrorCode = NodeExists for /brokers/ids (org.apache.zookeeper.server.PrepRequestProcessor)
[2017-03-22 22:42:32,943] INFO Accepted socket connection from /17.150.218.7:58233 (org.apache.zookeeper.server.NIOServerCnxnFactory)
[2017-03-22 22:42:32,944] WARN Connection request from old client /17.150.218.7:58233; will be dropped if server is in r-o mode (org.apache.zookeeper.server.ZooKeeperServer)
[2017-03-22 22:42:32,944] INFO Client attempting to establish new session at /17.150.218.7:58233 (org.apache.zookeeper.server.ZooKeeperServer)
[2017-03-22 22:42:32,947] INFO Established session 0x25af82f142c0001 with negotiated timeout 30000 for client /17.121.102.241:58233 (org.apache.zookeeper.server.ZooKeeperServer)
[2017-03-22 22:42:33,402] INFO Processed session termination for sessionid: 0x25af82f142c0001 (org.apache.zookeeper.server.PrepRequestProcessor)
[2017-03-22 22:42:33,405] INFO Closed socket connection for client /17.150.218.7:58233 which had sessionid 0x25af82f142c0001 (org.apache.zookeeper.server.NIOServerCnxn)
Thanks
I installed fresh zookeeper & kafka both. I started them both. Then when I want to see the list of topics with this command:
bin/kafka-topics.sh --list --zookeeper localhost 2181
It gives me the socket connection closed. Here is the screen shot:
darpanshah#darpan-ubuntu:/opt/Kafka$ bin/kafka-topics.sh --list --zookeeper localhost 2181
2016-08-17 10:44:44,053] INFO Accepted socket connection from /127.0.0.1:48452 (org.apache.zookeeper.server.NIOServerCnxnFactory)
[2016-08-17 10:44:44,059] INFO Client attempting to establish new session at /127.0.0.1:48452 (org.apache.zookeeper.server.ZooKeeperServer)
[2016-08-17 10:44:44,069] INFO Established session 0x15698f5ac360001 with negotiated timeout 30000 for client /127.0.0.1:48452 (org.apache.zookeeper.server.ZooKeeperServer)
[2016-08-17 10:44:44,095] INFO Processed session termination for sessionid: 0x15698f5ac360001 (org.apache.zookeeper.server.PrepRequestProcessor)
[2016-08-17 10:44:44,105] INFO Closed socket connection for client /127.0.0.1:48452 which had sessionid 0x15698f5ac360001 (org.apache.zookeeper.server.NIOServerCnxn)
darpanshah#darpan-ubuntu:/opt/Kafka$
Thanks in advance. Got stuck.
That is not an error. The topic details are fetched from Zookeeper. Hence the client (invoked by kafka-topics.sh) first connects to Zookeeper, then establishes a session, gets the data and then disconnects at the end.
This is the expected behavior of any clients that will get some data from Zookeeper.
when starting the kafka broker i am getting some error:
I am giving last few lines of the error log:
INFO zookeeper state changed (SyncConnected) (org.I0Itec.zkclient.ZkClient)
[2016-05-12 01:07:01,759] INFO Log directory '/var/logs/service-bridge-logs' not found, creating it. (kafka.log.LogManager)
[2016-05-12 01:07:01,778] INFO Loading logs. (kafka.log.LogManager)
[2016-05-12 01:07:01,796] INFO Logs loading complete. (kafka.log.LogManager)
[2016-05-12 01:07:01,797] INFO Starting log cleanup with a period of 300000 ms. (kafka.log.LogManager)
[2016-05-12 01:07:01,806] INFO Starting log flusher with a default period of 9223372036854775807 ms. (kafka.log.LogManager)
[2016-05-12 01:07:01,874] INFO Awaiting socket connections on gns3-d.cloudapp.net:9092. (kafka.network.Acceptor)
[2016-05-12 01:07:01,875] INFO [Socket Server on Broker 2], Started (kafka.network.SocketServer)
[2016-05-12 01:07:02,042] INFO Will not load MX4J, mx4j-tools.jar is not in the classpath (kafka.utils.Mx4jLoader$)
[2016-05-12 01:07:02,168] INFO 2 successfully elected as leader (kafka.server.ZookeeperLeaderElector)
[2016-05-12 01:07:02,386] INFO Registered broker 2 at path /brokers/ids/2 with address 10.1.0.4:9092. (kafka.utils.ZkUtils$)
[2016-05-12 01:07:02,416] INFO [Kafka Server 2], started (kafka.server.KafkaServer)
[2016-05-12 01:07:02,529] INFO New leader is 2 (kafka.server.ZookeeperLeaderElector$LeaderChangeListener)
[2016-05-12 01:07:25,798] ERROR Closing socket for /40.122.64.23 because of error (kafka.network.Processor)
java.io.IOException: Connection reset by peer
at sun.nio.ch.FileDispatcherImpl.read0(Native Method)
at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39)
at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223)
at sun.nio.ch.IOUtil.read(IOUtil.java:197)
at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:380)
at kafka.utils.Utils$.read(Utils.scala:380)
at kafka.network.BoundedByteBufferReceive.readFrom(BoundedByteBufferReceive.scala:54)
at kafka.network.Processor.read(SocketServer.scala:444)
at kafka.network.Processor.run(SocketServer.scala:340)
at java.lang.Thread.run(Thread.java:745)
while in zookeeper sidealso am getting few error:
INFO Established session 0x154a35b64f40000 with negotiated timeout 6000 for client /10.1.0.4:36673 (org.apache.zookeeper.server.ZooKeeperServer)
[2016-05-12 01:07:02,313] INFO Got user-level KeeperException when processing sessionid:0x154a35b64f40000 type:delete cxid:0x1d zxid:0x52 txntype:-1 reqpath:n/a Error Path:/admin/preferred_replica_election Error:KeeperErrorCode = NoNode for /admin/preferred_replica_election (org.apache.zookeeper.server.PrepRequestProcessor)
[2016-05-12 01:08:33,001] INFO Expiring session 0x154a35b64f40000, timeout of 6000ms exceeded (org.apache.zookeeper.server.ZooKeeperServer)
[2016-05-12 01:08:33,001] INFO Processed session termination for sessionid: 0x154a35b64f40000 (org.apache.zookeeper.server.PrepRequestProcessor)
[2016-05-12 01:08:33,017] INFO Closed socket connection for client /10.1.0.4:36673 which had sessionid 0x154a35b64f40000 (org.apache.zookeeper.server.NIOServerCnxn)
Any idea guys??
Thanks in advance..
As user avr pointed out, this is a known bug in Kafka 0.8.2.x
These are routine informational messages misclassified as ERROR.
As of Kafka v0.9.0.0, the loglevel has been corrected to WARN.
We are using Kafka high level consumer , and we are able to successfully consume messages but the zookeeper connections keep expiring and reestablishing.
I am wondering why are there no heartbeats to keep the connections alive:
Kafka Consumer Logs
====================
[localhost-startStop-1-SendThread(10.41.105.23:2181)] [ClientCnxn$SendThread] [line : 1096 ] - Client session timed out, have not heard from server in 2666ms for sessionid 0x153175bd3860159, closing socket connection and attempting reconnect
2016-03-08 18:00:06,750 INFO [localhost-startStop-1-SendThread(10.41.105.23:2181)] [ClientCnxn$SendThread] [line : 975 ] - Opening socket connection to server 10.41.105.23/10.41.105.23:2181. Will not attempt to authenticate using SASL (unknown error)
2016-03-08 18:00:06,823 INFO [localhost-startStop-1-SendThread(10.41.105.23:2181)] [ClientCnxn$SendThread] [line : 852 ] - Socket connection established to 10.41.105.23/10.41.105.23:2181, initiating session
2016-03-08 18:00:06,892 INFO [localhost-startStop-1-SendThread(10.41.105.23:2181)] [ClientCnxn$SendThread] [line : 1235 ] - Session establishment complete on server 10.41.105.23/10.41.105.23:2181, sessionid = 0x153175bd3860159, negotiated timeout = 4000
Zookeeper Logs
==================
[2016-03-08 17:44:37,722] INFO Accepted socket connection from /10.10.113.92:51333 (org.apache.zookeeper.server.NIOServerCnxnFactory)
[2016-03-08 17:44:37,742] INFO Client attempting to renew session 0x153175bd3860159 at /10.10.113.92:51333 (org.apache.zookeeper.server.ZooKeeperServer)
[2016-03-08 17:44:37,742] INFO Established session 0x153175bd3860159 with negotiated timeout 4000 for client /10.10.113.92:51333 (org.apache.zookeeper.server.ZooKeeperServer)
[2016-03-08 17:46:56,000] INFO Expiring session 0x153175bd3860151, timeout of 4000ms exceeded (org.apache.zookeeper.server.ZooKeeperServer)
[2016-03-08 17:46:56,001] INFO Processed session termination for sessionid: 0x153175bd3860151 (org.apache.zookeeper.server.PrepRequestProcessor)
[2016-03-08 17:46:56,011] INFO Closed socket connection for client /10.10.114.183:38324 which had sessionid 0x153175bd3860151 (org.apache.zookeeper.server.NIOServerCnxn)
Often ZooKeeper session timeouts are caused by "soft failures," which are most commonly a garbage collection pause. Turn on GC logging and see if a long GC occurs at the time the connection times out. Also, read about JVM tuning in Kafka.
[2016-03-08 17:46:56,000] INFO Expiring session 0x153175bd3860151,
timeout of 4000ms exceeded (org.apache.zookeeper.server.ZooKeeperServer)
What is Zookeeper's maxSessionTimeout?
If it's just 4000ms (4 seconds), then it's way too small.
In Cloudera distribution of Hadoop, ZK's maxSessionTimeout is by default 40s
(40000ms).
As explained in ZK configuration -
https://zookeeper.apache.org/doc/r3.4.5/zookeeperAdmin.html
it defaults 20 ticks
(and one tick by default is 2 seconds).