I've been trying to consume messages from broker using kcat, however for some reasons kcat tries to connect to localhost:9092 instead of the broker I've supplied to it using the -b switch.
I've been through all the official documentation and several how-to's but i can not figure this one out.
An example :
kcat -C -b <remote kafka broker ip> -t test-topic1 -o -1 -e
%3|1675853772.320|FAIL|rdkafka#consumer-1| [thrd:localhost:9092/1001]: localhost:9092/1001: Connect to ipv4#127.0.0.1:9092 failed: Connection refused (after 0ms in state CONNECT)
% ERROR: Local: Broker transport failure: localhost:9092/1001: Connect to ipv4#127.0.0.1:9092 failed: Connection refused (after 0ms in state CONNECT)
%3|1675853772.576|FAIL|rdkafka#consumer-1| [thrd:localhost:9092/1001]: localhost:9092/1001: Connect to ipv6#[::1]:9092 failed: Connection refused (after 0ms in state CONNECT)
% ERROR: Local: Broker transport failure: localhost:9092/1001: Connect to ipv6#[::1]:9092 failed: Connection refused (after 0ms in state CONNECT)
I think i'm missing something obvious here, but why does it want to connect to localhost?
If i issue a command like
kcat -b <remote kafka broker ip> -L
it works like a charm and lists topics and partitions from the <remote kafka broker ip>
broker
Have you double-checked your kcat configuration? It might be reading from that if you're not providing your broker explicitly.
Here's a blog post on how to set up your config. If you set it up that way, you don't have to pass in your -b flag every time you run it, either.
why does it want to connect to localhost
Because that is what Kafka's server.properties has listed under advertised.listeners.
https://www.confluent.io/blog/kafka-listeners-explained/
Thanks for all your answers!
In the end realizing that the error was on my side, i went with a properly configured kafka docker image. Worked right out of the box
Related
I'm trying to run Kafka. I'm able to start up zookeeper but when I start the Kafka server using
.\bin\windows\kafka-server-start.bat .\config\server.properties
I get the following error:
WARN [Controller id=0, targetBrokerId=0] Connection to node 0 could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient)
Might anyone know how to resolve this issue? Thanks in advance.
I had this error yesterday and just resolved it this morning. For others who run into a similar issue, my fix was in server.properties to change
listeners=PLAINTEXT://:9092 to listeners=PLAINTEXT://127.0.0.1:9092
I ran into an issue while trying to set up kafaka (version 2.12-2.4.0) on my local machine by following:
https://kafka.apache.org/quickstart
I created a very simple spring-boot app that had a producer and consumer by following some online tutorials. When I would start up my app it would spin for 30 seconds and then start throwing errors in the logs trying to create a topic with connection errors.
I thought maybe my spring-boot app was misconfigured so I tried creating a topic from the command line but I got a similar error:
bin/kafka-topics.sh --create --bootstrap-server localhost:9092 --replication-factor 1 --partitions 1 --topic testing
Error while executing topic command : org.apache.kafka.common.errors.TimeoutException: Timed out waiting for a node assignment.
[2020-02-11 21:57:06,545] ERROR java.util.concurrent.ExecutionException: org.apache.kafka.common.errors.TimeoutException: Timed out waiting for a node assignment.
at org.apache.kafka.common.internals.KafkaFutureImpl.wrapAndThrow(KafkaFutureImpl.java:45)
at org.apache.kafka.common.internals.KafkaFutureImpl.access$000(KafkaFutureImpl.java:32)
at org.apache.kafka.common.internals.KafkaFutureImpl$SingleWaiter.await(KafkaFutureImpl.java:89)
at org.apache.kafka.common.internals.KafkaFutureImpl.get(KafkaFutureImpl.java:260)
at kafka.admin.TopicCommand$AdminClientTopicService.createTopic(TopicCommand.scala:225)
at kafka.admin.TopicCommand$TopicService.createTopic(TopicCommand.scala:194)
at kafka.admin.TopicCommand$TopicService.createTopic$(TopicCommand.scala:189)
at kafka.admin.TopicCommand$AdminClientTopicService.createTopic(TopicCommand.scala:217)
at kafka.admin.TopicCommand$.main(TopicCommand.scala:61)
at kafka.admin.TopicCommand.main(TopicCommand.scala)
Caused by: org.apache.kafka.common.errors.TimeoutException: Timed out waiting for a node assignment.
Kafka and zookeeper were up and running but nothing could connect to kafka.
I noticed that the kafka logs said it was listening on 0.0.0.0:9092:
INFO Awaiting socket connections on 0.0.0.0:9092. (kafka.network.Acceptor)
I went into the server.properties and changed all the localhost values to:
# The address the socket server listens on. It will get the value returned from
# java.net.InetAddress.getCanonicalHostName() if not configured.
# FORMAT:
# listeners = listener_name://host_name:port
# EXAMPLE:
# listeners = PLAINTEXT://your.host.name:9092
listeners=PLAINTEXT://127.0.0.1:9092
This solved the problem. I don't know if this is an issue with my laptop but I wanted to save other people time. I didn't have to change my spring-boot connection configuration, connecting to localhost still worked.
i tried to start zookeeper with:
/opt/Kafka/kafka_2.10-0.10.0.1/bin/zookeeper-server-start.sh
/opt/Kafka/kafka_2.10-0.10.0.1/config/zookeeper.properties
but I got
ERROR Unexpected exception, exiting abnormally (org.apache.zookeeper.server.ZooKeeperServerMain)
java.net.BindException: Address already in use
I restarted the computer and I got the same problem
I tried netstat -nlp|grep 2181 the result:
tcp6 0 0 :::2181 :::* LISTEN
kill -9 2181 --> bash: kill: (2181) - No such process
./zookeeper-server-stop.sh # no result so it works but
./zookeeper-server-start.sh /opt/Kafka/kafka_2.10-0.10.0.1/config/zookeeper.properties # the same exception
what can I do?
Note: when starting kafka server there is no exception
kill -9 2181 --> bash: kill: (2181) - No such process
You're trying to kill a port number, not a process ID.
Address already in use
Okay, so you clearly already have something on that port.
Options:
Edit the zookeeper.properties to bind to a different port. Also edit kafka properties to use the new Zookeeper connection.
You already have a running Zookeeper, which would explain why Kafka would work when starting.
P.S. I would recommend using a later version of Kafka anyway. Lots of good upgrades.
I have been trying to test a kafka installation and using the guide created a producer and consumer. When trying to retrieve a message I get the following error:
WARN Session 0x0 for server null, unexpected error, closing socket connection and
attempting reconnect (org.apache.zookeeper.ClientCnxn)
java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1146)
[2014-03-04 18:01:20,628] INFO Terminate ZkClient event thread. (org.I0Itec.zkclient.ZkEventThread)
[2014-03-04 18:01:21,315] INFO Opening socket connection to server kafka-test/192.xxxxxx.110:2182 (org.apache.zookeeper.ClientCnxn)
[2014-03-04 18:01:21,418] INFO Session: 0x0 closed (org.apache.zookeeper.ZooKeeper)
Exception in thread "main" org.I0Itec.zkclient.exception.ZkTimeoutException: Unable to connect to zookeeper server within timeout: 6000
at org.I0Itec.zkclient.ZkClient.connect(ZkClient.java:880)
at org.I0Itec.zkclient.ZkClient.<init>(ZkClient.java:98)
at org.I0Itec.zkclient.ZkClient.<init>(ZkClient.java:84)
at kafka.consumer.ZookeeperConsumerConnector.connectZk(ZookeeperConsumerConnector.scala:151)
at kafka.consumer.ZookeeperConsumerConnector.<init>(ZookeeperConsumerConnector.scala:112)
at kafka.consumer.ZookeeperConsumerConnector.<init>(ZookeeperConsumerConnector.scala:123)
at kafka.consumer.Consumer$.create(ConsumerConnector.scala:89)
at kafka.consumer.ConsoleConsumer$.main(ConsoleConsumer.scala:178)
at kafka.consumer.ConsoleConsumer.main(ConsoleConsumer.scala)
[2014-03-04 18:01:21,419] INFO EventThread shut down (org.apache.zookeeper.ClientCnxn)
Kafka
Looks like you're not connecting to Zookeeper correctly. I'm not sure of your setup (multi-machine, VMs, containers) so it's hard to say what's wrong. From the debug output I see the following line hinting at your expected Zookeeper IP:
[2014-03-04 18:01:21,315] INFO Opening socket connection to server kafka-test/192.xxxxxx.110:2182 (org.apache.zookeeper.ClientCnxn)
Kafka looks for Zookeeper at the address specified by the zookeeper.connect configuration property in the $KAFKA_HOME/config/server.properties file. Be sure to edit that before starting Kafka. Also, try giving the actual public IP of your Zookeeper instance, not just 127.0.0.1 as that solves a lot of confusion if you're running in containers. In your case it looks like it would be:
zookeeper.connect=192.xxxxxx.110:2182
Also relevant to the Kafka config if you're running on AWS or operating in a container, don't forget to update the following two configuration properties to make sure clients who connect to Kafka see the correct public IP
advertised.host.name
advertised.port
and Kafka sees the correct internal IP
host.name
port
Zookeeper
Zookeeper has some gotchas when setting it up as well. On your Zookeeper instance, don't forget to edit the server configuration property in the zoo.cfg (usually in /etc/zookeeper/conf) file to point to the correct IP for your Zookeeper instance. In your case probably the following:
server.1=192.xxxxxx.110:2888:3888
Those last two ports (2888 3888) are only needed if you're running a Zookeeper cluster (for followers to connect to the leader and Zookeeper leader election, respectively, so be sure to unblock them on firewallish things if you have multiple Zookeeper servers).
Check your zookeeper connection with telnet command:
telnet 192.xxxxxx.110 2181
You probably get an error, in which case check that the process is running:
ps -ef | grep "zookeeper.properties"
If it's not running, start it by going into kafka home directory:
bin/zookeeper-server-start.sh config/zookeeper.properties &
Something wrong with your Zookeper configuration. Make sure your zookeeper is up and running. The default port it runs on is 2181
Bit more info and some code could be useful I believe.
I hit the same issue and the problem was the max client connections property in zookeeper config.
if you see something like "maxClientCnxns = 20" in the config file in /etc/zookeeper/conf, comment it out and restart zookeeper.
You may also check if the all the connections available have already been exhausted. If you are using an API to connect to ZK, make sure you free up the connection after you're done.
I also meet the problem. When I shutdown the firewall of the zk node, it will work.
Kafka 0.8 works great. I am able to use CLI as well as write my own producers/consumers!
Checking Zookeeper... and I see all the topics and partitions created successfully for 0.8.
Kafka 0.7 does not work!
Why Kafka 0.7? I am using Kafka Spout from Storm which is made for Kafka 0.7.
First I just want to run CLI based producer/consumer for Kafka 0.7, which I am unable to. I carry out the following steps:
I delete all the topics/partitions etc. in Zookeeper that were created from my Kafka 0.8
I change the dataDir in zoo.cfg to point to different location.
Now I start the kafka server 0.7. It starts successfully. However I don’t know why it again registers the broker topics I deleted?
Now I start the Kafka Producer :
bin/kafka-console-producer.sh --zookeeper localhost:2181 --topic topicime
& it starts successfully:
[2013-06-28 14:06:05,521] INFO zookeeper state changed (SyncConnected) (org.I0Itec.zkclient.ZkClient)
[2013-06-28 14:06:05,606] INFO Creating async producer for broker id = 0 at 0:0 (kafka.producer.ProducerPool)
Time to send some messages & oops I get this error:
[2013-06-28 14:07:19,650] INFO Disconnecting from 0:0 (kafka.producer.SyncProducer)
[2013-06-28 14:07:19,653] ERROR Connection attempt to 0:0 failed, next attempt in 1 ms (kafka.producer.SyncProducer)
java.net.ConnectException: Connection refused
at sun.nio.ch.Net.connect0(Native Method)
at sun.nio.ch.Net.connect(Net.java:364)
at sun.nio.ch.Net.connect(Net.java:356)
at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:623)
at kafka.producer.SyncProducer.connect(SyncProducer.scala:173)
at kafka.producer.SyncProducer.getOrMakeConnection(SyncProducer.scala:196)
at kafka.producer.SyncProducer.send(SyncProducer.scala:92)
at kafka.producer.SyncProducer.multiSend(SyncProducer.scala:135)
at kafka.producer.async.DefaultEventHandler.send(DefaultEventHandler.scala:58)
at kafka.producer.async.DefaultEventHandler.handle(DefaultEventHandler.scala:44)
at kafka.producer.async.ProducerSendThread.tryToHandle(ProducerSendThread.scala:116)
at scala.collection.immutable.Stream.foreach(Stream.scala:254)
at kafka.producer.async.ProducerSendThread.processEvents(ProducerSendThread.scala:70)
at kafka.producer.async.ProducerSendThread.run(ProducerSendThread.scala:41)
Note that Zookeeper is already running.
Any help would really be appreciated.
EDIT:
I don't even see the topic being created in zookeeper. I am running the following command:
bin/kafka-console-producer.sh --zookeeper localhost:2181 --topic topicime
After the command everything is fine & I get the following message:
[2013-06-28 14:30:17,614] INFO Session establishment complete on server localhost/127.0.0.1:2181, sessionid = 0x13f805c6673004b, negotiated timeout = 6000 (org.apache.zookeeper.ClientCnxn)
[2013-06-28 14:30:17,615] INFO zookeeper state changed (SyncConnected) (org.I0Itec.zkclient.ZkClient)
[2013-06-28 14:30:17,700] INFO Creating async producer for broker id = 0 at 0:0 (kafka.producer.ProducerPool)
However now when i type a string to send I get the above error (Connection refused!)
INFO Disconnecting from 0:0 (kafka.producer.SyncProducer)
The above line has the error hidden in it. 0:0 is not a valid host and port. The solution is to explicitly set the host ip to be registered in Zookeeper by setting the "hostname" property in server.properties.
Consider checking out the storm-kafka fork, available at https://github.com/wurstmeister/storm-kafka-0.8-plus
I'm installing it right now for our servers =).