i installed latest version of kafka on ubuntu 20.04 and when i try to create a topic , i have this error :
~/Kafka/bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic TutorialTopic
Exception in thread "main" kafka.zookeeper.ZooKeeperClientTimeoutException: Timed out waiting for connection while in state: CONNECTING
at kafka.zookeeper.ZooKeeperClient.waitUntilConnected(ZooKeeperClient.scala:271)
at kafka.zookeeper.ZooKeeperClient.<init>(ZooKeeperClient.scala:125)
at kafka.zk.KafkaZkClient$.apply(KafkaZkClient.scala:1948)
at kafka.admin.TopicCommand$ZookeeperTopicService$.apply(TopicCommand.scala:378)
at kafka.admin.TopicCommand$.main(TopicCommand.scala:56)
at kafka.admin.TopicCommand.main(TopicCommand.scala)```
i tried :
- to modify zookeeper timeout connection time to 18000->60000
- change zookeeper localhost:2181 to zookeeper zookeeper:2181
but still same issue, do you know why please ?
Related
I'm trying to run a kafka cluster with this command :
kafka-topics.sh --bootstrap-server 127.0.0.1:2181 --topic first_topic --create --partitions 3 --replication-factor 1
and i get this as an error:
[2022-02-03 11:25:28,635] WARN [AdminClient clientId=adminclient-1] Connection to node -1 (/127.0.0.1:2181) could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient)
So i tried to look at kafka_2.12-3.1.0\config\server.properties i have
listeners=PLAINTEXT://localhost:9092
Any help will be very appreciated.
2181 is typically the port used by ZooKeeper. If you want to specify that, and you're not running Kafka in KRaft (zookeeper-less mode) then you need to do as #Umeshwaran said and use the --zookeeper argument.
However, you can use --bootstrap-server, but if you are doing so then specify the broker address and port, which from your listeners config is 9092:
kafka-topics.sh --bootstrap-server 127.0.0.1:9092 --topic first_topic --create --partitions 3 --replication-factor 1
This article should clarify things.
This is because your zookeeper is not working ,I was facing the same issue but when I run the zookeeper with(I am using wsl)
"zookeeper-server-start.sh ~/kafka_2.13-3.3.1/config/zookeeper.properties"
command it starts working
I am running Kafka also if you don't how to run, use this command
"kafka-server-start.sh ~/kafka_2.13-3.3.1/config/server.properties"
hint- modify kafka_2.13-3.3.1 this with the correct version of kafka you are using
I am trying to create my first topic by using the command below:
./bin/kafka-topics.sh --bootstrap-server localhost:2181 --create --topic test --partitions 3 --replication-factor 1
and then I am getting the error below.
ost:2181 --create --topic test --partitions 3 --replication-factor 1
[2021-10-07 14:03:15,144] WARN [AdminClient clientId=adminclient-1] Connection to node -1 (localhost/127.0.0.1:2181) could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient)
[2021-10-07 14:03:15,251] WARN [AdminClient clientId=adminclient-1] Connection to node -1 (localhost/127.0.0.1:2181) could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient)
[2021-10-07 14:03:15,418] WARN [AdminClient clientId=adminclient-1] Connection to node -1 (localhost/127.0.0.1:2181) could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient)
Could you please advise on what exactly is the issue here and how this can be resolved?
In the Apache Kafka documentation, under the Notable changes in 2.2.0 states that
The bin/kafka-topics.sh command line tool is now able to connect directly to brokers with --bootstrap-server instead of zookeeper. The old --zookeeper option is still available for now. Please read KIP-377 for more information.
(As of Apache Kafka 3.0.0 --zookeper flag was removed).
It's unclear what version of Kafka you are using, but given that it accepted the flag --bootstrap-server then you're at least using 2.2.0 (probably < 3.0.0 given the dir name but not important for this question).
If you're using --bootstrap-server then you want to be connecting to the port associated to the Kafka server not Apache zookeper
Recall that
The bin/kafka-topics.sh command line tool is now able to connect directly to brokers
Therefore, typically port 9092 is used and so your command should be
./bin/kafka-topics.sh --bootstrap-server localhost:9092 --create --topic test --partitions 3 --replication-factor 1
However, --zookeeper is typically used with port 2181 since that's the port that Apache Zookeper tends to run on.
I understand my default retention period, but I want to see if any topics have special overrides for the retention period.
I've tried using the kafka-topics, but it doesn't connect with zookeeper.
kafka-topics --zookeeper localhost:2181 --describe --topics-with-overrides
I expect this output to be basically nothing but I get the error message:
Exception in thread "main" kafka.zookeeper.ZooKeeperClientTimeoutException: Timed out waiting for connection while in state: CONNECTING
at kafka.zookeeper.ZooKeeperClient$$anonfun$kafka$zookeeper$ZooKeeperClient$$waitUntilConnected$1.apply$mcV$sp(ZooKeeperClient.scala:268)
at kafka.zookeeper.ZooKeeperClient$$anonfun$kafka$zookeeper$ZooKeeperClient$$waitUntilConnected$1.apply(ZooKeeperClient.scala:264)
at kafka.zookeeper.ZooKeeperClient$$anonfun$kafka$zookeeper$ZooKeeperClient$$waitUntilConnected$1.apply(ZooKeeperClient.scala:264)
at kafka.utils.CoreUtils$.inLock(CoreUtils.scala:251)
at kafka.zookeeper.ZooKeeperClient.kafka$zookeeper$ZooKeeperClient$$waitUntilConnected(ZooKeeperClient.scala:264)
at kafka.zookeeper.ZooKeeperClient.<init>(ZooKeeperClient.scala:97)
at kafka.zk.KafkaZkClient$.apply(KafkaZkClient.scala:1693)
at kafka.admin.TopicCommand$.main(TopicCommand.scala:57)
at kafka.admin.TopicCommand.main(TopicCommand.scala)
Turns out my zookeeper param wasn't correct, zookeeper is not hosted on the kafka server. This worked:
kafka-topics --zookeeper myactualzookeeperserver.com:2181 --describe --topics-with-overrides
I have installed Kafka 1.1.0 with Zookeeper 3.4.12 under Windows SubSystem for Linux on my Windows 10 laptop. I'm able to procude and consume messages while I stay in ubuntu but when I want to produce a message from windows (with a java program or with the tool kafka-console-producer.bat) I have the following error :
[2018-05-11 15:31:01,449] ERROR Error when sending message to topic test with key: null, value: 15 bytes with error: (org.apache.kafka.clients.producer.internals.ErrorLoggingCallback)
org.apache.kafka.common.errors.TimeoutException: Expiring 1 record(s) for test-0: 1534 ms has passed since batch creation plus linger time
Any idea ?
I had a similar issue and I solved it testing kafka from console like this:
Creating a topic:
bin/kafka-topics.sh --zookeeper zooker-domain:2181 --create --topic test --replication-factor 1 --partitions 1
Inserting a message:
bin/kafka-console-producer.sh --broker-list kafka-domain:9092 --topic test
Consuming messages:
(old api)
bin/kafka-console-consumer.sh --zookeeper zookeeper-domain:2181 --topic test --from-beginning
(or with new api)
bin/kafka-console-consumer.sh --new-consumer --topic test --from-beginning --bootstrap-server kafka-domain:9092 --partition 0
(Instead of domains you can use directly IPs).
I tested it with kafka 0.11.0.1 and it works.
If you still have problems try with: https://stackoverflow.com/a/51071306/2493852
I've installed and setup Kafka using the following simple producer/consumer tutorial:
https://kafka.apache.org/quickstart
I have two machines, and both are using Ubuntu.
Resume of the problem:
If I use the producer and consumer on the same machine, everything works fine.
If I use the producer on machine 2, and the rest on machine 1, such as kafka, zookeeper server and the consumer, I never receive any messages on machine 1.
Machine 1 has IP: 192.168.1.100
Machine 2 has IP: 192.168.1.101
Working Example using just the Machine 1 only, with 4 console
applications
Console 1 - started zookeeper:
bin/zookeeper-server-start.sh config/zookeeper.properties
Console 2 - started kafka server
bin/kafka-server-start.sh config/server.properties
Console 3
Created a topic called test:
bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic test
Tested the topic:
bin/kafka-topics.sh --list --zookeeper localhost:2181
Started the consumer
bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic test
Console 4 - Send some messages:
bin/kafka-console-producer.sh --broker-list localhost:9092 --topic test
It works!
NOT Working Example using Machine 1 and 2
Machine 1 - Console 1 - started zookeeper:
bin/zookeeper-server-start.sh config/zookeeper.properties
Machine 1 - Console 2 - started kafka server
bin/kafka-server-start.sh config/server.properties
Machine 1 - Console 3
Created a topic called test:
bin/kafka-topics.sh --create --zookeeper 192.168.1.100:2181 --replication-factor 1 --partitions 1 --topic test
Tested the topic:
bin/kafka-topics.sh --list --zookeeper 192.168.1.100:2181
Started the consumer
bin/kafka-console-consumer.sh --bootstrap-server 192.168.1.100:9092 --topic test
Machine 2 - Console 1 - Send some messages to kafka on the IP 192.168.1.100
bin/kafka-console-producer.sh --broker-list 192.168.1.100:9092 --topic test
Doesn't work ...
What am I missing here?
EDIT:
I've used now the kafkacat to test the connection...here is the output.
root#ubuntu:~/kafka/kafka_2.11-0.11.0.1# kafkacat -b 192.168.1.100 -t test -L
Metadata for test (from broker -1: 192.168.1.100:9092/bootstrap):
1 brokers:
broker 0 at ubuntu:9092
1 topics:
topic "test" with 1 partitions:
partition 0, leader 0, replicas: 0, isrs: 0
%3|1507802180.807|FAIL|rdkafka#producer-1| [thrd:ubuntu:9092/0]: ubuntu:9092/0: Connect to ipv4#127.0.1.1:9092 failed: Connection refused
%3|1507802180.807|ERROR|rdkafka#producer-1| [thrd:ubuntu:9092/0]: ubuntu:9092/0: Connect to ipv4#127.0.1.1:9092 failed: Connection refused
Why is the 127.0.1.1:9092 set above?
Shouldn't be 192.168.1.100:9092?
From the FAQ, a common problem is the hostname:
When a broker starts up, it registers its ip/port in ZK. You need to make sure the registered ip is consistent with what's listed in metadata.broker.list in the producer config. By default, the registered ip is given by InetAddress.getLocalHost.getHostAddress(). Typically, this should return the real ip of the host. However, sometimes (e.g., in EC2), the returned ip is an internal one and can't be connected to from outside. The solution is to explicitly set the host ip to be registered in ZK by setting the hostname property in server.properties. In another rare case where the binding host/port is different from the host/port for client connection, you can set advertised.host.name and advertised.port for client connection.
So, open your server.properties and change the hostname:
host.name=<your hostname>
If this still doesn't work, try to modify the advertised.host.name:
advertised.host.name=<your ip>
If this really does not work, have a look at the advertised.listeners and ensure it is either 0.0.0.0:port or <your.ip>:port. Example:
advertised.listeners=PLAINTEXT://0.0.0.0:9092