I'm trying to run a kafka cluster with this command :
kafka-topics.sh --bootstrap-server 127.0.0.1:2181 --topic first_topic --create --partitions 3 --replication-factor 1
and i get this as an error:
[2022-02-03 11:25:28,635] WARN [AdminClient clientId=adminclient-1] Connection to node -1 (/127.0.0.1:2181) could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient)
So i tried to look at kafka_2.12-3.1.0\config\server.properties i have
listeners=PLAINTEXT://localhost:9092
Any help will be very appreciated.
2181 is typically the port used by ZooKeeper. If you want to specify that, and you're not running Kafka in KRaft (zookeeper-less mode) then you need to do as #Umeshwaran said and use the --zookeeper argument.
However, you can use --bootstrap-server, but if you are doing so then specify the broker address and port, which from your listeners config is 9092:
kafka-topics.sh --bootstrap-server 127.0.0.1:9092 --topic first_topic --create --partitions 3 --replication-factor 1
This article should clarify things.
This is because your zookeeper is not working ,I was facing the same issue but when I run the zookeeper with(I am using wsl)
"zookeeper-server-start.sh ~/kafka_2.13-3.3.1/config/zookeeper.properties"
command it starts working
I am running Kafka also if you don't how to run, use this command
"kafka-server-start.sh ~/kafka_2.13-3.3.1/config/server.properties"
hint- modify kafka_2.13-3.3.1 this with the correct version of kafka you are using
Related
Does anyone knows how to fix the error when creating a topic in Kafka?
C:\kafka\bin\windows>kafka-topics.bat --create --bootstrap-server localhost:2181 --replication-factor 1 --partition 1 --topic test
Exception in thread "main" joptsimple.UnrecognizedOptionException: partition is not a recognized option
at joptsimple.OptionException.unrecognizedOption(OptionException.java:108)
at joptsimple.OptionParser.handleLongOptionToken(OptionParser.java:510)
at joptsimple.OptionParserState$2.handleArgument(OptionParserState.java:56)
at joptsimple.OptionParser.parse(OptionParser.java:396)
at kafka.admin.TopicCommand$TopicCommandOptions.<init>(TopicCommand.scala:567)
at kafka.admin.TopicCommand$.main(TopicCommand.scala:47)
at kafka.admin.TopicCommand.main(TopicCommand.scala)
The parameter is partitions
The bootstrap server normally (default) runs in port 9092
C:\kafka\bin\windows>kafka-topics.bat --create --bootstrap-server localhost:9092 --replication-factor 1 --partitions 1 --topic test
In recent versions, you don't have to create topics on zookeeper. You can directly create topics on the bootstrap servers of Kafka. In the later version, they plan to remove the zookeeper altogether, so they are preparing for that in the current versions.
Use the below to create a new partition. I suggest adding the below parameters as well to control the topic behaviour appropriately.
kafka-topics.bat --create --bootstrap-server localhost:9092 --partitions 1 --replication-factor 1 --config retention.ms=604800000 segment.bytes=26214400 retention.bytes=1048576000 min.insync.replicas=1 --topic test
I am trying to create my first topic by using the command below:
./bin/kafka-topics.sh --bootstrap-server localhost:2181 --create --topic test --partitions 3 --replication-factor 1
and then I am getting the error below.
ost:2181 --create --topic test --partitions 3 --replication-factor 1
[2021-10-07 14:03:15,144] WARN [AdminClient clientId=adminclient-1] Connection to node -1 (localhost/127.0.0.1:2181) could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient)
[2021-10-07 14:03:15,251] WARN [AdminClient clientId=adminclient-1] Connection to node -1 (localhost/127.0.0.1:2181) could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient)
[2021-10-07 14:03:15,418] WARN [AdminClient clientId=adminclient-1] Connection to node -1 (localhost/127.0.0.1:2181) could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient)
Could you please advise on what exactly is the issue here and how this can be resolved?
In the Apache Kafka documentation, under the Notable changes in 2.2.0 states that
The bin/kafka-topics.sh command line tool is now able to connect directly to brokers with --bootstrap-server instead of zookeeper. The old --zookeeper option is still available for now. Please read KIP-377 for more information.
(As of Apache Kafka 3.0.0 --zookeper flag was removed).
It's unclear what version of Kafka you are using, but given that it accepted the flag --bootstrap-server then you're at least using 2.2.0 (probably < 3.0.0 given the dir name but not important for this question).
If you're using --bootstrap-server then you want to be connecting to the port associated to the Kafka server not Apache zookeper
Recall that
The bin/kafka-topics.sh command line tool is now able to connect directly to brokers
Therefore, typically port 9092 is used and so your command should be
./bin/kafka-topics.sh --bootstrap-server localhost:9092 --create --topic test --partitions 3 --replication-factor 1
However, --zookeeper is typically used with port 2181 since that's the port that Apache Zookeper tends to run on.
I'm basically doing the kafka quickstart using kafka_2.11-2.0.0 which comes with zookeeper.version=3.4.13-2d71af4dbe22557fda74f9a9b4309b15a7487f03, built on 06/29/2018 00:39 GMT.
I also use Ubuntu:
lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 16.04.5 LTS
Release: 16.04
Codename: xenial
I do these with the order below:
start zookeeper
start kafka server0 with port 9092
start kafka server1 with port 9093
start kafka server2 with port 9094
create a topic: kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 3 --partitions 1 --topic my-replicated-topic
produce some messages: kafka-console-producer.sh --broker-list localhost:9092 --topic my-replicated-topic
consume from server0: kafka-console-consumer.sh --bootstrap-server localhost:9092 --from-beginning --topic my-replicated-topic
consume from server1: kafka-console-consumer.sh --bootstrap-server localhost:9093 --from-beginning --topic my-replicated-topic
check the leader for my-replicated-topic and find it to be server0 -> here's the tricky part; one should kill server1 and then maybe server2 but never server0 and then restore them just in order to get server0 to be the leader of my-replicated-topic
kill server0
check for the new leader (happens to be server2): kafka-topics.sh --describe --zookeeper localhost:2181 --topic my-replicated-topic
produce some messages to server2: kafka-console-producer.sh --broker-list localhost:9094 --topic my-replicated-topic
consume from server2 (or server1): kafka-console-consumer.sh --bootstrap-server localhost:9094 --from-beginning --topic my-replicated-topic -> this will hang till restarting server0
starting again server0
consumer from server2 outputs all messages including the one sent to server2 when was the leader
What is wrong and how one would solve the problem so not to matter which server becomes the leader?
I think this is due to replication factor of "__consumer_offsets" topic. This is set to one in server.properties file for testing purpose.
set offsets.topic.replication.factor=3 for high availability.
Copied from KAFKA-7526 comment
I've installed and setup Kafka using the following simple producer/consumer tutorial:
https://kafka.apache.org/quickstart
I have two machines, and both are using Ubuntu.
Resume of the problem:
If I use the producer and consumer on the same machine, everything works fine.
If I use the producer on machine 2, and the rest on machine 1, such as kafka, zookeeper server and the consumer, I never receive any messages on machine 1.
Machine 1 has IP: 192.168.1.100
Machine 2 has IP: 192.168.1.101
Working Example using just the Machine 1 only, with 4 console
applications
Console 1 - started zookeeper:
bin/zookeeper-server-start.sh config/zookeeper.properties
Console 2 - started kafka server
bin/kafka-server-start.sh config/server.properties
Console 3
Created a topic called test:
bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic test
Tested the topic:
bin/kafka-topics.sh --list --zookeeper localhost:2181
Started the consumer
bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic test
Console 4 - Send some messages:
bin/kafka-console-producer.sh --broker-list localhost:9092 --topic test
It works!
NOT Working Example using Machine 1 and 2
Machine 1 - Console 1 - started zookeeper:
bin/zookeeper-server-start.sh config/zookeeper.properties
Machine 1 - Console 2 - started kafka server
bin/kafka-server-start.sh config/server.properties
Machine 1 - Console 3
Created a topic called test:
bin/kafka-topics.sh --create --zookeeper 192.168.1.100:2181 --replication-factor 1 --partitions 1 --topic test
Tested the topic:
bin/kafka-topics.sh --list --zookeeper 192.168.1.100:2181
Started the consumer
bin/kafka-console-consumer.sh --bootstrap-server 192.168.1.100:9092 --topic test
Machine 2 - Console 1 - Send some messages to kafka on the IP 192.168.1.100
bin/kafka-console-producer.sh --broker-list 192.168.1.100:9092 --topic test
Doesn't work ...
What am I missing here?
EDIT:
I've used now the kafkacat to test the connection...here is the output.
root#ubuntu:~/kafka/kafka_2.11-0.11.0.1# kafkacat -b 192.168.1.100 -t test -L
Metadata for test (from broker -1: 192.168.1.100:9092/bootstrap):
1 brokers:
broker 0 at ubuntu:9092
1 topics:
topic "test" with 1 partitions:
partition 0, leader 0, replicas: 0, isrs: 0
%3|1507802180.807|FAIL|rdkafka#producer-1| [thrd:ubuntu:9092/0]: ubuntu:9092/0: Connect to ipv4#127.0.1.1:9092 failed: Connection refused
%3|1507802180.807|ERROR|rdkafka#producer-1| [thrd:ubuntu:9092/0]: ubuntu:9092/0: Connect to ipv4#127.0.1.1:9092 failed: Connection refused
Why is the 127.0.1.1:9092 set above?
Shouldn't be 192.168.1.100:9092?
From the FAQ, a common problem is the hostname:
When a broker starts up, it registers its ip/port in ZK. You need to make sure the registered ip is consistent with what's listed in metadata.broker.list in the producer config. By default, the registered ip is given by InetAddress.getLocalHost.getHostAddress(). Typically, this should return the real ip of the host. However, sometimes (e.g., in EC2), the returned ip is an internal one and can't be connected to from outside. The solution is to explicitly set the host ip to be registered in ZK by setting the hostname property in server.properties. In another rare case where the binding host/port is different from the host/port for client connection, you can set advertised.host.name and advertised.port for client connection.
So, open your server.properties and change the hostname:
host.name=<your hostname>
If this still doesn't work, try to modify the advertised.host.name:
advertised.host.name=<your ip>
If this really does not work, have a look at the advertised.listeners and ensure it is either 0.0.0.0:port or <your.ip>:port. Example:
advertised.listeners=PLAINTEXT://0.0.0.0:9092
I first create a topic test using the command
/opt/cloudera/parcels/KAFKA/bin/kafka-topics --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic test
Then when I try to insert a new message to a topic in Kafka using the command
/opt/cloudera/parcels/KAFKA/bin/kafka-console-producer --broker-list localhost:2181 --topic test
Then I enter a message like "name":"Ashish"
I get the error
"received -1 from channel reading from source socket has likely been
closed" !
What am I doing wrong here?
Any pointers to the solution or help will be appreciated.
The --broker-list arguments expect to have the Kafka broker Id not the zookeeper one. In your case you have mentioned the zookeeper host:port which is running at port 2181. By default kafka broker listen on port 9092.
Change --broker-host localhost:2181 to --broker-host localhost:9092 to see if it helps (assuming the kafka broker is running in the same machine, otherwise replace lcoalhost with the machine ip)