I'm basically doing the kafka quickstart using kafka_2.11-2.0.0 which comes with zookeeper.version=3.4.13-2d71af4dbe22557fda74f9a9b4309b15a7487f03, built on 06/29/2018 00:39 GMT.
I also use Ubuntu:
lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 16.04.5 LTS
Release: 16.04
Codename: xenial
I do these with the order below:
start zookeeper
start kafka server0 with port 9092
start kafka server1 with port 9093
start kafka server2 with port 9094
create a topic: kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 3 --partitions 1 --topic my-replicated-topic
produce some messages: kafka-console-producer.sh --broker-list localhost:9092 --topic my-replicated-topic
consume from server0: kafka-console-consumer.sh --bootstrap-server localhost:9092 --from-beginning --topic my-replicated-topic
consume from server1: kafka-console-consumer.sh --bootstrap-server localhost:9093 --from-beginning --topic my-replicated-topic
check the leader for my-replicated-topic and find it to be server0 -> here's the tricky part; one should kill server1 and then maybe server2 but never server0 and then restore them just in order to get server0 to be the leader of my-replicated-topic
kill server0
check for the new leader (happens to be server2): kafka-topics.sh --describe --zookeeper localhost:2181 --topic my-replicated-topic
produce some messages to server2: kafka-console-producer.sh --broker-list localhost:9094 --topic my-replicated-topic
consume from server2 (or server1): kafka-console-consumer.sh --bootstrap-server localhost:9094 --from-beginning --topic my-replicated-topic -> this will hang till restarting server0
starting again server0
consumer from server2 outputs all messages including the one sent to server2 when was the leader
What is wrong and how one would solve the problem so not to matter which server becomes the leader?
I think this is due to replication factor of "__consumer_offsets" topic. This is set to one in server.properties file for testing purpose.
set offsets.topic.replication.factor=3 for high availability.
Copied from KAFKA-7526 comment
Related
Does anyone knows how to fix the error when creating a topic in Kafka?
C:\kafka\bin\windows>kafka-topics.bat --create --bootstrap-server localhost:2181 --replication-factor 1 --partition 1 --topic test
Exception in thread "main" joptsimple.UnrecognizedOptionException: partition is not a recognized option
at joptsimple.OptionException.unrecognizedOption(OptionException.java:108)
at joptsimple.OptionParser.handleLongOptionToken(OptionParser.java:510)
at joptsimple.OptionParserState$2.handleArgument(OptionParserState.java:56)
at joptsimple.OptionParser.parse(OptionParser.java:396)
at kafka.admin.TopicCommand$TopicCommandOptions.<init>(TopicCommand.scala:567)
at kafka.admin.TopicCommand$.main(TopicCommand.scala:47)
at kafka.admin.TopicCommand.main(TopicCommand.scala)
The parameter is partitions
The bootstrap server normally (default) runs in port 9092
C:\kafka\bin\windows>kafka-topics.bat --create --bootstrap-server localhost:9092 --replication-factor 1 --partitions 1 --topic test
In recent versions, you don't have to create topics on zookeeper. You can directly create topics on the bootstrap servers of Kafka. In the later version, they plan to remove the zookeeper altogether, so they are preparing for that in the current versions.
Use the below to create a new partition. I suggest adding the below parameters as well to control the topic behaviour appropriately.
kafka-topics.bat --create --bootstrap-server localhost:9092 --partitions 1 --replication-factor 1 --config retention.ms=604800000 segment.bytes=26214400 retention.bytes=1048576000 min.insync.replicas=1 --topic test
I'm trying to run a kafka cluster with this command :
kafka-topics.sh --bootstrap-server 127.0.0.1:2181 --topic first_topic --create --partitions 3 --replication-factor 1
and i get this as an error:
[2022-02-03 11:25:28,635] WARN [AdminClient clientId=adminclient-1] Connection to node -1 (/127.0.0.1:2181) could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient)
So i tried to look at kafka_2.12-3.1.0\config\server.properties i have
listeners=PLAINTEXT://localhost:9092
Any help will be very appreciated.
2181 is typically the port used by ZooKeeper. If you want to specify that, and you're not running Kafka in KRaft (zookeeper-less mode) then you need to do as #Umeshwaran said and use the --zookeeper argument.
However, you can use --bootstrap-server, but if you are doing so then specify the broker address and port, which from your listeners config is 9092:
kafka-topics.sh --bootstrap-server 127.0.0.1:9092 --topic first_topic --create --partitions 3 --replication-factor 1
This article should clarify things.
This is because your zookeeper is not working ,I was facing the same issue but when I run the zookeeper with(I am using wsl)
"zookeeper-server-start.sh ~/kafka_2.13-3.3.1/config/zookeeper.properties"
command it starts working
I am running Kafka also if you don't how to run, use this command
"kafka-server-start.sh ~/kafka_2.13-3.3.1/config/server.properties"
hint- modify kafka_2.13-3.3.1 this with the correct version of kafka you are using
I am new to kafka and zookepper, and I am trying to create a topic, but I am getting this error -
Exception in thread "main" joptsimple.UnrecognizedOptionException: zookeeper is not a recognized option
at joptsimple.OptionException.unrecognizedOption(OptionException.java:108)
at joptsimple.OptionParser.handleLongOptionToken(OptionParser.java:510)
at joptsimple.OptionParserState$2.handleArgument(OptionParserState.java:56)
at joptsimple.OptionParser.parse(OptionParser.java:396)
at kafka.admin.TopicCommand$TopicCommandOptions.<init>(TopicCommand.scala:517)
at kafka.admin.TopicCommand$.main(TopicCommand.scala:47)
at kafka.admin.TopicCommand.main(TopicCommand.scala)
I am using this command to create the topic -
.\bin\windows\kafka-topics.bat --create --zookeeper localhost:2181 --replication-factor 1 --partions 1 --topic TestTopic
Newer versions(2.2+) of Kafka no longer requires ZooKeeper connection string
--zookeeper localhost:2181
It throws the following exception while creating a topic
Exception in thread "main" joptsimple.UnrecognizedOptionException:
zookeeper is not a recognized option
Instead, add Kafka Broker --bootstrap-server localhost:9092 connection string.
./kafka-topics.sh --create --topic test-topic --bootstrap-server localhost:9092 --replication-factor 1 --partitions 4
In the latest version kafka_2.12-3.1.0 (2022) after unzipping and setting the properties and logs. keep the Kafka folder on the C drive and always run the command prompt with 'run as administrator'.
The .bat file is for windows
Terminal 1
C:\kafka\bin\windows>zookeeper-server-start.bat
..\..\config\zookeeper.properties
Terminal 2
C:\kafka\bin\windows>kafka-server-start.bat
..\..\config\server.properties
Terminal 3
C:\kafka\bin\windows>kafka-topics.bat --create --topic tutorialspedia
--bootstrap-server localhost:9092
Created topic tutorialspedia.
To checklist of topic created
C:\kafka\bin\windows>kafka-topics.bat --list --bootstrap-server
localhost:9092
tutorialspedia
Read the official Kafka documentation for the version you downloaded, and not some other blog/article that you might have copied the command from
zookeeper is almost never used for CLI commands in current versions
If you run bin\kafka-topics on its own with --help or no options, then it'll print the help messaging that shows all available arguments.
It's not --partions, but --partitions.
Same like you, see this link: Exception: partition is not a recognized option, when creating a kafka topic inside a docker.
Kakfa create Topic :
./kafka-topics.sh --create --topic test-topic --bootstrap-server localhost:9092 --replication-factor 1 --partitions 4
Kafka Created Topics for Windows:
kafka-topics.bat --create --topic learningkafka --bootstrap-server localhost:9092
After topic created we can check the kafka list:
kafka-topics.bat --list --bootstrap-server localhost:9092
Kafka Version : kafka_2.11-0.11.0.1
when I try to send messages from producer to consumer in a Multi-node Kafka Ecosystem, the message is getting sent, and broadcasted from Kafka server! These messages can be read by using Kafka Consumer via Zookeeper [which is going to be deprecated on following version!], and also via kafka-simple-consumer-shell.sh
./bin/kafka-simple-consumer-shell.sh --broker-list <broker-ip>:9092 --topic myTopic
But cannot be read from Consumer via consumer-bootstrap-server!
Call to Consumer via Zookeeper-server :
./bin/kafka-console-consumer.sh --topic myTopic --zookeeper <broker-ip>:2181 --from-beginning
Call to Consumer via Bootstrap-server :
./kafka-console-consumer.sh --topic myTopic --bootstrap-server <broker-ip>:9092 --consumer.config ../config/consumer.properties
My config/producer.properties contains:
bootstrap.servers= {broker-ip}:9092
compression.type=none
My config/consumer.properties contains(nothing):
# zookeeper.connect={private-ip-1}:2181,{private-ip-2}:2181
# timeout in ms for connecting to zookeeper
# zookeeper.connection.timeout.ms=6000
#consumer group id
group.id=test-consumer-group
#consumer timeout
# consumer.timeout.ms=7000
# consumer.timeout.ms=-1
My config/server.properties contains:
listeners=PLAINTEXT://{private-ip-1}:9092
advertised.host.name={private-ip-1}
advertised.listeners=PLAINTEXT://{private-ip-1}:9092
advertised.port=9092
host.name={private-ip-1}
zookeeper.connect={private-ip-1}:2181
I've installed and setup Kafka using the following simple producer/consumer tutorial:
https://kafka.apache.org/quickstart
I have two machines, and both are using Ubuntu.
Resume of the problem:
If I use the producer and consumer on the same machine, everything works fine.
If I use the producer on machine 2, and the rest on machine 1, such as kafka, zookeeper server and the consumer, I never receive any messages on machine 1.
Machine 1 has IP: 192.168.1.100
Machine 2 has IP: 192.168.1.101
Working Example using just the Machine 1 only, with 4 console
applications
Console 1 - started zookeeper:
bin/zookeeper-server-start.sh config/zookeeper.properties
Console 2 - started kafka server
bin/kafka-server-start.sh config/server.properties
Console 3
Created a topic called test:
bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic test
Tested the topic:
bin/kafka-topics.sh --list --zookeeper localhost:2181
Started the consumer
bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic test
Console 4 - Send some messages:
bin/kafka-console-producer.sh --broker-list localhost:9092 --topic test
It works!
NOT Working Example using Machine 1 and 2
Machine 1 - Console 1 - started zookeeper:
bin/zookeeper-server-start.sh config/zookeeper.properties
Machine 1 - Console 2 - started kafka server
bin/kafka-server-start.sh config/server.properties
Machine 1 - Console 3
Created a topic called test:
bin/kafka-topics.sh --create --zookeeper 192.168.1.100:2181 --replication-factor 1 --partitions 1 --topic test
Tested the topic:
bin/kafka-topics.sh --list --zookeeper 192.168.1.100:2181
Started the consumer
bin/kafka-console-consumer.sh --bootstrap-server 192.168.1.100:9092 --topic test
Machine 2 - Console 1 - Send some messages to kafka on the IP 192.168.1.100
bin/kafka-console-producer.sh --broker-list 192.168.1.100:9092 --topic test
Doesn't work ...
What am I missing here?
EDIT:
I've used now the kafkacat to test the connection...here is the output.
root#ubuntu:~/kafka/kafka_2.11-0.11.0.1# kafkacat -b 192.168.1.100 -t test -L
Metadata for test (from broker -1: 192.168.1.100:9092/bootstrap):
1 brokers:
broker 0 at ubuntu:9092
1 topics:
topic "test" with 1 partitions:
partition 0, leader 0, replicas: 0, isrs: 0
%3|1507802180.807|FAIL|rdkafka#producer-1| [thrd:ubuntu:9092/0]: ubuntu:9092/0: Connect to ipv4#127.0.1.1:9092 failed: Connection refused
%3|1507802180.807|ERROR|rdkafka#producer-1| [thrd:ubuntu:9092/0]: ubuntu:9092/0: Connect to ipv4#127.0.1.1:9092 failed: Connection refused
Why is the 127.0.1.1:9092 set above?
Shouldn't be 192.168.1.100:9092?
From the FAQ, a common problem is the hostname:
When a broker starts up, it registers its ip/port in ZK. You need to make sure the registered ip is consistent with what's listed in metadata.broker.list in the producer config. By default, the registered ip is given by InetAddress.getLocalHost.getHostAddress(). Typically, this should return the real ip of the host. However, sometimes (e.g., in EC2), the returned ip is an internal one and can't be connected to from outside. The solution is to explicitly set the host ip to be registered in ZK by setting the hostname property in server.properties. In another rare case where the binding host/port is different from the host/port for client connection, you can set advertised.host.name and advertised.port for client connection.
So, open your server.properties and change the hostname:
host.name=<your hostname>
If this still doesn't work, try to modify the advertised.host.name:
advertised.host.name=<your ip>
If this really does not work, have a look at the advertised.listeners and ensure it is either 0.0.0.0:port or <your.ip>:port. Example:
advertised.listeners=PLAINTEXT://0.0.0.0:9092