I list the brokers using: /opt/confluent-kafka/v5.5.0/bin/zookeeper-shell localhost:2181 ls /brokers/ids
Which returns:
Connecting to localhost:2181
WATCHER::
WatchedEvent state:SyncConnected type:None path:null
[0]
Which shows no brokers running.
I then create a topic using:
[test#bhop logs]$ /opt/confluent-kafka/v5.5.0/bin/kafka-topics --create \
> --zookeeper localhost:2181 \
> --replication-factor 1 \
> --partitions 1 \
> --topic test
Created topic test.
And can see the topic is created:
/opt/confluent-kafka/v5.5.0/bin/kafka-topics --list --zookeeper localhost:2181
returns:
__confluent.support.metrics
test-topic
As there are no brokers running how can a topic be created? Is a running broker not required to create a topic ?
[0] shows there's one broker with id 0. The ids znode would not exist if no brokers were running
You should get /brokers/ids/0 to see the broker data
If it looks correct, things are working as expected
Note: --zookeeper flag is deprecated in latest Kafka CLIs
Related
Does anyone knows how to fix the error when creating a topic in Kafka?
C:\kafka\bin\windows>kafka-topics.bat --create --bootstrap-server localhost:2181 --replication-factor 1 --partition 1 --topic test
Exception in thread "main" joptsimple.UnrecognizedOptionException: partition is not a recognized option
at joptsimple.OptionException.unrecognizedOption(OptionException.java:108)
at joptsimple.OptionParser.handleLongOptionToken(OptionParser.java:510)
at joptsimple.OptionParserState$2.handleArgument(OptionParserState.java:56)
at joptsimple.OptionParser.parse(OptionParser.java:396)
at kafka.admin.TopicCommand$TopicCommandOptions.<init>(TopicCommand.scala:567)
at kafka.admin.TopicCommand$.main(TopicCommand.scala:47)
at kafka.admin.TopicCommand.main(TopicCommand.scala)
The parameter is partitions
The bootstrap server normally (default) runs in port 9092
C:\kafka\bin\windows>kafka-topics.bat --create --bootstrap-server localhost:9092 --replication-factor 1 --partitions 1 --topic test
In recent versions, you don't have to create topics on zookeeper. You can directly create topics on the bootstrap servers of Kafka. In the later version, they plan to remove the zookeeper altogether, so they are preparing for that in the current versions.
Use the below to create a new partition. I suggest adding the below parameters as well to control the topic behaviour appropriately.
kafka-topics.bat --create --bootstrap-server localhost:9092 --partitions 1 --replication-factor 1 --config retention.ms=604800000 segment.bytes=26214400 retention.bytes=1048576000 min.insync.replicas=1 --topic test
I am new to kafka and zookepper, and I am trying to create a topic, but I am getting this error -
Exception in thread "main" joptsimple.UnrecognizedOptionException: zookeeper is not a recognized option
at joptsimple.OptionException.unrecognizedOption(OptionException.java:108)
at joptsimple.OptionParser.handleLongOptionToken(OptionParser.java:510)
at joptsimple.OptionParserState$2.handleArgument(OptionParserState.java:56)
at joptsimple.OptionParser.parse(OptionParser.java:396)
at kafka.admin.TopicCommand$TopicCommandOptions.<init>(TopicCommand.scala:517)
at kafka.admin.TopicCommand$.main(TopicCommand.scala:47)
at kafka.admin.TopicCommand.main(TopicCommand.scala)
I am using this command to create the topic -
.\bin\windows\kafka-topics.bat --create --zookeeper localhost:2181 --replication-factor 1 --partions 1 --topic TestTopic
Newer versions(2.2+) of Kafka no longer requires ZooKeeper connection string
--zookeeper localhost:2181
It throws the following exception while creating a topic
Exception in thread "main" joptsimple.UnrecognizedOptionException:
zookeeper is not a recognized option
Instead, add Kafka Broker --bootstrap-server localhost:9092 connection string.
./kafka-topics.sh --create --topic test-topic --bootstrap-server localhost:9092 --replication-factor 1 --partitions 4
In the latest version kafka_2.12-3.1.0 (2022) after unzipping and setting the properties and logs. keep the Kafka folder on the C drive and always run the command prompt with 'run as administrator'.
The .bat file is for windows
Terminal 1
C:\kafka\bin\windows>zookeeper-server-start.bat
..\..\config\zookeeper.properties
Terminal 2
C:\kafka\bin\windows>kafka-server-start.bat
..\..\config\server.properties
Terminal 3
C:\kafka\bin\windows>kafka-topics.bat --create --topic tutorialspedia
--bootstrap-server localhost:9092
Created topic tutorialspedia.
To checklist of topic created
C:\kafka\bin\windows>kafka-topics.bat --list --bootstrap-server
localhost:9092
tutorialspedia
Read the official Kafka documentation for the version you downloaded, and not some other blog/article that you might have copied the command from
zookeeper is almost never used for CLI commands in current versions
If you run bin\kafka-topics on its own with --help or no options, then it'll print the help messaging that shows all available arguments.
It's not --partions, but --partitions.
Same like you, see this link: Exception: partition is not a recognized option, when creating a kafka topic inside a docker.
Kakfa create Topic :
./kafka-topics.sh --create --topic test-topic --bootstrap-server localhost:9092 --replication-factor 1 --partitions 4
Kafka Created Topics for Windows:
kafka-topics.bat --create --topic learningkafka --bootstrap-server localhost:9092
After topic created we can check the kafka list:
kafka-topics.bat --list --bootstrap-server localhost:9092
Would like to share different ways to purge or delete a kafka topic in 2.1.0 version. I've found similar question here Purge Kafka Topic however, the accepted answer has been deprecated and it works on Kafka version 0.8 and below hence, creating this question with answer.
This is not a duplicate question.
Kafka by default keeps the messages for 168 hrs which is 7 days. If you wanted to force kafka to purge the topic, you can do this by multiple ways. Let’s see each in detail.
1. Using kafka-configs.sh command
Temporarily change the retention policy to 1 sec.
kafka-configs.sh --zookeeper localhost:2181 --alter --entity-type topics --add-config retention.ms=1000 --entity-name text_topic
You can check the current value of retention policy by running below command.
kafka-configs.sh --zookeeper localhost:2181 --entity-type topics --describe --entity-name text_topic
Configs for topic 'text_topic' are retention.ms=1000
Wait for 1 sec and remove the retention policy config which will set it back to default.
kafka-configs.sh --zookeeper localhost:2181 --alter --entity-type topics --delete-config retention.ms --entity-name text_topic
2. Delete topic and re-create
Before we delete an existing topic, first get the partition and replica of the current topic as you would need these to re-create the topic. You can get this information by running describe of the topic
kafka-topics.sh --zookeeper localhost:2181 --describe --topic text_topic
Topic:text_topic PartitionCount:3 ReplicationFactor:3 Configs:
Topic: text_topic Partition: 0 Leader: 0 Replicas: 0 Isr: 0
Delete the topic.
kafka-topics.sh --zookeeper localhost:2181 --delete --topic text_topic
Re-create the topic with replication and partition details.
kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 3 --partitions 3 --topic text_topic
3. Manually delete the data from kafka logs.
Stop zookeeper and kafka from all nodes.
Clean kafka logs from all nodes. kafka stores its log files at
/tmp/kafka-logs/MyTopic-0 where /tmp/kafka-logs is specified by the
log.dirattribute
Restart zookeeper and kafka.
Hope this helps !!
I'm basically doing the kafka quickstart using kafka_2.11-2.0.0 which comes with zookeeper.version=3.4.13-2d71af4dbe22557fda74f9a9b4309b15a7487f03, built on 06/29/2018 00:39 GMT.
I also use Ubuntu:
lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 16.04.5 LTS
Release: 16.04
Codename: xenial
I do these with the order below:
start zookeeper
start kafka server0 with port 9092
start kafka server1 with port 9093
start kafka server2 with port 9094
create a topic: kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 3 --partitions 1 --topic my-replicated-topic
produce some messages: kafka-console-producer.sh --broker-list localhost:9092 --topic my-replicated-topic
consume from server0: kafka-console-consumer.sh --bootstrap-server localhost:9092 --from-beginning --topic my-replicated-topic
consume from server1: kafka-console-consumer.sh --bootstrap-server localhost:9093 --from-beginning --topic my-replicated-topic
check the leader for my-replicated-topic and find it to be server0 -> here's the tricky part; one should kill server1 and then maybe server2 but never server0 and then restore them just in order to get server0 to be the leader of my-replicated-topic
kill server0
check for the new leader (happens to be server2): kafka-topics.sh --describe --zookeeper localhost:2181 --topic my-replicated-topic
produce some messages to server2: kafka-console-producer.sh --broker-list localhost:9094 --topic my-replicated-topic
consume from server2 (or server1): kafka-console-consumer.sh --bootstrap-server localhost:9094 --from-beginning --topic my-replicated-topic -> this will hang till restarting server0
starting again server0
consumer from server2 outputs all messages including the one sent to server2 when was the leader
What is wrong and how one would solve the problem so not to matter which server becomes the leader?
I think this is due to replication factor of "__consumer_offsets" topic. This is set to one in server.properties file for testing purpose.
set offsets.topic.replication.factor=3 for high availability.
Copied from KAFKA-7526 comment
I'm trying to run official "Kafka010Example.scala", but unortunatelly it doesn't read from input topic and write to output as expected. What am I missing or doing wrong? Any help or hints much appreciated. Here's exactly what I did:
Started kafka in docker container (spotify/kafka:latest)
$ docker run -d -p 2181:2181 -p 9092:9092 spotify/kafka:latest
Started bash session inside of the container:
$ docker exec -it 26d1cfced4cb /bin/bash
Created input and output topics:
$ /opt/kafka_2.11-0.10.1.0/bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic test-input
$ /opt/kafka_2.11-0.10.1.0/bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic test-output
Launched "Kafka010Example.scala" running flink 1.3.2 in local mode (MiniCluster), with flink-connector-kafka-0.10_2.11
with the following arguments:
--input-topic test-input --output-topic test-output --bootstrap.servers localhost:9092 --zookeeper.connect localhost:2181 --group.id myconsumer
Logs:
https://file.io/jWsqI8
Sent some messages to the topic:
$ /opt/kafka_2.11-0.10.1.0/bin/kafka-console-producer.sh --broker-list localhost:9092 --topic test-input
blah
blahh
blahhh
Checked offset on the output topic - NOTHING
$ /opt/kafka_2.11-0.10.1.0/bin/kafka-consumer-offset-checker.sh --zookeeper localhost:2181 --group myconsumer --topic test-output
Group Topic Pid Offset logSize
Lag Owner
myconsumer test-output 0 0 0
0 none
Check consumer group offset - NOTHING
$ /opt/kafka_2.11-0.10.1.0/bin/kafka-consumer-groups.sh --zookeeper localhost:2181 --describe --group myconsumer
No topic available for consumer group provided
GROUP TOPIC PARTITION
CURRENT-OFFSET LOG-END-OFFSET LAG OWNER
I suggest using the console-consumer to watch the test-output topic. I wouldn't expect the offset for the test-output topic to have advanced, since no one has read from it.
Step 7 should be:
$ /opt/kafka_2.11-0.10.1.0/bin/kafka-console-consumer.sh --zookeeper localhost:2181 --group myconsumer --topic test-output