I am using dockerised wurstmeister/kafka-docker. I created a topic using
bin/kafka-topics.sh --create --bootstrap-server localhost:9092 --replication-factor 1 --partitions 27 --topic raw-sensor-data --config retention.ms=86400000
After a few days I tried changing retention period by
bin/kafka-configs.sh --bootstrap-server localhost:9092 --alter --entity-type topics --entity-name raw-sensor-data --add-config retention.ms=3600000
I also tried
bin/kafka-topics.sh --zookeeper locahost:2181 --alter --topic raw-sensor-data --config retention.ms=3600000
and
./bin/kafka-topics.sh --zookeeper localhost:2181 --alter --topic raw-sensor-data --config cleanup.policy=delete
This also gets reflected in topic describe details
bin/kafka-topics.sh --zookeeper localhost:2181 --describe --topics-with-overrides
Topic: raw-sensor-data PartitionCount: 27 ReplicationFactor: 1 Configs: cleanup.policy=delete,retention.ms=3600000
But I can still see old data and data is not getting deleted in 1 hour time.
In server.properties I have
log.retention.check.interval.ms=300000
Only closed log segments will be deleted. The default segment size is 1GB.
So, if you have less data in the topic, it will remain, regardless of the time that has passed.
Related
kafka-topics.bat --list --bootstrap-server localhost:9092 is not returning anything
kafka-topics.bat --list --bootstrap-server localhost:9092 is not showing anything
The above command meant to show me the topic list but it wasn't returning anything.
It can only be two things
The cluster at localhost:9092 does not have any topics
There are ACL's preventing you from listing topics.
Have you tried creating a topic first? You can do this with
kafka-topics.sh --bootstrap-server localhost:9092 --topic first_topic --create --partitions 3 --replication-factor 1
we the following kafka config command we can set the retention bytes to 1000000
TOPIC_NAME=test
kafka-configs --alter --zookeeper localhost:2181 --entity-type topics --entity-name $TOPIC_NAME --add-config retention.bytes=1000000
but how to do the opposite way to find the retention bytes per topic ?
You can use describe key
./kafka-configs --zookeeper localhost:2181 --describe --entity-type topics --entity-name test
The altered properties are returned, otherwise they are default from the broker.
I have a Kafka topic that seems to simultaneously exist and not exist.
kafka-topics.sh --bootstrap-server localhost:9092 --topic my-topic --delete
returns an error:
Topics in [] does not exist
Meanwhile, trying to re-create the topic
kafka-topics.sh --bootstrap-server localhost:9092 --topic my-topic \
--create --replication-factor 1 --partitions 1
returns
Topic already exists
It does not show up in the topics list with
kafka-topics.sh --list
I suspect some form of corruption but it's no clear how I can fully delete the topic so that it can be re-created
Instead of passing --bootstrap-server try with --zookeeper
kafka-topics.sh --zookeeper localhost:2181 --topic my-topic --delete
Would like to share different ways to purge or delete a kafka topic in 2.1.0 version. I've found similar question here Purge Kafka Topic however, the accepted answer has been deprecated and it works on Kafka version 0.8 and below hence, creating this question with answer.
This is not a duplicate question.
Kafka by default keeps the messages for 168 hrs which is 7 days. If you wanted to force kafka to purge the topic, you can do this by multiple ways. Let’s see each in detail.
1. Using kafka-configs.sh command
Temporarily change the retention policy to 1 sec.
kafka-configs.sh --zookeeper localhost:2181 --alter --entity-type topics --add-config retention.ms=1000 --entity-name text_topic
You can check the current value of retention policy by running below command.
kafka-configs.sh --zookeeper localhost:2181 --entity-type topics --describe --entity-name text_topic
Configs for topic 'text_topic' are retention.ms=1000
Wait for 1 sec and remove the retention policy config which will set it back to default.
kafka-configs.sh --zookeeper localhost:2181 --alter --entity-type topics --delete-config retention.ms --entity-name text_topic
2. Delete topic and re-create
Before we delete an existing topic, first get the partition and replica of the current topic as you would need these to re-create the topic. You can get this information by running describe of the topic
kafka-topics.sh --zookeeper localhost:2181 --describe --topic text_topic
Topic:text_topic PartitionCount:3 ReplicationFactor:3 Configs:
Topic: text_topic Partition: 0 Leader: 0 Replicas: 0 Isr: 0
Delete the topic.
kafka-topics.sh --zookeeper localhost:2181 --delete --topic text_topic
Re-create the topic with replication and partition details.
kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 3 --partitions 3 --topic text_topic
3. Manually delete the data from kafka logs.
Stop zookeeper and kafka from all nodes.
Clean kafka logs from all nodes. kafka stores its log files at
/tmp/kafka-logs/MyTopic-0 where /tmp/kafka-logs is specified by the
log.dirattribute
Restart zookeeper and kafka.
Hope this helps !!
I created the new topic 'rahul' with the following command :
bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic rahul
Created topic "rahul".
I also re-checked the topics with
bin/kafka-topics.sh --list --zookeeper localhost:2181
__consumer_offsets
rahhy
rahul`
Now starting the producer:
bin/kafka-console-producer.sh --broker-list localhost:9092 --topic rahul
hey
hi
hello
But when the time comes to consumer to show the messages: there is nothing
As of Kafka 0.9, you don't use Zookeeper for consumption or production
Try kafka-console-consumer --topic rahul --bootstrap-server localhost:9092
There are other ways you can check messages were sent to Kafka - by checking that the offsets of the topic partitions have changed using GetOffsetShell