I'm trying to remove a Kafka topic with 8 partitions and 2 replications. First I delete that topic using kafka-topic.sh --delete command. Then I used zkCli.sh -server slave1.....slave3, and rmr /brokers/topics/MY_TOPIC.
However I still see that topic in /brokers/topics/. And I tried restart Kafka, everything still the same.
Btw, topic with 1 partition and 1 replica can be deleted successfully.
You can set server properties to enable delete of kafka topic
Add line mentioned below in service.properties
delete.topic.enable = true
If you removing manually using rmr /brokers/topics/MY_topic then you also need to remove topic related metadata from other nodes in zookeeper ex- consumer information about that topic. Also need to remove kafka topic director on kafka server.
It is cleaner to enable topic delete property and execute kafka-topics.sh --delete
Related
I have a Kafka cluster consisting on 3 servers all connected through Zookeeper. But when I delete a topic that has some information and create the topic again with the same name, the offset does not start from zero.
I tried restarting both Kafka and Zookeeper and deleting the topics directly from Zookeeper.
What I expect is to have a clean topic When I create it again.
I found the problem. A consumer was consuming from the topic and the topic was never actually deleted. I used this tool to have a GUI that allowed me to see the topics easily https://github.com/tchiotludo/kafkahq. Anyway, the consumers can be seen running this:
bin/kafka-consumer-groups.sh --list --bootstrap-server localhost:9092
I am using bin/kafka-topics.sh --zookeeper --delete --topic and i see in kafka logs of that indicate that the partitions for that topic are marked for deletion. However, I am still seeing the directories for those partitions present in the data dir.
Is this something expected and I am have manually delete them?
The topics haven't been removed from the zookeeper also. I still see the topics in zookeeper. Is this also expected?
Thanks!
There could be several reasons for topics not being deleted automatically.
In order to delete a topic delete.topic.enable should be set to true.
If it is set to true, it should ideally delete the directories from Zookeeper and kafka data.dir . But in case, if it doesn't, you should check the logs to make sure if there is any problem with kafka brokers or zookeeper due to some LEADER selection issue.
So in that case, you have to cleanup the dirs manually.
I was using a kafka topic, and it's metadata as well in my application. I hard deleted the topic from the zookeeper shell, by deleting the directories corresponding to that topic. After creating the topic again, I described the topic and found that no leaders have been assigned to this newly created topic. In the consumer, I can see repeated logs printing LEADER_NOT_AVAILABLE. Any reason as to what am I doing wrong? Or maybe is there a way to delete the metadata related to the kafka topic as well that I'm unaware of? Thanks in advance!
Deleting topics in Kafka hasn't been straightforward until recently. In general, you shouldn't attempt to delete Kafka topics by deleting metadata in Zookeeper. You should always use the included command line utilities.
First you need to make sure that deleting topics is enabled in the server.properties file on all brokers, and do a rolling restart if needed:
delete.topic.enable=true
After you restart the brokers to enable topic deletion, you should issue the delete command using the command line utilities:
./kafka-topics.sh —zookeeper <zookeeper_host>:2181 —delete —topic <topic_name>
If at this point, it's still stuck, try to run these two commands from the zookeeper shell to make sure and remove all metadata for that particular topic:
rmr /brokers/topics/<topic_name>
rmr /admin/delete_topics/<topic_name>
A few more details here:
https://medium.com/#contactsunny/manually-delete-apache-kafka-topics-424c7e016ff3
I am not able to delete kafka topic, Its marked for deletion but never gets deleted. Iam running kafka cluster with zookeeper cluster.
version of kafka : 0.10.2.1
Can anyone help me , with the list of steps that one needs to follow in order to delete a topic in kafka cluster.
Went through various queries in stackoverflow but could not find a valid workable answer.
You should have enabled its property at config before starting kafka server; it is disabled at default. To enable deletion property first stop kafka server and then open the server.properties in config file
and then uncomment #delete.topic.enable=true or add
delete.topic.enable=true
at the end of the file.
Now you can start kafka server and then you can delete any topic you want via:
bin/kafka-topics.sh --delete --zookeeper localhost:2181 --topic YOUR_TOPIC_NAME.
You could use Kafka Tool
Download link here
Then connect to your kafka server .
After that you could see the available topics in that server. From there you can select and delete the topics .
I'm facing issue with kafka topic deletion.
Using kafka rest API's to create/delete topic and for producing & consuming messages. I have tried to delete the topic where the topic will be deleted, but after some time say 10 sec, topic gets reappeared.
Have checked the Consumer Group offsets and LAG is listed as negative.
docker run --net=host --rm confluentinc/cp-kafka:3.1.0 kafka-consumer-groups --zookeeper localhost:2181 --describe --group grp1
GROUP TOPIC PARTITION CURRENT-OFFSET LOG-END-OFFSET LAG OWNER
grp1 topic1 0 3 0 -3 none
It seems deleting a kafka topic is still has some bugs.
* The only way to delete a topic permanently is as follows: *
stop the brokers
remove the directories on disk
sudo rm -rf kafka_data_dir/topic_name
remove the topic from zookeeper:
bin/zkCli.sh - to start zookeeper shell
rmr /config/topics/topic_name
rmr /brokers/topics/topic_name
rmr /admin/delete_topics/topic_name
You might have enabled the auto.create.topics.enable property which automatically creates a topic if any Producer [or] Consumer issues a request to Kafka broker with send / subscribe / assign request.
Once you have deleted the topic, either Consumer / Producer issued a request to the broker which in-turn created a new topic (with the same name).
You can disable the property(default: true) and re-test your setup. Use AdminUtils to create the topic. Topic deletion is much improved in the latest version of Kafka.