As far as I know version 1.0x and up, the config delete.topic.enable=true is the default configuration. Meaning, if I execute the delete command it should work as it expected.
However, I tried deleting the topic but it still there and was not deleted (when I describe the said topic in all brokers)
I read in some forums that I need to restart the broker servers and the zookeepers in order the deletion take effect.
Is there any other way to delete the topic without restarting the zookeeper and the brokers?
Related
I'm trying to delete Kafka topic __Consumer_offset as it is causing a lot of confusion for my brokers.
When i do so, it says this topic can't be marked for deletion.
i'm using the zookeeper cli to delete it such as rmr /brokers/topic __consumer_offset, but it is not working!
__consumer_offsets is a kafka internal topic and it is not allowed to be deleted through delete topic command. It contains information about committed offsets for each topic:partition for each group of consumers (groupID). If you want to wipe it out entirely you have to delete the zookeeper dataDir location. That implies, you lose all the metadata.
Also if you just want to get rid of the existing consumer groups, you can as well reset the offsets or consider deleting them.
AFAIK you cannot delete that topic. It is a internal topic and should not be deleted manually.
If it is must, then you will have to manually clean/remove your data directory. When you deploy Kafka brokers and Zookeepers it creates data directory.
Note: By removing data directory you will loose all topics and related data. So this is not feasible option in Production.
I was using a kafka topic, and it's metadata as well in my application. I hard deleted the topic from the zookeeper shell, by deleting the directories corresponding to that topic. After creating the topic again, I described the topic and found that no leaders have been assigned to this newly created topic. In the consumer, I can see repeated logs printing LEADER_NOT_AVAILABLE. Any reason as to what am I doing wrong? Or maybe is there a way to delete the metadata related to the kafka topic as well that I'm unaware of? Thanks in advance!
Deleting topics in Kafka hasn't been straightforward until recently. In general, you shouldn't attempt to delete Kafka topics by deleting metadata in Zookeeper. You should always use the included command line utilities.
First you need to make sure that deleting topics is enabled in the server.properties file on all brokers, and do a rolling restart if needed:
delete.topic.enable=true
After you restart the brokers to enable topic deletion, you should issue the delete command using the command line utilities:
./kafka-topics.sh —zookeeper <zookeeper_host>:2181 —delete —topic <topic_name>
If at this point, it's still stuck, try to run these two commands from the zookeeper shell to make sure and remove all metadata for that particular topic:
rmr /brokers/topics/<topic_name>
rmr /admin/delete_topics/<topic_name>
A few more details here:
https://medium.com/#contactsunny/manually-delete-apache-kafka-topics-424c7e016ff3
Having installed KAFKA and having looked at these posts:
kafka loses all topics on reboot
Kafka topic no longer exists after restart
and thus moving kafka-logs to /opt... location, I still note that when I reboot:
I can re-create the topic again.
the kafka-logs directory contains information on topics, offsets etc. but it gets corrupted.
I am wondering how to rectify this.
Testing of new topics prior to reboot works fine.
There can be two potential problems
If it is kafka running in docker, then docker image restart always cleans up the previous state and creates a new cluster hence all topics are lost.
Check the log.dir or Zookeeper data path. If either is set to /tmp directory, it will be cleaned on each reboot. Hence you will lose all logs and topics will be lost.
In this VM I noted the Zookeeper log was defined on /tmp. Changed that to /opt (presume it should be /var though) and the clearing of Kafka data when instance terminated was corrected. Not sure how to explain this completely.
I have set up kafka version 2.11-0.10.1.0 in multinode multicluster environment .In kafka server.properties I already added delete.enable.topic=true in all the 3 machines.
I am using command for delete topic is ,
./bin/kafka-topic.sh --zookeeper ip1:2181,ip2:2181,ip3:2181 --delete --topic topicname
but It's not deleting ,showing topic name -mark for deletion
So everytime I am clearing kafka-logs and zookeeper logs for delete topic .
Anybody having any idea to delete using command prompt .
In general everything you are doing sounds right, I suspect that your problem is a simple typo.
The parameter to enable topic deletion in Kafka is called: delete.topic.enable not, as you stated above: delete.enable.topic.
This would cause deletion to revert to its default value, which is false and result in the behavior you are seeing.
Correcting this and restarting should fix your issue and delete all topics.
I'm trying to delete a topic nodes in Zookeeper for Kafka. I tried: rmr /brokers/topics, as well as deleting individual node topics (delete /brokers/topics/topic-1).
It worked for most topics, but I 4 topics/nodes that just refuse to get deleted.
Any ideas?