I am using Kafka version 0.8.2. In the middle of development I thought I might need to delete a topic. So what I did is I put the following line in the server configuration files and started two kafka servers.
delete.topic.enable=true
When I needed to delete a topic and I ran following command,
bin/kafka-topics.sh --zookeeper localhost:2181 --delete --topic MyTopic
Right now its already 17 hours since when I ran delete topic command but Kafka is still showing me the topic is marked for deletion. Is it a bug of Kafka or I am doing something wrong here? Because it doesn't seem normal to me. Can someone clarify in this regard?
[N.B.: I have already read this thread. But couldn't find my answer there.]
AFAIK it is not possible to delete kafka topic if you previously had delete.topic.enable=false. You can change this value, but it does not affect currently available topics. So, you can try to delete it manually.
You can go to the log folder location and manually delete the files concerning the topic you want to delete.
Related
I marked a topic for deletion and it sat there forever not deleting (even though delete.topic.enable is set to true). So I followed the instructions and shelled into one of the zookeepers and ran the following to get it deleted:
rmr /brokers/topics/topicname
rmr /admin/delete_topics/topicname
The topic then appeared to be deleted (would not come back on a list command). But then when I tried to recreate it with new configuration (compaction turned on), the in-sync-replicas are empty and I can't consume from the topic. Consumption comes back with 'UNKNOWN_TOPIC_OR_PARTITION' errors even though the list command shows the topic as being there.
Is there a log somewhere I can look at to see why it is not able to get the topic setup properly after deletion and recreation? Am I missing a step and not properly deleting the topic to begin with? Why is the recreated topic not getting properly initialized?
What I ran to delete the topic initially before running the two commands above (this left the topic in 'marked for deletion' for a long time):
./kafka-topics.sh --zookeeper $KAFKAZKHOSTS --delete --topic topicname
What I ran to recreate the topic:
/usr/hdp/current/kafka-broker/bin/kafka-topics.sh --create --zookeeper $KAFKAZKHOSTS --replication-factor 3 --partitions 3 --topic topicname --config cleanup.policy=compact
Kafka version: 1.1.0.2.6.5.3005-27
So I read somewhere that you should restart the brokers and that may solve it. So I tried that and sure enough after the restart the ISRs are in the right state and the topic is consumable again.
I would still like to know under what circumstances this can happen and if there's a way to fix it without restarting brokers since in a production environment I would like to avoid doing that.
I am using bin/kafka-topics.sh --zookeeper --delete --topic and i see in kafka logs of that indicate that the partitions for that topic are marked for deletion. However, I am still seeing the directories for those partitions present in the data dir.
Is this something expected and I am have manually delete them?
The topics haven't been removed from the zookeeper also. I still see the topics in zookeeper. Is this also expected?
Thanks!
There could be several reasons for topics not being deleted automatically.
In order to delete a topic delete.topic.enable should be set to true.
If it is set to true, it should ideally delete the directories from Zookeeper and kafka data.dir . But in case, if it doesn't, you should check the logs to make sure if there is any problem with kafka brokers or zookeeper due to some LEADER selection issue.
So in that case, you have to cleanup the dirs manually.
I was using a kafka topic, and it's metadata as well in my application. I hard deleted the topic from the zookeeper shell, by deleting the directories corresponding to that topic. After creating the topic again, I described the topic and found that no leaders have been assigned to this newly created topic. In the consumer, I can see repeated logs printing LEADER_NOT_AVAILABLE. Any reason as to what am I doing wrong? Or maybe is there a way to delete the metadata related to the kafka topic as well that I'm unaware of? Thanks in advance!
Deleting topics in Kafka hasn't been straightforward until recently. In general, you shouldn't attempt to delete Kafka topics by deleting metadata in Zookeeper. You should always use the included command line utilities.
First you need to make sure that deleting topics is enabled in the server.properties file on all brokers, and do a rolling restart if needed:
delete.topic.enable=true
After you restart the brokers to enable topic deletion, you should issue the delete command using the command line utilities:
./kafka-topics.sh —zookeeper <zookeeper_host>:2181 —delete —topic <topic_name>
If at this point, it's still stuck, try to run these two commands from the zookeeper shell to make sure and remove all metadata for that particular topic:
rmr /brokers/topics/<topic_name>
rmr /admin/delete_topics/<topic_name>
A few more details here:
https://medium.com/#contactsunny/manually-delete-apache-kafka-topics-424c7e016ff3
I have set up kafka version 2.11-0.10.1.0 in multinode multicluster environment .In kafka server.properties I already added delete.enable.topic=true in all the 3 machines.
I am using command for delete topic is ,
./bin/kafka-topic.sh --zookeeper ip1:2181,ip2:2181,ip3:2181 --delete --topic topicname
but It's not deleting ,showing topic name -mark for deletion
So everytime I am clearing kafka-logs and zookeeper logs for delete topic .
Anybody having any idea to delete using command prompt .
In general everything you are doing sounds right, I suspect that your problem is a simple typo.
The parameter to enable topic deletion in Kafka is called: delete.topic.enable not, as you stated above: delete.enable.topic.
This would cause deletion to revert to its default value, which is false and result in the behavior you are seeing.
Correcting this and restarting should fix your issue and delete all topics.
During Testing I do some operation like delete a topic.
However I can see the topic after immeidately deleting it
Using: bin/kafka-topics.sh --list --zookeeper localhost:2181
It takes some time for deletion to actually occur.
This confused me.
Similarly when i produce data, i cannot consume immediately but have to wait for some time and re-run the consume command to consume data.
Is it because I am running a single node kafka set up and I am testing it too heavily.
If you want to delete a topic, you need to enable this via broker setting delete.topic.enable.
Deleting a topic happens "asynchronously", ie, after you did issues the delete command, the topic gets marked as "to be deleted" and the broker will delete the topic at some point later on.
About producing/consuming: not sure. If the consumer is online and the producer writes data it should up at the consumer shortly after...