I marked a topic for deletion and it sat there forever not deleting (even though delete.topic.enable is set to true). So I followed the instructions and shelled into one of the zookeepers and ran the following to get it deleted:
rmr /brokers/topics/topicname
rmr /admin/delete_topics/topicname
The topic then appeared to be deleted (would not come back on a list command). But then when I tried to recreate it with new configuration (compaction turned on), the in-sync-replicas are empty and I can't consume from the topic. Consumption comes back with 'UNKNOWN_TOPIC_OR_PARTITION' errors even though the list command shows the topic as being there.
Is there a log somewhere I can look at to see why it is not able to get the topic setup properly after deletion and recreation? Am I missing a step and not properly deleting the topic to begin with? Why is the recreated topic not getting properly initialized?
What I ran to delete the topic initially before running the two commands above (this left the topic in 'marked for deletion' for a long time):
./kafka-topics.sh --zookeeper $KAFKAZKHOSTS --delete --topic topicname
What I ran to recreate the topic:
/usr/hdp/current/kafka-broker/bin/kafka-topics.sh --create --zookeeper $KAFKAZKHOSTS --replication-factor 3 --partitions 3 --topic topicname --config cleanup.policy=compact
Kafka version: 1.1.0.2.6.5.3005-27
So I read somewhere that you should restart the brokers and that may solve it. So I tried that and sure enough after the restart the ISRs are in the right state and the topic is consumable again.
I would still like to know under what circumstances this can happen and if there's a way to fix it without restarting brokers since in a production environment I would like to avoid doing that.
Related
This is definitively not the way to do it, and it should probably be handled by the cleanup policy but that's not the point. Let's imagine the files in log.dirs has been deleted, what's the impact ?
The broker would crash ?
The offset would start over at 0 after restarting the service ?
Would it be necessary to do anything to fix ?
If you delete the files from log.dirs, the data will be deleted but topic will still exist in zookeeper metadata. The broker won't crash. Once you restart the brokers, it will read the topic as an empty one and you can produce new data.
If you delete the topic from zookeeper metadata as well, it will delete the topic from broker.
In order to check the offsets you can use below command:
// Before deleting the log.dirs directory for topic 'test1'
kafka_2.12-1.1.1 % bin/kafka-run-class.sh kafka.tools.GetOffsetShell --broker-list localhost:9092 --topic test1
test1:0:6
// After deleting the directory and restarting the broker
kafka_2.12-1.1.1 % bin/kafka-run-class.sh kafka.tools.GetOffsetShell --broker-list localhost:9092 --topic test1
test1:0:0
In fact it will depend how many brokers you have in your cluster, and from how many of them you delete the files at the same time. Luckily if you delete the files from one broker in a 3-broker cluster, and you have defined a replication factor of 3 for your topics, you will not lose anything and the files will be recreated on the broker where you deleted them.
I am using bin/kafka-topics.sh --zookeeper --delete --topic and i see in kafka logs of that indicate that the partitions for that topic are marked for deletion. However, I am still seeing the directories for those partitions present in the data dir.
Is this something expected and I am have manually delete them?
The topics haven't been removed from the zookeeper also. I still see the topics in zookeeper. Is this also expected?
Thanks!
There could be several reasons for topics not being deleted automatically.
In order to delete a topic delete.topic.enable should be set to true.
If it is set to true, it should ideally delete the directories from Zookeeper and kafka data.dir . But in case, if it doesn't, you should check the logs to make sure if there is any problem with kafka brokers or zookeeper due to some LEADER selection issue.
So in that case, you have to cleanup the dirs manually.
$ kafka-topics --zookeeper localhost:2181 --list | grep my-topic
my-topic
$ kafka-topics --zookeeper localhost:2181 --delete --topic my-topic
Topic my-topic is marked for deletion.
Note: This will have no impact if delete.topic.enable is not set to true.
$ kafka-topics --zookeeper localhost:2181 --list | grep my-topic
my-topic - marked for deletion
It's been hours since then, and the topic has still not been deleted.
I've seen suggestions that I should put delete.topic.enable in my server.properties and restart kafka. I've tried this. It hasn't worked.
$ grep -F delete.topic.enable /usr/local/etc/kafka/server.properties
delete.topic.enable = true
(Why wouldn't this be set by default?)
In any case, these suggestions have all been for kafka 0.* and I am on a newer version.
$ brew list --versions kafka
kafka 1.1.0
How do I delete topics?
I can shutdown kafka and zookeeper, run rm -r /usr/local/var/lib/kafka-logs /usr/local/var/run/zookeeper/data, then start zookeeper and kafka again. But this is rather drastic. Surely there should be some way of persuading kafka-topics to actually delete a topic?
The command you're using is the recommended way to delete a topic.
Unfortunately there are a few reasons why a topic can stay stuck in "marked for deletion" state.
If you check the controller logs, you might see why it's not able to delete it. In the controller logs, all lines from the Deletion Manager are prefixed with Topic Deletion Manager.
Periodically, you should see messages like Handling deletion for topics .... In case the controller has given up the deletion, it should have logged it, with this format: Halted deletion of topics ....
If you want to force the deletion, you can try:
Deleting the topic details in Zookeeper. Using zookeeper-shell.sh, run rmr /brokers/topics/TOPIC_NAME
Bouncing the controller
Finally if none of this works, you can, as hinted, delete the log data on disk/
On the suggestion of a colleague, I
shut down Kafka and Zookeeper
added the line listeners=PLAINTEXT://127.0.0.1:9092 to /usr/local/etc/kafka/server.properties
restarted Zookeeper and Kafka
Now I can delete topics again. I don't know why this should work, but suspect it is related to a recent macOS upgrade.
I have marked 3 topics for deletion using this command
kafka-topics --zookeeper localhost:2181 --delete --topic topic_name
Now I can't use them neither I can recreate them how to resolve this issue ?
Either to delete them totally then recreate them or unmark the status for deletion.
Yes, you can restore it if not yet deleted!
Login to zookeeper prompt and use rmr path command:
sudo ./zookeeper-shell.sh localhost:2181
rmr /admin/delete_topics/topic_name
Did you list and see whether those topics are there? In order to list:
kafka-topics.sh --list --zookeeper localhost:2181
Deleting a topic is supported since 0.8.2.x. You will need to enable topic deletion (setting delete.topic.enable to true) on all brokers first.
Make sure you've done the change in the broker config file and then try deleting the topics.
Ref: Is it possible to delete a topic
I'm not sure whether you'll be able to unmark the ones you've marked for deletion. As far as I know you'd be able to delete it totally and recreate them.
I am using Kafka version 0.8.2. In the middle of development I thought I might need to delete a topic. So what I did is I put the following line in the server configuration files and started two kafka servers.
delete.topic.enable=true
When I needed to delete a topic and I ran following command,
bin/kafka-topics.sh --zookeeper localhost:2181 --delete --topic MyTopic
Right now its already 17 hours since when I ran delete topic command but Kafka is still showing me the topic is marked for deletion. Is it a bug of Kafka or I am doing something wrong here? Because it doesn't seem normal to me. Can someone clarify in this regard?
[N.B.: I have already read this thread. But couldn't find my answer there.]
AFAIK it is not possible to delete kafka topic if you previously had delete.topic.enable=false. You can change this value, but it does not affect currently available topics. So, you can try to delete it manually.
You can go to the log folder location and manually delete the files concerning the topic you want to delete.