Delete topic in Kafka 0.8.1.1 - apache-kafka

I need to delete the topic test in Apache Kafka 0.8.1.1.
As expressed in the documentation here, I have executed:
bin/kafka-topics.sh --zookeeper localhost:2181 --delete --topic test
However, this results in the following message:
Command must include exactly one action: --list, --describe, --create or --alter
How can I delete this topic?

Deleting topic isn't always working in 0.8.1.1
Deletion should be working in the next release, 0.8.2
kafka-topics.sh --delete --zookeeper localhost:2181 --topic your_topic_name
Topic your_topic_name is marked for deletion.
Note: This will have no impact if delete.topic.enable is not set to true.
You may also pass in a bootstrap server instead of zookeeper:
kafka-topics.sh --bootstrap-server kafka:9092 --delete --topic your_topic_name
Is it possible to delete a topic?
Jira KAFKA-1397

It seems that the deletion command was not officially documented in Kafka 0.8.1.x because of a known bug (https://issues.apache.org/jira/browse/KAFKA-1397).
Nevertheless, the command was still shipped in the code and can be executed as:
bin/kafka-run-class.sh kafka.admin.DeleteTopicCommand --zookeeper localhost:2181 --topic test
In the meantime, the bug got fixed and the deletion command is now officially available from Kafka 0.8.2.0 as:
bin/kafka-topics.sh --delete --zookeeper localhost:2181 --topic test

Add below line in ${kafka_home}/config/server.properties
delete.topic.enable=true
Restart the kafka server with new config:
${kafka_home}/bin/kafka-server-start.sh ~/kafka/config/server.properties
Delete the topics you wish to:
${kafka_home}/bin/kafka-topics.sh --delete --zookeeper localhost:2181 --topic daemon12

Andrea is correct. we can do it using command line.
And we still can program it, by
ZkClient zkClient = new ZkClient("localhost:2181", 10000);
zkClient.deleteRecursive(ZkUtils.getTopicPath("test2"));
Actually I do not recommend you delete topic on Kafka 0.8.1.1. I can delete this topic by this method, but if you check log for zookeeper, deletion mess it up.

Steps to Delete 1 or more Topics in Kafka
#To delete topics in kafka the delete option needs to be enabled in Kafka server.
1. Go to {kafka_home}/config/server.properties
2. Uncomment delete.topic.enable=true
#Delete one Topic in Kafka enter the following command
kafka-topics.sh --delete --zookeeper localhost:2181 --topic <your_topic_name>
#To Delete more than one topic from kafka
(good for testing purposes, where i created multiple topics & had to delete them for different scenarios)
Stop the Kafka Server and Zookeeper
go to server folder where the logs are stored (defined in their config files) and delete the kafkalogs and zookeeper folder manually
Restart the zookeeper and kafka server and try to list topics,
bin/kafka-topics.sh --list --zookeeper localhost:2181
if no topics are listed then the all topics have been deleted successfully.If topics are listed, then the delete was not successful. Try the above steps again or restart your computer.

You can delete a specific kafka topic (example: test) from zookeeper shell command (zookeeper-shell.sh). Use the below command to delete the topic
rmr {path of the topic}
example:
rmr /brokers/topics/test

This steps will delete all topics and data
Stop Kafka-server and Zookeeper-server
Remove the data directories of both services, by default on windows they are
C:/tmp/kafka-logs and C:/tmp/zookeeper.
then start Zookeeper-server and Kafka-server

The command:
bin/kafka-topics.sh --delete --zookeeper localhost:2181 --topic test
unfortunately only marks topic for deletion.
Deletion does not happen.
That makes troubles, while testing any scripts, which prepares Kafka configuration.
Connected threads:
Purge Kafka Queue
Is there a way to delete all the data from a topic or delete the topic before every run?

As mentioned in doc here
Topic deletion option is disabled by default. To enable it set the server config
delete.topic.enable=true
Kafka does not currently support reducing the number of partitions for a topic or changing the replication factor.
Make sure delete.topic.enable=true

Adding to above answers one has to delete the meta data associated with that topic in zookeeper consumer offset path.
bin/zookeeper-shell.sh zookeeperhost:port
rmr /consumers/<sample-consumer-1>/offsets/<deleted-topic>
Otherwise the lag will be negative in kafka-monitoring tools based on zookeeper.

First, you run this command to delete your topic:
$ bin/kafka-topics.sh --delete --bootstrap-server localhost:9092 --topic <topic_name>
List active topics to check delete completely:
$ bin/kafka-topics.sh --list --bootstrap-server localhost:9092

If you have issues deleting the topics, try to delete the topic using:
$KAFKA_HOME/bin/kafka-topics.sh --delete --zookeeper localhost:2181 --topic your_topic_name
command. Then in order to verify the deletion process, go to the kafka logs directory which normally is placed under /tmp/kafka-logs/, then delete the your_topic_name file via rm -rf your_topic_name command.
Remember to monitor the whole process via a kafka management tool like Kafka Tool.
The mentioned process above will remove the topics without kafka server restart.

There is actually a solution without touching those bin/kafka-*.sh: If you have installed kafdrop, then simply do:
url -XPOST http://your-kafdrop-domain/topic/THE-TOPIC-YOU-WANT-TO-DELETE/delete

bin/kafka-topics.sh --delete --zookeeper localhost:2181 --topic <topic-name>

Step 1: Make sure you are connected to zookeeper and Kafka running
Step 2: To delele the Kafka topic run kafka-topics script, add the port and --topic with name of your topic and --delete it just delete the topic with success.
# Delete the kafka topic
# it will delete the kafka topic
bin/kafka-topics.sh --zookeeper 127.0.0.1:2181 --topic name_of_topic --delete

This worked for me:
kafka-topics --delete --bootstrap-server localhost:9092 --topic user-commands
I am using Confluent CLI and run this on the directory I installed it.

Recent Kafka versions are about to remove the Zookeeper dependency. Therefore, you should instead reference the brokers (through --boostrap-server):
kafka-topics \
--bootstrap-server localhost:9092,localhost:9093,localhost:9094 \
--delete \
--topic topic_for_deletion

for confluent cloud use -
$ confluent kafka topic delete my_topic

Related

Delete topic level config

In order to delete all of the data in a topic I set the retention.ms config of it to 1000.
./bin/kafka-topics.sh --zookeeper $KAFKAZKHOSTS --alter --topic <topic> --config retention.ms=1000
This worked fine. All the data was deleted after a very short wait.
Before altering the config, the retention.ms was not set on the topic and so the server default property log.retention.hours=168 was the previous retention policy. (log.retention.minutes and log.retention.ms had not been set in the server properties).
Now I would like to remove the retention.ms config from this topic completely and go back to using the server level config.
Commands like
./bin/kafka-topics.sh --zookeeper $KAFKAZKHOSTS --alter --topic <topic> --config retention.ms=
or
./bin/kafka-topics.sh --zookeeper $KAFKAZKHOSTS --alter --topic <topic> --config retention.ms=null
throw an error.
I know that the delete option for kafka-topics.sh actually deletes the entire topic, so I'm not going to try play around with that.
Question: How do I completely remove a topic level config so that the topic reverts to using the server default?
To remove a topic configuration override, you can use the kafka-config tool. For example:
./bin/kafka-configs.sh --zookeeper <zookeeper> --alter \
--entity-type topics --entity-name <topic> --delete-config retention.ms
Another traditional solution which I used is manually deleting the particular folder for particular topic.
First stop the kafka server
Go to /var/lib/kafka/data (where your kafka data ios being stored specifies by you at the time of installation)
rm -rf /var/lib/kafka/data/yourTopicName-0

Kafkacat: how to delete a topic or all its messages?

I am looking for a way to delete a topic or all its messages using kafkacat.
Is it possible or the only way is through the script listed here?
bin/kafka-topics.sh --zookeeper localhost:2181 --delete --topic mytopic
There is no delete topic feature at this stage in kafkacat according to man page and github source code. So the only way is to use kafka-topics script.
github source code
man page
kafkacat is a generic non-JVM producer and consumer for Apache Kafka
0.8, think of it as a
netcat for Kafka.
In producer mode ( -P ), kafkacat reads messages from stdin, delimited with a configurable
delimeter and produces them to the provided Kafka cluster, topic and partition. In consumer
mode ( -C ), kafkacat reads messages from a topic and partition and prints them to stdout
using the configured message delimiter.
If neither -P or -C are specified kafkacat attempts to figure out the mode automatically
based on stdin/stdout tty types.
kafkacat also features a metadata list mode ( -L ), to display the current state of the
Kafka cluster and its topics and partitions.
As #Naween Banuka pointed out, you can also use zookeeper-shell.sh or zkCli.sh (found under zookeeper/bin) to do this:
List the existing topics : ./zookeeper-shell.sh localhost:2181 ls /brokers/topics
Remove Topic : ./zookeeper-shell.sh localhost:2181 rmr /brokers/topics/yourtopic
Yes, It's possible.
But first, You have to enable topic deletion on all brokers.
change delete.topic.enable to true.
By default, it's false (In server.properties file)
Then,
Use topic delete command.
bin/kafka-topics.sh --zookeeper localhost:2181 --delete --topic mytopic
If you want to delete that topic permanently,
You can use the zookeeper delete command.
List the existing topics : ./zookeeper-shell.sh localhost:2181 ls /brokers/topics
Remove Topic : ./zookeeper-shell.sh localhost:2181 rmr /brokers/topics/yourtopic

Not able to delete Topics from kafka

even after enabling delete.topic.enable=true in server.config
deletion of topics not working . i am getting following error on recreating the topic again
Topic 'test' already exists.
[2017-05-23 06:47:05,757] ERROR
org.apache.kafka.common.errors.TopicExistsException: Topic 'test' already exists.
You can not delete a topic when consuming it. Use bin/kafka-consumer-groups.sh or simple ps -aux|grep Consumer to find any possible consumers which block the operation.
remove meta data in zookeeper
./bin/zookeeper-shell.sh localhost:2181
rmr /brokers/topics/mytopic
rmr /admin/delete_topics/mytopic
If you use the latest Kafka (v. ~0.10.) then after you enabled the delete.topic.enable=true option you have to:
Restart Kafka
Delete topic:
kafka-topics.sh --zookeeper localhost:2181 --topic mytopic --delete
Check that it was marked for deletion:
kafka-topics.sh --zookeeper localhost:2181 --list
mytopic - marked for deletion
And wait a bit.
And if you use some old version of Kafka, then try to delete the topic from a zookeeper-shell.
If the zookeeper is a standalone instance (not on the localhost), mark for deletion of topics won't delete it properly.
One suggestion would be to use Zookeeper Exhibitor & delete it from admin & brokers.
Exhibitor give a UI interface to visualise how the topics & kafka brokers are arranged.

How do I delete a Kafka Consumer Group to reset offsets?

I want to delete a Kakfa consumer group so that when the application creates a consumer and subscribes to a topic it can start at the beginning of the topic data.
This is with a single node development vm using the current latest Confluent Platform 3.1.2 which uses Kafka 0.10.1.1.
I try the normal syntax:
sudo /usr/bin/kafka-consumer-groups --new-consumer --bootstrap-server localhost:9092 --delete --group my_consumer_group
I get the error:
Option [delete] is only valid with [zookeeper]. Note that there's no need to delete group metadata for the new consumer as the group is deleted when the last committed offset for that group expires.
If I try the zookeeper variant:
sudo /usr/bin/kafka-consumer-groups --zookeeper localhost:2181 --delete --group my_consumer_group
I get:
Delete for group my_consumer_group failed because group does not exist.
If I list using the "old" consumer, I do not see my consumer group (or any other consumer groups)
sudo /usr/bin/kafka-consumer-groups --zookeeper localhost:2181 --list
If I list using the "new" consumer, I can see my consumer group but apparently I can't delete it:
sudo /usr/bin/kafka-consumer-groups --new-consumer --bootstrap-server localhost:9092 --list
This can be done with Kafka 1.1.x. From the documentation:
bin/kafka-consumer-groups.sh --bootstrap-server localhost:9092 --delete --group my-group --group my-other-group
In Kafka 0.11 (or Confluent 3.3) you can reset the offsets of any existing consumer group without having to delete the topic. In fact you can change the offsets to any absolute offset value or timestamp or any relative position as well.
These new functions are all added with the new --reset-offsets flag on the kafka-consumer-groups command line tool.
See KIP-122 details here https://cwiki.apache.org/confluence/display/KAFKA/KIP-122%3A+Add+Reset+Consumer+Group+Offsets+tooling
Upgrading to the just released Confluent Platform 3.2 with Kafka 0.10.2 solved my underlying issue. When I delete a topic, offset information is now correctly reset. So when I create a topic with the same name, consumers start from the beginning of the new data.
I still can't delete new style consumer groups with the kafka-consumer-groups tool, but my underlying issue is solved.
Before Kafka 0.10.2, there were hacks, but no clean solution to this issue.
If you use Java client, you can first get the beginning offset.
TopicPartition partition = new TopicPartition("YOUR_TOPIC", YOUR_PARTITION);
Map<TopicPartition, Long> map = consumer.beginningOffsets(Collections.singleton(partition));
And the offset that consumer using to start processing, (if not delete the consumer group).
Long committedOffset = consumer.committed(partition).offset();
Now, if you think start from committedOffset is ok, just poll records.
if you want the beginning offset,
consumer.seek(partition, map.get(partition));
you can also reset the offset of single topic without deleting the entire consumer group :
bin/kafka-consumer-groups.sh --bootstrap-server localhost:9092 --group my-group --topic my-topic --reset-offsets --to-earliest --execute
If you're using Windows and you need a JAAS config with password & username to access your kafka cluster, you can use the following commands
#consumer.properties
sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required username="XXXXXXXXXXXXXXX" password="XXXXXXXXXXXXXXXXXXX";
security.protocol=SASL_SSL
sasl.mechanism=PLAIN
.\kafka\bin\windows\kafka-consumer-groups.bat --bootstrap-server xxxxxxxxxx.confluent.cloud:9092 --group cop-group --topic topic_name_ini --reset-offsets --to-earliest --execute --command-config .\kafka\config\consumer.properties
In the command above we used --command-config to pass a properties file for the JAAS config

Confusion about delete Kafka Topic

I'm using Kafka 0.8.0, it's Cloudera version.
When I deleted the topic such as: kafka-topics --zookeeper 10.0.0.11:2181/ --delete --topic test
it response:
Topic test is already marked for deletion.
But afterwards I recreated it, it throw exception as following:
kafka-topics --create --zookeeper 10.0.0.11:2181 --partitions 90 --replication-factor 2 --topic test
Error while executing topic command Topic "test" already exists.
kafka.common.TopicExistsException: Topic "test" already exists.
Any ideas please? How should I delete the topic and it's data.
My Kakfa version is kafka_2.10-0.8.2.2, below link works for me (from Delete topic in Kafka 0.8.1.1)
Add below line in ${kafka_home}/config/server.properties
delete.topic.enable=true
Restart the kafka server with new config:
${kafka_home}/bin/kafka-server-start.sh ~/kafka/config/server.properties
Delete the topics you wish to:
${kafka_home}/bin/kafka-topics.sh --delete --zookeeper localhost:2181 --topic daemon12
More information from Kafka FAQ:
Deleting a topic is supported since 0.8.2.x. You will need to enable
topic deletion (setting delete.topic.enable to true) on all brokers
first.
Make sure that, In kafka's config.poperties file 'delete.topic.enable' property should be true.
use below command
bin/kafka-topics.sh --delete --zookeeper localhost:2181 --topic test
Even if topic is not deleted then follow next steps.
open terminal zookeeper client mode:
A. stop zookeeper
B. rmr /broker/topics
C. check given topic using below command
/bin/kafka-topics.sh --zookeeper maxiq:2181 --list
If delete.topic.enable is set to false by default, the topics are not deleted on execution of --delete command (as displayed in the command response).
Topic test is marked for deletion.
Note: This will have no impact if delete.topic.enable is not set to true.
In order to overcome this, use steps mentioned by #Shawn
The best and easy to make delete.topic.enable=true is not modify the server.property file.
Because, if kafka restart from ambari, it will overwrite this file, delete.topic.enable =false again.
Only at ambari, click on kafak/config/advance kafka broker/delete.topic.enable =true, it will work.
I just found that out as now.
Delete topic in kafka
step 1 -->> cd /usr/lib/zookeeper/bin
step 2 -->> zkCli.sh -server 127.0.0.1:2181
step 3 -->>rmr /brokers/topics/topic_name