I am very new to Kafka. Following a few tutorials, I have the following questions regarding consuming actual Kafka topics.
The situation: there is a server in my workplace that is streaming Kafka topics. I have the topic name. I would like to consume this topic from my machine (Windows WSL2 Ubuntu). From this tutorial, I am able to
Run zookeeper with this command:
bin/zookeeper-server-start.sh config/zookeeper.properties
Create a broker with:
bin/kafka-server-start.sh config/server.properties
Run a producer console, with a fake topic named quickstart-events at port localhost:9092:
bin/kafka-console-producer.sh --topic quickstart-events --bootstrap-server localhost:9092
Run a consumer console listening to localhost:9092 and receive the streaming data from the producer:
bin/kafka-console-consumer.sh --topic quickstart-events --from-beginning --bootstrap-server localhost:9092
Now for my real situation: if I know the topic name, what else do I need in order to apply the same steps above to listen to it as a consumer? What are the steps involved? I read in other threads about tunnelling with Jumphost. How to do that?
I understand this question is rather generic. Appreciate any pointers to any relevant readings or guidance.
Based on your company nameserver the next procedure should be done in your wsl instance
To gain outside connection
unable to access network from WSL2
You need to set bootstrap-server to your company server
--bootstrap-server my.company.com:9092
I'm trying to test run a single Kafka node with 3 brokers & zookeeper. I wish to test using the console tools. I run the producer as such:
kafka-console-producer --broker-list localhost:9092,localhost:9093,localhost:9094 --topic testTopic
Then I run the consumer as such:
kafka-console-consumer --zookeeper localhost:2181 --topic testTopic --from-beginning
And I can enter messages in the producer and see them in the consumer, as expected. However, when I run the updated version of the consumer using bootstrap-server, I get nothing. E.g
kafka-console-consumer --bootstrap-server localhost:9092,localhost:9093,localhost:9094 --topic testTopic --from-beginning
This worked fine when I had one broker running on port 9092 so I'm thoroughly confused. Is there a way I can see what zookeeper is providing as the bootstrap server? Is the bootstrap server different from the broker list? Kafka compiled with Scala 2.11.
I have no idea what was wrong. Likely I put Kafka or Zookeeper in a weird state. After deleting the topics in the log.dir of each broker AND the zookeeper topics in /brokers/topics then recreating the topic, Kafka consumer behaved as expected.
Bootstrap servers are same as kafka brokers. And if you want to see the list of bootstrap server zookeeper is providing, you can query ZNode information via any ZK client. All active brokers are registered under /brokers/ids/[brokerId]. All you need is zkQuorum address. Below command will give you
list of active bootstrap servers :
./zookeeper-shell.sh localhost:2181 <<< "ls /brokers/ids"
I experienced the same problem when using mismatched versions of:
Kafka client libraries
Kafka scripts
Kafka brokers
In my exact scenario I was using Confluent Kafka client libraries version 0.10.2.1 with Confluent Platform 3.3.0 w/ Kafka broker 0.11.0.0. When I downgraded my Confluent Platform to 3.3.2 which matched my client libraries the consumer worked as expected.
My theory is that the latest kafka-console-consumer using the new Consumer API was only retrieving messages using the latest format. There were a number of message format changes introduced in Kafka 0.11.0.0.
Could not find or load main class org.apache.kafka.connect.cli.ConnectStandalone
I get this error when I try to run connect-standalone in kafka.
First I realized that there was no connect folder in the libexec where this file was, I downloaded that. Still the same error.
So I tried to compile ConnectStandalone.java file separately, and I saw that a lot of packages are missing from my folder of kafka. There were supposed to be connect and common folders with some packages in org/apache/kafka but there weren't.
Package org.slf4j is also missing.
I tried to download a few manually, but if someone could tell me where I can get the correct folder.
All I want to do is to create a simple kafka producer that reads a csv and sends it line by line to a consumer.
Please tell me if I am at least heading in the correct direction.
EDIT: I used brew to download kafka, if that is relevant.
It seems that the kafka install has incomplete binaries. To have producer consumer of csv file. You can run sample thing as an example from Linux machine.
Assuming kafka borker is at localhost:9092 and zookeeper at localhost:2181, Open 2 console and go to kafka home
In first console create topic test:
bin/kafka-topics.sh --zookeeper localhost:2181 --create --topic test --partitions 1 --replication-factor 1
Then open as consumer in that console by:
bin/kafka-console-consumer.sh --zookeeper localhost:2181 --topic test
In second console send csv data to test topic:
bin/kafka-console-producer.sh --broker-list localhost:9092 --topic test < abc.csv
See on first console the csv data consumed.
If above commands also not running then you need a fresh re-install of kafka and zookeeper.
I'm using Kafka 0.8.0, it's Cloudera version.
When I deleted the topic such as: kafka-topics --zookeeper 10.0.0.11:2181/ --delete --topic test
it response:
Topic test is already marked for deletion.
But afterwards I recreated it, it throw exception as following:
kafka-topics --create --zookeeper 10.0.0.11:2181 --partitions 90 --replication-factor 2 --topic test
Error while executing topic command Topic "test" already exists.
kafka.common.TopicExistsException: Topic "test" already exists.
Any ideas please? How should I delete the topic and it's data.
My Kakfa version is kafka_2.10-0.8.2.2, below link works for me (from Delete topic in Kafka 0.8.1.1)
Add below line in ${kafka_home}/config/server.properties
delete.topic.enable=true
Restart the kafka server with new config:
${kafka_home}/bin/kafka-server-start.sh ~/kafka/config/server.properties
Delete the topics you wish to:
${kafka_home}/bin/kafka-topics.sh --delete --zookeeper localhost:2181 --topic daemon12
More information from Kafka FAQ:
Deleting a topic is supported since 0.8.2.x. You will need to enable
topic deletion (setting delete.topic.enable to true) on all brokers
first.
Make sure that, In kafka's config.poperties file 'delete.topic.enable' property should be true.
use below command
bin/kafka-topics.sh --delete --zookeeper localhost:2181 --topic test
Even if topic is not deleted then follow next steps.
open terminal zookeeper client mode:
A. stop zookeeper
B. rmr /broker/topics
C. check given topic using below command
/bin/kafka-topics.sh --zookeeper maxiq:2181 --list
If delete.topic.enable is set to false by default, the topics are not deleted on execution of --delete command (as displayed in the command response).
Topic test is marked for deletion.
Note: This will have no impact if delete.topic.enable is not set to true.
In order to overcome this, use steps mentioned by #Shawn
The best and easy to make delete.topic.enable=true is not modify the server.property file.
Because, if kafka restart from ambari, it will overwrite this file, delete.topic.enable =false again.
Only at ambari, click on kafak/config/advance kafka broker/delete.topic.enable =true, it will work.
I just found that out as now.
Delete topic in kafka
step 1 -->> cd /usr/lib/zookeeper/bin
step 2 -->> zkCli.sh -server 127.0.0.1:2181
step 3 -->>rmr /brokers/topics/topic_name
I need to delete the topic test in Apache Kafka 0.8.1.1.
As expressed in the documentation here, I have executed:
bin/kafka-topics.sh --zookeeper localhost:2181 --delete --topic test
However, this results in the following message:
Command must include exactly one action: --list, --describe, --create or --alter
How can I delete this topic?
Deleting topic isn't always working in 0.8.1.1
Deletion should be working in the next release, 0.8.2
kafka-topics.sh --delete --zookeeper localhost:2181 --topic your_topic_name
Topic your_topic_name is marked for deletion.
Note: This will have no impact if delete.topic.enable is not set to true.
You may also pass in a bootstrap server instead of zookeeper:
kafka-topics.sh --bootstrap-server kafka:9092 --delete --topic your_topic_name
Is it possible to delete a topic?
Jira KAFKA-1397
It seems that the deletion command was not officially documented in Kafka 0.8.1.x because of a known bug (https://issues.apache.org/jira/browse/KAFKA-1397).
Nevertheless, the command was still shipped in the code and can be executed as:
bin/kafka-run-class.sh kafka.admin.DeleteTopicCommand --zookeeper localhost:2181 --topic test
In the meantime, the bug got fixed and the deletion command is now officially available from Kafka 0.8.2.0 as:
bin/kafka-topics.sh --delete --zookeeper localhost:2181 --topic test
Add below line in ${kafka_home}/config/server.properties
delete.topic.enable=true
Restart the kafka server with new config:
${kafka_home}/bin/kafka-server-start.sh ~/kafka/config/server.properties
Delete the topics you wish to:
${kafka_home}/bin/kafka-topics.sh --delete --zookeeper localhost:2181 --topic daemon12
Andrea is correct. we can do it using command line.
And we still can program it, by
ZkClient zkClient = new ZkClient("localhost:2181", 10000);
zkClient.deleteRecursive(ZkUtils.getTopicPath("test2"));
Actually I do not recommend you delete topic on Kafka 0.8.1.1. I can delete this topic by this method, but if you check log for zookeeper, deletion mess it up.
Steps to Delete 1 or more Topics in Kafka
#To delete topics in kafka the delete option needs to be enabled in Kafka server.
1. Go to {kafka_home}/config/server.properties
2. Uncomment delete.topic.enable=true
#Delete one Topic in Kafka enter the following command
kafka-topics.sh --delete --zookeeper localhost:2181 --topic <your_topic_name>
#To Delete more than one topic from kafka
(good for testing purposes, where i created multiple topics & had to delete them for different scenarios)
Stop the Kafka Server and Zookeeper
go to server folder where the logs are stored (defined in their config files) and delete the kafkalogs and zookeeper folder manually
Restart the zookeeper and kafka server and try to list topics,
bin/kafka-topics.sh --list --zookeeper localhost:2181
if no topics are listed then the all topics have been deleted successfully.If topics are listed, then the delete was not successful. Try the above steps again or restart your computer.
You can delete a specific kafka topic (example: test) from zookeeper shell command (zookeeper-shell.sh). Use the below command to delete the topic
rmr {path of the topic}
example:
rmr /brokers/topics/test
This steps will delete all topics and data
Stop Kafka-server and Zookeeper-server
Remove the data directories of both services, by default on windows they are
C:/tmp/kafka-logs and C:/tmp/zookeeper.
then start Zookeeper-server and Kafka-server
The command:
bin/kafka-topics.sh --delete --zookeeper localhost:2181 --topic test
unfortunately only marks topic for deletion.
Deletion does not happen.
That makes troubles, while testing any scripts, which prepares Kafka configuration.
Connected threads:
Purge Kafka Queue
Is there a way to delete all the data from a topic or delete the topic before every run?
As mentioned in doc here
Topic deletion option is disabled by default. To enable it set the server config
delete.topic.enable=true
Kafka does not currently support reducing the number of partitions for a topic or changing the replication factor.
Make sure delete.topic.enable=true
Adding to above answers one has to delete the meta data associated with that topic in zookeeper consumer offset path.
bin/zookeeper-shell.sh zookeeperhost:port
rmr /consumers/<sample-consumer-1>/offsets/<deleted-topic>
Otherwise the lag will be negative in kafka-monitoring tools based on zookeeper.
First, you run this command to delete your topic:
$ bin/kafka-topics.sh --delete --bootstrap-server localhost:9092 --topic <topic_name>
List active topics to check delete completely:
$ bin/kafka-topics.sh --list --bootstrap-server localhost:9092
If you have issues deleting the topics, try to delete the topic using:
$KAFKA_HOME/bin/kafka-topics.sh --delete --zookeeper localhost:2181 --topic your_topic_name
command. Then in order to verify the deletion process, go to the kafka logs directory which normally is placed under /tmp/kafka-logs/, then delete the your_topic_name file via rm -rf your_topic_name command.
Remember to monitor the whole process via a kafka management tool like Kafka Tool.
The mentioned process above will remove the topics without kafka server restart.
There is actually a solution without touching those bin/kafka-*.sh: If you have installed kafdrop, then simply do:
url -XPOST http://your-kafdrop-domain/topic/THE-TOPIC-YOU-WANT-TO-DELETE/delete
bin/kafka-topics.sh --delete --zookeeper localhost:2181 --topic <topic-name>
Step 1: Make sure you are connected to zookeeper and Kafka running
Step 2: To delele the Kafka topic run kafka-topics script, add the port and --topic with name of your topic and --delete it just delete the topic with success.
# Delete the kafka topic
# it will delete the kafka topic
bin/kafka-topics.sh --zookeeper 127.0.0.1:2181 --topic name_of_topic --delete
This worked for me:
kafka-topics --delete --bootstrap-server localhost:9092 --topic user-commands
I am using Confluent CLI and run this on the directory I installed it.
Recent Kafka versions are about to remove the Zookeeper dependency. Therefore, you should instead reference the brokers (through --boostrap-server):
kafka-topics \
--bootstrap-server localhost:9092,localhost:9093,localhost:9094 \
--delete \
--topic topic_for_deletion
for confluent cloud use -
$ confluent kafka topic delete my_topic