I'm trying to list all topics from all group ids with total number of messages in each topic but couldn't find such command anywhere. Tried the below 2 commands but expecting the results of the below commands in a single command. Please help.
Command 1:(To list all topics in Kafka server)
bin/kafka-topics.sh --list --zookeeper localhost:2181
Command 2:(To know the count of total messages in a topic in Kafka server)
bin/kafka-run-class.sh kafka.tools.GetOffsetShell --broker-list 0.0.0.0:6667 --topic topicname --time -1
I have never heard of a kafka command that can do what you want. However, you can achieve this with basic shell scripting. This command should do the job :
./bin/kafka-topics.sh --zookeeper localhost:2181 --list | while read x; do ./bin/kafka-run-class.sh kafka.tools.GetOffsetShell --broker-list localhost:9092 --topic $x --time -1; done
Tested with kafka 0.10.2 running on Linux.
Related
I used kafka version 2.10-0.9.0.1, all information goes to zookeeper so i used the following command to checke kafka Lag:
./kafka-run-class.sh kafka.tools.ConsumerOffsetChecker --group
group-name --topic topicName --zookeeper zooIp:2181
Now we used latest kafka version 2.11-0.11.0.0, here this command not works.
How we see the kafka lag of topics.
The offsets are not stored anymore in Zookeeper but directly in Kafka now.
You can use the kafka-consumer-groups tool (kafka.admin.ConsumerGroupCommand) to retrieve the position/lag of a group.
bin/kafka-consumer-groups.sh --bootstrap-server KAFKA_HOST:KAFKA_PORT --describe --group GROUP_NAME
./bin/kafka-console-consumer.sh --zookeeper 127.0.0.1:2181 --topic test
./bin/kafka-console-producer.sh --broker-list 127.0.0.1:9092 --topic test
Why to consume kakfa needed zookeeper?
could I consume kafka from brokers ?
--zookeeper is an outdated option (for the old Consumer).
Use the newer --bootstrap-server option with kafka-console-consumer.
More info: http://kafka.apache.org/documentation/#newconsumerconfigs
I use Ubuntu server 16.04 to try using Kafka. For the command to start a producer and a consumer console I use the following.
producer console :
bin/kafka-console-producer.sh --broker-list localhost:9092 --topic hello-topic
consumer console :
bin/kafka-console-consumer.sh --zookeeper localhost:2181 --from-beginning --topic hello-topic
but the command above only subscribes to one topic. How can I subscribe to multiple topics?
First you should connect with the option bootstrap-server to the Kafka server itself not the zookeeper server.
For multiple topics you use the whitelist option. This will get interpreted as a regular expression and has to get quoted, see Kafka documentation.
So a correct command would be:
kafka-console-consumer.sh --bootstrap-server localhost:9092 --whitelist 'hello-topic|world-topic|another-topic'
Other expressions are also possible, like
kafka-console-consumer.sh --bootstrap-server localhost:9092 --whitelist '.*'
BE AWARE
For convenience we allow the use of ',' instead of '|' to specify a list of topics.
Does not work with Kafka 2.0, perhaps only when mirroring, which I did not try yet.
Hi The actual problem lies in the syntax, from version to verison it keeps on changing.
./kafka-console-consumer.sh --bootstrap-server localhost:9092 --from-beginning --topic muleesb
would work for you
Reason
For kafka versions above kafka_2.11-2.1.0: ./kafka-console-consumer.sh --bootstrap-server IP:PORT --from-beginning --topic TOPIC
For Kafka versions kafka_2.11_0.9.0.0 and below: /kafka-console-consumer.sh --zookeeper IP:PORT —-topic TOPIC --from-beginning --whitelist TOPIC
Exactly one of whitelist/blacklist/topic is required
As mentioned by Harald, use whitelist / blacklist option to include / exclude a set of topics for consumption.
sh kafka-console-consumer.sh --zookeeper localhost:2181 --from-beginning --whitelist Hello,World
To launch a Kafka consumer I have to run it with the following params.
kafka-console-consumer.bat --zookeeper localhost:2181 --topic MyTopic
Was wondering why it needs to have a zookeeper as a param, if I use the broker param will it not work similar to how the producer is launched. Consumer needs to be aware of the broker/cluster and not the zookeeper location.
.\kafka-console-producer.bat --broker --list localhost:9092 --topic MyTopic
This is on windows, am not sure how its in Unix flavors.
-Chandra
You're invoking the old Kafka Consumer. Old consumer requires zookeeper to co-ordinate from which broker to fetch data.
The new consumer don't requires the zookeeper parameter.
.\kafka-console-consumer.bat --bootstrap-server localhost:9092 --topic MyTopic --new-consumer
Optionally, you can add --from-beginning argument to read the old records in the topic.
Is this any bug with the Apache Kafka 0.9.0.0? I'm using the following command to list the topics and I get nothing even though the server was started with a topic being created!
Joes-MacBook-Pro:kafka_2.11-0.9.0.0 joe$ bin/kafka-topics.sh --list --zookeeper localhost:2181
Joes-MacBook-Pro:kafka_2.11-0.9.0.0 joe$
Here is the command that I used to start Apache Kafka:
bin/kafka-server-start.sh config/server.properties & bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic test_topic
It is getting harsh for me to just try out the basic stuff around the much hyped Apache Kafka. Here is one another example where I'm pretty much stuck!
Apache Kafka 0.9.0.0 Show all Topics with Partitions
I don't see the zookeeper started in your code.
Please start zookeeper and then kafka in the same order.
bin/zookeeper-server-start.sh config/zookeeper.properties &
bin/kafka-server-start.sh config/server.properties &
Try to list the topics with the below command.
bin/kafka-topics.sh --list --zookeeper localhost:2181
You can create the topic with the below command
bin/kafka-topics.sh --zookeeper localhost:2181 --create --topic test --partitions 1 --replication-factor 1
Once the topic is created, you get a message saying "test" created.
Now you can try list command to see your topic listed.
Note: The replication factor depends upon the number of brokers that you run on a single server.