I use Ubuntu server 16.04 to try using Kafka. For the command to start a producer and a consumer console I use the following.
producer console :
bin/kafka-console-producer.sh --broker-list localhost:9092 --topic hello-topic
consumer console :
bin/kafka-console-consumer.sh --zookeeper localhost:2181 --from-beginning --topic hello-topic
but the command above only subscribes to one topic. How can I subscribe to multiple topics?
First you should connect with the option bootstrap-server to the Kafka server itself not the zookeeper server.
For multiple topics you use the whitelist option. This will get interpreted as a regular expression and has to get quoted, see Kafka documentation.
So a correct command would be:
kafka-console-consumer.sh --bootstrap-server localhost:9092 --whitelist 'hello-topic|world-topic|another-topic'
Other expressions are also possible, like
kafka-console-consumer.sh --bootstrap-server localhost:9092 --whitelist '.*'
BE AWARE
For convenience we allow the use of ',' instead of '|' to specify a list of topics.
Does not work with Kafka 2.0, perhaps only when mirroring, which I did not try yet.
Hi The actual problem lies in the syntax, from version to verison it keeps on changing.
./kafka-console-consumer.sh --bootstrap-server localhost:9092 --from-beginning --topic muleesb
would work for you
Reason
For kafka versions above kafka_2.11-2.1.0: ./kafka-console-consumer.sh --bootstrap-server IP:PORT --from-beginning --topic TOPIC
For Kafka versions kafka_2.11_0.9.0.0 and below: /kafka-console-consumer.sh --zookeeper IP:PORT —-topic TOPIC --from-beginning --whitelist TOPIC
Exactly one of whitelist/blacklist/topic is required
As mentioned by Harald, use whitelist / blacklist option to include / exclude a set of topics for consumption.
sh kafka-console-consumer.sh --zookeeper localhost:2181 --from-beginning --whitelist Hello,World
Related
I used kafka version 2.10-0.9.0.1, all information goes to zookeeper so i used the following command to checke kafka Lag:
./kafka-run-class.sh kafka.tools.ConsumerOffsetChecker --group
group-name --topic topicName --zookeeper zooIp:2181
Now we used latest kafka version 2.11-0.11.0.0, here this command not works.
How we see the kafka lag of topics.
The offsets are not stored anymore in Zookeeper but directly in Kafka now.
You can use the kafka-consumer-groups tool (kafka.admin.ConsumerGroupCommand) to retrieve the position/lag of a group.
bin/kafka-consumer-groups.sh --bootstrap-server KAFKA_HOST:KAFKA_PORT --describe --group GROUP_NAME
I'm trying to list all topics from all group ids with total number of messages in each topic but couldn't find such command anywhere. Tried the below 2 commands but expecting the results of the below commands in a single command. Please help.
Command 1:(To list all topics in Kafka server)
bin/kafka-topics.sh --list --zookeeper localhost:2181
Command 2:(To know the count of total messages in a topic in Kafka server)
bin/kafka-run-class.sh kafka.tools.GetOffsetShell --broker-list 0.0.0.0:6667 --topic topicname --time -1
I have never heard of a kafka command that can do what you want. However, you can achieve this with basic shell scripting. This command should do the job :
./bin/kafka-topics.sh --zookeeper localhost:2181 --list | while read x; do ./bin/kafka-run-class.sh kafka.tools.GetOffsetShell --broker-list localhost:9092 --topic $x --time -1; done
Tested with kafka 0.10.2 running on Linux.
./bin/kafka-console-consumer.sh --zookeeper 127.0.0.1:2181 --topic test
./bin/kafka-console-producer.sh --broker-list 127.0.0.1:9092 --topic test
Why to consume kakfa needed zookeeper?
could I consume kafka from brokers ?
--zookeeper is an outdated option (for the old Consumer).
Use the newer --bootstrap-server option with kafka-console-consumer.
More info: http://kafka.apache.org/documentation/#newconsumerconfigs
I am playing aroung Kafka, when I use
bin/kafka-console-consumer.sh --zookeeper localhost:2181 --from-beginning --topic test
Kafka will automatically create a consumer group. I am wondering how to get the consumer group name?
You should use kafka-consumer-groups.sh. The following command will list you all consumer groups.
bin/kafka-consumer-groups.sh --list --bootstrap-server localhost:9092
Note: This will only show information about consumers that use the Java Consumer API (non-Zookeeper-based consumers).
To launch a Kafka consumer I have to run it with the following params.
kafka-console-consumer.bat --zookeeper localhost:2181 --topic MyTopic
Was wondering why it needs to have a zookeeper as a param, if I use the broker param will it not work similar to how the producer is launched. Consumer needs to be aware of the broker/cluster and not the zookeeper location.
.\kafka-console-producer.bat --broker --list localhost:9092 --topic MyTopic
This is on windows, am not sure how its in Unix flavors.
-Chandra
You're invoking the old Kafka Consumer. Old consumer requires zookeeper to co-ordinate from which broker to fetch data.
The new consumer don't requires the zookeeper parameter.
.\kafka-console-consumer.bat --bootstrap-server localhost:9092 --topic MyTopic --new-consumer
Optionally, you can add --from-beginning argument to read the old records in the topic.