I am trying consume the messages in kafka topic using below command. But it is not working.
bin/kafka-console-consumer.sh --bootstrap-server BROKER_URL --topic TOPIC_NAME --from-beginning
My colleague used the below query and it is working.
bin/kafka-console-consumer.sh --bootstrap-server BROKER_URL --topic TOPIC_NAME --offset 0 --partition 0
I'm not sure about the what is the difference between the 2 commands and why the second one is working but not first.
One reason why this could happen is if the consumer is part of a consumer group. In which case, an offset is established for the consumer group and --from-beginning will not go back to offset 0. However, in the command presented above, I don't see any mention of the (--group) consumer group in the command. Anyway, like other mentioned, the --from-beginning works for stand-alone consumers every time.
Related
Apache Kafka installed on Mac (Intel).
Single local producer and single local consumer.
1 topic with 3 partitions and 1 replication factor is created:
bin/kafka-topics.sh --bootstrap-server localhost:9092 --create --topic animal --partitions 3 --replication-factor 1
Producer code:
bin/kafka-console-producer.sh --bootstrap-server localhost:9092 --topic animal
Producer Messages:
>alligator
>crocodile
>tiger
When producing messages (manually via producer-console), all go into the same partition. Shouldn't they get distributed across partitions?
I've tried with 3 records (as above), but they get sent to 1 partition only. Checked within tmp/kafka-logs/topic-0/00**00.log
Other logs in topic- are empty.
I've tried with tens of records, but no luck.
I even increased the default partition configuration (num.partitions=3) within 'config/server.properties', but no luck.
I've also tried with different topics, but no luck.
Starting with kafka 2.4, the default partitioner was changed from round-robin to sticky, which will stick to the same partition (pun intended) for an entire batch.
With my kafka version, the kafka-console-producer uses a default batch size of 16384, so once you produce enough messages to fill that buffer, the partition will change.
If a producer, produces messages with the same key then it’s guaranteed to be produced on the same partition. so in your case if you want it to be consumed by different partitions than make sure to publish it with different keys.
You will need to set below property.
--property parse.key=true
See below command to produce record with key.
kafka-console-producer --broker-list 127.0.0.1:9092 --topic first_topic --property parse.key=true --property key.separator=,
> key1,value1
> key2,value2
My input is 1:45$ and we are processing the message and next I am updating the 1:null.
I do see still the 1:45$ in the topic along with the 1:null (I can see both the messages)
I want output to be 1:null in the same topic.
I have used this code:
kafka-topics --create --zookeeper zookeeper:2181 --topic latest- product-price --replication-factor 1 --partitions 1 --config "cleanup.policy=compact" --config "delete.retention.ms=100" --config "segment.ms=100" --config "min.cleanable.dirty.ratio=0.01"
kafka-console-producer --broker-list localhost:9092 --topic latest- product-price --property parse.key=true --property key.separator=::
1::45$
1::null
kafka-console-consumer --bootstrap-server localhost:9092 --topic latest-product-price --property print.key=true --property key.separator=:: --from-beginning
But I do not find any compaction in my case and need some inputs to make the value as 1::null
Compaction in Kafka is not immediate. If you send two messages with the same key to a compacted topic, and you have a live consumer on that topic, that consumer will see both messages come through.
Periodically, there's a background cleaner thread that goes looking for duplicate keys in compacted topics, and removes the overwritten records, so that a consumer that pulls down the data after that log cleaner has run, will only see the last change/update for a particular key. So, topic compaction seems to be better suited for consumers that run periodically, not ones that are active 100% of the time.
One thing that you can tune how often this background log cleaner thread runs, to maybe run those consumers more often. Look for the log.cleaner configuration parameters in the Kafka documentation: https://kafka.apache.org/documentation/#brokerconfigs
There's a good explanation on how Kafka log compaction works at this link:
https://medium.com/swlh/introduction-to-topic-log-compaction-in-apache-kafka-3e4d4afd2262
I'm trying to list all topics from all group ids with total number of messages in each topic but couldn't find such command anywhere. Tried the below 2 commands but expecting the results of the below commands in a single command. Please help.
Command 1:(To list all topics in Kafka server)
bin/kafka-topics.sh --list --zookeeper localhost:2181
Command 2:(To know the count of total messages in a topic in Kafka server)
bin/kafka-run-class.sh kafka.tools.GetOffsetShell --broker-list 0.0.0.0:6667 --topic topicname --time -1
I have never heard of a kafka command that can do what you want. However, you can achieve this with basic shell scripting. This command should do the job :
./bin/kafka-topics.sh --zookeeper localhost:2181 --list | while read x; do ./bin/kafka-run-class.sh kafka.tools.GetOffsetShell --broker-list localhost:9092 --topic $x --time -1; done
Tested with kafka 0.10.2 running on Linux.
I use Ubuntu server 16.04 to try using Kafka. For the command to start a producer and a consumer console I use the following.
producer console :
bin/kafka-console-producer.sh --broker-list localhost:9092 --topic hello-topic
consumer console :
bin/kafka-console-consumer.sh --zookeeper localhost:2181 --from-beginning --topic hello-topic
but the command above only subscribes to one topic. How can I subscribe to multiple topics?
First you should connect with the option bootstrap-server to the Kafka server itself not the zookeeper server.
For multiple topics you use the whitelist option. This will get interpreted as a regular expression and has to get quoted, see Kafka documentation.
So a correct command would be:
kafka-console-consumer.sh --bootstrap-server localhost:9092 --whitelist 'hello-topic|world-topic|another-topic'
Other expressions are also possible, like
kafka-console-consumer.sh --bootstrap-server localhost:9092 --whitelist '.*'
BE AWARE
For convenience we allow the use of ',' instead of '|' to specify a list of topics.
Does not work with Kafka 2.0, perhaps only when mirroring, which I did not try yet.
Hi The actual problem lies in the syntax, from version to verison it keeps on changing.
./kafka-console-consumer.sh --bootstrap-server localhost:9092 --from-beginning --topic muleesb
would work for you
Reason
For kafka versions above kafka_2.11-2.1.0: ./kafka-console-consumer.sh --bootstrap-server IP:PORT --from-beginning --topic TOPIC
For Kafka versions kafka_2.11_0.9.0.0 and below: /kafka-console-consumer.sh --zookeeper IP:PORT —-topic TOPIC --from-beginning --whitelist TOPIC
Exactly one of whitelist/blacklist/topic is required
As mentioned by Harald, use whitelist / blacklist option to include / exclude a set of topics for consumption.
sh kafka-console-consumer.sh --zookeeper localhost:2181 --from-beginning --whitelist Hello,World
To launch a Kafka consumer I have to run it with the following params.
kafka-console-consumer.bat --zookeeper localhost:2181 --topic MyTopic
Was wondering why it needs to have a zookeeper as a param, if I use the broker param will it not work similar to how the producer is launched. Consumer needs to be aware of the broker/cluster and not the zookeeper location.
.\kafka-console-producer.bat --broker --list localhost:9092 --topic MyTopic
This is on windows, am not sure how its in Unix flavors.
-Chandra
You're invoking the old Kafka Consumer. Old consumer requires zookeeper to co-ordinate from which broker to fetch data.
The new consumer don't requires the zookeeper parameter.
.\kafka-console-consumer.bat --bootstrap-server localhost:9092 --topic MyTopic --new-consumer
Optionally, you can add --from-beginning argument to read the old records in the topic.