Kafka 2.12 Consumer Group (with command prompt) - Second consumer can't seem to receive the message? - apache-kafka

I am starting out with Apache Kafka 2.12 and I'm running into problems with practicing the Consumer Group while using the command prompt: When two consumers are created and set to subscribe to a topic in a group, only one consumer seems to receive all the message, the other consumer receive nothing.
So, after running the Zookeeper and Kafka Servers (imagine that we're starting fresh), this is what I typed on the 3rd command prompt to create the topic:
kafka-topics.bat --create --bootstrap-server localhost:9092 --replication-factor 1 --partitions 2 --topic test-TOPIC
Next, I opened the 4th command prompt and entered this:
kafka-console-consumer.bat --bootstrap-server localhost:9092 --topic test-TOPIC --group learning
Finally, I opened the 5th command prompt and entered this:
kafka-console-consumer.bat --bootstrap-server localhost:9092 --topic test-TOPIC --group learning
Now I go back to the 3rd command prompt and started entering these messages (separated with a new line):
One
Two
Three
Four
Five
Six
But only the 5th command prompt receive the messages. The 4th command prompt gets nothing. From the tutorial I see in the net (I could reveal to you, but I'm not sure if it will be ethical, and it's in Indonesian, so... not yet), it worked as planned, yielding this result:
5th command prompt gets:
One
Three
Five
4th command prompt gets:
Two
Four
Six
So... where did I go wrong in this? Help will be appreciated. Thank you!

Related

Test if kafka ready, topics are reachable from cli

Was looking around and couldn't find anything other than kafka-topic --list.
I'm running Kafka in a K8s environment and have an init container that creates a couple of topics. I want my main container to start only when the topics are created and the topics are "subscribable" to. kafka-topic --list I believe is just reaching zookeeper as I can see my pod is showing error messages about the topic.
I did try kafka-console-consumer but even if the topic is not present it doesn't exit with status 1. It does exit with status one if the bootstrap server is not reachable. I'm looking for a solution similar to below
kafka-console-consumer --bootstrap-server correct-bootstrap-server:9092 --topic correct-topic --timeout-ms 100
exits with 0 ( this one works )
kafka-console-consumer --bootstrap-server wrong-bootstrap-server:9092 --topic wrong-topic --timeout-ms 100
exits with a non zero exit code (this one works too).
kafka-console-consumer --bootstrap-server correct-bootstrap-server:9092 --topic wrong-topic --timeout-ms 100
exits with a non zero exit code ( this one doesn't work as it exits with code 0)
Thanks.
It is not totally trivial to make sure from CLI if a Kafka topic is "ready". Many things can go wrong.
We had the same issues, and the current approach we are taking involves several call to the kafka-topic CLI
We make sure the topic exists with kafka-topics.sh --describe --topic FOO
Check that all partitions have a leader kafka-topics.sh --describe --topic FOO --unavailable-partitions (output should be empty)
Check that all partitions are fully replicated kafka-topics.sh --describe --topic FOO --under-replicated-partitions (output should be empty)
That still doesn't make it 100% certain that the topic is "ready", but it works for us
kafka-topics can list under-replicated, offline and under-min-isr partitions.
The best bet is to check that your topic is not under-replicated. If it isn't, it should be ready.

--from-beginning parameter is not working in kafka-console-consumer.sh but --offset 0 --partition 0 works

I am trying consume the messages in kafka topic using below command. But it is not working.
bin/kafka-console-consumer.sh --bootstrap-server BROKER_URL --topic TOPIC_NAME --from-beginning
My colleague used the below query and it is working.
bin/kafka-console-consumer.sh --bootstrap-server BROKER_URL --topic TOPIC_NAME --offset 0 --partition 0
I'm not sure about the what is the difference between the 2 commands and why the second one is working but not first.
One reason why this could happen is if the consumer is part of a consumer group. In which case, an offset is established for the consumer group and --from-beginning will not go back to offset 0. However, in the command presented above, I don't see any mention of the (--group) consumer group in the command. Anyway, like other mentioned, the --from-beginning works for stand-alone consumers every time.

Kafka 10.2 new consumer vs old consumer

I've spent some hours to figure out what was going on but didn't manage to find the solution.
Here is my set up on a single machine:
1 zookeeper running
3 broker running (on port 9092/9093/9094)
1 topic with 3 partitions and 3 replications (each partition are properly assigned between brokers)
I'm using kafka console producer to insert messages. If i check the replication offset (cat replication-offset-checkpoint), I see that my messages are properly ingested by Kafka.
Now I use the kafka console consumer (new):
sudo bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --from-beginning --topic testTopicPartitionned2
I dont see anything consumed. I tried to delete my logs folder (/tmp/kafka-logs-[1,2,3]), create new topics, still nothing.
However when I use the old kafka consumer:
sudo bin/kafka-console-consumer.sh --zookeeper localhost:2181 --topic testTopicPartitionned2
I can see my messages.
Am I missing something big here to make this new consumer work ?
Thanks in advance.
Check to see what setting the consumer is using for auto.offset.reset property
This will affect what a consumer group without a previously committed offset will do in terms of setting where to start reading messages from a partition.
Check the Kafka docs for more on this.
Try providing all your brokers to --bootstrap-server argument to see if you notice any differnce:
sudo bin/kafka-console-consumer.sh --bootstrap-server localhost:9092,localhost:9093,localhost:9094 --from-beginning --topic testTopicPartitionned2
Also, your topic name is rather long. I assume you've already made sure you provide the correct topic name.

Apache Kafka console Producer-Consumer example

I have installed Apache Kafka to a Windows system and have tried a console Producer-Consumer example. In case I add new consumers to the topic, only the messages after adding the consumer are printed to the console. Is there any way to get all the messages of the particular topic?
You need to add --from-beginning flag to your console consumer command to get all messages.
Example command:
bin/kafka-console-consumer.sh --zookeeper localhost:2181 --topic test --from-beginning

Kafka both consumer gets same message withing same group name

I have single machine set up for kafka and wanted to run que based kaffa. I have set up group name in consumer.properties like this.
consumer group id
group.id=test-consumer-group
and when I run 2 consumer both for same topic both gets the message. Is there anything else I need to do other than this?
sudo bin/kafka-console-consumer.sh config/consumer.properties --zookeeper localhost:2181 --topic 219Topic --from-beginning
I am using command prompt only no client currently for testing.
Thanks,