We have a working topic-consumer setup in kafka. While trying to create the same in another environment, the consumer does not start reading from the topic. I've tried restarting, deleting, renaming and many things but none of them worked.
When I describe the consumer group with the command kafka-consumer-groups --describe I got this result:
GROUP TOPIC PARTITION CURRENT-OFFSET LOG-END-OFFSET LAG
G__BRC_SENDSMS BRC_SENDSMS 0 - 31 -
Mind how current offset is not zero. Since it is not zero, there is also no lag too. Is this normal? What should I do?
create the same in another environment
Only one consumer in group.id=G__BRC_SENDSMS can read from BRC_SENDSMS, partition 0 at the same time. Change the group id, and it should start to read data. Or add more partitions to the topic, and you should see the group rebalance and distribute messages between all consumer instances. If neither work, then check networking configurations in this "other environment", such as the Kafka advertised.listeners.
If there is no current offset, then has your consumer actually committed yet?
Related
Hope you are having good day.
I have an issue with kafka consumers on kubernetes. I am running 3 replicas inside a consumer group
I have a topic with 3 partitions and 3 brokers with offsets replication factor set to 3. My offset in consumer group is set to earliest.
When I start the consumer group, all are working fine with each consumer replica taking different partition and processing the data.
Issue: When by any means if a consumer replica inside the consumer group "abc-consumer-group" restarts OR if a broker(leader) restarts, it is not resuming from the point where it stopped. It states that I am up to date and no messages I have to process.
Any suggestions please where to look at?
Tried increasing rebalance, heartbeat, session timeout on broker level, no luck.
And yes whenever any new consumer is added or removed to the consumer group rebalacing is taken care by kafka. I do see it happening but still not consumers are not resuming messages. It states nothing to process.
Kafka machines are installed as part of hortonworks packages , kafka version is 0.1X
We run the deeg_data applications, consuming data from kafka topics
On last days we saw that our application – deeg_data are failed and we start to find the root cause
On kafka cluster we see the following behavior
/usr/hdp/current/kafka-broker/bin/kafka-consumer-groups.sh --group deeg_data --describe --bootstrap-server kafka1:6667
To enable GC log rotation, use -Xloggc:<filename> -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=<num_of_files>
where num_of_file > 0
GC log rotation is turned off
Consumer group ‘deeg_data’ is rebalancing
from kafka side kafka cluster is healthy and all topics are balanced and all kafka brokers are up and signed correctly to zookeeper
After some time ( couple hours ) , we run again the following , but without the errors about - Consumer group ‘deeg_data’ is rebalancing
And we get the following correctly results
/usr/hdp/current/kafka-broker/bin/kafka-consumer-groups.sh --group deeg_data --describe --bootstrap-server kafka1:6667
To enable GC log rotation, use -Xloggc:<filename> -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=<num_of_files>
where num_of_file > 0
GC log rotation is turned off
GROUP TOPIC PARTITION CURRENT-OFFSET LOG-END-OFFSET LAG OWNER
deeg_data pot.sdr.proccess 0 6397256247 6403318505 6062258 consumer-1_/10.3.6.237
deeg_data pot.sdr.proccess 1 6397329465 6403390955 6061490 consumer-1_/10.3.6.237
deeg_data pot.sdr.proccess 2 6397314633 6403375153 6060520 consumer-1_/10.3.6.237
deeg_data pot.sdr.proccess 3 6397258695 6403320788 6062093 consumer-1_/10.3.6.237
deeg_data pot.sdr.proccess 4 6397316230 6403378448 6062218 consumer-1_/10.3.6.237
deeg_data pot.sdr.proccess 5 6397325820 6403388053 6062233 consumer-1_/10.3.6.237.
.
.
.
So we want to understand why we get:
Consumer group ‘deeg_data’ is rebalancing
What is the reason for above state , and why we get rebalancing
we also have good post (https://www.confluent.io/blog/kafka-consumer-multi-threaded-messaging/)
Group rebalancing
Consumer group rebalancing is triggered when partitions need to be reassigned among consumers in the consumer group: A new consumer joins the group; an existing consumer leaves the group; an existing consumer changes subscription; or partitions are added to one of the subscribed topics.
Rebalancing is orchestrated by the group coordinator and it involves communication with all consumers in the group. To dive deeper into the consumer group rebalance protocol, see Everything You Always Wanted to Know About Kafka’s Rebalance Protocol But Were Afraid to Ask by Matthias J. Sax from Kafka Summit and The Magical Rebalance Protocol of Apache Kafka by Gwen Shapira.
Regarding consumer client code, some of the partitions assigned to it might be revoked during a rebalance. In the older version of the rebalancing protocol, called eager rebalancing, all partitions assigned to a consumer are revoked, even if they are going to be assigned to the same consumer again. With the newer protocol version, incremental cooperative rebalancing, only partitions that are reassigned to another consumer will be revoked. You can learn more about the new rebalancing protocol in this blog post by Konstantine Karantasis and this blog post by Sophie Blee-Goldman.
Regardless of protocol version, when a partition is about to be revoked, the consumer has to make sure that record processing is finished and the offset is committed for that partition before informing the group coordinator that the partition can be safely reassigned.
With automatic offset commit enabled in the thread per consumer model, you don’t have to worry about group rebalancing. Everything is done by the poll method automatically. However, if you disable automatic offset commit and commit manually, it’s your responsibility to commit offsets before the join group request is sent. You can do this in two ways:
Note - also good post is from you-tube - https://www.youtube.com/watch?v=QaeXDh12EhE
Note - good stack-overflow post - Kafka Consumer Rebalancing takes too long
Note - from ENV side , since our zookeeper servers are installed on VM machines and VM machine are using non ssd disks , and regarding to swap consuming , then I think we need to consider also the post - https://community.cloudera.com/t5/Community-Articles/Zookeeper-Sizing-and-Placement/ta-p/247885
The rebalance in Kafka is a protocol and is used by various components (Kafka connect, Kafka streams, Schema registry etc.) for various purposes.
In the most simplest form, a rebalance is triggered whenever there is any change in the metadata.
Now, the word metadata can have many meanings - for example:
In the case of a topic, it's metadata could be the topic partitions and/or replicas and where (which broker) they are stored
In the case of a consumer group, it could be the number of consumers that are a part of the group and the partitions they are consuming the messages from etc.
The above examples are by no means exhaustive i.e. there is more metadata for topics and consumer groups but I wouldn't go into more details here.
So, if there is any change in:
The number of partitions or replicas of a topic such as addition, removal or unavailability
The number of consumers in a consumer group such as addition or removal
Other similar changes...
A rebalance will be triggered. In the case of consumer group rebalancing, consumer applications need to be robust enough to cater for such scenarios.
So rebalances are a feature. However, in your case it appears that it is happening very frequently so you may need to investigate the logs on your client application and the cluster.
Following are a couple of references that might help:
Rebalance protocol - A very good article on medium on this subject
Consumer rebalancing - Another post on SO focusing on consumer rebalancing
I'm running this kafka command:
/opt/kafka_2.11/bin/kafka-consumer-groups.sh --bootstrap-server xxxxx:9092 \
--describe --group flink-cg
Result is like this:
TOPIC PARTITION CURRENT-OFFSET LOG-END-OFFSET LAG
my_topic 0 481239571 484028280 2788709
The offset keep stuck although my flink is running and have no error in the log file.
How to check if the number of my offset is correct? I'm afraid my current-offset is having wrong number so the value is stuck.
The sole fact that Flink Job is running doesn't necesarily mean that the offset should change. This depends on the configuration of Your job, but by default the offset is only commited on checkpoint, so the first thing to check is if Your job is properly checkpointing (maybe You have configured long time between checkpoints).
If it is or if You have enabled enable.auto.commit then You should check if there is possibly a backpressure for some operators that may be causing problems with reading of records.
It would be easier to tell if You could provide more info about configuration and the job itself.
My kafka topic has 10 records/messages in total and 2 partitions having 5 messages each. My consumer group has 2 consumers and each of the consumer has already read 5 messages from their assigned partition respectively. Now, I want to re-process/read messages from my topic from start/beginning (offset 0).
I stopped my kafka consumers and ran following command to reset consumer group offset to 0.
./kafka-consumer-groups.sh --group cg1 --reset-offsets --to-offset 0 --topic t1 --execute --bootstrap-server "..."
My expectation was that once I restart my kafka consumers they will start reading records from offset 0 i.e. beginning, but that didn't happen and they polled from their last position i.e. offset 5. Why is that so? I then have to make each of my consumers, explicitly seek to offset 0 (beginning) to re-process/read records from the beginning. And in later tests cycles, I didn't even ran above command to reset offset for kafka consumer group.
My question is, if I have to make my consumers explicitly seek to beginning to make them re-process/read messages again, then what's the purpose of resetting the offset of kafka consumer group?
Handling Kafka consumer offsets is bit more tricky. Consumer program uses auto.offset.reset config only when consumer group used does not have a valid offset committed in an internal Kafka topic.(Other supported offset storage is Zookeeper but internal Kafka topic is used as offset storage in latest Kafka versions).
Consider below scenarios:
Consumer in consumer group named 'group1' has consumed 5 messages from topic 'testtopic' and offset details are committed to internal Kafka topic- Next time when the consumer starts, it will not use 'auto.offset.reset' config. Instead it will fetch the stored offset from storage and will continue fetch messages from the retrieved offset.
Consumer in consumer group named 'group2' is started as a new consumer to fetch messages from 'testtopic'. This is new group and there is no offset details available in internal Kafka topic- 'auto.offset.reset' config is used now to decide where to start; either from beginning of the topic or from latest(only new messages will be consumed).
The issue as per your question is that the command to reset offset not working, you have to manually seek to beginning and start consumer.
kafka-consumer-groups.sh --bootstrap-server <kafka_host:port> --group <group_id> [--topic <topic_name> or --all-topics] --reset-offsets [--to-earliest or --to-offset <offset>] --execute
There are three possibilities for reset command not working.
The log retention period is smaller and offset you are trying to reset is no longer available
A consumer instance in the consumer group is running. In both cases, reset offset command may not work.
Kafka version is <0.11. Reset offset API is available only from Kafka 0.11
From your question, first and third case is unlikely. Please check for second case. Stop any consumer instance running and then try resetting offsets.
Below command can be used to check whether a consumer group has active consumer instance.
kafka-consumer-groups.sh --bootstrap-server <kafka_host:port> --group <group_id> --describe
Sample output:
Consumer group 'group1' has no active members.
TOPIC PARTITION CURRENT-OFFSET LOG-END-OFFSET LAG CONSUMER-ID HOST CLIENT-ID
intro 0 0 99 99
I'm new to Kafka and I've tried the Kafka-Python package.
I managed to setup a simple producer and consumer, which can send and receive messages. In this case the consumer is without using consumer group as below:
consumer = KafkaConsumer(queue_name, bootstrap_servers='kafka:9092')
However, when I started to use the group_id as below, it stops receiving any messages:
consumer = KafkaConsumer(bootstrap_servers='kafka:9092', auto_offset_reset='earliest', group_id='my-group')
consumer.subscribe([queue_name])
For comparison, I've also tried the confluent-kafka-python package, where I have the following consumer code, which also doesn't work:
consumer = Consumer({
'bootstrap.servers': 'kafka:9092',
'group.id': 'mygroup',
'auto.offset.reset': 'earliest'
})
consumer.subscribe([queue_name])
Also running ./kafka-consumer-groups.sh --bootstrap-server localhost:9092 --list gives empty result.
Any configuration I'm missing here?
By default, the consumer starts consuming from the last committed offsets which is probably the last offset in your case.
The auto.offset.reset only applies when there are no committed offsets. As by default the consumer automatically commits offsets, it usually only applies the first time your run it (there are a few other cases but they don't matter in this example).
So to see messages flowing, you need to either start producing once your consumer is running or use a different group name to allow auto.offset.reset to apply.