I am using Kafka: 2.11-1.0.1. The application contains consumers with concurrency=5 for the topic 'X' with partitions=5.
When the application is restarted and the message is published on topic 'X' before partition assignment,
5 consumers of topic 'X' find group coordinator and send the join group request to the group coordinator. It is expected to get a response from the group coordinator but no response is received.
I have Checked Kafka server logs but I could not find relevant logs found DEBUG log level.
When I run describe consumer group command, the following observation is made:
consumer group is rebalancing
Old consumers with some lag
New consumers with some random names. As time goes new consumer numbers are increasing.
New messages are published on the topic 'X', but it is not being received by the consumers.
heartbeat and session.time.out is set as default.
This problem occurs if the message is published before the partition assignment for the topic 'X' and its consumers.
My doubt is: Why rebalancing is not getting complete so that new consumer starts consuming the newly produced message?
Application have below consumers in consumer group
Consumer A listens to Topic1. Topic1 have 1 partition.
max.poll.interval.time.ms=4 hours for this consumer.
Consumer B listens to Topic2. Topic2 have 5 partiition.
Consumer B concurrency=5.
max.poll.interval.time.ms=1 hour for this consumer.
What is happening on application restart and if one of the topic has already published message
When the application restarts one instance of consumer (consumerA1)
created and it subscribes to topic1. ConsumerA1 finds Group Coordinate (GC) and sends join group request.
ConsumerA1 gets response from GC and becomes leader.Till this step not other consumer has initialized.
ConsumerA1 assigns partitions and sends SyncGroup request to GC. New
assignment generation happens. In this way first rebalance completed.
Message on topic1 is already published , consumerA1 fetches this message
and starts processing. Processing of completion of this message take
significant time. (Say 2 hours)
Now 5 consumer instances initialize one by one and all of them subscribes to topic2. These consumer finds GC and sends join group request.
but GC does not respond them.
When consumerA1 sends heartbeat to GC , GC responds that rebalancing is going on but consumerA1 does not revoke partition since it is processing the message.
According to Rebalancing protocol (Nice article on rebalancing) , GC waits till all consumer sends join group request. In this case , GC waits to get join group request from consumerA1. Maximum wait is max.poll.interval.time.ms i.e. 4 hours in this case.
Root Cause:
Group Coordinator did not wait to all consumers initialization after application restart therefore first unnecessary rebalance happened therefore consumerA1 fetched the message from partition and started processing it.
Solution:
To avoid such unnecessary initial rebalance , kafka provides one configuration in which Group Coordinator waits till consumer join new consumer group. Documentation
group.initial.rebalance.delay.ms
Checked my kafka server.properties , it was set to 0.
Tries with default i.e. 3 seconds.
Initial rebalance avoided , GC wait 3 seconds on application restart and in this time all other consumers initialized.All consumers sent join group request , as all GC got request from all consumers. GC responded without any delay , rebalancing procedded and completed successfully.
Related
How long does Partition Rebalancing take when a new Consumer joins the group?
What factors can affect this?
In my understanding, these factors play role:
Consumers needs to finish processing the data they polled last time.
Coordinator waits for Consumer to send JoinGroup request - for how long?
Consumers send SyncGroup request (is there a delay between receiving JoinGroup response and sending SyncGroup request?)
In a normal situation, assuming that Consumers process data instantly, how long should one expect to Partition Rebalancing to take place?
Coordinator waits for at most rebalance.timeout.ms (1 minute by default). If Consumer does not join during this time it is considered dead. But if all consumers re-join before this, the Coordinator will not wait further (my assumption).
Usual Consumers send Sync group as soon as they received the JoinGroup response. Leader needs more time (to perform partition assignment logic).
Source:
https://www.confluent.io/online-talks/everything-you-always-wanted-to-know-about-kafkas-rebalance-protocol-but-were-afraid-to-ask-on-demand/
I've below consumer settings.
auto.offset.reset=earliest
enable.auto.commit=true (default value)
session.timeout.ms=10000 (default value)
max.poll.interval.ms= 300000 (default value)
With the above configuration, let's say i have five messages( m1, m2, m3, m4 and m5) in a topic A (with only 1 partition). Now I've consumer subscribed to this topic and was able to process first two messages (m1 and m2) without any issues and committed offset.
Now, Let us say the consumer got the third message m3 and trying to process it and it took 300100 ms for processing because of some network latency. Now, as per my understanding, the offset commit will not happen because the record processing took more than max.poll.interval.ms and hence the consumer would be considered as dead and removed from the group.
Now I've two questions
What happens to the message m3? I mean, would it be picked in the next poll because it's offset was not committed
What happens to the other messages m4 and m5?
Expiring max.poll.inteval.ms without calling poll() is one of the reasons of rebalance. When rebalance starts in a consumer group, all the consumers in this consumer group are revoked. (removed from consumer list) During rebalance Kafka waits all healthy consumers to send joinGroupRequest by calling poll() until rebalance timeout (rebalance timeout equals to max.poll.interval.ms). Upon completion of joinGroupRequests of healthy consumers or rebalance timeout, Kafka assign partitions to consumers that sends joinGroupRequests.
In your case:
What happens to the message m3? I mean, would it be picked in the next
poll because it's offset was not committed
Answer: Its process continues even after your consumer is revoked unless you have a logic to interrupt process thread in case of revoke. So all the messages returned from previous poll are processed. But offset cannot be committed. If this partition is assigned to another consumer at the result of the rebalance, then new consumer will get same messages starts from M3. So message(s) will be processed twice. When first consumer sends poll request again, that means joinGroupRequests and again rebalance will be triggered.
What happens to the other messages m4 and m5?
Answer: If these messages are returned from poll() as well as m3, then result will be the same. They will be processed, but cannot be committed by the old consumer. New consumer will process messages and commit offset.
Topic Name : testTopic
Total # of message at topic : 1
Partition : 8
Consumer group Name : Consumer1
Consumer language : Java with partition listener impl
Infrastructure : having 4 jvms running parallel (which means, 4 consumers are running with same group name)
problem : when i start my first consumer Lister call back methods are called and partition assignments are done.. this consumer started processing my messages.
take an example, this consumer is holding a message MSG-1 and my processor is processing the message(I intentionally put 20 mille second as thread wait). so, did not committed MSG-1 back to topic with offset.
properties of the consumer
session.timeout.ms = 15 mille seconds.
In the mean time, consumer 2 started,
this consumer started, assigned partition(properly call back methods are called) and did not consumed the messages because, these 2 messages are hold by consumer 1 .
now, from consumer heart beat interval exceed and broker considered that consumer-1 is dead and reassigned the partition at consumer-2(all)
now call back method called at consumer-2 (assigned & revoked). In the mean time, my session timeout is expired and msg-1 and msg-2 put back to topic and picked up consumer 2.
now, i have processed msg-1 && msg-2 two times.... one time from consumer-1 and consumer-2
My problem here is,
Consumer-1 did not get called with partition revoked call back method ?
after my Thread sleep completes (from consumer -1 ) he is trying to commit the offset with partition.... we are getting partition re-assignment is done .. u cant commit. this is correct, but how can i get the call back method form consumer-1....
-Naresh.
Consumer-1 did not get called with partition revoked call back method ?
The consumer can only get a partition revoked call back, if it participates in a rebalance. However, because it timed out and did drop out of the group, it does not participate in a rebalance and the broker does not send a any information to the consumer. Therefore, the consumer does not know that the partitions is revoked (and hence, no call back).
after my Thread sleep completes (from consumer -1 ) he is trying to commit the offset with partition.... we are getting partition re-assignment is done .. u cant commit. this is correct
Not sure what you mean by we are getting partition re-assignment is done: because the consumer does not participate in the rebalance, it still thinks it owns the partitions. Hence, it tries to commit, and as you correctly said, it is (correctly) not allowed to commit as it dropped out of the group.
but how can i get the call back method form consumer-1....
You need to re-join the group by calling poll() again to get back into healthy state.
General comment: Your timeout configurations seems rather low and it's not recommended to have such small timeouts in practice. I think it will be hard to get a stable group with such low timeouts, as most likely consumers will hit the timeout regularly, dropping out of the group, and need to rejoin again.
I am try to update my KAFKA client from 0.8.2 to 0.9.0.1 to reduce the pressure to zookeeper cluster. And I'm running into follow questions:
The KAFKA consumer protocal say that "The join group request will park at the coordinator until all expected members have sent their own join group request". Then I found that the join group request is triggered by poll() and the method will not return before the group rebalancing finished. So does that means I will need as the same number of consumer thread as the consumer numbers to make sure all the consumers can send out the group join request at the same time? If I have more than 10000 partitions and I want each partition has it's own consumer, does that means I need more than 10000 consumer threads?
To trigger the heart beat, I need to call poll(). But if I don't want to get new messages since the old messages is still consuming, could I do that by consumer.pause() -> consumer.poll() -> consumer.resume()? Is there a better way to do that?
Consumers can read multiple partitions. So in general, a single consumer is sufficient -- it can assign all partitions to itself. However, if you "want that each partition has it's own consumer", you will of course need one consumer per partition...
About joining groups: if you have multiple consumers and you are in a rebalance, the rebalance will not block forever. There is a timeout applied. If a consumer does not send a join-request within the timeout, it drops out of the group (for now) and rebalance can finish. If this late consumer gets live again sending a join group request, a new rebalance will get triggered.
Pausing, poll, resume would be the right thing to do. Heads-up: this is going to get changed via KIP-62 that introduces a heartbeat background thread in the consumer.
I am a new user to Kafka and have been trialling it for about 2-3 weeks now. I believe at the moment I have a good understand of how Kafka works for the most part, but after attempting to fit the API for my own Kafka consumer (this is obscure but I'm following the guidelines for the new KafkaConsumer that is supposed to be available for v 0.9, which is out on the 'trunk' repo atm) I've had latency issues consuming from a topic if I have multiple consumers with the same groupID.
In this setup, my console consistently logs issues regarding a 'rebalance triggering'. Do rebalances occur when I add new consumers to a consumer group and are they triggered in order to figure out which consumer instance in the same groupID will get which partitions or are rebalances used for something else entirely?
I also came across this passage from https://cwiki.apache.org/confluence/display/KAFKA/Kafka+0.9+Consumer+Rewrite+Design and I just can't seem to understand it, so if someone could help me make sense of it that would be much appreciated:
Rebalancing is the process where a group of consumer instances
(belonging to the same group) co-ordinate to own a mutually exclusive
set of partitions of topics that the group is subscribed to. At the
end of a successful rebalance operation for a consumer group, every
partition for all subscribed topics will be owned by a single consumer
instance within the group. The way rebalancing works is as follows.
Every broker is elected as the coordinator for a subset of the
consumer groups. The co-ordinator broker for a group is responsible
for orchestrating a rebalance operation on consumer group membership
changes or partition changes for the subscribed topics. It is also
responsible for communicating the resulting partition ownership
configuration to all consumers of the group undergoing a rebalance
operation.
When a new consumer joins a consumer group the set of consumers attempt to "rebalance" the load to assign partitions to each consumer. If the set of consumers changes while this assignment is taking place the rebalance will fail and retry. This setting controls the maximum number of attempts before giving up.
the command for this is: rebalance.max.retries and is set to 4 by default.
also, it might be happening if the following is true:
ZooKeeper session timeout. If the consumer fails to send a heartbeat to ZooKeeper for this period of time it is considered dead and a rebalance will occur.
Hope this helps!
Rebalance is the re-assignment of partition ownership among consumers within a given consumer group. Remember that every consumer in a consumer group is assigned one or more topic partitions exclusively.
A Rebalance happens when:
a consumer JOINS the group
a consumer SHUTS DOWN cleanly
a consumer is considered DEAD by the group coordinator. This may happen after a crash or when the consumer is busy with a long-running processing, which means that no heartbeats has been sent in the meanwhile by the consumer to the group coordinator within the configured session interval
new partitions are added
Being a group coordinator (one of the brokers in the cluster) and a group leader (the first consumer that joins a group) designated for a consumer group, Rebalance can be more or less described as follows:
the leader receives a list of all consumers in the group from the
group coordinator (this will include all consumers that sent a
heartbeat recently and which are therefore considered alive) and is
responsible for assigning a subset of partitions to each consumer.
After deciding on the partition assignment (Kafka has a couple built-in partition assignment policies), the group leader sends
the list of assignments to the group coordinator, which sends this
information to all the consumers.
This applies to Kafka 0.9, but I'm quite sure for newer versions is still valid.
Consumer rebalance decide which consumer is responsible for which subset of all available partitions for some topic(s).
For example, you might have a topic with 20 partitions and 10 consumers; at the end of a rebalance, you might expect each consumer to be reading from 2 partitions. If you shut down 10 of those consumers, you might expect each consumer to have 1 partition after a rebalance has completed. Consumer rebalance is a dynamic partition assignment that can handle automatically by Kafka.
A Group Coordinator is one of the brokers responsible to communicate with consumers to achieve rebalances between consumers.In earlier version Zookeeper stored metadata details but the latest version, it stores on brokers. The consumer coordinators receive heartbeat and polling from all consumers of the consumer groups so be aware of each consumer's heartbeat and manager their offset on partitions.
Group Leader:
One of a consumer Group work as group leader which is chosen by the Group coordinator and will responsible for making partition assignment decision on behalf of all consumers in a group.
Rebalance Scenario:
Consumer Group subscribes to any topics
A Consumer instance could not able to send a heartbeat with a session.heart.beat time interval.
Consumer long process exceeds the poll timeout
Consumer of Consumer group through exception
New partition added.
Scaling Up and Down consumer. Added new consumer or remove existing consumer manually for
Consumer Rebalance
Consumer rebalance initiated when consumer requests to join a group or leave a group. The Group Leader receives a list of all active consumers from the Group Coordinator. Group Leader decides partition(s) assigned to each consumer by using PartitionAssigner.
Once Group Leader finalize partition assignment it sends assignments list to Group Coordinator which send back this information to all consumer. Group only sends applicable partitions to their consumer not other consumer assigned partitions. Only the Group Leader aware of all consumers and their assigned partitions.
After the rebalance is complete, consumers start sending Heartbeat to the Group Coordinator that it's alive.
Consumers send an OffsetFetch request to the Group Coordinator to get the last committed offsets for their assigned partitions.
Consumers start consuming messaged for newly assigned partition.
State Management
While rebalancing, the Group coordinator set its state to Rebalance and wait for all consumers to re-join the group.
When the Group starts rebalancing, the group coordinator first switches its state to rebalance so that all interacting consumers are notified to rejoin the group.
Once rebalance completed Group coordinator create new generation ID and notified to all consumers and group proceed to sync stage where consumers send sync request and go to wait until group Leader finish generating new assign partition. Once consumers received a new assigned partition they moved to a stable stage.
Static Membership
This rebalancing is quite a heavy operation as it required to stop all consumers and wait to get the new assigned partition. On each rebalance always create new generation id means refresh everything. To solve this overhead Kafka 2.3+ introduced Static Membership to reduce unnecessary Rebalance. KIP-345
In Static Membership, the consumer state will persist and on Rebalance the same assignment will get apply. It uses a new group.instance.id to persist member identity. So even in the worst-case scenario member id get reshuffle to assign a new partition but still, the same consumer instance-id will get the same partition assignment
instanceId: A, memberId: 1, assignment: {0, 1, 2}
instanceId: B, memberId: 2, assignment: {3, 4, 5}
instanceId: C, memberId: 3, assignment: {6, 7, 8}
And after the restart:
instanceId: A, memberId: 4, assignment: {0, 1, 2}
instanceId: B, memberId: 2, assignment: {3, 4, 5}
instanceId: C, memberId: 3, assignment: {6, 7, 8}
Ref:
https://www.confluent.io/blog/kafka-rebalance-protocol-static-membership
https://cwiki.apache.org/confluence/display/KAFKA/KIP-345%3A+Introduce+static+membership+protocol+to+reduce+consumer+rebalances
Consumer Group, Consumer and Partition Rebalance
Kafka Consumer can consume/Subscribe to multiple topics and start receiving the messages. Kafka Consumer are typically part of consumer group. When multiple consumers are subscribed to a topic and belong to same consumer group, each consumer in the group will receive messages from a different subset of partitions in the topic.
So consumers in a consumer group share ownership of the partitions in the topics they subscribe to. When we add a new consumer to the group, it starts consuming messages from partitions previously consumed by another consumer. The same thing happen when a consumer shuts down or crashes; it leaves the group, and the partition it used to consume will be consumed by one of the remaining consumers. Reassignment of partitions to consumer also happen when the consumer group is consuming are modified like new partition are added.
"Moving partition ownership from one consumer to another is called rebalance" During a rebalance, consumers can not consumer messages so we can say that rebalance is a short window of unavailability to entire consumer group. It also leads to some other activity on consumer side like when partitions are moved from one consumer t another consumer , cosnumer lose its current state like if any data is cache then it need to refresh its cache , slowing down the overall application until consumer is setup its state again.
heartbeat.interval.ms
Consumer maintain membership in a consumer group and ownership of the partitions assigned to them is by sending heartbeats to Kafka broker designated as a group coordinator and it will be different for different consumer group. As long as consumer is sending heartbeat at a regular intervals then it is considered t be alive and continue processing messages from designated assigned partition Heartbeat are sent when consumer call the poll method ( to retrieve records from partition) and when it commit the records it has consumed.
If a consumer stop sending heartbeat for long time and its session will time out (controlled by session.timeout.ms) then group coordinator will consider it dead and as a result trigger a rebalance. If a consumer crashed and are not processing messages it will take group coordinator few seconds without heartbeat to decide it is dead and trigger rebalance. When closing a consumer cleanly , consumer will notify the group coordinator that it is leaving the group and coordinator will trigger the rebalance immediately , reducing the time of unavailability of messages.