When stop one consumer and start it instantly, the consumer group needs about 10 seconds to do rebalance - apache-kafka

Question

1. problem desc
If stop one consumer and start this consumer instantly, the consumer group needs about 10 seconds to do rebalance.
But if stop one consumer and wait for about 10s, then start that consumer, this problem will not appear.
2. Env
kakfa-version:0.11.0.1
3. demo
Producer1Demo.java(Producer-A send message to topic:topic1)
Producer2Demo.java(Producer-B send message to topic:topic2)
Consumer1Demo.java(Consumer-A get message from topic:topic1)
Consumer2Demo.java(Consumer-B get message from topic:topic2)
Consumer-A and Consumer-B in same group group1,
the producer's codes like below
public static void main(String[] args) throws Exception{
Properties props = new Properties();
props.put("bootstrap.servers", "ip:9092");
props.put("acks", "all");
props.put("retries", 0);
props.put("batch.size", 16384);
props.put("linger.ms", 1);
props.put("buffer.memory", 33554432);
props.put("key.serializer", "org.apache.kafka.common.serialization.StringSerializer");
props.put("value.serializer", "org.apache.kafka.common.serialization.StringSerializer");
Producer<String, String> producer = new KafkaProducer<>(props);
boolean flag = true;
int i = 0;
while(flag){
if(i>=1000){
flag = false;
}
producer.send(new ProducerRecord("topic1", "topic<1>-"+ i));
i++;
Thread.sleep(1000);
}
producer.close();
}
The consumer's codes like below
public static void main(String []args){
Properties props = new Properties();
props.put("bootstrap.servers", "ip:9092");
props.put("group.id", "group1");
props.put("enable.auto.commit", "true");
props.put("auto.commit.interval.ms", "1000");
props.put("key.deserializer", "org.apache.kafka.common.serialization.StringDeserializer");
props.put("value.deserializer", "org.apache.kafka.common.serialization.StringDeserializer");
KafkaConsumer<String, String> consumer = new KafkaConsumer<>(props);
consumer.subscribe(Arrays.asList("topic1"));
boolean flag = true;
while (flag) {
ConsumerRecords<String, String> records = consumer.poll(100);
for (ConsumerRecord<String, String> record : records)
System.out.printf("offset = %d, key = %s, value = %s%n", record.offset(), record.key(), record.value());
}
consumer.close();
}
4.how to reappear this problem
First start above four java's main method, then, Consumer-A and
Consumer-B can get message from their corresponding topic
continuously
Then, I stop the Consumer1Demo.java's process and start
Consumer1Demo.java's main method instantly, Consumer-B will bock for
about 10 seconds to rejoin group group1, below is Consumer-A's log's
log:
Discovered coordinator 10.105.168.92:9092 (id: 2147483646 rack: null) for group group1.
Revoking previously assigned partitions [] for group group1
(Re-)joining group group1
<will sleep about 10s>
Successfully joined group group1 with generation 52
Setting newly assigned partitions [topic1-0, topic1-1, topic1-6, topic1-7, topic1-2, topic1-3, topic1-4, topic1-5] for group group1
Consumer-B's log
Revoking previously assigned partitions [topic2-0, topic2-1, topic2-2, topic2-7, topic2-3, topic2-4, topic2-5, topic2-6] for group group1
(Re-)joining group group1
<will sleep about 10s>
Successfully joined group group1 with generation 52
Setting newly assigned partitions [topic2-0, topic2-1, topic2-2, topic2-7, topic2-3, topic2-4, topic2-5, topic2-6] for group group1
offset = 215, key = null, value = testMessageTopic<2>-318
Through log, we can see, the Consumer-A and Consumer-B will also cost about 10 seconds to join the group, it seems they both need to get generation same value:52, this step seems to cost 10 seconds.
5.How to avoid this problem.
First, stop Consumer-A process, and wait about 15 seconds, then,
start Consumer1Demo.java main to start Consumer-A, this problem will gone.
Then let's analyze the log.
<5.2.1>Before doing step1, the Consumer-A and Consumer-B's generation value are also 63
<5.2.2>after stop Consumer-A process, Consumer-B will rejoin the group instantly and get the generation value:64
<5.2.3>after wait about 15 seconds, start Consumer1Demo.java main again. the Consumer-A will rejoin the group instantly and get the generation value:65
<5.2.4>Then, Consumer-B will also rejoin the group with the generation value:65 when the generation value is changed due to Consumer-A rejoin again.
5.The root reason
When one consumer's down, consumer group will rebalance and get new
generation value(oldValue + 1)
When one consumer startup, consumber group will also do rebalance
and get new generation value(oldValue + 1)
In this case, it seems step5.1 and step5.2 happen at the same time, so may exist resource competition, so the coordinator may cost 10 seconds to solve the problem.
6.my question
I don't know whether I analyzed the problem right, there was another reason?

Answer 1

Because the two consumer1 and consumer2 in same consumer group,when restart or stop one of the consumers, the consumer group will do do rebalance operation, it will cost about 10s.
After changed to different consumer groups, the problem gone.