Kafka: bizarre assignment of partitions in a topic - apache-kafka

Question

I'm not sure how to explain the issue I'm facing with Kafka, but I'll try my best. I have a set of 4 consumers in the same consumer group named :
absolutegrounds.helper.processor
consuming from a topic with 5 partitions; therefore each consumer in the group is being assigned to 1 partition and 1 consumer to 2 partitions in order to distribute 5 partitions between 4 consumers fairly.
But for some reason I cannot figure out, the initial assignment turned into only 2 consumers being assigned into all the available partitions, i.e. 1 consumer with 3 partitions and 1 consumer with 2 partitions. However, there are still 4 consumers theoretically in the same consumer group:
[medinen#ocvlp-rks001 kafka_2.11-0.10.0.1]$ ./bin/kafka-run-class.sh kafka.admin.ConsumerGroupCommand --new-consumer --bootstrap-server localhost:9092 --describe --group absolutegrounds.helper.processor
GROUP TOPIC PARTITION CURRENT-OFFSET LOG-END-OFFSET LAG OWNER
absolutegrounds.helper.processor AG_TASK_SOURCE 0 27286 31535 4249 consumer-1_/10.132.9.128
absolutegrounds.helper.processor AG_TASK_SOURCE 1 28015 28045 30 consumer-1_/10.132.9.128
absolutegrounds.helper.processor AG_TASK_SOURCE 2 35437 40091 4654 consumer-1_/10.132.9.128
absolutegrounds.helper.processor AG_TASK_SOURCE 3 31765 31874 109 consumer-1_/10.132.8.23
absolutegrounds.helper.processor AG_TASK_SOURCE 4 33279 38003 4724 consumer-1_/10.132.8.23
The most bizarre behaviour is that the other 2 consumers left out of the consumer group (as per the response from Kafka above) seem to be still consuming from the topic as per the logs I see in my application, although I cannot find them anywhere as part of the consumer group. And even more bizarre is the fact that 1 consumer is supposed to be assigned to all the partitions in the topic whereas the other one is assigned only partition 4. See logs from the application (this is a Spring Boot application using Spring Kafka):
First left consumer:
- - 08/05/2017 12:27:29.119 - [-kafka-consumer-1] INFO o.a.k.c.c.i.ConsumerCoordinator - Setting newly assigned partitions [AG_TASK_SOURCE-0, AG_TASK_SOURCE-1, AG_TASK_SOURCE-2, AG_TASK_SOURCE-3, AG_TASK_SOURCE-4] for group absolutegrounds.helper.processor
Second left consumer:
- - 08/05/2017 12:27:19.044 - [-kafka-consumer-1] INFO o.a.k.c.c.i.ConsumerCoordinator - Setting newly assigned partitions [AG_TASK_SOURCE-4] for group absolutegrounds.helper.processor
Trying to understand the reasoning behind this behaviour, I've taken a look into the topic that stores all the offsets for the consumers:
__consumer_offsets
using this command:
kafka/kafka_2.11-0.10.0.1/bin/kafka-console-consumer.sh --consumer.config /tmp/consumer.config --formatter "kafka.coordinator.GroupMetadataManager\$GroupMetadataMessageFormatter" --zookeeper ocvlp-rks003:2181 --topic __consumer_offsets --from-beginning | grep "absolutegrounds.helper.processor"
and this is what I've found:
absolutegrounds.helper.processor::[absolutegrounds.helper.processor,consumer,Stable,Map(consumer-1-170fb8f6-c8d3-4782-8940-350673b859cb -> [consumer-1-170fb8f6-c8d3-4782-8940-350673b859cb,consumer-1,/10.132.8.23,10000], consumer-1-b8d3afc0-159e-4660-bc65-faf68900c332 -> [consumer-1-b8d3afc0-159e-4660-bc65-faf68900c332,consumer-1,/10.132.9.128,10000], consumer-1-dddf10ad-187b-4a29-9996-e05edaad3caf -> [consumer-1-dddf10ad-187b-4a29-9996-e05edaad3caf,consumer-1,/10.132.8.22,10000], consumer-1-2e4069f6-f3a8-4ede-a4f4-aadce6a3adb7 -> [consumer-1-2e4069f6-f3a8-4ede-a4f4-aadce6a3adb7,consumer-1,/10.132.9.129,10000])]
absolutegrounds.helper.processor::[absolutegrounds.helper.processor,consumer,Stable,Map(consumer-1-66de4a46-538c-425f-8e95-5a00ff5eb5fd -> [consumer-1-66de4a46-538c-425f-8e95-5a00ff5eb5fd,consumer-1,/10.132.9.129,10000])]
absolutegrounds.helper.processor::[absolutegrounds.helper.processor,consumer,Stable,Map(consumer-1-5b96166e-e528-48f7-8f6e-18a67328eae6 -> [consumer-1-5b96166e-e528-48f7-8f6e-18a67328eae6,consumer-1,/10.132.9.128,10000], consumer-1-dcfff37a-8ad3-403c-a070-cca82a1f6d21 -> [consumer-1-dcfff37a-8ad3-403c-a070-cca82a1f6d21,consumer-1,/10.132.8.23,10000])]
absolutegrounds.helper.processor::[absolutegrounds.helper.processor,consumer,Stable,Map(consumer-1-5b96166e-e528-48f7-8f6e-18a67328eae6 -> [consumer-1-5b96166e-e528-48f7-8f6e-18a67328eae6,consumer-1,/10.132.9.128,10000], consumer-1-dcfff37a-8ad3-403c-a070-cca82a1f6d21 -> [consumer-1-dcfff37a-8ad3-403c-a070-cca82a1f6d21,consumer-1,/10.132.8.23,10000])]
From the response of Kafka, I can see that at some point in time all 4 consumers where properly distributed between the partitions:
absolutegrounds.helper.processor::[absolutegrounds.helper.processor,consumer,Stable,Map(consumer-1-170fb8f6-c8d3-4782-8940-350673b859cb -> [consumer-1-170fb8f6-c8d3-4782-8940-350673b859cb,consumer-1,/10.132.8.23,10000], consumer-1-b8d3afc0-159e-4660-bc65-faf68900c332 -> [consumer-1-b8d3afc0-159e-4660-bc65-faf68900c332,consumer-1,/10.132.9.128,10000], consumer-1-dddf10ad-187b-4a29-9996-e05edaad3caf -> [consumer-1-dddf10ad-187b-4a29-9996-e05edaad3caf,consumer-1,/10.132.8.22,10000], consumer-1-2e4069f6-f3a8-4ede-a4f4-aadce6a3adb7 -> [consumer-1-2e4069f6-f3a8-4ede-a4f4-aadce6a3adb7,consumer-1,/10.132.9.129,10000])]
However, at some point later, the assignment changed to the current scenario where only 2 of the 4 consumers in the consumer group are assigned partitions.
I've struggled to understand what could have led to this situation, but I cannot find a valid answer to figure it out and fix it.
Anyone can help here? Thanks.