We are using kafka cluster with 3 servers, all of them are having zookeeper as well. we have 7 topics each with 6 partitions. We have 3 java consumers for each topic. When I start consumer it takes almost 3-5 min to assign partitions to consumers. The same behavior is encountered when we stop one of the consumer and start is again. How can I control or reduce it?
Please note, I am using kafka 0.9, with new consumer
I have added below properties in server.properties of each kafka
auto.leader.rebalance.enable=true
leader.imbalance.check.interval.seconds=10
Let me know if you need more information.
Thanks
Check the value your consumer is using for 'session.timeout.ms'.
The default is 30 seconds and the co-ordination won't trigger a rebalance until this time has passed E.g. no heartbeat for 30 seconds.
The danger in making this lower is if you take too long to process the messages a rebalance might occur because the co-ordinator will think your consumer is dead.
Related
We have a single node Kafka service running Kafka 2.13 version. For one of the topics, we have configured 20 partitions and there is just one consumer group consuming from this topic. This setup has been working fine for a long time now. Recently we have been seeing issues w.r.t Kafka rebalancing this consumer group frequently. After a while, the consumers start consuming again, but on one of the partitions, the current offset doesn't move forward at all indicating the messages are stuck in that partition. Other messages from other partitions get consumed without any issues. Logs from Kafka service doesn't show any issue. Any hints on what is going wrong and how to identify / rectify it?
A possible explanation is that the specific partition has a message that takes a long time to process, this could explain both the frequent rebalancing and the stuck offset
I use Confluent.Kafka 1.9.2 C# library to create single Kafka consumer for listening topic with several partitions. Currently consumer drain out all messages from first partition and only then goes to next. As I know from KIP, I can avoid such behavior and achieve round-robin by changing max.partition.fetch.bytes parameter to lower value. I changed this value to 5000 bytes and pushed 10000 messages to first partition and 1000 to second, average size of messages is 2000 bytes, so consumer should to move between partitions every 2-3 messages (if I understand correctly). But it still drains out first partition before consuming second one. My only guess why it don't work as should is latest comment here that such approach can't work with several brokers, btw Kafka server that I use just has 6 brokers. Could it be the reason or maybe something else?
Kafka image :confluentinc/cp-kafka:5.2.1
Kafka client: apache.kafka.client 2.5.0
load:One thread is handling 4 partitions
Noticing some of the offsets are getting skipped (missing) in a partition.First thought it was an issue on the consumer side but In both thread consumer groups logs offsets are getting skipped(also another observation: Consumer thread is taking significant amount of time to jump the skipped offset which is causing lag)This happened when the load on the cluster was high . We are not using transactional kafka or using idempotent configurations.What can be the possible reasons for this?
Producer properties:
ACKS_CONFIG=1
RETRIES_CONFIG=1
BATCH_SIZE_CONFIG=16384);
LINGER_MS_CONFIG=1
BUFFER_MEMORY_CONFIG=33554432
KEY_SERIALIZER_CLASS_CONFIG= StringSerializer
VALUE_SERIALIZER_CLASS_CONFIG= ByteArraySerializer
ProducerConfig.RECONNECT_BACKOFF_MS_CONFIG, 1000
Consumer properties:
HEARTBEAT_INTERVAL_MS_CONFIG=6 seconds
SESSION_TIMEOUT_MS_CONFIG=30 sec
ConsumerConfig.MAX_POLL_RECORDS_CONFIG=10
FETCH_MAX_WAIT_MS_CONFIG=200
MAX_PARTITION_FETCH_BYTES_CONFIG=1048576
FETCH_MAX_BYTES_CONFIG=31457280
AUTO_OFFSET_RESET_CONFIG=latest
ENABLE_AUTO_COMMIT_CONFIG=false
Edit
there were no consumer rebalances checked the logs during this time so no consumer rebalances
we have two consumer groups (in different data centers )and both the groups skipped these offsets.So ruling out any issue from the consumer side.
-Both had the same pattern the consumer eg-> both consumer threads stopped consuming till offset 112300 and after 30 mins started consuming after skipping 12k offsets. And the threads were consuming other partitions. This only happened for 3-4 partitions.
-so what I’m wondering is it normal to have such huge offset gaps ? (During high loads)Didn’t find anything concrete when going through docs. And what can cause this issue - is it from the broker side or the producer side to have ghost offsets
I was running a test where kafka consumer was reading data from multiple partitions of a topic. While the process was running I added more partitions. It took around 5 minutes for consumer thread to read data from the new partition. I have found this configuration "topic.metadata.refresh.interval.ms", but this is for producer only. Is there a similar config for consumer too?
When we add more partitions to an existing topic then a rebalance process gets initiated.
Every consumer in a consumer group is assigned one or more topic partitions exclusively, and Rebalance is the re-assignment of partition ownership among consumers.
A Rebalance happens when:
consumer JOINS the group
consumer SHUTS DOWN cleanly
consumer is considered DEAD by the group coordinator. This may happen after a
crash or when the consumer is busy with long-running processing, which means
that no heartbeats have been sent in the meanwhile by the consumer to the
group coordinator within the configured session interval
new partitions are added
We need to provide two parameters to reduce the time to rebalance.
request.timeout.ms
max.poll.interval.ms
More detailed information is available at the following.
https://medium.com/streamthoughts/apache-kafka-rebalance-protocol-or-the-magic-behind-your-streams-applications-e94baf68e4f2
I changed "metadata.max.age.ms" parameter value to refresh matadata https://kafka.apache.org/documentation/#consumerconfigs_metadata.max.age.ms
Scenario:
Running spring-boot project consuming from partition named 'test' that has 10 partitions. Partition assignment happens at 13:00:00
At ~13:00:30 adding partitions to the topic using: ./kafka-topics.sh --alter --zookeeper zookeeper:2181 --topic test --partitions 100
At ~13:05:30 partition reassignment is triggered.
I ran those steps few times and it looks like that reassignment happens every ~5 minutes.
Is there a way to change the reassignment check operation frequency?
We would like it to be every few seconds. Does this operation is heavy and this is the reason that it happens every 5 minutes? Or it's pretty negligible?
EDIT:
My use case is the following: we have integration tests which boots our microservices. When a consumer of a topic boots first, it creates the topic if it does not exists and the number of partitions it's creating equals to the configured concurrency (10 for example). Then the producer of this topic boots and his configured partitonCount (20 for example) is bigger than the number of created partitions, so spring-cloud-stream adds the missing partitions, in the mean time the consumer assigned partitions , haven't changed and it keeps consuming from the first 10 partitions (1-10). The problem is that the producer is publishing messages to all 20 partitions so messages that are sent to the last 10 partitions (11-20) will not be consumed until the consumer is assigned the new partitions. This behavior causes problems to our tests and we cannot wait for 5 minutes until all of the partitions are assigned to consumers. Also we would not want to create the topic with the number of desired partitions in advance, and we would like that it will still be handled by spring-cloud-stream.
EDIT 2:
It seems like that the relevant property that controls the "reassignment" is metadata.max.age.ms.
The period of time in milliseconds after which we force a refresh of metadata even if we haven't seen any partition leadership changes to proactively discover any new brokers or partitions.
So there are couple of concerns here.
First, "spring-cloud-stream" and/or "spring-kafka" are NOT doing any type of rebalancing, partition reassignment etc. This is all done inside of Kafka. There is a client-side property in Kafka that defaults to 5 min (i believe) if a consumer didn't poll for that much time consider it dead etc. In any event I would refer you to apache-kafka channel to get more information on Kafka internals.
Also, adding partitions, reassignments and rebalancing are expensive operations and should not be attempted without serious consideration of their impact. So, I'd be curious to know what is your use case for constantly adding partitions?