I was consuming from a Kafka topic (with a retention of 21 days) in a Kafka Cluster as a consumer (earliest/from beginning) for 15 days continously with x consumer group and on 15th day producer's team stopped producing and I stopped consumer on my side conforming that no messages left over to consume. Then Kafka Cluster was also turned off. Then on 16th day Kafka Cluster turned on and Producer started his producer on 23rd day and I started my consumer on myside as well. But when I started, I was getting messages from beginning not from where I left out eventhough am consuming with same x consumer group. So my question is why this happened? Does Kafka Broker lost information about consumer group?
When a consumer group loses all its consumers its offsets will be kept for the period configured in the broker property offsets.retention.minutes. This property defaults to 10080 which is the equivalent to 7 days — roughly the time taken when your consumer stopped (the 16th day) and when it was resumed (the 23rd day).
You can tweak this property to increase the retention period of offsets. Alternatively, you can also tweak the property offsets.retention.check.interval.ms that dictates how often a check for stale offsets will occur.
Related
I have a service with kafka consumers. Previously, I created and closed consumers after receiving records every time. I made a change and started using resume / pause without closing consumers (with ConsumerConfig.ENABLE_AUTO_COMMIT_CONFIG = false and consumer.commitSync(offsetAndMetadataMap);). The service worked great all week. After 7 days it was restarted. After the restart, all offsets disappeared and consumers began to receive all old records (). How could this happen? Where did the offsets go?
I guess your consumers of that consumer group were not up for the 7 days before the restart?
The internal offset topic which contains data about the offsets of your groups is defined as compacted and delete topic policy,
it means it compact the records to save last value of a key
and also deletes old records from the topic,
the default is 7 days, offset topic retention ,
KAFKA-3806: Increase offsets retention default to 7 days (KIP-186) #4648
it is configurable as any other topic configuration
Offset expiration semantics has slightly changed in this version. According to the new semantics, offsets of partitions in a group will not be removed while the group is subscribed to the corresponding topic and is still active (has active consumers). If group becomes empty all its offsets will be removed after default offset retention period (or the one set by broker) has passed (unless the group becomes active again). Offsets associated with standalone (simple) consumers, that do not use Kafka group management, will be removed after default offset retention period (or the one set by broker) has passed since their last commit.
I have Kafka server version 2.4 and set log.retention.hours=168(so that messages in the topic will get deleted after 7 days) and auto.offset.reset=earliest(so that if the consumer doesn't get the last committed offset then it should be processed from the beginning). And since I am using Kafka 2.4 version so by default value offsets.retention.minutes=10080 (since I am not setting this property in my application).
My Topic data is : 1,2,3,4,5,6,7,8,9,10
current consumer offset before shutting down consumer: 10
End offset:10
last committed offset by consumer: 10
So let's say my consumer is not running for the past 7 days and I have started the consumer on the 8th day. So my last committed offset by the consumer will get expired(due to offsets.retention.minutes=10080 property) and topic messages also will get deleted(due to log.retention.hours=168 property).
So wanted to know what consumer offset will be set by auto.offset.reset=earliest property now?
Although no data is available in the Kafka topic, your brokers still know the "next" offset within that partition. In your case the first and last offset of this topic is 10 whereas it does not contain any data.
Therefore, your consumer which already has committed offset 10 will try to read 11 when started again, independent of the consumer configuration auto.offset.reset.
Your example will get even more interesting when your topic has had offsets, say, until 15 while the consumer was shut down after committing offset 10. Now, imagine all offsets were removed from the topic due to the retention policy. If you then start your consumer only then the consumer configuration auto.offset.reset comes into effect as stated in the documentation:
"What to do when there is no initial offset in Kafka or if the current offset does not exist any more on the server (e.g. because that data has been deleted)"
As long as the Kafka topic is empty there is no offset "set" for the consumer. The consumer just tries to find the next available offset, either based on
the last committed offset or,
in case the last committed offset does not exist anymore, the configuration given through auto.offset.reset.
Just as an additional note: Even though the messages seem to get cleaned by the retention policy you may still see some data in the topic due to Data still remains in Kafka topic even after retention time/size
Once the consumer group gets deleted from log, auto.offset.reset will take the precedence and consumers will start consuming data from beginning.
My Topic data is : 1,2,3,4,5,6,7,8,9,10
If the topic has the above data, the consumer will start from beginning, and all 1 to 10 records will be consumed
My Topic data is : 11,12,13,14,15,16,17,18,19,20
In this case if old data is purged due to retention, the consumer will reset the offset to earliest (earliest offset available at that time) and start consuming from there, for example in this scenario it will consume all from 11 to 20 (since 1 to 10 are purged)
I'm trying to understand, how does the ConsumerConfig.auto.offset.reset = latest would affect the message consumption.
For example I've a consumer, sending 100 messages initially at time t1 and then my consumer is up and running at t1+30 sec, then would my consumer consume the messages published after t1+30 sec or will it consume messages published t1 onwards?
It depends.
auto.offset.reset only applies when there is no stored offset for the consumer group.
It applies to the following conditions:
the first time a consumer group consumes
if a consumer doesn't commit any offsets, the next time it is started
if a consumer group has been expired (7 days by default with modern brokers)
if the message the stored offset points to has been removed due to message retention policies (an attempt to read a message that has been purged triggers the application of the rule)
If a consumer commits an offset; it will start at the last committed offset the next time it is started.
Does retention period of zero makes sense in kafka borker?
We want to quickly forward message from producer to consumer via kafka broker. From buffercache/pagecache on broker machine without flushing to disk. We do not need replication and assume our broker will never crash.
When a message is produced to a Kafka topic it is written to the disk. Once the message has been consumed, the offset of this message is committed by the consumer (if you are using the high-level consumer API) however, there is no functionality that deletes only the messages that have been consumed (many consumers may subscribe to the same topic and some of them might have consumed that message while some others might have not).
What I would suggest in your case is to set a short retention period (which by default is set to 7 days) but allow a reasonable amount of time in order to allow your consumer to consume the messages. To do this, you simply need to configure the following parameter in server.properties:
log.retention.ms=X
Note that there is no guarantee that the deleted message(s) have been successfully consumed by your consumer(s). For example, if you set the retention period to 2 seconds (i.e. log.retention.ms=2000) and your consumer crashes, then every message which is sent to the topic while the consumer is down will be lost.
We are using kafka cluster with 3 servers, all of them are having zookeeper as well. we have 7 topics each with 6 partitions. We have 3 java consumers for each topic. When I start consumer it takes almost 3-5 min to assign partitions to consumers. The same behavior is encountered when we stop one of the consumer and start is again. How can I control or reduce it?
Please note, I am using kafka 0.9, with new consumer
I have added below properties in server.properties of each kafka
auto.leader.rebalance.enable=true
leader.imbalance.check.interval.seconds=10
Let me know if you need more information.
Thanks
Check the value your consumer is using for 'session.timeout.ms'.
The default is 30 seconds and the co-ordination won't trigger a rebalance until this time has passed E.g. no heartbeat for 30 seconds.
The danger in making this lower is if you take too long to process the messages a rebalance might occur because the co-ordinator will think your consumer is dead.