I have a retention policy set for 48 hours. So old logs are eventually flushed. But topic's offset number keeps growing. When does this number get reset? What happens when the max offset number is reached? Also, new segments are rolled with base offset as filename at the time of creating new segment.What will be the filenames of .log and .index files when this limit is reached?
The following is the current base offset for log segment :
The offset is never reset because the max offset value is so big (int64) that you won't ever reach it.
Related
I was following the book "Kafka: The Definitive Guide" First Edition to understand when log segments are deleted by the broker.
As per the text I understood, a segment will not become eligible for deletion until it is closed. A segment can be closed only when it has reached log.segment.bytes size (considering log.segment.ms is not set) . Once a segment becomes eligible for deletion, the log.retention.ms policy would apply to finally decide when to delete this segment.
However this seems to contradict the behaviour I see in our production cluster ( Kafka ver 2.5).
The log segment gets deleted as soon as log.retention.ms is satisfied, even when the segment size is less than log.segment.bytes.
[2020-12-24 15:51:17,808] INFO [Log partition=Topic-2,
dir=/Folder/Kafka_data/kafka] Found deletable segments with base
offsets [165828] due to retention time 604800000ms breach
(kafka.log.Log)
[2020-12-24 15:51:17,808] INFO [Log partition=Topic-2,
dir=/Folder/Kafka_data/kafka] Scheduling segments for deletion
List(LogSegment(baseOffset=165828, size=895454171,
lastModifiedTime=1608220234000, largestTime=1608220234478))
(kafka.log.Log)
The size is still less than 1GB, but the segment got deleted.
The book mentions at the time of press release the Kafka version was 0.9.0.1 . So was this setting changed in later versions of Kafka. ( I could not find any specific mention of this change in the Kafka docs). Below is the snippet from the book.
Setting: log.retention.ms and log.retention.bytes
The most common configuration for how long Kafka broker will retain messages (actually, “log segments”) is by time (in milliseconds), and is specified using log.retention.ms parameter (default to 1 week). If set to -1, no time limit is applied.
Another way to expire is based on the total number of bytes of messages retained. This value is set using the log.retention.bytes parameter, and it is applied per partition. Its default value is -1, which allows for infinite retention. This means that if you have a topic with 8 partitions, and log.retention.bytes is set to 1 GB, the amount of data retained for the topic will be 8 GB at most. If you have specified both log.retention.bytes and log.retention.ms, messages may be removed when either criterion is met.
Setting: log.segment.bytes and log.segment.ms
As messages are produced to the Kafka broker, they are appended to the current log segment for the partition. Once the log segment has reached the size specified by the log.segment.bytes parameter (default 1 GB), the log segment is closed and a new one is opened. Only once a log segment has been closed, it can be considered for expiration (by log.retention.ms or log.retention.bytes).
Another way to control when log segments are closed is by using the log.segment.ms parameter, which specifies the amount of time after which a log segment should be closed. Kafka will close a log segment either when the size limit is reached or when the time limit is reached, whichever comes first.
A smaller log-segment size means that files must be closed and allocated more often, which reduces the overall efficiency of disk writes. Adjusting the size of the log segment can be important if topics have a low produce rate. For example, if a topic receives only 100 megabytes per day of messages, and log.segment.bytes is set to the default, it will take 10 days to fill one segment. As messages cannot be expired until the log segment is closed, if log.retention.ms is set to 1 week, they will actually be up to 17 days of messages retained until the closed segment expires. This is because once the log segment is closed with the current 10 days of messages, that log segment must be retained 7 days before it expires based on the time policy.
Hope this becomes clearer.
segment.ms => the maximum age of the segment file (from the date of
creation)
retention.ms => the maximum age of any message in a segment (that is
closed) beyond which this segment is eligible for deletion (if delete
policy is set)
So if the segment is "active segment" then it can be rolled over based on segment.ms (or segment.bytes) but NOT by retention.ms. The retention only comes into play on closed (not active) segments.
So the behavior that is quoted from the book is correct. However you think that the segment is active and the INFO logs specify that the segment is setup for deletion.
This cannot happen on an active segment (assuming no bug). The segment has to be closed (not active) before any of the retention.* properties can take effect.
See this.
What you observe is the expected behavior. In short, if you have an active segment that is not full yet, and segment.ms has passed, then it will be closed and turn into an "old log segment" even if it is not full.
i have a kafka setting for retention like this:
# A size-based retention policy for logs. Segments are pruned from the log as long as the remaining$
# segments don't drop below log.retention.bytes.$
log.retention.bytes=1073741824$
$
# The maximum size of a log segment file. When this size is reached a new log segment will be created.$
log.segment.bytes=1073741824$
So the size of log.retention.bytes and log.segment.bytes are 1gb, and then i created a topic with only one partition. After flushing message to my topic, i observed that there is alway two log files, one file already reached 1gb and another one is an active one which is receiving messages.
My question is, does this log.retention.bytes defines the maximum total size of inactive segments files and not including the active one?
Thanks
Yes that's roughly correct. I usually don't like to define this setting as the "maximum size" as it's not completely right.
One way to see it is to consider log.retention.bytes the minimum amount of data that must be left after Kafka deletes segments. Or the amount of data Kafka guarantees to keep at anytime (obviously only if the time retention limit is not reached!)
The active segment is not eligible for deletion. So as you noticed when the first segment fills up, Kafka does not delete anything even though you reached 1GB. Instead it rolled a new segment (the new active one). Once this new segment also reaches 1GB, you effectively have 2GB of data on disk.
At that point a new segment is rolled again and you have 2 inactive segments. Only now Kafka can delete a segment and still satisfy log.retention.bytes, as there will be 1GB of data on disk + the active segment.
From Kafka Docs I got interested and tried the following 2 retention types together
log.retention.bytes:
The maximum size of the log before deleting it
Type: longDefault: -1Valid Values:Importance: highUpdate Mode: cluster-wide
log.retention.ms
The number of milliseconds to keep a log file before deleting it (in
milliseconds), If not set, the value in log.retention.minutes is used.
If set to -1, no time limit is applied. Type: longDefault: nullValid
Values:Importance: highUpdate Mode: cluster-wide
AS
log.retention.bytes = 1Gb
log.retention.ms = 7 days
Problem Situation
I have currently on my topic all messages belonging two different log files both of which are < 1GB
Lets say log.1 files has 400 MB of messages with oldest message > 7 days old.
which is on the top of
log.2 file has 500 MB with newest message > 7 days old.
I understand kafka would clean up all records belonging to log.2 file in other words remove this log from the topic.
What happens to the records in the log.1 which are older than 7 days?
There are two properties which defines message retention in Kafka - log.retention.bytes and log.retention.ms (per topic per partition level). The strategy for data removal works on FIFO basic, i.e., the message which was pushed to a topic first would be deleted first.
You have rightly said that the default values for the same are:
log.retention.bytes = 1Gb (per topic per partition)
log.retention.ms = 7 days (per topic)
It means that whichever limit is breached first, would lead to data purge in Kafka.
For example, let's assume that the size of messages in your topic takes 500 MB of space (which is less than log.retention.bytes) but older than 7 days (i.e. greater than the default log.retention.ms). In this case the data older than 7 days would be purged (on FIFO basis).
Likewise, if, for a given topic, the space occupied by the messages exceeds the log.retention.bytes but are not older than log.retention.ms, in this case too, the data would be purged (on FIFO basis).
Concept of making data expire is called as Cleanup & the messages on a topic are not immediately removed after they are consumed/expired. What happens in the background is, once either of the limit is breached, the messages are marked deleted. There are 3 logs cleanup policies in Kafka - DELETE (default), COMPACT, DELETE AND COMPACT. Kafka Log Cleaner does log compaction, a pool of background compaction threads.
To turn on compaction for a topic use topic config log.cleanup.policy=compact. To set delay to start compacting records after they are written use topic config log.cleaner.min.compaction.lag.ms. Records won’t get compacted until after this period. The setting gives consumers time to get every record. This could be reason that older messages are not getting deleted immediately. You can check the value of property for compaction delay.
Below links might be helpful:
https://medium.com/#sunny_81705/kafka-log-retention-and-cleanup-policies-c8d9cb7e09f8
http://cloudurable.com/blog/kafka-architecture-log-compaction/index.html
https://www.learningjournal.guru/courses/kafka/kafka-foundation-training/broker-configurations/
I'm paraphrasing here, from the relevant section of a book, Kafka - Definitive Guide. It'll most likely clear your doubt.
log.retention.bytes : This denotes the total number of bytes of messages retained per partition. So, if we have a topic with 8 partitions, and log.retention.bytes is set to 1GB, then the amount of data retained for the topic will be 8GB at most. This means if we ever choose to increase the number of partitions for a topic, total amount of data retained will also increase.
log.retention.ms : The most common configuration for how long Kafka will retain messages is by time. The default is specified in the configuration file using the log.retention.hours parameter, and it is set to 168 hours, or one week. However, there are two other parameters allowed, log.retention.minutes and log.retention.ms. All three of these specify the same configuration—the amount of time after which messages may be deleted—but the recommended parameter to use is log.retention.ms, as the smaller unit size will take precedence if more than one is specified. This will make sure that the value set for log.retention.ms is always the one used. If more than one is specified, the smaller unit size will take precedence.
Retention By Time and Last Modified Times : Retention by time is performed by examining the last modified time (mtime) on each log segment file on disk. Under normal cluster operations, this is the time that the log segment was closed, and represents the timestamp of the last message in the file. However, when using administrative tools to move partitions between brokers, this time is not accurate and will result in excess retention for these partitions.
Configuring Retention by Size and Time : If you have specified a value for both log.retention.bytes and log.retention.ms (or another parameter for retention by time), messages may be removed when either criteria is met. For example, if log.retention.ms is set to 86400000 (1 day) and log.retention.bytes is set to 1000000000 (1 GB), it is possible for messages that are less than 1 day old to get deleted if the total volume of messages over the course of the day is greater than 1 GB. Conversely, if the volume is less than 1 GB, messages can be deleted after 1 day even if the total size of the partition is less than 1 GB.
I am new to kafka, when we use kafka,we can set the retention.bytes. say we set to 1GB, if the amount of message reach 1GB,kafka will delete messages.I want to ask that the offset will be reset to zero?
second, the consumer set auto.offset.reset to largest, after kafka delete the messages, what offset will the consumer start?
For your question #1, with honoring both size-based and time-based policies, log might be rolled over to a new empty log segment. New log segment file's starting offset will be the offset of the next message that will be appended to the log.
For your question #2, it depends. If the offset tracked by consumer is out of range due to the message deletion, then it will be reset to the largest offset.
Kafka Topic Partition offset position always start from 0 or random value and How to ensure the consumer record is the first record in the partition ? Is there any way to find out ? If any please let me know. Thanks.
Yes and no.
When you start a new topic, the offset start at zero. Depending on the Kafka version you are using, the offsets are
logical – and incremented message by message (since 0.8.0: https://issues.apache.org/jira/browse/KAFKA-506) – or
physical – ie, the offset is increased by the number of bytes for each message.
Furthermore, old log entries are cleared by configurable conditions:
retention time: eg, keep message of the last week only
retention size: eg, use at max 10GB of storage; delete old messages that cannot be stored any more
log-compaction (since 0.8.1): you only preserve the latest value for each key (see https://cwiki.apache.org/confluence/display/KAFKA/Log+Compaction)
Thus, the first offset might not be zero if old messages got deleted. Furthermore, if you turn on log-compaction, some offsets might be missing.
In any case, you can always seek to any offset safely, as Kafka can figure out if the offset is valid or not. For an invalid offset, is automatically advances to the next valid offset. Thus, if you seek to offset zero, you will always get the oldest message that is stored.
Yes, Kafka offset starts from 0 and ends with byte length of the complete record and then next record picks the offset from there onward.
As Kafka is distributed so we can not assure that Consumer will get the data in ordered way.