I've a Kafka topic with one partition. I'm trying to send messages to broker. The source is of 1.5 TB in size. My broker has two directories to store the Kafka partitions
/dev/sdc1 1.1T 567G 460G 56% /data_disk_0
/dev/sdd1 1.1T 1.1T 0 100% /data_disk_1
Each one with 1.1 TB size. As my topic has only one partition, Kafka is storing all the messages to /dev/sdd1. Eventually the disk fills up completely because the source size is greater than the target disk size. Can I span my topic partition to store half data in disk0 and the other half in disk1 without changing the number of partitions?
Please advice
I couldn't find any configuration related changes that I can add to Kafka
This isn't possible at the kafka configuration level. You'd have to use RAID or logical volume groups to pool the disks together as one volume
In the Kafka documentation, it mentions
You can either RAID these drives together into a single volume or format and mount each drive as its own directory
If your data is so heavily skewed to one disk, meaning certain partitions, you should be checking how your producers are partitioning the data, start to persist such a large topic somewhere, or turn on compaction / retention periods for these topics
Related
When doing stateful processing in kafka streams we can hold large state. We can provision more disks space for the client as the data grows. But what about the changelog topic? The local state is backed up in this compacted topic. Are there any limitations in how much data we can store in this topic?
We did not encounter any issues yet. But i see that some cloud services do have limitations on the size for a compacted topic. Is this a kafka limitation? An if yes, do these limitations also apply for non compacted topics?
Infinite retention of any topic log segments can be achieved by setting
log.retention.bytes = -1
log.retention.hours = -1
This option is available from version 0.9.0.0 which indicates a mature feature on Kafka.
However, many suggest that using Kafka as permanent storage is not what it was designed to do and as the amount of data stored in Kafka increases, users eventually hit a “retention cliff,” at which point it becomes significantly more expensive to store, manage, and retrieve data. The infrastructure costs will be increased as the longer the retention period the more hardware is required.
Having said that, it seems that people do use Kafka for persistence storage, for example, The New York Times uses Kafka as a source of truth, storing 160 years of journalism going back to the 1850s.
I would suggest using a small message size if you decide to use
Kafka as a System Of Record (SOR) and to hold the state of an entity.
Kafka makes it very clear that its performance is greatly based on the event/message size, so there is a size limit on them.
Kafka has a default limit of 1MB per message in the topic. This is
because very large messages are considered inefficient and an
anti-pattern in Apache Kafka.
more for handling larger messages here.
By default, each Kafka topic partition log will start at a minimum size of 20MB and grow to a maximum size of 100MB on disk before a new log file is created. It's possible to have multiple log files in a partition at any one time.
we have kafka cluster with 3 nodes
kafka contain 5 topics and each topic include 100 partitions
bow we want to set the retention bytes and the retention segment in way that each topic partition will not high the 5G ( because we are limited according to kafka disk size )
is it possible to tune the values of retention bytes and segment bytes , so no way that any topic partition will be high then 5G ?
There is no way to cap the size of a topic. It's even possible that retention will go above retention.bytes if you push data into the topic faster than the LogCleaner thread has time to clean it up.
Also note that upcoming versions of Kafka will offer infinite retention
Or you could similarly use tiered storage features of Apache Pulsar instead of Kafka
I have a Kafka cluster with 3 brokers (v2.3.0) and each broker has 2 disks attached. I added a new topic (heavyweight) and was surprised that even if the topic has 15 partitions, those weren't distributed evenly on the disks. Thus I got one disk that's almost empty and the other almost filled up. Is there any way to have Kafka evenly distribute data on its disks?
In designing a streaming processing pipeline what cost might be incurred if I were to have many topics which would have at least one partition but potentially no data going into it?
As an example, with one consumer and I could choose to have one "mega topic" which contains all of the data and many partitions or I could choose to split that data (by tenant, account, or user etc.) into many topics with, by default, a single partition. My worry about the second case is that there would be many topics/partitions which would see no data. So, is this unused partition costing anything or is there no cost that is incurred by an unused topic.
First of all, there is no difference between one fat topic and lots of partitions and more than one topic that contains a few partitions. Topic is just for logical distinction between events. Kafka only cares about number of partitions.
Secondly, having lots of partitions can lead some problems:
Too many open files:
Each partition maps to a directory in the file system in the broker.
Within that log directory, there will be two files (one for the index
and another for the actual data) per log segment.
More partitions requires more memory both in broker and consumer
sides:
Brokers allocate a buffer the size of replica.fetch.max.bytes for each
partition they replicate. If replica.fetch.max.bytes is set to 1 MiB,
and you have 1000 partitions, about 1 GiB of RAM is required.
More Partitions may increase unavailability:
If a broker which is controller is failed, then zookeeper elect another broker as controller. At that point newly elected broker should read metadata for every partition from Zookeeper during initialization.
For example, if there are 10,000 partitions in the Kafka cluster and
initializing the metadata from ZooKeeper takes 2 ms per partition,
this can add 20 more seconds to the unavailability window.
You may get more information from these links:
https://www.confluent.io/blog/how-choose-number-topics-partitions-kafka-cluster/
https://docs.cloudera.com/documentation/kafka/latest/topics/kafka_performance.html
Assuming the mentioned topics are not compacted, there is the initial overhead of retaining any initially produced data, but after which, an empty topic is just
metadata in zookeeper
metadata in any consumer group coordinator, and wasted processing by any active consumer threads
empty directories on disk
For the first two, having lots of topics may increase request latency, causing an unhealthy cluster.
I have a question about Kafka's disk.
Kafka will fail when its disk become full.
So I want to reduce the disk usage to less than x% by discarding the old data stored on the Kafka disk (or discarding a copy of the data) when the Kafka disk usage reaches x%. Do I need to modify the Kafka source code to do this?
You can configure retention.bytes for your topics.
This configuration controls the maximum size a partition (which consists of log segments) can grow to before we will discard old log segments to free up space if we are using the "delete" retention policy. By default there is no size limit only a time limit. Since this limit is enforced at the partition level, multiply it by the number of partitions to compute the topic retention in bytes.
See https://kafka.apache.org/documentation/#topicconfigs