Distribute messages equally into partitions in kafka - apache-kafka

Question

I am new in Kafka, so i have some issues related on basic things for Kafka. I wanted to distribute all messages equally to all over partitions.
As I know, Producer chose the partition based on key hashing (If key is available) using default Partitioner hash algorithm (Random, Consistent, Murmur2, sticky etc.). Which is great. But I want to distribute the messages to all partitions. Like:
Topic: "Test"
Partition: 3
Now, If i produce messages (Key Available) then I want to distribute those messages equally like:
Partition 1: 1,4,7,10
Partition 2: 2,5,8
Partition 3: 3,6,9
So, how can i distribute messages equally to all partition

Answer 1

The default partitioner chooses partition based on the hash of key if a key is available and no partition is specified in the record itself. Otherwise (i.e. no key is present and no partition is specified) it chooses the partition in a round-robin fashion (Kafka<2.4, read below).
public int partition(String key, int partitionNum) {
byte[] keyBytes = key.getBytes();
return toPositive(murmur2(keyBytes)) % partitionNum;
}
For a handful number of keys, using the default partitioner may not give you even data distribution, as toPositive(murmur2(keyBytes)) % numberOfPartitions will have collisions. The best way is for the producer to take responsibility and decide which partition to send the message to using CustomPartitioner based on your business use-case.
Kafka guarantees that any consumer of a given topic-partition will always read that partition's events in exactly the same order as they were written.
https://kafka.apache.org/documentation.html#introduction
One thing to note here is, that although eliminating data skewness is important - The order of messages going in different partitions in a topic may or may not be in-order - this may have consequences based on your use-case. But within a Partition will they'll be stored in the order, Thus keep related messages in the same partition.
For e.g. In an E-commerce delivery-related environment, Messages related to an OrderID should come in order (you don't want "Out-For-Delivery" to be after "Delivered"), thus messages for specific order_id should go into the same partition.
Update:
As mentioned in the comment, Kafka ≥ v2.4 uses Sticky Partitioner as the default partitioner.
The sticky partitioner addresses the problem of spreading out records without keys into smaller batches by picking a single partition to send all non-keyed records. Once the batch at that partition is filled or otherwise completed, the sticky partitioner randomly chooses and “sticks” to a new partition. That way, over a larger period of time, records are about evenly distributed among all the partitions while getting the added benefit of larger batch sizes.
https://www.confluent.io/blog/apache-kafka-producer-improvements-sticky-partitioner/
This means Kafka producers don’t immediately send records but keeps a batch of records for a specific topic with no keys and no assigned partition and will send to the same partition until the batch is ready to be sent. When a new batch is created, a new partition is chosen.
Effectively, the partitioner assigns records to the same partition until the batch is sent based on batch.size and linger.ms, once that batch is sent, a new partition will be used. Thus messages may not necessarily be evenly distributed.
Further Reading:
https://cwiki.apache.org/confluence/display/KAFKA/KIP-480%3A+Sticky+Partitioner
https://cwiki.apache.org/confluence/display/KAFKA/KIP-794%3A+Strictly+Uniform+Sticky+Partitioner#KIP794:StrictlyUniformStickyPartitioner-UniformStickyBatchSize
https://www.confluent.io/blog/5-things-every-kafka-developer-should-know/#tip-2-new-sticky-partitioner
https://aiven.io/blog/balance-data-across-kafka-partitions#challenge-of-uneven-record-distribution

Answer 2

I think this answers your question best:
https://rajatjain-ix.medium.com/whats-wrong-with-kafka-b53d0549677a
So, there are two solutions available..
You don't specify any partition_key. In this case, the DefaultPartitioner will automatically round-robin the messages across the partitions.
You use a (incremental uuid) % (count of partitions) as the partition number in Producer API. This way you are manually telling it to round-robin the messages to partitions.

Answer 3

Ronak explained very precisely.
You could achieve distribution of the messages over partitions evenly by implementing Partitioner interface regardless of the key.
New sticky version
public class SimplePartitioner implements Partitioner {
private final StickyPartitionCache stickyPartitionCache = new StickyPartitionCache();
public void configure(Map<String, ?> configs) {
}
#Override
public int partition(String topic, Object key, byte[] keyBytes, Object value, byte[] valueBytes, Cluster cluster) {
return partition(topic, key, keyBytes, value, valueBytes, cluster, cluster.partitionsForTopic(topic).size());
}
public int partition(String topic, Object key, byte[] keyBytes, Object value, byte[] valueBytes, Cluster cluster,
int numPartitions) {
return stickyPartitionCache.partition(topic, cluster);
}
#Override
public void close() {
}
}
Old version - see this link: hhttps://github.com/sharefeel/kafka-simple-partitioner/blob/0.8.2/SimplePartitioner.java
Don't forget this. Target partitions of SimplePartitioner and DefaultPartitioner are not same. But normally same.
If key is given, DefaultPartitioner will return one number from 0 to numPartition-1.
But SimplePartition always returns number of stikyPartitionCache.partitionCache.partition()'s value.
If there's an unavailable partition (all replicas of that parition down), producing will fail with DefaulPartitioner. But Simpartition can make producing success.
I tested about this with old version of SimplePartitioner but did not with newer one.