I want to use key/value pattern writing to Kafka in order to keep the same order of data writing while reading it.
My question is should the number of partitions in the topic be equal to the number of different keys in the incoming data.
I already know that with the key/value pattern data having the same key will go to the same partition.
Hence if number of partitions is not equal to the number of different keys in data, we can have data having different keys in the same partition? In this case how data order is kept?
My question is should the number of partitions in the topic be equal to the number of different keys in the incoming data.
I don't think that this is generally a good idea. It totally depends on the data you are processing. In case you have a fixed amount of keys (such as female, male and diverse) it might make sense. However, even then you need to be careful as the this could lead to an unbalance of data load over the broker as there might be less diverse. So you could end up having most of the data in one partition whereas the other partition(s) would be left empty. In general, the amount of partitions should be adjusted to your throughput requirements.
Hence if number of partitions is not equal to the number of different keys in data, we can have data having different keys in the same partition? In this case how data order is kept?
Yes, you could end up having different key in the same partition. Then the ordering is kept for this particular partition but not guaranteed in the topic overall.So assume, you have the keys A, B, and C and a topic with two partitions. A and C goes to the first partition and B is stored in the second partition. If data is flowing like this:
A/V1, A/V2, B/V1, C/V1, B/V2
Then your partitions will be filled like this:
partition0: A/V1, A/V2, C/V1
partition1: B/V1, B/V2
When consuming this topic it is not clear how the ordering between A and C messages are in relation to B messages. However, it is always guaranteed that the message A/V1 is consumed before A/V2, A/V2 before C/V1, and B/V1 before B/V2.
If you are looking for a more flexible way of directing your messages into partitions you can also think of writing a custom partitioner.
From Kafka docs:
Each partition is an ordered, immutable sequence of records that is
continually appended to a structured commit log. The records in the
partitions are each assigned a sequential id number called the offset
that uniquely identifies each record within the partition.
Messages sent by a producer to a particular topic partition will be
appended in the order they are sent. That is, if a record M1 is sent
by the same producer as a record M2, and M1 is sent first, then M1
will have a lower offset than M2 and appear earlier in the log.
A
consumer instance sees records in the order they are stored in the
log.
These are basic rules about Kafka and sending messages with different keys to same partition will not change this. You can even send all messages to same partition still the first message will be appended to the log before subsequent ones and will have lower offset value. Therefore order will be preserved.
Related
We want to put messages/records of a different customers on different partitions of a kafka topic.
But number of customers is not known in prior. So how can we set partition count for kafka topic in this case? Do we need any other way where partition count changes at runtime based on keys (customer_id in this case). Thanks in advance.
need to know number of partitions
Assuming Java, use AdminClient.describeTopics() method call and get partitions of each response object.
Regarding the rest of the question, consumer instances automatically distribute partition assignment when subscribing to topics.
Producers should not know about consumers, so you don't "put records on partitions" based on any factor of (possible) consumers.
partition count changes at runtime based on keys (customer_id)
Unclear what this means. Partition count can only increase, and if you do increase it, then your partitions will become unordered, so you should consider how large your keyspace is before creating the topic. For example, if you have a numeric ID, and use the first two digits as the partition value, then you could create a topic up to 100 partitions.
If I have 1 Kafka topic with 1 partition and multiple sources are posted in the same partition. What happens if 2 servers are trying to post in the same partition at the same time? Would it mix the information between both of those servers or one of them would wait until the other finishes?
The producers will mix the messages in the partition.
As per theory, events will be guaranteed to be appended in order per partition per producer. But if we are talking about multiple producers, then the behaviour will depend on the configuration set at the producer side. In particular, max.in.flight.requests.per.connection = 1. The reason being is if there are multiple in flight events and the first one failed, the second will get appended to the log earlier, thus breaking the ordering.
Have a glance at https://blog.softwaremill.com/does-kafka-really-guarantee-the-order-of-messages-3ca849fd19d2
If somehow keys are same for both sources and every record, all of them will be recorded in the same partition (other partitions will remain empty)
If every source has a different key from each other but this key is used for every message from same source, then messages from different sources will be recorded at different partitions (if partition count is no less than source count).
If each value has a different key, regardless of sources, still kafka will mix them in partitions as I know.
In short, keys determine the partition of a message. Values with same key go to same partition. If every record has a unique key, Kafka will apply Round-Robin for incoming messages and each partition will have almost same amount of records.
I want to share a problem and a solution I used, as I think it may be beneficial for others, if people have any other solutions please share.
I have a table with 1,000,000 rows, which I want to send to kafka, and spread the data between 20 partitions.
I want to notify the consumer when producer reached end of data, I don't want to have direct connection between producer and consumer.
I know kafka is designed as logical endless stream of data, but I still need to mark the end of the specific table.
There was a suggestion to count the number of items per logical section, and send this data (to a metadata topic), so the consumer will be able to count items, and know when the logical section ended.
There are several disadvantages for this approach:
As data is spread between partitions, I can tell there are total x items at my logical section, however if there are multiple consumers (one per partition), they'll need to share a counter of consumed messages per logical section. I want to avoid this complexity. Also when consumer is stopped and resumed, it will need to know how many items were already consumed and keep context.
Regular producer session guarantees at-least-once delivery, which means I may have duplicated messages. Counting the messages will need to take this into account (and avoid counting duplicated messages).
There is also the case where I don't know in advance the number of items per logical session, (I'm also kind of consumer, consuming stream of event and signaled when data ended), so at this case, the producer will also need to have a counter, keep it when stopped and resumed etc. Having several producers will need to share the counter etc. So it adds a lot of complexity to the process.
Solution 1:
I actually want the last message at each partition indicate it is the last message.
I can do some work in advance, create some random message keys, send messages partitioned by key, and test to which partition each message is directed. As partitioning by keys is deterministic (for given number of partitions), I want to prepare a map of keys and the target partition. For example key: ‘xyz’ is directed to partition #0, key ‘hjk’ is directed to partition #1 etc, and finally have the reversed map, so for partition 0, use key ‘xyz’, for partition 1, use key ‘hjk’ etc.
Now I can send the entire table (except of the last 20 rows) with partition strategy random, so the data is spread between partitions, for almost entire table.
When I come to the last 20 rows, I’ll send them using partition key and I’ll set for each message partition key which will hash the message to a different partition. This way, each one of the 20 partitions will get one of the last 20 messages. For each one of the last 20 messages, I'll set a relevant header which will state it is the last one.
Solution 2:
Similar to solution 1, but send the entire table spread to random partitions. Now send 20 metadata messages, which I’ll direct to the 20 partitions using the partition by key strategy (by setting appropriate keys).
Solution 3:
Have additional control topic. After the table was sent entirely to the data topic, send a message to the control topic saying table is completed. The consumer will need to test the control topic from time to time, when it gets the 'end of data' message, it will know that if it reached the end of the partition, it actually reached the end of the data for that partition. This solution is less flexible and less recommended, but I wrote it as well.
Another one solution is to use open source analog of S3 (e.g. minio.io). Producers can uplod data, send message with link to object storage. Consumers will remove data frome object storage after collecting.
I had a few questions from Kafka. Please help me in understanding the problem.
As per official documentation, each partition will have one unique sequential id which called offset.
How does the offset numbers will be generated i.e based on the message arrival into a partition or offset numbers will be generated whenever the partitions are created?
do the same offset ID/number generates/exists in another partition because each partition is independent each other?
If the same offset can be possible in another partition then, How consumer uniquely identifies the message across multiple partitions?
How does consumer know the particular offset belongs to a particular partition? Please let me understand in both situations like a message with key & without a key?
Each partition maintains the messages it has received in a sequential order where they are identified by an offset. This offset is a sequential number and it automatically generated and assigned to messages.
Yes this is correct. Message ordering is guaranteed only on the partition level. This means that if you have a topic with multiple partitions, messages on different partitions might have the same offset. Therefore, an offset has a true meaning only within a single partition (as you can also see in the picture below, which is taken from Kafka Docs).
3/4. The consumers are subscribed to topics, but behind the scenes they are subscribed to particular partitions (well, if you have a single consumer in the consumer group it will subscribe to all of the partitions). Therefore, when the consumer reads messages from a particular partition, it can uniquely identify messages using their unique offsets which are maintained throughout the partition. As I already mentioned, the message order is guaranteed only within a single partition.
Note that messages without key, will be evenly distributed across the partitions of the topic, in a round-robin fashion. On the other hand, messages with the same key will be stored in the same partition and hence, you can use the key to store and order messages having the same key. For example, if you need to process users and you'd like order guarantee for each distinct user, you can use userID as a key, so that all the events of that user are stored in the same partition. Later on, you will be able to consume these user-specific messages, in the order they were originally received.
I have a topic with 10 partitions, 1 consumer group with 4 consumers and worker size is 3.
I could see there is an uneven distribution of messages in the partitions, One partition is having so much data and another one is free.
How can I make my producer to evenly distribute the load into all the partitions, so that all partitions are being utilized properly?
According to the JavaDoc comment in the DefaultPartitioner class itself, the default partitioning strategy is:
If a partition is specified in the record, use it.
If no partition is specified but a key is present choose a partition based on a hash of the key.
If no partition or key is present choose a partition in a round-robin fashion.
https://github.com/apache/kafka/blob/trunk/clients/src/main/java/org/apache/kafka/clients/producer/internals/DefaultPartitioner.java
So here are two possible reasons that may be causing the uneven distribution, depending on whether you are specifying a key while producing the message or not:
If you are specifying a key and you are getting an uneven distribution using the DefaultPartitioner, the most apparent explanation would be that you are specifying the same key multiple times.
If you are not specifying a key and using the DefaultPartitioner, a non-obvious behavior could be happening. According to the above you would expect round-robin distribution of messages, but this is not necessarily the case. An optimization introduced in 0.8.0 could be causing the same partition to be used. Check this link for a more detailed explanation: https://cwiki.apache.org/confluence/display/KAFKA/FAQ#FAQ-Whyisdatanotevenlydistributedamongpartitionswhenapartitioningkeyisnotspecified? .
Instead of going for the default partitioner class you can assign the producer with a partition number so that message directly goes to the specified partition,
ProducerRecord<String, String> record = new ProducerRecord<String, String>(topicName, partitionNumber,key, value);
Seems like your problem is uneven consumption of messages rather than uneven producing of messages to Kafka topic. In other words, your amount of reading threads doesn't match amount of partitions you have (they do not need to match 1:1 though, only be the same amout of partitions to read from per each consumer thread).
See short explanation for more details.
You can make use of the key parameter of the producer record. Here is a thing that for a specific key the data goes in to the same partition always now, I don’t know the structure of your producer record but as you said you have 10 partition then you can use simply n%10 as your producer record key.
Where n is 0 to 9 now your for record 0 key will be 0 and then kafka will generate a hash key and put it in some partition say partition 0, and for record 1 it will be one and then it will go into the 1st partition and so on.
This way you will able to apply round robin on your producer record your key will be independent from the fields in your record so you can have a variable n and key as n%10.
Or you can specify the partition in your producer record. So either you use the key or the partition field of the producer record.
If you have defined partitioner from record let's say in Kafka key is string and value is student Pojo.
In student Pojo let's say based on student country field, I want to go in a specific partition. Imagine that there is 10 partitions in a topic and for example, in value, "India" is a country and based on "India" we got partition number 5.
Whenever country is "India", Kafka will allocate the 5 number partition and that record goes to the partition number 5 always (if the partition has not changed).
Let's say that in your pipeline there are lots of records which are coming and have a country "India", all those records will go to partition number 5, and you will see uneven distribution in Kafka partition.
In my case, I used the default partitioner but still had much much more records in one partition than in others. The problem was I unexpectedly had many records with the same key. Check your keys!
As I was unable to resolve this with Faust, the approach I am using is to implement the 'round-robin' distribution myself.
I iterate over my records to produce and do for example:
for index, message in enumerate(messages):
topic.send(message, partition=index % num_partitions)
I.e. bound my index to within the range of partitions I have.
There could still be unevenness - consider you repeatedly run this but your number of records is less than your num_partitions - then your first partitions will keep getting the major share of messages. You can avoid this issue by adding a random offset:
import random
initial_partition = random.randrange(0, num_partitions)
for index, message in enumerate(messages):
topic.send(message, partition=(initial_partition + index) % num_partitions)