A general question. Assume a topic has 3 kafka partitions on different servers (brokers), each partition has 10 message with offset as its timestamp (0,1,...,9, greater number means stayed shorter time in partition, also means is newly came message). Let's say one partition happen to shut down since the server is done. What's the strategy for Kafka to re-balance the 10 message in the shut down partition into other partitions?
Visually, we have
broker 1 partition: |1-0|1-1|1-2|1-3|1-4|1-5|1-6|1-7|1-8|1-9|
broker 2 partition: |2-0|2-1|2-2|2-3|2-4|2-5|2-6|2-7|2-8|2-9|
broker 3 partition: |3-0|3-1|3-2|3-3|3-4|3-5|3-6|3-7|3-8|3-9|
Now if broker 3 is done, how will 3-0 to 3-9 be inserted into broker 1 and broker 2?
( My assumption is by default it will be spread half half randomly and inserted based on timestamp of broker 3, attached to tail of broker 1 and 2, and maybe there is somewhere one can configure behavior by code?)
Thanks in advance.
If a partition only exists on a single broker (replication factor 1) then when this broker is offline, the partition is not available. This is what you drew in your question.
To keep data available even when brokers go down you have to create topics with a replication factor greater than 1.
Then the data of the partition will be replicated onto several brokers and if one of them go offline, user traffic will be rediected to the available replicas.
I suggest you to go through the Replication section in the docs to understand how this works.
The below diagram will help you understand how Kafka replicates partitions. If one broker is down, the consumer can read from the other broker because Kafka has a replication ability. (Of course, you need to set it like below)
For example, if broker 1 dies, broker 2 will become a leader of topic1-part1, and a consumer can read from it.
Zookeeper will know if a broker( partition) is down, it will appoint another leader.
Related
For example, I have a topic that has 2 partitions and a producer using defaultpartitioner (round-robin I assumed) writes to the topic. At some point, partition 1 becomes unavailable because all of the replica brokers go offline. Assuming the messages have no specified keys, will the producer resend the messages to partition 2? or simply gets stuck?
That is an interesting question and we should look at it from a broader (cluster) perspective.
At some point, partition 1 becomes unavailable because all of the replica brokers go offline.
I see the following scenarios:
All replica brokers of partition one are different to the replica brokers of partition two.
All replica brokers of partition one are the same as for partition two.
Some replica brokers are the same as for partition two.
In scenario "1" it means you still have enough brokers alive as the replication factor is a topic-wide not a partition-based configuration. In that case as soon as the first broker goes down its data will be moved to another broker to ensure that your partition always has enough in-sync replicas.
In scenarios "2" both partitions become unavailable and your KafkaProducer will eventually time out. Now, it depends if you have other brokers that are alive and can take on the data of the partitions.
In scenario "3" the dead replicas would be shifted to running brokers. During that time the KafkaProducer will only write to partition 2 as this is the only available partition in the topic. As soon as partition 1 has enough in-sync replicas the producer will start producing again to both partitions.
Actually, I could think of many more scenarios. If you need a more concrete answer you need to specify
how many brokers you have,
what your replication factor actually is and
in what timely order which broker goes down.
Assuming the messages have no specified keys, will the producer resend the messages to partition 2?
The KafkaProducer will not re-send the data that was previously send to partition 1 to partition 2. Whatever was written to partition 1 will stay in partition 1.
Suppose default.replication.factor is set to one. For the sake of simplicity, let's say we have a topic with only one partition. We have a Kafka setup with three brokers. The topic we are interested in lives on the broker that just went down. Obviously, we won't have access to messages on this topic until the broker will be brought back, but my question is what will happen to messages for this topic that will come from producers while the broker is down? Will they be rejected?
The Producer will not be able to locate the primary replica of the partition because there will be no available primary replica because there will be no ISR's (in-sync replicas) at failover time. There will be an error, but I'm not exactly sure it's on the send, especially if you're batching sends.
Can you explain what kafka-topics.sh --describe is showing? me
I am following a tutorial video and also was reading the Apache documentation but I need a little more clarify as to what I'm looking at for the following columns in this graphic.
Leader: Is this pointing to the 3rd broker or is this pointing to the 3rd partition [2]?
Replicas: Is this pointing to brokers:partitions?
Isr: Is this pointing to brokers:partitions?
I would greatly appreciate it if someone explains what the columns A, B , C, D are.
Topic name: "install_test2"
4 partitions (partition 0, partition 1, partition 2, partition 3) and your replication factor for this topic is 2. It means that data in your topic will be stored (replicated) in 2 brokers for redundancy. In Kafka every partition has a leader and all the requests from producers and consumers are sent to the leader.
Leader column (column B in your image) shows broker ids of the leader for each partition. (Kafka evenly distributes partition leadership between brokers for load balancing)
Replicas column (column C in your image) shows ids of brokers that replicates data for each partition. The first id represents preferred leader. It means Kafka will try to make this broker leader of partition.
ISR (column D in your image) means in-sync-replica. In Kafka when a message is sent to a topic-partition (firstly message is received and stored in leader) and if you have replication factor greater than 1, then replica broker(s) send fetch request and this data is replicated to other broker(s). A follower (replica) broker is in-sync if it is not far behind the leader (explained in below). If a partition leader fails, Kafka chooses an ISR as the new leader for failover.
From Kafka docs:
Configuration parameter replica.lag.time.max.ms now refers not just to
the time passed since last fetch request from replica, but also to
time since the replica last caught up. Replicas that are still
fetching messages from leaders but did not catch up to the latest
messages in replica.lag.time.max.ms will be considered out of sync.
Trying to Understand the relationship between replication factor and Consumer group . Example : Number of partition = 2 Number of replication = 3 Number consumers in consumer group = 4 . In this case ,
How many consumer will receive the message ?
How This replication will impact the number of consumer to receive .
For your first question, since you have two partitions in your example, only 2 of the 4 consumers will actually get data. The other two consumers will not have any partitions assigned to them, because there aren't any partitions left for that consumer group. If you had a different consumer group, then those consumers would still be assigned partitions.
Additionally, in this case, you mention there's only a single message coming through. Depending on which partition it's assigned to, the message will only be sent to that partition. So in this case, only one of the four consumers will get the message, the one that had that partition assigned to it.
As for your second question, replication factor configuration in Kafka doesn't impact the number of messages consumers receive. Replication, as far as consumers and producers are concerned, is an internal kafka cluster detail that they don't need to worry about. As long as they're producing/consuming to/from the leader of the partition, that's all they need to know. A topic could have replication factor 2, and another one could have replication factor 10, and they would both behave identically to producers and consumers.
There's a few more details in the official Kafka documentation: https://kafka.apache.org/documentation/#theconsumer
To give some additional details on the replication factor, it doesn't have any relation whatsoever to the number of consumers receiving messages from the topic. Replication serves only one major purpose, and that is High Availability. So, let's say you have 3 brokers in a cluster, and for a topic my-topic you've set replication factor as 2. Now, if at-most one broker goes down at some point of time, you'd still be okay, as the messages are replicated in another broker for the topic.
From the Kafka documentation and from some other blogs I read, I concluded that one kafka-broker consists of one partition of topics. Here It says one Kafka-broker holds only one partition. I have only one broker in my system, But I can able to create a topic with 3 partitions and 1 replication factor. I also tried to create a topic with 3 partitions and 3 replication factor with only one broker. It throws below error
Error while executing topic command : replication factor: 3 larger than available brokers: 1
[2017-10-21 15:35:25,928] ERROR org.apache.kafka.common.errors.InvalidReplicationFactorException: replication factor: 3 larger than available brokers: 1
(kafka.admin.TopicCommand$).
So I have a query.
Whether Kafka-broker holds replication instead of a partition?
If I create 3 partitions with a single broker, what happens?
In such case of 1 broker, 1 replica and 3 partition , how many partitions of single topic can kafka-broker hold?
Somebody, please explain what happens here.
The post you are reffering to doesn't say that one broker can store only one partitions. It just says that partition is not splittable between brokers (topic is). Actually I manage a brokers with thousands of partitions. So, for your questions:
Kafka brokers hold many partitions. Replication is the way to store multiple copies of partitions across cluster.
If you create topic with 3 partitions on single node cluster, the broker will hold the data for all the partitions. Replication is not possible since it requires more nodes.
All of them.
Summary: The replication factor has to be equal or less in number compared to the number of brokers you have.