I just started exploring kafka.
I have query regarding kafka topic and partitions.
Let assume we have 3 machines x.x.x.1 , x.x.x.2 , x.x.x.3
We have a Topic Test and it has 3 partitions and 3 replica set as the 3 machines above 1,2,3.
Can it be possible that we can write 1st partition data to machine 1. 2nd partition data to machine 2 and 3rd partition data to 3rd machine always?
If it is possible than how?
The way the partition assignment works is the following.
Starting from a random broker-id (which could not be x.x.x.1 but x.x.x.3 instead), the partition leader 0 will be assigned to such broker, the partition leader 1 to the next one and so on.
So for example, if the broker x.x.x.2 is chosen then partition leader 0 will be on it (2), partition leader 1 on x.x.x.3 and finally partition leader 2 on x.x.x.1.
Regarding the follower replicas they are assigned increasing by one the starting broker : in this example the first follower for partition 0 will be on x.x.x.3, the second follower will be on x.x.x.1. The same will happen for follower replicas for partition 1 and 2. In this way the replication allows HA and the traffic is balanced across the cluster.
Btw there is a tool named "kafka-reassign-partitions.sh" you can use for specifying your preferred assignment through a JSON. You can find more here : https://cwiki.apache.org/confluence/display/KAFKA/Replication+tools#Replicationtools-4.ReassignPartitionsTool
Related
For example, I have a topic that has 2 partitions and a producer using defaultpartitioner (round-robin I assumed) writes to the topic. At some point, partition 1 becomes unavailable because all of the replica brokers go offline. Assuming the messages have no specified keys, will the producer resend the messages to partition 2? or simply gets stuck?
That is an interesting question and we should look at it from a broader (cluster) perspective.
At some point, partition 1 becomes unavailable because all of the replica brokers go offline.
I see the following scenarios:
All replica brokers of partition one are different to the replica brokers of partition two.
All replica brokers of partition one are the same as for partition two.
Some replica brokers are the same as for partition two.
In scenario "1" it means you still have enough brokers alive as the replication factor is a topic-wide not a partition-based configuration. In that case as soon as the first broker goes down its data will be moved to another broker to ensure that your partition always has enough in-sync replicas.
In scenarios "2" both partitions become unavailable and your KafkaProducer will eventually time out. Now, it depends if you have other brokers that are alive and can take on the data of the partitions.
In scenario "3" the dead replicas would be shifted to running brokers. During that time the KafkaProducer will only write to partition 2 as this is the only available partition in the topic. As soon as partition 1 has enough in-sync replicas the producer will start producing again to both partitions.
Actually, I could think of many more scenarios. If you need a more concrete answer you need to specify
how many brokers you have,
what your replication factor actually is and
in what timely order which broker goes down.
Assuming the messages have no specified keys, will the producer resend the messages to partition 2?
The KafkaProducer will not re-send the data that was previously send to partition 1 to partition 2. Whatever was written to partition 1 will stay in partition 1.
I'm trying to understand how does the no of partitions would affect the partition leader (broker)?
Let's say, I've a kafka cluster with 1 zookeeper, 3 brokers and 1 schema registry. My topic replication factor is 1. Now, If I've two topics A and B with 5 partitions.
Now, let's say if i send a message to topic A with key key1 and assume that based on the partitioning strategy, it is ended up being redirected to partition 5 of topic A and the leader for the partition 5 of topic A is broker 2.
In this scenario, If i send a message to topic B with key key1 (same as key as the message that was sent onto Topic A), then can we assume that it would go to partition 5 on the broker 2?
There is no correlation between leadership and partitioning
You can guarantee that the same key will be hashed the same, and go to the same partition (assuming matching counts), but you cannot guarantee which broker will be the leader
Can you explain what kafka-topics.sh --describe is showing? me
I am following a tutorial video and also was reading the Apache documentation but I need a little more clarify as to what I'm looking at for the following columns in this graphic.
Leader: Is this pointing to the 3rd broker or is this pointing to the 3rd partition [2]?
Replicas: Is this pointing to brokers:partitions?
Isr: Is this pointing to brokers:partitions?
I would greatly appreciate it if someone explains what the columns A, B , C, D are.
Topic name: "install_test2"
4 partitions (partition 0, partition 1, partition 2, partition 3) and your replication factor for this topic is 2. It means that data in your topic will be stored (replicated) in 2 brokers for redundancy. In Kafka every partition has a leader and all the requests from producers and consumers are sent to the leader.
Leader column (column B in your image) shows broker ids of the leader for each partition. (Kafka evenly distributes partition leadership between brokers for load balancing)
Replicas column (column C in your image) shows ids of brokers that replicates data for each partition. The first id represents preferred leader. It means Kafka will try to make this broker leader of partition.
ISR (column D in your image) means in-sync-replica. In Kafka when a message is sent to a topic-partition (firstly message is received and stored in leader) and if you have replication factor greater than 1, then replica broker(s) send fetch request and this data is replicated to other broker(s). A follower (replica) broker is in-sync if it is not far behind the leader (explained in below). If a partition leader fails, Kafka chooses an ISR as the new leader for failover.
From Kafka docs:
Configuration parameter replica.lag.time.max.ms now refers not just to
the time passed since last fetch request from replica, but also to
time since the replica last caught up. Replicas that are still
fetching messages from leaders but did not catch up to the latest
messages in replica.lag.time.max.ms will be considered out of sync.
A general question. Assume a topic has 3 kafka partitions on different servers (brokers), each partition has 10 message with offset as its timestamp (0,1,...,9, greater number means stayed shorter time in partition, also means is newly came message). Let's say one partition happen to shut down since the server is done. What's the strategy for Kafka to re-balance the 10 message in the shut down partition into other partitions?
Visually, we have
broker 1 partition: |1-0|1-1|1-2|1-3|1-4|1-5|1-6|1-7|1-8|1-9|
broker 2 partition: |2-0|2-1|2-2|2-3|2-4|2-5|2-6|2-7|2-8|2-9|
broker 3 partition: |3-0|3-1|3-2|3-3|3-4|3-5|3-6|3-7|3-8|3-9|
Now if broker 3 is done, how will 3-0 to 3-9 be inserted into broker 1 and broker 2?
( My assumption is by default it will be spread half half randomly and inserted based on timestamp of broker 3, attached to tail of broker 1 and 2, and maybe there is somewhere one can configure behavior by code?)
Thanks in advance.
If a partition only exists on a single broker (replication factor 1) then when this broker is offline, the partition is not available. This is what you drew in your question.
To keep data available even when brokers go down you have to create topics with a replication factor greater than 1.
Then the data of the partition will be replicated onto several brokers and if one of them go offline, user traffic will be rediected to the available replicas.
I suggest you to go through the Replication section in the docs to understand how this works.
The below diagram will help you understand how Kafka replicates partitions. If one broker is down, the consumer can read from the other broker because Kafka has a replication ability. (Of course, you need to set it like below)
For example, if broker 1 dies, broker 2 will become a leader of topic1-part1, and a consumer can read from it.
Zookeeper will know if a broker( partition) is down, it will appoint another leader.
From the Kafka documentation and from some other blogs I read, I concluded that one kafka-broker consists of one partition of topics. Here It says one Kafka-broker holds only one partition. I have only one broker in my system, But I can able to create a topic with 3 partitions and 1 replication factor. I also tried to create a topic with 3 partitions and 3 replication factor with only one broker. It throws below error
Error while executing topic command : replication factor: 3 larger than available brokers: 1
[2017-10-21 15:35:25,928] ERROR org.apache.kafka.common.errors.InvalidReplicationFactorException: replication factor: 3 larger than available brokers: 1
(kafka.admin.TopicCommand$).
So I have a query.
Whether Kafka-broker holds replication instead of a partition?
If I create 3 partitions with a single broker, what happens?
In such case of 1 broker, 1 replica and 3 partition , how many partitions of single topic can kafka-broker hold?
Somebody, please explain what happens here.
The post you are reffering to doesn't say that one broker can store only one partitions. It just says that partition is not splittable between brokers (topic is). Actually I manage a brokers with thousands of partitions. So, for your questions:
Kafka brokers hold many partitions. Replication is the way to store multiple copies of partitions across cluster.
If you create topic with 3 partitions on single node cluster, the broker will hold the data for all the partitions. Replication is not possible since it requires more nodes.
All of them.
Summary: The replication factor has to be equal or less in number compared to the number of brokers you have.