Who manages ISR in Kafka? - apache-kafka

I understand Kafka has the ISR mechanism to manage leader-follower data replication, but I'm just wondering who exactly updates the ISR?


For each partition, the current leader tracks and manages the current in-sync replicas. Followers send Fetch requests to the leader to retrieve records. That enables the leader to keep track where each follower is and determine which ones are in-sync.
Before Kafka 2.7, each leader would update its partition state znode in ZooKeeper. Now (with KIP-497) when the in-sync replicas change, leaders send an AlterISR request to the controller and it's the controller that is responsible for updating ZooKeeper.

Related

A topic has 3 replicas and min.insyinc.replicas to 2, what happens when both the follower and in-syinc replica go down and the consumer wants to read?

I am preparing for the CCDAK certification and I stepped into this question:
"A topic has three replicas and you set min.insync.replicas to 2. If two out of three replicas are not available, what happens when a consume request is sent to broker?"
The given answer is:
Data will be returned by the remaining in-sync replica
I had a doubt over a corner case: what happens when both the leader and the in-sync replica go down at the same time (third replica still not in-sync) and the consumer wants to read a message not present in the third replica (it may has not been copied yet since it is a not in-sync replica)?
Would a practical example be something like this: let's say consumer has committed offset 10 and wants to read from 11 but third replica has messages up to offset 9. Leader and follower are down, what would happen? Will the consumer just wait for the other brokers or will Kafka throw an error I have to deal with?

Your consumer will not be rebalanced because from Kafka side, It can not select a leader for that partition. The partition is not Synced and then Kafka will wait to start of other replicas to select a leader. In this scenario, Your consumer will hang.

Kafka - Controller Broker

I come across this phrase from https://niqdev.github.io/devops/kafka/
and https://livebook.manning.com/book/kafka-streams-in-action/chapter-2/109 (Kafka Streams in Action )
The controller broker is responsible for setting up leader/follower relationships for all partitions of a topic. If a Kafka node dies or is unresponsive (to ZooKeeper heartbeats), all of its assigned partitions (both leader and follower) are reassigned by the controller broker.
I think it is not correct assignment of follower partitions to other brokers - as the partitions wont heal themselves unless the broker comes back . I know it ONLY happens for leader replica where if the broker that has leader replica gone down, one of the broker that contains follower will become leader. But, I dont think "reassigment" of followers will happen automatically unless reassignment is initiated manually. Please add your inputs

The terminology might be a little off indeed but still applies. Followers are not necessarily assigned to other brokers but they need to change the endpoint to where they are going to send fetch requests. The follower's job is to stay in-sync with the leader, and if the leader has been assigned to a new broker because the old one failed then the followers need to send their fetch requests to the new elected broker. I think that is what reassignment means in the context that you shared.

What happens to partition of Kafka topic with replication factor of 1 when a broker goes down?

Suppose default.replication.factor is set to one. For the sake of simplicity, let's say we have a topic with only one partition. We have a Kafka setup with three brokers. The topic we are interested in lives on the broker that just went down. Obviously, we won't have access to messages on this topic until the broker will be brought back, but my question is what will happen to messages for this topic that will come from producers while the broker is down? Will they be rejected?

The Producer will not be able to locate the primary replica of the partition because there will be no available primary replica because there will be no ISR's (in-sync replicas) at failover time. There will be an error, but I'm not exactly sure it's on the send, especially if you're batching sends.

Kafka's Replicas handling mechanism?

I have two questions.
I wonder how to sychronized leader partion and follower partions.
If leader partition receive a message, then the leader broadcasting to follower partition on background communication? but It seemed kafka config file does not include these features(synchronization port info etc.)
If assume the following architecture.
Two brokers - Two partition - Two replicas
Broker#1 - leader partition#1, follower partition#2
Broker#2 - leader partition#2, follower partition#1
Sending messages will be round-robin to these two brokers...
If message#1 go to Broker#1(partition#1) and Broker#1 was shut down,
then broker#2 open the follower partition#1 and broker#2 has active two leader partition (for delivering the message#1)?

This is already handled by Kafka. You only need to define the replication factor for a topic. According to Kafka docs,
The partitions of the log are distributed over the servers in the
Kafka cluster with each server handling data and requests for a share
of the partitions. Each partition is replicated across a configurable
number of servers for fault tolerance.
Each partition has one server which acts as the "leader" and zero or
more servers which act as "followers". The leader handles all read and
write requests for the partition while the followers passively
replicate the leader. If the leader fails, one of the followers will
automatically become the new leader. Each server acts as a leader for
some of its partitions and a follower for others so load is well
balanced within the cluster.
Your question is not clear. I believe my answer to this question should shed some light with regards to kafka partitions, distribution of messages and fault tolerance.

Does producers and consumers can write and read data, while we run kafka-reassign-partition tool to balance partition across brokers?

While we are running kafka-reassign-partition tool to balance partitions across brokers.
Does we have to stop producers and consumers to write and read data from kafka topic?

No we don't have to stop producers and consumers for partition re-assignment. Partition reassignment happens ,step by step , first the new assigned replicas will catch up with the leaders and once they become in-sync replicas ,then when preferred replica election happens ( either triggered manually or automatically triggered based on configuration) , then the depending on the leadership imbalance and new assignment, new leaders would be elected. Once new leaders are elected the producer and consumers would get an updated metadata, hence the producer and consumers continue to work as usual.