Try to understanding consistency maintenance in Kafka. Please find the scenario and help to understand.
Number of partition = 2
Replication factor = 3
Number of broker in the cluster = 4
In that case, for achieving the strong consistency how many nodes should acknowledge. Either ack = all or ack = 3 or any other value. Please confirm for the same.
You might be interested in seeing When it Absolutely, Positively, Has to be There talk from Kafka Summit.
Which was given by an engineer at Cloudera, and Cloudera has their own documenation on Kafka availability
To summarize, more than 1 replica and higher than 1 in-sync replica is a good start. Then on the producer, if you are okay with sacrificing throughput for data availability, meaning you must have all replicas be written before continuing, then acks=all. Otherwise, if you trust the leader broker to be highly available with unclean leader election is false, then acks=1 should be okay in most cases.
acks=3 isn't a valid config, by the way. I think you are looking for min.insync.replicas=2 and acks=all with a replication factor of 3; from above link
If min.insync.replicas is set to 2 and acks is set to all, each message must be written successfully to at least two replicas. This guarantees that the message is not lost unless both hosts crash
Also, you can enable the transactional producer, as of Kafka 0.11 to work towards exactly once processing
enable.idempotence=true
In your setting, what you have is
4 brokers
Replication factor = 3
That means each message in a given partition will be replicated to 3 out of 4 brokers, including the leader for that partition.
In-order to achieve strong consistency guarantees, you have to set min.insync.replicas to 2 and use acks=all. This way, you are guaranteed that each write goes to at-least 2 out of 3 brokers which hold the data, before which it is acknowledged.
Setting acks to all provides the highest consistency guarantee at the expense of slower writes to the cluster.
If you use older versions of Kafka where unclean leader election is true by default, you should also consider setting that to false explicitly. This way, an out of sync. broker won't be elected as the leader in case of leader crashes (effectively compromising availability).
Also, Kafka is a system where all the reads go through the leader. This is a bit different from some other distributed system such as zookeeper which supports read replicas. So you do not have a situation where a client ends up reading directly from a stale broker. Leader ensures that writes are ordered and replicated to designated number of in-sync replicas and acknowledged based on your acks setting.
If you are looking for consistency as in realm of ACID property, all replicas need to be acknowledged. Since you have 3 replicas, all of those 3 nodes should be acknowledged.
Related
I'm studying kafka and I see that a lot of articles about it are talking about kafka cluster with 3 nodes. And I'm curious why 3 nodes?
Do we have something technically specific about this number ?
This is all about high availability, the recommended replication factor for Kafka is 3, meaning that you are going to have 3 replicas for every piece of data sent to your cluster. If you have only 1 node, and that node crashes or has HW issues such as disk issues, probably you will lose all your data, as opposed to have healthy replicas on the other 2 nodes.
Here there are some additional explanations about it as well Kafka replicas
One of the reasons it's about electing the leader. One of the algorithms for electing a leader is Raft algorithm https://raft.github.io/ it will elect a new leader only when the major of brokers (n (alive) + 1 / 2 > all / 2) agree on one value. Long story short. If we have 3 brokers it's guaranteed that we will survive one failure. If there will be 2 brokers and one broker fails another broker will not be able to elect itself.
P.S. I'm not sure if kafka using the raft algorithm now for electing a leader (older versions used zookeeper). But I'm just writing an interesting case with electing leaders in distributed systems and if they use Raft so why it should be minimum 3.
The main developer of Kafka said Kafka is CA but P in CAP theorem. But I'm so confused, is Kafka not Partition tolerate? I think it does, when one replication is down the other would become leader and continue work!
Also, I would like to know what if Kafka uses P? Would P hurt C or A?
If you read how CAP defines C, A and P, "CA but not P" just means that when an arbitrary network partition happens, each Kafka topic-partition will either stop serving requests (lose A), or lose some data (lose C), or both, depending on its settings and partition's specifics.
If a network partition splits all ISRs from Zookeeper, with default configuration unclean.leader.election.enable = false, no replicas can be elected as a leader (lose A).
If at least one ISR can connect, it will be elected, so it can still serve requests (preserve A). But with default min.insync.replicas = 1 an ISR can lag behind the leader by approximately replica.lag.time.max.ms = 10000. So by electing it Kafka potentially throws away writes confirmed to producers by the ex-leader (lose C).
Kafka can preserve both A and C for some limited partitions. E.g. you have min.insync.replicas = 2 and replication.factor = 3, and all 3 replicas are in-sync when a network partition happens, and it splits off at most 1 ISR (either a single-node failures, or a single-DC failure or a single cross-DC link failure).
To preserve C for arbitrary partitions, you have to set min.insync.replicas = replication.factor. This way, no matter which ISR is elected, it is guaranteed to have the latest data. But at the same time it won't be able to serve write requests until the partition heals (lose A).
CAP Theorem states that any distributed system can provide at most two out of the three guarantees: Consistency, Availability and Partition tolerance.
According to the Engineers at LinkedIn (where Kafka was initially founded) Kafka is a CA system:
All distributed systems must make trade-offs between guaranteeing
consistency, availability, and partition tolerance (CAP Theorem). Our
goal was to support replication in a Kafka cluster within a single
datacenter, where network partitioning is rare, so our design focuses
on maintaining highly available and strongly consistent replicas.
Strong consistency means that all replicas are byte-to-byte identical,
which simplifies the job of an application developer.
However, I would say that it depends on your configuration and more precisely on the variables acks, min.insync.replicas and replication.factor. According to the docs,
If a topic is configured with only two replicas and one fails (i.e.,
only one in sync replica remains), then writes that specify acks=all
will succeed. However, these writes could be lost if the remaining
replica also fails. Although this ensures maximum availability of the
partition, this behavior may be undesirable to some users who prefer
durability over availability. Therefore, we provide two topic-level
configurations that can be used to prefer message durability over
availability:
Disable unclean leader election - if all replicas become unavailable, then the partition will remain unavailable until the most
recent leader becomes available again. This effectively prefers
unavailability over the risk of message loss. See the previous section
on Unclean Leader Election for clarification.
Specify a minimum ISR size - the partition will only accept writes if the size of the ISR is above a certain minimum, in order to prevent
the loss of messages that were written to just a single replica, which
subsequently becomes unavailable. This setting only takes effect if
the producer uses acks=all and guarantees that the message will be
acknowledged by at least this many in-sync replicas. This setting
offers a trade-off between consistency and availability. A higher
setting for minimum ISR size guarantees better consistency since the
message is guaranteed to be written to more replicas which reduces the
probability that it will be lost. However, it reduces availability
since the partition will be unavailable for writes if the number of
in-sync replicas drops below the minimum threshold.
CAP is a proofed theorem so there is no distributed system that can have features C, A and P altogether during failure. In case Kafka uses the P, that is when the cluster split into two or more isolate part it can continue the functioning, one of the C or A should be sacrificed.
Maybe if we consider Kafka and Zookeeper nodes as a whole cluster, because Kafka needs zookeeper nodes, we can not consider it partition tolerant in case of losing connection to zookeeper nodes.
in kafka, for
replication-factor = 2
minimum ISR size = 1
unclean.leader.election.enable = false
is there a chance that(like network partition), two broker think they'are leader and both accept write, so finally some msg lost? and the producer does't even notice this.
producer use acks = all
Similar question has been answered here :How does kafka handle network partitions?
In your case, I think there is no problem when network partitioning. Since unclean.leader.election.enable is false, one of two side cannot elect new leader so only the other side can accept write.
With minimum ISR set to 1, your cluster can have only a single broker with the data at any time, so if the disk of this broker was to blow up, you risk losing data.
If you want stronger guarantees, you need to increase the minimum ISR size. For example, if you set it to 2, at any time at least 2 brokers will have all the data. So in order to lose data in this configuration, you would need to lose the disks of both brokers within the same time frame which is a lot less likely than just losing a single disk.
If you increase minimum ISR, to ease maintenance, you probably also want to bump up the number of replicas so you can have 1 broker down and still be able to produce with acks = all.
Since you have replication factor as 2. Having 1 ISR out of two is sufficient. It means that Even if the leader goes down you have 1 replica to handle the transactions. Having more replicas will lead to higher write overhead and might slow down the throughput. You can have higher number of replicas at the cost of performance for reliability.
Kafka has the concept of a in-sync replica set, which is the set of nodes that aren't too far behind the leader.
What happens if the network cleanly partitions so that a minority containing the leader is on one side, and a majority containing the other in-sync nodes on the other side?
The minority/leader-side presumably thinks that it lost a bunch of nodes, reduces the ISR size accordingly, and happily carries on.
The other side probably thinks that it lost the leader, so it elects a new one and happily carries on.
Now we have two leaders in the same cluster, accepting writes independently. In a system that requires a majority of nodes to proceed after a partition, the old leader would step down and stop accepting writes.
What happens in this situation in Kafka? Does it require majority vote to change the ISR set? If so, is there a brief data loss until the leader side detects the outages?
I haven't tested this, but I think the accepted answer is wrong and Lars Francke is correct about the possibility of brain-split.
Zookeeper quorum requires a majority, so if ZK ensemble partitions, at most one side will have a quorum.
Being a controller requires having an active session with ZK (ephemeral znode registration). If the current controller is partitioned away from ZK quorum, it should voluntarily stop considering itself a controller. This should take at most zookeeper.session.timeout.ms = 6000. Brokers still connected to ZK quorum should elect a new controller among themselves. (based on this: https://stackoverflow.com/a/52426734)
Being a topic-partition leader also requires an active session with ZK. Leader that lost a connection to ZK quorum should voluntarily stop being one. Elected controller will detect that some ex-leaders are missing and will assign new leaders from the ones in ISR and still connected to ZK quorum.
Now, what happens to producer requests received by the partitioned ex-leader during ZK timeout window? There are some possibilities.
If producer's acks = all and topic's min.insync.replicas = replication.factor, then all ISR should have exactly the same data. The ex-leader will eventually reject in-progress writes and producers will retry them. The newly elected leader will not have lost any data. On the other hand it won't be able to serve any write requests until the partition heals. It will be up to producers to decide to reject client requests or keep retrying in the background for a while.
Otherwise, it is very probable that the new leader will be missing up to zookeeper.session.timeout.ms + replica.lag.time.max.ms = 16000 worth of records and they will be truncated from the ex-leader after the partition heals.
Let's say you expect longer network partitions than you are comfortable with being read-only.
Something like this can work:
you have 3 availability zones and expect that at most 1 zone will be partitioned from the other 2
in each zone you have a Zookeeper node (or a few), so that 2 zones combined can always form a majority
in each zone you have a bunch of Kafka brokers
each topic has replication.factor = 3, one replica in each availability zone, min.insync.replicas = 2
producers' acks = all
This way there should be two Kafka ISRs on ZK quorum side of the network partition, at least one of them fully up to date with ex-leader. So no data loss on the brokers, and available for writes from any producers that are still able to connect to the winning side.
In a Kafka cluster, one of the brokers is elected to serve as the controller.
Among other things, the controller is responsible for electing new leaders. The Replica Management section covers this briefly: http://kafka.apache.org/documentation/#design_replicamanagment
Kafka uses Zookeeper to try to ensure there's only 1 controller at a time. However, the situation you described could still happen, spliting both the Zookeeper ensemble (assuming both sides can still have quorum) and the Kafka cluster in 2, resulting in 2 controllers.
In that case, Kafka has a number of configurations to limit the impact:
unclean.leader.election.enable: False by default, this is used to prevent replicas that were not in-sync to ever become leaders. If no available replicas are in-sync, Kafka marks the partition as offline, preventing data loss
replication.factor and min.insync.replicas: For example, if you set them to 3 and 2 respectively, in case of a "split-brain" you can prevent producers from sending records to the minority side if they use acks=all
See also KIP-101 for the details about handling logs that have diverged once the cluster is back together.
Is there any tools or operation to use to mitigate data loss issues when kafka broker fail in multi node kafka cluster.
well, replication is an important features of Kafka and a key element to avoid data loss. In particular, should one of your broker go down, the replica on other brokers will be used by the consumers just as nothing happened (from the business side). Of course, this has consequences on the connections, band width etc.
However, a message must have been properly produced to be replicated.
So basically, if you have a replication set at higher than 1, this should be safe, as long as your producers don't go down.
The default.replication.factor is 1, so set replication (at the topic or general level) to 2 or 3. Of course you need 2 or 3 brokers.
http://kafka.apache.org/documentation.html#basic_ops_increase_replication_factor