When there's a failure writing a message to all ISR replicas, but the message is already persisted on the leader, what happens?
Even if the request fails, is the data still is available for consumers?
Are consumers able to read "uncommitted" data?
The message is not visible to the KafkaConsumer until the topic configuration min.insync.replicas is fulfilled.
Kafka is expecting the failed replica broker to get up and running again, so the replication can complete.
Note, that this scenario is only relevant if your KafkaProducer configuration acks is set to all. When you set it to 0 or 1, the Consumer will be able to consume the message as soon as the data arrives at the partition leader.
In general, the client (here KafkaProducer) only communicates with the partition leader and depending on its mode such as synchronous, asynchronous or fire-and-forget and its acks configuration waits or does not wait for a reply. The replication of the data itself is independent of the producer and is handled by a (single-thread) on the partition leader broker. The leader will only notify on the success or failure on the replication and the leader/replicas continue to ensure to satisfy the replication set with the topic configuration replication.factor.
The producer may still try to resend the message in case the message was only acknowledged by producer but not by the replicas, depending on its acks setting and if the exception is retriable or not. In that case you will end up having duplicate values. You can avoid this by enabling idempotence.
Related
I know I can set acks=all in Kafka producer configuration to make producer to wait for acknowledgement from leader after all replicas receive the message sent. If the acknowledgement timeout occurs, producer retries sending message. This happens transparently without requiring any code changes. Is it possible to have some stats of those retries. Is it possible to know which message involved retries and how many retries. Does Kafka provide any kind of hook to be called before / after retry so that we can log some message?
I had a set up of 3 Node Zookeeper and 3 Broker Cluster when one of my brokers goes down in Cluster, the producer is not giving any Error but, consumers will throw an error saying that...
Marking coordinator Dead for the group... Discovered coordinator for
the group.
According to my knowledge if any one Broker available across the cluster I should not be stopped consuming messages.
But, as of now Server.1, server.2, server.3 if my server.2 goes down my all consumers stops consuming messages.
What are the exact parameters to set to achieve failover of producers and as well as consumers?
if my server.2 goes down my all consumers stops consuming messages.
For starters, you disable unclear leader election in the brokers, and create your topics with --replication-factor=3 and a configuration of min.insync.replicas=2.
To ensure that a producer has at least two durable writes (as set by the in-sync replcicas), then set acks=all
Then, if any broker fails, and assuming a leader election does not have any error, a producer and consumer should seemlessly re-connect to the new leader TopicPartitions.
If Kafka consumers consume messages in sync mode, but their dependent downstream service is broken(eg. cannot connect to DB). Kafka coordinator keeps triggering group rebalance, will all the consumers be dead in the end?
As far as consumer is responding for heart beat intervals, Kafka does not do the re-balancing. If suppose you consumer failed to respond for heart beat to zookeeper then Kafka assume the consumer is dead and perform re balancing. it is based on couple of properties here
heartbeat.interval.ms
The expected time between heartbeats to the consumer coordinator when using Kafka's group management facilities. Heartbeats are used to ensure that the consumer's session stays active and to facilitate rebalancing when new consumers join or leave the group. The value must be set lower than session.timeout.ms, but typically should be set no higher than 1/3 of that value. It can be adjusted even lower to control the expected time for normal rebalances.
session.timeout.ms
The timeout used to detect consumer failures when using Kafka's group management facility. The consumer sends periodic heartbeats to indicate its liveness to the broker. If no heartbeats are received by the broker before the expiration of this session timeout, then the broker will remove this consumer from the group and initiate a rebalance. Note that the value must be in the allowable range as configured in the broker configuration by group.min.session.timeout.ms and group.max.session.timeout.ms
zookeeper.session.timeout.ms
ZooKeeper session timeout. If the consumer fails to heartbeat to ZooKeeper for this period of time it is considered dead and a rebalance will occur.
I'm a bit confused on the Topic partitioning in Apache Kafka. So I'm charting down a simple use case and I would like to know what happens in different scenarios. So here it is:
I have a Topic T that has 4 partitions TP1, TP2, TP4 and TP4.
Assume that I have 8 messages M1 to M8. Now when my producer sends these messages to the topic T, how will they be received by the Kafka broker under the following scenarios:
Scenario 1: There is only one kafka broker instance that has Topic T with the afore mentioned partitions.
Scenario 2: There are two kafka broker instances with each node having same Topic T with the afore mentioned partitions.
Now assuming that kafka broker instance 1 goes down, how will the consumers react? I'm assuming that my consumer was reading from broker instance 1.
I'll answer your questions by walking you through partition replication, because you need to learn about replication to understand the answer.
A single broker is considered the "leader" for a given partition. All produces and consumes occur with the leader. Replicas of the partition are replicated to a configurable amount of other brokers. The leader handles replicating a produce to the other replicas. Other replicas that are caught up to the leader are called "in-sync replicas." You can configure what "caught up" means.
A message is only made available to consumers when it has been committed to all in-sync replicas.
If the leader for a given partition fails, the Kafka coordinator will elect a new leader from the list of in-sync replicas and consumers will begin consuming from this new leader. Consumers will have a few milliseconds of added latency while the new leader is elected. A new coordinator will also be elected automatically if the coordinator fails (this adds more latency, too).
If the topic is configured with no replicas, then when the leader of a given partition fails, consumers can't consume from that partition until the broker that was the leader is brought back online. Or, if it is never brought back online, the data previously produced to that partition will be lost forever.
To answer your question directly:
Scenario 1: if replication is configured for the topic, and there exists an in-sync replica for each partition, a new leader will be elected, and consumers will only experience a few milliseconds of latency because of the failure.
Scenario 2: now that you understand replication, I believe you'll see that this scenario is Scenario 1 with a replication factor of 2.
You may also be interested to learn about acks in the producer.
In the producer, you can configure acks such that the produce is acknowledged when:
the message is put on the producer's socket buffer (acks=0)
the message is written to the log of the lead broker (acks=1)
the message is written to the log of the lead broker, and replicated to all other in-sync replicas (acks=all)
Further, you can configure the minimum number of in-sync replicas required to commit a produce. Then, in the event when not enough in-sync replicas exist given this configuration, the produce will fail. You can build your producer to handle this failure in different ways: buffer, retry, do nothing, block, etc.
I have a topic called Topic1 with two partitions. Assume that serverA is leader for Topic1, partition1 and serverB is a follower.
What will happen if in my client I publish to serverB (in the broker.list I specify only serverB)? How does the message propagate? Is it sent to serverB and then to serverA.
I've found this document to be very helpful in explaining what's going on internally with Kafka.
There's an API used by the producer to ask any one Kafka server for the list of partitions and all metadata for those partitions. That metadata includes the leader broker for each partition. The producer calls the partitioner to get the target partition and then talks directly with that partition's Kafka broker leader to write the message onto the partition. The leader will handle communications with any other brokers managing replicas of the partition.
To publish a message to a partition, the client first finds the leader of the partition from Zookeeper and sends the message to the leader.
The leader writes the message to its local log. Each follower constantly pulls new messages from the leader using a single socket channel.The follower writes each received message to its own log and sends an acknowledgment back to the leader. Once the leader receives the acknowledgment from all replicas in ISR, the message is committed
So to answer your question, if the client publishes to serverB, it consults zookeeper to know the leader for Topic1 and partition1. Zookeeper responds saying serverA is the leader for partition1. So the client sends the message to serverA.(Here I assumed that the partitioner will send the message to partition1.)
All these are handled by the kafka producer. End user application need not worry about these details.
You can read more about this here