Trying to understand the difference between following configuration to handle Order of the message and Duplicate message in Kafka . I could not find any detailed explanation in anywhere .
Could you please help me understand with some use case .
enable.idempotence=true
Idempotent producers can handle duplicate messages and preserve message order even with request pipelining—there is no message duplication because the broker ignores duplicate sequence numbers, and message ordering is preserved because when there are failures, the producer temporarily constrains to a single message in flight until sequencing is restored.
max.in.flight.requests.per.connection=1 to ensure that only one request can be sent to the broker at a time. To preserve message order while allowing request pipelining, set the configuration
parameter retries=0 if the application is able to tolerate some message loss
When you set enable.idempotence=true these configurations are automatically set if you don't set them manually.
retries=Integer.MAX_VALUE
max.in.flight.requests=5 (if Kafka version >= 1.0 You can check this for more information.)
acks=all
And this is the ideal configuration for idempotent producer. If you set retries=0, in case of network failure you couldn't even send the message to broker.
enable.idempotence: When set to 'true', the producer will ensure that exactly one copy of each message is written in the stream. If
'false', producer retries due to broker failures, etc., may write
duplicates of the retried message in the stream. Note that enabling
idempotence requires max.in.flight.requests.per.connection to be less
than or equal to 5, retries to be greater than 0 and acks must be
'all'. If these values are not explicitly set by the user, suitable
values will be chosen. If incompatible values are set, a
ConfigException will be thrown.
Related
I'm struggling to figure out what kind of guarantees a system with this producer configuration provides
acks=1
min.insync.replicas=1
retries=10
The first two parameters basically give me at-most-once guarantees since the leader can write to local storage, reply to the producer and then die before replication to the followers.
That part is clear but the retries would mean that when there are network issues between the producer and the cluster the leader may write to the local disk and fail to confirm this fact to the producer. After that, the producer would retry the writing and duplicate the message.
Is my reasoning correct that this setup technically provides no specific delivery guarantee? Or is there a process on the cluster that prevents the latter from happening?
You are correct, the first two configuration properties provide "at most once" guarantee only.
For "at least once" guarantee, you need to change acks to all (or -1). This would only send an acknowledgment when all members of the ISR (leader and followers) have received the record.
Even with acks=all, in some cases this will still only guarantee "at most once" delivery IF there is only 1 replica in the ISR due to failures.
To ensure acks=all always guarantees "at least once" delivery, you need to set min.insyc.replicas to at least 2 (assuming you have more than 1 broker in your cluster). This would mean that at least 2 replicas must have received the record for the producer write to be considered successful. If min.insyc.replicas is not met, it will throw a NotEnoughReplicas exception.
On to the second part of your question. Yes, you can receive duplicate messages with this configuration. To prevent duplicates, you can set enable.idempotence=true.
However, when setting enable.idempotence=true, this requires max.in.flight.requests.per.connection to be less than or equal to 5, acks must be all and retries have to be greater than 0.
the current (3.2) producer retry documentantion in Kafka is:
Allowing retries while setting enable.idempotence to false and max.in.flight.requests.per.connection to 1 will potentially change the ordering of records because if two batches are sent to a single partition, and the first fails and is retried but the second succeeds, then the records in the second batch may appear first.
Previously, the documentation for 2.8 was:
Allowing retries without setting max.in.flight.requests.per.connection to 1 will potentially change the ordering of records because if two batches are sent to a single partition, and the first fails and is retried but the second succeeds, then the records in the second batch may appear first.
Do the two docs contradict themselves?
From previous experience, setting max.in.flight.requests.per.connection=1 will ensure ordering even for enable.idempotence=false , which is not what the current documentation states.
UPDATE:
I've found that the default acks configurations changed and it might be a clue:
Notable changes in 3.0.0:
The producer has stronger delivery guarantees by default: idempotence is enabled and acks is set to all instead of 1. See KIP-679 for details.
However, it is more related to data loss than to ordering.
In the documentation is stated
From Kafka 0.11, the KafkaProducer supports two additional modes: the idempotent producer
and the transactional producer. The idempotent producer strengthens Kafka's delivery
semantics from at least once to exactly once delivery.
...
To take advantage of the idempotent producer, it is imperative to avoid application
level re-sends since these cannot be de-duplicated. As such, if an application enables
idempotence, it is recommended to leave the retries config unset, as it will be defaulted
to Integer.MAX_VALUE. Additionally, if a send(ProducerRecord) returns an error even with
infinite retries (for instance if the message expires in the buffer before being sent),
then it is recommended to shut down the producer and check the contents of the last
produced message to ensure that it is not duplicated.
Finally, the producer can only guarantee idempotence for messages sent within a single session.
I don't exactly understand how to avoid application level resends in failure scenarios, particularly in the scenarios when ACKs are lost due to network error in combination with the producer app being down.
Would you be able to point me to the strategies used to ensure exactly once delivery ?
Read this article about message ordering in topic partition: https://blog.softwaremill.com/does-kafka-really-guarantee-the-order-of-messages-3ca849fd19d2
Allowing retries without setting max.in.flight.requests.per.connection
to 1 will potentially change the ordering of records because if two
batches are sent to a single partition, and the first fails and is
retried but the second succeeds, then the records in the second batch
may appear first.
According it there are two types of producer configs possible to achieve ordering guarantee:
max.in.flight.requests.per.connection=1 // can impact producer throughput
or alternative
enable.idempotence=true
max.in.flight.requests.per.connection //to be less than or equal to 5
max.retries // to be greater than 0
acks=all
Can anybody explain how second configuration achieves order guarantee? Also in the second config exactly-once semantics enabled.
idempotence:(Exactly-once in order semantics per partition)
Idempotent delivery enables the producer to write a message to Kafka exactly
once to a particular partition of a topic during the lifetime of a
single producer without data loss and order per partition.
Idempotent is one of the key features to achieve Exactly-once Semantics in Kafka. To set “enable.idempotence=true” eventually get exactly-once semantics per partition, meaning no duplicates, no data loss for a particular partition. If an error occurred even producer send messages multiple times will get written to Kafka once.
Kafka producer concept of PID and Sequence Number to achieve idempotent as explained below:
PID and Sequence Number
Idempotent producers use product id(PID) and sequence number while producing messages. The producer keeps incrementing the sequence number on each message published which map with unique PID. The broker always compares the current sequence number with the previous one and it rejects if the new one is not +1 greater than the previous one which avoids duplication and the same time if more than greater show lost in messages.
In a failure scenario it will still maintain sequence number and avoid duplication as shown below:
Note: When the producer restarts, new PID gets assigned. So the idempotency is promised only for a single producer session
If you are using enable.idempotence=true you can keep max.in.flight.requests.per.connection up to 5 and you can achieve order guarantee which brings better parallelism and improve performance.
Idempotence feature introduced in Kafka 0.11+ before we can achieve some level level of guaranteed using max.in.flight.requests.per.connection with retries and Acks setting:
max.in.flight.requests.per.connection to 1
max.retries bigger number
acks=all
max.in.flight.requests.per.connection=1: to make sure that while messages are retrying, additional messages will not be sent.
This gives guarantee at-least-once and comes with cost on performance and throughput and that's encourage introduced enable.idempotence feature to improve the performance and at the same time guarantee ordering.
exactly_once: To achieve exactly_once along with idempotence we need to set transaction as read_committed and will not allow to overwrite following parameters:
isolation.level:read_committed( Consumers will always read committed
data only)
enable.idempotence=true (Producer will always haveidempotency enabled)
MAX_IN_FLIGHT_REQUESTS_PER_CONNECTION=5 (Producer will
always have one in-flight request per connection)
enable.idempotence is a newer setting that was introduced as part of kip-98 (implemented in kafka 0.11+). before it users would have to set max.inflight to 1.
the way it works (abbreviated) is that producers now put sequence numbers on ourgoing produce batches, and brokers keep track of these sequence numbers per producer connected to them. if a broker receives a batch out of order (say batch 3 after 1) it rejects it and expects to see batch 2 (which the producer will retransmit). for complete details you should read kip-98
If I am using Kafka Async producer, assume there are X number of messages in buffer.
When they are actually processed on the client, and if broker or a specific partition is down for sometime, kafka client would retry and if a message is failed, would it mark the specific message as failed and move on to the next message (this could lead to out of order messages) ? Or, would it fail the remaining messages in the batch in order to preserve order?
I next to maintain the ordering, so would ideally want to kafka to fail the batch from the place where it failed, so I can retry from the failure point, how would I achieve that?
Like it says in the kafka documentation about retries
Setting a value greater than zero will cause the client to resend any
record whose send fails with a potentially transient error. Note that
this retry is no different than if the client resent the record upon
receiving the error. Allowing retries will potentially change the
ordering of records because if two records are sent to a single
partition, and the first fails and is retried but the second succeeds,
then the second record may appear first.
So, answering to your title question, no kafka doesn't have order guarantees under async sends.
I am updating the answers base on Peter Davis question.
I think that if you want to send in batch mode, the only way to secure it I would be to set the max.in.flight.requests.per.connection=1 but as the documentation says:
Note that if this setting is set to be greater than 1 and there are
failed sends, there is a risk of message re-ordering due to retries
(i.e., if retries are enabled).
Starting with Kafka 0.11.0, there is the enable.idempotence setting, as documented.
enable.idempotence: When set to true, the producer will ensure that
exactly one copy of each message is written in the stream. If false,
producer retries due to broker failures, etc., may write duplicates of
the retried message in the stream. Note that enabling idempotence
requires max.in.flight.requests.per.connection to be less than or
equal to 5, retries to be greater than 0 and acks must be all. If
these values are not explicitly set by the user, suitable values will
be chosen. If incompatible values are set, a ConfigException will be
thrown.
Type: boolean Default: false
This will guarantee that messages are ordered and that no loss occurs for the duration of the producer session. Unfortunately, the producer cannot set the sequence id, so Kafka can make these guarantees only per producer session.
Have a look at Apache Pulsar if you need to set the sequence id, which would allow you to use an external sequence id, which would guarantee ordered and exactly-once messaging across both broker and producer failovers.