I am using exactly once semantics provided by kafka. Therefore, my producer writes a message within a transaction. While my producer was sending 100th message, right between send() and commitTransaction(); I killed the producer process.
I read last few uncommitted messages in my topic.
Consumer record offset - message number
0 - 1
2 - 2
196 - 99 <- Last committed message by producer
198 - 100 <- 100th message was sent but not committed
Now, when I run consumer with read_committed isolation level. It exactly reads from 1-99 messages. But for that I have read entire topic. Eventually, I am going to store millions of messages in a topic. So, reading entire topic is not preferable.
Also, assume that consumer is polling messages from a broker and there is some communication issue with a kafka borker and consumer. Last message read by consumer is let's say offset#50. This means that I could not identify the last committed message in a topic reliably.
I used other methods i.e.
seekToEnd() - took me to offset#200
endOffsets() - took me to offset#200
Is there a way to get message that was committed by the Kafka producer reliably ? (In my case, Offset#196)
Related
I'm using AWS Kafka MSK and I have a topic with 2 partitions.
I also have a 2 consumers which are part of the same consumer group.
I'm wondering that will happen in the following case:
Consumer A - took messages 1 - 100
Consumer B - took messages 101 - 200
Consumer A failed
Consumer B succeeded
What happens to the messages 1 - 100?
Will the auto Kafka rebalance set consumer B to read messages 1 - 100?
or the new consumer that will startup instead of Consumer A will read the messages?
Thanks in advance.
Offset ranges are for partitions, not topics.
This scenario is not possible for a fresh consumer application unless one of the following is true
Offsets 0-100 of the partition assigned to consumer B have been removed due to retention
Your code calls seek method to skip those offsets
On the other hand, if the consumer group already existed and consumed none of the records of partition assigned to consumer A (say, it had failed before), and did commit offset 100 of the other partition. In this case, perhaps the same thing would happen; the consumer group might fail reading offset 0 of the "first" partition.
When any consumer instance fails, the group will rebalance. Depending on how you handle errors/failures, the previously healthy instance may then be assigned both partitions, and then fail consuming the "first" partition again (since it'll be the same code that died previously). Or, writing code differently, you'll ignore consumer exceptions and optionally mark bad offsets in a dead-letter queue. When logged or ignored, you'd commit offsets for the original consumer and skip those records.
I am new to Streaming Broker [like Kafka], and coming from Queueing Messaging System [like JMS, Rabbit MQ].
I read from Kafka docs that, messages are stored in Kafka partitions in offset as record. And consumer reads from offset.
What is the difference between message and record [does multiple/partial messages constitute a record?]
When comsumer reads from offset, is there a possibility that consumer reads partial message? IS there a need for consumer to string these parital messages based on some logic?
OR
1 message = 1 record = 1 offset
EDIT1:
The question was popped because, the "batch size" decides how many bytes of message should be published on to the borker. Lets say there are 2 messages with message1 = 100bytes and message2= 200 bytes and batchsize is set to 150bytes. Does this mean 100 bytes from message1 and 50 bytes from message2 are sent to broker at once? If yes, how are these 2 messages stored in offset?
In Kafka, a Producer sends messages or records (both terms can be used interchangeably) to Topics. A topic is divided into one or more Partitions that are distributed among the Kafka Cluster, which is generally composed of at least three Brokers.
A message/record is sent to a leader partition (which is owned by a single broker) and associated to an Offset. An Offset is a monotonically increasing numerical identifier used to uniquely identify a record inside a topic/partition, e.g. the first message stored in a record partition will have the offset 0 and so on.
Offsets are used both to identify the position of a message in a topic/partition as well as for the position of a Consumer Group.
For optimisation purpose , a producer will batch messages per partition. A batch is considered to be ready when either the configured batch.sized or linger.ms are reached. For example, if you have a batch.size set to 200KB and you send two messages (150KB and 100KB), they will be part potentially of the same batch. But the producer will never fragment a single message into chuncks.
No, a consumer cannot read partial messages.
I new to Kafka and i have a configuration where i have a source Kafka topic which has messages with a default retention for 7 days. I have 3 brokers with 1 partition and 1 replication.
When i try to consume messages from source Kafka topic and to my target Kafka topic i was able to consume messages in the same order. Now my question is if i am trying to reprocess all the messages from my source Kafka and consume in ,y Target Kafka i see that my Target Kafka is not consuming any messages. I know that duplication should be avoided but lets say i have a scenario where i have 100 messages in my source Kafka and i am expecting 200 messages in my target Kafka after running it twice. But i am just getting 100 messages in my first run and my second run returns nothing.
Can some one please explain why this is happening and what is the functionality behind it ?
Kafka consumer reads data from a partition of a topic. One consumer can read from one partition at one time only.
Once a message has been read by the consumer, it can't be re-read again. Let me first explain the current offset. When we call a poll method, Kafka sends some messages to us. Let us assume we have 100 records in the partition. The initial position of the current offset is 0. We made our first call and received 100 messages. Now Kafka will move the current offset to 100.
The current offset is a pointer to the last record that Kafka has already sent to a consumer in the most recent poll and that has been committed. So, the consumer doesn't get the same record twice because of the current offset. Please go through the following diagram and URL for complete understanding.
https://www.learningjournal.guru/courses/kafka/kafka-foundation-training/offset-management/
Currently in my kafka consumer i have turned off auto commit, so currently when processing of messages failed for ex: three invalid messages, the manual ack fails and the lag increases to three.
TOPIC PARTITION CURRENT-OFFSET LOG-END-OFFSET LAG
orders 0 35 38 3
After that if a new incoming valid message comes through and the processing of that message is successfully completed, it is manually acked and
after that consumer looks like this
TOPIC PARTITION CURRENT-OFFSET LOG-END-OFFSET LAG
orders 0 39 39 0
Why does consumer set the current-offset to 39 when the messages with offset 36, 37, 38 were not successfully processed
and they are never read again by the same consumer
Can anyone pls explain this behavior? Thanks in advance!
In Kafka, consumers don't ack every single messages. Instead they ack (commit) the offset of the last message they processed.
For example, if you commit offset 15, it implicitly means you've processed all messages before from 0 to 15. Also when committing 15, you overwrite any previous commit, so you cannot know if you committed 13 or 14 before.
I suggest you read the Consumer Position section in the docs that go over this concept.
Regarding reprocessing, Kafka offers a few options. When hitting a processing failure, before polling for more messages and processing new records, you can try to reprocess the message. Another option is to skip it as invalid and carry on (what you are currently doing).
On the other hand, you could ensure data is good by running a Streams job to pipe valid messages into a checked topic and forward bad messages to a DLQ. Then consume from this checked topic where you know you only have good messages. See validation for kafka topic messages
Kafka messaging use at-least-once message delivery to ensure every message to be processed, and uses a message offset to indicates which message is to deliver next.
When there are multiple consumers, if some deadly message cause a consumer crash during message processing, will this message be redelivered to other consumers and spread the death? If some slow message blocked a single consumer, can other consumers keep going and process subsequent messages?
Or even worse, if a slow and deadly message caused a consumer crash, will it cause other consumers start from its offset again?
There are a few things to consider here:
A Kafka topic partition can be consumed by one consumer in a consumer group at a time. So if two consumers belong to two different groups they can consume from the same partition simultaneously.
Stored offsets are per consumer group. So each topic partition has a stored offset for each active (or recently active) consumer group with consumer(s) subscribed to that partition.
Offsets can be auto-committed at certain intervals, or manually committed (by the consumer application).
So let's look at the scenarios you described.
Some deadly message causes a consumer crash during message processing
If offsets are auto-committed, chances are by the time the processing of the message fails and crashes the consumer, the offset is already committed and the next consumer in the group that takes over would not see that message anymore.
If offsets are manually committed after processing is done, then the offset of that message will not be committed (for simplicity, I am assuming one message is read and processed at a time, but this can be easily generalized) because of the consumer crash. So any other consumer in the group that is (will be) subscribed to that topic will read the message again after taking over that partition. So it's possible that it will crash other consumers too. If offsets are committed before message processing, then the next consumers won't see the message because the offset is already committed when the first consumer crashed.
Some slow message blocks a single consumer: As long as the consumer is considered alive no other consumer in the group will take over. If the slowness goes beyond the consumer's session.timeout.ms the consumer will be considered dead and removed from the group. So whether another consumer in the group will read that message depends on how/when the offset is committed.
Slow and deadly message causes a consumer crash: This scenario should be similar to the previous ones in terms of how Kafka handles it. Either slowness is detected first or the crash occurs first. Again the main thing is how/when the offset is committed.
I hope that helps with your questions.