Imagine there are 1 producer and 1000 consumers with same group id (the producer and consumer group id is not the same).
When message arrived and Kafka place it to the queue, does Kafka send notification to 1000 consumers that new message has been arrived (and after that, only one consumer takes the message)?
If it's not, how does consumer know that new message has been arrived?
Does Kafka send notification fo all consumers that new message has arrived?
Kafka works differently.
In the case you describe, all consumers would regularly try to fetch messages from the brokers. Thus, it's not necessary for the broker to send a notification, because the consumer pro-actively poll for new messages anyway.
Related
Kafka - is a message available directly after the producer receives ACK? (assuming that ack=all is set in current configuration)
Scenario:
consumer seeks to end on partition 1
my producer produces a message on Kafka topic (partition 1)
ack-all is completed
consumer poll the partition 1
Is it possible that the consumer will not receive that new message, assuming that the poll duration will be respectively small? Or ack-all guarantees that in moment of request is processed data is available to read
When a consumer drops from a group and a rebalance is triggered, I understand no messages are consumed -
But does an in-flight request for messages stay queued passed the max wait time?
Or does Kafka send any payload back during the rebalance?
UPDATE
For clarification, I'm referring specifically to the consumer polling process.
From my understanding, when one of the consumers drop from the consumer group, a rebalance of the partitions to consumers is performed.
During the rebalance, will an error be sent back to the consumer if it's already polled and waiting for max time to pass?
Or does Kafka wait the max time and send an empty payload?
Or does Kafka queue the request passed max wait time until the rebalance is complete?
Bottom line - I'm trying to explain periodic timeouts from consumers.
This may be in the docs, but I'm not sure where to find it.
Kafka producers doesn't directly send messages to their consumers, rather they send them to the brokers.
The inflight requests corresponds to the producer and not to the consumer.
Whether the consumer leaves a group and a rebalance is triggered or not is quite immaterial to the behaviour of the producer.
Producer messages are queued in the buffer, batched, optionally compressed and sent to the Kafka broker as per the configuration.
In-flight requests are the maximum number of unacknowledged requests
the client will send on a single connection before blocking.
Note that when we say ack, it is acknowledgement by the broker and not by the consumer.
Does Kafka send any payload back during the rebalance?
Kafka broker doesn't notify of any rebalance to its producers.
I need to send message for all consumers, but before detect who should get this message, how to do that using Kafka?
Should I use Kafks stream to filter data then send to consumers?
As I know each consumers should be added to unique consumer group, but how to detect in real time, who must receive message ?
Kafka decouples consumer and producer and when you write into a topic, you don't know which consumers might read the data.
Thus, in Kafka you never "send a message to a consumer", you just write the message into a topic and that's it.
Consumers just read from topics.
I'm working on a module where it consumes messages from a Kafka topic and publish to a downstream system. In the event of downstream system is unavailable consumer do not acknowledge the Kakfa message. Because of this when my consumer receives messages when downstream system is unavailable offset of the kakfa will not be committed. But if I receive new message after downstream system comes up and when I acknowledge that message, latest offset will be committed and consumer never receive those messages which were in the topic without the offset commit.
i.e Let's say my consumer is consumed up to offset 4. Consumer receive two messages when downstream is unavailable and because of that my consumer didn't commit the offset. So number of messages in the toipc is now 6, but offset is still 4. Now downstream system is available and consumer receive a new message (7th message). Since there is no issue from downstream, consumer acknowledge the 7th message and offset of the topic will be set to 7.
Is there any method that my consumer can receive the 5th and 6th messages before it receives the 7th message? I use spring cloud stream in the implementation.
See this answer.
You need a SeekToCurrentErrorHandler and throw an exception so that the offsets are reset.
How can be archive Kafka messages like if we want to send a particular message to some topic so we archive that message and send to that topic or some other topic?
Can we replay that message to the topic?
Can we replay based on particular offset?
send a particular message to some topic
That is a regular producer send request
we archive that message
Kafka persists data for a configurable amount of time on its own. Default is a week
send to that topic or some other topic
Again, a producer request can send to a specific topic. Kafka Streams or MirrorMaker can send to other topics, if needed
replay that message to the topic
Not clear... Replay from where? Generally, read the message and produce to a topic
replay based on particular offset
Yes, you can consume from a given TopicPartition + Offset coordinate within Kafka