Suppose that kafka message producer send an event message to a topic. And then a consumer process this event message. However, this consumer process it throw exception because business error, so he want to let the message producer know it and resent again.
Are there any solution?
You should, generally, not create such coupling between producers and consumers.
If a consumer fails to read a message then it ideally shouldn't commit that offset. If the record didn't expire in Kafka, then there's no need for the producer to send anything as data is still in the broker, so having the consumer application be restarted will attempt to read the failed offsets again.
If you want it to never miss a message but might duplicate the occasional message, you can use delivery semantic "at-least-once".
Related
I am trying to deveopemet synchronous poc. My objective is i want to send message to request topic and get back updated message. I have request-topic for producer, and reply-topic with consumer.
i am able send the message to consumer and able to update the message as well. but i am not getting response on consumer side. getting an error " No pending reply-timeout, perhaps using shared reply topic.
In a spring boot app, i use JMS with QPID to receive messages from an Azure ServiceBus Queue.
I create my own connection factory with properties:
SessionsAcknownlegdeMode: CLIENT_ACKNOWlEDGE
RedeliveryPolicy Outcome: REJECTED
MaxRedelivery: 5
I use the annotation #JmsListener
Problem: When I consume the message, we try to send a mail with JavaMail, this normally works but it happened that the smtp server we use was having problem so the org.springframework.mail.MailSendException was thrown.
The message is correctly retried and put in DLQ after max retries but after a few messages in error, my #JmsListener method is not invoked for the following messages, and they are put directly in DLQ. That is not what I want.
I tried to replicate this behavior locally by manually throwing exceptions in the listener for given messages, but the consumer correctly sends bad messages to DLQ and consumes good messages.
Does anyone know what is happening ?
I'm trying very hard to understand the flow of a web request to a server which has a queue or message broker in the middle, but I can't find information about when and where the reply is given.
Imagine this use case:
Client A:
sends a invoice order request
the invoice is enqueued
the request is processed and dequeued.
at which time the client will receive a response?
right after the message is received by the queue?
right after the message is processed and dequeued? Other?
I'm asking because if the reply only comes after the message being processed the client might wait a long time. Imagine the message takes 3 minutes to process, would the client need to keep requesting the server to see if it is processed? or a connection is maintained using something like long polling?
I'm interested in scenarios using RabbitMq and kafka.
Advantage of having a messaging system is to ensure the frontend webserver and backend processing is decoupled. Best practice is Web server should publish the message and just wait for the messaging system to acknowledge receiving the message.
I have a message a want to dequeue then right after its dequeued I want to queue another message to a different queue. I want to do all this in the same transaction. is this possible with rabbitmq or any other queueing service?
The closest you can get to what you want with RabbitMQ is:
Use acks and publisher confirms
You receive a message and do not ack it.
Send your reply message.
Wait for confirm from the broker.
Once confirm had arrived, ack initial message.
But then, consider this failure situation:
Initial message received
Reply message sent
Your service failed before ACKing initial message
When your service is back, it will receive the initial message again
So you will need to use some deduplication mechanism etc.
I am implementing a service that dispatches messages to clients using an arbitrary message broker. A single user may have multiple clients and the message will be dispatched to each one of them. Once the user has read the message on one client, I want the message removed from the user's other clients.
Do message brokers typically implement this functionality, or will I need a custom solution?
For a custom solution, I was thinking that the broker could maintain a separate reply topic to which a client will deliver a message to say that the user has read the message. The service can consume messages on this reply topic, and dispatch another message to the user's other clients that tells them to remove the message.
Is this how such a solution might typically be implemented?
If it helps, I am considering using MQTT as a message protocol.
There is no concept of even end to end message delivery notification in the MQTT protocol1, let alone read notification. You are likely to need to implement this your self.
If I was doing this I would have 2 topics per user something like this:
[user id]/msg
and
[user id]/read
I would make the payload of the messages delivered to the [user id]/msg contain a message id. I would then publish the message id on the [user id]/read topic. All clients would subscribe to both, that way they could easily mark as read/remove messages as they were consumed on other clients.
1confirmation for higher QOS levels are between the publisher and the broker and then between the broker and the subscriber