I am currently using an IntegrationFlow to trigger a Job execution when RabbitMq messages arrive in a queue. Both the IntegrationFlow's AmqpInboundChannelAdapter and the job's first step's ItemReader are configured to read messages from the same queue.
The issue that I am having is that the IntegrationFlow's AmqpInboundChannelAdapter reads the RabbitMQ message, and the ItemReader then can no longer find that message. Probably because the IntegrationFlow acknowledges the message before the Job launches.
Is there a way to keep the IntegrationFlow from consuming/acknowledging the message, leaving it in the queue so that the ItemReader can work as intendend? I tried configuring the AmqpInboundChannelAdapter to requeue the Message, but that simply caused an infinite loop of the Adapter re-reading it's own message.
This question describes my problem somewhat, except that I am doing no processing, I am simply trying to use the IntegrationFlow as a JobLaunching trigger. So the solution seems like an anti-pattern.
Spring Batch Integration - pass data b/w integration and batch
Any help would be greatly appreciated
If the batch job only needs information from that one message, I would suggest binding a second queue, with the same routing key; one queue for the trigger and one for the item reader.
If the first message is a trigger and the item reader then reads multiple messages, you could add the message contents to the JobParameters. You would also need to set the adapter's prefetch to 1 so it doesn't get sent any other messages while this one is being processed.
Related
I am new to Kafka and would like to seek advice on what is the best practice to handle such scenario.
Scenario:
I have a spring boot application that has a consumer method that is listening for messages via the #KafkaListner annotation. Once an incoming message has occurred, the consumer method will process the message, which simply performs database updates to different tables via JdbcTemplate.
If the updates to the tables are successful, I will manually commit the message by calling the acknowledge() method. If the database update fails, instead of calling the acknowledge() method, I will call the nack() method with a given duration (E.g. 10 seconds) such that the message will reappear again to be consumed.
Things to note
I am not concerned with the ordering of the messages. Whatever event comes I just have to consume and process it, that's all.
I am only given a topic (no retryable topic and no dead letter topic)
Here is the problem
If I do the above method, my consumer becomes inconsistent. Let's say if I call the nack() method with a duration of 1min, meaning to say after 1 min, the same message will reappear.
Within this 1 min, there could "x" number of incoming messages to be consumed and processed. The observation made was none of these messages are getting consumed and processed.
What I want to know
Hence, I hope someone will advise me what I am doing wrongly and what is the best practice / way to handle such scenarios.
Thanks!
Records are always received in order; there is no way to defer the current record until later, but continue to process other records after this one when consuming from a single topic.
Kafka topics are a linear log and not a queue.
You would need to send it to another topic; the #RetryableTopic (non-blocking retrties) feature is specifically designed for this use case.
https://docs.spring.io/spring-kafka/docs/current/reference/html/#retry-topic
You could also increase the container concurrency so at least you could continue to process records from other partitions.
We're using ActiveMQ Artemis on JBoss EAP 7.1.
We noticed that once a message with a specific _AMQ_DUPL_ID value is passed through the queue, if the message producer tries to send a message with the same _AMQ_DUPL_ID value to the same queue again it is discarded by the broker. However, our need is to discard duplicate messages only if they are still in queue.
Is there a way to achieve this goal?
We use the primary key from the database as _AMQ_DUPL_ID value. This is the code we use
public void sendMessage(final T msg, final String id) {
jmsTemplate.send(destination, new MessageCreator() {
#Override
public Message createMessage(Session session) throws JMSException {
Message message = session.createObjectMessage(msg);
message.setStringProperty("_AMQ_DUPL_ID", id);
return message;
}
});
}
We're looking for a solution because we have a timer that every 30 seconds loads from DB all records with a specific value for status field and puts them into the JMS queue.
Consumers consume JMS messages, processes them, updates their status field, insert/update them into the DB and opens a websocket connection with another application that we don't control. Sometimes the consumer hangs on the websocket call and, consequently, it remains busy while the timer continues to fill the queue.
To solve this problem we thought that something like Artemis duplicate message detection would help. However, when the external app hangs our consumer we need our timer to be able to put the message on the queue again.
The duplicate message detection on ActiveMQ Artemis is working as designed. Its goal is to avoid any chance that a consumer will receive a duplicate message which means that even though a message may no longer be in the queue (e.g. because it was consumed) any duplicate of that message should still be rejected.
What you're asking for here is akin to asking how you can insert multiple records with the same primary key into a database table. It simply can't because because the entire point of having a primary key is to avoid duplicate records.
I recommend you implement some kind of timeout for the websocket call otherwise your application will be negatively impacted by a resource you have no control over.
Aside from that you may be able to use a last-value queue using the primary key as the value for _AMQ_LVQ_NAME. This will guarantee that only 1 instance of the message will be in a queue at any point. Read the documentation for more details.
So I got some annoying offset commiting case with my kafka consumers.
I use 'kafka-node' for my project.
I created a topic.
Created 2 consumers within a consumer-group over 2 servers.
Auto-commit set to false.
For every mesaage my consumers get, they start an async process which can take between 1~20sec, when the process done the consumer commits the offset..
My problem is:
There is a senarios in which,
Consumer 1 gets a message and takes 20sec to process.
In the middle of the process he gets another message which takes 1s to process.
He finish the second message processing, commit the offset, then crashes right away.
Causing the previous message processing to fail.
If I re run the consumer, hes not reading the first message again, because the second message already commited the offsst which is greater than the first.
How can i avoid this?
Kafkaconsumer.on('message', async(message)=>{
await SOMETHING_ASYNC_1~20SEC;
Kafkaconsumer.commit(()=>{});
});
You essentially want to throttle messages and handle concurrency by utilizing async.queue.
Create a async.queue with message processor and concurrency of one (the message processor itself is wrapped with setImmediate so it will not freeze up the event loop)
Set the queue.drain to resume the consumer
The handler for consumer's message event pauses the consumer and pushes the message to the queue.
The kafka-node README details this here.
An example implementation, similar to your problem, can be found here.
We are implementing a Kafka Consumer using Spring Kafka. As I understand correctly if processing of a single message fails, there is the option to
Don't care and just ACK
Do some retry handling using a RetryTemplate
If even this doesn't work do some custom failure handling using a RecoveryCallback
I am wondering what your best practices are for that. I think of simple application exceptions, such as DeserializationException (for JSON formatted messages) or longer local storage downtime, etc. Meaning there is needed some extra work, like a hotfix deployment, to fix the broken application to be able to re-process the faulty messages.
Since losing messages (i. e. not processing them) is not an option for us, the only option left is IMO to store the faulty messages in some persistence store, e. g. another "faulty messages" Kafka topic for example, so that those events can be processed again at a later time and there is no need to stop event processing totally.
How do you handle these scenarios?
One example is Spring Cloud Stream, which can be configured to publish failed messages to another topic errors.foo; users can then copy them back to the original topic to try again later.
This logic is done in the recovery callback.
We have a use case where we can't drop any messages at all, even for faulty messages. So when we encounter a faulty message, we will send a default message in place of that faulty record and at the same time send the message to a failed-topic for retry later.
TL;DR
When a Flume source fails to push a transaction to the next channel in the pipeline, does it always keep event instances for the next try?
In general, is it safe to have a stateful Flume interceptor, where processing of events depends on previously processed events?
Full problem description:
I am considering the possibility of leveraging guarantees offered by Apache Kafka regarding the way topic partitions are distributed among consumers in a consumer group to perform streaming deduplication in an existing Flume-based log consolidation architecture.
Using the Kafka Source for Flume and custom routing to Kafka topic partitions, I can ensure that every event that should go to the same logical "deduplication queue" will be processed by a single Flume agent in the cluster (for as long as there are no agent stops/starts within the cluster). I have the following setup using a custom-made Flume interceptor:
[KafkaSource with deduplication interceptor]-->()MemoryChannel)-->[HDFSSink]
It seems that when the Flume Kafka source runner is unable to push a batch of events to the memory channel, the event instances that are part of the batch are passed again to my interceptor's intercept() method. In this case, it was easy to add a tag (in the form of a Flume event header) to processed events to distinguish actual duplicates from events in a failed batch that got re-processed.
However, I would like to know if there is any explicit guarantee that Event instances in failed transactions are kept for the next try or if there is the possibility that events are read again from the actual source (in this case, Kafka) and re-built from zero. In that case, my interceptor will consider those events to be duplicates and discard them, even though they were never delivered to the channel.
EDIT
This is how my interceptor distinguishes an Event instance that was already processed from a non-processed event:
public Event intercept(Event event) {
Map<String,String> headers = event.getHeaders();
// tagHeaderName is the name of the header used to tag events, never null
if( !tagHeaderName.isEmpty() ) {
// Don't look further if event was already processed...
if( headers.get(tagHeaderName)!=null )
return event;
// Mark it as processed otherwise...
else
headers.put(tagHeaderName, "");
}
// Continue processing of event...
}
I encountered the similar issue:
When a sink write failed, Kafka Source still hold the data that has already been processed by interceptors. In next attempt, those data will send to interceptors, and get processed again and again. By reading the KafkaSource's code, I believe it's bug.
My interceptor will strip some information from origin message, and will modify the origin message. Due to this bug, the retry mechanism will never work as expected.
So far, The is no easy solution.