We are using remote chunking with one master and 2 workers. While testing with small job all chunk messages has been sent to Queue in one go and then Master went down but workers are continuing to work on chunks from request Queue.
When Master restarted , all the replies from the worker are queued in Reply-Queue but due to miss match in correlation ID of reply messages and new message container created , messages are not consumed from the reply queue.
How to deal with such scenarios while using Spring Batch.
<int-jms:outbound-gateway
id="masterOutboundGateway"
connection-factory="masterJMSConnectionFactory"
correlation-key="JMSCorrelationID"
request-channel="ChunkMasterRequestChannel"
request-destination-name="ChunkRequestQueue"
receive-timeout="50000"
reply-channel="ChunkMasterReplyChannel"
reply-destination-name="ChunkReplyQueue" async="true" >
<int-jms:reply-listener/>
</int-jms:outbound-gateway>
Related
We have containerized ActiveMQ Artemis 2.16.0 and deployed it as a K8s deployment for KEDA.
We use STOMP using stomp.py python module. The ACK-mode is set as client-individual and consumerWindowSize = 0 on the connection. We are promptly acknowledging the message as soon as we read it.
The problem is, sometimes, the message count in the web console does not become zero even after all the messages are actually consumed and acknowledged. When I browse the queue, I don't see any messages in it. This is causing KEDA to spin up pods unnecessarily. Please refer to the attached screenshots I attached in the JIRA for this issue.
I fixed the issue in my application code. My requirement was one queue listener should consume only one message and exit gracefully. So, soon after sending ACK for the consumed message, I disconnected the connection, instead of waiting for the sleep duration to disconnect.
Thanks, Justin, for spending time on this.
I need to create an independent server which continuously polls data from a Kafka Topic. Till now I have created a vertx server and a Kafka Consumer. The issue is when we start the server it drains the queue and polls all the messages currently present but when new messages will arrive later it wont be able to consume them. I need to implement some logic in my server which able it to continuously poll the data and drain the queue.
According to Kafka's commitTransaction documentation, commitTransaction will fail with TimeoutException if it doesn't receive a response in certain time
Note that this method will raise TimeoutException if the transaction cannot be committed before expiration of max.block.ms. Additionally, it will raise InterruptException if interrupted. It is safe to retry in either case, but it is not possible to attempt a different operation (such as abortTransaction) since the commit may already be in the progress of completing. If not retrying, the only option is to close the producer.
Consider an application in which a Kafka producer sends a group of records as Transaction A.
After the records have been successfully sent to the topic, Kafka producer will execute commitTransaction .
Kafka cluster receives the commit transaction request and successfully commits records that are part of transaction A. Kafka cluster sends an acknowledgement regarding successful commit.
However, due to some issue this acknowledgement is lost, causing a Timeout exception at Kafka producer's commitTransaction call. Thus even though the records have been committed on Kafka cluster, from producer's perspective the commit failed.
Generally in such a scenario the application would retry sending the transaction A records in a new transaction B, but this would lead to duplication of records as they were already committed as part of transaction A.
Is the above described scenario possible?
How do you handle loss of commitTransaction acknowledgement and the eventual duplication of records that is caused by it?
There is a retry feature on Kafka clients. I am struggling to find out when a retry happens. Would a retry happen if the connection to the broker in interrupted briefly? How about if the brokers were not reachable for 5 mins? Will the messages get delivered once the brokers are back up? Or does retry only happen on known scenarios to the kafka clients?
Kafka Producer consists of a pool of buffer space that holds records that haven't yet been transmitted to the server as well as a background I/O thread that is responsible for turning these batch records into requests and transmitting them to the cluster.
For example if records are sent faster than they can be delivered to the server the producer will block for max.block.ms after which it will throw an exception. Then client assumes batch is failed and will retry to send the batch based on retries config
org.apache.kafka.common.errors.TimeoutException: Expiring 1 record(s) for my-test-topic-4 due to 30024 ms has passed since batch creation plus linger time
Suppose if the retries config is set to 3 and if all retries fails then the batch is lost
error: Failed to send message after 3 tries
How about if the brokers were not reachable for 5 mins?
If the broker is down and in mean time if retry is exhausted then you lost the data
Scenario :
Stream create [StreamName] --definition " Kafka -zkconnect=10.10.10.1:2181 --topic=<topic name> | MyCompositeModule " --deploy
We are running this stream in distributed mode and redis is the transport bus.
Per my understanding, kafka source maintains the offsets for messages consumed by MyCompositeModule (which is a sink as its a module created through 'module compose' process) through [streamname]-kafka-offsets topic. Which is unreadable and I would appreciate if there was a way to read the data from this topic.
Also, when I push the messages from kafka source the messages are queued in redis transport, then the module fetches them from this queue.
If kafka consumer module starts consuming 1000 messages from kafka redis queue- and composite module fails after receiving 10 messages or randomly processed 10 messages.So how to identify remaining 990 [ 1000 ( consumed ) - 10 (processed) = 990 ] unprocessed messages .
Even if we check kafka offsets it will show consumed messages count. example: -kafka.offsets - which is unreadable in our process.
So all the unprocessed messages will be in Redis queue as we are using Redis in SpringXD. So can anyone help me out how to identify the unprocessed messages and
re sending it to composite module to process.
Basically, I am looking for the recommendations on an elegant solution for robust delivery, adding failure handling capability in spring xd stream when consuming from from kafka source.
If the messages are effectively consumed from Kafka and moved to the bus, then they will be acknowledged as consumed from the offset manager's perspective.
You can try enabling retry and dead lettering for the Redis Message Bus as described here: http://docs.spring.io/spring-xd/docs/current/reference/html/#error-handling-message-delivery-failures.
Cheers,
Marius