Is that possible to have mutex in RabbitMQ queue, i.e. If a client is reading from the queue, no other client should read from the queue. is that possible?
Let me explain my scenario:
Two application running in two different servers. reading the same queue. But, if one application is running and reading the messages from the Queue, the other application should not do anything. if the Main application fails or stopped, then the other application should
start reading from this queue.
This is kind of a fail over mechanism. Have anyone tried this before. Any help is much appreciated.
As long as i have searched, no solutions found...A simple solution is
create a queue call it as Lock Queue.
Have only one message make the application to read it from the queue.
When ever the application starts in a another server, it will wait for the message in the Queue. so, if the first one fails second
one will read the message and start processing the message in desired queue from which it should read.
A Mutex in Queue, that's it.
Note: This approach will work only if there is only message in the lock queue. make sure you handle it in your application.
This talk explicitly explains why this is a bad idea:
http://www.youtube.com/watch?v=XiXZOF6dZuE&feature=share&t=29m55s
from ~ 29m 55s in
Related
I am using Spring Cloud Stream over RabbitMQ for my project. I have a processor that reads from a source, process the message and publish it to the sink.
Is my understanding correct that if my application picks up an event from the stream and fails (e.g. app sudden death):
unless I ack the message or
I save the message after reading it from the queue
then my event would be lost? What other option would I have to make sure not to lose the event in such case?
DIgging through the Rabbit-MQ documentation I found this very useful example page for the different types of queues and message deliveries for RabbitMQ, and most of them can be used with AMPQ.
In particular looking at the work queue example for java, I found exactly the answer that I was looking for:
Message acknowledgment
Doing a task can take a few seconds. You may wonder what happens if
one of the consumers starts a long task and dies with it only partly
done. With our current code, once RabbitMQ delivers a message to the
consumer it immediately marks it for deletion. In this case, if you
kill a worker we will lose the message it was just processing. We'll
also lose all the messages that were dispatched to this particular
worker but were not yet handled. But we don't want to lose any tasks.
If a worker dies, we'd like the task to be delivered to another
worker.
In order to make sure a message is never lost, RabbitMQ supports
message acknowledgments. An ack(nowledgement) is sent back by the
consumer to tell RabbitMQ that a particular message has been received,
processed and that RabbitMQ is free to delete it.
If a consumer dies (its channel is closed, connection is closed, or
TCP connection is lost) without sending an ack, RabbitMQ will
understand that a message wasn't processed fully and will re-queue it.
If there are other consumers online at the same time, it will then
quickly redeliver it to another consumer. That way you can be sure
that no message is lost, even if the workers occasionally die.
There aren't any message timeouts; RabbitMQ will redeliver the message
when the consumer dies. It's fine even if processing a message takes a
very, very long time.
Manual message acknowledgments are turned on by default. In previous
examples we explicitly turned them off via the autoAck=true flag. It's
time to set this flag to false and send a proper acknowledgment from
the worker, once we're done with a task.
Thinking about it, using the ACK seems to be the logic thing to do. The reason why I didn't think about it before, is because I thought of a ACK just under the perspective of the publisher and not of the broker. The piece of documentation above was very useful to me.
I'm new to Service Fabric Reliable Actors technology and trying to figure out best practices for this specific scenario:
Let's say we have some legacy code that we want to run new code built on SF Reliable Actors. Actors of certain type "ActorExecutor" are going to asynchronously call some third-party service that sometimes could stuck for pretty long time, longer than actor's calling client is ready to wait, or even experience some prolonged underling communication issues. We do not want client (legacy code) to get blocked by any sort of issues in ActorExecutor, it does not expect to receive any value or status back from actor. Should we use SF ReliableQueue for that? Should we use some sort of actor-broker to receive requests from client and storing them to queue: Client->ActorBroker->ActorExecutor? Are reminders could be helpful here?
One more question in this regard: Giving the situation is possible when many thousands of actors might stuck in 'third-party incomplete call' in the same time, and we want to reactivate and repeat the very last call for them, should we write a new tool for that? In NServiceBus you can create an error queue in MSMQ where all failed like 'unable to process' messages to be landed, and then we were able to simply re-process them anytime in the future. From my understanding, there is no such thing in Service Fabric and it's something we need to built on our own.
An event driven approach can help you here. Instead of waiting for the Actor to return from the call to a service, you can enqueue some task on it, to request it to perform some action. The service calling Actor would function autonomously, processing items from it's task queue. This will allow it to perform retries and error handling. After a successful call, a new event can notify the rest of the system.
Maybe this project can help you to get started.
edits:
At this time, I don't believe you can use reliable collections in Actors. So a queue inside the state of an Actor, is a regular (read-only) collection.
Process the queue using an Actor Timer. Don't use the threadpool, as it's not persistent and won't survive crashes and Actor garbage collections.
Is there any way to configure an MSMQ queue to send copies of all messages it receives to another MSMQ queue? I have a memory leak on a production application that services a queue. I have a test version (that hopefully fixes the memory leak) on a test server, that services a test queue. I want to deluge the test version with the production stream of messages, to ensure that the memory leak has been fixed. After I am done testing, I would like to shut off this "message forwarding"
I had the same problem on my application, I was faced with 2 solutions, the easiest one I would recommend you to do is to make a very simple application that Peeks every message in a Queue via a transaction, and send a copy of the Message object to another queue, and you're done, just Abort() the transaction, that way you can be sure it'll be restored and wait for the production app to process the messages.
The other alternative would be to have the Message Queue apps just send the messages to yet another message queue, that way you don't have to mess around with peeks in Production and you'll have full access to your own queue in a test environment.
No. MSMQ is a protocol for delivering messages. You would need an application to read the delivered messages and send new copies to a different queue.
I am very much new to Mule ESB.
I have created a flow which has multiple queues(I am using RabbitMQ). The flow is something like some messages are put into first queue which will be read by second queue which will be read be third and so on.
Note: I am sending messages concurrently using JMeter.
Let's say before all message(s) can be put into third queue from second queue, my rabbitmq server is stopped. Now, in this case, I want to recover my messages. Also, I should be able to know what messages have been put into third queue and what are still left.
I might not have put my question in elegant or understandable way but i hope you understood what i want to achieve.
You can use Rollback Exception strategy (http://www.mulesoft.org/documentation/display/current/Rollback+Exception+Strategy) along with transactions: http://www.mulesoft.org/documentation/display/current/Transaction+Management when properly implemented messages that have not been delivered to the second queue will be rolled back automatically.
In rollback exception strategy you can write your custom behavior. Why don't you use rabbitmq client to see what messages where in third queue?
So, I am new to MSMQ and NServiceBus. I played around with demos and got a working scenario going with NServiceBus. (Getting my own up and running was even easier than following the demo thanks to the new Modeling tools!)
I then went and presented my plan (based off my work with the demo and my own model) to my co-workers. Two of them were versed in using MSMQ and started asking me questions about how I will handle "Dead Letters".
I had never heard of "Dead Letters". They explained that it is queue used for messages that things cannot be sent (either because the other end refuses them or if the other end is not there).
The concern of my co-workers is that if we don't have Dead Letter queues then how will we stop a message from blocking the queue? (If the queue is FIFO and the top message can't be sent, then it blocks the other messages behind it right?)
On the other hand, if we have "Dead Letter" queues how are they managed? (Do I get an event from NServiceBus that tells me a new message is in the Dead Letter queue? How do I configure when a message will go to the Dead Letter queue? How can I try to re-send a Dead Letter message?)
So basically, how does NServiceBus deal with undeliverable messages?
Typically, you specify an error queue as well, which is where messages that couldn't be sent will go. Look at their MSMQ example:
http://docs.particular.net/nservicebus/msmq/transportconfig &
http://docs.particular.net/nservicebus/msmq/connection-strings
<MsmqTransportConfig InputQueue="MyClient" ErrorQueue="error"
NumberOfWorkerThreads="1" MaxRetries="5"/>
In this scenario, the error queue will reside on the same machine as the input queue. This may also be a remote queue, if preferred. MaxRetries refers to the number of attempts to send before it gets put into the error queue. How you choose to handle the error queue is up to you, however.