Is there any way to configure an MSMQ queue to send copies of all messages it receives to another MSMQ queue? I have a memory leak on a production application that services a queue. I have a test version (that hopefully fixes the memory leak) on a test server, that services a test queue. I want to deluge the test version with the production stream of messages, to ensure that the memory leak has been fixed. After I am done testing, I would like to shut off this "message forwarding"
I had the same problem on my application, I was faced with 2 solutions, the easiest one I would recommend you to do is to make a very simple application that Peeks every message in a Queue via a transaction, and send a copy of the Message object to another queue, and you're done, just Abort() the transaction, that way you can be sure it'll be restored and wait for the production app to process the messages.
The other alternative would be to have the Message Queue apps just send the messages to yet another message queue, that way you don't have to mess around with peeks in Production and you'll have full access to your own queue in a test environment.
No. MSMQ is a protocol for delivering messages. You would need an application to read the delivered messages and send new copies to a different queue.
Related
I use NServiceBus with MSMQ on Windows Server 2008.
In one of our scenario we send a command to another server queue but sometimes the message never arrives in the queue and MSQM doesn't throw an exception.
I have written a small application that sends a message directly to the queue without NServiceBus and I have encountered the same problem.
I think it isn't a nservicebus problem but a MSMQ problem : windows security, network,...
How can I force MSMQ to return an error when the message can't be saved in queue ?
What are the possible reasons of this behavior ?
Where are the messages that have disappeared ?
In the MSMQ Management snap-in take a look at the system queues and look at transactional dead letter and non-transactional dead letter queues.
Also be sure to have setup queue ACL correctly.
Windows Server 2012
MSMQ 6 Workgroup Mode
We've had issues trying to recover MSMQ messages that were sent to the transaction dead letter queue. We've tried moving them to the outbound queue, the message seems to send fine (even the Event Log says so) however it never gets to the destination queue.
After trial and error we've figured out how to get them to another queue on the same server but not to the destination queue on a remote server. We don't want to lose anymore messages. Does anyone have any suggestion on how we can deliver these messages?
Thank you,
David
As I understood your question, it's a one time problem with some number of messages you already have in MSMQ, and not general connectivity issue between machines? If so, you should be able to solve it with some MSMQ management tool. Disclaimer: I'm the author of one such tool - QueueExplorer. I don't know what other tools can do, but with QueueExplorer you can copy/paste or drag/drop messages to another machine opened in separate tab/window. In order to do that QueueExplorer has to perform MSMQ Send operation, so messages will have to pass through MSMQ between these two machines.
So if there's still that issue that prevented original delivery you'll still be stuck. In that case you can save all messages to a file, transfer it to another machine through file system and load it there to whichever queue they should go. This is obviusly just a manual workaround for one time situation. Btw. this could be done in QueueExplore's trial mode.
If however problem is with connectivity and messages always end up in dead letter queue, it's better to check them from Computer Management. It's one area where it's better than our tool - you can turn on "Class" column and see reason why messages couldn't be delivered. For instance if you see "The time-to-be-received has elapsed" you'll know what's the problem.
Is there some standard configuration setting, service, or tool that accepts messages from one queue and moves them on to another one? Automatically handling the dead message problem, and providing some of retry capability? I was thinking this is what "MSMQ Message Routing" does but can't seem to find documentation on it (except for on Windows Mobile 6, and I don't know if that's relevant).
Context:
I understand that when using MSMQ you should always write to a local queue so that failure is unlikely, and then X should move that message to a remote queue. Is my understanding wrong? Is this where messaging infrastructure like Biztalk comes in? Is it unnecessary to write to a local queue first to absolutely ensure success? Am I supposed to build X myself?
As Hugh points out, you need only one MSMQ Queue to Send messages in one direction from a source to a destination. Source and destination can be on the same server, same network or across the internet, however, both source and destination must have the MSMQ service running.
If you need to do 'message' routing (e.g. a switch which processes messages from several source or destination queues, or routing a message to one or more subscribers based on the type of message etc) you would need more than just MSMQ queue.
Although you certainly can use BizTalk to do message routing, this would be expensive / overkill if you didn't need to use other features of BizTalk. Would recommend you look at open source, or building something custom yourself.
But by "Routing" you might be referring to the queue redirection capability when using HTTP as the transport e.g. over the internet (e.g. here and here).
Re : Failed delivery and retry
I think you have most of the concepts - generally the message DELIVERY retry functionality should be implicit in MSMQ. If MSMQ cannot deliver the message before the defined expiry, then it will be returned on the Dead Letter Queue, and the source can then process messages from the DLQ and then 'compensate' for them (e.g. reverse the actions of the 'send', indicate failure to the user, etc).
However 'processing' type Retries in the destination will need to be performed by the destination application / listener (e.g. if the destination system is down, deadlocks, etc)
Common ways to do this include:
Using 2 Phase commit - under a distributed unit of work, pull the message off MSMQ and process it (e.g. insert data into a database, change the status of some records etc), and if any failure is encountered, then leave the message back onto the queue and the DB changes will be rolled back.
Application level retries - i.e. on the destination system, in the event of 'retryable' type errors (timeout due to load, deadlocks etc) then to sleep for a few seconds and then retry the same transaction.
However, in most cases, indefinite processing retries are not desirable and you would ultimately need to admit defeat and implement a mechanism to log the message and the error and remove it from the queue.
But I wouldn't 'retry' business failures (e.g. Business Rules, Validation etc) and the behaviour should be defined in your requirements of how to handle these (e.g. account is overdrawn, message is not in a correct format or not valid, etc), e.g. by returning a "NACK" type message back to the source.
HTH
MSMQ sends messages from one queue to another queue.
Let's say you have a queue on a remote machine. You want to send a message to that queue.
So you create a sender. A sender is an application that can use the MSMQ transport to send a message. This can be a .Net queue client (System.Messaging), a WCF service consumer (either over netMsmqBinding or msmqIntegrationBinding, BizTalk using the MSMQ adapter, etc etc.
When you send the message, what actually happens is:
The MSMQ queue manager on the sender machine writes the message to a temporary local queue.
The MSMQ queue manager on the sender machine connects to the MSMQ manager on the receiving machine and transmits the message.
The MSMQ queue manager on the receivers machine puts the message onto the destination queue.
In certain situations MSMQ will encounter messages which for some reason or another cannot be received on the destination queue. In these situations, if you have indicated that a message will use the dead-letter queue then MSMQ will make sure that the message is forwarded to the dead-letter queue.
We have a Pub / Sub system based on NServiceBus, where we have intermittent issues with messages getting stuck on the Publishers outgoing queue indefinitely, rather than being transmitted to the Subscribers input queues.
Points to note:
When we restart the Publisher Service and Subscriber services, message flow resumes normally for a while.
The problem seems to occur more often if a sustained period of time between messages occurs.
The publisher service resides on the LAN, the subscribers on the otherside of a firewall.
Some messages get through! As mentioned after service restarts, things go fine for a while.
Using QueueExplorer, I can see the messages on the Outgoing queue have a state of WAITING.
Annoyingly our development environment does not exhibit this behaviour, but then again the publisher and subscribers all reside on the same LAN in this environment.
MSMQ messages being stuck in an outgoing queue is purely an MSMQ issue. Restarting the Publisher and Subscriber services should make no difference as they are not directly involved in message delivery. If you can fix the problem by ONLY restarting the Pub/Sub services and NOT the Message Queuing services then it looks like a resources/memory leak problem.
I imagine something like this happening:
Messages flow to destination, which uses up kernel memory in storing them
For some reason, kernel memory runs out (too many messages, memory leak, whetever)
Destination now rejects new messages as they cannot be loaded into memory from the wire
Connection is reset and not re-connected until WaitTime value reached; Queue is "waiting" at this point
System loops through (3) and (4) until ...
Pub/Sub services are restarted and now there is sufficient resources for messages to be delivered
Goto (2)
Occasional messages get through when just enough kernel memory is temporarily freed up by one of the many services and device drivers that use it.
Item 4 of this blog post is the most likely culprit:
http://blogs.msdn.com/b/johnbreakwell/archive/2006/09/18/insufficient-resources-run-away-run-away.aspx
Cheers
John Breakwell
We had a similar scenario in production, it turned out we migrated one of our subscriber endpoints to a new physical host and forgot to unsubscribe before shutting down the old endpoint. Our publisher was trying to deliver messages to both the old and new endpoints but could only reach the new one. Eventually the publishers outbound queue grew so large that it started affecting all outgoing messages.
I have run into this issue as well, I know it is not Item 4, as I don't send anything to it before it gets stuck in the outgoing queue. If I let both publisher and subscriber sit for about 10 minutes before sending a message, it never leaves the outgoing queue. If I send a message before that amount of time, it flows fine. Also, if I restart the subscriber the message will then flow. This is reproducible every time I let them sit idle for 10 minutes.
I think I found the answer here, at least this fixed the issue I was having:
http://support.microsoft.com/kb/2554746
Also, in my case it had nothing to do with restarting, so don't let that throw you off, I did exhibit the symptoms in the netstat and messages would initially go through when the client was first started up.
Just to throw my 2p in:
We had an issue where the message queuing service had some kind of memory leak and would consume large amounts of memory which is did not release.
This lead to messages getting stuck for long periods of time - although they would eventually be delivered (sometimes after 3 days).
We have not bothered fixing this yet as it only happens when the service is under heavy load which does not happen often.
I need to work with MSMQ (Microsoft Message Queuing). What is it, what is it for, how does it work? How is it different from web services?
With all due respect to #Juan's answer, both are ways of exchanging data between two disconnected processes, i.e. interprocess communication channels (IPC). Message queues are asynchronous, while webservices are synchronous. They use different protocols and back-end services to do this so they are completely different in implementation, but similar in purpose.
You would want to use message queues when there is a possibility that the other communicating process may not be available, yet you still want to have the message sent at the time of the client's choosing. Delivery will occur the when process on the other end wakes up and receives notification of the message's arrival.
As its name states, it's just a queue manager.
You can Send objects (serialized) to the queue where they will stay until you Receive them.
It's normally used to send messages or objects between applications in a decoupled way
It has nothing to do with webservices, they are two different things
Info on MSMQ:
https://msdn.microsoft.com/en-us/library/ms711472(v=vs.85).aspx
Info on WebServices:
http://msdn.microsoft.com/en-us/library/ms972326.aspx
Transactional Queue Management 101
A transactional queue is a middleware system that asynchronously routes messages of one sort of another between hosts that may or may not be connected at any given time. This means that it must also be capable of persisting the message somewhere. Examples of such systems are MSMQ and IBM MQ
A Transactional Queue can also participate in a distributed transaction, and a rollback can trigger the disposal of messages. This means that a message is guaranteed to be delivered with at-most-once semantics or guaranteed delivery if not rolled back. The message won't be delivered if:
Host A posts the message but Host B
is not connected
Something (possibly but not
necessarily initiated from Host A)
rolls back the transaction
B connects after the transaction is
rolled back
In this case B will never be aware the message even existed unless informed through some other medium. If the transaction was rolled back, this probably doesn't matter. If B connects and collects the message before the transaction is rolled back, the rollback will also reverse the effects of the message on B.
Note that A can post the message to the queue with the guarantee of at-most-once delivery. If the transaction is committed Host A can assume that the message has been delivered by the reliable transport medium. If the transaction is rolled back, Host A can assume that any effects of the message have been reversed.
Web Services
A web service is remote procedure call or other service (e.g. RESTFul API's) published by a (typically) HTTP Server. It is a synchronous request/response protocol and has no guarantee of delivery built into the protocol. It is up to the client to validate that the service has been correctly run. Typically this will be through a reply to the request or timeout of the call.
In the latter case, web services do not guarantee at-most-once semantics. The server can complete the service and fail to deliver a response (possibly through something outside the server going wrong). The application must be able to deal with this situation.
IIRC, RESTFul services should be idempotent (the same state is achieved after any number of invocations of the same service), which is a strategy for dealing with this lack of guaranteed notification of success/failure in web service architectures. The idea is that conceptually one writes state rather than invoking a service, so one can write any number of times. This means that a lack of feedback about success can be tolerated by the application as it can re-try the posting until it gets a 'success' message from the server.
Note that you can use Windows Communication Foundation (WCF) as an abstraction layer above MSMQ. This gives you the feel of working with a service - with only one-way operations.
For more information, see:
http://msdn.microsoft.com/en-us/library/ms789048.aspx
Actually there is no relation between MSMQ and WebService.
Using MSMQ for interprocess communication (you can use also sockets, windows messaging, mapped memory).
it is a windows service that responsible for keeping messages till someone dequeue them.
you can say it is more reliable than sockets as messages are stored on a harddisk but it is slower than other IPC techniques.
You can use MSMQ in dotnet with small lines of code, Just Declare your MessageQueue object and call Receive and Send methods.
The Message itself can be normal string or binary data.
As everyone has explained MSMQ is used as a queue for messages. Messages can be wrapper for actual data, object and anything that you can serialize and send across the wire. MSMQ has it's own limitations. MSMQ 1.0 and MSMQ 2.0 had a 4MB message limit. This restriction was lifted off with MSMQ 3.0. Message oriented Middleware (MOM) is a concept that heavily depends on Messaging. Enterprise Service Bus foundation is built on Messaging. All these new technologies, depend on Messaging for asynchronous data delivery with reliability.
MSMQ stands for Microsoft Messaging Queue.
It is simply a queue that stores messages formatted so that it can pass to DB (may on same machine or on Server). There are different types of queues over there which categorizes the messages among themselves.
If there is some problem/error inside message or invalid message is passed, it automatically goes to Dead queue which denotes that it is not to be processed further. But before passing a message to dead queue it will retry until a max count and till it is not processed. Then it will be sent to the Dead queue.
It is generally used for sending log message from client machine to server or DB so that if there is any issue happens on client machine then developer or support team can go through log to solve problem.
MSMQ is also a service provided by Microsoft to Get records of Log files.
You get Better Idea from this blog http://msdn.microsoft.com/en-us/library/ms711472(v=vs.85).aspx.