I'm wondering if its possible to write a Powershell script to move a message from one MSMQ queue to another?
I have a few failed messages (i.e. messages that did not complete because they involved, say, a call to a server that was down, for example) in an error queue that I would like to retry by moving them back into their original processing queue. I am using NServiceBus v6 handlers to listen for messages that enter the original processing queue that will resend the message if they are put back into that queue.
Has anyone done this before using Powershell?
In particular, I would like to do the following in Powershell:
Find a message based on a specific identifier (string) within the message body
Move this message to a specific queue
I know Powershell has commands such as Move-MsmqMessage. I have been unable to find an example online that I can re-use for this function.
Any help much appreciated.
If you have messages in the error queue from NServiceBus, you can use this tool to return them to the original queue they were in.
https://github.com/ParticularLabs/MsmqReturnToSourceQueue
It checks the headers what the original queue was and sends them there again. The idea is that any endpoint can send messages to the same error queue. They can then either be digested by ServiceControl or any other tool. If you leave them in the error queue, this ReturnToSourceQueue can send them back to the original queue.
Related
I have a transactional private message queue (among other message queues on which I have not seen this problem) on a Windows Server 2008 R2 server.
This particular queue has a recurring problem happening every few weeks where the console shows a nonzero count of messages in the queue, but it does not have any messages in the queue itself or any subqueue. Queue Explorer shows the same thing. Performance counters indicate there are messages like the count in the built-in msmq console and queue explorer.
I cannot find any messages. I understand that I could see a situation like this for outgoing queues with dead letter tracking such that it may have been delivered to a remote machine but not yet processed. This is not an outgoing queue, though. Messages are sourced from remote machines and have landed here on this machine.
Also, I am certain that the count I'm seeing are not journal messages or subqueues.
Does this make any sense? Is there a logical explanation for this and under some circumstance this is expected? If so, what is it?
EDIT: Removed info about purging queue removing the count - that was incorrect. Purging actually does nothing and leaves me in the same state as before with a count reflected, but no messages showing.
As you noted, you can see a message count on an outgoing queue if source journaling is in use. The invisible messages are there in case they need to be moved to the DLQ.I would expect your problem to be similar - there should be a visible message in the outgoing queue and an invisible message in the destination queue because delivery hasn't completed. I assume a handshaking or storage acknowledgement has been lost along the way. Or maybe the message has been processed and removed from the queue but MSMQ couldn't update the sender of the fact. Check the outgoing queues on the remote machines sending TO this queue.
I'm working on a BizTalk application that sends messages to a Transactional MSMQ location. If it is unable to write to the Queue, the message gets sent to the Dead Letter Queue on the BizTalk server. Is there a better way to handle this? I would prefer the message to suspend so we would get notification from BizTalk360. If it is not possible to suspend the message, and the message must go to the Dead Letter Queue, how can I recover the message once the issue that prevented writing to the Queue is corrected?
Ok, so the best way that I have found to accomplish this is a bit of a hack in my opinion, but it works. I am setting up a receive port/location listening to the Dead Letter Queue at DIRECT=.\SYSTEM$;DEADXACT. I also added an orchestration that listens for messages coming from that receive port and immediately throws an exception suspending the message allowing BizTalk360 to send me notification that there is an issue.
Is there some standard configuration setting, service, or tool that accepts messages from one queue and moves them on to another one? Automatically handling the dead message problem, and providing some of retry capability? I was thinking this is what "MSMQ Message Routing" does but can't seem to find documentation on it (except for on Windows Mobile 6, and I don't know if that's relevant).
Context:
I understand that when using MSMQ you should always write to a local queue so that failure is unlikely, and then X should move that message to a remote queue. Is my understanding wrong? Is this where messaging infrastructure like Biztalk comes in? Is it unnecessary to write to a local queue first to absolutely ensure success? Am I supposed to build X myself?
As Hugh points out, you need only one MSMQ Queue to Send messages in one direction from a source to a destination. Source and destination can be on the same server, same network or across the internet, however, both source and destination must have the MSMQ service running.
If you need to do 'message' routing (e.g. a switch which processes messages from several source or destination queues, or routing a message to one or more subscribers based on the type of message etc) you would need more than just MSMQ queue.
Although you certainly can use BizTalk to do message routing, this would be expensive / overkill if you didn't need to use other features of BizTalk. Would recommend you look at open source, or building something custom yourself.
But by "Routing" you might be referring to the queue redirection capability when using HTTP as the transport e.g. over the internet (e.g. here and here).
Re : Failed delivery and retry
I think you have most of the concepts - generally the message DELIVERY retry functionality should be implicit in MSMQ. If MSMQ cannot deliver the message before the defined expiry, then it will be returned on the Dead Letter Queue, and the source can then process messages from the DLQ and then 'compensate' for them (e.g. reverse the actions of the 'send', indicate failure to the user, etc).
However 'processing' type Retries in the destination will need to be performed by the destination application / listener (e.g. if the destination system is down, deadlocks, etc)
Common ways to do this include:
Using 2 Phase commit - under a distributed unit of work, pull the message off MSMQ and process it (e.g. insert data into a database, change the status of some records etc), and if any failure is encountered, then leave the message back onto the queue and the DB changes will be rolled back.
Application level retries - i.e. on the destination system, in the event of 'retryable' type errors (timeout due to load, deadlocks etc) then to sleep for a few seconds and then retry the same transaction.
However, in most cases, indefinite processing retries are not desirable and you would ultimately need to admit defeat and implement a mechanism to log the message and the error and remove it from the queue.
But I wouldn't 'retry' business failures (e.g. Business Rules, Validation etc) and the behaviour should be defined in your requirements of how to handle these (e.g. account is overdrawn, message is not in a correct format or not valid, etc), e.g. by returning a "NACK" type message back to the source.
HTH
MSMQ sends messages from one queue to another queue.
Let's say you have a queue on a remote machine. You want to send a message to that queue.
So you create a sender. A sender is an application that can use the MSMQ transport to send a message. This can be a .Net queue client (System.Messaging), a WCF service consumer (either over netMsmqBinding or msmqIntegrationBinding, BizTalk using the MSMQ adapter, etc etc.
When you send the message, what actually happens is:
The MSMQ queue manager on the sender machine writes the message to a temporary local queue.
The MSMQ queue manager on the sender machine connects to the MSMQ manager on the receiving machine and transmits the message.
The MSMQ queue manager on the receivers machine puts the message onto the destination queue.
In certain situations MSMQ will encounter messages which for some reason or another cannot be received on the destination queue. In these situations, if you have indicated that a message will use the dead-letter queue then MSMQ will make sure that the message is forwarded to the dead-letter queue.
I am writing an acceptor application and using a persistent FIX session. I am trying to write a recovery mode, such that if I go offline or my program restarts, when I reconnect I want to reprocess all the messages sent to me during the day to get back to the current state.
To do this, when I start up I send a resend request for all messages to the server. They fire me back all the relevant messages, and they are marked possdupflag=Y and possresend=Y. Before each message, they send a sequence reset for the repeated message they are about to send.
The problem is though, these messages do not seem to be processed by my message cracker. Both fromAdmin and fromApp do not get these messages. I assume they are being ignored because of the dup flag and/or resend. So is there a way for me to tell QuickFIX that I want to see these messages?
On that note- if anyone has any recommendations on better recovery processes I would be open to them.
Thanks.
There's at least a couple of potential problems with this recovery strategy. The first is that it's not very friendly to your trading counterparty. If you only receive a small number of messages during your session then it may not be an issue, but if you receive hundreds of thousands of messages then your counterparty might complain about the massive resends.
The other issue is that message resend is intended for error recovery and is managed by the session protocol layer. In QuickFIX/J (and other FIX engines) the session maintains recovery state in addition to sending the ResendRequest automatically when it detects a sequence number gap. Your approach might work if you reset the next expected incoming sequence number to 1. When the session receives the next message with a higher sequence number it will detect the gap and request the missing messages. If the messages are validated, they will be forwarded to application layer with the PossDup flag set. If you send the ResendRequest message yourself the behavior is undefined since the session state will not have been set up properly.
I recommend using a MessageLog implementation to store your incoming messages in a form you can use for recovery when your application starts. You can look at the implementation of the existing message logs (FileLog, JdbcLog) to get some ideas.
The behaviour occurs because the engine's persistance system tells it that the recieved messages are resent messages and so (per the FIX protocol specification) are discarded. Here we save FIXml strings into our database to provide a similar recovery ability to that which you describe(they are also written to xml files on disk for other reasons). I don't believe that there is any way to tell quickfix that you want to see duplicate messages but it is probably better to use a different form or persistance to save on connection overheads. Quickfix does provide a way of outputting messages to file as they come in if that helps.
I too have the same issue and What Frank Says is absolutely correct ,
Just use the below method to set the target sequence number to the begin seq number of the desired resend req .
getSession()->setNextTargetMsgSeqNum(atoi(seq.c_str()));
The engine internally identifies that the target number is way too large and automatically sends resend request , and all messages will be captured in onMessage call back itself as usual
So, I am new to MSMQ and NServiceBus. I played around with demos and got a working scenario going with NServiceBus. (Getting my own up and running was even easier than following the demo thanks to the new Modeling tools!)
I then went and presented my plan (based off my work with the demo and my own model) to my co-workers. Two of them were versed in using MSMQ and started asking me questions about how I will handle "Dead Letters".
I had never heard of "Dead Letters". They explained that it is queue used for messages that things cannot be sent (either because the other end refuses them or if the other end is not there).
The concern of my co-workers is that if we don't have Dead Letter queues then how will we stop a message from blocking the queue? (If the queue is FIFO and the top message can't be sent, then it blocks the other messages behind it right?)
On the other hand, if we have "Dead Letter" queues how are they managed? (Do I get an event from NServiceBus that tells me a new message is in the Dead Letter queue? How do I configure when a message will go to the Dead Letter queue? How can I try to re-send a Dead Letter message?)
So basically, how does NServiceBus deal with undeliverable messages?
Typically, you specify an error queue as well, which is where messages that couldn't be sent will go. Look at their MSMQ example:
http://docs.particular.net/nservicebus/msmq/transportconfig &
http://docs.particular.net/nservicebus/msmq/connection-strings
<MsmqTransportConfig InputQueue="MyClient" ErrorQueue="error"
NumberOfWorkerThreads="1" MaxRetries="5"/>
In this scenario, the error queue will reside on the same machine as the input queue. This may also be a remote queue, if preferred. MaxRetries refers to the number of attempts to send before it gets put into the error queue. How you choose to handle the error queue is up to you, however.