Occasionally when NServiceBus picks up a message, it fails with the exception below. This causes NServiceBus to retry (up to it's configured retry limit). Sometimes one of the retries result in the message being handled successfully, but it's common for all retries to fail with the same exception. In this case, the message is routed to the error queue as expected when all retries fail.
My question is... what would cause this exception in the first place? It doesn't appear related to my message handler code, as my code doesn't appear in the stack trace.
NServiceBus version: 2.6.0.1504
OS: Windows Server 2003
Handler code is targeting .NET 3.5 or earlier
Here is the full exception message and stack trace:
NServiceBus.Unicast.Transport.Msmq.MsmqTransport [(null)] - Failed raising 'finished message processing' event. System.Messaging.MessageQueueException: Cannot enlist the transaction.
at System.Messaging.MessageQueue.SendInternal(Object obj, MessageQueueTransaction internalTransaction, MessageQueueTransactionType transactionType)
at System.Messaging.MessageQueue.Send(Object obj, MessageQueueTransactionType transactionType)
at NServiceBus.Unicast.Transport.Msmq.MsmqTransport.SendMsmqMessage(Message m, String destination)
at NServiceBus.Unicast.Transport.Msmq.MsmqTransport.Send(TransportMessage m, String destination)
at NServiceBus.Unicast.UnicastBus.SendReadyMessage(Boolean startup)
at NServiceBus.Unicast.UnicastBus.TransportFinishedMessageProcessing(Object sender, EventArgs e)
at NServiceBus.Unicast.Transport.Msmq.MsmqTransport.OnFinishedMessageProcessing()
I see the method NServiceBus.Unicast.UnicastBus.SendReadyMessage(Boolean startup) in your stack trace.
This tells me that your endpoint is connected to a Distributor. After all your message handlers have completed, an endpoint connected to a distributor will send the ReadyMessage back to the Distributor's control queue to say "I'm done with that work. Please send more!"
This is supposed to happen within the same transaction as the rest of your work, but apparently the MSMQ send is having trouble enlisting in that transaction. This could point to a problem with DTC. It would be interesting to know what else you have going on in your message handlers for that message type that also enlist in the transaction. You aren't manually committing or rolling anything back are you?
Related
I'm using Kafka 0.11.0.0. I have a test program that publishes to a Kafka topic; if the zookeeper and Kafka servers are down (which is normal in my development environment; I bring them up as needed) then the call to KafkaProducer<>.send() hangs indefinitely.
I either need to have send() return, preferably indicating the error; or I need a way to check whether the servers are up or down. Basically, I want my test tool to be able tell me, "Hey, dummy, start up Kafka!" instead of hanging.
Is there a way for my producer task to determine whether the servers are up or down?
I'm calling the send() like this:
kafkaProducer.send(new ProducerRecord<>(KAFKA_TOPIC, KAFKA_KEY,
message), (rm, ex) -> {
System.out.println("**** " + rm + "\n**** " +ex);
});
I have linger.ms = 1; I've tried retries=0, 1, and 2, and send() still blocks. I've never seen the callback called.
Older messages suggest setting metadata.fetch.timeout.ms to a small value, but that's gone in 0.11. Others suggest calling command line utilities to see if the servers are OK...but the referenced utilities also seem to be gone.
What's the graceful way to get this done?
We can send messages to broker in three ways :
Fire-and-forget :
We send a message to the server and don’t really care if it arrives successfully or not. Most of the time, it will arrive successfully, since Kafka is highly available and the producer will retry sending messages automatically. However, some messages will get lost using this method.
Asynchronous send
We call the send() method with a callback function, which gets triggered when it receives a response from the Kafka broker.
Synchronous send
We send a message, the send() method returns a Future object, and we use get() to wait on the future and see if the send() was successful or not.
The simplest way to send a message synchronously is as follows:
ProducerRecord<String, String> record =
new ProducerRecord<>(KAFKA_TOPIC, KEY, message);
try {
producer.send(record).get();
} catch (Exception e) {
e.printStackTrace();
}
Here, we are using Future.get() to wait for a reply from Kafka. This method will throw an exception if the record is not sent successfully to Kafka. If there were no errors, we will get a RecordMetadata object that we can use to retrieve the offset the message was written to.
hope this helps.
That is strange. It should return with an error saying either "Failed to update metadata" or "Expiring x number of records".
Check request.timeout.ms and max.block.ms setting for your producer. By default request.timeout.ms is 60 seconds long
I have one MSMQ queue which is listened by five windows services. I used BeginPeek and PeekCompleted event for this purpose. My problem is among five services, only one service is the right recipient of the message. All four just read message, but no action is performed. This can only be identified when we read MQ message.
Now, I added a code in my services to check, if the criteria matches and the message is being processed by the right service, then I am using Receive to dequeue the message from MSMQ. Is that a good idea?
Secondly, If the message doesnot satisfy condition and all five services just peeked it, but not received, the message still lies in queue. I understand. But the same message is being processed infinite times, as the message was never removed.
private void queue_PeekCompleted(object sender, PeekCompletedEventArgs e)
{
MessageQueue queue = (MessageQueue)sender;
//Message msg = queue.EndPeek(e.AsyncResult);
Message msg = e.Message;
//Read message and check if the criteria matches
if(CriteriaMatches)
{
queue.ReceiveById(e.Message.Id);
}
queue.EndPeek(e.AsyncResult);
queue.BeginPeek();
}
Appreciate your help.
Thanks,
Fayaz
Set the messages to expire after a set (short) period. They will then move to the dead letter queue where you can have another service waiting for arrivals. This service could then raise an alert, for example, as soon as a message arrives.
I want to consume msmq service. But unable to send message to queue.
Here is my code.
System.Messaging.MessageQueue msmQ = new System.Messaging.MessageQueue("net.msmq://myServerName/private/MyQueueName");
msg ="<nodeDetails><node>Node1</node></nodeDetails>";//Dummy value. it is XML structure consist of multiple node
msmQ.Send(msg);
It gives me an error on msmQ.Send(msg)
Length cannot be less than zero. Parameter name: length
The following things are installed on my m/c:
Microsoft Message Queue(MSMQ)Server
Window Activation Process
Also when I tried as
bool msmQExits = MessageQueue.Exists("net.msmq://myServerName/private/MyQueueName");
it gives "Path syntax is invalid". I am not able to get anything on it.
I just have a msmq URL net.msmq://myServerName/private/MyQueueName.
How can I consume such URL and send my message to "MyQueueName"?
Change your queue name to this:
var queueName = #"FormatName:DIRECT=HTTP://URLAddressSpecification/net.msmq://myServerName/private/MyQueueName";
And you cannot check if a remote query exists by MessageQueue.Exists method. It will always throw an exception.
You can check these links for more info:
MSMQ FormatName
Checking if a remote query exists
Also, the problem is not with the message you see that length is less than 0. If you go deeper and check the stack trace you’ll see that your queue name has an invalid format. It tries to find FORMAT occurrence inside your queue name, doesn’t find it and Substring() method returns -1 there.
Stacktrace:
at System.String.Substring(Int32 startIndex, Int32 length)
at System.Messaging.MessageQueue.ResolveFormatNameFromQueuePath(String queuePath, Boolean throwException)
at System.Messaging.MessageQueue.get_FormatName()
at System.Messaging.MessageQueue.SendInternal(Object obj, MessageQueueTransaction internalTransaction, MessageQueueTransactionType transactionType)
at System.Messaging.MessageQueue.Send(Object obj)
at MessagingTest.Program.SendMessage(String str, Int32 x) in c:\Users\ivan.yurchenko\Documents\Visual Studio 2013\Projects\MSMQTestProjects\MessagingTest\MessagingTest\Program.cs:line 21
at MessagingTest.Program.<Main>b__1() in c:\Users\ivan.yurchenko\Documents\Visual Studio 2013\Projects\MSMQTestProjects\MessagingTest\MessagingTest\Program.cs:line 38
at System.Threading.Tasks.Task.InnerInvoke()
at System.Threading.Tasks.Task.Execute()
Here is an example for how to consume the service.
It has Wcf Service,Physical MSMQ, and the client project. So you have to have a WCF service to receive the message and msmq to store the message and a client to send the message.
http://www.codeproject.com/Articles/326909/Creating-a-WCF-Service-with-MSMQ-Communication-and
We are using NServiceBus on production, and in our log files we see the following error:
ERROR Our.Namespace.SomeMessageHandler [(null)] <(null)> - MethodName --> end with exception: NServiceBus.Unicast.Queuing.FailedToSendMessageException: Failed to send message to address: our.namespace.worker#somemachinename ---> System.Messaging.MessageQueueException: Cannot enlist the transaction.
at System.Messaging.MessageQueue.SendInternal(Object obj, MessageQueueTransaction internalTransaction, MessageQueueTransactionType transactionType)
at NServiceBus.Unicast.Queuing.Msmq.MsmqMessageSender.NServiceBus.Unicast.Queuing.ISendMessages.Send(TransportMessage message, Address address)
--- End of inner exception stack trace ---
at NServiceBus.Unicast.Queuing.Msmq.MsmqMessageSender.ThrowFailedToSendException(Address address, Exception ex)
at NServiceBus.Unicast.Queuing.Msmq.MsmqMessageSender.NServiceBus.Unicast.Queuing.ISendMessages.Send(TransportMessage message, Address address)
at NServiceBus.Unicast.UnicastBus.SendMessage(List`1 addresses, String correlationId, MessageIntentEnum messageIntent, Object[] messages)
at NServiceBus.Unicast.UnicastBus.SendMessage(Address address, String correlationId, MessageIntentEnum messageIntent, Object[] messages)
at NServiceBus.Unicast.UnicastBus.NServiceBus.IBus.Send(Address address, Object[] messages)
at Our.Namespace.SomeMessageHandler.MethodName(EventLogVO eventLog, IApplicationContext applContext, CreateEventLogHistory message)
The queue on the target machine exists (double checked). The strange thing here is that it doesn't happen all the time and for each message sent to that queue, but happens occasionally (which means that there are messages that arrive to that queue).
Searched and didn't find a similar case.
What am I missing here?
We recently had this error and tracked it down to be one of the following:
Database performance. We had a long running query that we tuned up, but the problem persisted.
Large transaction scope. You may be doing something that would cause too many resource managers to be involved.
MSMQ Resources. Ultimately our disk was not fast enough to perform the IO required for what we were doing with MSMQ.
I would try to track down the true source (hopefully you get some ideas from above). But if all else fails, first turn on the System.Transactions logging to see if it is truly a timeout. If it is, then use this section in the app.config
I had this happen to me in a handler that took over a minute to complete,
I ended up increasing the timeout by adding this to the app.config
<system.transactions>
<defaultSettings timeout="00:05:00"/>
</system.transactions>
As per this:
https://erictummers.wordpress.com/2014/05/28/cannot-enlist-the-transaction/
Additionally, I changed machine config as per this post:
http://blogs.msdn.com/b/ajit/archive/2008/06/18/override-the-system-transactions-default-timeout-of-10-minutes-in-the-code.aspx
I also wrote a quick app that edits the machine.configs:
https://github.com/hfortegcmlp/MachineConfigEditor
Let's say I try to send to an authenticated transactional queue,
by calling msg.send(object,MessageQueueTransactionType.Single), message does not receive in transactional queue, no exception thrown.
What I want to accomplish is after sending, if message fail to send, perform some function and abort transaction, yet it doesn't throw exception, so I am unable to process it.
I am sending object from Web Application in local to local message queue.
My code is as follows in my web application:
MessageQueueTransaction mqTran=new MessageQueueTransaction();
try
{
using(System.Messaging.Message msg=new System.Messaging.Message(){
mqTran.Begin();
MessageQueue adminQ = new MessageQueue(AdminQueuePath);
MessageQueue msgQ = new MessageQueue(queuePath);
msgQ.DefaultPropertiesToSend.Recoverable = true;
msg.body = object;
msg.Recoverable=true;
msg.Label="Object";
msg.TimeToReachQueue=new TimeSpan(0,0,30);
msg.AcknowledgeType=AcknowledgeTypes.FullReachQueue;
msg.ResponseQueue=adminQ;
msg.AdministrationQueue=adminQ;
msgQ1.Send(msg,MessageQueueTransactionType.Single);
mqTran.Commit();
}
catch(Exception e)
{
mqTran.Abort();
//Do some processing if fail to send
}
It's not going to throw an exception for failure to deliver, only for failure to place on the queue. One of the points of message queueing is that the messages are durable so that you can take appropriate measures if delivery fails. This means you need to program another process to read the dead letter queue. The image below is taken from MSDN.
Because the entire process is asynchronous, your code flow is not going to be exception-driven the way your code block would like. Your transaction is simply the "sending transaction" in this workflow.
Recommendation: Check your message queue to find the messages, either in the outgoing queue or the transactional dead-letter queue.