I have one MSMQ queue which is listened by five windows services. I used BeginPeek and PeekCompleted event for this purpose. My problem is among five services, only one service is the right recipient of the message. All four just read message, but no action is performed. This can only be identified when we read MQ message.
Now, I added a code in my services to check, if the criteria matches and the message is being processed by the right service, then I am using Receive to dequeue the message from MSMQ. Is that a good idea?
Secondly, If the message doesnot satisfy condition and all five services just peeked it, but not received, the message still lies in queue. I understand. But the same message is being processed infinite times, as the message was never removed.
private void queue_PeekCompleted(object sender, PeekCompletedEventArgs e)
{
MessageQueue queue = (MessageQueue)sender;
//Message msg = queue.EndPeek(e.AsyncResult);
Message msg = e.Message;
//Read message and check if the criteria matches
if(CriteriaMatches)
{
queue.ReceiveById(e.Message.Id);
}
queue.EndPeek(e.AsyncResult);
queue.BeginPeek();
}
Appreciate your help.
Thanks,
Fayaz
Set the messages to expire after a set (short) period. They will then move to the dead letter queue where you can have another service waiting for arrivals. This service could then raise an alert, for example, as soon as a message arrives.
edited at 2015-11-25 02:10
My ejabberd version is 14.12 and erlang R17B, so this code seems not useful because erlang:system_info(otp_release) in R17B retruns "17"
ejabberd_listener.erl
SockOpts2 =
try erlang:system_info(otp_release) >= "R13B" of
true -> [{send_timeout_close, true} | SockOpts];
false -> SockOpts
catch
_:_ -> []
end,
I added {send_timeout_close, true} manually in listen option, my problem sees to be solved because socket is closed at the same time of send timeout, trying to send follow-up messages in the queue would receive a {error,enotconn} response.
when a {gen_event, 'closed'} msg comes, c2s process terminate normally.
edited at 2015-11-24 03:40
Maybe I found method to reproduce this problem:
1. build a normal c2s connection with xmpp client
2. cut the client's network with some tools, eg. clumsy(drops all the tcp packet from server)
3. keep sending large packets to the c2s process
At first, gen_tcp:send returns ok before sendbuffer fills
Then, gen_tcp:send retruns {error,timeout} because of sendbuffer is filled
the process calls ejabberd_socket:close(Socket) to close the connection
send_text(StateData, Text) when StateData#state.mgmt_state == active ->
catch ?INFO_MSG("Send XML on stream = ~ts", [Text]),
case catch (StateData#state.sockmod):send(StateData#state.socket, Text) of
{'EXIT', _} ->
(StateData#state.sockmod):close(StateData#state.socket),
error;
_ ->
ok
end;
But ejabberd_socket:close/1 seems to be an async call, so the c2s process would handle next message in message_queue, keep calling gen_tcp:send/2, waiting for a send_timeout.
But at this time, ejabberd_receiver called gen_tcp:close(Socket), the socket is closed, so previous gen_tcp:send/2 never returns. I have tried several times with this method, it happens 100%.
Briefly, if I send packets to a client socket which is unable to receive packet and the sendbuffer is fullfilled, i would receive a {error, timeout} after sendtimeout. But, if another async process closed the socket when i am waiting for a sendtimeout with gen_tcp:send/2, I would never get a response.
so, I did this with erl, and gen_tcp:send/2 no response ( cuting network at step3, keep sending packet, async close).
I want to know is this a problem or because reason of myself?
original post below
Generally in ejabberd , i route message to client process, send to tcp socket via this function. And it works well most time.
Module ejabberd_c2s.erl
send_text(StateData, Text) when StateData#state.mgmt_state == active ->
catch ?INFO_MSG("Send XML on stream = ~ts", [Text]),
case catch (StateData#state.sockmod):send(StateData#state.socket, Text) of
{'EXIT', _} ->
(StateData#state.sockmod):close(StateData#state.socket),
error;
_ ->
ok
end;
But in some cases the c2s pid blocked on gen_tcp:send like this
erlang:process_info(pid(0,8353,11)).
[{current_function,{prim_inet,send,3}},
{initial_call,{proc_lib,init_p,5}},
{status,waiting},
{message_queue_len,96},
{messages ...}
...
Most cases happened when user's network status not so good, the receiver process should send 2 messages to c2s pid , and c2s would terminate session or wait for resume
{'$gen_event',closed}
{'DOWN',#Ref<0.0.1201.250595>,process,<0.19617.245>,normal}
I printed message queue in the c2s process, the 2 msg are in the queue, waiting to be handled. Unfortunately,
the queue does not move any more becasue the process had blocked before handling these messages, as described above, stacked at prim_inet:send/3 when tring to do gen_tcp:send/2.
The queue grows very large after days, and ejabberd crahes when the process asking for more memory.
prim_inet:send/3 source :
send(S, Data, OptList) when is_port(S), is_list(OptList) ->
?DBG_FORMAT("prim_inet:send(~p, ~p)~n", [S,Data]),
try erlang:port_command(S, Data, OptList) of
false -> % Port busy and nosuspend option passed
?DBG_FORMAT("prim_inet:send() -> {error,busy}~n", []),
{error,busy};
true ->
receive
{inet_reply,S,Status} ->
?DBG_FORMAT("prim_inet:send() -> ~p~n", [Status]),
Status
end
catch
error:_Error ->
?DBG_FORMAT("prim_inet:send() -> {error,einval}~n", []),
{error,einval}
end.
It seems the port driver did not reply {inet_reply,S,Status} after erlang:port_command(S, Data, OptList) .
the gen_tcp:send function would block infinity, Can anyone explain this?
It depends on the version of Erlang you are using. The option to timeout on gen_tcp send is not used on old ejabberd version because it was not available at that time in Erlang. Moreover, you have to use a very recent version of Erlang as some bug were fixed in Erlang itself regarding that options.
We are using NServiceBus on production, and in our log files we see the following error:
ERROR Our.Namespace.SomeMessageHandler [(null)] <(null)> - MethodName --> end with exception: NServiceBus.Unicast.Queuing.FailedToSendMessageException: Failed to send message to address: our.namespace.worker#somemachinename ---> System.Messaging.MessageQueueException: Cannot enlist the transaction.
at System.Messaging.MessageQueue.SendInternal(Object obj, MessageQueueTransaction internalTransaction, MessageQueueTransactionType transactionType)
at NServiceBus.Unicast.Queuing.Msmq.MsmqMessageSender.NServiceBus.Unicast.Queuing.ISendMessages.Send(TransportMessage message, Address address)
--- End of inner exception stack trace ---
at NServiceBus.Unicast.Queuing.Msmq.MsmqMessageSender.ThrowFailedToSendException(Address address, Exception ex)
at NServiceBus.Unicast.Queuing.Msmq.MsmqMessageSender.NServiceBus.Unicast.Queuing.ISendMessages.Send(TransportMessage message, Address address)
at NServiceBus.Unicast.UnicastBus.SendMessage(List`1 addresses, String correlationId, MessageIntentEnum messageIntent, Object[] messages)
at NServiceBus.Unicast.UnicastBus.SendMessage(Address address, String correlationId, MessageIntentEnum messageIntent, Object[] messages)
at NServiceBus.Unicast.UnicastBus.NServiceBus.IBus.Send(Address address, Object[] messages)
at Our.Namespace.SomeMessageHandler.MethodName(EventLogVO eventLog, IApplicationContext applContext, CreateEventLogHistory message)
The queue on the target machine exists (double checked). The strange thing here is that it doesn't happen all the time and for each message sent to that queue, but happens occasionally (which means that there are messages that arrive to that queue).
Searched and didn't find a similar case.
What am I missing here?
We recently had this error and tracked it down to be one of the following:
Database performance. We had a long running query that we tuned up, but the problem persisted.
Large transaction scope. You may be doing something that would cause too many resource managers to be involved.
MSMQ Resources. Ultimately our disk was not fast enough to perform the IO required for what we were doing with MSMQ.
I would try to track down the true source (hopefully you get some ideas from above). But if all else fails, first turn on the System.Transactions logging to see if it is truly a timeout. If it is, then use this section in the app.config
I had this happen to me in a handler that took over a minute to complete,
I ended up increasing the timeout by adding this to the app.config
<system.transactions>
<defaultSettings timeout="00:05:00"/>
</system.transactions>
As per this:
https://erictummers.wordpress.com/2014/05/28/cannot-enlist-the-transaction/
Additionally, I changed machine config as per this post:
http://blogs.msdn.com/b/ajit/archive/2008/06/18/override-the-system-transactions-default-timeout-of-10-minutes-in-the-code.aspx
I also wrote a quick app that edits the machine.configs:
https://github.com/hfortegcmlp/MachineConfigEditor
Let's say I try to send to an authenticated transactional queue,
by calling msg.send(object,MessageQueueTransactionType.Single), message does not receive in transactional queue, no exception thrown.
What I want to accomplish is after sending, if message fail to send, perform some function and abort transaction, yet it doesn't throw exception, so I am unable to process it.
I am sending object from Web Application in local to local message queue.
My code is as follows in my web application:
MessageQueueTransaction mqTran=new MessageQueueTransaction();
try
{
using(System.Messaging.Message msg=new System.Messaging.Message(){
mqTran.Begin();
MessageQueue adminQ = new MessageQueue(AdminQueuePath);
MessageQueue msgQ = new MessageQueue(queuePath);
msgQ.DefaultPropertiesToSend.Recoverable = true;
msg.body = object;
msg.Recoverable=true;
msg.Label="Object";
msg.TimeToReachQueue=new TimeSpan(0,0,30);
msg.AcknowledgeType=AcknowledgeTypes.FullReachQueue;
msg.ResponseQueue=adminQ;
msg.AdministrationQueue=adminQ;
msgQ1.Send(msg,MessageQueueTransactionType.Single);
mqTran.Commit();
}
catch(Exception e)
{
mqTran.Abort();
//Do some processing if fail to send
}
It's not going to throw an exception for failure to deliver, only for failure to place on the queue. One of the points of message queueing is that the messages are durable so that you can take appropriate measures if delivery fails. This means you need to program another process to read the dead letter queue. The image below is taken from MSDN.
Because the entire process is asynchronous, your code flow is not going to be exception-driven the way your code block would like. Your transaction is simply the "sending transaction" in this workflow.
Recommendation: Check your message queue to find the messages, either in the outgoing queue or the transactional dead-letter queue.
Occasionally when NServiceBus picks up a message, it fails with the exception below. This causes NServiceBus to retry (up to it's configured retry limit). Sometimes one of the retries result in the message being handled successfully, but it's common for all retries to fail with the same exception. In this case, the message is routed to the error queue as expected when all retries fail.
My question is... what would cause this exception in the first place? It doesn't appear related to my message handler code, as my code doesn't appear in the stack trace.
NServiceBus version: 2.6.0.1504
OS: Windows Server 2003
Handler code is targeting .NET 3.5 or earlier
Here is the full exception message and stack trace:
NServiceBus.Unicast.Transport.Msmq.MsmqTransport [(null)] - Failed raising 'finished message processing' event. System.Messaging.MessageQueueException: Cannot enlist the transaction.
at System.Messaging.MessageQueue.SendInternal(Object obj, MessageQueueTransaction internalTransaction, MessageQueueTransactionType transactionType)
at System.Messaging.MessageQueue.Send(Object obj, MessageQueueTransactionType transactionType)
at NServiceBus.Unicast.Transport.Msmq.MsmqTransport.SendMsmqMessage(Message m, String destination)
at NServiceBus.Unicast.Transport.Msmq.MsmqTransport.Send(TransportMessage m, String destination)
at NServiceBus.Unicast.UnicastBus.SendReadyMessage(Boolean startup)
at NServiceBus.Unicast.UnicastBus.TransportFinishedMessageProcessing(Object sender, EventArgs e)
at NServiceBus.Unicast.Transport.Msmq.MsmqTransport.OnFinishedMessageProcessing()
I see the method NServiceBus.Unicast.UnicastBus.SendReadyMessage(Boolean startup) in your stack trace.
This tells me that your endpoint is connected to a Distributor. After all your message handlers have completed, an endpoint connected to a distributor will send the ReadyMessage back to the Distributor's control queue to say "I'm done with that work. Please send more!"
This is supposed to happen within the same transaction as the rest of your work, but apparently the MSMQ send is having trouble enlisting in that transaction. This could point to a problem with DTC. It would be interesting to know what else you have going on in your message handlers for that message type that also enlist in the transaction. You aren't manually committing or rolling anything back are you?