Consider the following sequence diagram which depicts the communication between a FIX initiator and the acceptor. Please note I am refering to FIX.4.4 here.
As you can see the message with sequence number 2 gets lost in transit. The initiator sends another message (with seq. number 3), the acceptor detects that gap and issues a resend request, asking again for message with seq. number 2 and everything that might have followed (7=2|16=0).
A couple of questions that I couldn't get answered by digging through the spec:
What happens if the "Resend Request" gets lost in transit?
What happens if the initiator is not able to resend the requested messages?
What happens if the "Resend Request" gets lost in transit?
The gap will be detected on the subsequent message just as you lined out.
However, the ResendRequest will not actually be resent because the only session-level message that must be resent is the Reject message.
Instead, either a SequenceReset with 123/GapFillFlag=Y (description) will be sent or the message will be skipped with a SequenceReset message with 36/NewSeqNo set to a higher sequence number, effectively skipping the message that will not be resent.
This is stated in the FIX spec in chapter "message recovery" (link)
What happens if the initiator is not able to resend the requested
messages?
As stated above it should either send a GapFill instead, or skip to a higher sequence number.
Related
Imagine I have a message queue on to which I place messages saying that I want to send a user an email.
If I’m using a message broker that provides “at least once” delivery guarantees, then from all the resources I’ve been reading, they say “you must make sure your processing is idempotent”.
However, in the case of a side effect like sending an email, I don’t see how this is possible.
I have two possible choices:
Store the message ID in my database, then send the email.
Send the email, then store the message ID in the database.
When a message is received, I would then check the database to see if the message ID exists. If it does, I would skip the message as a duplicate.
However, the first case leaves me with “at most once” semantics on sending my email (if sending the email fails, it will be skipped next time the message is seen), and the second case gives me “at least once” semantics (if storing the ID in the database fails, I’ll end up sending multiple emails).
Some things I have read say “you need an email API that supports idempotency”, but as far as I can tell that just pushes the problem on to their servers - they still have the same dilemma.
Am I missing something here? Or is it just not possible to have idempotent message processing when that processing has external side effects?
I am going to claim that email inherently does not support exactly-once semantics. When SMTP protocol is used, your side can always crash in between the far side confirms and you persist the confirmation. Your best bet is to use at-least once semantics.
But you may get exactly-once UX experience. If every time you send/resend a message you use the same Message ID, then the client side (mail client) may de duplicate those messages. See https://en.wikipedia.org/wiki/Message-ID for details.
I have an [Azure Storage] queue where I put e-mail messages to be sent. Then, there is a separate service which monitors that queue and send e-mails using some service. In this particular case I'm using SendGrid.
So, theoretically, if the sender crashes right after a successful call to SendGrid Mail Send API (https://sendgrid.com/docs/API_Reference/Web_API_v3/Mail/index.html), the message will be returned to the queue and retried later. This may result in the same e-mail being delivered more than once, which could be really annoying for some type of e-mail.
The normal way to avoid this situation would be to provide some sort of idempotency key to Send API. Then the side being called can make sure the operation is performed at most once.
After careful reading of SendGrid documentation and googling, I could not find any way to achieve what I'm looking for here (at most once semantic). Any ideas?
Without support for an idempotency key in the API itself your options are limited I think.
You could modify your email sending service to dequeue and commit before calling the Send API. That way if the service fails to send the message will not be retried as it has already been removed from the queue, it will be sent at most once.
Additionally, you could implement some limited retries on particular http responses (e.g. 429 & 5xx) from SendGrid where you are certain the message was not sent and retrying might be useful - this would maintain "at most once" whilst lowering the failure rate. Probably this should include some backoff time between each attempt.
Here, at the end of this page. last paragraph ,
They mentioned some problems that occurs in This protocol.
i am unable to understand what are these problems. ?
for example. He told. "If a request processing long time"
I am unable to understand this statement. Where is the request which processing taking long time, on client ? or on server ?
Or i am unable to understand where is the Clock(time) ? is it on Client side or Server Side? because here mentioned in the end of 2 point. "if the reply is not received within the time period , the kernel of the client machine re-transmits the request message."
Consider this:
The client sends a message. If it doesn't get a reply from the server within - say - 1 minute it will transmit the message again.
When the server receives a message, it only sends a reply after having generated a full response to the message that the client sent.
No suppose you, as client, send a message to the server. The server receives your message, and starts processing it. At this time, you, the client, have no idea of whether the server got the message or not. Assume you send a complicated task to the server, which takes it 1 minute and 5 seconds to complete. After 1 minute (ignoring transmission times), the server is still busy doing your work, but you as the client don't know of any of this and send your message again.
Now, depending on the actual protocol implementation, there are a few potential issues:
It's possible that by sending the message again, you increase some sequence count and are therefore unable to receive the reply to the original message afterwards.
It's possible that the server isn't able to determine whether a message that arrives is the first message or a message that had to be send again. So it could be doing work that it already did, leading either to needless processing or in the worst case to (business) logic errors.
Additionally, by sending both the message and the reply possibly needless more than once, you increase the amount of total data transmitted, without gaining anything from it.
To "solve" this, you could increase the waiting time before the client sends its message again. This will "fix" the issue with long running tasks on the server, but will also hurt in case the message actually got lost on the way, because you're waiting longer to even send a new message.
The "real" solution here is to have the server acknowledge as soon as it receives a message from the client, just as saying "i got your message, i'll send the reply soon!" before even starting to actually process the message.
I am using org.hornetq.api.core.client
how can I guarantee the message that I am sending actually reached the queue (not the client, just the queue) ?
producer.send("validQueue",clientMessage)
please note that the queue is a valid queue .
this similar question is referring to invalid queue. other ones such as this one is relevant to the delivery to the client .
It really depends on how you are sending.
First question of yours was about
First of all, on JMS you have to way to send to an invalid queue since the producer will validate the queue's existence. On HornetQ core api you send to an address (not to a queue), and you may have an unbound queue. So you have to query if the address has queues or not.
Now, for confirmation the message was received:
Scenario I, persistent messages, non transactionally
Every message is sent blocked. The client will unblock as soon as the server acknowledged receiving the message. This is done automatically.. you don't have to do anything.
Scenario II, non persistent messages, non transactionally
There are no confirmations by default. The message is sent asynchronously. We assume the message is transient and it's not a big deal if you lost it. you can change that by setting block-on-non-persistent-send on the ServerLocator.
Scenario III, transactionally (either persistent or not).
As soon as you call commit the message is on the queues.
Scenario IV, Confirmation send
You set a callback and you get a method call as soon as the server acked it on the queues. Look on the manual for confirmation callback. There's also the same feature on JMS2.
If a quick fix session is created by server(acceptor) at say 9AM, but the StartTime is at 11AM. This means the session exists but not active.
If the server receives an unsolicited message from an exchange that it needs to send on this session, will it persist this if I have configuration PersistMessages=Y and sends it to the client(initiator) when it connects after 11AM?
No, it would not persist messages received before start time and would send you a reject message. The message will be rejected at the interface itself, message isn't handled. You would have to resend it to get a response.
QuickFIX does persist (but not send) messages before a session is connected. The sequence numbers are updated and when the session is connected and the first message is sent, the counterparty FIX engine will see the gap in the sequence numbers and request a resend. QuickFIX will then resend the persisted messages. However, depending on your QuickFIX configuration, the outgoing messages might be considered to be too old and rejected locally.
As I understand, these are kept to take into account timings under which corresponding exchange would accept the orders.
Application or its sub-modules do not need to maintain timings and take some action on closing the fix session. Rather, QuickFix shall automatically deactivate the session.
Persistence of the message or re-sesnding when the session becomes active does not look desirable to me.
You can rather maintain some kind of queue to buffer such messages in sending application, and send them only when the time matches with active session timings.
That's my thoughts, hope that helps.