If I connect after some messages have already come from the connection, I receive a log like this:
20220707-14:35:33.000000000 : ResendRequest for messages FROM: 969 TO: 1000 has been satisfied.
20220707-14:35:33.000000000 : Processing QUEUED message: 1001
20220707-14:35:33.000000000 : MsgSeqNum too low, expecting 1002 but received 1001
I want to receive all the messages in the queue and then proceed as normal but the connection keeps saying MsgSeqNum too low and then lets the MsgSeqNum increment and ask for resends and then keeps repeating this process.
Thanks!
PS: I can't use ResetOnLogon = Y
Related
As per LINGER_MS_DOC in ProducerConfig java class:
"The producer groups together any records that arrive in between
request transmissions into a single batched request. Normally this
occurs only under load when records arrive faster than they can be
sent out. However in some circumstances the client may want to reduce
the number of requests even under moderate load. This setting
accomplishes this by adding a small amount of artificial delay; that
is, rather than immediately sending out a record the producer will
wait for up to the given delay to allow other records to be sent so
that the sends can be batched together. This can be thought of as
analogous to Nagle's algorithm in TCP. This setting gives the upper
bound on the delay for batching: once we get "BATCH_SIZE_CONFIG" worth
of records for a partition it will be sent immediately regardless of
this setting, however if we have fewer than this many bytes
accumulated for this partition we will 'linger' for the specified time
waiting for more records to show up. This setting defaults to 0 (i.e.
no delay). Setting "LINGER_MS_CONFIG=5" for example, would have the
effect of reducing the number of requests sent but would add up to 5ms
of latency to records sent in the absence of load."
I searched for a suggested value for linger.ms but nowhere found a higher value suggested for this. Most of the places 5 ms is mentioned for linger.ms.
For testing, I have set "batch.size" to 16384 (16 KB)
and "linger.ms" to 60000 (60 seconds)
as per doc I felt if I send a message of size > 16384 bytes then the producer will not wait and send the message immediately, but I am not observing the same behavior.
I am sending events of size > 16384 bytes but it still waits for 60 seconds. Am I missing to understand the purpose of "batch.size"? My understanding of "batch.size" and "linger.ms" is that whichever meets first the messages/batch will be sent.
In this case, if it is going to be the minimum wait time and do not give preference to "batch.size" then I guess setting a high value for linger.ms is not right.
Here is the kafka properties used in yaml:
producer:
properties:
acks: all
retries: 5
batch:
size: 16384
linger:
ms: 10
max:
request:
size: 1046528
I have a quickfixj initiator connecting to vendor's acceptor and receiving messages. I keep the fix messages in a buffer which is processed by a thread. To avoid loosing the message in case crash with message in the buffer, I have the last seqnum processed, and plan to send resend message for that next seqnum on my side when I reconnect.
I know the better solution would be that I save the messages before I receives them, but the design is to avoid doing any db access in the onMessage call.
I didn't find any example how this could be done, resending request for a specific seqnum. Should I simply overload the logon message and send the seqnum?
Anyone has an example?
I guess you are already in synch as per the last thread if quickfixj crash in onmessage, will I lose my current message?.
QuickFixJ manages 2 sequence numbers:
SenderSequenceNum: Sequence number used in sending messages.
TargetSequenceNum: Sequence number expected to receive.
So you have two options:
Option 1: Process the receive messages on the QuickfixJ onMessage() callback thread. So that in case of an exception the sequence number does not increment. And QuickFixJ automatically sends the resend request on receiving next fix message as it will detect the sequence gap.
Option 2: Persist the sequence number that you have successfully processed. In case of crash, on restart you can set the expected receive sequence number using:
Session.lookupSession(session_).setNextTargetMsgSeqNum();
So if you receive a sequence number higher than that, QuickfixJ automatically sends resend the request.
Note: Do not change the sender sequence number then another party will receive a sequence number lower than expected and can cause disconnection.
I am trying to implement Fix.4.2 protocol, but It is difficult to understand the message log I attached below. Here Logon(35=A) request was sent with MsgSeqNum(34=1) from client. Then for testing ResendRequest and SequenceReset session level messages I sent a NewOrderSingle request with MsgSeqNum=7 (instead of MsgSeqNum=2, as subsequent messages should have incremeted msgseqnum after logon request). As expected MsgSeqNum is too high than recieved one Fiximulator responded with a ResendRequest(35=2) to send from 2 to 0 (i.e., from 2 to 7). Here why the Fiximulator is not waiting for client's reply ? instead it is sending an heartbeat message. Why the client is sending ResendRequest in response to ResendRequest of Fiximulator instead of sending SequenceReset message ?.
Also explain remaining cases if possible.
Thanks in advance.
What is your status of ResetOnLogon in your config file for the acceptor ? Default value is N so it isn't being reset. Always check your config file or try debugging to figure out issues.
ResetOnLogon Determines if sequence numbers should be reset when recieving a logon request. Acceptors only
I have send Udp packet to the Server. If the server is OK then I can receive the response packet nicely but when the server is down then I did not get any response packet. Anybody can help me that how can I send my packet to server multiple time when fail to receive the response packet. Moreover, want to keep alive the connection with server. Thanks in advance.
Well,
After you've sent the packet, you wait for the ACK (response) package from the server. You could use the DatagramSocket.setSoTimeout() to an appropriate time, if you get the Timeout Exception increment a counter, if that counter is less than 2/3 send the packet again and repeat these steps. If the counter is bigger than 2/3 the server is down, just quit.
According to Java documentation, receive will block until a package is received or a timeout has expired.
To keep the connection alive you need to implement a ping-pong. In another thread of your program, you send a Keep-Alive packet (any small packet will do) and wait for a response. I suggest using a different port number for this purpose so that these packets won't mess up with the normal data packets. These packets can be send every 2 seconds o 2 minutes depends on your particular needs. When the thread receives the ACK packet it will update a private time variable with the current time, for example:
lastTimeSeen = System.currentTimeMillis();
put a method in your thread class to access the value of that variable.
I was wondering how exactly does TCP implement in-order delivery.
lets say this is list of events
packet1 sent , ack received.
packet2 sent , ack not-received.
packet3 sent.
packet4 sent.
ack4 received.
ack3 received.
ack2 received.
Can you describe to me what exactly happens sequentially?
The short answer is that each packet contains offset information (disguised as sequence number), specifying where in the stream its payload lies.
Let's say the following occurred: packet 1 is received, packet 2 is not received, and packet 3 and 4 are received. At this point receiving TCP stack knows where to copy contents of packets 3 and 4 on the buffer, and it knows that it still hasn't received prior data, so it will make packet 1 data available for reading, but it won't make packet 3 or 4 data available until packet 2 is received.
Transmitting TCP stack generally does not wait for acknowledgements for any single packet before sending out the next one, but if it does not receive an acknowledgement for a given packet (and ACKs can and are bundled together in a single packet for efficiency), it will retransmit it until an ACK is received.
Exact sequence of events depends on network conditions, TCP stack implementation, chosen TCP policy, socket options and other factors.
TCP packets have sequence numbers (byte offsets since the start, from memory) and the ACK messages acknowledge that a specific offset has been received:
So you could end up with a situation like:
data 1 (10 bytes) ->
<- ack (10, data1)
data 2 (15 bytes) ->
data 3 (10 bytes) ->
data 4 ( 8 bytes) ->
<- ack (25, data1/2/3)
<- ack (33, data1/2/3/4)
In other words, the sender can continue to send regardless of acknowledgments up to the point where its buffers are full (it has to keep unacknowledged packets in case it needs to re-transmit them).
This "disconnect" between sends and acknowledgments can greatly speed up the data flow.
At the receiving end, the packets can arrive out of order, and they're kept until something can be delivered to the higher levels in order.
For example, if data 3 arrived before data 2, the receiving end would hold on to it until data 2 arrived, then both of them would be sent upwards for delivery.