Multiple Data length receive from Socket streaming Server - PYTHON3 - sockets

Question

I'm intermittent to python, I'm using socket module to receive market data and Order data from Broker Application via API. I'm still confusing how to code for receiving the multiple data length and header, based on the header info, I will treat the data.
How to receive multiple Data length and decide to unpack with correct struct format function?
while True:
try:
"""
continously receive data from Server API (stock Market data streaming)
"""
brecvd = self.sock.recv(1024)
self.brecvdsize = len(brecvd)
# Unpack the header for correct struct formate to unpack
unpkr = self.struct.Struct('<lh')
recvd =self.struct.Struct.unpack_from(unpkr, brecvd)
Marketdepth = recvd[0] == 284 and recvd[1] == 26
Indices = recvd[0] == 228 and recvd[1] == 27
Feed = recvd[0] == 384 and recvd[1] == 22
BidOffer = recvd[0] == 145 and recvd[1] == 28
Msg = recvd[0] == 360 and recvd[1] == 99
#Msg to be checked for 260 or 360
if Marketdepth:
self.Marketdepthresponse(brecvd)
Marketdepth = False
elif Indices:
self.Indicesresponse(brecvd)
Indices = False
elif Feed:
self.feedresponse(brecvd)
Feed = False
elif BidOffer:
self.Bidoffer(brecvd)
BidOffer = False
elif Msg:
self.GeneralMsg(brecvd)
Msg = False
else:
Marketdepth = False
Indices = False
Feed = False
BidOffer = False
Msg = False
pass
except Exception as e:
self.errorcount += 1
print('***Run Loop Receive Issue: {0}'.format(str(e)))

Answer 1

When you call self.sock.recv(1024) you could recieve anywhere from 0 (socket was closed) to 1024 bytes of data. You may not receive enough data to unpack a complete message, may have to continue receiving and buffering the data until you have a complete message to unpack, and may receive part of the next message as well. TCP is a streaming protocol with no message boundary indicators. A protocol using TCP has to define what a complete message looks like.
Without more details of the protocol, I can only make up a protocol and give an example. This is completely untested but should give you an idea.
Protocol: 4 byte length (big-endian) followed by data bytes.
def get_message(self):
# read the 4-byte length
data = self.get_bytes(4)
if not data:
return None # indicates socket closed
if len(data) < 4:
raise IncompleteMessageError
# process the length and fetch the data
length = struct.unpack('>L',data)
data = self.get_bytes(length)
if len(data) < length:
raise IncompleteMessageError
return data
def get_bytes(self,count):
# buffer data until the requested size is present
while len(self.buffer) < count:
new_data = self.sock.recv(1024)
if not new_data: # socket closed
remaining,self.buffer = self.buffer,b''
return remaining # return whatever is left.
self.buffer += new_data
# split off the requested data from the front of the buffer
data,self.buffer = self.buffer[:count],self.buffer[count:]
return data