I have a socket.io server and during testing I captured all live traffic with wireshark. The server switched protocol to WebSocket and I have been getting this sort of packets (this is at the websocket layer):
WebSocket
1... .... = Fin: True
.100 .... = Reserved: 0x4
.... 0001 = Opcode: Text (1)
1... .... = Mask: True
.000 1000 = Payload length: 8
Masking-Key: 435cfdf7
Masked payload
Payload
Line-based text data
2*(\312OJ\005\000
I understand that the websocket data from client to server is masked, but what exactly does the line based text data mean and how can I decode it back into readable plaintext? The traffic isn't encrypted so I don't really understand how it works.
Related
What will happen if I will establish a connection between a client and a server, and configure a different buffer size for each of them.
This is my client's code:
import socket,sys
TCP_IP = sys.argv[1]
TCP_PORT = int(sys.argv[2])
BUFFER_SIZE = 1024
MESSAGE = "World! Hello, World!"
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.connect((TCP_IP, TCP_PORT))
s.send(MESSAGE)
data = s.recv(BUFFER_SIZE)
s.close()
print "received data:", data
Server's code:
import socket,sys
TCP_IP = '0.0.0.0'
TCP_PORT = int(sys.argv[1])
BUFFER_SIZE = 5
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.bind((TCP_IP, TCP_PORT))
s.listen(1)
while True:
conn, addr = s.accept()
print 'New connection from:', addr
while True:
data = conn.recv(BUFFER_SIZE)
if not data: break
print "received:", data
conn.send(data.upper())
conn.close()
That means I will be limited to only 5 bytes? Which means I won't be able to receive the full packet and will lose 1024-5 packets?
I or does it mean I am able to get only packets of 5 bytes, which means that instead of receiving one packets of 1024 bytes as the client sent it, I'll have to divide 1024 by 5 and get 204.8 packets (?) which sounds not possible.
What in general is happing in that code?
Thanks.
Your arguments are based on the assumption that a single send should match a single recv. But this is not the case. TCP is a byte stream and not a message based protocol. This means all what matters are the transferred bytes. And for this is does not matter if it does not matter if one or 10 recv are needed to read 50 bytes.
Apart from that send is not guaranteed to send the full buffer either. It might only send parts of the buffer, i.e. the sender need actually check the return code to find out how much of the given buffer was actually send now and how much need to be retried for sending later.
And note that the underlying "packet" is again a different thing. If there is a send for 2000 bytes it will usually need multiple packets to be send (depending on the maximum transfer unit of the underlying data link layer). But this does not mean that one also need multiple recv. If all the 2000 bytes are already transferred to the OS level receive buffer at the recipient then they can be also be read at once, even if they traveled in multiple packets.
Your socket won't lose the remaining 1024 - 5 (1019) bytes.it just stored on the socket and ready to read again! so , all you need to do is to read from the socket again. the size of buffer you want to read to is decided by yourself. and you are not limited to 5 bytes, you are just limiting the read buffer for each single read to 5 bytes. so for 1024 bytes to read you have to read for 204 times plus another time read which would be the last one. but remember that the last time read fills your last buffer index with null. and that means there is no more bytes available for now.
Im using flutter socket.io to connect to our server. The server doesnt use http protocol etc, straight connect, read and write, so I cant use WebSockets. I have the following:
Socket socket;
Uint8List BTBuffer = new Uint8List(10); //a buffer to hold data
Socket.connect("192.168.68.120", 21000);
.then((Socket sock) {
socket = sock;
socket.listen(dataHandler); //read data from server (data handler reads the data from server)
BTBuffer[0]=1;BTBuffer[1]=2;
socket.write(BTBuffer);
socket.destroy();
});
The above works, but it sends the values as strings and not decimal values. How can I send the values 1 and 2 for example in decimal? I was assuming there would be a write command pointing to a buffer and number of bytes to send.
Many Thanks
Scott
I have a trouble to tune TCP client-server communication.
My current project has a client, running on PC (C#) and a server,
running on embedded Linux 4.1.22-ltsi.
Them use UDP communication to exchanging data.
The client and server work in blocking mode and
send short messages one to 2nd
(16, 60, 200 bytes etc.) that include either command or set of parameters.
The messages do note include any header with message length because
UDP is message oriented protocol. Its recvfrom() API returns number of received bytes.
For my server's program structure is important to get and process entire alone message.
The problem is raised when I try to implement TCP communication type instead of UDP.
The server's receive buffer (recv() TCP API) is 2048 bytes:
#define UDP_RX_BUF_SIZE 2048
numbytes = recv(fd_connect, rx_buffer, UDP_RX_BUF_SIZE, MSG_WAITALL/*BLOCKING_MODE*/);
So, the recv() API returns from waiting when rx_buffer is full, i.e after it receives
2048 bytes. It breaks all program approach. In other words, when client send 16 bytes command
to server and waits an answer from it, server's recv() keeps the message
"in stomach", until it will receive 2048 bytes.
I tried to fix it as below, without success:
On client side (C#) I set the socket parameter theSocket.NoDelay.
When I checked this on the sniffer I saw that client sends messages "as I want",
with requested length.
On server side I set TCP_NODELAY socket option to 1
int optval= 1;
setsockopt(fd,IPPROTO_TCP, TCP_NODELAY, &optval, sizeof(optval);
On server side (Linux) I checked socket options SO_SNDLOWAT/SO_RCVLOWAT and they are 1 byte each one.
Please see the attached sniffer's log picture. 10.0.0.10 is a client. 10.0.0.106 is a server. It is seen, that client activates PSH flag (push), informing the server side to move the incoming data to application immediately and do not fill a buffer.
Additional question: what is SSH encrypted packets that runs between the sides. I suppose that it is my Eclipse debugger on PC (running server application through the same Ethernet connection) sends them. Am I right?
So, my problem is how to cause `recv() API to return each short message (16, 60, 200 bytes etc.) instead of accumulating them until receiving buffer fills.
TCP is connection oriented and it also maintains the order in which packets are sent and received.
Having said that, in TCP client, you will receive the stream of bytes and not the individual udp message as in UDP. So you will need to send the packet length and marker as the initial bytes.
So client can first find the packet length and then read data till packet length is reached and then expect new packet length.
You can also check for library like netty, zmq to do this extra work
My client side cannot recv the two messages if the sender sends too quickly.
sender.py
sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
sock.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
sock.bind(('', int(port)))
sock.listen(1)
conn, addr = sock.accept()
#conn.setsockopt(socket.IPPROTO_TCP, socket.TCP_NODELAY, 1)
# sends message 1 and message 2
conn.send(pickle.dumps(message1))
#time.sleep(1)
conn.send(pickle.dumps(message2))
Where both message 1 and message 2 are pickled objects.
client.py
sock = socket.socket(socket.AF_INET,socket.SOCK_STREAM)
sock.connect((ip,int(port)))
message1 = pickle.loads(sock.recv(1024))
print(message1)
message2 = pickle.loads(sock.recv(1024))
When i run this code as it is, i am able to print out message1 but i am unable to receive message2 from the sender. The socket blocks at message2.
Also, if i uncomment time.sleep(1) in my sender side code, i am able to receive both messages just fine. Not sure what the problem is. I tried to flush my TCP buffer everytime by setting TCP_NODELAY but that didnt work. Not sure what is actually happening ? How would i ensure that i receive the two messages
Your code assumes that each send on the server side will match a recv on the client side. But, TCP is byte stream and not a message based protocol. This means that it is likely that your first recv will already contain data from the second send which might be simply discarded by pickle.loads as junk after the pickled data. The second recv will only receive the remaining data (or just block since all data where already received) so pickle.loads will fail.
The common way to deal with this situation is to construct a message protocol on top of the TCP byte stream. This can for example be done by prefixing each message with a fixed-length size (for example as 4 byte uint using struct.pack('L',...)) when sending and for reading first read the fixed-length size value and then read the message with the given size.
I'm getting my feet wet with unix system programming, and am currently trying to implement a minimal http server with sockets.
Currently I have the following code (it's ocaml, the flow is the same as in any language)
let establish_server () =
let socket = Unix.socket Unix.PF_INET Unix.SOCK_STREAM 0 in
Unix.bind socket (Unix.ADDR_INET ((Unix.inet_addr_of_string "127.0.0.1"), 8888));
Unix.listen socket 128;
print_endline "accepting now";
while true do
let (service_socket, addr) = Unix.accept socket in
print_endline "Something arrived!";
let b = Bytes.create 1024 in
let _ = Unix.recv service_socket b 0 1024 [Unix.MSG_PEEK] in
print_endline ## Bytes.to_string b;
let response = Bytes.of_string "HTTP/1.1 200 OK\n\nHey from Ocaml!" in
let _ = Unix.send service_socket response 0 (Bytes.length response) [Unix.MSG_PEEK] in
Unix.close service_socket;
done
Why do I have to close the socket to receive something in the browser when I visit 127.0.0.1:8888? If I don't close the socket on the last line, my browser keeps on spinning forever.
Can't you just send something to the other end of the socket? Why doesn't the browser show anything, if I don't close the socket?
There are (at least) three big things you need to do to be handling HTTP/1.1 properly:
1) You need to buffer input, because the browser's request might not arrive in a single packet. This means that the Unix.recv call may receive a partial request, and the next call will retrieve the rest of the request (or the next chunk). The simplest approach is to wrap the stream in some sort of reader that lets you request an entire line synchronously.
2) When sending a response, you have to supply a Content-Length header. Without this, the browser doesn't know when the data is finished -- and without it, the browser can't tell the difference between the response being completed and the connection being unexpected closed.
3) In HTTP 1.1, connections are assumed to be persistent unless the server explicitly tells the client that they aren't. Your server will perform better if you support Connection: keep-alive, but for a minimalist implementation, you should at least send a Connection: close response header.