Look at this small basic python programs:
import socket
tcpsock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
tcpsock.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
tcpsock.bind(("", 10000))
tcpsock.listen(10)
(sock, (ip, port)) = tcpsock.accept()
s = sock.recv(1024)
print(s)
Second program:
import socket
import time
sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
sock.connect(('localhost', 10000))
time.sleep(1)
sock.sendall(b'hello world')
The first program is a socket server. It recv a message through the socket and display it on the console. The second program is a client which connects to the server and sends it a message.
As you can see, the server reads a 1024 bytes max length message. My client send a few bytes.
My question is: How does the server knows the message ends after the 'd' char ?
I am working with sockets since years and i have always implemented a delimiter mechanism in order to know when the message stops.
But it seems to work automaticly. My question is: How ?
I know TCP car fragment messages. So what's happen if the paquet is trucated in the middle of my message ? Is it managed by OS ?
Thanks
How does the server knows the message ends after the 'd' char ?
It does not. There is not even a concept of a message in TCP. recv simply returns what is there: it blocks if no data are available and returns what can be read up to the given size if data are available. "Data available" means that there are data in the sockets receive buffer, which are put by the OS kernel there. In other words: recv will not block until the requested number of bytes can be returned but it will already return when at least a single byte is in the sockets receive buffer.
For example if the client would do two send or sendall shortly after each other a single recv might return both "messages" together. This can be easily triggered by deferring the recv (add some sleep before it) so that both "messages" are guaranteed to be arrived at the client.
Related
I'm really new to Ada and would like to create a TCP socket between Ada and Python. The Ada programm should act as server and the Python program as client. The main use case is to receive string commands from Python and confirm their execution.
Without the Ada.Streams.Write(Channel.All, Data); in the Ada Server and the data = s.recv(512) it is at least possible to receive a Hello World from the Python Client.
I would like to send an answer from the Ada server to the Python client, that's the point where I stuck. I get an "Socket Error Connection Timed Out".
Ada Server:
use GNAT.Sockets;
Server : Socket_Type;
Socket : Socket_Type;
Address : Sock_Addr_Type;
Channel : Stream_Access;
Data : Stream_Element_Array(1 .. 512);
Last : Stream_Element_Offset;
S : Unbounded_String;
begin
Put_Line("Server Config Started..");
Create_Socket(Server);
Set_Socket_Option(Server,
Socket_Level,
(Reuse_Address, True));
Set_Socket_Option(Server, Socket_Level,(Receive_Timeout, Timeout => 5.0));
Bind_Socket(Server, Address => (Family => Family_Inet, Addr => Inet_Addr("127.0.0.2"), Port => 65432));
Listen_Socket(Server);
Accept_Socket(Server, Socket, Address);
Put_Line("Client connected from:" & Image(Address));
Channel := Stream(Socket);
Ada.Streams.Read(Channel.All, Data, Last);
Put_Line("Received:");
for I in 1 .. Last loop
Put(Character'Val(Data(I)));
end loop;
Ada.Streams.Write(Channel.All, Data);
Python Client:
HOST = '127.0.0.2'
PORT = 65432
with socket.socket(socket.AF_INET, socket.SOCK_STREAM) as s:
s.connect((HOST, PORT))
s.sendall(b'Hello, world')
data = s.recv(512)
What am I doing wrong? Has someone a hint?
Thanks in advance.
The reason things don’t go the way you hoped is that you’ve put the Receive_Timeout socket option on the Server socket; you need to put it on Socket instead, since that’s the socket you’re receiving from.
As things stand: the call
Ada.Streams.Read(Channel.All, Data, Last);
will terminate either when it has read Data’Length (512) bytes or when the other end closes the socket.
Before you included the data = s.recv(512) at the end of the Python script, the Python was closing the socket.
Now, it’s waiting for the Ada side to send it 512 bytes back - and the Ada side is waiting for the Python side to send it the remaining (512 - 12) bytes. Classic deadly embrace.
How to fix? possibly, have the Ada side read character-by-character until it reads a terminator (e.g. \0). Or you could use datagrams. In any case, you need a protocol to determine message boundaries on the wire.
On macOS, the Ada side prints out the message it has received as soon as the Python side sends it. This is not what the man page for setsockopt() says re: SO_RCVTIMEO! It should wait for the timeout set (5 seconds) before deciding to give up.
What will happen if I will establish a connection between a client and a server, and configure a different buffer size for each of them.
This is my client's code:
import socket,sys
TCP_IP = sys.argv[1]
TCP_PORT = int(sys.argv[2])
BUFFER_SIZE = 1024
MESSAGE = "World! Hello, World!"
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.connect((TCP_IP, TCP_PORT))
s.send(MESSAGE)
data = s.recv(BUFFER_SIZE)
s.close()
print "received data:", data
Server's code:
import socket,sys
TCP_IP = '0.0.0.0'
TCP_PORT = int(sys.argv[1])
BUFFER_SIZE = 5
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.bind((TCP_IP, TCP_PORT))
s.listen(1)
while True:
conn, addr = s.accept()
print 'New connection from:', addr
while True:
data = conn.recv(BUFFER_SIZE)
if not data: break
print "received:", data
conn.send(data.upper())
conn.close()
That means I will be limited to only 5 bytes? Which means I won't be able to receive the full packet and will lose 1024-5 packets?
I or does it mean I am able to get only packets of 5 bytes, which means that instead of receiving one packets of 1024 bytes as the client sent it, I'll have to divide 1024 by 5 and get 204.8 packets (?) which sounds not possible.
What in general is happing in that code?
Thanks.
Your arguments are based on the assumption that a single send should match a single recv. But this is not the case. TCP is a byte stream and not a message based protocol. This means all what matters are the transferred bytes. And for this is does not matter if it does not matter if one or 10 recv are needed to read 50 bytes.
Apart from that send is not guaranteed to send the full buffer either. It might only send parts of the buffer, i.e. the sender need actually check the return code to find out how much of the given buffer was actually send now and how much need to be retried for sending later.
And note that the underlying "packet" is again a different thing. If there is a send for 2000 bytes it will usually need multiple packets to be send (depending on the maximum transfer unit of the underlying data link layer). But this does not mean that one also need multiple recv. If all the 2000 bytes are already transferred to the OS level receive buffer at the recipient then they can be also be read at once, even if they traveled in multiple packets.
Your socket won't lose the remaining 1024 - 5 (1019) bytes.it just stored on the socket and ready to read again! so , all you need to do is to read from the socket again. the size of buffer you want to read to is decided by yourself. and you are not limited to 5 bytes, you are just limiting the read buffer for each single read to 5 bytes. so for 1024 bytes to read you have to read for 204 times plus another time read which would be the last one. but remember that the last time read fills your last buffer index with null. and that means there is no more bytes available for now.
My client side cannot recv the two messages if the sender sends too quickly.
sender.py
sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
sock.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
sock.bind(('', int(port)))
sock.listen(1)
conn, addr = sock.accept()
#conn.setsockopt(socket.IPPROTO_TCP, socket.TCP_NODELAY, 1)
# sends message 1 and message 2
conn.send(pickle.dumps(message1))
#time.sleep(1)
conn.send(pickle.dumps(message2))
Where both message 1 and message 2 are pickled objects.
client.py
sock = socket.socket(socket.AF_INET,socket.SOCK_STREAM)
sock.connect((ip,int(port)))
message1 = pickle.loads(sock.recv(1024))
print(message1)
message2 = pickle.loads(sock.recv(1024))
When i run this code as it is, i am able to print out message1 but i am unable to receive message2 from the sender. The socket blocks at message2.
Also, if i uncomment time.sleep(1) in my sender side code, i am able to receive both messages just fine. Not sure what the problem is. I tried to flush my TCP buffer everytime by setting TCP_NODELAY but that didnt work. Not sure what is actually happening ? How would i ensure that i receive the two messages
Your code assumes that each send on the server side will match a recv on the client side. But, TCP is byte stream and not a message based protocol. This means that it is likely that your first recv will already contain data from the second send which might be simply discarded by pickle.loads as junk after the pickled data. The second recv will only receive the remaining data (or just block since all data where already received) so pickle.loads will fail.
The common way to deal with this situation is to construct a message protocol on top of the TCP byte stream. This can for example be done by prefixing each message with a fixed-length size (for example as 4 byte uint using struct.pack('L',...)) when sending and for reading first read the fixed-length size value and then read the message with the given size.
I've followed the code in this answer to create a pair of programs which send and receive datagrams via a Unix socket.
What's awkward about this: On the side which creates the first socket (i.e. the "server"), I can't use calls to send, recv, read or write because there is no destination set (those calls fail with "Destination address required" error.
I've tried working around this by adding an initial call to recvfrom and using the address given back through there, but it never has the correct value (on OSX at least). It also doesn't work to use sendto since we don't know the client address.
The way which I have got it working is roughly following this process:
Start server program, which:
Calls socket and bind to create the server socket.
It waits here.
Start client program, which:
Calls socket and bind to create the client socket.
It knows the path to the server socket and calls connect.
This side is now set up correctly.
Server program:
Accepts the path to the client socket via stdin
Copies the path to a struct sockaddr_un and uses that to call connect (as in the linked answer).
This is pretty awkward! If I was doing this with SOCK_STREAM sockets, I could use listen and accept; the flow is much more straight-forward without the server needing to know the client's socket path.
Is there a more elegant way of getting these sockets connected?
SOCK_DGRAM (UDP) sockets are "Connectionless", so you cannot "connect" the two sockets. They only send packets to the designated destination address and the client simply captures it. So you'll to first decide if you are going to use a SOCK_DGRAM (UDP) or SOCK_STREAM (TCP).
If you are using UDP sockets the client side socket need not connect, you simply sendto the destination address (Server in this case) after creating and binding.
So if you need a dedicated connected connection you are better off using TCP socket. Or if you are using this over the internet the closest thing you can find for UDP is Hole punching.
One way to solve the problem:
Your messages probably have common header.
Add address information of sender to the header.
Then your server can respond to the correct client by using sendto.
Pseudo example:
void handle_my_message(const my_message_t *msg)
{
struct sockaddr_un client_address = msg->header.sender;
my_message_response_t response_msg;
... handle the message and fill the response...
// Send response message
sendto(fd, &response_msg, sizeof(response_msg), 0,
(struct sockaddr*)&client_address, sizeof(client_address));
}
This way your server programs does not need to keep book of connections.
Instead of struct sockaddr_un in the header you maybe should use something smaller and more portable format, that can be converted to struct sockaddr_un.
You should also bind the client side socket to an address. If the client socket is bound (i.e. has its own name), then you don't need an out-of-band mechanism to communicate the client's address to the server. The OS sends it along with each datagram.
Sample code for client (in python because it's quick and easy to prototype -- should be easy to translate to the equivalent C):
#!/usr/bin/env python3
import os
import socket
server_addr = "/tmp/ux_server"
client_addr = "/tmp/ux_client"
if os.path.exists(client_addr):
os.remove(client_addr)
sock = socket.socket(socket.AF_UNIX, socket.SOCK_DGRAM)
sock.bind(client_addr)
for n in range(5):
data = "Hello " + str(n)
data = data.encode()
print("Sent '{}' to {}".format(data, server_addr))
sock.sendto(data, server_addr)
data, addr = sock.recvfrom(16000)
print("Got '{}' back from {}".format(data, addr))
Furthermore, you can execute a connect on the client side. Since it's a datagram socket, that doesn't actually create a connection between the two but it does fix the address of the server endpoint, relieving you of the need to provide the server address on every send (i.e. you can use simple send rather than sendto).
For completeness, here's the echo server corresponding to the above:
#!/usr/bin/env python3
import os
import socket
server_addr = "/tmp/ux_server"
if os.path.exists(server_addr):
# Bind will fail if endpoint exists
os.remove(server_addr)
sock = socket.socket(socket.AF_UNIX, socket.SOCK_DGRAM)
sock.bind(server_addr)
while True:
data, addr = sock.recvfrom(16000)
print("Got '{}' from {}".format(data, addr))
sock.sendto(data, addr)
EDIT
Hmm... I see now that you say you're already binding the client socket, and then connecting to the server side. But that means you simply need to have the server use recvfrom once initially to obtain the client's address. The OS will send the address along and you don't need to use an out-of-band mechanism.
The downside to connecting the socket is that if the client goes down, the server won't know that unless it attempts to send, but the client won't be able to reconnect because the server's socket is already connected. That's why datagram servers typically use recvfrom and sendto for all messages.
Updated server with initial recvfrom followed by connect:
#!/usr/bin/env python3
import os
import socket
server_addr = "/tmp/ux_server"
if os.path.exists(server_addr):
# Bind will fail if endpoint exists
os.remove(server_addr)
sock = socket.socket(socket.AF_UNIX, socket.SOCK_DGRAM)
sock.bind(server_addr)
client_addr = None
while True:
if client_addr:
data = sock.recv(16000)
else:
data, client_addr = sock.recvfrom(16000)
sock.connect(client_addr)
print("Got '{}' from {}".format(data, client_addr))
sock.send(data)
Updated client with connected socket.
#!/usr/bin/env python3
import os
import socket
server_addr = "/tmp/ux_server"
client_addr = "/tmp/ux_client"
if os.path.exists(client_addr):
os.remove(client_addr)
sock = socket.socket(socket.AF_UNIX, socket.SOCK_DGRAM)
sock.bind(client_addr)
sock.connect(server_addr)
for n in range(5):
data = ("Hello " + str(n)).encode()
print("Sent '{}'".format(data))
sock.send(data)
data = sock.recv(16000)
print("Got '{}' back".format(data))
I am trying to use the sockets package for Octave on my Ubuntu. I am using the Java Sockets API for connecting to Octave. The Java program is the client, Octave is my server. I just tried your code example:
http://pauldreik.blogspot.de/2009/04/octave-sockets-example.html
There are two problems:
1.)
Using SOCK_STREAM, for some strange reason, certain bytes are being received by recv() right after accept(), even if I'm not sending anything from the client. Subsequent messages I send with Java have no effect, it seems the Octave socket completely has its own idea about what it thinks it receives, regardless of what I'm actually sending.
2.)
Using SOCK_DGRAM, there is another problem:
I do get a reception of my actual message this way, but it seems that a recv() command doesn't remove the first element from the datagram queue. Until I send the second datagram to the socket, any subsequent recv() commands will repeatedly read the first datagram as if it were still in the queue. So the recv() function doesn't even block to wait for an actually new available datagram. Instead, it simply reads the same old one again. This is useless, since I cannot tell my server to wait for news from the client.
Is this how UDP is supposed to behave? I thought datagram packets are really removed from the datagram queue by recv().
This is my server side code:
s=socket(AF_INET, SOCK_DGRAM, 0);
bind(s,12345);
[config,count] = recv(s, 10)
[test,count] = recv(s, 4)
And this is my Java client:
public LiveSeparationClient(String host, int port, byte channels, byte sampleSize, int sampleRate, int millisecondsPerFrame) throws UnknownHostException, IOException {
this.port = port;
socket = new DatagramSocket();
this.host = InetAddress.getByName(host);
DatagramPacket packet = new DatagramPacket(ByteBuffer.allocate(10)
.put(new byte[]{channels, sampleSize})
.putInt(sampleRate)
.putInt(millisecondsPerFrame)
.array(), 10, this.host, port
);
socket.send(packet);
samplesPerFrame = (int) Math.floor((double)millisecondsPerFrame / 1000.0 * (double)sampleRate);
}
As you see, I'm sending 10 Bytes and receiving all 10 (this works so far) with recv(s, 10). In the later part of my Java program, packets will be generated and send also, but this may take some seconds. In the mean time, the second receive, recv(s, 4), in Octave should wait for a really new datagram package. But this doesn't happen, is simply reads the first 4 Bytes of the same old package again. recv() doesn't block the second time.
I hope it is not a problem for you to fix this?
Thanks in advance :-)
Marvin
P.S.: Also, I don't undertstand why listen() and accept() are both necessary when using SOCK_STREAM, but not for SOCK_DGRAM.