In a nutshell:
I am trying to make a socket server to which clients connect and send/receive messages (based on the sockserv code in Learn you some erlang tutorial http://learnyousomeerlang.com/buckets-of-sockets)
Server side components:
supervisor - unique, started at the very beginning, spawns processes with gen_server behaviour
gen_server behaviour processes - each one deals with a connection.
Client side:
client which connects to the socket and sends a few bytes of data and then disconnects.
Code details
My code is pretty much the same as in the presented tutorial. The supervisor is identical. The gen_server component is simplified so that it has only one handle_info case which is supposed to catch everything and just print it.
Problem
The connection succeeds, but when the client sends data, the server behaves as though no data is received (I am expecting that handle_info is called when that happens).
handle_info does get called but only when the client disconnects and this event is reported with a message.
My attempts
I have played around with different clients written in Erlang or Java, I have tried setting the active/passive state of the socket. The author of the tutorial sets {active, once} after sending a message. I ended up just setting {active, true} after the AcceptSocket is created as such: (the gen_server proc is initialized with a state which contains the original ListenSocket created by the supervisor)
handle_cast(accept, S = #state{socket=ListenSocket}) ->
{ok, AcceptSocket} = gen_tcp:accept(ListenSocket),
io:format("accepted connection ~n", []),
sockserv_sup:start_socket(), % a new acceptor is born, praise the lord
inet:setopts(AcceptSocket, [{active, true}]),
send(AcceptSocket, "Yellow", []),
{noreply, S#state{socket=AcceptSocket, next=name}}.
send(Socket, Str, Args) ->
ok = gen_tcp:send(Socket, io_lib:format(Str++"~n", Args)),
ok.
handle_info(E, S) ->
io:format("mothereffing unexpected: ~p~n", [E]),
{noreply, S}.
It has aboslutely no effect. handle_info only gets called when the connection is lost because the client disconnects. whenever the client sends data nothing happens.
What could be the problem? I have spend quite some time on this, I really have no idea.
Many thanks.
Have you tried setting the other options in http://www.erlang.org/doc/man/inet.html#setopts-2
inet:setopts(AcceptSocket, [{active, true}])
for example:
{packet, line} to read in a line at a time
and
binary to read in data as a binary.
I also was working through a similar exercise based on that tutorial recently and my options used were:
inet:setopts(LSocket, [{active,true}, {packet, line}, binary, {reuseaddr, true}]),
To conclude, watch out for the options. I was indeed not paying attention to the implications of the set of options. I tried with a more narrowed down situation and worked it out. My problem was the {packet, line} option which implies that \n is considered a message delimiter.
Related
I'm trying to create a server that sets up a Unix socket and listens for clients which send/receive data. I've made a small repository to recreate the problem.
The server runs and it can receive data from the clients that connect, but I can't get the server response to be read from the client without an error on the server.
I have commented out the offending code on the client and server. Uncomment both to recreate the problem.
When the code to respond to the client is uncommented, I get this error on the server:
thread '' panicked at 'called Result::unwrap() on an Err value: Os { code: 11, kind: WouldBlock, message: "Resource temporarily unavailable" }', src/main.rs:77:42
MRE Link
Your code calls set_read_timeout to set the timeout on the socket. Its documentation states that on Unix it results in a WouldBlock error in case of timeout, which is precisely what happens to you.
As to why your client times out, the likely reason is that the server calls stream.read_to_string(&mut response), which reads the stream until end-of-file. On the other hand, your client calls write_all() followed by flush(), and (after uncommenting the offending code) attempts to read the response. But the attempt to read the response means that the stream is not closed, so the server will wait for EOF, and you have a deadlock on your hands. Note that none of this is specific to Rust; you would have the exact same issue in C++ or Python.
To fix the issue, you need to use a protocol in your communication. A very simple protocol could consist of first sending the message size (in a fixed format, perhaps 4 bytes in length) and only then the actual message. The code that reads from the stream would do the same: first read the message size and then the message itself. Even better than inventing your own protocol would be to use an existing one, e.g. to exchange messages using serde.
I currently have a server that handles multiple connections from clients, and client that connects to the server using two connections. My client has two processes that handle respectively sending and receiving to and from the server, but not both. The problem I currently have is when I want to close the socket, my reading process is stuck on the gen_tcp:recv/2 block. If I put a timeout, the socket is closed when the timeout has been reached. My question is, is it possible to have gen_tcp:recv/3 call that doesn't closes the socket.
This is how my reading process looks like.
read(Socket, Control) ->
Control ! ok,
receive
read ->
case gen_tcp:recv(Socket, 0) of
{ok, Data} ->
%% handling for messages;
Other ->
io:format(Other)
end,
read(self()), %% this sends "read" to itself with "!"
read(Socket, Control);
{error, Reason} ->
io:format(Reason)
end;
close ->
io:format("Closing Reading Socket.~n"),
gen_tcp:close(Socket)
end.
As you can see here, the process will never be able to receive a close if recv/2 doesn't read anything.
Sure, gen_tcp:recv/3 with timeout set to infinity will not close the socket :) See the official documentation.
Edit:
From the documentation:
This function receives a packet from a socket in passive mode.
Check the documentation for setopts/2 to understand the difference between passive and active modes. In particular:
If the value is false (passive mode), the process must explicitly receive incoming data by calling gen_tcp:recv/2,3.
Your process can only do one thing at a time - either listen for the close message from another process or wait for the TCP packet. You could try to use the gen_tcp:controlling_process/2 but I don't know the details. Another solution would be to handle the recv/3 in a separate (third) linked process and kill the process when the close is received.
The better way would be to use an active socket instead, see the Examples section in the official documentation for some guide how to do that.
The best way in my opinion would be to use the OTP gen_server to handle both, the close message and incoming TCP packets in the same process. There is an excellent tutorial in the Erlang and OTP in Action book on how to implement that and here is the code example on Github.
We are writing a message broker in Haskell (HMB). Therefore messages have to be parsed (Data.Binary) after they are received from socket (Network.Socket). We've been testing on loopback (localhost) so far - for producing and parsing messages. This worked quiet well. If we benchmark by producing messages from another machine we are facing problems: Suddenly the parser does not have enough bytes to parse.
The first 4 bytes of each message defines the length of the message and thus describes the message to be parsed. As hinted above, we do parsing with Data.Binary - so this is lazy. For testing purposes we switched parsing of the first 4 bytes to strict by using the cereal library. This the same problem. We now even tried to completely parse the requests with cereal only and the problem also remains.
In the code you'll see that we do threading. However, we also tried without a channel (single threaded setup) but this didn't solve the problem either.
Here is a part of the code (Thread1) where the received bytes are written to a channel to be further consumed/parsed. (As mentioned, nothing changes if we omit channeling and directly parse input):
runConnection :: (Socket, SockAddr) -> RequestChan -> Bool -> IO()
runConnection conn chan False = return ()
runConnection conn chan True = do
r <- recvFromSock conn
case (r) of
Left e -> do
handleSocketError conn e
runConnection conn chan False
Right input -> do
threadDelay 5000 -- THIS FIXES THE PROBLEM!?
writeToReqChan conn chan input
runConnection conn chan True
Here is the part (Thread2) where input is beeing parsed:
runApiHandler :: RequestChan -> ResponseChan -> IO()
runApiHandler rqChan rsChan = do
(conn, req) <- readChan rqChan
case readRequest req of -- readRequest IS THE PARSER
Left (bs, bo, e) -> handleHandlerError conn $ ParseRequestError e
Right (bs, bo, rm) -> do
res <- handleRequest rm
case res of
Left e -> handleHandlerError conn e
Right bs -> writeToResChan conn rsChan bs
runApiHandler rqChan rsChan
Now I figured out, that if the process of parsing is delayed a bit (see threadDelay in the first code block), everything works fine. Which basically means, the parser doesn't wait for bytes received from the socket.
Why is that? Why does the parser not wait for the socket the have enough bytes? Is there a general mistake in our setup?
I would bet that the problem has nothing to do with the parser but is instead due to the blocking semantics of UNIX sockets.
While a loopback
interface will likely pass the packet directly from the sender to the receiver,
an Ethernet interface may need to break up the packet to fit in the Maximum
Transmission Unit (MTU) of the link. This is known as packet fragmentation.
The len
argument to the recv system call is merely
the upper bound on the received length (e.g. the size of the target buffer); the
call may produce less data than you ask for. To quote the manpage,
If no messages are available at the socket, the receive calls wait for a
message to arrive, unless the socket is nonblocking (see fcntl(2)), in which
case the value -1 is returned and the external variable errno is set to
EAGAIN or EWOULDBLOCK. The receive calls normally return any data
available, up to the requested amount, rather than waiting for receipt of
the full amount requested.
For this reason, you may need multiple recv calls to retrieve the entire packet. Your example works if you delay the recv as the operating system can reassemble the original packet since all fragments have arrived by the time it is requested.
As meiersi pointed out, there are a variety of streaming I/O libraries that have developed in the Haskell world for solving this problem, among others. These include pipes, conduit, io-streams, and others. Depending upon your goals, this may be a natural way to handle this issue.
You might want to try the socket support in conduit-extra combined with binary-conduit to properly handle the parsing of the chunked streaming, which happens due to the reasons pointed out by bgamari.
First of all, consider yourself lucky to observe this. On many platforms perhaps only one out of a thousand packets exhibit this behaviour, causing a lot of such (sorry) bad networking code to fail seldom and randomly.
The problem is that you start processing before the data is ready. Instead of the threadDelay (which introduces a permanent delay and might not be long enough in all cases), the solution is to make sure you have at least one item/message/packet to process before you start processing it. Your protocol where the first 32bit word contains the length is perfect for this. Read data until you have at least 4 bytes (the length). Then read data until you have the required number of bytes. If any calls to recvFromSock returns less than the required number, call it again to get some more. Remember to also handle the case of 0 bytes, this means the other party closed the connection.
I have implemented this for a similar protocol (SMPP, packets also starts with the length) and it works perfectly.
I'm trying to run the following client and server socket example code in matlab:
http://www.mathworks.com/help/instrument/using-tcpip-server-sockets.html
This is my code.
Server:
t=tcpip('0.0.0.0', 9994, 'NetworkRole', 'server');
fopen(t);
data=fread(t, t.BytesAvailable, 'double');
plot(data);
Client:
data=sin(1:64);
t=tcpip('localhost', 9994, 'NetworkRole', 'client');
fopen(t);
fwrite(t, data, 'double');
This is what happens: I run the server code-> The program waits for the connection from the client-> I run the client code ->In the server console I get:
Error using icinterface/fread (line 163)
SIZE must be greater than 0.
Error in socketTentativaMatlab (line 3)
data=fread(t, t.BytesAvailable, 'double');
What am I doing wrong? It looks like the server doesn't wait for the client to send anything to try to read the data, so there's no data to read (It waits for the client connection thought).
Edit1:
Ok, I'm sending chars now, so we know for sure that t.BytesAvaiable = number of elements.
I have been able to successfully receive synchronously in the following way (this is server code, client code is the same but I send chars now and pause 1 second after establishing the connection with the server):
t=tcpip('0.0.0.0', 30000, 'NetworkRole', 'server');
fopen(t);
data=strcat(fread(t, 1, 'uint8')');
if get(t,'BytesAvailable') > 1
data=strcat(data,fread(t, t.BytesAvailable, 'uint8')');
end
data
This is because I suspected that bytesAvaiable is the number of bytes left to read after attempting to read at least once... this doesn't seem very logical, but it apparently is what happens. Since I have to read at least once to know how many bytes the message has...I choose to read 1 byte only the first time. I then read what's left, if there is something left...
I can make this work between matlab processes, but I can't do it between C++ and matlab. The C++ client successfully connects to the matlab server, and can send the data without problems or errors. However, on the matlab server side, I can't read it.
Something seems very wrong with all this matlab tcpip implementation!
Edit2:
If I properly close all the sockets in both client and server (basically don't let the program exit with open sockets), the above code seams to work consistently. I went to console and typed "netstat" to see all the connections ...It turns out since I was leaving open sockets, some connections were in the FIN_WAIT_2 state, which apparently rendered the ports of those connections unusable. Eventually the connection times out definitely, but it takes a minute or more, so, it's really best practice to just make sure the sockets are always properly closed.
I don't understand thought what is the logic behind t.BytesAvaiable... it doesn't seam to make much sense the way it is. If I loop and wait for it to become greater then 0, it eventually happens, but this is not the way things are supposed to be with synchronous sockets. My code lets one do things synchronously, even though I don't understand why t.BytesAvaiable isn't properly set the first time.
Final server code:
t=tcpip('0.0.0.0', 30000, 'NetworkRole', 'server');
fopen(t);
data=strcat(fread(t, 1, 'uint8'));
if get(t,'BytesAvailable') > 1
data=strcat(data,fread(t, t.BytesAvailable, 'uint8')');
end
fclose(t);
Final client code:
Your typical socket client, implemented in any language, but you will have to make sure that between successive calls of send() method/function (or between calling connect() and send()), at least 100ms (lower number seam to be risky) are ellapsed.
You are right, the server doesn't appear to be waiting for the client, even though the default mode of communication is synchronous. You can implement the waiting yourself, for example by inserting
while t.BytesAvailable == 0
pause(1)
end
before the read.
However, I've found that there are more problems – it's weird that the code from the MathWorks site is so bad – namely, t.BytesAvailable gives a number of bytes, while fread expects a number of values, and since one double value needs 8 bytes it has to say
data=fread(t, floor(t.BytesAvailable / 8), 'double');
Moreover, if the client writes the data immediately after opening the connection, I've found that the server simply overlooks them. I was able to fix this by inserting a pause(1) in the client code, like this
data=sin(1:64);
t=tcpip('localhost', 9994, 'NetworkRole', 'client');
fopen(t);
pause(1)
fwrite(t, data, 'double');
My impression is that Matlab's implementation of TCP/IP server client communication is quite fragile and needs a lot of workarounds...
Simple question...
This code ..
client() ->
SomeHostInNet = "localhost" % to make it runnable on one machine
{ok, Sock} = gen_tcp:connect(SomeHostInNet, 5678,
[binary, {packet, 0}]),
ok = gen_tcp:send(Sock, "Some Data"),
ok = gen_tcp:close(Sock).
is very clear except that I don't quite understand what [binary, {packet,0}] means ?
Any one cares to explain ?
MadSeb
As per the gen_tcp:connect documentation:
[binary, {packet, 0}] is the list of options that's passed to the connect function.
binary means that the data that is sent/received on the socket is in binary format (as opposed to, say, list format.
{packet, 0}, is a little confusing and it doesn't appear to be covered in the documentation. After talking to some knowledgeable chaps in #erlang on Freenode, I found that the packet option specifies how many bytes indicate the packet length. Behind the scenes, the length is stripped from the packet and erlang just sends you the packet without the length. Therefore {packet, 0} is the same as a raw packet without a length and everything is handled but the receiver of the data. For more information on this, check out inet:setopts.
Hope that helps.
{packet,0} is used to indicate that TCP data is delivered directly to the application in an unmodified form.
binary means that received packet is delivered as a binary. (but you can still use gen_tcp:send with a message like "message")