I found an interesting problem when using gen_tcp behavior. I have a server and a client. The server accepts connections and the client creates many processes that all try to connect to the listening server.
If I try to start the client which spawns many processes that all try to connect to the socket at the same time then many fail. However if I put timer:sleep(x) then every socket is being accepted.
Does this mean that gen_tcp:accept() has a limit where it can accept some connection request?
Code for the server and the client follows:
accept(State = #state{lsocket = LSocket, num = Num}) ->
case gen_tcp:accept(LSocket) of
{ok, Socket} ->
io:format("Accepted ~p ~n", [Num]),
{sockets, List} = hd(ets:lookup(csockets, sockets)),
NewList = [Socket | List],
ets:insert(csockets, {sockets, NewList}),
Pid = spawn(fun() -> loop(Socket) end),
gen_tcp:controlling_process(Socket, Pid),
accept(State#state{num = Num + 1});
{error, closed} -> State
end.
loop(Socket) ->
case gen_tcp:recv(Socket, 0) of
{ok, Data} ->
gen_tcp:send(Socket, Data),
loop(Socket);
{error, closed} ->
io:format(" CLOSED ~n"),
ok
end.
Client:
send(State = #state{low = Low, high = Low}) ->
State;
send(State = #state{low = Low}) ->
N = Low rem 10,
Dest = lists:nth(N + 1, State#state.dest),
spawn(?MODULE, loop, [Dest, Low]),
%%timer:sleep(1),
NewState = State#state{low = Low + 1},
send(NewState).
loop({IP, Port}, Low) ->
case gen_tcp:connect(IP, Port, [binary]) of
{ok, Socket} ->
io:format("~p Connected ~n", [Low]),
gen_tcp:send(Socket, "Hi"),
receive
{tcp, RecPort, Data} ->
io:format("I have received ~p on port ~p ~p ~n", [Data, RecPort, Low])
end;
_Else ->
io:format("The connection failed ~n"),
loop({IP, Port}, Low)
end.
It is true that a single process can only gen_tcp:accept/1 so fast, though I'm not sure that that's the bottleneck you're running in to.
You might be interested in Ranch, the TCP library for the Cowboy webserver. The manual includes a section on internal features that talks about using multiple acceptors.
In your case, you should try to produce more debugging output for yourself. Printing the error when the client fails to connect would be a good start -- there are lots reasons why a TCP client might fail to connect.
Related
I have socket server with this method:
#impl true
def handle_call({:tcp, socket, packet}, state) do
Logger.info("Received packet: \x02#{packet}\x03 and send response")
{:reply, {:ok, packet}, state}
end
I wrote script in python that send "\x02Test\x03" to socket:
import socket
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.connect(("127.0.0.1", port))
s.send("\x02Test\x03".encode())
print(s.recv(1024))
But the response printed by python is b'\x02\x02Test\x03\x03'
What does handle_call() have to do with gen_tcp?
gen_tcp can be configured to read from a socket and send() messages to whatever process called :gen_tcp.accept(), the so called controlling process, which can be a gen_server; but a gen_server handles messages sent to its mailbox with handle_info()--not handle_call(). handle_call() handles messages sent by :gen_server.call().
Here's an example:
defmodule TcpServer do
use GenServer
require Logger
def start_link() do
ip = Application.get_env :gen_tcp, :ip, {127,0,0,1}
port = Application.get_env :gen_tcp, :port, 6666
IO.puts "gen_tcp is listening on port: #{port}"
GenServer.start_link(__MODULE__, {ip, port},[])
end
def init({ip, port}) do
{:ok, listen_socket}= :gen_tcp.listen(
port,
[:binary, {:packet,0}, {:active,true}, {:ip,ip}]
)
{:ok, socket } = :gen_tcp.accept listen_socket
{:ok, %{ip: ip, port: port, socket: socket} }
end
def handle_call({:tcp, _socket, packet}, state) do
Logger.info("handle_call(): Received packet: #{inspect packet}")
{:reply, {:ok, packet}, state}
end
def handle_info({:tcp,socket,packet},state) do
Logger.info "handle_info(:tcp, ...): incoming packet: #{inspect packet}"
:gen_tcp.send(socket, "****#{packet}*****")
{:noreply,state}
end
def handle_info({:tcp_closed, _socket}, state) do
Logger.info("handle_info({:tcp_closed, ...): Client closed socket.")
{:noreply, state}
end
def handle_info({:tcp_error, socket, reason}, state) do
Logger.info("Connection closed due to #{reason}: #{socket}")
{:noreply,state}
end
end
To start the server:
~/elixir_programs/tcp_server$ iex -S mix
Erlang/OTP 20 [erts-9.3] [source] [64-bit] [smp:4:4] [ds:4:4:10] [async-threads:10] [hipe] [kernel-poll:false]
Interactive Elixir (1.8.2) - press Ctrl+C to exit (type h() ENTER for help)
iex(1)> TcpServer.start_link()
gen_tcp is listening on port: 6666
In another terminal window:
~/python_programs$ cat 5.py
import socket
port = 6666
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.connect(("127.0.0.1", port))
s.send("\x02Test\x03".encode())
print(s.recv(1024))
~/python_programs$ p36 5.py
b'****\x02Test\x03*****'
~/python_programs$
As you can see, there was no duplication of the \x02 and \x03 characters.
Back in the server window:
{:ok, #PID<0.123.0>}
iex(2)>
21:54:18.363 [info] handle_info(:tcp, ...): incoming packet: <<2, 84, 101, 115, 116, 3>>
21:54:18.369 [info] handle_info({:tcp_closed, ...): Client closed socket.
Do you have another process that is the controlling process and is calling :gen_server.call({:tcp, socket, packet})?
By the way, in this code:
def handle_info({:tcp,socket,packet},state) do
packet may be the whole packet, 1/2 the packet or 1/10 of the packet. That's the way sockets work. The config option {packet, 0} tells gen_tcp that there is no length header (0 bytes) on the front of the packet, while {packet, 1|2|4} tells gen_tcp that the length of the packet is contained in the first byte, the first 2 bytes, or the first 4 bytes respectively. That way gen_tcp can read the first 1|2|4 bytes to get the packet length, say L, then keep reading from the socket until it has received L bytes. Then gen_tcp packages up the pieces into one message, and sends the whole message to the gen_server. On the other hand, when you specify {packet, 0}, you are telling gen_tcp that the packet has no length header; and because a socket splits up a single packet into an indeterminate number of chunks, gen_tcp has no idea where the end of the packet is, so gen_tcp's only option is to read a chunk from the socket and send the chunk to the gen_server; then read another chunk and send the chunk to the gen_server, etc., etc. That means the gen_server has to figure out where the end of the packet is.
Therefore, your server and your client have to agree on a protocol to signal the end of a packet; and handle_info(:tcp, ...) will have to store the pieces of the packet in the state (or in a db) until it has read all the chunks that make up a packet.
One protocol that you can use to signal the end of the packet is: the client closes the socket. In that case,
def handle_info({:tcp_closed, _socket}, state)
will be called, and inside that function clause you can assemble the chunks stored in state (or in a db) into a complete message, then do whatever is necessary, e.g. send the message back to the client.
If you use STX and ETX as your begin message, end message protocol, then handle_info(:tcp, ...) will still have to look for the STX character to signal that it should start storing chunks in state, and when handle_info(:tcp, ...) finds a chunk with an ETX character in it, then you have to assemble the entire message.
I'm new in Erlang.
My Problem is, that when I start the client for the 1st time everything seems okay, I get the sorted list: <<1,5,72,97,108,108,111>>.
But by the 2nd time it won't receive the sorted list, because I think the socket is closed. The output from the Client is "Connection closed".
Here is my code:
Client
-module(client).
-export([client/0]).
client() ->
case gen_tcp:connect("localhost", 6000, [{mode, binary}]) of
{ok, Sock} ->
Data = [1, "Hallo", 5],
gen_tcp:send(Sock, Data),
receive
{tcp, _, Bin} ->
io:fwrite("Received sorted list from server: ~w~n", [Bin]);
{tcp_closed, _} ->
io:fwrite("Connection closed"),
gen_tcp:close(Sock)
end;
{error,_} ->
io:fwrite("Connection error! Quitting...~n")
end.
Server
-module(server).
-export([server/0]).
-import(mergeSort,[do_recv/1]).
%creates a tcp socket on Port 6000
server() ->
{ok, Listen} = gen_tcp:listen(6000, [{keepalive, true}, %send keepalive packets
{reuseaddr, true}, %reuse address
{active, once}, %socket is active once
{mode, list}]), %binary traffic
spawn(fun() -> parallel_connection(Listen) end).
%server is listening
%accepts the connection
%starts MergeSort
parallel_connection(Listen) ->
io:fwrite("Listening connections..~n"),
{ok, Socket} = gen_tcp:accept(Listen),
io:fwrite("Connection accepted from ~w~n", [Socket]),
spawn(fun() -> parallel_connection(Listen) end),
do_recv(Socket).
MergeSort
-module(mergeSort).
-export([do_recv/1]).
merge_sort(List) -> m(List, erlang:system_info(schedulers)).
%break condition
m([L],_) ->
[L];
%for more than one scheduler
m(L, N) when N > 1 ->
{L1,L2} = lists:split(length(L) div 2, L),
%self () returns Pid, make_ref() returns almost unique reference
{Parent, Ref} = {self(), make_ref()},
%starts a new process for each half of the list
%and sends Message to Parent
spawn(fun()-> Parent ! {l1, Ref, m(L1, N-2)} end),
spawn(fun()-> Parent ! {l2, Ref, m(L2, N-2)} end),
{L1R, L2R} = receive_results(Ref, undefined, undefined),
lists:merge(L1R, L2R);
m(L, _) ->
{L1,L2} = lists:split(length(L) div 2, L),
lists:merge(m(L1, 0), m(L2, 0)).
receive_results(Ref, L1, L2) ->
receive
{l1, Ref, L1R} when L2 == undefined -> receive_results(Ref, L1R, L2);
{l2, Ref, L2R} when L1 == undefined -> receive_results(Ref, L1, L2R);
{l1, Ref, L1R} -> {L1R, L2};
{l2, Ref, L2R} -> {L1, L2R}
after 5000 -> receive_results(Ref, L1, L2)
end.
do_recv(Socket) ->
%{ok, {Address, _}} = inet:peername(Socket),
receive
{tcp, Socket, List} ->
try
Data = merge_sort(List),
gen_tcp:send(Socket, Data),
io:fwrite("Sent sorted list to ~w | Job was done! Goodbye :)~n", [Socket]),
gen_tcp:close(Socket)
catch
_:_ ->
io:fwrite("Something went wrong with ~w | Worker terminated and connection closed!~n", [Socket]),
gen_tcp:close(Socket)
end;
{tcp_closed, _} ->
io:fwrite("Connection closed ~n");
{error, _} ->
io:fwrite("Connection error from ~w | Worker terminated and connection closed!~n", [Socket]),
gen_tcp:close(Socket)
end.
Can anyone help me?
When you call client:client/0, it creates a connection, sends its data, receives the response, then returns. Meanwhile, the server closes the socket. When you call client:client/0 again, it again creates a connection and sends data, but then it receives the tcp_closed message for the previous socket, and then it returns.
You can fix this by specifying the client socket in your receive patterns:
receive
{tcp, Sock, Bin} ->
io:fwrite("Received sorted list from server: ~w~n", [Bin]);
{tcp_closed, Sock} ->
io:fwrite("Connection closed"),
gen_tcp:close(Sock)
end;
In this code, the variable Sock replaces both the underscores in the original code, in the {tcp, _, Bin} and {tcp_closed, _} tuples. This forces the messages to match only for the specified socket.
I need to stream database data/string to client using Erlang/Yaws. I found this documentation to achieve this but this example uses open_port to send data.
this is the example from [http://yaws.hyber.org/stream.yaws][1]
out(A) ->
%% Create a random number
{_A1, A2, A3} = now(),
random:seed(erlang:phash(node(), 100000),
erlang:phash(A2, A3),
A3),
Sz = random:uniform(100000),
Pid = spawn(fun() ->
%% Read random junk
S="dd if=/dev/urandom count=1 bs=" ++
integer_to_list(Sz) ++ " 2>/dev/null",
P = open_port({spawn, S}, [binary,stream, eof]),
rec_loop(A#arg.clisock, P)
end),
[{header, {content_length, Sz}},
{streamcontent_from_pid, "application/octet-stream", Pid}].
rec_loop(Sock, P) ->
receive
{discard, YawsPid} ->
yaws_api:stream_process_end(Sock, YawsPid);
{ok, YawsPid} ->
rec_loop(Sock, YawsPid, P)
end,
port_close(P),
exit(normal).
rec_loop(Sock, YawsPid, P) ->
receive
{P, {data, BinData}} ->
yaws_api:stream_process_deliver(Sock, BinData),
rec_loop(Sock, YawsPid, P);
{P, eof} ->
yaws_api:stream_process_end(Sock, YawsPid)
end.
I need to stream string, I managed to understand the process until here except port_close(p)-which obviously closes the port.
rec_loop(Sock, P) ->
receive
{discard, YawsPid} ->
yaws_api:stream_process_end(Sock, YawsPid);
{ok, YawsPid} -> rec_loop(Sock, YawsPid, P)
end,
port_close(P),
exit(normal).
What I do not understand is this part.
rec_loop(Sock, YawsPid, P) ->
receive
{P, {data, BinData}} ->
yaws_api:stream_process_deliver(Sock, BinData),
rec_loop(Sock, YawsPid, P);
{P, eof} ->
yaws_api:stream_process_end(Sock, YawsPid)
end.
Now, There is no documentation on {P, {data, BinData}} -> nor {P, eof} -> and I need to change the content type
{streamcontent_from_pid, "application/octet-stream", Pid}. to {streamcontent_from_pid, "text/html; charset=utf-8", Pid}
So the question is How do I stream text/string without using port?
The Yaws example creates an OS process to read from /dev/urandom, a special file that delivers pseudorandom values, and it uses a port to communicate with that process. It runs the port within an Erlang process that serves as the content streamer.
The content streamer process first awaits directions from Yaws by receiving either {discard, YawsPid} or {ok, YawsPid}. If it gets the discard message, it has no work to do, otherwise it calls rec_loop/3, which loops recursively, taking in data from the port and streaming it to a Yaws HTTP socket. When rec_loop/3 gets an end-of-file indication from the port, it terminates its streaming by calling yaws_api:stream_process_end/2 and then returns to rec_loop/2, which in turn closes the port and exits normally.
For your application, you need a streaming process that, just like the Yaws example, first handles either {discard, YawsPid} or {ok, YawsPid}. If it gets {ok, YawsPid}, it should then go into a loop receiving messages that supply the text you want to stream to the HTTP client. When there's no more text to send, it should receive some sort of message telling it to stop, after which it should exit normally.
I create one udp client, and need to send message every 5s, so i write
start() ->
{ok, Sock} = gen_udp:open(0, []),
send(Sock).
send(Sock) ->
gen_udp:send(Sock, "127.0.0.1", 3211, "hello world"),
timer:sleep(5000),
send(Sock).
I want to know a good place to close the socket
If your goal is to send a message every 5 seconds, then why would you want to close the socket? If you have some logic to determine when you have sent enough messages (you count them for example), then that would be the place to close the socket.
Here's an example of how you could count the messages in a long-running process:
start() ->
{ok, Sock} = gen_udp:open(...),
send(Sock, 0),
gen_udp:close(Sock).
send(Sock, N) when N >= ?MAX_MESSAGE_COUNT ->
ok;
send(Sock, N) ->
...
send(Sock, N+1).
By counting up to a given number, instead of down, you can change this number while the process is running by simply reloading the code.
After reading this answer, I want to understand if the same applies to the calls to gen_tcp:recv(Socket, Length). My understanding of the documentation is that this if more than Length bytes are available in the buffer, they remain there; if there is less than Length bytes, the call blocks until enough is available or connection closes.
In particular, this should work when packets are prefixed by 2 bytes holding packet length in little-endian order:
receive_packet(Socket) ->
{ok, <<Length:16/integer-little>>} = gen_tcp:recv(Socket, 2),
gen_tcp:recv(Socket, Length).
Is this correct?
Yes (or No, see comments for details).
Consider:
Shell 1:
1> {ok, L} = gen_tcp:listen(8080, [binary, {packet, 0}, {active, false}]).
{ok,#Port<0.506>}
2> {ok, C} = gen_tcp:accept(L). %% Blocks
...
Shell 2:
1> {ok, S} = gen_tcp:connect("localhost", 8080, [binary, {packet, 0}]).
{ok,#Port<0.516>}
2> gen_tcp:send(S, <<0,2,72,105>>).
ok
3>
Shell 1 cont:
...
{ok,#Port<0.512>}
3> {ok, <<Len:16/integer>>} = gen_tcp:recv(C, 2).
{ok,<<0,2>>}
4> Len.
2
5> {ok, Data} = gen_tcp:recv(C, Len).
{ok,<<"Hi">>}
6>
However this is useful if you only want to confirm the behaviour. In reality you would change the {packet, N} option to define how many bytes that should be the packet length (on big-endian systems).
Same as before but without extracting length explicitly (note packet length = 2 in shell 1):
Shell 1:
1> {ok, L} = gen_tcp:listen(8080, [binary, {packet, 2}, {active, false}]).
{ok,#Port<0.506>}
2> {ok, C} = gen_tcp:accept(L). %% Blocks
...
In this case Erlang will strip the first 2 bytes and recv/2 will block until as many bytes it needs. In this case read-length must be 0 in recv/2.
Shell 2:
1> {ok, S} = gen_tcp:connect("localhost", 8080, [binary, {packet, 0}]).
{ok,#Port<0.516>}
2> gen_tcp:send(S, <<0,2,72,105>>).
ok
3>
Shell 1:
...
{ok,#Port<0.512>}
3> {ok, Data} = gen_tcp:recv(C, 0).
{ok,<<"Hi">>}
In this case I don't specify the {packet, N} option in shell 2 just to show the idea but normally it is not 0. If the packet option is set then gen_tcp will automatically append/strip that many bytes from the package.
If you specify packet 0 then you must do a recv/2 with a length >= 0 and the behaviour is the same as in C. You can simulate non-blocking receives by giving a short time out when doing the receive and this will return {error, timeout} in that case.
More on that can be read here:
http://www.erlang.org/doc/man/gen_tcp.html
http://www.erlang.org/doc/man/inet.html#setopts-2
Hope this clears things up.