Increasing max outbound connections on CentOS - sockets

I learned from this articl: Scaling to 12 Million Concurrent Connections: How MigratoryData Did It that it's possible to make more than 64K connections from a single client with multiple IP.
Now I have an AWS ec2 machine that has 10 IPs for testing. The config in /etc/sysctl.conf is
fs.nr_open = 2000000
fs.file-max = 2000000
And the config in /etc/security/limits.d/def.conf is
* soft nofile 2000000
* hard nofile 2000000
I start one process (written in C) and create 60000 connections from the first IP address. Everything works fine.
Than I started another process and try to create 60000 connections from the second IP address but it gets error when the number of connections reaches about 7500 (total number: 67500). The error message is Connection timed out.
The problem doesn't seem to be file descriptor limitations because I still can open/read/write files in the client machine. But any out going connection to any remote server gets timed out.
The problem is not in the server side because the server can accept many more connection from different client machine.
It looks like there's some kind of settings rather than number of open files that limits the number of outgoing connections. Can anyone help?

In order to be able to open more than 65536 TCP socket connections from your client machine, you have to use indeed more IP addresses.
Then, for each TCP socket connection, you should tell the kernel which IP address and which ephemeral port to use.
So, after the TCP client creates a socket and before it connects to the remote address, the TCP client should explicitly bind one of the local IP addresses available on your client machine to the socket.
The MigratoryData Benchmark Tools are written in Java so I cannot provide you the exact code that we use to open any number of TCP connections on the client side. But, here is a quick example written in C++.
Suppose your TCP server listens on 192.168.1.1:8800 and suppose 192.168.1.10 is one of the IP addresses of your client machine, then you can create a socket connection from the local IP address 192.168.1.10 and an ephemeral local port -- let's say 12345 -- to the remote IP address 192.168.1.1 and the remote port 8800 using something like:
#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>
#include<sys/socket.h>
#include <arpa/inet.h>
int main(int argc, char *argv[])
{
int n, sockfd;
char buffer[1024];
struct sockaddr_in localaddr, remoteaddr;
sockfd = socket(AF_INET, SOCK_STREAM, 0);
localaddr.sin_family = AF_INET;
localaddr.sin_addr.s_addr = inet_addr("192.168.1.10");
localaddr.sin_port = htons(12345);
bind(sockfd, (struct sockaddr *) &localaddr, sizeof(localaddr));
remoteaddr.sin_family = AF_INET;
remoteaddr.sin_addr.s_addr = inet_addr("192.168.1.1");
remoteaddr.sin_port = htons(80);
connect(sockfd, (struct sockaddr *) &remoteaddr, sizeof(remoteaddr));
n = read(sockfd, buffer, 512);
// ...
close(sockfd);
return 0;
}

Related

Binding to UDP socket *from* a specific IP address

I have packets coming from a specific device directly connected to my machine. When I do a tcpdump -i eno3 -n -n, I can see the packets:
23:58:22.831239 IP 192.168.0.3.6516 > 255.255.255.255.6516: UDP, length 130
eno3 is configured as 192.168.0.10/24
When I set the socket the typical way:
gOptions.sockfd = socket(AF_INET, SOCK_DGRAM, 0);
memset((void *)&gOptions.servaddr, 0, sizeof(struct sockaddr_in));
gOptions.servaddr.sin_family = AF_INET;
inet_pton(AF_INET, gOptions.sourceIP, &(gOptions.servaddr.sin_addr));
gOptions.servaddr.sin_port = htons(gOptions.udpPort);
bind(gOptions.sockfd, (struct sockaddr *)&gOptions.servaddr, sizeof(struct sockaddr_in));
And I use the sourceIP of "255.255.255.255" on port "6516" - it connects and reads.
What I want to do, however, is bind such that I am limiting my connection from the source IP - "192.168.0.3". I have figured out how to connect on the device using either device name ("eno3") of the iface of that device ("192.168.0.10") - but that doesn't help as I may have multiple devices connected to "192.168.0.10" that blab on that port, but I only want the packets from 192.168.0.3 for port 6516.
I thought s_addr - part of sin.addr - was the source IP... but it is not.
You can't bind() to a remote IP/port, only to a local IP/port. So, for what you have described, you need to bind() to the IP/port where the packets are being sent to (192.168.0.10:6516).
Now, you have two options to choose from. You can either:
use recvfrom() to receive packets, using its src_addr parameter to be given each sender's IP/port, and then you can discard packets that were not sent from the desired sender (192.168.0.3:6516).
or, use connect() to statically assign the desired sender's IP/port (192.168.0.3:6516), and then you can use recv() (not recvfrom()) to receive packets from only that sender.

Can I detect whether an UDP-socket or a connected UDP socket is used?

Can I detect whether a client application uses an UDP-socket or a connected UDP-socket?
If yes, how? If no, why?
As I said in my comment above, code call connect on a UDP socket. That enforces only traffic to/from the connection address is allowed (and all other packets get dropped) and allows you to use send instead of sendto, but the traffic is still UDP.
But you can use the netstat command from the command line to see if the datagram socket has a remote address association:
For example, imagine if the code did this:
// create a datagram socket that listens on port 12345
sock = socket(AF_INET, SOCK_DGRAM, 0);
port = 12345;
addrLocal.sin_family = AF_INET;
addrLocal.sin_port = htons(port);
result = bind(sock, (sockaddr*)&addrLocal, sizeof(addrLocal));
// associate the socket only with packets arriving from 1.2.3.4:6666
addrRemote.sin_family = AF_INET;
addrRemote.sin_port = htons(6666);
addrRemote.sin_addr.s_addr = ipaddress; // e.g. "1.2.3.4"
result = connect(sock, (sockaddr*)&addrRemote, sizeof(addrRemote));
A corresponding netstat -a -u will reveal the following:
ubuntu#ip-10-0-0-15:~$ netstat -u -a
Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address Foreign Address State
udp 0 0 ip-10-0-0-15:12345 1.2.3.4:6666 ESTABLISHED
The presence of a value that isn't *:* in the Foreign Address column for the UDP socket will reveal if the socket has connection address associated with it.

how does Operating System map port number to process that use it

For a simple python server using TCP socket as below, when there comes a TCP packet, and transport layer get port number, how does OS/transport layer know which thread/process to wake up(assuming the thread/process is blocking because of recv() system call)? for the code below, both parent thread and child thread have the connectionsocket file descriptor, how OS know which one to wake up? Thanks
host = 'localhost'
port = 55567
buf = 1024
addr = (host, port)
welcomesocket = socket(AF_INET, SOCK_STREAM)
welcomesocket.bind(addr)
welcomesocket.listen(2)
while 1:
connectionsocket, clientaddr = serversocket.accept()
thread.start_new_thread(handler, (connectionsocket, clientaddr))
serversocket.close()
There is an hash map tracking all used port in the kernel space.
When a packet arrives, kernel lookup the table using the port information in the packet, find the associated socket, and notify it
Here is how linux do it http://lxr.free-electrons.com/source/net/ipv4/udp.c#L489

How do I get client's port number from the server?

I have written a simple client-server program. I am able to print the port number of client in the client program. The values are dynamic.
But when I try to print it in the server program, it gives me the port number of the server and not the client.
connfd = accept(listenfd, (struct sockaddr*) &cliaddr, &clilen);
cout<<"Server: Server's Port: "<< ntohs(servaddr.sin_port)<<endl; /*23112*/
cout<<"Server: Client's Port: "<< ntohs(cliaddr.sin_port)<<endl; /*23112*/
I am using the same in client program, and it is printing the random ports correctly.
I have initialized them this way:
struct sockaddr_in cliaddr, servaddr;
socklen_t clilen = sizeof(cliaddr);
You need to initialize 'clilen':
The addrlen argument is a value-result argument: the caller must
initialize it to contain the size (in bytes) of the structure pointed
to by addr; on return it will contain the actual size of the peer
address.

Socket remote connection problem C

I wrote a simple server application in C. This server do nothing except print the received message, then exit. Here is the code
int listenfd,connfd,n;
struct sockaddr_in servaddr,cliaddr;
socklen_t clilen;
char *mesg = (char*) malloc(1000*sizeof(char));
listenfd=socket(PF_INET,SOCK_STREAM,0);
bzero(&servaddr,sizeof(servaddr));
servaddr.sin_family = AF_INET;
servaddr.sin_addr.s_addr = INADDR_ANY;
servaddr.sin_port=htons(20600);
bind(listenfd,(struct sockaddr *)&servaddr,sizeof(servaddr));
listen(listenfd,5);
clilen=sizeof(cliaddr);
connfd = accept(listenfd,(struct sockaddr *)&cliaddr,&clilen);
n = (int) recvfrom(connfd,mesg,1000,0,(struct sockaddr *)&cliaddr,&clilen);
sendto(connfd,mesg,n,0,(struct sockaddr *)&cliaddr,sizeof(cliaddr));
printf("-------------------------------------------------------\n");
mesg[n] = 0;
printf("Received the following:\n");
printf("%s\n",mesg);
printf("-------------------------------------------------------\n");
close(connfd);
close(listenfd);
I managed to establish a connection using telnet and running
telnet 192.168.1.2 20600
where 192.168.1.2 is the local ip of the server.
The machine runs behind a router ZyXel p-660HW-61 (192.168.0.1).
The problem is I cannot reach the server if I specify the public ip of the machine (151.53.150.45).
I set NAT configuration to the server local ip on all port from 20000 to 21000
http://img593.imageshack.us/img593/3959/schermata20110405a22492.png
port 20600 seems to be open, according to canyouseeme.org/ and yougetsignal.com/tools/open-ports/ (in fact I can read in the console that a packet has been received), but if I run
telnet 151.53.150.45 20600
I get a "Connection Refused" error.
Firewall is disabled, both on the router and on the server machine (that is the same running telnet).
Any help?
If you are typing:
telnet 151.53.150.45 20600
from the LAN rather than from the WAN, then your NAT most probably does not handle hairpin situations properly. This means it only expects you to use the translated address from the WAN.
The solution is check whether you can change the configuration of your NAT to enable usage of translated address on the LAN too (it is sometimes a requirement for P2P systems). If such functionalities are not available, then you need a new NAT.