Increasing max outbound connections on CentOS - sockets

Question

I learned from this articl: Scaling to 12 Million Concurrent Connections: How MigratoryData Did It that it's possible to make more than 64K connections from a single client with multiple IP.
Now I have an AWS ec2 machine that has 10 IPs for testing. The config in /etc/sysctl.conf is
fs.nr_open = 2000000
fs.file-max = 2000000
And the config in /etc/security/limits.d/def.conf is
* soft nofile 2000000
* hard nofile 2000000
I start one process (written in C) and create 60000 connections from the first IP address. Everything works fine.
Than I started another process and try to create 60000 connections from the second IP address but it gets error when the number of connections reaches about 7500 (total number: 67500). The error message is Connection timed out.
The problem doesn't seem to be file descriptor limitations because I still can open/read/write files in the client machine. But any out going connection to any remote server gets timed out.
The problem is not in the server side because the server can accept many more connection from different client machine.
It looks like there's some kind of settings rather than number of open files that limits the number of outgoing connections. Can anyone help?

Answer 1

In order to be able to open more than 65536 TCP socket connections from your client machine, you have to use indeed more IP addresses.
Then, for each TCP socket connection, you should tell the kernel which IP address and which ephemeral port to use.
So, after the TCP client creates a socket and before it connects to the remote address, the TCP client should explicitly bind one of the local IP addresses available on your client machine to the socket.
The MigratoryData Benchmark Tools are written in Java so I cannot provide you the exact code that we use to open any number of TCP connections on the client side. But, here is a quick example written in C++.
Suppose your TCP server listens on 192.168.1.1:8800 and suppose 192.168.1.10 is one of the IP addresses of your client machine, then you can create a socket connection from the local IP address 192.168.1.10 and an ephemeral local port -- let's say 12345 -- to the remote IP address 192.168.1.1 and the remote port 8800 using something like:
#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>
#include<sys/socket.h>
#include <arpa/inet.h>
int main(int argc, char *argv[])
{
int n, sockfd;
char buffer[1024];
struct sockaddr_in localaddr, remoteaddr;
sockfd = socket(AF_INET, SOCK_STREAM, 0);
localaddr.sin_family = AF_INET;
localaddr.sin_addr.s_addr = inet_addr("192.168.1.10");
localaddr.sin_port = htons(12345);
bind(sockfd, (struct sockaddr *) &localaddr, sizeof(localaddr));
remoteaddr.sin_family = AF_INET;
remoteaddr.sin_addr.s_addr = inet_addr("192.168.1.1");
remoteaddr.sin_port = htons(80);
connect(sockfd, (struct sockaddr *) &remoteaddr, sizeof(remoteaddr));
n = read(sockfd, buffer, 512);
// ...
close(sockfd);
return 0;
}