Nginx websocket proxy uses three connections per socket - sockets

Question

I am trying to create an Nginx configuration that will serve as a proxy to incoming websocket connections (mainly for SSL offloading), but I am running into connection limits. I followed several guides and SO answers to accommodate more connections but something weird caught my attention. I currently have 18K clients connected and when I run ss -s on the Nginx machine, this is the report:
Total: 54417 (kernel 54537)
TCP: 54282 (estab 54000, closed 280, orphaned 0, synrecv 0, timewait 158/0), ports 18263
Transport Total IP IPv6
* 54537 - -
RAW 0 0 0
UDP 1 1 0
TCP 54002 36001 18001
INET 54003 36002 18001
FRAG 0 0 0
I understand how there can be 36K IP connections, but what I do not get is where those additional IPv6 connections come from. I am having problems scaling above 25K connections and I think part of that comes from the fact that somehow there are three connections set up for each socket. So, my question is this: does anyone know where those extra connections are coming from?
The entire system is running within a Kubernetes cluster, with the configuration as follows:
nginx.conf:
user nginx;
worker_processes auto;
worker_rlimit_nofile 500000;
error_log /dev/stdout warn;
pid /var/run/nginx.pid;
# Increase worker connections to accommodate more sockets
events {
worker_connections 500000;
use epoll;
multi_accept on;
}
http {
include /etc/nginx/mime.types;
default_type application/octet-stream;
log_format main '$remote_addr - $remote_user [$time_local] "$request" '
'$status $body_bytes_sent "$http_referer" '
'"$http_user_agent" "$http_x_forwarded_for"';
access_log off; # don't use it, so don't waste cpu, i/o and other resources.
tcp_nopush on;
tcp_nodelay on;
include /etc/nginx/conf.d/*.conf;
}
proxy.conf (included via conf.d):
server {
listen 0.0.0.0:443 ssl backlog=100000;
# Set a big keepalive timeout to make sure no connections are dropped by nginx
# This should never be less than the MAX_CLIENT_PING_INTERVAL + MAX_CLIENT_PING_TIMEOUT in the ws-server config!
keepalive_timeout 200s;
keepalive_requests 0;
proxy_read_timeout 200s;
ssl_certificate /app/secrets/cert.chain.pem;
ssl_certificate_key /app/secrets/key.pem;
ssl_prefer_server_ciphers On;
ssl_protocols TLSv1.2;
location / {
proxy_pass http://127.0.0.1:8443;
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "upgrade";
}
}
I also set the following options in Unix:
/etc/sysctl.d/custom.conf:
fs.file-max = 1000000
fs.nr_open = 1000000
net.ipv4.netfilter.ip_conntrack_max = 1048576
net.core.somaxconn = 1048576
net.ipv4.tcp_max_tw_buckets = 1048576
net.ipv4.ip_local_port_range 1024 65000
net.ipv4.tcp_max_syn_backlog = 3240000
net.nf_conntrack_max = 1048576
net.ipv4.tcp_tw_reuse= 1
net.ipv4.tcp_fin_timeout= 15
/etc/security/limits.d/custom.conf:
root soft nofile 1000000
root hard nofile 1000000
* soft nofile 1000000
* hard nofile 1000000

Answer 1

With help of some colleagues I found out that this is actually Kubernetes confusing everything by joining containers within a Pod in one IP namespace (so that each container can access the other via localhost (link)). So what I see there:
Incoming connections from the proxy
Outgoing connections from the proxy
Incoming connections from the server
Although this does not help me to achieve more connections on a single instance, it does explain the weird behaviour.