Mongo 3.4 on Amazon Linux - Maximum simultaneous connections stops at 4077 - mongodb

Question

I'm running mongoDB 3.4 on a t2.micro EC2 instance (Amazon Linux 2.0 (2017.12))
Following is the ulimit -a configuration in the instance.
core file size (blocks, -c) 0
data seg size (kbytes, -d) unlimited
scheduling priority (-e) 0
file size (blocks, -f) unlimited
pending signals (-i) 3867
max locked memory (kbytes, -l) 64
max memory size (kbytes, -m) unlimited
open files (-n) 50000
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority (-r) 0
stack size (kbytes, -s) 8192
cpu time (seconds, -t) unlimited
max user processes (-u) 3867
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited
You can see that the number of open files set is 50000.
So I expected that mongo will allow nearly 50000 connections to the running mongo instance. But I'm unable to get more than 4077 connections simultaneously. In the /var/log/mongodb/mongod.log I can see that the current open connections is 4077 and new connections are getting rejected because it fails to create threads for new connection requests.
I'm not even able to connect to the mongo shell from the terminal. Its unable to create the sockets. I can connect to the DB if I release the 4077 connections that are open now.
How can I specify the maximum simultaneous connections within the mongo config file? Do I change any other parameters in the OS environment like ulimit?

Answer 1

I can see that the current open connections is 4077 and new connections are getting rejected because it fails to create threads for new connection requests.
A t2.micro instance only provides 1GiB of RAM. Each database connection will use up to 1 MB of RAM, so with 4000+ connections you are likely to have exhausted the available resources of your server. Assuming you are using the default WiredTiger storage engine in MongoDB 3.4, you probably have 256MB of RAM allocated to the WiredTiger cache by default and the remaining memory has to be shared between connection threads, your O/S, and any other temporary memory allocation required by mongod.
How can I specify the maximum simultaneous connections within the mongo config file? Do I change any other parameters in the OS environment like ulimit?
Resource limits are intended to impose a reasonable upper bound so a system administrator can intervene before the system becomes non-responsive. There are two general categories of limits for connections: those imposed by your MongoDB server configuration (in this case net.maxIncomingConnections ) and those imposed by your operating system (ulimit -a).
In MongoDB 3.4, net.maxIncomingConnections defaults to 65,536 simultaneous connections, so ulimit settings for files or threads are typically reached before the connection limit.
For a server with more capacity than a t2.micro, it typically makes sense to increase limits from the default. However, given the limited resources of a t2.micro instance I would actually recommend reducing limits if you want your deployment to be stable.
For example, a more realistic limit would be to set net.maxIncomingConnections to 100 connections (or an expected max of 100MB of RAM for connections). In your case you are aiming for 50,000 connections so you could either set that value or leave the default (65,536) and rely on ulimit restrictions.
Your ulimit settings already allow more consumption than your instance can reasonably cope with, but the MongoDB manual has a reference if you'd like to Review and Set Resource Limits. You could consider increasing your -u value (max processes/threads) as this is likely the current ceiling you are hitting, but as with connections I would consider what is reasonable given available resources and your workload.