MongoDB Atlas has rate limit of 100 requests per minute per project but if I host mongodb on my own server then will it have a rate limit or I can make unlimited read/write calls per minute to database. (obviously it is relative to the specifications of server) I am using Node js for making calls with mongoose
No, when you install MongoDB in your own premises, then you don't have any limit. Of course, the system has some MongoDB Limits and Thresholds but I don't think you will hit any of them.
Related
What is the maximum theoretical number of parallel requests that we can squize from single mongodb instance before deciding to shard?
Considering the database and indexes fit in memory and all requests are find() queries fetching single document based on indexed field. The hosting OS is Ubuntu , the data partition is SSD. ulimits are set to max.
In my laptop with simple test on single instance I reach near 40k/sec , after that the avg execution times start to increase significantly, but wondering what can be the upper theoretical limit?
It depends. If your active dataset can fit in the memory - if most of the requests don't need to perform any disk I/O - then you can achieve 24k+ requests pretty easily. If not on a (bigger) single machine, then at least use a replica set cluster with multiple secondaries.
If an active dataset is much larger than the available RAM then you have the same problem as with any other database. The advantage of MongoDB's new engine WiredTiger (since v3.0) is a transparent compression - it can reduce the amount of data and I/O and thus improve performance - even despite the fact that compression adds CPU load.
For more performance it really helps:
if the most accessed documents are small so it takes less time to
load them, transfer them, and less time to deserialize in your app List item
If you use projections in find(), for the same reasons
if you use bulk operations to reduce networking I/O and context switches
Even MongoDB itself has an option to limit the maximum number of incoming connections. It defaults to 64k.
for more information you can refer link
We have an M10 cluster and the official page states that we get a max of 100 IOPS.
I cant run mongoperf on the cluster as we have direct mongo shell and compass access and mongoperf needs to be run on the instance that has MongoDB installed.
Is there any way to test the maximum requests per second that this cluster can handle and if not, is there any rough estimate available as to how many read/write operations it can handle concurrently?
PS:- Assume the queries being run aren't particularly complex and are only entering small sets of data such as Name, Email Address, Age, etc.
Thanks in advance!
Is there any way to test the maximum requests per second that this cluster can handle and if not, is there any rough estimate available as to how many read/write operations it can handle concurrently?
The answer to this, really depends on a lot of variables. i.e. document sizes, indexes utilisation, network latency between application and servers, etc.
To provide a rough estimation however, assuming your MongoDB cluster is hosted on AWS (GCP and Azure would be different), the specs would be:
M10, 2GB RAM and 10GB included storage.
In addition to this, you can select different EBS storage sizes as well as provisioned IOPS to match your required performance.
See also MongoDB Atlas: FAQ
We have an M10 cluster and the official page states that we get a max of 100 IOPS.
The number of IOPS advertised is, what would be advised by the cloud provider, i.e. AWS. Not taking account the network latency and your database usage that affects the server resources i.e. CPU/RAM.
Is there any limit of concurrent queries that mongodb can run in a second?
I am trying to implement an API that runs 300 queries in each request in mongodb.
So if there are 100 client requests in a second so the number of queries becomes 100 x 300 which is resulting in high latency.
Any clue?
The only limit is the number of conections mongo can have in parallel. Check out net.maxIncomingConnections
The maximum number of simultaneous connections that mongos or mongod will accept. This setting has no effect if it is higher than
your operating system’s configured maximum connection tracking
threshold.
But still you'll need to monitor your system to figure out what's actually happening.
And yes, 300 queries per API request is a bit too much even from networking perspective/requests parsing overhead.
I followed this tutorial and there is configuration connections per host.
What is this?
connectionsPerHost are the amount of physical connections a single Mongo client instance (it's singleton so you usually have one per application) can establish to a mongod/mongos process. At time of writing the java driver will establish this amount of connections eventually even if the actual query throughput is low (in order words you will see the "conn" statistic in mongostat rise until it hits this number per app server).
There is no need to set this higher than 100 in most cases but this setting is one of those "test it and see" things. Do note that you will have to make sure you set this low enough so that the total amount of connections to your server do not exceed
Found here How to configure MongoDB Java driver MongoOptions for production use?
I am using pecl mongo 1.4.x driver(http://pecl.php.net/package/mongo/1.4.1), with the setup mentioned in the title on a moderate traffic services (5K - 10K request per min).
And I've found that, in mongoDB the Auth command is taking a large chunck of traffic, and connection request rate is like 30-50 per second.
This impacts the performance seriously(lock ratio up, memory management doesn't cops nicely)
And if I do netstat in a box(which I have 5-8 boxes in total), I see 2-3K mongo connections in total (some in WAIT some in ESTABLISHED) per box.
My question is how can I reduce the # of connection to mongoDB, how to setup persistent connection properly?
It seems the way of persistent connection working in PECL mongoDB Driver has been changing since 1.2 then 1.3 and it performs slightly differently in 1.4.
Here is the way I invoke the client with the driver:
$mongo = new MongoClient(
"host1:11004,host2:11004", array(
'replicaSet' => MG_REPLICASET,
'password'=>"superpasswd",
'username'=>"myuser",
'db'=>"mydb",
'journal' => true,
"readPreference"=> MongoClient::RP_SECONDARY_PREFERRED
)
);
In the 1.4 version all connections are persistent, unless you close them yourself - which you should never do. You will see a connection per IP/username/password/database combination from each PHP processing unit. In your case, per each PHPFPM process. In order to reduce the number of connections, you need to have less username/password/database combinations. However, with 8 boxes, and 50 FPM processes and 3 nodes in your replicaset you're already at 1200 connections - without even taken into account database/username/password differences. There is not much that you can do about that, but it shouldn't have a big influence in performance. You are much more likely to hit RAM/slow disk limitations.
I guess I found a solution to avoid exceessive mongo connection request.
We need to set PHP_FCGI_MAX_REQUESTS (or pm. max_requests in php-fpm) to a larger number, so the process won't recycle too often.
Also I need to make sure pm.request_terminate_timeout is not too small, so workers won't be killed too often.