My loadbalancer's GUI shows that it's backend has a limit of 20.000 sessions, no matter the maxconn parameter I set in global or in the backend sections.
When running a test, the max parameter in Sessions exceeds the limit.
Any idea how to solve it or if it's even a problem?
HAProxy version: 2.2.11-1ppa1~bionic 2021/03/18
Nothing complex but slightly tricky, the number in the backend session limit is fullconn,
Adding fullconn to your backend will change that limit on the stats page just in case.
backend my-backend
fullconn 200000
However, if fullconn is not set, it sums up all the session limit values for frontends that route to this backend, divided by 10. And there is lot more in the official documentation explained.
Reference documentation
Related
Is throughput the max number of requests a server instance can handle or is it the number of requests that the server instance is currently handling?
Edit: By "currently handling" I mean, the number of requests the server is receiving for a given time interval in recent time. For eg: The server is currently handling 400 reqs every min.
For eg:, I might have a server instance with a lot of hardware which can have high throughput, but I might be only receiving small amount of traffic. What does throughput measure in such a situation. Also, what about the inverse case, i.e if my instance can only handle x requests per min. but is receiving y>>>x requests per min.
If throughput is the max no. of requests a server can handle, how is it measured? Do we do a load/stress test, where we keep increasing the requests per min on the server until it cannot handle them anymore?
No, Throughput is an aggregation that depends on execution time, you can send 1000 requests in the same second and your server won't handle, but when you'll send 1000 requests in an hour and your server will handle it normally.
Throughput is calculated as requests/unit of time. The time is calculated from the start of the first sample to the end of the last sample. This includes any intervals between samples, as it is supposed to represent the load on the server.
The formula is: Throughput = (number of requests) / (total time).
You want to find the number of concurrent users that your server can handle by increasing JMeter threads until server reach his maximum
Throughput is the number of Samplers which JMeter executes within the duration of your test. If you want to see the actual amount of requests which are being sent - consider using i.e. Server Hits Per Second listener (can be installed using JMeter Plugins Manager)
If you see that your server resources consumption doesn't increase as you increase the load in JMeter the reasons are in:
Your application middleware configuration is not suitable for high load (i.e. there is a limit of concurrent sessions and requests are queuing up instead of being processed), check out Web Performance Optimization: Top 3 Server and Client-Side Performance Tips for overall ideas what could be looked at
Your application code doesn't utilize underlying OS resources efficiently, consider using profiler tool to see what's going on under the hood.
JMeter may fail to send requests fast enough, make sure to follow JMeter Best Practices and if JMeter's machine is overloaded - consider going for Distributed Testing
I would like to be able to use leastconn in HAProxy while still having different weights on backend hosts.
Context: I have ~100 backend hosts that are being accessed by ~2000 front end hosts. All backend hosts process requests the same way (no faster hosts), however some backend hosts can process more requests (they have more cores). The problem is that I cannot use round robin as it is, because sometimes some backend host get stuck with long connections, and with round robin it will keep receiving more and more front end connections, which it never recovers from. In the current situation, I use leastconn, so all backend hosts process ~ the same number of requests, but I don't optimize their CPU usage.
What I would like to achieve is to be able to still use leastconn, but allowing more connections to certain hosts. For example if we have only 2 hosts: host A with 1 core and host B with 2 cores. At any moment, I would like HAProxy to decide which host to pick based on:
x= num_current_connections_A, y = 0.5*num_current_connections_B. If x<=y go to A, otherwise go to B.
I read this post which states the same issue, but no answer really solved my problem: http://haproxy.formilux.narkive.com/I6hSmq8H/balance-leastconn-does-not-honor-weight
Thank you
I followed this tutorial and there is configuration connections per host.
What is this?
connectionsPerHost are the amount of physical connections a single Mongo client instance (it's singleton so you usually have one per application) can establish to a mongod/mongos process. At time of writing the java driver will establish this amount of connections eventually even if the actual query throughput is low (in order words you will see the "conn" statistic in mongostat rise until it hits this number per app server).
There is no need to set this higher than 100 in most cases but this setting is one of those "test it and see" things. Do note that you will have to make sure you set this low enough so that the total amount of connections to your server do not exceed
Found here How to configure MongoDB Java driver MongoOptions for production use?
I am using httperf to benchmark web-servers. My configuration, i5 processor and 4GB RAM. How to stress this configuration to get accurate results...? I mean I have to put 100% load on this server(12.04 LTS server).
you can use httperf like this
$httperf --server --port --wsesslog=200,0,urls.log --rate 10
Here the urls.log contains the different uri/path to be requested. Check the documention for details.
Now try to change the rate value or session value, then see how many RPS you can achieve and what is the reply time. Also in mean time monitor the cpu and memory utilization using mpstat or top command to see if it is reaching 100%.
What's tricky about httperf is that it is often saturating the client first, because of 1) the per-process open files limit, 2) TCP port number limit (excluding the reserved 0-1024, there are only 64512 ports available for tcp connections, meaning only 1075 max sustained connections for 1 minute), 3) socket buffer size. You probably need to tune the above limit to avoid saturating the client.
To saturate a server with 4GB memory, you would probably need multiple physical machines. I tried 6 clients, each of which invokes 300 req/s to a 4GB VM, and it saturates it.
However, there are still other factors impacting hte result, e.g., pages deployed in your apache server, workload access patterns. But the general suggestions are:
1. test the request workload that is closest to your target scenarios.
2. add more physical clients to see if the changes of response rate, response time, error number, in order to make sure you are not saturating the clients.
I am using pecl mongo 1.4.x driver(http://pecl.php.net/package/mongo/1.4.1), with the setup mentioned in the title on a moderate traffic services (5K - 10K request per min).
And I've found that, in mongoDB the Auth command is taking a large chunck of traffic, and connection request rate is like 30-50 per second.
This impacts the performance seriously(lock ratio up, memory management doesn't cops nicely)
And if I do netstat in a box(which I have 5-8 boxes in total), I see 2-3K mongo connections in total (some in WAIT some in ESTABLISHED) per box.
My question is how can I reduce the # of connection to mongoDB, how to setup persistent connection properly?
It seems the way of persistent connection working in PECL mongoDB Driver has been changing since 1.2 then 1.3 and it performs slightly differently in 1.4.
Here is the way I invoke the client with the driver:
$mongo = new MongoClient(
"host1:11004,host2:11004", array(
'replicaSet' => MG_REPLICASET,
'password'=>"superpasswd",
'username'=>"myuser",
'db'=>"mydb",
'journal' => true,
"readPreference"=> MongoClient::RP_SECONDARY_PREFERRED
)
);
In the 1.4 version all connections are persistent, unless you close them yourself - which you should never do. You will see a connection per IP/username/password/database combination from each PHP processing unit. In your case, per each PHPFPM process. In order to reduce the number of connections, you need to have less username/password/database combinations. However, with 8 boxes, and 50 FPM processes and 3 nodes in your replicaset you're already at 1200 connections - without even taken into account database/username/password differences. There is not much that you can do about that, but it shouldn't have a big influence in performance. You are much more likely to hit RAM/slow disk limitations.
I guess I found a solution to avoid exceessive mongo connection request.
We need to set PHP_FCGI_MAX_REQUESTS (or pm. max_requests in php-fpm) to a larger number, so the process won't recycle too often.
Also I need to make sure pm.request_terminate_timeout is not too small, so workers won't be killed too often.