I am using nghttp2 to implement a RESTful API server. I have defined two GET APIs:
/api/ping and /api/wait. While the response to the former is sent immediately, the server does some processing before responding to the latter. I allotted 4 threads to the server.
From a client (also implemented using nghttp2), I made a connection to the server and made the API calls one by one, /api/wait first followed by /api/ping. I observed using Wireshark that the two GET requests are sent over two different TCP packets. However, until the server completes processing of the /api/wait, it does not not process /api/ping, although it has other threads available.
I established two TCP connections from the client and made the two API calls on the different connections and the server processed those in parallel.
Does this mean that nghttp2 processes one TCP connection exclusively on one thread and requests from one TCP connection are processed sequentially by design? Is there any setting in nghttp2 to circumvent this? This may be a good feature for a web application (processing requests sequentially) but not an API server where APIs are independent of each other.
I'm building a software agent that run on a server, this software agent act as a server manager i.e. starting/stoping Docker container, monitoring etc.
This server will host/serve many services, these services are programs running in Docker container, 1 program/service per container.
There may be so many servers and these servers aren't necessary be a high performance server, they ranges from a small VM to high performance computer. Right now, I assume that every service uses HTTP to serve request.
The function that I want to implement in this software agent is tracking the number of clients that are currently connecting (requesting) to server for every service (e.x. server A is processing 500 requests) or specific program is ok (e.x. program A is processing 100 requests, program B is processing 200 request).
I want to know this number because I want to do workload balancing across servers that host the same service.
The following is ideas that I have.
Implementing load balancer/reverse proxy inside this agent (I would use this load balancer https://github.com/nwoodthorpe/Load-Balancer-Golang). This may be the last choice because I think it will use pretty much resources for load balancing.
Letting service programs that are running on server tell agent whenever they start and finish processing request. I simply implement UDP socket server in agent to listen for a datagram that tell unique ID of request (actually, can be anything that help me distinguish specific request that being processed) and status whether is being processed or finish processing.
So, I would like to ask for a suggestion for above approaches, which one is better or how should I implement it? Is there any better approach to do this?
For example we have 2 EJB stateless facades UserFacade and ContactFacade.
Client is tomcat web application which sometimes do remote calls to UserFacade and to ContactFacade.
In JBoss we have configuration like
<!-- Maximum number of connections in client invoker's -->
<!-- connection pool (socket transport). Defaults to 50. -->
<!--entry><key>clientMaxPoolSize</key> <value>20</value></entry-->
Link
Is this configuration for every EJB3 facade? For example if clientMaxPoolSize is 50, it means what it is 50 for UserFacade and 50 for ContactFacade? Or it means what it is 25 for each facade?
And is it useful in client side/application keep connection pool and do remote call via connection pool and if for each facade connection count increased over than 25(50) clear connections or do another actions.
In some client applications I used connection pool, it has some advantages and disadvantages.
Any suggestions?
Thanks
Is this configuration for every EJB3 facade? For example if clientMaxPoolSize is 50, it means what it is 50 for UserFacade and 50 for ContactFacade? Or it means what it is 25 for each facade?
Ans : It will Allow 50 for UserFacade and 50 for ContacrFacade
And is it useful in client side/application keep connection pool and do remote call via connection pool and if for each facade connection count increased over than 25(50) clear connections or do another actions.
Ans : It will go in wait mode and wait for the connections to be available.
In some client applications I used connection pool, it has some advantages and disadvantages. Any suggestions?
Ans : Connection pool is good for those who wants to serve large number of processing with lesser hardware. But make sure your configuration are according to your average application load
Happy Spring Festival - the Chinese New Year.
I'm working on server programming, and stucked in 10055 Error.
I have a TCP client application, which can simulate a huge amount of clients.
Hearing that 65534 is the maximum value of tcp client connections of one computer,
I use Asio to implement simulation client which start 50000 asynchronous tcp connects.
pseudocode:
for (int i=0: i<50000 ; ++i)
asyn_connect(...);
Development Environment is:
windows xp , x86 , 4G memory, 4 core CPU
HKLM\SYSTEM\CurrentControlSet\Services\Tcpip\Parameters\MaxUserPort=65000
The result is:
when connects come up to 17000 , 10055 Error occur.
I tried another computer ,the Error occur at 30000 connections, better but not enough good.
( the server app runs at another computer, also using Asio ).
The question is:
How to successfully start 50000 client connections at one computer?
Cou could try to do it more blockwise:
Eg. start with 10000 connections. As soon as 5000 connections were successful start the next 5000 async_connect calls. Then repeat that until you have reached your target. That would at least put less stress on the IO completion port. If it doesn't work I would try with even smaller blocks.
However depending on where the OS runs out of memory that still won't help.
Do you start asynchronous reads directly after the connect succeeds? These will also drain the memory resources.
I have a question about my haproxy config:
#---------------------------------------------------------------------
# Global settings
#---------------------------------------------------------------------
global
log 127.0.0.1 syslog emerg
maxconn 4000
quiet
user haproxy
group haproxy
daemon
#---------------------------------------------------------------------
# common defaults that all the 'listen' and 'backend' sections will
# use if not designated in their block
#---------------------------------------------------------------------
defaults
mode http
log global
option abortonclose
option dontlognull
option httpclose
option httplog
option forwardfor
option redispatch
timeout connect 10000 # default 10 second time out if a backend is not found
timeout client 300000 # 5 min timeout for client
timeout server 300000 # 5 min timeout for server
stats enable
listen http_proxy localhost:81
balance roundrobin
option httpchk GET /empty.html
server server1 myip:80 maxconn 15 check inter 10000
server server2 myip:80 maxconn 15 check inter 10000
As you can see it is straight forward, but I am a bit confused about how the maxconn properties work.
There is the global one and the maxconn on the server, in the listen block. My thinking is this: the global one manages the total number of connections that haproxy, as a service, will queue or process at one time. If the number gets above that, it either kills the connection, or pools in some linux socket? I have no idea what happens if the number gets higher than 4000.
Then you have the server maxconn property set at 15. First off, I set that at 15 because my php-fpm, this is forwarding to on a separate server, only has so many child processes it can use, so I make sure I am pooling the requests here, instead of in php-fpm. Which I think is faster.
But back on the subject, my theory about this number is each server in this block will only be sent 15 connections at a time. And then the connections will wait for an open server. If I had cookies on, the connections would wait for the CORRECT open server. But I don't.
So questions are:
What happens if the global connections get above 4000? Do they die? Or pool in Linux somehow?
Are the global connection related to the server connections, other than the fact you can't have a total number of server connections greater than global?
When figuring out the global connections, shouldn't it be the amount of connections added up in the server section, plus a certain percentage for pooling? And obviously you have other restrains on the connections, but really it is how many you want to send to the proxies?
Thank you in advance.
Willy got me an answer by email. I thought I would share it. His answers are in bold.
I have a question about my haproxy config:
#---------------------------------------------------------------------
# Global settings
#---------------------------------------------------------------------
global
log 127.0.0.1 syslog emerg
maxconn 4000
quiet
user haproxy
group haproxy
daemon
#---------------------------------------------------------------------
# common defaults that all the 'listen' and 'backend' sections will
# use if not designated in their block
#---------------------------------------------------------------------
defaults
mode http
log global
option abortonclose
option dontlognull
option httpclose
option httplog
option forwardfor
option redispatch
timeout connect 10000 # default 10 second time out if a backend is not found
timeout client 300000 # 5 min timeout for client
timeout server 300000 # 5 min timeout for server
stats enable
listen http_proxy localhost:81
balance roundrobin
option httpchk GET /empty.html
server server1 myip:80 maxconn 15 check inter 10000
server server2 myip:80 maxconn 15 check inter 10000
As you can see it is straight forward, but I am a bit confused about how the
maxconn properties work.
There is the global one and the maxconn on the server, in the listen block.
And there is also another one in the listen block which defaults to something
like 2000.
My thinking is this: the global one manages the total number of connections
that haproxy, as a service, will que or process at one time.
Correct. It's the per-process max number of concurrent connections.
If the number
gets above that, it either kills the connection, or pools in some linux
socket?
The later, it simply stops accepting new connections and they remain in the
socket queue in the kernel. The number of queuable sockets is determined
by the min of (net.core.somaxconn, net.ipv4.tcp_max_syn_backlog, and the
listen block's maxconn).
I have no idea what happens if the number gets higher than 4000.
The excess connections wait for another one to complete before being
accepted. However, as long as the kernel's queue is not saturated, the
client does not even notice this, as the connection is accepted at the
TCP level but is not processed. So the client only notices some delay
to process the request.
But in practice, the listen block's maxconn is much more important,
since by default it's smaller than the global one. The listen's maxconn
limits the number of connections per listener. In general it's wise to
configure it for the number of connections you want for the service,
and to configure the global maxconn to the max number of connections
you let the haproxy process handle. When you have only one service,
both can be set to the same value. But when you have many services,
you can easily understand it makes a huge difference, as you don't
want a single service to take all the connections and prevent the
other ones from working.
Then you have the server maxconn property set at 15. First off, I set that at
15 because my php-fpm, this is forwarding to on a separate server, only has
so many child processes it can use, so I make sure I am pooling the requests
here, instead of in php-fpm. Which I think is faster.
Yes, not only it should be faster, but it allows haproxy to find another
available server whenever possible, and also it allows it to kill the
request in the queue if the client hits "stop" before the connection is
forwarded to the server.
But back on the subject, my theory about this number is each server in this
block will only be sent 15 connections at a time. And then the connections
will wait for an open server. If I had cookies on, the connections would wait
for the CORRECT open server. But I don't.
That's exactly the principle. There is a per-proxy queue and a per-server
queue. Connections with a persistence cookie go to the server queue and
other connections go to the proxy queue. However since in your case no
cookie is configured, all connections go to the proxy queue. You can look
at the diagram doc/queuing.fig in haproxy sources if you want, it explains
how/where decisions are taken.
So questions are:
What happens if the global connections get above 4000? Do they die? Or
pool in Linux somehow?
They're queued in linux. Once you overwhelm the kernel's queue, then they're
dropped in the kernel.
Are the global connection related to the server connections, other than
the fact you can't have a total number of server connections greater than
global?
No, global and server connection settings are independant.
When figuring out the global connections, shouldn't it be the amount of
connections added up in the server section, plus a certain percentage for
pooling? And obviously you have other restrains on the connections, but
really it is how many you want to send to the proxies?
You got it right. If your server's response time is short, there is nothing
wrong with queueing thousands of connections to serve only a few at a time,
because it substantially reduces the request processing time. Practically,
establishing a connection nowadays takes about 5 microseconds on a gigabit
LAN. So it makes a lot of sense to let haproxy distribute the connections
as fast as possible from its queue to a server with a very small maxconn.
I remember one gaming site queuing more than 30000 concurrent connections
and running with a queue of 30 per server ! It was an apache server, and
apache is much faster with small numbers of connections than with large
numbers. But for this you really need a fast server, because you don't
want to have all your clients queued waiting for a connection slot because
the server is waiting for a database for instance.
Also something which works very well is to dedicate servers. If your site
has many statics, you can direct the static requests to a pool of servers
(or caches) so that you don't queue static requests on them and that the
static requests don't eat expensive connection slots.
Hoping this helps,
Willy