I'm trying to think through the advantages and disadvantages of haproxy health checks happening on a different port from regular traffic.
If a server becomes overloaded having health checks on a different port may mark the server as being up even when overloaded. I think this is a good thing because taking servers offline may make an overloading problem worse, but want to confirm that that makes sense. I can't seem to find any good docs on the tradeoffs though and was wondering if someone has a good analysis on the tradeoffs.
The port keyword is often used with address to send health checks somewhere else than directly to the service you are checking. One example might be enabling option httpchk to monitor a non-HTTP service. What you then do is have a HTTP-compatible service that when queried can execute complex health checks against the service you are actually testing.
The above is often done with agent-check nowdays, but some people prefer to use an HTTP interface.
This also has nothing to do with server load, the only idea is to send health checks to some other service, not the one directly monitored, which is more capable of testing the actual service (possibly by using a more-complex logic) and returning a result. As an example, one could have a MySQL backend which instead of being tested just for authentication by option mysql-check, could be tested by a PHP script that, for example, checks if backup is running and if it is returns a 5xx HTTP error. The configuration could be something like:
backend mysql
mode tcp
option httpchk GET /mysql-status.php
server mysqlserver 10.0.0.1:3306 check port 80
Related
I am currently working on a multiplayer game that is meant to handle 20-50 player connections to a single game instance.
My current client connection model:
Client requests connection from server rest endpoint
Server creates 2 new sockets bound to random ports (1 tcp and 1 udp)
Client gets response and connects
I don't see anything glaringly wrong with this, but I am now questioning whether this is the general way that game server connections are done.
To explain further, I am in the process of learning how to use Kubernetes and Agones to deploy and manage app/game instances by wrapping them in Kubernetes pods. I am mostly working off of information found in the official guides (https://agones.dev/site/docs/getting-started/create-gameserver/) and associated github examples (https://github.com/googleforgames/agones/blob/release-1.15.0/examples).
For Agones, my understanding is that client connections are made via the port specified in "hostPort" in the "GameServer" yaml. I have previously deployed some instances with plain Kubernetes, using the "hostNetwork=true" option, which enables my above network model to work by allowing the game instance to bind directly to host ports and be exposed to the outside network. With Agones though, it seems that using this option is, at the very least, not encouraged (https://github.com/googleforgames/agones/issues/1389).
I'm certainly not an expert on networking, so please forgive my ignorance, but how are the client connections meant to be handled here if I'm only exposing one port? Is all the traffic multiplexed, or can I directly pass off connections somehow to other sockets/ports and have them automatically be exposed to the outside network?
Is all the traffic multiplexed, or can I directly pass off connections somehow to other sockets/ports and have them automatically be exposed to the outside network?
I would multiplex the traffic. It sounds like right now you are using the incoming port to determine "who is who". But you could also include that information in the packet flow to a shared port instead.
I have a web application spread over multiple servers and the incoming traffic is handled by HAProxy in order to balance the load. When we do the distribution, we do it at night because the users are much less and therefore we are less in service. To make the distribution we use the following strategy:
we shut down half of the servers
we deploy on servers that are turned off
we reactivate the servers that are turned off
we perform the same procedure on the other servers
The problem is that in any case I turn off the servers we close connections to users. Is there a better strategy for doing this? How could I improve this and avoid disservices and maybe be able to make distributions even during the day?
I hope I was clear. Thanks
I strongly suggest to use health checks for the servers.
Using HAProxy as an API Gateway, Part 3 [Health Checks]
You should have a URL ("/health") which you can use for health check of the backend server and add option redispatch to the config.
Now when you want to maintain the backend server just "remove" the "/health" URL and haproxy automagically routes the user to the other available servers.
I'm facing a configuration issue with HAProxy (1.8).
Context:
In a HAProxy config, I have a several severs in a backend, and an additional backup server in case the other servers are down.
Once a client gets an answer from a server, it must stick to this server for its next queries.
For some good reasons, I can't use a cookie for this concern, and I had to use a stick-table instead.
Problem:
When every "normal" server is down, clients are redirected to the backup server, as expected.
BUT the stick-table is then filled with an association between the client and the id of the backup server.
AND when every "normal" server is back, the clients which are present in the stick table and associated with the id of the backup server will continue to get redirected to the backup server instead of the normal ones!
This is really upsetting me...
So my question is: how to prevent HAProxy to stick clients to a backup server in a backend?
Please find below a configuration sample:
defaults
option redispatch
frontend fe_test
bind 127.0.0.1:8081
stick-table type ip size 1m expire 1h
acl acl_test hdr(host) -i whatever.domain.com
...
use_backend be_test if acl_test
...
backend be_test
mode http
balance roundrobin
stick on hdr(X-Real-IP) table fe_test
option httpchk GET /check
server test-01 server-01.lan:8080 check
server test-02 server-02.lan:8080 check
server maintenance 127.0.0.1:8085 backup
(I've already tried to add a lower weight to the backup server, but it didn't solve this issue.)
I read in the documentation that the "stick-on" keyword has some "if/unless" options, and maybe I can use it to write a condition based on the backend server names, but I have no clue about the syntax to use, or even if it is possible.
Any idea is welcome!
So silly of me! I was so obsessed by the stick table configuration that I didn't think to look in the server options...
There is a simple keyword that perfectly solves my problem: non-stick
Never add connections allocated to this sever to a stick-table. This
may be used in conjunction with backup to ensure that stick-table
persistence is disabled for backup servers.
So the last line of my configuration sample simply becomes:
server maintenance 127.0.0.1:8085 backup non-stick
...and everything is now working as I expected.
we’ve got a strange little problem we’re experiencing for months now:
The load on our cluster (http, long lasting keepalive connections with a lot of very short (<100ms) requests) is distributed very uneven.
All servers are configured the same way but some connections that push through thousands of requests per second just end up being sent to only one server.
We tried both load balancing strategies but that does not help.
It seems to be strictly keepalive related.
The misbehaving backend has the following settings:
option tcpka
option http-pretend-keepalive
Is the option http-server-close made to cover that issue?
If I get it right it will close and re-open a lot of connections which means load to the systems? Isn't there a way to keep the connections open but evenly balance the traffic anyway?
I tried to enable that option but it kills all of our backends when under load.
HAProxy currently only support keep-alive HTTP-connections toward the client, not the server. If you want to be able to inspect (and balance) each HTTP request, you currently have to use one of the following options
# enable keepalive to the client
option http-server-close
# or
# disable keepalive completely
option httpclose
The option http-pretend-keepalive doesn't change the actual behavior of HAProxy in regards of connection handling. Instead, it is intended as a workaround for servers which don't work well when they see a non-keepalive connection (as is generated by HAProxy to the backend server).
Support for keep-alive towards the backend server is scheduled to be in the final HAProxy 1.5 release. But the actual scope of that might still vary and the final release date is sometime in the future...
Just FYI, it's present in the latest release 1.5-dev20 (but take the fixes with it, as it shipped with a few regressions).
Just curious. When developing with Casini development server, one has an infinite number of ports. But, the production servers seem to give a particular importance to port 80.
Has that to do with a technical requirement, a convention, or both? I've checked the web but haven't been able to find a clear response so far.
Thanks for helping.
Many services have specifically-assigned ports This allows users to type, for example http://stackoverflow.com and get the website for SO, without needing to enter a port as well. This isn't a technical requirement; however, using a different port requires the user to know an extra piece of information, which must be entered into the URL every time.
When you connect to a server via TCP/IP you specify particular port you connect to. You do not connect to a server and hope that server guesses which port you would like to talk to.
So in most cases you tell browser to use protocol http, say "http://example.com/" then browser uses default port number assigned to that protocol (http) to connect to server "example.com". In this case port is 80. If for example you specify "https://example.com/" then browser looks for default port for https and then connects to port 443 instead.
So if you do not want to tell to every of your users to specify some non-default port for your service (say "http://example.com:60765/") you better use default one.
BTW there is a way to get port number your service listens to by it's protocol name (by asking a service's host's daemon at port 0) but this method seems to be rarely used (if at all).
See also other answers: default protocol numbers are assigned by IANA
It's a convention: you can use whatever port you feel like. You can look at the evolution of RFCs to see when the convention was official (http://www.faqs.org/rfcs/rfc1700.html)
You can see in the RFC 1060 (http://www.faqs.org/rfcs/rfc1060.html ) that it's the ISO Internet Protocol :)
In a production environment your web server is embedded in a server infrastructure (firewalls, proxies) protecting you against attacks from the internet. In such an environment port 80 is normally open for HTTP traffic. If you use this port there is no need to configure your server infrastructure.