I was successfully using Spilo (HA PostgreSQL Cluster with Docker) in Docker Swarm behind HAProxy. I used one of the HAProxy configuration posted by one of the users.
It was working fine for HAProxy 2.1. I updated HAProxy to 2.2 and suddenly it doesn't work anymore. In the announce of HAProxy 2.2 I found that there was some changes for the Health Checks.
This is my backend section of the master that was working before:
backend backend_master
option httpchk OPTIONS /master
server dbnode1 spilo1:5432 maxconn 100 check port 8008 resolvers docker_resolver resolve-prefer ipv4
server dbnode2 spilo2:5432 maxconn 100 check port 8008 resolvers docker_resolver resolve-prefer ipv4
server dbnode3 spilo3:5432 maxconn 100 check port 8008 resolvers docker_resolver resolve-prefer ipv4
After reading HAProxy 2.2 documentation I'm not sure why the current configuration doesn't work anymore.
This is the message from the logs:
Server be-postgres-master/dbnode1 is DOWN, reason: Layer7 invalid response, info: "TCPCHK got an empty response at step 1", check duration: 5ms. 2 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
Server be-postgres-master/dbnode2 is DOWN, reason: Layer7 invalid response, info: "TCPCHK got an empty response at step 1", check duration: 4ms. 1 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
Server be-postgres-master/dbnode3 is DOWN, reason: Layer7 invalid response, info: "TCPCHK got an empty response at step 1", check duration: 4ms. 0 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
[ALERT] 235/144508 (6) : backend 'be-postgres-master' has no server available!
I downgraded HAProxy to 2.1 and it works again but how to make it work with 2.2 ?
Don't know whether you're still struggling with the issue or not, but changing the request method from OPTIONS to GET in the httpchk section helped me.
Related
I’ve been working on creating a new syslog setup and have run into an issue, that i cannot find a solution for, so i thought maybe someone here could help me out.
I have a setup with 2 syslog servers and 2 haproxy nodes(in HA with keepalived). i have 2 endpoints on configured on the haproxy nodes: “endpoint_X” and “endpoint_Y” for different types of logs. I would like to control the flow of syslog messages, so that when syslog is send to “endpoint_X”:514 its send to syslog01 and when “endpoint_Y”:514 its send to syslog02. this is normally done with the use of ACL’s for normal frontends. But for syslog I use HAproxy’s “log-forward” function, where ACL’s is not supported for.
Below is an example of my config:
ring syslog01
description " "
format rfc3164
maxlen 1200
size 357913941
server syslog01 XXXXX_01:514 source YYYYY check
timeout client 90s
timeout connect 10s
timeout server 90s
timeout check 10s
ring syslog02
description " "
format rfc3164
maxlen 1200
size 357913941
server syslog02 XXXXX_02:514 source YYYYYY check
timeout client 90s
timeout connect 10s
timeout server 90s
timeout check 10s
log-forward syslog
bind 0.0.0.0:514
bind [::]:514
dgram-bind 0.0.0.0:514
dgram-bind [::]:514
log ring#syslog01 local0
log ring#syslog02 local0
does anyone have an idea if there is something i can do to get around this issue , so i can control the data flow in log-forward, other than using differen ports? I use haproxy version 2.6
i have tried some like the following, but as stated ACL does not work with log-forward:
acl acl_endpoint_X hdr(host) -i endpoint_X
acl acl_endpoint_X hdr(host) -i endpoint_Y
log ring#syslog01 local0 if endpoint_X hdr(host)
log ring#syslog02 local0 if endpoint_Y hdr(host)
I would like to print some log messages from the external check script of HAPROXY to rsyslog.
For now, I use “echo” in my external-check.sh script but it does not show up the echo messages. It only shows the haproxy log messages.
Is that possible?
The content of haproxy.cfg:
global
log 127.0.0.1 local0
external-check
insecure-fork-wanted
defaults
mode http
log global
option httplog
timeout queue 1m
timeout connect 10m
timeout client 1m
timeout server 10m
timeout http-keep-alive 10s
timeout check 10m
timeout tunnel 10m
maxconn 2048
frontend pa
bind *:443
use_backend back-servers
backend back-servers
option external-check
option log-health-checks
external-check command /etc/haproxy/external-check.sh
server PA-A xxx check inter 30s fall 6 rise 1 ssl verify none
server PA-B xxx backup check inter 30s fall 6 rise 1 ssl verify none
the content of /etc/rsyslog.d/haproxy.conf
$ModLoad imudp
$UDPServerRun 514
$template Haproxy,"%msg%\n"
local0.* -/var/log/haproxy.log
example of log output:
Health check for server back-servers/PA-A failed, reason: External check timeout, code: 1, check duration: 30009ms, status: 0/1 DOWN.
Health check for server back-servers/PA-A failed, reason: External check error, code: 1, check duration: 738ms, status: 0/1 DOWN.
Health check for backup server back-servers/PA-B failed, reason: External check timeout, code: 1, check duration: 30022ms, status: 0/1 DOWN.
Health check for backup server back-servers/PA-B failed, reason: External check error, code: 1, check duration: 1590ms, status: 0/1 DOWN
As you can see there are no printed messages of my "echo"(s) that I have added to the external check script
I got the answer:
Currently, I am using echo but I should use logger to log messages from the external script to the rsyslog socket via 127.0.0.1. By default, HAPROXY does not do it for us. It only redirects the log messages of the haproxy.cfg events, but not the external script messages.
The trick is to replace all echo messages by:
logger -p local0.info -t external-script -n 127.0.0.1 "My message"
I am trying to perform a load balancing for my Node.js application which uses websockets. I need haproxy to stop load balance new connections on a server, which has reached its maximum number of connections, keeping the existing ones intact in the same time.
I do this by performing an agent-check for each of my servers. If a server cannot accept new connections it responds with "drain" to an agent-check. If a server is able to respond to a new connection, it responds with "ready" to an agent-check.
Here is my haproxy.cfg configuration file:
global
daemon
maxconn 240000
log /dev/log local0 debug
log /dev/log local1 notice
tune.ssl.default-dh-param 2048
defaults
mode http
log global
option httplog
option dontlognull
option dontlog-normal
option http-server-close
option redispatch
timeout connect 20000ms
timeout http-request 1m
timeout client 2100000ms
timeout server 2100000ms
timeout queue 30s
timeout check 5s
timeout http-keep-alive 180s
timeout tunnel 3600s
timeout tarpit 60s
frontend stats
bind *:8084
stats enable
stats uri /stats
stats refresh 10s
stats admin if TRUE
frontend test
mode http
bind *:5000
default_backend ws
backend ws
mode http
fullconn 100000
balance roundrobin
cookie SERVERID insert indirect nocache
server 1 backend1:9999 check agent-check agent-port 8080 cookie 1 inter 500 fall 1 rise 2
And here is how I respond to haproxy agent-check in my Node.js app:
const healthCheckServer = net.createServer((c) => {
let data = '';
if (currentConn < MAX_CONN) {
data += 'ready';
} else {
data += 'drain';
}
c.write(data + '\r\n');
c.destroy();
});
healthCheckServer.listen(8080, '0.0.0.0');
When number of connections to my application is reaching its maximum, haproxy correctly changes server status to DRAIN (agent) (I can observe this in haproxy web dashboard). The problem is, that new connections are still accepted by the application.
I am new to haproxy, so can someone point me to where I am wrong?
Found out that when server is drained by agent (status is set to DRAIN (agent)) and if server is the only one existing in the backend it will still accept new connections.
When there are multiple servers present and each server is drained the behavior is just like expected: haproxy returns HTTP 503.
UPDATE:
Turned out that I was looking in the wrong direction all the time.
First I had to mark my backend which processes WebSocket connections as a non-http (remove mode http line). My guess is haproxy incorrectly counts current sessions for http backend when using WebSockets. Removing mode http solved my problem.
Second, returning maxconn:<conn> in agent-check looks like much simple and more idiomatic way to limit the number of concurrent connections.
Sources:
https://cbonte.github.io/haproxy-dconv/1.8/configuration.html#5.2-agent-check
https://www.haproxy.com/blog/websockets-load-balancing-with-haproxy/
I hope this will help someone.
I'm using Fiddler v4.6.20171.26113 on Windows 8.1. I have enabled the Hide CONNECTs option under the Rules menu and even tried put this script in the custom rules file:
if (oSession.HTTPMethodIs("CONNECT"))
{
oSession["ui-hide"] = "true";
}
However the CONNECT requests are still shown even when their flag UI-HIDE: true is set.
SESSION STATE: Done.
Response Entity Size: 0 bytes.
== FLAGS ==================
BitFlags: [ResponseGeneratedByFiddler, IsDecryptingTunnel, ProtocolViolationInRequest, RequestBodyDropped] 0x10a100
HTTPS-CLIENT-SESSIONID: empty
HTTPS-CLIENT-SNIHOSTNAME: mtalk.google.com
LOG-DROP-REQUEST-BODY: yes
LOG-DROP-RESPONSE-BODY: yes
UI-BACKCOLOR: LightYellow
UI-HIDE: true
X-CLIENTIP: ::ffff:***.***.**.**
X-CLIENTPORT: 5033
X-EGRESSPORT: 55428
X-HOSTIP: **.***.***.***
X-HTTPPROTOCOL-VIOLATION: [ProtocolViolation] HTTP/1.1 Request was missing the required HOST header.
X-ORIGINAL-HOST:
X-REQUESTBODYFINALLENGTH: 1,384
X-RESPONSEBODYTRANSFERLENGTH: 0
== TIMING INFO ============
ClientConnected: 07:05:03.136
ClientBeginRequest: 07:05:03.339
GotRequestHeaders: 07:05:03.339
ClientDoneRequest: 07:05:03.339
Determine Gateway: 0ms
DNS Lookup: 0ms
TCP/IP Connect: 61ms
HTTPS Handshake: 215ms
ServerConnected: 07:05:03.777
FiddlerBeginRequest: 07:05:03.777
ServerGotRequest: 07:05:03.777
ServerBeginResponse: 00:00:00.000
GotResponseHeaders: 00:00:00.000
ServerDoneResponse: 00:00:00.000
ClientBeginResponse: 07:05:03.777
ClientDoneResponse: 07:05:03.777
Overall Elapsed: 0:00:00.437
The response was buffered before delivery to the client.
== WININET CACHE INFO ============
This URL is not present in the WinINET cache. [Code: 2]
* Note: Data above shows WinINET's current cache state, not the state at the time of the request.
* Note: Data above shows WinINET's Medium Integrity (non-Protected Mode) cache only.
So what should I do now?
How does haproxy deal with static file , like .css, .js, .jpeg ? When I use my configure file , my brower says :
503 Service Unavailable
No server is available to handle this request.
This my config :
global
daemon
group root
maxconn 4000
pidfile /var/run/haproxy.pid
user root
defaults
log global
option redispatch
maxconn 65535
contimeout 5000
clitimeout 50000
srvtimeout 50000
retries 3
log 127.0.0.1 local3
timeout http-request 10s
timeout queue 1m
timeout connect 10s
timeout client 1m
timeout server 1m
timeout check 10s
listen dashboard_cluster :8888
mode http
stats refresh 5s
balance roundrobin
option httpclose
option tcplog
#stats realm Haproxy \ statistic
acl url_static path_beg -i /static
acl url_static path_end -i .css .jpg .jpeg .gif .png .js
use_backend static_server if url_static
backend static_server
mode http
balance roundrobin
option httpclose
option tcplog
stats realm Haproxy \ statistic
server controller1 10.0.3.139:80 cookie controller1 check inter 2000 rise 2 fall 5
server controller2 10.0.3.113:80 cookie controller2 check inter 2000 rise 2 fall 5
Does my file wrong ? What should I do to solve this problem ? ths !
What I think is the cause:
There was no default_backend defined. 503 will be sent by HAProxy---this will appear as NOSRV in the logs.
Another Possible Cause
Based on one of my experiences, the HTTP 503 error I receive was due to my 2 bindings I have for the same IP and port x.x.x.x:80.
frontend test_fe
bind x.x.x.x:80
bind x.x.x.x:443 ssl blah
# more config here
frontend conflicting_fe
bind x.x.x.x:80
# more config here
Haproxy configuration check does not warn you about it and netstat doesn't show you 2 LISTEN entries, that's why it took a while to realize what's going on.
This can also happen if you have 2 haproxy services running. Please check the running processes and terminate the older one.
Try making the timers bigger and check that the server is reachable.
From the HAproxy docs:
It can happen from many reasons:
The status code is always 3-digit. The first digit indicates a general status :
- 1xx = informational message to be skipped (eg: 100, 101)
- 2xx = OK, content is following (eg: 200, 206)
- 3xx = OK, no content following (eg: 302, 304)
- 4xx = error caused by the client (eg: 401, 403, 404)
- 5xx = error caused by the server (eg: 500, 502, 503)
503 when no server was available to handle the request, or in response to
monitoring requests which match the "monitor fail" condition
When a server's maxconn is reached, connections are left pending in a queue
which may be server-specific or global to the backend. In order not to wait
indefinitely, a timeout is applied to requests pending in the queue. If the
timeout is reached, it is considered that the request will almost never be
served, so it is dropped and a 503 error is returned to the client.
if you see SC in the logs:
SC The server or an equipment between it and haproxy explicitly refused
the TCP connection (the proxy received a TCP RST or an ICMP message
in return). Under some circumstances, it can also be the network
stack telling the proxy that the server is unreachable (eg: no route,
or no ARP response on local network). When this happens in HTTP mode,
the status code is likely a 502 or 503 here.
Check ACLs, check timeouts... and check the logs, that's the most important...