I need to health check backend servers with HEAD request by looking for /serverup.html file and expect 200 status
I tried using "option httpchk HEAD /serverup" and "http-check expect status 200"
frontend test1
mode http
bind 192.168.100.1:80
default_backend test1
backend test1
description Test Webpage
balance leastconn
mode http
option httpchk /serverup
http-check expect status 200
server webserver1 192.168.100.101:80 check inter 3s fall 3 rise 5 downinter 1m
server webserver2 192.168.100.102:80 check inter 3s fall 3 rise 5 downinter 1m
server webserver3 192.168.100.103:80 check inter 3s fall 3 rise 5 downinter 1m
server webserver4 192.168.100.104:80 check inter 3s fall 3 rise 5 downinter 1m
some servers requires also the host header.
can you try the following line for the check
option httpchk HEAD /serverup\n\rHost: your-backend-http-hostname\n\r\n\r
Related
I would like to print some log messages from the external check script of HAPROXY to rsyslog.
For now, I use “echo” in my external-check.sh script but it does not show up the echo messages. It only shows the haproxy log messages.
Is that possible?
The content of haproxy.cfg:
global
log 127.0.0.1 local0
external-check
insecure-fork-wanted
defaults
mode http
log global
option httplog
timeout queue 1m
timeout connect 10m
timeout client 1m
timeout server 10m
timeout http-keep-alive 10s
timeout check 10m
timeout tunnel 10m
maxconn 2048
frontend pa
bind *:443
use_backend back-servers
backend back-servers
option external-check
option log-health-checks
external-check command /etc/haproxy/external-check.sh
server PA-A xxx check inter 30s fall 6 rise 1 ssl verify none
server PA-B xxx backup check inter 30s fall 6 rise 1 ssl verify none
the content of /etc/rsyslog.d/haproxy.conf
$ModLoad imudp
$UDPServerRun 514
$template Haproxy,"%msg%\n"
local0.* -/var/log/haproxy.log
example of log output:
Health check for server back-servers/PA-A failed, reason: External check timeout, code: 1, check duration: 30009ms, status: 0/1 DOWN.
Health check for server back-servers/PA-A failed, reason: External check error, code: 1, check duration: 738ms, status: 0/1 DOWN.
Health check for backup server back-servers/PA-B failed, reason: External check timeout, code: 1, check duration: 30022ms, status: 0/1 DOWN.
Health check for backup server back-servers/PA-B failed, reason: External check error, code: 1, check duration: 1590ms, status: 0/1 DOWN
As you can see there are no printed messages of my "echo"(s) that I have added to the external check script
I got the answer:
Currently, I am using echo but I should use logger to log messages from the external script to the rsyslog socket via 127.0.0.1. By default, HAPROXY does not do it for us. It only redirects the log messages of the haproxy.cfg events, but not the external script messages.
The trick is to replace all echo messages by:
logger -p local0.info -t external-script -n 127.0.0.1 "My message"
I am getting occasional layer 7 health check failures. This happens on production machine seemingly at random, maybe once a minute or every few minutes on average. Here is the configuration:
backend api
mode http
option httpchk GET /api/v1/status HTTP/1.0
http-check expect status 200
balance roundrobin
server api1 127.0.0.1:8001 check fall 3 rise 2
server api2 127.0.0.1:8002 check fall 3 rise 2
The HAproxy log tells me the following:
Health check for server api/api2 failed, reason: Layer7 timeout, check duration: 10001ms, status: 2/3 UP.
Strange thing is when I run a script to fetch the same URL at a much faster pace than HAproxy, it never fails to return 200 response. It never hangs like it seems to do for HAproxy.
In addition, I'm getting occasional HAProxy error for various API calls, not just health checks, all looking quite similar:
https-in~ api/api1 45/0/0/-1/30045 504 194 - - sHVN 50/49/13/10/0 0/0 "POST /api/v1/accounts HTTP/1.1"
What could be the issue here? This one really got me stumped.
Is it possible to perform HTTP healthcheck on individual backend servers and load balance among active backend servers.
This works fine with single backend server when we mention host name in httpchk (option httpchk GET /info HTTP/1.1 Host:\ abc.mysrv1.com)
However i am unable to successfully configure it with multiple backend servers as i can't give individual host name on httpchk command and with below backend configuration getting http 400 error -
backend svc.op
balance roundrobin
mode http
http-send-name-header Host
option httpchk GET /info HTTP/1.1
http-check send-state
http-request set-uri /test/abcService
server abc.mysrv1.com abc.mysrv1.com:80 check
server abc.mysrv2.com abc.mysrv2.com:80 check
[WARNING] 115/162226 (4788) : Server svc.op/abc.mysrv1.com is DOWN, reason: Layer7 wrong status, code: 400, info: "Bad Request: missing required Host header", check duration: 1ms. 1 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
[WARNING] 115/162227 (4788) : Server svc.op/abc.mysrv2.com is DOWN, reason: Layer7 wrong status, code: 400, info: "Bad Request: missing required Host header", check duration: 0ms. 0 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
[ALERT] 115/162227 (4788) : backend ' svc.op' has no server available!
$ ./haproxy -version
HA-Proxy version 1.6.7 2016/07/13
UPDATE:
I didn't pursue this but someone suggested following -
backend h_a01
mode http
option httpchk GET /health HTTP/1.1\r\nHost:\ a01\r\n
http-check expect rstatus ^200
http-request set-uri /hello/world
http-request set-header Host a01
server a01 a01 check
backend h_a02
mode http
option httpchk GET /health HTTP/1.1\r\nHost:\ a02\r\n
http-check expect rstatus ^200
http-request set-uri /hello/world
http-request set-header Host a02
server a02 a02 check
backend a
mode http
option forwardfor
http-send-name-header Host
http-request set-uri /hello/world
server a1 a1 track h_a01/a01
server a2 a2 track h_a02/a02
I'm setting up HAProxy to load-balance a resource between 3 back-ends. Here is the HAProxy config : (In the following snippets I replaced the actual domain name by example.net)
global
log 127.0.0.1 local2
log-send-hostname
maxconn 2000
pidfile /var/run/haproxy.pid
stats socket /var/run/haproxy.sock mode 600 level admin
stats timeout 30s
daemon
# SSL ciphers
...
defaults
mode http
option forwardfor
option contstats
option http-server-close
option log-health-checks
option redispatch
timeout connect 5000
timeout client 10000
timeout server 10000
...
frontend front
bind *:443 ssl crt /usr/local/etc/haproxy/front.pem
reqadd X-Forwarded-Proto:\ https if { ssl_fc }
stats uri /haproxy?stats
option httpclose
option forwardfor
default_backend back
balance source
backend back
balance roundrobin
option httpchk GET /healthcheck HTTP/1.0
server server1 xxx.xxx.xxx.xxx:80 check inter 5s fall 2 rise 1
server server2 yyy.yyy.yyy.yyy:8003 check backup
server mysite example.net:80 check backup
The issue is the following: even though the first 2 servers respond correctly, the domain-based one always shows as a 404:
What is counter-intuitive to me is that if I use curl to access this same healthcheck, I get an HTTP 200 (like I would expect to see in the HAProxy stats) :
curl -I http://example.net/healthcheck
HTTP/1.1 200 OK
When I ping my site, I get:
# ping example.net
PING example.net (217.160.0.195) 56(84) bytes of data.
64 bytes from 217-160-0-195.elastic-ssl.ui-r.com (217.160.0.195): icmp_seq=1 ttl=50 time=45.7 ms
Is it because the IP of my domain is shared with other domains (1&1 shared hosting) that HAProxy can't access it? Why is that and how to make HAProxy reach it correctly?
How does haproxy deal with static file , like .css, .js, .jpeg ? When I use my configure file , my brower says :
503 Service Unavailable
No server is available to handle this request.
This my config :
global
daemon
group root
maxconn 4000
pidfile /var/run/haproxy.pid
user root
defaults
log global
option redispatch
maxconn 65535
contimeout 5000
clitimeout 50000
srvtimeout 50000
retries 3
log 127.0.0.1 local3
timeout http-request 10s
timeout queue 1m
timeout connect 10s
timeout client 1m
timeout server 1m
timeout check 10s
listen dashboard_cluster :8888
mode http
stats refresh 5s
balance roundrobin
option httpclose
option tcplog
#stats realm Haproxy \ statistic
acl url_static path_beg -i /static
acl url_static path_end -i .css .jpg .jpeg .gif .png .js
use_backend static_server if url_static
backend static_server
mode http
balance roundrobin
option httpclose
option tcplog
stats realm Haproxy \ statistic
server controller1 10.0.3.139:80 cookie controller1 check inter 2000 rise 2 fall 5
server controller2 10.0.3.113:80 cookie controller2 check inter 2000 rise 2 fall 5
Does my file wrong ? What should I do to solve this problem ? ths !
What I think is the cause:
There was no default_backend defined. 503 will be sent by HAProxy---this will appear as NOSRV in the logs.
Another Possible Cause
Based on one of my experiences, the HTTP 503 error I receive was due to my 2 bindings I have for the same IP and port x.x.x.x:80.
frontend test_fe
bind x.x.x.x:80
bind x.x.x.x:443 ssl blah
# more config here
frontend conflicting_fe
bind x.x.x.x:80
# more config here
Haproxy configuration check does not warn you about it and netstat doesn't show you 2 LISTEN entries, that's why it took a while to realize what's going on.
This can also happen if you have 2 haproxy services running. Please check the running processes and terminate the older one.
Try making the timers bigger and check that the server is reachable.
From the HAproxy docs:
It can happen from many reasons:
The status code is always 3-digit. The first digit indicates a general status :
- 1xx = informational message to be skipped (eg: 100, 101)
- 2xx = OK, content is following (eg: 200, 206)
- 3xx = OK, no content following (eg: 302, 304)
- 4xx = error caused by the client (eg: 401, 403, 404)
- 5xx = error caused by the server (eg: 500, 502, 503)
503 when no server was available to handle the request, or in response to
monitoring requests which match the "monitor fail" condition
When a server's maxconn is reached, connections are left pending in a queue
which may be server-specific or global to the backend. In order not to wait
indefinitely, a timeout is applied to requests pending in the queue. If the
timeout is reached, it is considered that the request will almost never be
served, so it is dropped and a 503 error is returned to the client.
if you see SC in the logs:
SC The server or an equipment between it and haproxy explicitly refused
the TCP connection (the proxy received a TCP RST or an ICMP message
in return). Under some circumstances, it can also be the network
stack telling the proxy that the server is unreachable (eg: no route,
or no ARP response on local network). When this happens in HTTP mode,
the status code is likely a 502 or 503 here.
Check ACLs, check timeouts... and check the logs, that's the most important...