haproxy heartbeat with backend based on http post - haproxy

I want to create a configuration such that the heartbeat between haproxy and the backend is based on HTTP POST.
Does anyone have any idea about this?
I have tried the below configuration, but it only sent the http HEAD to the backend server (I want HTTP POST):
backend mlp
mode http
balance roundrobin
server mlp1 192.168.12.165:9210 check
server mlp2 192.168.12.166:9210 check
Thanks for your help.
#Mohsin,
Thank you so much. I indeed work.
But I want to specify the request message, seems my configure doesn't work. I appreciate that if you can help too.
[root#LB_vAPP_1 tmp]# more /var/www/index.txt
POST / HTTP/1.1\r\nHost: 176.16.0.8:2234\r\nContent-Length: 653\r\n\r\n<?xml version=\"1.0\" encoding=\"gb2312\"?>\r\n<svc_init ver=\"3.2.0\">\r\n<hdr ver=\"3.2.0\">\r\n<client>\r\n<id>915948</id>\r\n<pwd>915948</pwd>\r\n<serviceid></serviceid>\r\n</client>\r\n<requestor><id>13969041845</id></requestor>\r\n</hdr>\r\n<slir ver=\"3.2.0\" res_type=\"SYNC\">\r\n<msids><msid enc=\"ASC\" type=\"MSISDN\">00000000000</msid></msids>\r\n<eqop>\r\n<resp_req type=\"LOW_DELAY\"/>\r\n<hor_acc>200</hor_acc>\r\n</eqop>\r\n<geo_info>\r\n<CoordinateReferenceSystem>\r\n<Identifier
>\r\n<code>4326</code>\r\n<codeSpace>EPSG</codeSpace>\r\n<edition>6.1</edition>\r\n</Identifier\r\n</CoordinateReferenceSystem>\r\n</geo_info>\r\n<loc_type type=\"CURRENT_OR_LAST\"/>\r\n<prio type=\"HIGH\"/>\r\n</slir>\r\n</svc_init>\r\n\r\n\r\n\r\n
my haproxy.conf file is as bellowing:
#---------------------------------------------------------------------
# Example configuration for a possible web application. See the
# full configuration options online.
#
# http://haproxy.1wt.eu/download/1.4/doc/configuration.txt
#
#---------------------------------------------------------------------
#---------------------------------------------------------------------
# Global settings
#---------------------------------------------------------------------
global
# to have these messages end up in /var/log/haproxy.log you will
# need to:
#
# 1) configure syslog to accept network log events. This is done
# by adding the '-r' option to the SYSLOGD_OPTIONS in
# /etc/sysconfig/syslog
#
# 2) configure local2 events to go to the /var/log/haproxy.log
# file. A line like the following can be added to
# /etc/sysconfig/syslog
#
# local2.* /var/log/haproxy.log
#
log 127.0.0.1 local7
chroot /var/lib/haproxy
pidfile /var/run/haproxy.pid
ulimit-n 65536
daemon
nbproc 1
# turn on stats unix socket
stats socket /var/lib/haproxy/stats
defaults
mode tcp
retries 3
log global
option redispatch
# option abortonclose
retries 3
timeout queue 28s
timeout connect 28s
timeout client 28s
timeout server 28s
timeout check 1s
maxconn 32000
#---------------------------------------------------------------------
# main frontend which proxys to the backends
#---------------------------------------------------------------------
frontend mlp
mode tcp
option persist
# bind 10.68.97.42:9211 ssl crt /etc/ssl/server.pem
#bind 10.68.97.42:9211
bind 10.68.97.42:9210
default_backend mlp
frontend supl
mode tcp
option persist
bind 10.68.97.42:7275
default_backend supl
#-------------
# option1 http check
#------------
backend mlp
mode http
balance roundrobin
option httpchk POST / HTTP/1.1\r\nHost: 176.16.0.8:2234\r\nContent-Length: 653\r\n\r\n{<?xml version=\"1.0\" encoding=\"gb2312\"?>\r\n<svc_init ver=\"3.2.0\">\r\n<hdr ver=\"3.2.0\">\r\n<client>\r\n<id>915948</id>\r\n<pwd>915948</pwd>\r\n<serviceid></serviceid>\r\n</client>\r\n<requestor><id>13969041845</id></requestor>\r\n</hdr>\r\n<slir ver=\"3.2.0\" res_type=\"SYNC\">\r\n<msids><msid enc=\"ASC\" type=\"MSISDN\">00000000000</msid></msids>\r\n<eqop>\r\n<resp_req type=\"LOW_DELAY\"/>\r\n<hor_acc>200</hor_acc>\r\n</eqop>\r\n<geo_info>\r\n<CoordinateReferenceSystem>\r\n<Identifier>\r\n<code>4326</code>\r\n<codeSpace>EPSG</codeSpace>\r\n<edition>6.1</edition>\r\n</Identifier>\r\n</CoordinateReferenceSystem>\r\n</geo_info>\r\n<loc_type type=\"CURRENT_OR_LAST\"/>\r\n<prio type=\"HIGH\"/>\r\n</slir>\r\n</svc_init>\r\n\r\n\r\n\r\n}
http-check expect rstring <result resid=\"4\">UNKNOWN SUBSCRIBER</result>
server mlp1 192.168.12.165:9210 check
server mlp2 192.168.12.166:9210 check
#server mlp2 192.168.12.166:9210 check
backend supl
mode tcp
source 0.0.0.0 usesrc clientip
balance roundrobin
server supl1 192.168.12.165:7275 check
server supl2 192.168.12.166:7275 check
#server supl2 192.168.12.166:7275 check

#Mohsin,
Thanks for your answer, it gave me the critical clue to resolve this issue.
However, my message is as bellowing, right now it can work as I want(send the specified request and check the specified response). I post it, hopefully, it may help others also. One point is, the content-length is very important.
backend mlp
mode http
balance roundrobin
option httpchk POST / HTTP/1.1\r\nUser-Agent:HAProxy\r\nHost:176.16.0.8:2234\r\nContent-Type:\ text/xml\r\nContent-Length:516\r\n\r\n91594891594813969041845000000000003200
http-check expect rstring <result resid=\"4\">UNKNOWN SUBSCRIBER</result>
server mlp1 192.168.12.165:9210 check
server mlp2 192.168.12.166:9210 check

I was able to get this working after a bit of experimenting.
This was my setup
HAProxy -> NGINX -> Backend
I was sniffing the requests at the NGINX stage with tcpdump to see what was actually happening.
In order to change the health check request we have to follow a hack described in the documentation to change the HTTP version and send headers:
It is possible to send HTTP headers after the string by concatenating them using rn and backslashes spaces. This is useful to send Host headers when probing a virtual host
This is the raw http check I want to send:
POST ${ENDPOINT} HTTP/1.0
Content-Type: application/json
{"body": "json"}
The big issue here is that HAProxy adds a new header by itself: Connection: close, so this is what NGINX gets:
POST ${ENDPOINT} HTTP/1.0
Content-Type: application/json
{"body": "json"}
Connection: close
This leads, at least in my case to error 400s due to a malformed request.
The fix is to add a Content-Length header:
POST ${ENDPOINT} HTTP/1.0
Content-Type: application/json
Content-Length: 16
{"body": "json"}
Connection: close
Since the Content-Length should take precedence over the actual length, this forces the last header to be ignored. This is what NGINX passes to the backend:
POST ${ENDPOINT} HTTP/1.0
Host: ~^(.+)$
X-Real-IP: ${IP}
X-Forwarded-For: ${IP}
Connection: close
Content-Length: 16
Content-Type: application/json
{"body": "json"}
This is my final check:
option httpchk POST ${ENDPOINT} HTTP/1.0\r\nContent-Type:\ application/json\r\nContent-Length:\ 16\r\n\r\n{\"body\":\"json\"}
If it's just JSON you should be ok copying and pasting this and adjusting the content length.
However, I do recommend that you follow the same procedure and sniff the actual health checks, because, with the characters one has to escape in the config file, creating the request properly can be tricky.

Open haproxy/conf/haproxy.conf file. Goto end of the page, you will see that there is a line 'option httpchk GET /', change GET to POST and you are done.
Let me know if you face any problem.

Related

Disable non-backend request and non haproxy restart log in haproxy

i just want to save log for backend request and the restarting of haproxy service in haproxy,
this is the configuration of my haproxy.
[appadmin#sltxh5gvt4c rsyslog.d]$ cat /etc/rsyslog.conf
# Provides UDP syslog reception
$ModLoad imudp
$UDPServerRun 514
# Save haproxy log
local3.* /var/log/haproxylog/haproxy.log
# Save keepalived log
local2.* /var/log/keepalived/keepalived.log
++++++++++ this is haproxy.cfg
defaults
option forwardfor
log global
option httplog
log 127.0.0.1 local3
frontend case3
bind :80
mode http
log global
option httpclose
timeout client 5000
acl sabrix path_beg -i /sabrix
acl geolink path_beg -i /axis2
use_backend sabrix_servers if sabrix
use_backend us_geolink if geolink
capture request header Host len 64
capture request header User-Agent len 128
capture request header X-Forwarded-For len 100
capture request header Referer len 200
capture response header Server len 40
capture response header Server-ID len 40
log-format %ci:%cp\ %si:%sp\ %ST\ %r\ %b\ %f\ %bi\
+++++++++++++++++++++++++ haproxy.log
2020-06-14T23:33:36+08:00 localhost haproxy[27891]: 10.12.12.12:42164 -:- 400 <BADREQ> case3 case3 -
2020-06-14T23:33:42+08:00 localhost haproxy[27891]: 10.12.12.12:42204 -:- 400 <BADREQ> case3 case3 -
2020-06-14T23:33:48+08:00 localhost haproxy[27891]: 10.12.12.12:42244 -:- 400 <BADREQ> case3 case3 -
i don't want to save such logs in my haproxy.log, how to change my configuration files ?
thanks
You can't do this in haproxy, the best way to discard this log lines is with rsyslog.
:msg, contains, "BADREQ" ~
There are more information about discarding unwanted messages in this document.
https://www.rsyslog.com/discarding-unwanted-messages/
This looks similar to this question.
rsyslog filtering and forwarding
i tried to edit /etc/rsyslog.conf
# ignore BADREQ in haproxy.log
:msg, contains, "BADREQ" /var/log/haproxylog/haproxy.log
or
# ignore BADREQ in haproxy.log
:msg, contains, "BADREQ" stop
both don't work.
i check haproxy.cfg, i got the answer, we just need add
option dontlognull
to haproxy.cfg

How to redirect to domain name with https using haproxy

I tried to receive request and want to redirect it to other host using dns name and exposed with https protocol. For example, my server is http://8.8.8.8:10101/partnerA/getUser. I want haproxy redirect this to https://partner.com/partnerA/getUser (same path as the source).
I also want to filter by path for another redirect destination such as http://8.8.8.8:10101/partnerB/getMarketShare will redirected by HAProxy to https://subdomainb.differentpartner.com/partnerB/getMarketShare(notice the path also follow the same rule, but based on path it will give different host name.
I tried below haproxy.cfg
global
log /dev/log local0
log /dev/log local1 notice
chroot /var/lib/haproxy
stats socket /run/haproxy/admin.sock mode 660 level admin
stats timeout 30s
user haproxy
group haproxy
daemon
defaults
log global
mode http
option httplog
option dontlognull
timeout connect 5000
timeout client 50000
timeout server 50000
#---------------------------------------------------------------------
# main frontend which proxys to the backends
#---------------------------------------------------------------------
frontend main
bind *:10101
acl url_partnerA path_beg -i /partnerA
acl url_partnerB path_beg -i /partnerB
http-request redirect scheme https if url_partnerA
http-request redirect scheme https if url_partnerB
http-request redirect prefix https://partnerA.com if url_partnerA
http-request redirect prefix https://subdomainb.differentpartner.com/ if url_partnerA
default_backend app
#---------------------------------------------------------------------
# round robin balancing between the various backends
backend app
balance roundrobin
# server app1 127.0.0.1:11003 check
But everytime I access (I use http) POST http://8.8.8.8:10101/partnerA/getUser, the log from haproxy -f haproxy10101.cfg -d will give me this
00000000:main.accept(0005)=0009 from [8.8.8.8:48554] ALPN=<none>
00000000:main.clireq[0009:ffffffff]: POST /partnerA/getUser HTTP/1.1
00000000:main.clihdr[0009:ffffffff]: Host: 8.8.8.8:10101
00000000:main.clihdr[0009:ffffffff]: User-Agent: curl/7.47.0
00000000:main.clihdr[0009:ffffffff]: Accept: */*
00000000:main.clihdr[0009:ffffffff]: Authorization: Basic dGNhc2g6RzBqM2tmMHJsMWYzIQ==
00000000:main.clihdr[0009:ffffffff]: Content-Type: application/json
00000000:main.clihdr[0009:ffffffff]: Postman-Token: 45a236c-740a-4859-a13a-1c45195a99f2
00000000:main.clihdr[0009:ffffffff]: cache-control: no-cache
00000000:main.clihdr[0009:ffffffff]: Content-Length: 218
00000000:main.clicls[0009:ffffffff]
00000000:main.closed[0009:ffffffff]
Anything I miss to make it work? Thanks

HAProxy environment refusing to connect

I have an installation with 2 webservices behind a load balancer with HAProxy. While on service run by 3 servers responds quite fine, the other service with just one server doesn't.
So basically here's what should happen:
loadbalancer --> rancherPlatformAdministration if certain url is used
loadbalancer --> rancherServices for all other requests
Here's my haproxy.cfg:
#---------------------------------------------------------------------
# Global settings
#---------------------------------------------------------------------
global
log 127.0.0.1 local2
chroot /var/lib/haproxy
pidfile /var/run/haproxy.pid
maxconn 4000
user haproxy
group haproxy
daemon
# turn on stats unix socket
stats socket /var/lib/haproxy/stats
#---------------------------------------------------------------------
# common defaults that all the 'listen' and 'backend' sections will
# use if not designated in their block
#---------------------------------------------------------------------
defaults
mode http
log global
option httplog
option dontlognull
option http-server-close
option forwardfor except 127.0.0.0/8
option redispatch
retries 3
timeout http-request 10s
timeout queue 1m
timeout connect 10s
timeout client 1m
timeout server 1m
timeout http-keep-alive 10s
timeout check 10s
maxconn 3000
frontend http-in
bind *:80
# Define hosts
acl host_rancherAdmin hdr(host) -i admin.mydomain.tech
use_backend rancherPlatformAdministration if host_rancherAdmin
default_backend rancherServices
backend rancherServices
balance roundrobin
server rancherserver91 192.168.20.91:8080 check
server rancherserver92 192.168.20.92:8080 check
server rancherserver93 192.168.20.93:8080 check
backend rancherPlatformAdministration
server rancherapi01 192.168.20.20:8081 check
wget --server-response foo.mydomain.tech answers with a 401 which is respected behaviour as I am not providing a username nor a password. I can also open up foo.mydomain.tech with my browser an log in. So this part works as I said before.
wget --server-response 192.168.20.20:8081 (yes, this Tomcat really is running under 8081) locally from the loadbalancer responds with 200 and thus works just fine, while trying wget --server-response admin.mydomain.tech results in the following:
--2018-06-10 20:51:56-- http://admin.mydomain.tech/
Aufl"osen des Hostnamens admin.mydomain.tech (admin.mydomain.tech)... <PUBLIC IP>
Verbindungsaufbau zu admin.mydomain.tech (admin.mydomain.tech)|<PUBLIC IP>|:80 ... verbunden.
HTTP-Anforderung gesendet, auf Antwort wird gewartet ...
HTTP/1.0 503 Service Unavailable
Cache-Control: no-cache
Connection: close
Content-Type: text/html
2018-06-10 20:51:56 FEHLER 503: Service Unavailable.
I am pretty sure I am missing something here; I am aware of the differences in forwarding the request as a layer 4 or a layer 7 request – which seems to work just fine. I am providing mode http so I am on layer7...
Any hints on what's happening here or on how I can debug this?
Turns out that in my case the selinux was the showstopper – after putting it to permissive mode by setenforce 0, it just worked...
Since this change is not restart-persistent, I had to follow the instructions found here: https://www.tecmint.com/disable-selinux-temporarily-permanently-in-centos-rhel-fedora/

Why does HAProxy show that a server's check URL is a 404 when running curl on this URL is successful?

I'm setting up HAProxy to load-balance a resource between 3 back-ends. Here is the HAProxy config : (In the following snippets I replaced the actual domain name by example.net)
global
log 127.0.0.1 local2
log-send-hostname
maxconn 2000
pidfile /var/run/haproxy.pid
stats socket /var/run/haproxy.sock mode 600 level admin
stats timeout 30s
daemon
# SSL ciphers
...
defaults
mode http
option forwardfor
option contstats
option http-server-close
option log-health-checks
option redispatch
timeout connect 5000
timeout client 10000
timeout server 10000
...
frontend front
bind *:443 ssl crt /usr/local/etc/haproxy/front.pem
reqadd X-Forwarded-Proto:\ https if { ssl_fc }
stats uri /haproxy?stats
option httpclose
option forwardfor
default_backend back
balance source
backend back
balance roundrobin
option httpchk GET /healthcheck HTTP/1.0
server server1 xxx.xxx.xxx.xxx:80 check inter 5s fall 2 rise 1
server server2 yyy.yyy.yyy.yyy:8003 check backup
server mysite example.net:80 check backup
The issue is the following: even though the first 2 servers respond correctly, the domain-based one always shows as a 404:
What is counter-intuitive to me is that if I use curl to access this same healthcheck, I get an HTTP 200 (like I would expect to see in the HAProxy stats) :
curl -I http://example.net/healthcheck
HTTP/1.1 200 OK
When I ping my site, I get:
# ping example.net
PING example.net (217.160.0.195) 56(84) bytes of data.
64 bytes from 217-160-0-195.elastic-ssl.ui-r.com (217.160.0.195): icmp_seq=1 ttl=50 time=45.7 ms
Is it because the IP of my domain is shared with other domains (1&1 shared hosting) that HAProxy can't access it? Why is that and how to make HAProxy reach it correctly?

Haproxy 503 Service Unavailable . No server is available to handle this request

How does haproxy deal with static file , like .css, .js, .jpeg ? When I use my configure file , my brower says :
503 Service Unavailable
No server is available to handle this request.
This my config :
global
daemon
group root
maxconn 4000
pidfile /var/run/haproxy.pid
user root
defaults
log global
option redispatch
maxconn 65535
contimeout 5000
clitimeout 50000
srvtimeout 50000
retries 3
log 127.0.0.1 local3
timeout http-request 10s
timeout queue 1m
timeout connect 10s
timeout client 1m
timeout server 1m
timeout check 10s
listen dashboard_cluster :8888
mode http
stats refresh 5s
balance roundrobin
option httpclose
option tcplog
#stats realm Haproxy \ statistic
acl url_static path_beg -i /static
acl url_static path_end -i .css .jpg .jpeg .gif .png .js
use_backend static_server if url_static
backend static_server
mode http
balance roundrobin
option httpclose
option tcplog
stats realm Haproxy \ statistic
server controller1 10.0.3.139:80 cookie controller1 check inter 2000 rise 2 fall 5
server controller2 10.0.3.113:80 cookie controller2 check inter 2000 rise 2 fall 5
Does my file wrong ? What should I do to solve this problem ? ths !
What I think is the cause:
There was no default_backend defined. 503 will be sent by HAProxy---this will appear as NOSRV in the logs.
Another Possible Cause
Based on one of my experiences, the HTTP 503 error I receive was due to my 2 bindings I have for the same IP and port x.x.x.x:80.
frontend test_fe
bind x.x.x.x:80
bind x.x.x.x:443 ssl blah
# more config here
frontend conflicting_fe
bind x.x.x.x:80
# more config here
Haproxy configuration check does not warn you about it and netstat doesn't show you 2 LISTEN entries, that's why it took a while to realize what's going on.
This can also happen if you have 2 haproxy services running. Please check the running processes and terminate the older one.
Try making the timers bigger and check that the server is reachable.
From the HAproxy docs:
It can happen from many reasons:
The status code is always 3-digit. The first digit indicates a general status :
- 1xx = informational message to be skipped (eg: 100, 101)
- 2xx = OK, content is following (eg: 200, 206)
- 3xx = OK, no content following (eg: 302, 304)
- 4xx = error caused by the client (eg: 401, 403, 404)
- 5xx = error caused by the server (eg: 500, 502, 503)
503 when no server was available to handle the request, or in response to
monitoring requests which match the "monitor fail" condition
When a server's maxconn is reached, connections are left pending in a queue
which may be server-specific or global to the backend. In order not to wait
indefinitely, a timeout is applied to requests pending in the queue. If the
timeout is reached, it is considered that the request will almost never be
served, so it is dropped and a 503 error is returned to the client.
if you see SC in the logs:
SC The server or an equipment between it and haproxy explicitly refused
the TCP connection (the proxy received a TCP RST or an ICMP message
in return). Under some circumstances, it can also be the network
stack telling the proxy that the server is unreachable (eg: no route,
or no ARP response on local network). When this happens in HTTP mode,
the status code is likely a 502 or 503 here.
Check ACLs, check timeouts... and check the logs, that's the most important...