I have the following Haproxy config:
frontend http-in
mode http
bind :80
option forwardfor
option httplog
reqadd X-Forwarded-Proto:\ http
default_backend http-routers
errorfile 502 /var/haproxy/404.http
acl is_internal_error status eq 404
rspdeny . if is_internal_error
When I hit the url for non existing domain, ex: http://test.example.com
I receive a timeout error (408).
However, when I retrieve the line errorfile 502 /var/haproxy/404.http, I can see the default error msg of 502 error code.
Can anyone tell why Haproxy cannot read my file /var/haproxy/404.http?
Sounds like permissions problem right off the bat. What user is haproxy running as? Can you post the permissions of /var/haproxy/404.http?
I think this is a duplicate of Haproxy behind ELB.
TL;DR;
Problems with erro_file newline format.
You need a ^M on HTTP INFO lines, not needed on the response body.
Dont know why, but it behaves that way. hope it helps.
Example file has those when open in vi(m). Don't copy the contents, it looks the same but isn't.
Cheers
Related
We have several (100+) clients in the field with a bug in the HTTP request. The request was previously working when directly routed to our Windows Server, but now with it fails with HAProxy v1.7 in front of it.
Here is an example request:
GET /index.aspx HTTP/1.1 \nHost: host\n\n
There is an extra space after the HTTP version before the \n.
Here is a snapshot of the relevant config.
frontend http_port_80
bind :80
mode http
reqrep (.)\ HTTP/1.1\ (.*) \1\ HTTP/1.1\2
option forwardfor
option accept-invalid-http-request
stats enable
use_backend cert_update if is_updater
use_backend getConsoleHTTP if is_getconsole
default_backend schedule_server
I have tried rewriting the request to remove the extra space and set the option accept-invalid-http-request to address the issue, but we still receive the same error.
{
type: haproxy,
timestamp: 1506545591,
termination_state: PR-,
http_status:400,
http_request:,
http_version:,
remote_addr:192.168.1.1,
bytes_read:187,
upstream_addr:-,
backend_name:http_port_80,
retries:0,
bytes_uploaded:92,
upstream_response_time:-1,
upstream_connect_time:-1,
session_duration:2382,
termination_state:PR
}
Does anyone have any ideas of how to fix the malformed request prior to haproxy rejecting it?
I may be using the wrong terms to search, but I'm having trouble finishing connections to my backend servers though HAProxy. I am able to initially login to the servers, but then the application communicates through two others ports as well. I can go directly to one of the servers and it logs in and lunches the application correctly. Through HAProxy however, I can authenticate, and then I get a communication error. I'm thinking that HAProxy is not passing through data on the other ports. How can this be achieved?
My setup:
2 HAProxy server connected to 2 identical VM servers. HAProxy seems to be working correctly as far as telling when the service on the mcahines are running. It is passing through authentication on port 8443. But that's as far as it will go. It will not launch the VM which uses ports 3000 and 5432. Any ideas on the HAProxy setup?
Here is my configuration file:
global
ssl-server-verify none
tune.ssl.default-dh-param 2048
maxconn 256
defaults
log global
mode http
option httplog
option dontlognull
retries 3
option redispatch
timeout connect 5000
timeout client 10000
timeout server 10000
errorfile 400 /etc/haproxy/errors/400.http
errorfile 403 /etc/haproxy/errors/403.http
errorfile 408 /etc/haproxy/errors/408.http
errorfile 500 /etc/haproxy/errors/500.http
errorfile 502 /etc/haproxy/errors/502.http
errorfile 503 /etc/haproxy/errors/503.http
errorfile 504 /etc/haproxy/errors/504.http
listen vm-port-3000
bind *:5432
server qvd4 10.0.0.1:3000
server qvdnode02 10.0.0.2:3000
listen vm-port-5432
bind *:5432
server qvd4 10.0.0.1:5432
server qvdnode02 10.0.0.2:5432
listen stats
bind :1936
stats enable
stats hide-version
stats realm Loadbalanced\ Servers
stats uri /haproxy?stats
stats auth haproxy:haproxy
frontend vm-initial-conn
bind *:8443 ssl crt /etc/ssl/certs/qvd/haproxy.pem
default_backend vmConn
backend vmConn
option forwardfor
option httpchk GET /qvd/ping HTTP/1.1
http-check expect status 200
balance roundrobin
http-request add-header X-Forwarded-Proto https if { ssl_fc }
http-request set-header X-Forwarded-Port %[dst_port]
server qvd4 10.0.0.1:8443 ssl verify none check
server qvdnode02 10.0.0.2:8443 ssl verify none check
In your vm-port-3000 you are actually binding to port 5432 instead of how it appears you intended to port 3000.
Thus, requests to port 5432 are randomly handled by either your vm-port-3000 or your vm-port-5432 listener while connections to port 3000 are not handled by HAProxy at all.
I have just installed HAproxy on a server which should do nothing to serve as redirection endpoint of any incoming naked domain request (http and https), to the www.-version. Just like the service wwwizer.com. This is my cfg:
global
log /dev/log local0
log /dev/log local1 notice
chroot /var/lib/haproxy
stats socket /run/haproxy/admin.sock mode 660 level admin
stats timeout 30s
user haproxy
group haproxy
daemon
# Default SSL material locations
ca-base /etc/ssl/certs
crt-base /etc/ssl/private
# Default ciphers to use on SSL-enabled listening sockets.
# For more information, see ciphers(1SSL). This list is from:
# https://hynek.me/articles/hardening-your-web-servers-ssl-ciphers/
ssl-default-bind-ciphers ECDH+AESGCM:DH+AESGCM:ECDH+AES256:DH+AES256:ECDH+AES128:DH+AES:ECDH+3DES:DH+3DES:RSA+AESGCM:RSA+AES:RSA+3DES:!aNULL:!MD5:!DSS
ssl-default-bind-options no-sslv3
defaults
log global
mode http
option httplog
option dontlognull
timeout connect 5000
timeout client 50000
timeout server 50000
errorfile 400 /etc/haproxy/errors/400.http
errorfile 403 /etc/haproxy/errors/403.http
errorfile 408 /etc/haproxy/errors/408.http
errorfile 500 /etc/haproxy/errors/500.http
errorfile 502 /etc/haproxy/errors/502.http
errorfile 503 /etc/haproxy/errors/503.http
errorfile 504 /etc/haproxy/errors/504.http
frontend http
bind *:80
bind *:443
acl has_www hdr_beg(host) -i www
http-request redirect code 301 location www.%[hdr(host)]%[req.uri] unless has_www
I have the A record of my domain DNS pointing to the IP of the server. However if I goto http://exmaple_url.com I get forwarded to: http://exmaple_url.com/www.exmaple_url.com instead of just http://www.exmaple_url.com
What am I doing wrong?
I followed the instruction in the documentation: https://www.haproxy.com/doc/aloha/7.0/haproxy/http_redirection.html
I expect adding https:// to the redirect location is the fix you need:
http-request redirect code 301 location https://www.%[hdr(host)]%[req.uri] unless has_www
Update for anyone using this answer. The spec must have changed, and I needed to use this modified http-request to work:
http-request redirect code 301 location https://www.%[hdr(host)]%[capture.req.uri] unless has_www
Per section 7.3.6. of the docs regarding req.uri:
https://www.haproxy.org/download/2.4/doc/configuration.txt
This is my experience with HA-Proxy version 2.3.9-53945bf 2021/03/30
I have HAProxy (v1.5.8) as a load balancer with keepalived (v1.2.2) to maintain high availability. I also have several Jetty servers on a different machine on the local network as backend server with a webapp.
The problem I have is: I can access to the web app through it IP (z.z.z.z in the code below), but when I try to access by the VIP(y.y.y.y in the code below) I get a connection refused error. I dont have any iptables rule set (Debian 7).
My haproxy.cfg file is this one:
global
log 127.0.0.1 local0
log 127.0.0.1 local1 notice
chroot /var/lib/haproxy
#stats socket /run/haproxy/admin.sock mode 660 level admin
#stats timeout 30s
maxconn 4096
user haproxy
group haproxy
defaults
log global
mode http
option httplog
option dontlognull
retries 3
option redispatch
maxconn 2000
timeout connect 5000
timeout client 50000
timeout server 50000
errorfile 400 /etc/haproxy/errors/400.http
errorfile 403 /etc/haproxy/errors/403.http
errorfile 408 /etc/haproxy/errors/408.http
errorfile 500 /etc/haproxy/errors/500.http
errorfile 502 /etc/haproxy/errors/502.http
errorfile 503 /etc/haproxy/errors/503.http
errorfile 504 /etc/haproxy/errors/504.http
listen stats x.x.x.x:9000
mode http
balance
option httplog
timeout client 5000
timeout connect 4000
timeout server 30000
stats uri /haproxy_stats
stats realm HAProxy/Statistics
stats auth user:passwd
stats admin if TRUE
listen webfarm y.y.y.y:8080
mode http
stats enable
stats auth someuser:somepassword
balance roundrobin
cookie JSESSIONID prefix
option httpclose
option forwardfor
option httpchk HEAD /check.txt HTTP/1.0
server webA z.z.z.z:9400 check
In the keepalived conf file I have defined the VIP I will use (Is the same one defined in the haproxy.cfg :
virtual_ipaddress {
y.y.y.y
}
In the stats web I get the next results:
HAProxy Stats
After this, I believe that haproxy cannot get to the backends as they are displayed as frontend servers. I am missing something in the conf files?
Thank you
No, 'frontend' and 'backend' are summary lines not titles. It is normal for the servers to be between the two.
Your backend is down because it is returning a 404 to haproxy. You need to make /check.txt return a valid response.
I'd recommend simply disabling your checks until you get a bit more comfortable with HAProxy as a whole. Id also recommend you turn on the stats page, which will give you a better visualization of what HAProxy sees. See below:
###########################################
#
# HAProxy Stats page
#
###########################################
listen stats
bind *:9090
mode http
maxconn 10
stats enable
stats hide-version
stats realm Haproxy\ Statistics
stats uri /
stats auth -----:-----
I want to create a configuration such that the heartbeat between haproxy and the backend is based on HTTP POST.
Does anyone have any idea about this?
I have tried the below configuration, but it only sent the http HEAD to the backend server (I want HTTP POST):
backend mlp
mode http
balance roundrobin
server mlp1 192.168.12.165:9210 check
server mlp2 192.168.12.166:9210 check
Thanks for your help.
#Mohsin,
Thank you so much. I indeed work.
But I want to specify the request message, seems my configure doesn't work. I appreciate that if you can help too.
[root#LB_vAPP_1 tmp]# more /var/www/index.txt
POST / HTTP/1.1\r\nHost: 176.16.0.8:2234\r\nContent-Length: 653\r\n\r\n<?xml version=\"1.0\" encoding=\"gb2312\"?>\r\n<svc_init ver=\"3.2.0\">\r\n<hdr ver=\"3.2.0\">\r\n<client>\r\n<id>915948</id>\r\n<pwd>915948</pwd>\r\n<serviceid></serviceid>\r\n</client>\r\n<requestor><id>13969041845</id></requestor>\r\n</hdr>\r\n<slir ver=\"3.2.0\" res_type=\"SYNC\">\r\n<msids><msid enc=\"ASC\" type=\"MSISDN\">00000000000</msid></msids>\r\n<eqop>\r\n<resp_req type=\"LOW_DELAY\"/>\r\n<hor_acc>200</hor_acc>\r\n</eqop>\r\n<geo_info>\r\n<CoordinateReferenceSystem>\r\n<Identifier
>\r\n<code>4326</code>\r\n<codeSpace>EPSG</codeSpace>\r\n<edition>6.1</edition>\r\n</Identifier\r\n</CoordinateReferenceSystem>\r\n</geo_info>\r\n<loc_type type=\"CURRENT_OR_LAST\"/>\r\n<prio type=\"HIGH\"/>\r\n</slir>\r\n</svc_init>\r\n\r\n\r\n\r\n
my haproxy.conf file is as bellowing:
#---------------------------------------------------------------------
# Example configuration for a possible web application. See the
# full configuration options online.
#
# http://haproxy.1wt.eu/download/1.4/doc/configuration.txt
#
#---------------------------------------------------------------------
#---------------------------------------------------------------------
# Global settings
#---------------------------------------------------------------------
global
# to have these messages end up in /var/log/haproxy.log you will
# need to:
#
# 1) configure syslog to accept network log events. This is done
# by adding the '-r' option to the SYSLOGD_OPTIONS in
# /etc/sysconfig/syslog
#
# 2) configure local2 events to go to the /var/log/haproxy.log
# file. A line like the following can be added to
# /etc/sysconfig/syslog
#
# local2.* /var/log/haproxy.log
#
log 127.0.0.1 local7
chroot /var/lib/haproxy
pidfile /var/run/haproxy.pid
ulimit-n 65536
daemon
nbproc 1
# turn on stats unix socket
stats socket /var/lib/haproxy/stats
defaults
mode tcp
retries 3
log global
option redispatch
# option abortonclose
retries 3
timeout queue 28s
timeout connect 28s
timeout client 28s
timeout server 28s
timeout check 1s
maxconn 32000
#---------------------------------------------------------------------
# main frontend which proxys to the backends
#---------------------------------------------------------------------
frontend mlp
mode tcp
option persist
# bind 10.68.97.42:9211 ssl crt /etc/ssl/server.pem
#bind 10.68.97.42:9211
bind 10.68.97.42:9210
default_backend mlp
frontend supl
mode tcp
option persist
bind 10.68.97.42:7275
default_backend supl
#-------------
# option1 http check
#------------
backend mlp
mode http
balance roundrobin
option httpchk POST / HTTP/1.1\r\nHost: 176.16.0.8:2234\r\nContent-Length: 653\r\n\r\n{<?xml version=\"1.0\" encoding=\"gb2312\"?>\r\n<svc_init ver=\"3.2.0\">\r\n<hdr ver=\"3.2.0\">\r\n<client>\r\n<id>915948</id>\r\n<pwd>915948</pwd>\r\n<serviceid></serviceid>\r\n</client>\r\n<requestor><id>13969041845</id></requestor>\r\n</hdr>\r\n<slir ver=\"3.2.0\" res_type=\"SYNC\">\r\n<msids><msid enc=\"ASC\" type=\"MSISDN\">00000000000</msid></msids>\r\n<eqop>\r\n<resp_req type=\"LOW_DELAY\"/>\r\n<hor_acc>200</hor_acc>\r\n</eqop>\r\n<geo_info>\r\n<CoordinateReferenceSystem>\r\n<Identifier>\r\n<code>4326</code>\r\n<codeSpace>EPSG</codeSpace>\r\n<edition>6.1</edition>\r\n</Identifier>\r\n</CoordinateReferenceSystem>\r\n</geo_info>\r\n<loc_type type=\"CURRENT_OR_LAST\"/>\r\n<prio type=\"HIGH\"/>\r\n</slir>\r\n</svc_init>\r\n\r\n\r\n\r\n}
http-check expect rstring <result resid=\"4\">UNKNOWN SUBSCRIBER</result>
server mlp1 192.168.12.165:9210 check
server mlp2 192.168.12.166:9210 check
#server mlp2 192.168.12.166:9210 check
backend supl
mode tcp
source 0.0.0.0 usesrc clientip
balance roundrobin
server supl1 192.168.12.165:7275 check
server supl2 192.168.12.166:7275 check
#server supl2 192.168.12.166:7275 check
#Mohsin,
Thanks for your answer, it gave me the critical clue to resolve this issue.
However, my message is as bellowing, right now it can work as I want(send the specified request and check the specified response). I post it, hopefully, it may help others also. One point is, the content-length is very important.
backend mlp
mode http
balance roundrobin
option httpchk POST / HTTP/1.1\r\nUser-Agent:HAProxy\r\nHost:176.16.0.8:2234\r\nContent-Type:\ text/xml\r\nContent-Length:516\r\n\r\n91594891594813969041845000000000003200
http-check expect rstring <result resid=\"4\">UNKNOWN SUBSCRIBER</result>
server mlp1 192.168.12.165:9210 check
server mlp2 192.168.12.166:9210 check
I was able to get this working after a bit of experimenting.
This was my setup
HAProxy -> NGINX -> Backend
I was sniffing the requests at the NGINX stage with tcpdump to see what was actually happening.
In order to change the health check request we have to follow a hack described in the documentation to change the HTTP version and send headers:
It is possible to send HTTP headers after the string by concatenating them using rn and backslashes spaces. This is useful to send Host headers when probing a virtual host
This is the raw http check I want to send:
POST ${ENDPOINT} HTTP/1.0
Content-Type: application/json
{"body": "json"}
The big issue here is that HAProxy adds a new header by itself: Connection: close, so this is what NGINX gets:
POST ${ENDPOINT} HTTP/1.0
Content-Type: application/json
{"body": "json"}
Connection: close
This leads, at least in my case to error 400s due to a malformed request.
The fix is to add a Content-Length header:
POST ${ENDPOINT} HTTP/1.0
Content-Type: application/json
Content-Length: 16
{"body": "json"}
Connection: close
Since the Content-Length should take precedence over the actual length, this forces the last header to be ignored. This is what NGINX passes to the backend:
POST ${ENDPOINT} HTTP/1.0
Host: ~^(.+)$
X-Real-IP: ${IP}
X-Forwarded-For: ${IP}
Connection: close
Content-Length: 16
Content-Type: application/json
{"body": "json"}
This is my final check:
option httpchk POST ${ENDPOINT} HTTP/1.0\r\nContent-Type:\ application/json\r\nContent-Length:\ 16\r\n\r\n{\"body\":\"json\"}
If it's just JSON you should be ok copying and pasting this and adjusting the content length.
However, I do recommend that you follow the same procedure and sniff the actual health checks, because, with the characters one has to escape in the config file, creating the request properly can be tricky.
Open haproxy/conf/haproxy.conf file. Goto end of the page, you will see that there is a line 'option httpchk GET /', change GET to POST and you are done.
Let me know if you face any problem.