HAProxy only logging starting messages? - haproxy

I use a remote service for logs. The directive:
log logs2.papertrailapp.com:41859 local0 correctly sends my logs to the remote service. However when I view my logs I only see the startup messages:
Proxy my-service started
Proxy my-service-2 started
Proxy my-service-3 started
Why don't I see any requests? What am I doing wrong? (I've manually testing making requests and triggering 404 errors but I don't see any results)
Haproxy config below:
global
log logs2.papertrailapp.com:41859 local0
log 127.0.0.1 local0
log 127.0.0.1 local1 notice
log-send-hostname
tune.ssl.default-dh-param 2048
chroot /var/lib/haproxy
daemon
maxconn 8000
user haproxy
group haproxy
defaults
log global
mode http

Append debug to your log line for more verbosity:
log logs2.papertrailapp.com:41859 local0 debug
For more details see https://cbonte.github.io/haproxy-dconv/configuration-1.5.html#8

Related

HAProxy log-forward section doesn't run

I try to send firewall logs to syslog using UDP. Between the syslog and the firewall I have an HAProxy cluster.
I saw that the flow arrives to the HAProxy but when I activate debug mode there is no communication between the HAProxy and the syslog.
I use port 2000 with UDP protocol.
My HAproxy version is :
HAProxy version 2.5.5-1~bpo11+1 2022/03/14 - https://haproxy.org/
Status: stable branch - will stop receiving fixes around Q1 2023.
Known bugs: http://www.haproxy.org/bugs/bugs-2.5.5.html
Running on: Linux 5.10.0-12-amd64 #1 SMP Debian 5.10.103-1 (2022-03-07) x86_64
The configuration of HA proxy :
global
log stderr format iso local7
# log /dev/log local0
# log /dev/log local1 notice
# chroot /var/lib/haproxy
# stats socket /run/haproxy/admin.sock mode 660 level admin expose-fd listeners
# stats timeout 30s
# user root
# group root
# daemon
# Default SSL material locations
#ca-base /etc/ssl/certs
#crt-base /etc/ssl/private
# See: https://ssl-config.mozilla.org/#server=haproxy&server-version=2.0.3&config=intermediate
#ssl-default-bind-ciphers ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY13>
#ssl-default-bind-ciphersuites TLS_AES_128_GCM_SHA256:TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256
#ssl-default-bind-options ssl-min-ver TLSv1.2 no-tls-tickets
defaults
log global
mode http
option httplog
option dontlognull
timeout connect 5000
timeout client 50000
timeout server 50000
errorfile 400 /etc/haproxy/errors/400.http
errorfile 403 /etc/haproxy/errors/403.http
errorfile 408 /etc/haproxy/errors/408.http
errorfile 500 /etc/haproxy/errors/500.http
errorfile 502 /etc/haproxy/errors/502.http
errorfile 503 /etc/haproxy/errors/503.http
errorfile 504 /etc/haproxy/errors/504.http
log-forward sylog-loadb
# VIP of HA proxy
dgram-bind 10.10.10.1:2000
bind 10.10.10.1:2000
# IP of my syslog server
log 10.10.10.2:2000 sample 1:2 local0
log 10.10.10.3:2000 sample 2:2 local0
Please have you any idea about where the problem could be ?
Thank you.
Best regards.

HAProxy Backend Layer7 Invalid Response

I am trying to load balance two server using HAProxy v1.8 but in my case the backends are domain names instead of IP addresses.
My HAProxy config looks like this:
global
log /dev/log local0
log /dev/log local1 notice
chroot /var/lib/haproxy
pidfile /var/run/rh-haproxy18-haproxy.pid
user haproxy
group haproxy
daemon
stats socket /run/haproxy/admin.sock mode 660 level admin expose-fd listeners
spread-checks 21
# Default SSL material locations
ca-base /etc/ssl/certs
crt-base /etc/ssl/private
# Default ciphers to use on SSL-enabled listening sockets.
# For more information, see ciphers(1SSL). This list is from:
# https://hynek.me/articles/hardening-your-web-servers-ssl-ciphers/
# An alternative list with additional directives can be obtained from
# https://mozilla.github.io/server-side-tls/ssl-config-generator/?server=haproxy
ssl-default-bind-ciphers ECDH+AESGCM:DH+AESGCM:ECDH+AES256:DH+AES256:ECDH+AES128:DH+AES:RSA+AESGCM:RSA+AES:!aNULL:!MD5:!DSS
ssl-default-bind-options no-sslv3
defaults
mode http
log global
option httplog
option dontlognull
option http-server-close
option redispatch
retries 3
timeout http-request 10s
timeout queue 1m
timeout connect 10s
timeout client 1m
timeout server 1m
timeout http-keep-alive 10s
timeout check 10s
maxconn 10000
balance roundrobin
frontend https-443
bind *:443
mode http
option httplog
acl ACL_global.domain.com hdr(host) -i global.domain.com
use_backend www-443-app if ACL_global.domain.com
backend www-443-app
balance roundrobin
mode http
option httpchk GET /health
option forwardfor
http-check expect status 200
server backendnode1 app1.domain.com:443 check
server backendnode2 app2.domain.com:443 check
frontend health-443
bind *:8443
acl backend_dead nbsrv(www-443-app) lt 1
monitor-uri /haproxy_status
monitor fail if backend_dead
listen stats # Define a listen section called "stats"
bind :9000 # Listen on localhost:9000
mode http
stats enable # Enable stats page
stats hide-version # Hide HAProxy version
stats realm Haproxy\ Statistics # Title text for popup window
stats uri /haproxy_stats # Stats URI
stats auth haproxy:passwd # Authentication credentials
However, the health check is not passing. When I checked the stat page it says: Layer7 invalid response.
I checked if I can connect to the backend domains from my HAProxy server and I am successfully able to do so.
curl -X GET -I https://app1.domain.com/health
HTTP/2 200
cache-control: no-cache, private, max-age=0
content-type: application/json
expires: Thu, 01 Jan 1970 00:00:00 UTC
pragma: no-cache
x-accel-expires: 0
x-content-type-options: nosniff
x-xss-protection: 1; mode=block
date: Wed, 28 Jul 2021 12:05:09 GMT
content-length: 18
x-envoy-upstream-service-time: 0
endpoint: health
version: 1.0.0
server: istio-envoy
Is there something that I am missing in my configuration or something that I need to change to make this work?
You're missing ssl keyword for server lines. You may also want to set sni
backend foo
default-server ssl check verify none
server backendnode1 app1.domain.com:443 sni str('app1.domain.com')
server backendnode2 app2.domain.com:443 sni str('app2.domain.com')
You should also decide if you want to verify SSL certificates of your backend servers. Can you trust the connection? Is it your network? Haproxy encourages you to verify, but requires supplying CA certificate for them to verify. You can also add verifyhost and check-sni settings if you verify certificate:
backend foo
default-server ssl check verify required
server backendnode1 app1.domain.com:443 sni str('app1.domain.com') check-sni 'app1.domain.com' verifyhost 'app1.domain.com' ca-file /path/to/CA1.pem
server backendnode2 app2.domain.com:443 sni str('app2.domain.com') check-sni 'app2.domain.com' verifyhost 'app2.domain.com' ca-file /path/to/CA2.pem

Haproxy + percona 5.7 xtradb error

i configure
Hello, I configure haproxy by digitalocean manual, roundrobin for percona 5.7 bases, but on the haproxy server, when I try to connect to the database I getting error.
On the haproxy server:
mysql -h 127.0.0.1 -u haproxy_root -p -e "SHOW DATABASES"
And i get error:
ERROR 2013 (HY000): Lost connection to MySQL server at 'reading initial communication packet', system error: 2
Haproxy config:
lobal
log 127.0.0.1 local0
log 127.0.0.1 local1 notice
#log loghost local0 info
maxconn 1024
#chroot /usr/share/haproxy
user haproxy
group haproxy
daemon
#debug
#quiet
defaults
log global
mode http
option tcplog
option dontlognull
retries 3
option redispatch
maxconn 1024
timeout connect 5000ms
timeout client 50000ms
timeout server 50000ms
listen galera_cluster
bind 127.0.0.1:3306
mode tcp
option httpchk
balance leastconn
server galera-node01 192.168.0.101:3306 check port 9200
server galera-node02 192.168.0.102:3306 check port 9200
server galera-node03 192.168.0.103:3306 check port 9200
If I connect directly to the database 192.168.0.101, everything works, I get a response from the database, but when I make the request through to haproxy 127.0.0.1 I get this error:
ERROR 2013 (HY000): Lost connection to MySQL server at 'reading
initial communication packet', system error: 2
My xinetd config on mysql:
# default: on
# description: mysqlchk
service mysqlchk
{
# this is a config for xinetd, place it in /etc/xinetd.d/
disable = no
flags = REUSE
socket_type = stream
type = UNLISTED
port = 9200
wait = no
user = nobody
server = /usr/bin/clustercheck
server_args = percona percona
log_on_failure += USERID
only_from = 0.0.0.0/0
#
# Passing arguments to clustercheck
# <user> <pass> <available_when_donor=0|1> <log_file> <available_when_readonly=0|1> <defaults_extra_file>"
# Recommended: server_args = user pass 1 /var/log/log-file 0 /etc/my.cnf.local"
# Compatibility: server_args = user pass 1 /var/log/log-file 1 /etc/my.cnf.local"
# 55-to-56 upgrade: server_args = user pass 1 /var/log/log-file 0 /etc/my.cnf.extra"
#
# recommended to put the IPs that need
# to connect exclusively (security purposes)
per_source = UNLIMITED
}
If i telnet to PXC node on port 9200, i got:
telnet 192.168.0.101 9200
Trying 192.168.0.101...
Connected to 192.168.0.101.
Escape character is '^]'.
HTTP/1.1 503 Service Unavailable
Content-Type: text/plain
Connection: close
Content-Length: 57
Percona XtraDB Cluster Node is not synced or non-PRIM.
Connection closed by foreign host.
The most common reason for this is that all nodes in the cluster is down. If you have enabled your HAProxy stats, check that all nodes are up. If not, you mysqlchk service is likely not being able to connect to the cluster nodes properly.
Check your mysqlchk xinetd service should have the proper server_args configured. Once these are set, restart xinetd, and telnet to port 9200 to validate
[root#node02 log]# cat /etc/xinetd.d/mysqlchk
# default: on
# description: mysqlchk
service mysqlchk
{
# this is a config for xinetd, place it in /etc/xinetd.d/
...
server = /usr/bin/clustercheck
server_args = percona percona
...
# Passing arguments to clustercheck
# <user> <pass> <available_when_donor=0|1> <log_file> <available_when_readonly=0|1> <defaults_extra_file>"
# Recommended: server_args = user pass 1 /var/log/log-file 0 /etc/my.cnf.local"
}
UPDATE:
A more thorough procedure, including making sure mysqlchk is configured, can be found here https://www.percona.com/doc/percona-xtradb-cluster/5.6/howtos/virt_sandbox.html

HAProxy not running stats socket

I installed haproxy from aur in Arch Linux and modified the config file a bit:
global
maxconn 20000
log 127.0.0.1 local0
user haproxy
stats socket /run/haproxy/haproxy.sock mode 660 level admin
stats timeout 30s
chroot /usr/share/haproxy
pidfile /run/haproxy.pid
daemon
defaults
mode http
stats enable
stats uri /stats
stats realm Haproxy\ Statistics
frontend www-http
bind 127.0.0.1:80
default_backend www-backend
backend www-backend
mode http
balance roundrobin
timeout connect 5s
timeout server 30s
timeout queue 30s
server app1 127.0.0.1:5001 check
server app2 127.0.0.1:5002 check
I have made sure that the directory /run/haproxy exists and has permissions for the user haproxy to write to it:
ツ ls -al /run/haproxy
total 0
drwxr-xr-x 2 haproxy root 40 May 13 21:37 .
drwxr-xr-x 27 root root 720 May 13 22:00 ..
When I launch haproxy using systemctl start haproxy.service, it loads fine. I can even go to the /stats page and view stats, however, socat reports the following error:
ツ sudo socat unix-connect:/run/haproxy/haproxy.sock stdio
2016/05/13 22:04:11 socat[24202] E connect(5, AF=1 "/run/haproxy/haproxy.sock", 27): No such file or directory
I am at wits end and not able to understand what is happening. This is what I get from journalctl -xe:
May 13 21:56:31 rohanarch.local systemd[1]: Starting HAProxy Load Balancer...
May 13 21:56:31 rohanarch.local systemd[1]: Started HAProxy Load Balancer.
May 13 21:56:31 rohanarch.local haproxy-systemd-wrapper[20454]: haproxy-systemd-wrapper: executing /usr/bin/haproxy -f /etc/haproxy/haproxy.cfg -p /run/haproxy.pid -Ds
May 13 21:56:31 rohanarch.local haproxy-systemd-wrapper[20454]: [WARNING] 133/215631 (20456) : config : missing timeouts for frontend 'www-http'.
May 13 21:56:31 rohanarch.local haproxy-systemd-wrapper[20454]: | While not properly invalid, you will certainly encounter various problems
May 13 21:56:31 rohanarch.local haproxy-systemd-wrapper[20454]: | with such a configuration. To fix this, please ensure that all following
May 13 21:56:31 rohanarch.local haproxy-systemd-wrapper[20454]: | timeouts are set to a non-zero value: 'client', 'connect', 'server'.
Basically, no errors/warnings or not even so much as an indication about the stats socket. Others who have faced a problem with the stats socket fail to get haproxy started. In my case, it starts up fine, but the socket just isn't creating.
You need to manually create the directory yourself. Please ensure
/run/haproxy exists. If it doesn't, then first create it with:
sudo mkdir /run/haproxy
This should resolve your issue.
try to make selinux permissive with the command belowe and restart HAproxy service.
selinux command

HAProxy doesn't start, can not bind UNIX socket [/run/haproxy/admin.sock]

I'm trying to start haproxy (version 1.5.8 2014/10/31) with an "empty" config file and I get:
user#server:~$ sudo service haproxy start
[....] Starting haproxy: haproxy[ALERT] 126/120540 (7363) : Starting frontend GLOBAL: cannot bind UNIX socket [/run/haproxy/admin.sock]
altough it's enabled:
user#server:~$ cat /etc/default/haproxy
# Set ENABLED to 1 if you want the init script to start haproxy.
ENABLED=1
Configuration file:
global
log /dev/log local0
log /dev/log local1 notice
chroot /var/lib/haproxy
stats socket /run/haproxy/admin.sock mode 660 level admin
stats timeout 30s
user haproxy
group haproxy
daemon
# Default SSL material locations
ca-base /etc/ssl/certs
crt-base /etc/ssl/private
# Default ciphers to use on SSL-enabled listening sockets.
# For more information, see ciphers(1SSL).
ssl-default-bind-ciphers kEECDH+aRSA+AES:kRSA+AES:+AES256:RC4-SHA:!kEDH:!LOW:!EXP:!MD5:!aNULL:!eNULL
ssl-default-bind-options no-sslv3
defaults
log global
mode http
option httplog
option dontlognull
timeout connect 5000
timeout client 50000
timeout server 50000
errorfile 400 /etc/haproxy/errors/400.http
errorfile 403 /etc/haproxy/errors/403.http
errorfile 408 /etc/haproxy/errors/408.http
errorfile 500 /etc/haproxy/errors/500.http
errorfile 502 /etc/haproxy/errors/502.http
errorfile 503 /etc/haproxy/errors/503.http
errorfile 504 /etc/haproxy/errors/504.http
Does anyone have an idea why it can't start?
Haproxy needs to write to /run/haproxy/admin.sock but it wont create the directory for you. Create the directory /run/haproxy/ first or set stats socket to a different path.
I ran into this problem and had to remove the /run/haproxy/admin.sock file for HAProxy to restart successfully. I can only think it became corrupted after I aborted a yum update command. Oops! 😅
After updating pfSense from 2.4.5 to 2.5.2 I was facing this issue.
As #datacarl said, using command mkdir -p /run/haproxy/ from pfSense CLI works great.
Couple things with this. Know not the newest convo.
Anything i create in the /run folder disappears after reboot.
If I move to /var/lib/haproxy rather than /run/haproxy it starts fine manually as root.
If I reboot it fails. Not sure if it's because it's trying to use haproxy on reboot? if I su haproxy it says the account isn't available but think that's because it's set to nologin.