Issue with HAProxy not retrying on retry-on - haproxy

Question

We were having issues with Apache mod_proxy getting random 502/503 errors from our backend server (that we don’t control), so we decided to give HAProxy a shot in testing. We setup HAProxy and got the same errors, so decided to try the retry-on all-retryable-errors but keep getting the same errors. We would have thought that HAProxy would have attempted retries on this but it doesn’t seem to be happening.
For testing, we do a WGET every half a second for 10000 tries. out of the 10000 tries, we get about 10 errors.
Didn’t know if someone could look at our setup and logs to help us determine why the retry isn’t occuring.
haproxy.cfg
global
log 127.0.0.1 local2 debug
chroot /var/lib/haproxy
pidfile /var/run/haproxy.pid
maxconn 4000
user haproxy
group haproxy
daemon
stats socket /var/lib/haproxy/stats
defaults
mode http
log global
option httplog
option dontlognull
option http-server-close
option redispatch
retries 3
timeout http-request 10s
timeout queue 1m
timeout connect 10s
timeout client 1m
timeout server 1m
timeout http-keep-alive 10s
timeout check 10s
maxconn 3000
frontend main
bind 127.0.0.1:5000
default_backend app
mode http
backend app
balance roundrobin
http-send-name-header Host
retry-on all-retryable-errors
retries 10
http-request disable-l7-retry if METH_POST
server srv1 backend-server:443 ssl verify none
server srv2 backend-server:443 ssl verify none
server srv3 backend-server:443 ssl verify none
haproxy.log (you see the 502 error in the middle of the log)
Apr 27 19:01:29 localhost haproxy[26058]: 127.0.0.1:58028 [27/Apr/2022:19:01:29.769] main app/srv1 0/0/5/111/116 200 1932 - - ---- 1/1/0/0/0 0/0 "GET /PBI_PBI1151/Login/RemoteInitialize/053103585 HTTP/1.1"
Apr 27 19:01:30 localhost haproxy[12306]: 127.0.0.1:58032 [27/Apr/2022:19:01:30.430] main app/srv2 0/0/2/119/121 200 1932 - - ---- 1/1/0/0/0 0/0 "GET /PBI_PBI1151/Login/RemoteInitialize/053103585 HTTP/1.1"
Apr 27 19:01:31 localhost haproxy[8726]: 127.0.0.1:58036 [27/Apr/2022:19:01:31.099] main app/srv2 0/0/6/114/120 200 1932 - - ---- 1/1/0/0/0 0/0 "GET /PBI_PBI1151/Login/RemoteInitialize/053103585 HTTP/1.1"
Apr 27 19:01:33 localhost haproxy[26058]: 127.0.0.1:58040 [27/Apr/2022:19:01:31.764] main app/srv2 0/0/6/-1/1385 502 209 - - SH-- 1/1/0/0/0 0/0 "GET /PBI_PBI1151/Login/RemoteInitialize/053103585 HTTP/1.1"
Apr 27 19:01:33 localhost haproxy[8726]: 127.0.0.1:58044 [27/Apr/2022:19:01:33.695] main app/srv3 0/0/10/112/122 200 1932 - - ---- 1/1/0/0/0 0/0 "GET /PBI_PBI1151/Login/RemoteInitialize/053103585 HTTP/1.1"
Apr 27 19:01:34 localhost haproxy[26058]: 127.0.0.1:58048 [27/Apr/2022:19:01:34.362] main app/srv3 0/0/3/113/116 200 1932 - - ---- 1/1/0/0/0 0/0 "GET /PBI_PBI1151/Login/RemoteInitialize/053103585 HTTP/1.1"
Apr 27 19:01:45 localhost haproxy[8726]: 127.0.0.1:58052 [27/Apr/2022:19:01:35.023] main app/srv1 0/0/16/10552/10568 200 1932 - - ---- 1/1/0/0/0 0/0 "GET /PBI_PBI1151/Login/RemoteInitialize/053103585 HTTP/1.1"
output from the wget:
--2022-04-27 19:01:31-- http://localhost:5000/PBI_PBI1151/Login/RemoteInitialize/053103585
Resolving localhost (localhost)... 127.0.0.1
Connecting to localhost (localhost)|127.0.0.1|:5000... connected.
HTTP request sent, awaiting response... 502 Bad Gateway
2022-04-27 19:01:33 ERROR 502: Bad Gateway.