Recently our servers were facing status_code: 400 errors. We added logs to check what is happening.. these are our observations:
Log added in nginx.conf file:
log_format custom '$remote_addr STATUS_CODE:$status $request_length $bytes_sent REQUEST "$request" COOKIE "$http_cookie" Hello';
access_log /var/log/nginx/access.log custom;
We are observing that for 400 errors the below log is getting set:
xxx.xxx.xxx.xxx STATUS_CODE:400 0 326 REQUEST "GET www.xyz.abc" COOKIE "-"
Can you please help in finding what could be the possibility of request_length getting set to zero.
Related
I am trying to create ACL in Haproxy to query Authorization from request header and route to backend based on AccessID. I have used map file which are populated with AccessID and backend server. I am sure that my ACL is not working and hence I am getting 503 for incoming requests. Any help is appreciated!
Config File:
frontend main
bind *:80
capture request header Authorization len 50
acl GET_calls method GET HEAD OPTIONS
acl PUT_calls method PUT
use_backend %[urlp,map_sub(/etc/haproxy/PUT_Header.map)] if PUT_calls
Map File:
# AccessID backend server
JMYQ get_s1
P2BH get_s1
WEA1 get_s2
I have captured the request header in log and I see AccessID.
Apr 8 10:10:29 localhost haproxy[79517]: 0.11.4.1:929 [08/Apr/2022:10:10:29.232] main main/<NOSRV> -1/-1/-1/-1/0 503 212 - - SC-- 0/0/0/0/0 0/0 {Credential=WEA1} "PUT /common/Demo2.file HTTP/1.1"
I am facing following situations when configuring haproxy with node/express application. I am trying to
achieve following.
(https) (http)
browser ======> haproxy =====> node application
When loading the node application through the browser I am getting http 504 gateway time-out error.
Below is my haproxy configurtions.
haproxy configurations
Following are the haproxy logs.
vm-2 haproxy[21255]: 127.0.0.1:45948 [23/Dec/2019:10:57:51.411] https-in~ servers/server1 0/0/0/-1/100001 504 194 - - sH-- 1/1/0/0/0 0/0 "GET / HTTP/1.1"
vm-2 haproxy[21255]: 127.0.0.1:45948 [23/Dec/2019:10:57:51.411] https-in~ servers/server1 0/0/0/-1/100001 504 194 - - sH-- 1/1/0/0/0 0/0 "GET / HTTP/1.1"
vm-2 haproxy[21255]: 127.0.0.1:46122 [23/Dec/2019:10:59:31.435] https-in~ servers/server1 0/0/0/-1/100002 504 194 - - sH-- 1/1/0/0/0 0/0 "GET /favicon.ico HTTP/1.1"
Any help would be appreciated.
You're haproxy logs indicate that it's taking over 100 seconds (ie 100001/100002) for the request to complete and that it's being aborted (ie -1) before your backend server can send the full response.
If you're looking for a strictly haproxy solution (ie. you can't/won't tune your application) then you would need to play with haproxy timeout settings.
We faced the same problem, the client requests to the server were 504s sent by the HAProxy. We found out that the defaults configurations in the haproxy.cfg file had the timeout server property that defined the 504 response (setting it to a lower value, 1s in our case, would automatically result in a 504). Increasing that value is a way to have a longer connection between the proxy and the backend.
Good day everyone!
I’m migrated from haproxy 1.5 to 1.7.11 and I have some troubles with logging
I have a following in config file for logging
capture request header Host len 200
capture request header Referer len 200
capture request header User-Agent len 200
capture request header Content-Type len 200
capture request header Cookie len 300
log-format %[capture.req.hdr(0),lower]\ %ci\ -\ [%t]\ \"%HM\ %HP\ %HV\"\ %ST\ \"%[capture.req.hdr(3)]\"\ %U\ \"%[capture.req.hdr(1)]\"\ \"%[capture.req.hdr(2)]\"\ \"%[capture.req.hdr(4)]\"\ %Tq\ \"%s\"\ 'NGINX-CACHE-- "-"'\ \"%ts\»
Logformat is almost the same with Nginx
But is some cases it works incorrectly
For example log output
Nov 20 10:41:56 lb.loc haproxy[12633]: example.com 81.4.227.173 - [20/Nov/2019:10:41:56.095] "GET /piwik.php H" 200 "-" 2396 "https://example.com/" "Mozilla/5.0" "some.cookie data" 19 "vm06.lb.loc" NGINX-CACHE-- "-" "—"
Problem is that "GET /piwik.php H" must be "GET /piwik.php HTTP/1.1"
its %HV parameter in log-format
A part of "HTTP/1.1" randomly cut’s off. It may be "HT" or "HTT" or "HTTP/1."
I think we have discussed this on the HAProxy mailing list.
https://www.mail-archive.com/haproxy#formilux.org/msg35426.html
There are some bug fixes in the buffer handling therefore please try to update to the latest 1.7.
As you mentioned on the HAProxy list that you use CentOS 6 and you use the packages from ius repo please install 1.7.12 which is listed on the page below.
https://repo.ius.io/6/x86_64/packages/h/
As described in documentation:
req.hdr(): [...] The function considers any comma as a delimiter for distinct values. If full-line headers are desired instead, use req.fhdr(). [...]
So, you should use req.fhdr() to have the full header value.
For example, like this:
http-request capture req.fhdr(User-Agent) len 256k
Information from issue thread in official repository.
When using HAProxy, I've been getting the error 413: Request Entity Too Large
This error occurs when I'm trying to upload some files which are too large, however I can not find any documentation on how to increase this limit.
How can you increase the maximum upload limit to a specified amount of MB's?
This is not a HAProxy error, as you can see here http://cbonte.github.io/haproxy-dconv/configuration-1.7#1.3.1, 413 Error is not in the list.
So this probably an error returned from the server and HAProxy is just "forwarding" the error to the client.
To be 100% sure, you can see the logs:
An error returned by HAProxy:
127.0.0.1:35494 fe_main be_app/websrv1 0/0/-1/-1/3002 503 212 - - SC-- 0/0/0/0/3 0/0 "GET /test HTTP/1.1"
An error returned by the backend server:
127.0.0.1:39055 fe_main be_app/websrv2 0/0/0/0/0 404 324 - - --NI 1/1/0/1/0 0/0 "GET /test HTTP/1.1"
Notice the "-1" in the timers.
"upstream sent too big header while reading response header from upstream"
I keep getting this when I try and do an authentication from facebook. I've increased my buffers:
proxy_buffer_size 256k;
proxy_buffers 8 256k;
proxy_busy_buffers_size 512k;
fastcgi_buffers 8 256k;
fastcgi_buffer_size 128k;
But it doesn't seem to help. Any thoughts as to why this might occur?
nginx.conf file:
user www-data;
worker_processes 1;
error_log /var/log/nginx/error.log;
pid /var/run/nginx.pid;
events {
worker_connections 1024;
# multi_accept on;
}
http {
include /etc/nginx/mime.types;
proxy_buffer_size 256k;
proxy_buffers 8 256k;
proxy_busy_buffers_size 512k;
fastcgi_buffers 8 256k;
fastcgi_buffer_size 128k;
access_log /var/log/nginx/access.log;
sendfile on;
#tcp_nopush on;
#keepalive_timeout 0;
keepalive_timeout 65;
tcp_nodelay on;
gzip on;
gzip_disable "MSIE [1-6]\.(?!.*SV1)";
include /etc/nginx/conf.d/*.conf;
include /etc/nginx/sites-enabled/*;
}
/etc/nginx/sites-enabled/default
server {
listen 80 default;
server_name localhost;
access_log /var/log/nginx/localhost.access.log;
location / {
root /var/www/nginx-default;
index index.html index.htm;
}
location /doc {
root /usr/share;
autoindex on;
allow 127.0.0.1;
deny all;
}
location /images {
root /usr/share;
autoindex on;
}
}
In codeigniter I had the same error. This works for me:
http://forum.nginx.org/read.php?2,192785,196003#msg-196003
In .conf
location ~* \.php$ {
fastcgi_pass 127.0.0.1:9001;
fastcgi_index index.php;
fastcgi_split_path_info ^(.+\.php)(.*)$;
include fastcgi_params;
fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name;
# 16-sept-2012 parametros para evitar el 502
fastcgi_temp_file_write_size 10m;
fastcgi_busy_buffers_size 512k;
fastcgi_buffer_size 512k;
fastcgi_buffers 16 512k;
fastcgi_connect_timeout 300;
fastcgi_send_timeout 300;
fastcgi_read_timeout 300;
fastcgi_intercept_errors on;
fastcgi_next_upstream error invalid_header timeout http_500;
}
I had the same exact issue this morning. However, increasing buffer size worked for me. This is the settings that I used:
proxy_buffer_size 128k;
proxy_buffers 4 256k;
proxy_busy_buffers_size 256k;
proxy_temp_file_write_size 256k;
The only setting I don't see in your config is
proxy_temp_file_write_size 256k;
Also, I added these values just for that vhost. I don't think it should matter, but might be worth trying.
Turns out Codeigniter sets its own max size. I haven't figured out how to limit that, but changing nginx won't change anything unfortunately. Thanks for all the help VBart and gsharma.
We are moving our production environment and the old one works without problem and had the same "upstream sent too big header while reading response header from upstream" problem. This is a Codeigniter 2.x application.
Like #gsharma said, after change the server config with this the error log disappeared.
fastcgi_buffers 256 4k;
fastcgi_buffer_size 8k;
However, still had some problems: login did'n work anymore.
The problem was around $config['sess_encrypt_cookie']=TRUE;
When using sess_encrypt_cookie, Codeigniter tries to use mcrypt library but if it doesn't exist uses a method called '_xor_encode'. Ok, I think this method it's buggy.
After install php-mcrypt everything worked without problems.
(sorry for my english)
I am getting this error, on a page that is 800 bytes long, 4 headers. It was a signout page to delete cookies. To expire cookies I was setting them back to my birthday. This did not work in nginx, they must be expired by less than a month to pass validation to remove the cookies.
I ran a check on a few more different, but invalid headers and got the same result. If nginx cannot validate the header it throws: upstream sent too big header while reading response header from upstream
2015: more information from experience:
upstream sent too big header while reading response header from upstream is nginx's generic way of saying "I don't like what I'm seeing"
Your upstream server thread crashed
The upstream server sent an invalid header back
The Notice/Warnings sent back from STDERR broke their buffer and both it and STDOUT were closed
3: Look at the error logs above the message, is it streaming with logged lines preceding the message? PHP message: PHP Notice: Undefined index:
Example snippet from a loop my log file:
2015/11/23 10:30:02 [error] 32451#0: *580927 FastCGI sent in stderr: "PHP message: PHP Notice: Undefined index: Firstname in /srv/www/classes/data_convert.php on line 1090
PHP message: PHP Notice: Undefined index: Lastname in /srv/www/classes/data_convert.php on line 1090
... // 20 lines of same
PHP message: PHP Notice: Undefined index: Firstname in /srv/www/classes/data_convert.php on line 1090
PHP message: PHP Notice: Undefined index: Lastname in /srv/www/classes/data_convert.php on line 1090
PHP message: PHP Notice:
2015/11/23 10:30:02 [error] 32451#0: *580927 FastCGI sent in stderr: "ta_convert.php on line 1090
PHP message: PHP Notice: Undefined index: Firstname
you can see in the 3rd line (from the 20 previous errors) the buffer limit was hit, broke, and the next thread wrote in over it. Nginx then closed the connection and returned 502 to the client.
2: log all the headers sent per request, review them and make sure they conform to standards (nginx does not permit anything older than 24 hours to delete/expire a cookie, sending invalid content length because error messages were buffered before the content counted...)
examples include:
<?php
//expire cookie
setcookie ( 'bookmark', '', strtotime('2012-01-01 00:00:00') );
// nginx will refuse this header response, too far past to accept
....
?>
and this:
<?php
header('Content-type: image/jpg');
?>
<?php //a space was injected into the output above this line
header('Content-length: ' . filesize('image.jpg') );
echo file_get_contents('image.jpg');
// error! the response is now 1-byte longer than header!!
?>
1: verify, or make a script log, to ensure your thread is reaching the correct end point and not exiting before completion.