I have a requirement where i have to visualize data using two log files from two different sources , one is haproxy logs and other is nginx logs. I am trying to ELK stack but i am not able to parse the two different log files in a common format using logstash.
Below is the format for haproxy log:
**Feb 22 21:17:32** ap haproxy[1235]: 10.172.80.45:32071 10.31.33.34:44541 10.31.33.34:32772 13.127.229.72:443 [22/Feb/2020:21:17:32.006] this_machine~ backend_app/app_32772 40/0/5/1/**836** 200 701381 - - ---- 0/0/0/0/0 0/0 {cdn.net} {hitting} "GET **/ob/720/output00007.ts** HTTP/1.1"
Below is the Nginx Log:
183.87.179.64 - - [**24/Feb/2020:16:58:08** +0000] "GET **/felaapp/hd_images/apple.jpg** HTTP/1.1" 200 3414616 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:73.0) Gecko/20100101 Firefox/73.0" **1.049** - .
I want the highlighted content and a extra filed to identify the source of data in below common format
"Date" "Time" "Content" "Responsetime in Milliseconds" "Source"
"24-02-2020" "16:58:08" "/felaapp/hd_images/apple.jpg" "836" "Nginx"
"22-02-2020" "21:17:32" "/ob/720/output00007.ts" "1.04" "Haproxy"
I am trying to get this done through logstash using gork and mutate plugin, but i am open to csv format also, but i will be needing the data in real time so the csv has to be updated on real time basis.
Please help.
Both Nginx and Haproxy have a Filebeat module that can be used to convert the log to the ECS format. From there, you could query both the same way, and also use elastic SIEM features as a bonus.
Related
Good day everyone!
I’m migrated from haproxy 1.5 to 1.7.11 and I have some troubles with logging
I have a following in config file for logging
capture request header Host len 200
capture request header Referer len 200
capture request header User-Agent len 200
capture request header Content-Type len 200
capture request header Cookie len 300
log-format %[capture.req.hdr(0),lower]\ %ci\ -\ [%t]\ \"%HM\ %HP\ %HV\"\ %ST\ \"%[capture.req.hdr(3)]\"\ %U\ \"%[capture.req.hdr(1)]\"\ \"%[capture.req.hdr(2)]\"\ \"%[capture.req.hdr(4)]\"\ %Tq\ \"%s\"\ 'NGINX-CACHE-- "-"'\ \"%ts\»
Logformat is almost the same with Nginx
But is some cases it works incorrectly
For example log output
Nov 20 10:41:56 lb.loc haproxy[12633]: example.com 81.4.227.173 - [20/Nov/2019:10:41:56.095] "GET /piwik.php H" 200 "-" 2396 "https://example.com/" "Mozilla/5.0" "some.cookie data" 19 "vm06.lb.loc" NGINX-CACHE-- "-" "—"
Problem is that "GET /piwik.php H" must be "GET /piwik.php HTTP/1.1"
its %HV parameter in log-format
A part of "HTTP/1.1" randomly cut’s off. It may be "HT" or "HTT" or "HTTP/1."
I think we have discussed this on the HAProxy mailing list.
https://www.mail-archive.com/haproxy#formilux.org/msg35426.html
There are some bug fixes in the buffer handling therefore please try to update to the latest 1.7.
As you mentioned on the HAProxy list that you use CentOS 6 and you use the packages from ius repo please install 1.7.12 which is listed on the page below.
https://repo.ius.io/6/x86_64/packages/h/
As described in documentation:
req.hdr(): [...] The function considers any comma as a delimiter for distinct values. If full-line headers are desired instead, use req.fhdr(). [...]
So, you should use req.fhdr() to have the full header value.
For example, like this:
http-request capture req.fhdr(User-Agent) len 256k
Information from issue thread in official repository.
I've had SonarQube running for a good while but haven't used it very much but in general stuff seems to have been working. I'm running it inside Docker.
I just updated it to LTS (6.7) and after that it seems to have gone into some limbo state. I'm able to log in and browse the website but as soon as I try to perform some operation (seems to not matter what that operation is), I get redirected to the login page. If I log in again, everything repeats. So I'm unable to actually perform any action it seems.
At first I thought this had to have something to do with old data conflicting with the new setup. So I cleaned everything out and set it up from scratch. The problem remains, I'm unable to do anything and get redirected to the login page every time.
For example, after the clean setup, I log in with admin/admin and I get the "first time tutorial" where I'm offered to create a token. I tried to do that but get directed to the login page. I log in again and this time I try to skip the tutorial but then I get redirected to the login page. Below is a part of the access.log for when I try to skip the tutorial:
10.3.1.119 - - [16/Nov/2017:00:12:48 +0000] "POST /gor-sq/api/users/skip_onboarding_tutorial HTTP/1.0" 401 - "https://build.acme.com/gor-sq/projects" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/61.0.3163.100 Safari/537.36" "AV/CJhNZndR3RsZuAAA4"
10.3.1.119 - - [16/Nov/2017:00:12:48 +0000] "GET /gor-sq/api/users/identity_providers HTTP/1.0" 200 24 "https://build.acme.com/gor-sq/sessions/new?return_to=%2Fgor-sq%2Fprojects" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/61.0.3163.100 Safari/537.36" "AV/CJhNZndR3RsZuAAA5"
10.3.1.119 - - [16/Nov/2017:00:12:48 +0000] "GET /gor-sq/api/navigation/global HTTP/1.0" 200 573 "https://build.acme.com/gor-sq/sessions/new?return_to=%2Fgor-sq%2Fprojects" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/61.0.3163.100 Safari/537.36" "AV/CJhNZndR3RsZuAAA6"
The first line indicates that the POST is getting a 401 response. Without being absolutely certain, it does look like it's the POST operations that are getting 401 responses while GET works.
This setup does sit behind a reverse proxy but as I said before, the setup has been working fine before and no changes have been made to the reverse proxy setup.
Hope I am not so late. I had the same issue. What worked for me is the deleting cookies from the browser and everything else works like a charm.
I had the same issue.
https://myserver.com/sonar/api/users/skip_onboarding_tutorial
I got 401 and I was redirected to the Login page. I looked at the source code and request.ts was erroring out at line 108.
submit(): Promise<Response> {
const { url, options } = this.getSubmitData({ ...getCSRFToken() });
return window.fetch((window as any).baseUrl + url, options);}
Looked like an issue with the CSRFToken. Since I have Sonarqube running behind a Nginx Reverse Proxy, there might have been something to do with the way I was handling the cookies.
So when I looked a little bit, I found the solution here:
https://stackoverflow.com/a/47909810/3221249
Basically, they changed the way you handle secure cookies after v6.0. Since I was making the cookie secure and httponly to true(not letting the client browser interact with the js code) I was having the above issue. I was doing this even before my non-ssl traffic hits Nginx. I have another proxy server running HAProxy which was handling this so I commented that part of the definitions.
#rspirep ^(Set-cookie:.*) \1;\ Secure if ! secure
#rspirep ^(Set-cookie:.*) \1;\ httponly
I hope this helps you.
I'd like to be able to have two sets of slides, produced from two different notebooks, open at once in my browser. This use case is not supported, as far as I can tell, by the option --post serve of ipython nbconvert --to slides (of course, I'd be happy to be disproved).
My tactic has been to start a local server, as in
python -m SimpleHTTPServer 8001
and open the slide shows like this
google-chrome http://127.0.0.1:8001/my.slides.html
but now I get a bunch of messages alike
127.0.0.1 - - [31/Mar/2015 12:03:49] code 404, message File not found
127.0.0.1 - - [31/Mar/2015 12:03:49] "GET /reveal.js/css/reveal.css HTTP/1.1" 404 -
whose meaning is quite clear to me... so I did
ln -s /path/to/local/copy/of/reveal.js/ .
google-chrome http://127.0.0.1:8001/`
but now I have
127.0.0.1 - - [31/Mar/2015 12:07:29] code 404, message File not found
127.0.0.1 - - [31/Mar/2015 12:07:29] "GET /custom.css HTTP/1.1" 404 -
examining the source of my.slides.html I see the lines
<!-- Custom stylesheet, it must be in the same directory as the html file -->
<link rel="stylesheet" href="custom.css">
so I'm bound to the conclusion that --post serve does an awful lot of things at my back and that I'm out of luck in my attempt to save a standalone slide-show and have it served by a local HTTP server.
How can I have a properly served slide show without resorting to --post serve?
In my project, we have a component that includes a JavaScript file & in that file, we are making an AJAX call to a Servlet (defined in an OSGi bundle).
When the package is installed in the Publish instance (along with the OSGi bundle), I'm able to see results after I click on a link which is bound to the AJAX call.
When accessing the same page through the Dispatcher however, the page is getting displayed but the link which should show content from the OSGi bundle is not working. The same link is working fine when accessed directly via the Publish instance
Updated:
The access.log in Dispatcher (Apache Web Server)
Success log of dispatcher
domain - - [11/Jul/2014:15:25:11 +0530] "GET /content/sample.html HTTP/1.1" 200 25805
Failure log on one of the links on the above page
domain - - [11/Jul/2014:15:25:12 +0530] "GET /bin/servlet/SampleServlet?action=GET_SAMPLE_USAGE HTTP/1.1" 404 230
It is not finding GET_SAMPLE_USAGE servlet, but the same is already available in the OSGi bundle and is working perfectly fine via the Publish instance (logs below).
Publisher access.log
domain - admin 11/Jul/2014:15:24:31 +0530 "GET /content/sample.html HTTP/1.1" 200 25805 "-" "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:21.0) Gecko/20100101 Firefox/21.0"
domain - admin 11/Jul/2014:15:24:31 +0530 "GET /bin/servlet/SampleServlet?action=GET_SAMPLE_USAGE HTTP/1.1" 200 - [this is not a link] ( "domain:4503/content/sample.html" ) "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:21.0) Gecko/20100101 Firefox/21.0"
The application has a page with thumbnails. Clicking on a thumbnail calls the SetURL() javascript function on the player object.
In Safari on Windows, about 75% of the time, the plugin makes the request, sends the cookie, and life is good. The other 25% of the time, it fails to load at all half the time and when it does load, it won't loop. When it fails, we see the following requests:
127.0.0.1 - [20/May/2009:11:15:19 -0400] "GET /full/?id=1 HTTP/1.1" 302 - "-" 80 7542 0 QuickTime/7.6 (qtver=7.6;os=Windows NT 5.1Service Pack 3)
127.0.0.1 - [20/May/2009:11:15:19 -0400] "GET /denied/ HTTP/1.1" 200 3385 "-" 80 9050 0 QuickTime/7.6 (qtver=7.6;os=Windows NT 5.1Service Pack 3)
127.0.0.1 - [20/May/2009:11:15:20 -0400] "GET /full/?id=1 HTTP/1.1" 200 2639638 "-" 80 2005787 2 Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US) AppleWebKit/525.27.1 (KHTML, like Gecko) Version/3.2.1 Safari/525.27.1
The first request comes directly from the plugin and the request does not contain the session cookie, so the app redirects it to the "unauthorized access" page. We then see the plugin make the request to the redirected page. Then we see the same initial request from the browser itself. That request contains the cookie so it succeeds. About half the time, the movie plays, the other half it doesn't.
As I said, if we have 10 thumbnails on a page, everything works fine for at least 7 of them so we know the plugin is actually loaded. There are no javascript errors.
I have seen similar behavior in IE, but have not been able to reproduce it consistantly.
Thoughts?
On our development servers we password protect the sites at the server level (via htpasswd).
In Firefox, the first time the SetURL() was called for a new movie, the browser would wait and pop the password alert, regardless of whether or not it was "remembered" in the keychain. Once the credentials were sent, then QuickTime would load the movie.
In Chrome, it was remembered.
IE still won't swap the file. Downloading Charles Proxy now...