How to scrape multiple log files for an exception - perl

Question

I'm writing a node.js function to ssh to a remote machine, and attempt to scrape logs for exceptions from a variety of different log files. The important bit of the log file will look something like this:
.... gunk ....
2013-01-29 04:06:39,133 com.blahblah.BaseServlet processRequest Thread-1629 Site-102 Cons-0 Url-http://theurlthat.com/caused/the/problem App-26 yada yada yada
java.lang.NullPointerException
at com.blahblah.MyClass.hi(MyClass.java:173)
at com.blahblah.StandardStackTrace.main(StandardStackTrace.java:125)
at com.blahblah.SoOnAndSo.forth(SoOnAndSo.java:109)
at java.lang.Thread.run(Thread.java:595)
2013-01-29 04:06:39,133 com.blahblah.BaseServlet defaultThrowableHandler Thread-1629 Site-102 Cons-0 Url-http://theurlthat.com/caused/the/problem App-26 yad yada yada
TechnicalDifficultiesException: TD page delivered by handleThrowable
http://theurlthat.com/caused/the/problem
....more gunk....
I need to find the exception and corresponding date in the log file that meets the following three requirements:
The exception must be the first that precedes this static text:
TechnicalDifficultiesException: TD page delivered by handleThrowable
The Exception must be directly between two lines that have "BaseServlet.*Site-102"
The exception must be the most recent (last) in the log files that meets the above conditions. The log is rolled over periodically, so it need to be the last in Log, or if that doesn't exist the last in Log.001, or if that doesn't exist the last in Log.002, etc.
Since this program has to ssh into one of many potential servers, it's better to only have to maintain the logic in the node.js program and not on the machines with the logs. Thus, a one-liner in perl/sed/awk/grep/etc would be most ideal.

Answer 1

So your question looks like this, if I understand correctly:
The log file has a number of sections seperated by double newline.
Each is headed by a line with a date etc..
We are only interested in the sections whose headers match /BaseServlet.*?Site-102/.
If the body of a section matches /^TechnicalDifficultiesException: TD page delivered by handleThrowable/, we want to select the body of the previously matched section, which we should maybe validate to look like a java exception.
We process the whole log file, and return the last exception found this way.
Fair enough.
#!/usr/bin/perl
use strict; use warnings;
local $/ = ""; # paragraph mode
my ($prev_sec, $prev_err);
SECTION:
while (my $head = <>) {
my $body = <>;
defined $body or die "Can't read from empty filehandle.";
next SECTION unless $head =~ /BaseServlet.*?Site-102/;
if ($body =~ /^TechnicalDifficultiesException: TD page delivered by handleThrowable/) {
$prev_err = $prev_sec;
}
$prev_sec = $body;
}
die "No error found" unless defined $prev_err;
print $prev_err;
(not really tested that much, but prints out the error from your snippet)
The code is a bit to long for a one-liner. You could always pipe the source into the perl interpreter, if you wanted.
perl -ne'BEGIN{$/=""}END{print$prev_err}$b=<>;defined$b or die"empty FH";/BaseServlet.*?Site-102/ or next;$prev_err=$prev_sec if $b=~/^TechnicalDifficultiesException: TD page delivered by handleThrowable/;$prev_sec=$b'
Specify the log file as a command line argument, or pipe the file contents directly into that program. Finding the correct log file isn't hard. In a snippet of Perl:
my $log_dir = ...;
my ($log) = sort glob "$log_dir/LOG*";
die "no log in $log_dir" unless defined $log;
Update
If the date should be captured as well, the code would change to
#!/usr/bin/perl
use strict; use warnings;
local $/ = ""; # paragraph mode
my (#prev, #prev_err);
SECTION:
while (my $head = <>) {
my $body = <>;
defined $body or die "Can't read from empty filehandle.";
next SECTION unless $head =~ /BaseServlet.*?Site-102/;
if ($body =~ /^TechnicalDifficultiesException: TD page delivered by handleThrowable/) {
#prev_err = #prev;
}
#prev = ($head, $body);
}
die "No error found" unless #prev_err;
my ($date) = $prev_err[0] =~ /^(\d{4}-\d\d-\d\d \d\d:\d\d:\d\d),/;
print "$date\n\n$prev_err[1]";
And as the one-liner:
perl -ne'BEGIN{$/=""}END{#perr||die"No error found";($date)=$perr[0]=~/^(\d{4}-\d\d-\d\d \d\d:\d\d:\d\d),/;print"$date\n\n$perr[1]"}$b=<>;defined$b or die"empty FH";/BaseServlet.*?Site-102/ or next;#perr=#p if $b=~/^TechnicalDifficultiesException: TD page delivered by handleThrowable/;#p=($_,$b)'
I don't understand how it could only return the first match; this code should process the whole file. If you could provide a more complete testcase, I could verify that this code works as required.