PERL: reading Log file and cmd line input together - perl

I have written a script which is reading some data from log file and transform the data to simpler form and writing it back to another file. the reading is done line by line with a delay of 5 seconds i.e. sleep(5).
Meanwhile, on the command line if a user enters 'suspend' (through STDIN) then the program went on to sleep unless 'resume' is not entered and then read the next line.
Since, with every iteration in the loop I am checking STDIN whether 'suspend' is entered or not by the user.
if not then read the next line from the file. but when my programs runs I have to at least hit a ENTER key, otherwise it does not picks the next line from the input log file albeit i put an if statement to check if STDIN is undefined or not.
I am not a perl expert and this the first time I am writing a code in PERL. infact i have never done this file parsing thing before :'-(
my code implementation is like this;
#!/usr/local/bin/perl
my $line_no = 0;my $cancel = 0; my $line = "";
my $curpos = 0; my $whence = 0;
my $log_file = "/var/log/tmp_nagios.log";
#open(LOGFILE, "+< $log_file")or die "Failed to open $log_file, $!";
my $inp_file = "/var/log/sec_input";
my $logbuffer="";
#open(LOGFILE, "+< $log_file")or die "Failed to open $log_file, $!";
my $in;
while(1){
print "in While (1) Pos: $curpos and Whence:$whence\n";
open(LOGFILE, "+< $log_file")or die "Failed to open $log_file, $!";
seek(LOGFILE, $curpos, $whence);
next if(eof(LOGFILE));
print "Beginning\n";
while(<LOGFILE>){
#chomp($in = <STDIN>);
#if(defined($in) && $in =~ /^suspend$/i){
### Problem here ###
if(defined(<STDIN>) && <STDIN> =~ /^suspend\n$/i){ ## checking if 'suspend' is entered
print "Suspend Mode";
do{
sleep(5);
}while(!(<STDIN> =~ /^resume\n$/i));
print "Resume now\n";
close(LOGFILE);
last;
}
else{
$line = $_;
if($line =~ m/^\[(\d+)\]\sCURRENT\sSERVICE\sSTATE:\s(\w+);(\w+|\_|\d+)+;(CRITICAL|OK);.+$/){
$logbuffer = "$1,$2-$3,$4\n";
print $logbuffer;
open(INPFILE, ">> $inp_file")or die "Failed! to open $inp_file, $!";
#print INPFILE $logbuffer;
close(INPUTFILE);
sleep(5);
$curpos = tell(LOGFILE);
$whence = 1;
}
}
}
print "\nRe openning the file from Pos=$curpos and Whence=$whence\n";
}
close(LOGFILE);
here is the sample log file (/var/log/tmp_nagios.log) data;
[1284336000] CURRENT SERVICE STATE: host1;event1;CRITICAL; s
[1284336000] CURRENT SERVICE STATE: host2;event1;CRITICAL; f
[1284336000] CURRENT SERVICE STATE: host3;event3;CRITICAL; g
[1284336000] CURRENT SERVICE STATE: host4;event4;CRITICAL; j
[1284336000] CURRENT SERVICE STATE: host5;event1;CRITICAL; s
[1284336000] CURRENT SERVICE STATE: host6;event1;CRITICAL; f
[1284336000] CURRENT SERVICE STATE: host7;event7;CRITICAL; s
Sorry guys! Typo mistake
In the beginning I said, 'my script is reading data from log file with a delay of 5 seconds i.e. sleep(5)'
but actually i forget to mention it in my code, therefore, uncomment this line: #sleep(3); and make 'sleep(5);'
thanks

If I have understood your question correctly: check out the Term::ReadKey CPAN Module.
You can use it to do non-blocking buffer reads. (If there is nothing in the buffer, your script does not pause for user input)
https://metacpan.org/pod/Term::ReadKey
You may also like to approach this problem slightly differently - using signals:
http://perldoc.perl.org/perlipc.html. You can have your program run normally, but capture interrupts (e.g. CTRL-C)
Alternatively, you could just use CTRL-Z and fg to make your script sleep and wake.

Another option is POE::Wheel::FollowTail for the log and POE::Wheel::ReadLine or Term::Visual for user input. Though this might be a little overkill

Related

How to make perl to keep perform action until the match is found

I am new to Perl and trying to write a code to keep executing an action until the match is found and else give an error.
I am trying to execute a command ps -ef and check if it has got any process running in the name of "box", if there is no process named "box" found, I want to repeat ps -ef command execution until it gets the "box" process and then proceed to next action.
#!/usr/bin/perl -w
open (FH, "ps -ef |") or die "Cannot run the command:$!\n";
$line = "box";
while (<FH>) {
if (/$line/i) { next; }
else {
print ("ps -ef |") or die "Cannot run the command:$!\n");
}
}
close (FH);
You need to use an infinite loop and an exit-condition. Your condition is that the ps -ef command contains the word box. There is no need to open a pipe to that command explicitly, you can just run it as a system call with the qx operator (same as backticks).
use strict;
use warnings;
my $ps;
PS: while (1) {
$ps = qx/ps -ef/;
last PS if $ps =~ m/box/i;
print '.'; # do something in every run
}
print $ps;
As this has come up in the comments as well as in in AdrianHHH's answer: it might make sense to sleep after every run to make sure you don't hog the CPU. Depending on the nature of the process you are looking for, either the sleep builtin or usleep from Time::HiRes might be appropriate. The latter let's your program rest for milliseconds, while the builtin only works with full seconds. These might be too long if the target box process is very quick.
Explanation of your code:
Note that you have some issues in your implementation. I'll explain what your code does. This is taken from the question, comments are mine.
#!/usr/bin/perl -w
# open a filehandle to the ps command
open (FH, "ps -ef |") or die "Cannot run the command:$!\n";
$line = "box";
# read the output of one run line by line, for each line execute
# the block
while (<FH>) {
# if there is 'box' case-insensitive, skip the line
if (/$line/i) { next; }
else {
# else output (not run!) the command
print ("ps -ef |") or die "Cannot run the command:$!\n");
}
}
close (FH);
After it went through all the lines of the output of your command once it will stop.
I would recommend using pgrep(1) instead of ps because it lets you do a more granular search. With ps -ef, you potentially have to deal with cases like:
boxford 6254 6211 0 08:23 pts/1 00:00:00 /home/boxford/box --bounding-box=123
It's hard to tell if you're matching a process being run by a user with box in their username, a process that has box somewhere in its path, a process named box, or a process with box somewhere in its argument list.
pgrep, on the other hand, lets you match against just the process name or the full path, a specific user or users, and more. The following prints a message when a process named box appears (this looks for an exact match, so it will not match processes named dropbox, for example):
use strict;
use warnings;
use 5.010;
use String::ShellQuote qw(shell_quote);
sub is_running {
my ($proc) = #_;
my $cmd = 'pgrep -x ' . shell_quote($proc) . ' >/dev/null 2>&1';
system($cmd);
if ($? == -1) {
die "failed to execute pgrep: $!";
}
elsif ($? & 127) {
die "pgrep died with signal ", $? & 127;
}
else {
my $status = $? >> 8;
die "pgrep exited with error: exit status $status" if $status > 1;
return $status == 0;
}
}
my $proc = 'box';
until ( is_running($proc) ) {
sleep 1;
}
say "Process '$proc' is running";
Note that pgrep doesn't have a case-insensitive flag, probably because process names in *nix are almost always lowercase. If you really need to do a case-insensitive match, you can pass [Bb][Oo][Xx] to the is_running function.
The ps command outputs the current list of processes, then it completes. The code in the question reads that output. Suppose that the first ps command that is executed does not contain the wanted line, then there is nothing in the code in the question to run the ps command again.
The next statement in the question makes the script move on to the next line in the output from ps, not to rerun the command. The else print ... after the next will probably be executed for the first line of the output from ps. The outcome is that the print is run for each line in the ps output that does not have the wanted text and that the next command has no significant effect. In the code print ... or die "..." the or die "..." part is not very useful, the print is unlikely to fail and even if it did the die message would be wrong.
Perhaps you should write some code in the following style. Here the ps is run repeatedly until the wanted text is found. Note the sleep call, without that the script will keep running without pause, possibly preventing real work or at least slowing it down.
# This code is not tested.
use strict;
use warnings;
my $found_wanted_line = 0; # Boolean, set to false
my $line = "box";
while ( ! $found_wanted_line ) {
open (my $FH, "ps -ef |") or die "Cannot run the command:$!\n";
while (<$FH>) {
if (/$line/i) {
$found_wanted_line = 1; # Boolean, set to true
last;
}
}
close ($FH);
if ( ! $found_wanted_line )
sleep 2; # Pause for 2 seconds, to prevent this script hogging the CPU.
}
}

Open the latest log file and print lines later than a certain timestamp

I'm writing a Perl script and I need to capture some lines from a garbage collection log and write them to a file.
The log is located on a remote host and I'm connecting using the Net::OpenSSH module.
I need to read the latest log file available.
In the shell I can locate the latest log with the following commands:
cd builds/5.7.1/5.7.1.126WRF_B/jboss-4.2.3/bin
ls -lat | grep '.log$' | tail -1
Which will return the latest log:
-rw-r--r-- 1 load other 2406173 Jul 11 11:53 18156.stdout.log
So in Perl I'd like to be able write something that locates and opens that log for reading.
When I have that log file, I want to print all lines that have a timestamp greater than a specified time. The specified timestamp is a $Runtime variable subtracted from the latest log message time.
Here are the last messages of the garbage collection log:
...
73868.629: [GC [PSYoungGen: 941984K->14720K(985216K)] 2118109K->1191269K(3065984K), 0.2593295 secs] [Times: user=0.62 sys=0.00, real=0.26 secs]
73873.053: [GC [PSYoungGen: 945582K->12162K(989248K)] 2122231K->1189934K(3070016K), 0.2329005 secs] [Times: user=0.60 sys=0.01, real=0.23 secs]
So if $Runtime had a value of 120 seconds, I would need to print all the lines from timestamp (73873.053 - 120) seconds up.
In the end my script would look something like this...
open GARB, ">", "./report/archive/test-$now/GC.txt" or die "Unable to create file: $!";
my $ssh2 = Net::OpenSSH->(
$pathHost,
user => $pathUser,
password => $pathPassword
);
$ssh2->error and die "Couldn't establish SSH connection: ". $ssh2->error;
# Something to find and open the log file.
print GARB #Something to return certain lines.
close GARB;
I realize this is somewhat similar to this question, but I can't think of a way to tailor it to what I'm looking for. Any help is greatly appreciated!
Find the latest file and feed it to perl:
LOGFILE=`ls -t1 $DIR | grep '.log$' | head -1`
if [ -z $LOGFILE ]; then
echo "$0: No log file found - exiting"
exit 1;
fi
perl myscript.pl $LOGFILE
The pipe in the first line lists the file in the directory, name-only, in one column, most recent first; filters for log files, and then only returns the first one.
I have no idea how to translate your timestamps into something I can understand and do math and comparisons upon. but in general:
$threshold_ts = $time_specified - $offset;
while (<>) {
my ($line_ts) = split(/\s/, $_, 2);
print if compare_time_stamps($line_ts, $threshold_ts);
}
Writing the threshold manipulation and comparison is left as an exercise for the reader.
I think that the page for Net::OpenSSH gives a pretty good baseline for this:
my ($rout, $pid) = $ssh->pipe_out("cat /tmp/foo") or
die "pipe_out method failed: " . $ssh->error;
while (<$rout>) { print }
close $rout;
But instead, you want to do some discarding work:
my ($rout, $pid) = $ssh->pipe_out("cat /tmp/foo") or
die "pipe_out method failed: " . $ssh->error;
my $line;
while ( $line = <$rout>
and substr( $line, 0, index( $line, ':' )) < $start
) {}
while ( $line = <$rout>
and substr( $line, 0, index( $line, ':' )) <= $start + $duration
) {
print $line;
}
close $rout;
Here's an untested approach. I've not used Net::OpenSSH so there might be better ways to do it. I'm not even sure it works. What does work is the parsing part which I have tested.
use strict; use warnings;
use Net::OpenSSH;
my $Runtime = 120;
my $now = time;
open my $garb, '>',
"./report/archive/test-$now/GC.txt" or die "Unable to create file: $!";
my $ssh2 = Net::OpenSSH->(
$pathHost,
user => $pathUser,
password => $pathPassword
);
$ssh2->error and die "Couldn't establish SSH connection: ". $ssh2->error;
# Something to find and open the log file.
my $fileCapture = $ssh2->capture(
q~ls -lat builds/5.7.1/5.7.1.126WRF_B/jboss-4.2.3/bin |grep '.log$' |tail -1~
);
$fileCapture =~ m/\s(.+?)$/; # Look for the file name
my $filename = $1; # And save it in $filename
# Find the time of the last log line
my $latestTimeCapture = $ssh2->capture(
"tail -n 1 builds/5.7.1/5.7.1.126WRF_B/jboss-4.2.3/bin/$filename");
$latestTimeCapture =~ m/^([\d\.]+):/;
my $logTime = $1 - $Runtime;
my ($in, $out, $pid) = $ssh2->open2(
"builds/5.7.1/5.7.1.126WRF_B/jboss-4.2.3/bin/$filename");
while (<$in>) {
# Something to return certain lines.
if (m/^([\d\.]+):/ && $1 > $logTime) {
print $garb $_; # Assume the \n is still in there
}
}
waitpid($pid);
print $garb;
close $garb;
It uses your ls line to look up the file with the capture method. It then opens a pipe through the SSH tunnel to read that file. $in is a filehandle to that pipe which we can read.
Since we are going to process the file line by line, starting at the top, we need to first grab the last line to get the last timestamp. That is done with tail and, again, the capture method.
Once we have that, we read from the pipe line by line. This now is a simple regex (the same used above). Grab the timestamp and compare it to the time we have set earlier (minus the 120 seconds). If it is higher, print the line to the output filehandle.
The docs say we have to use waitpid on the $pid returned from $ssh2->open2 so it reaps the subprocess, so we do that before closing our output file.
You will need to either keep an accumulator containing all the lines (more memory) or iterate through the log more than once (more time).
With an accumulator:
my #accumulated_lines;
while (<$log_fh>) {
push #accumulated_lines, $_;
# Your processing to get $Runtime goes here...
if ($Runtime > $TOO_BIG) {
my ($current_timestamp) = /^(\d+(?:\.\d*))/;
my $start_timestamp = $current_timestamp - $Runtime;
for my $previous_line (#accumulated_lines) {
my ($previous_timestamp) = /^(\d+(?:\.\d*))/;
next unless $previous_timestamp <= $current_timestamp;
next unless $previous_timestamp >= $start_timestamp;
print $previous_line;
}
}
}
Or you can iterate through the log twice, which is similar, but without the nested loop. I've assumed you might have more than one of these spans in your log.
my #report_spans;
while (<$log_fh>) {
push #accumulated_lines, $_;
# Your processing to get $Runtime goes here...
if ($Runtime > $TOO_BIG) {
my ($current_timestamp) = /^(\d+(?:\.\d*))/;
my $start_timestamp = $current_timestamp - $Runtime;
push #report_spans, [ $start_timestamp, $current_timestamp ];
}
}
# Don't bother continuing if there's nothing to report
exit 0 unless #report_spans;
# Start over
seek $log_fh, 0, 0;
while (<$log_fh>) {
my ($previous_timestamp) = /^(\d+(?:\.\d*))/;
SPAN: for my $span (#report_spans) {
my ($start_timestamp, $current_timestamp) = #$span;
next unless $previous_timestamp <= $current_timestamp;
next unless $previous_timestamp >= $start_timestamp;
print; # same as print $_;
last SPAN; # don't print out the line more than once, if that's even possible
}
}
If you might have overlapping spans, the latter has the advantage of not showing the same log lines twice. If you don't have overlapping spans, you could optimize the top one by resetting the accumulator every time you output:
my #accumulator = ();
which would save memory.
Use SFTP to access the remote filesystem. You can use Net::SFTP::Foreign (alone or via Net::OpenSSH).
It will allow you to list the contents of the remote filesystem, pick the file you want to process, open it and manipulate it as a local file.
The only tricky thing you would need to do is to read lines backward, for instance reading chunks of the file starting from the end and breaking them in lines.

simulating tail -f in Perl

As per solution provided in perldoc, I am trying to emulate tail -f but it's not working as expected. The below code could print all lines first time but not the newly lines appended. Could you please elaborate if I am missing any thing here.
#!/usr/bin/perl
open (LOGFILE, "aa") or die "could not open file reason $! \n";
for (;;)
{
seek(LOGFILE,0,1); ### clear OF condition
for ($curpos = tell(LOGFILE); <LOGFILE>; $curpos = tell(LOGFILE))
{
print "$_ \n";
}
sleep 1;
seek(LOGFILE,$curpos,0); ### Setting cursor at the EOF
}
works fine for me. How are you updating "aa" ?
You wont see the data immediately if it is a buffered write to "aa".
can you try the following in a different terminal and check whether you are seeing any update.
while ( 1 )
echo "test" >> aa
end
If you are using perl to update aa, check this section on buffering and how to disable.

File locking and unlocking from input

I have a subroutine in which i'm trying to exclusively lock a set of files (one at a time) and hold that lock for a certain amount of given time(through sleep). I'm trying to add the functionality of letting the user unlock the current locked(sleeping) file whenever they press a certain key, such as the enter key. I'm just not sure what direction to go in to get this to work. Every attempt at using STDIN and checking for \n has not worked. Thanks.
Below is the subroutine. I change to the directory in which I want the files. Create the files from 1 to how ever many files are specified. For each file an exclusive lock is put on then sleeps for a specified amount of time.
Edit: It's my fault for not mentioning this but, this script is going to be run in a Windows environment. I'd ideally like a solution that will not require additional module installation that are not included in Perl. (This was brought up because the module in breqwas' solution does not support Windows). Thanks.
sub lockFiles{
#creates files that are locked for a specific amount of seconds.
chdir("lock files")or die "Unable to enter dir $!\n";
opendir(DIR,".") or die "Can't open the current directory: $!\n";
#file iterator
my $i=1;
#for each file lock it and sleep the given amount of time
while($i<=$numberOfFiles){
open FILE, ">>", "test$i.txt" or die $!;
flock(FILE, 2) or die "Could not lock test$i.txt\n";
print "test$i.txt locked, locking for $timeToSleep seconds\n";
print "Press ctrl+c to kill this\n";
sleep($timeToSleep);
$i++;
close(FILE);
}
closedir(DIR);
#change back to the parent folder
chdir("..") or die "Can't change to the directory: $!\n";
print "subRoutine lockFiles success\n";
}
I don't have a windows machine w/perl installed to check if that works there, but documentation on Term::ReadKey implies that it should. Term::ReadKey is a module that provides non-blocking and timed read functionality. It has some limited Windows support.
use Time::HiRes qw(time sleep);
use Term::ReadKey;
sub wait_for_key {
my $timeout = shift;
my $started = time();
while (1) {
last if $started + $timeout < time();
my $str = '';
while (my $char = ReadKey(-1)) {
$str .= $char;
};
last if $str =~ m/\n/s;
sleep 0.1;
}
}
I'm sure there are better ways to do that, though. Maybe someone with perl-on-windows experience will show up.
Curse on you, Windows. On any other system the code above would look like this:
sub wait_for_key { ReadLine(shift) }
Try smth like that:
use Sys::SigAction qw(sig_alarm set_sig_handler);
sub wait_for_key {
my $timeout = shift;
eval {
my $sa = set_sig_handler('ALRM', sub { die "enough" }, {} );
eval {
sig_alarm($timeout);
my $keyboard = <>;
sig_alarm(0);
};
sig_alarm(0);
}
}
This function will exit after any input ("Enter" key), or by timeout, so you just call it instead of sleep(). The basic idea is:
1) wait infinitely for the input
2) but set an alarm for the timeout you need.
Looks evil, but works fine.
That's actually a generic way to call some code with a timeout - just make this function accept a second parameter which is a function reference, and call that reference instead of my $keyboard = <> line. And, well, check the errors and do all the boring stuff I left out in this sample.

How can I run a system command and die if anything is written to STDERR?

I'm writing a Perl script which uses an external script. The external script must run from a specific directory so I found the following useful:
use IPC::System::Simple qw(capture);
my #args = ('external script path...', 'arg1', ...);
my $out = capture( [0], "cd $dir ; #args" );
Sometimes the external script writes stuff to STDERR but still returns 0. I wish to capture these times and confess (or die). Since I don't control the return value of the external script, I thought maybe I could capture its STDERR so I'll have something like this:
my ($out, $err) = cool_capture( [0], "cd $dir ; #args" );
say "Output was: $out";
if ($err) {
die "Error: this was written to STDERR: $err";
}
What can I do?
This is covered in the Perl FAQ.
Presuming test_app is a program that outputs one line to stdout and one line to stderr:
use IPC::Open3;
use Symbol 'gensym';
my($wtr, $rdr, $err);
$err = gensym;
my $pid = open3($wtr, $rdr, $err, 'test_app');
waitpid($pid, 0);
my $status = $? >> 8;
my $stdout = <$rdr>;
my $stderr = <$err>;
print "out output: $stdout\n";
print "err output: $stderr\n";
print "Exit code: $status\n";
EDIT: Per the request updated to include capturing the exit code. You could also have asked perldoc IPC::Open3 which says
waitpid( $pid, 0 );
my $child_exit_status = $? >> 8;
And which you should read anyway for its cautions and caveats.
If significant output is being written to stdout and/or stderr or you're both reading and writing to the process. You need to be a lot more careful with your I/O handling to avoid various blocking problems.
my ($wtr, $rdr, $err) ;
my $pid = IPC::Open3::open3($wtr, $rdr, $err, #_);
close($wtr);
my $stdout = '';
my $stderr = '';
my $s = IO::Select->new;
$s->add($rdr) if $rdr;
$s->add($err) if $err;
while (my #ready = $s->can_read) {
foreach my $ioh (#ready) {
my $bytes_read = sysread($ioh, my $chunk = '', 1024);
die "read error: $!" unless $bytes_read >= 0;
if ($bytes_read) {
($ioh eq $rdr? $stdout: $stderr) .= $chunk;
}
else {
$s->remove($ioh);
}
}
}
my $pid1;
for (;;) {
last if kill(0, $pid);
$pid1 = wait();
#
# Wait until we see the process or -1 (no active processes);
#
last if ($pid1 == $pid || $pid1 <= 0);
}
Finish reading before you shutdown the process. If you're writing to the process's stdin, you'd also need to add $wtr and syswrite to the above select loop.
EDIT
Rationale:
The above is probably overkill for simple cases. This advanced handling of input and output comes into play when you're likely to move more than a few K of data.
You wouldn't need it if you were executing a 'df' command for example.
However, it's when system buffers for any of stdin, stdout or stderr fill up that blocking becomes likely and things can get more involved.
If the child process fills up the stderr and/or stdout buffers, it'll likely block and wait for you to clear them. But if you're waiting for the process finish before you read from stdout or stderr; thats a deadlock. You'll likely to see that the system call never finishes and the child process never completes.
There's a similar possibility of deadlock if stdin is being written to, but the child process is unable to consume the input. This is particularly likely in a 'pipe' situation where the child process is consuming input and writing to stdout.
The select loop is about progressively clearing the buffers to avoid blocking. Both stdout and stderr are monitored concurrently.
If you're writing to stdin and reading from stdout (a pipe), you'll want to keep stdout and stderr clear and only write to stdin when its ready to receive input.
Simply waiting for the process to finish, then reading stdout/stderr probably works 90% of the time. This reply is just to give you somewhere to go if things get more complicated and processes start to block or go into deadlock.
EDIT2
As for which to use, I'd say start simple, test hard.
Go with Sorpigal's approach, but try to stress test with higher data volumes and under more difficult loads and conditionals that you'd ever expect in a live system.