Run external command and process its log file in parallel - perl

I need to run external tool from within my Perl code. This command works for a pretty long time, prints almost nothing to STDOUT but creates a log file.
I would like to run it and in parallel read and process its log file. How can I do it in Perl?
Thanks in advance.

If you use something like File::Tail to read the log file, then you can do a simple fork and exec to run the external command. Something like the following should work:
use strict;
use warnings;
use File::Tail;
my $pid = fork;
if ( $pid ) {
# in the parent process; open the log file and wait for input
my $tail = File::Tail->new( '/path/to/logfile.log' );
while( my $line = $tail->read ) {
# do stuff with $line here
last if $line eq 'done running'; # we need something to escape the loop
# or it will wait forever for input.
}
} else {
# in the child process, run the external command
exec 'some_command', 'arg1', 'arg2';
}
# wait for child process to exit and clean it up
my $exit_pid = wait;
If there are problems running the child process, the exit return code will be in the special variable $?; see the documentation for wait for more information.
Also, if the logging output does not provide a clue for when to stop tailing the file, you can install a handler in $SIG{CHLD} which will catch the child process's termination signal and allow you to break out of the loop.

Related

How to use Perl to check when a Unix command has finished processing

I am working on a capstone project and am hoping for some insight.
This is the first time I've worked with Perl and it's pretty much a basic Perl script to automate a few different Unix commands that need to be executed in a specific order. There are two lines throughout the script which executes a Unix command that needs to finish processing before it is acceptable for the rest of the script to run (data will be incorrect otherwise).
How am I able to use Perl (or maybe this is a Unix question?) to print a simple string once the Unix command has finished processing? I am looking into ways to read in the Unix command name but am not sure how to implement a way to check if the process is no longer running and to print a string such as "X command has finished processing" upon it's completion.
Example:
system("nohup scripts_pl/RunAll.pl &");
This runs a command in the background that takes time to process. I am asking how I can use Perl (or Unix?) to print a string once the process has finished.
I'm sorry if I didn't understand your asking context.
But couldn't you use perl process fork function instead of & if you would like to do parallel process?
# parent process
if (my $pid = fork) {
# this block behaves as a normal process
system("nohup scripts_pl/RunAll2.pl"); # you can call other system (like RunAll2.pl)
wait; # wait for the background processing
say 'finished both';
}
# child process
else {
# this block behaves as a background process
system("nohup scripts_pl/RunAll.pl"); # trim &
}
You could try to use IPC::Open3 instead of system:
use IPC::Open3;
my $pid = open3("<&STDIN", ">&STDOUT", ">&STDERR", 'nohup scripts_pl/RunAll.pl');
waitpid( $pid, 0 );
Or, if you need to run nohup through the shell:
my $pid = open3("<&STDIN", ">&STDOUT", ">&STDERR", 'bash','-c', 'nohup scripts_pl/RunAll.pl & wait');
Update: Thanks to #ikegami. A better approach if you would like STDIN to stay open after running the command:
open(local *CHILD_STDIN, "<&", '/dev/null') or die $!;
my $pid = open3("<&CHILD_STDIN", ">&STDOUT", ">&STDERR", 'nohup scripts_pl/RunAll.pl');

Non-blocking child process blocks file

Consider this scenario:
We have three scripts:
script.pl
use strict;
use warnings;
print "\nStarting a blocking process";
print "\nRedirect the output of the blocking process to execution.log";
my $cmd = "perl d:\\blocking_script.pl >d:\\execution.log";
my $exitCode = system ($cmd);
print "\nAfter the execution of the blocking process";
print "\nNow I try to rename the log";
rename "d:\\execution.log", "d:\\execution.err" or print "\nCouldn't rename because : $!";
blocking_script.pl
use strict;
use warnings;
print "\nFrom the blocking_process I run a non-blocking process";
my $cmd = "start perl d:\\non_blocking_script.pl";
my $exitCode = system ($cmd);
print "\nAfter I started the non-blocking process";
non_blocking_script.pl
use strict;
use warnings;
print "\nI am an independent non-blocking process";
sleep 5;
print "\nStill here";
sleep 2;
print "\nYou can't rename the log because you didn't wait for me";
sleep 3;
print "\n.";
sleep 1;
What will result from this?
Couldn't rename because : Permission denied
While another command promopt will be hanging ironically :
I am an independent non-blocking process
Still here
You can't rename the log because you didn't wait for me
.
In my situation from perl I run an external application in a blocking way, but that application was starting some non-blocking process which were holding my log.
How can I overcome this situation?
Here is the documentation for start (which you should also be able to read by using start /? on the command line. I do not have access to a Windows system right now, so I can't verify.
/b
Starts an application without opening a new Command Prompt window. CTRL+C handling is ignored unless the application enables CTRL+C processing. Use CTRL+BREAK to interrupt the application.
blocking_script.pl is waiting for the cmd window which start opened to run non_blocking_script.pl.
In the short run, using start /b might help.
Or, you could try
my #cmd = start => qw(perl d:\\non_blocking_script.pl);
my $exitCode = system #cmd;
However, you should change your design.

Perl : Cancel a batch process, ran using a system call, in a thread, from main

Hello pips,
I have a Tkx gui which runs a batch file using button.
The batch file is executed in a different thread, cause I want the GUI to still be usable. I want to implement a cancel button to cancel the execution of the batch file.
I tried sending a Kill signal but it will only terminate the thread and not the batch file. Below are the codes for the run and cancel subroutines.
Oh and I am not allowed to edit the batch file.
my $t1;
sub runbutton{
$bar->g_grid();
$bar->start();
$t1 = threads->create(sub {
local $SIG{'KILL'} = sub { threads->exit };
system("timer.bat");
});
$t1->set_thread_exit_only(1);
my $start = time;
my $end = time;
while ($t1->is_running()) {
$end = time();
$mytext = sprintf("%.2f\n", $end - $start);
Tkx::update();
}
$bar->stop();
$bar->g_grid_forget();
$b4->g_grid_forget();
}
sub cancelbutton
{
$t1->kill('KILL')->detach();
}
You are running this on windows, since you say 'batch'?
I believe you'll have to 'identify' and 'kill' the process using OS-specific tools, e.g. pslist/pskill (sysinternals)
I suspect that the Perl thread is waiting for system to return before "firing" the kill signal is executed.
I would suggest using Win32::Process to start the batch file as a separate process and then have a variable to signal the process should be terminated. When the variable is set, the thread can kill the process and then exit.
Here is the small test case I used to check it out using Win::Process to create and kill a batch file as a separate process using Active State Perl Version 5.16.1:
use strict;
# Modules needed for items
use Win32::Process;
use Win32;
# Subroutine to format the last error message which occurred
sub ErrorReport
{
print Win32::FormatMessage( Win32::GetLastError() );
}
print "Starting Process\n";
# Declare a scalar for the process object
my $ProcessObj;
# Create the process to run the batch file
Win32::Process::Create($ProcessObj,
"C:\\Users\\Glenn\\temp.bat",
"C:\\Users\\Glenn\\temp.bat",0,
NORMAL_PROIRITY_CLASS,
".") || ErrorReport();
print "Process Started\n";
# Sleep for a few seconds to let items start running
sleep(2);
# Kill the process with a -9
# $ProcessObj->Kill(0) does not seem to work to stop the
# process. Kill will kill the process with the process id
kill -9,$ProcessObj->GetProcessID();
# Done with efforts
print "Complete\n";
If you are using Windows 7, you will need to run as administrator to allow the process to be created, otherwise you will get an Access Denied message when trying to create the process.

Perl How to exit from child script which does a telnet?

I have a script which executes few commands and then telnets to machine. Now I need to call this script from another perl script.
$result = `some_script.pl`;
The script some_script.pl executes successfully but I am not able to exit from the main script as the script waits at the telnet prompt.
I also need to capture the exit status of the script in order to make sure that some_script.pl executed successfully.
I cannot modify some_script.pl.
Is there some way by which I can issue quit after the some_script.pl is executed successfully?
Try this out, this 'magic' close the standard in/out/err and may let your program finish.
$result = `some_script.pl >&- 2>&- <&-';
Otherwise you could use open2 and expect to watch for a specific string (like Done!) in your program output and close it when done.
http://search.cpan.org/~rgiersig/Expect-1.15/Expect.pod
Regards
I don't like the way you are actually executing your perl script with a "backtick" call to the system.
I suggest you actually fork (or something equivalent) and run the program in a more controlled manner.
use POSIX ":sys_wait_h";
my $pid = fork();
if($pid) { # on the parent proc, $pid will point to the child
waitpid($pid); # wait for the child to finish
} else { # this is the child, where we want to run the telnet
exec 'some_script.pl'; # this child will now "become" some_script.pl
}
Since I don't know how some_script.pl actually works, I cannot really help you more here. But for example, if all you need to do is print "quit" on the command line of some_script.pl you could use IPC::Open2 like suggested in another question. Doing something like:
use IPC::Open2;
$pid = open2(\*CHLD_OUT, \*CHLD_IN, 'some_script.pl');
print CHLD_IN "quit\n";
waitpid( $pid, 0 );
my $child_exit_status = $? >> 8;
You do need to tweak this a little, but the idea should solve your problem.

How can I run a system command in Perl asynchronously?

I currently have a Perl script that runs an external command on the system, gathers the output, and performs some action based on what was returned. Right now, here is how I run this (where $cmd is a string with the command setup):
#output = `$cmd`;
I'd like to change this so if the command hangs and does not return a value after so much time then I kill the command. How would I go about running this asynchronously?
There's a LOT of ways to do this:
You can do this with a fork (perldoc -f fork)
or using threads (perldoc threads). Both of these make passing the returned information back to the main program difficult.
On systems that support it, you can set an alarm (perldoc -f alarm) and then clean up in the signal handler.
You can use an event loop like POE or Coro.
Instead of the backticks, you can use open() or respectively open2 or open3 (cf. IPC::Open2, IPC::Open3) to start a program while getting its STDOUT/STDERR via a file handle. Run non-blocking read operations on it. (perldoc -f select and probably google "perl nonblocking read")
As a more powerful variant of the openX()'s, check out IPC::Run/IPC::Cmd.
Probably tons I can't think of in the middle of the night.
If you really just need to put a timeout on a given system call that is a much simpler problem than asynchronous programming.
All you need is alarm() inside of an eval() block.
Here is a sample code block that puts these into a subroutine that you could drop into your code. The example calls sleep so isn't exciting for output, but does show you the timeout functionality you were interested in.
Output of running it is:
/bin/sleep 2 failure: timeout at
./time-out line 15.
$ cat time-out
#!/usr/bin/perl
use warnings;
use strict;
my $timeout = 1;
my #cmd = qw(/bin/sleep 2);
my $response = timeout_command($timeout, #cmd);
print "$response\n" if (defined $response);
sub timeout_command {
my $timeout = (shift);
my #command = #_;
undef $#;
my $return = eval {
local($SIG{ALRM}) = sub {die "timeout";};
alarm($timeout);
my $response;
open(CMD, '-|', #command) || die "couldn't run #command: $!\n";
while(<CMD>) {
$response .= $_;
}
close(CMD) || die "Couldn't close execution of #command: $!\n";
$response;
};
alarm(0);
if ($#) {
warn "#cmd failure: $#\n";
}
return $return;
}
If your external program doesn't take any input, look for the following words in the perlipc manpage:
Here's a safe backtick or pipe open for read:
Use the example code and guard it with an alarm (which is also explained in perlipc).
I coded below to run rsync on 20 directories simultaneously (in parallel instead of sequentially requiring me to wait hours for it to complete):
use threads;
for my $user ( keys %users ) {
my $host = $users{$user};
async {
system <<~ "SHELL";
ssh $host \\
rsync_user $user
SHELL
}
}
$ pgrep -lf rsync | wc -l
20
Not sure if it's best or even a good solution, but I was glad that it worked for my use case.
With this you get a mixed output on screen (what I ignored anyway), but it does its job successfully.
threads pragma exports the (very useful) async function by default.
rsync_user is my Perl script that wraps rsync command with options, and source and target directories set.
Ran on FreeBSD 13.1 with Perl 5.32.1