I'm building a regression system (not unit testing) for some Perl scripts.
A core component of the system is
`perl script.pl #params 1>stdoutfile 2>stderrfile`;
However, in the course of actually working on the scripts, they sometimes don't compile(Shock!). But perl itself will execute correctly. However, I don't know how to detect on stderr whether Perl failed to compile (and therefore wrote to stderr), or my script barfed on input (and therefore wrote to stderr).
How do I detect whether a program executed or not, without exhaustively finding Perl error messages and grepping the stderr file?
It might be easiest to do this in two steps:
system('$^X -c script.pl');
if ($? == 0) {
# it compiled, now let's see if it runs
system('$^X script.pl', #params, '1>stdoutfile', '2>stderrfile');
# check $?
}
else {
warn "script.pl didn't compile";
}
Note the use of $^X instead of perl. This is more flexible and robust. It ensures that you're running from the same installation instead of whatever interpreter shows up first in your path. The system call will inherit your environment (including PERL5LIB), so spawning a different version of perl could result in hard-to-diagnose compatibility errors.
When I want to check that a program compiles, I check that it compiles :)
Here's what I put into t/compile.t to run with the rest of my test suite. It stops all testing with the "bail out" if the script does not compile:
use Test::More tests => 1;
my $file = '...';
print "bail out! Script file is missing!" unless -e $file;
my $output = `$^X -c $file 2>&1`;
print "bail out! Script file does not compile!"
unless like( $output, qr/syntax OK$/, 'script compiles' );
Scripts are notoriously hard to test. You have to run them and then scrape their output. You can't unit test their guts... or can you?
#!/usr/bin/perl -w
# Only run if we're the file being executed by Perl
main() if $0 eq __FILE__;
sub main {
...your code here...
}
1;
Now you can load the script like any other library.
#!/usr/bin/perl -w
use Test::More;
require_ok("./script.pl");
You can even run and test main(). Test::Output is handy for capturing the output. You can say local #ARGV to control arguments or you can change main() to take #ARGV as an argument (recommended).
Then you can start splitting main() up into smaller routines which you can easily unit test.
Take a look at the $? variable.
From perldoc perlvar:
The status returned by the last pipe
close, backtick ("``") command,
successful call to wait() or
waitpid(), or from the system()
operator. This is just the 16-bit
status word returned by the
traditional Unix wait() system call
(or else is made up to look like it).
Thus, the exit value of the subprocess
is really ("$? >> 8"), and "$? & 127"
gives which signal, if any, the
process died from, and "$? & 128"
reports whether there was a core dump.
It sounds like you need IPC::Open3.
Related
I have a perl script which takes 2 arguments as follows and calls appropriate function depending on the argument. I call this script from bash, but i want to call it from perl, is it possible?
/opt/sbin/script.pl --group="value1" --rule="value2";
Also the script exits with a return value that I would like to read.
The Perl equivalent of sh command
/opt/sbin/script.pl --group="value1" --rule="value2"
is
system('/opt/sbin/script.pl', '--group=value1', '--rule=value2');
You could also launch the command in a shell by using the following, though I'd avoid doing so:
system(q{/opt/sbin/script.pl --group="value1" --rule="value2"});
Just like you'd have to do in sh, you'll have to follow up with some error checking (whichever approach you took). You can do so by using use autodie qw( system );. Check the docs for how to do it "manually" if you want more flexibility.
If you want to capture the output:
$foo = `/opt/sbin/script.pl --group="value1" --rule="value2"`;
If you want to capture the exit status, but send script.pl's output to stdout:
$status = system "/opt/sbin/script.pl --group=value1 --rule=value2";
If you want to read its output like from a file:
open SCRIPT, "/opt/sbin/script.pl --group=value1 --rule=value2 |" or die $!;
while (<SCRIPT>) ...
Yes. You can use system, exec, or <backticks>.
The main difference between system and exec is that exec "executes a command and never returns.
Example of system:
system("perl", "foo.pl", "arg");
I have a Perl script that takes user input and creates another script that will be run at a later date. I'm currently going through and writing tests for these scripts and one of the tests that I would like to perform is checking if the generated script compiles successfully (e.g. perl -c <script>.) Is there a way that I can have Perl perform a compile on the generated script without having to spawn another Perl process? I've tried searching for answers, but searches just turn up information about compiling Perl scripts into executable programs.
Compiling a script has a lot of side-effects. It results in subs being defined. It results in modules being executed. etc. If you simply want to test whether something compiles, you want a separate interpreter. It's the only way to be sure that one testing one script doesn't cause later tests to give false positives or false negatives.
To execute dynamically generated code, use eval function:
my $script = join /\n/, <main::DATA>;
eval($script); # 3
__DATA__
my $a = 1;
my $b = 2;
print $a+$b, "\n";
However if you want to just compile or check syntax, then you will not be able to do it within same Perl session.
Function syntax_ok from library Test::Strict run a syntax check by running perl -c with an external perl interpreter, so I assume there is no internal way.
Only work-around that may work for you would be:
my $script = join /\n/, <main::DATA>;
eval('return;' . $script);
warn $# if $#; # syntax error at (eval 1) line 3, near "1
# my "
__DATA__
my $a = 1
my $b = 2;
print $a+$b, "\n";
In this case, you will be able to check for compilation error(s) using $#, however because the first line of the code is return;, it will not execute.
Note: Thanks to user mob for helpfull chat and code correction.
Won't something like this work for you ?
open(FILE,"perl -c generated_script.pl 2>&1 |");
#output=<FILE>;
if(join('',#output)=~/syntax OK/)
{
printf("No Problem\n");
}
close(FILE);
See Test::Compile module, particularly pl_file_ok() function.
I have a Perl script that runs a different utility (called Radmind, for those interested) that has the capability to edit the filesystem. The Perl script monitors output from this process, so it would be running throughout this whole situation.
What would happen if the utility being run by the script tried to edit the script file itself, that is, replace it with a newer version? Does Perl load the script and any linked libraries at the start of its execution and then ignore the script file itself unless told specifically to mess with it? Or perhaps, would all hell break loose, and executions might or might not fail depending on how the new file differed from the one being run?
Or maybe something else entirely? Apologies if this belongs on SuperUser—seems like a gray area to me.
It's not quite as simple as pavel's answer states, because Perl doesn't actually have a clean division of "first you compile the source, then you run the compiled code"[1], but the basic point stands: Each source file is read from disk in its entirety before any code in that file is compiled or executed and any subsequent changes to the source file will have no effect on the running program unless you specifically instruct perl to re-load the file and execute the new version's code[2].
[1] BEGIN blocks will run code during compilation, while commands such as eval and require will compile additional code at run-time
[2] Most likely by using eval or do, since require and use check whether the file has been loaded already and ignore it if it has.
For a fun demonstration, consider
#! /usr/bin/perl
die "$0: where am I?\n" unless -e $0;
unlink $0 or die "$0: unlink $0: $!\n";
print "$0: deleted!\n";
for (1 .. 5) {
sleep 1;
print "$0: still running!\n";
}
Sample run:
$ ./prog.pl
./prog.pl: deleted!
./prog.pl: still running!
./prog.pl: still running!
./prog.pl: still running!
./prog.pl: still running!
./prog.pl: still running!
Your Perl script will be compiled first, then run; so changing your script while it runs won't change the running compiled code.
Consider this example:
#!/usr/bin/perl
use strict;
use warnings;
push #ARGV, $0;
$^I = '';
my $foo = 42;
my $bar = 56;
my %switch = (
foo => 'bar',
bar => 'foo',
);
while (<ARGV>) {
s/my \$(foo|bar)/my \$$switch{$1}/;
print;
}
print "\$foo: $foo, \$bar: $bar\n";
and watch the result when run multiple times.
The script file is read once into memory. You can edit the file from another utility after that -- or from the Perl script itself -- if you wish.
As the others said, the script is read into memory, compiled and run. GBacon shows that you can delete the file and it will run. This code below shows that you can change the file and do it and get the new behavior.
use strict;
use warnings;
use English qw<$PROGRAM_NAME>;
open my $ph, '>', $PROGRAM_NAME;
print $ph q[print "!!!!!!\n";];
close $ph;
do $PROGRAM_NAME;
... DON'T DO THIS!!!
Perl scripts are simple text files that are read into memory, compiled in memory, and the text file script is not read again. (Exceptions are modules that come into lexical scope after compilation and do and eval statements in some cases...)
There is a well known utility that exploits this behavior. Look at CPAN and its many versions which is probably in your /usr/bin directory. There is a CPAN version for each version of Perl on your system. CPAN will sense when a new version of CPAN itself is available, ask if you want to install it, and if you say "y" it will download the newer version and respawn itself right where you left off without loosing any data.
The logic of this is not hard to follow. Read /usr/bin/CPAN and then follow the individualized versions related to what $Config::Config{version} would generate on your system.
Cheers.
I'm writing a perl script that mimics gcc. This my script needs to process some stdout from gcc. The part for processing is done, but I can't get the simple part working: how can I forward all the command line parameters as is to the next process (gcc in my case). Command lines sent to gcc tend to be very long and can potentially contain lots of escape sequences and I don't want now to play that game with escaping and I know that it's tricky to get it right on windows in complicated cases.
Basically,
gcc.pl some crazies\ t\\ "command line\"" and that gcc.pl has to forward that same command line to real gcc.exe (I use windows).
I do it like that: open("gcc.exe $cmdline 2>&1 |") so that stderr from gcc is fed to stdout and I my perl script processes stdout. The problem is that I can't find anywhere how to construct that $cmdline.
I would use AnyEvent::Subprocess:
use AnyEvent::Subprocess;
my $process_line = sub { say "got line: $_[0]" };
my $gcc = AnyEvent::Subprocess->new(
code => ['gcc.exe', #ARGV],
delegates => [ 'CompletionCondvar', 'StandardHandles', {
MonitorHandle => {
handle => 'stdout',
callback => $process_line,
}}, {
MonitorHandle => {
handle => 'stderr',
callback => $process_line,
}},
],
);
my $running = $gcc->run;
my $done = $running->recv;
$done->is_success or die "OH NOES";
say "it worked";
The MonitorHandle delegate works like redirection, except you have the option of using a separate filter for each of stdout and stderr. The "code" arg is an arrayref representing a command to run.
"Safe Pipe Opens" in the perlipc documentation describes how to get another command's output without having to worry about how the shell will parse it. The technique is typically used for securely handling untrusted inputs, but it also spares you the error-prone task of correctly escaping all the arguments.
Because it sidesteps the shell, you'll need to create the effect of 2>&1 yourself, but as you'll see below, it's straightforward to do.
#! /usr/bin/perl
use warnings;
use strict;
my $pid = open my $fromgcc, "-|";
die "$0: fork: $!" unless defined $pid;
if ($pid) {
while (<$fromgcc>) {
print "got: $_";
}
}
else {
# 2>&1
open STDERR, ">&STDOUT" or warn "$0: dup STDERR: $!";
no warnings "exec"; # so we can write our own message
exec "gcc", #ARGV or die "$0: exec: $!";
}
Windows proper does not support open FH, "-|", but Cygwin does so happily:
$ ./gcc.pl foo.c
got: gcc: foo.c: No such file or directory
got: gcc: no input files
Read up on the exec function and the system function in Perl.
If you provide either of these with an array of arguments (rather than a single string), it invokes the Unix execve() function or a close relative directly, without letting the shell interpret anything, exactly as you need it to do.
Thanks for answers, I came to conclusion that I made a big mistake that I touched perl again: hours of time wasted to find out that it can't be done properly.
Perl uses different way to split command line parameters than all other apps that use MS stdlib (which is standard on win32).
Because of that some commandline parameters that were meant to be interpreted as a signle commandline argument, by perl can be interpreted as more than one argument. That means that all what I'm trying to do is waste of time because of that buggy behavior in perl. It's impossible to get this task done correctly if I 1) can't access original command line as is and 2) perl doesn't split command line arguments correctly.
as a simple test:
script.pl """test |test"
on win32 will incorrectly interpret command line as:
ARGV=['"test', '|test']
Whereas, the correct "answer" on windows has to be
ARGV=['"test |test']
I used activestate perl, I tried also latest version of strawberry perl: both suck. It appears that perl that comes with msys works properly, most likely because it was built against mingw instead of cygwin runtime?..
The problem and reason with perl is that it has buggy cmd line parser and it won't work on windows NO MATTER WHAT cygwin supports or not.
I have a simple case where an environment variable (which I cannot control) expands to
perl gcc.pl -c "-IC:\ffmpeg\lib_avutil\" rest of args
Perl sees that I have two args only: -c and '-IC:\ffmpeg\lib_avutil" rest of args'
whereas any conforming windows implementation receives second cmd line arg as: '-IC:\ffmpeg\lib_avutil\', that mean that perl is a huge pile of junk for my simple case, because it doesn't provide adequate means to access cmd line arguments. I'm better off using boost::regex and do all my parsing in c++ directly, at least I won't ever make dumb mistakes like ne and != for comparing strings etc. Windows's escaping rules for command line arguments are quite strange, but they are standard on windows and perl for some strange reason doesn't want to follow OS's rules.
Is it possible to run an external process from Perl, capture its stderr, stdout AND the process exit code?
I seem to be able to do combinations of these, e.g. use backticks to get stdout, IPC::Open3 to capture outputs, and system() to get exit codes.
How do you capture stderr, stdout, and the exit code all at once?
(Update: I updated the API for IO::CaptureOutput to make this even easier.)
There are several ways to do this. Here's one option, using the IO::CaptureOutput module:
use IO::CaptureOutput qw/capture_exec/;
my ($stdout, $stderr, $success, $exit_code) = capture_exec( #cmd );
This is the capture_exec() function, but IO::CaptureOutput also has a more general capture() function that can be used to capture either Perl output or output from external programs. So if some Perl module happens to use some external program, you still get the output.
It also means you only need to remember one single approach to capturing STDOUT and STDERR (or merging them) instead of using IPC::Open3 for external programs and other modules for capturing Perl output.
If you reread the documentation for IPC::Open3, you'll see a note that you should call waitpid to reap the child process. Once you do this, the status should be available in $?. The exit value is $? >> 8. See
$? in perldoc perlvar.
If you don't want the contents of STDERR, then the capture() command from IPC::System::Simple module is almost exactly what you're after:
use IPC::System::Simple qw(capture system $EXITVAL);
my $output = capture($cmd, #args);
my $exit_value = $EXITVAL;
You can use capture() with a single argument to invoke the shell, or multiple arguments to reliably avoid the shell. There's also capturex() which never calls the shell, even with a single argument.
Unlike Perl's built-in system and backticks commands, IPC::System::Simple returns the full 32-bit exit value under Windows. It also throws a detailed exception if the command can't be started, dies to a signal, or returns an unexpected exit value. This means for many programs, rather than checking the exit values yourself, you can rely upon
IPC::System::Simple to do the hard work for you:
use IPC::System::Simple qw(system capture $EXIT_ANY);
system( [0,1], "frobincate", #files); # Must return exitval 0 or 1
my #lines = capture($EXIT_ANY, "baznicate", #files); # Any exitval is OK.
foreach my $record (#lines) {
system( [0, 32], "barnicate", $record); # Must return exitval 0 or 32
}
IPC::System::Simple is pure Perl, has no dependencies, and works on both Unix and Windows systems. Unfortunately, it doesn't provide a way of capturing STDERR, so it may not be suitable for all your needs.
IPC::Run3 provides a clean and easy interface into re-plumbing all three common filehandles, but unfortunately it doesn't check to see if the command was successful, so you'll need to inspect $? manually, which is not at all fun. Providing a public interface for inspecting $? is something which is on my to-do list for IPC::System::Simple, since inspecting $? in a cross-platform fashion is not a task I'd wish on anyone.
There are other modules in the IPC:: namespace that may also provide you with assistance. YMMV.
All the best,
Paul
There are three basic ways of running external commands:
system $cmd; # using system()
$output = `$cmd`; # using backticks (``)
open (PIPE, "cmd |"); # using open()
With system(), both STDOUT and STDERR will go the same place as the script's STDOUT and STDERR, unless the system() command redirects them. Backticks and open() read only the STDOUT of your command.
You could also call something like the following with open to redirect both STDOUT and STDERR.
open(PIPE, "cmd 2>&1 |");
The return code is always stored in $? as noted by #Michael Carman.
If you're getting really complicated, you might want to try Expect.pm. But that's probably overkill if you don't need to also manage sending input to the process as well.
I found IPC:run3 to be very helpful. You can forward all child pipes to a glob or a variable; very easily! And exit code will be stored in $?.
Below is how i grabbed stderr which i knew would be a number. The cmd output informatic transformations to stdout (which i piped to a file in the args using >) and reported how many transformations to STDERR.
use IPC::Run3
my $number;
my $run = run3("cmd arg1 arg2 >output_file",\undef, \undef, \$number);
die "Command failed: $!" unless ($run && $? == 0);