What's the difference between Perl's backticks, system, and exec? - perl

Can someone please help me? In Perl, what is the difference between:
exec "command";
and
system("command");
and
print `command`;
Are there other ways to run shell commands too?

exec
executes a command and never returns.
It's like a return statement in a function.
If the command is not found exec returns false.
It never returns true, because if the command is found it never returns at all.
There is also no point in returning STDOUT, STDERR or exit status of the command.
You can find documentation about it in perlfunc,
because it is a function.
system
executes a command and your Perl script is continued after the command has finished.
The return value is the exit status of the command.
You can find documentation about it in perlfunc.
backticks
like system executes a command and your perl script is continued after the command has finished.
In contrary to system the return value is STDOUT of the command.
qx// is equivalent to backticks.
You can find documentation about it in perlop, because unlike system and execit is an operator.
Other ways
What is missing from the above is a way to execute a command asynchronously.
That means your perl script and your command run simultaneously.
This can be accomplished with open.
It allows you to read STDOUT/STDERR and write to STDIN of your command.
It is platform dependent though.
There are also several modules which can ease this tasks.
There is IPC::Open2 and IPC::Open3 and IPC::Run, as well as
Win32::Process::Create if you are on windows.

In general I use system, open, IPC::Open2, or IPC::Open3 depending on what I want to do. The qx// operator, while simple, is too constraining in its functionality to be very useful outside of quick hacks. I find open to much handier.
system: run a command and wait for it to return
Use system when you want to run a command, don't care about its output, and don't want the Perl script to do anything until the command finishes.
#doesn't spawn a shell, arguments are passed as they are
system("command", "arg1", "arg2", "arg3");
or
#spawns a shell, arguments are interpreted by the shell, use only if you
#want the shell to do globbing (e.g. *.txt) for you or you want to redirect
#output
system("command arg1 arg2 arg3");
qx// or ``: run a command and capture its STDOUT
Use qx// when you want to run a command, capture what it writes to STDOUT, and don't want the Perl script to do anything until the command finishes.
#arguments are always processed by the shell
#in list context it returns the output as a list of lines
my #lines = qx/command arg1 arg2 arg3/;
#in scalar context it returns the output as one string
my $output = qx/command arg1 arg2 arg3/;
exec: replace the current process with another process.
Use exec along with fork when you want to run a command, don't care about its output, and don't want to wait for it to return. system is really just
sub my_system {
die "could not fork\n" unless defined(my $pid = fork);
return waitpid $pid, 0 if $pid; #parent waits for child
exec #_; #replace child with new process
}
You may also want to read the waitpid and perlipc manuals.
open: run a process and create a pipe to its STDIN or STDERR
Use open when you want to write data to a process's STDIN or read data from a process's STDOUT (but not both at the same time).
#read from a gzip file as if it were a normal file
open my $read_fh, "-|", "gzip", "-d", $filename
or die "could not open $filename: $!";
#write to a gzip compressed file as if were a normal file
open my $write_fh, "|-", "gzip", $filename
or die "could not open $filename: $!";
IPC::Open2: run a process and create a pipe to both STDIN and STDOUT
Use IPC::Open2 when you need to read from and write to a process's STDIN and STDOUT.
use IPC::Open2;
open2 my $out, my $in, "/usr/bin/bc"
or die "could not run bc";
print $in "5+6\n";
my $answer = <$out>;
IPC::Open3: run a process and create a pipe to STDIN, STDOUT, and STDERR
use IPC::Open3 when you need to capture all three standard file handles of the process. I would write an example, but it works mostly the same way IPC::Open2 does, but with a slightly different order to the arguments and a third file handle.

Let me quote the manuals first:
perldoc exec():
The exec function executes a system command and never returns-- use system instead of exec if you want it to return
perldoc system():
Does exactly the same thing as exec LIST , except that a fork is done first, and the parent process waits for the child process to complete.
In contrast to exec and system, backticks don't give you the return value but the collected STDOUT.
perldoc `String`:
A string which is (possibly) interpolated and then executed as a system command with /bin/sh or its equivalent. Shell wildcards, pipes, and redirections will be honored. The collected standard output of the command is returned; standard error is unaffected.
Alternatives:
In more complex scenarios, where you want to fetch STDOUT, STDERR or the return code, you can use well known standard modules like IPC::Open2 and IPC::Open3.
Example:
use IPC::Open2;
my $pid = open2(\*CHLD_OUT, \*CHLD_IN, 'some', 'cmd', 'and', 'args');
waitpid( $pid, 0 );
my $child_exit_status = $? >> 8;
Finally, IPC::Run from the CPAN is also worth looking at…

What's the difference between Perl's backticks (`), system, and exec?
exec -> exec "command"; ,
system -> system("command"); and
backticks -> print `command`;
exec
exec executes a command and never resumes the Perl script. It's to a script like a return statement is to a function.
If the command is not found, exec returns false. It never returns true, because if the command is found, it never returns at all. There is also no point in returning STDOUT, STDERR or exit status of the command. You can find documentation about it in perlfunc, because it is a function.
E.g.:
#!/usr/bin/perl
print "Need to start exec command";
my $data2 = exec('ls');
print "Now END exec command";
print "Hello $data2\n\n";
In above code, there are three print statements, but due to exec leaving the script, only the first print statement is executed. Also, the exec command output is not being assigned to any variable.
Here, only you're only getting the output of the first print statement and of executing the ls command on standard out.
system
system executes a command and your Perl script is resumed after the command has finished. The return value is the exit status of the command. You can find documentation about it in perlfunc.
E.g.:
#!/usr/bin/perl
print "Need to start system command";
my $data2 = system('ls');
print "Now END system command";
print "Hello $data2\n\n";
In above code, there are three print statements. As the script is resumed after the system command, all three print statements are executed.
Also, the result of running system is assigned to data2, but the assigned value is 0 (the exit code from ls).
Here, you're getting the output of the first print statement, then that of the ls command, followed by the outputs of the final two print statements on standard out.
backticks (`)
Like system, enclosing a command in backticks executes that command and your Perl script is resumed after the command has finished. In contrast to system, the return value is STDOUT of the command. qx// is equivalent to backticks. You can find documentation about it in perlop, because unlike system and exec, it is an operator.
E.g.:
#!/usr/bin/perl
print "Need to start backticks command";
my $data2 = `ls`;
print "Now END system command";
print "Hello $data2\n\n";
In above code, there are three print statements and all three are being executed. The output of ls is not going to standard out directly, but assigned to the variable data2 and then printed by the final print statement.

The difference between 'exec' and 'system' is that exec replaces your current program with 'command' and NEVER returns to your program. system, on the other hand, forks and runs 'command' and returns you the exit status of 'command' when it is done running. The back tick runs 'command' and then returns a string representing its standard out (whatever it would have printed to the screen)
You can also use popen to run shell commands and I think that there is a shell module - 'use shell' that gives you transparent access to typical shell commands.
Hope that clarifies it for you.

Related

Calling perl script from perl

I have a perl script which takes 2 arguments as follows and calls appropriate function depending on the argument. I call this script from bash, but i want to call it from perl, is it possible?
/opt/sbin/script.pl --group="value1" --rule="value2";
Also the script exits with a return value that I would like to read.
The Perl equivalent of sh command
/opt/sbin/script.pl --group="value1" --rule="value2"
is
system('/opt/sbin/script.pl', '--group=value1', '--rule=value2');
You could also launch the command in a shell by using the following, though I'd avoid doing so:
system(q{/opt/sbin/script.pl --group="value1" --rule="value2"});
Just like you'd have to do in sh, you'll have to follow up with some error checking (whichever approach you took). You can do so by using use autodie qw( system );. Check the docs for how to do it "manually" if you want more flexibility.
If you want to capture the output:
$foo = `/opt/sbin/script.pl --group="value1" --rule="value2"`;
If you want to capture the exit status, but send script.pl's output to stdout:
$status = system "/opt/sbin/script.pl --group=value1 --rule=value2";
If you want to read its output like from a file:
open SCRIPT, "/opt/sbin/script.pl --group=value1 --rule=value2 |" or die $!;
while (<SCRIPT>) ...
Yes. You can use system, exec, or <backticks>.
The main difference between system and exec is that exec "executes a command and never returns.
Example of system:
system("perl", "foo.pl", "arg");

From Perl, spawn a shell, configure it, and fork the STDOUT

I use a Perl script to configure and spawn a compiled program, that needs a subshell configured a certain way, so I use $returncode = system("ulimit -s unlimited; sg ourgroup 'MyExecutable.exe'");
I want to capture and parse the STDOUT from that, but I need it forked, so that the output can be checked while the job is still running. This question comes close:
How can I send Perl output to a both STDOUT and a variable? The highest-rated answer describes a function called backtick() that creates a child process, captures STDOUT, and runs a command in it with exec().
But the calls I have require multiple lines to configure the shell. One solution would be to create a disposable shell script:
#disposable.sh
#!/bin/sh
ulimit -s unlimited
sg ourgroup 'MyExecutable.exe'
I could then get what I need either with backtick(disposable.sh) or open(PROCESS,'disposable.sh|').
But I'd really rather not make a scratch file for this. system() happily accepts multi-line command strings. How can I get exec() or open() to do the same?
If you want to use shell's power (that includes loops, variables, but also multiple command execution), you have to invoke the shell (open(..., 'xxx|') doesn't do that).
You can pass your shell script to the shell with the -c option of the shell (another possibility would be to pipe the commands to the shell, but that's more difficult IMHO).
That means calling the backtick function from the other answer like this:
backtick("sh", "-c", "ulimit -s unlimited; sg ourgroup 'MyExecutable.exe'");
The system tee with backticks will do this, no?
my $output = `(ulimit -s unlimited; sg ourgroup 'MyExecutable.exe') | tee /dev/tty`;
or modify Alnitak's backticks (so it does use a subshell)?
my $cmd = "ulimit -s unlimiited ; sg ourgroup 'MyExecutable.exe'";
my $pid = open(CMD, "($cmd) |");
my $output;
while (<CMD>) {
print STDOUT $_;
$output .= $_;
}
close CMD;
Expect should be used as you are interacting with your program: http://metacpan.org/pod/Expect
Assuming /bin/bash on your *nix matches something like bash-3.2$ the below program can be used to launch number of commands using $exp->send on bash console and output from each command can then be parsed for further actions.
#!/usr/bin/perl
use Expect;
my $command="/bin/bash";
my #parameters;
my $exp= new Expect;
$exp->raw_pty(1);
$exp->spawn($command);
$exp->expect(5, '-re', 'bash.*$');
$exp->send("who \n");
$exp->expect(10, '-re', 'bash.*$');
my #output = $exp->before();
print "Output of who command is #output \n";
$exp->send("ls -lt \n");
$exp->expect(10, '-re', 'bash.*$');
my #output = $exp->before();
print "Output of ls command is #output \n";

Calling command from Perl, need to see output

I need to call some shell commands from perl. Those commands take quite some time to finish so I'd like to see their output while waiting for completion.
The system function does not give me any output until it is completed.
The exec function gives output; however, it exits the perl script from that point, which is not what I wanted.
I am on Windows. Is there a way to accomplish this?
Backticks, or the qx command, run a command in a separate process and returns the output:
print `$command`;
print qx($command);
If you wish to see intermediate output, use open to create a handle to the command's output stream and read from it.
open my $cmd_fh, "$command |"; # <--- | at end means to make command
# output available to the handle
while (<$cmd_fh>) {
print "A line of output from the command is: $_";
}
close $cmd_fh;

How can I check the status of the first program in pipeline in Perl's system()?

perl -e 'system ("crontab1 -l");print $?'
returns -1 as expected (program crontab1 doesn't exist)
perl -e 'system ("crontab1 -l|grep blah");print $?'
returns 256.
What is the way to check the status of the first (or both) programs?
You are getting the exit status of the entire command, just as you should. If you want the exit status separately, you're going to have to run the commands separately.
#!/usr/bin/perl -e
system("crontab1 -l > /tmp/junk.txt"); print $?;
system("grep blah /tmp/junk.txt"); print $?;
as an example.
If you tolerate using something other than system, there are easier solutions. For example, the results method in IPC::Run returns all exit codes of the chain.
Remember, you're supposed to use $?>>8 to get the exit code, not $?
perl -e 'system("set -o pipefail;false | true");print $?>>8,"\n"'
1
This ("pipefail") only works if your shell is bash 3. Cygwin and some Linux flavors ship with it; not sure about mac.
You should be aware that 256 is an error return. 0 is what you'd get on success:
perl -e 'system("true");print $?>>8,"\n"'
0
I didn't know system returned -1 for a single command that isn't found, but $?>>8 should still be non-zero in that case.
[This was composed as an answer to another question which was closed as a duplicate of this one.]
Executing a shell command requires executing a shell. To that end,
system($shell_command)
is equivalent to
system('/bin/sh', '-c', $shell_command)
As such, all of your examples run a single program (/bin/sh). If you want the exit statuses of multiple children, you're going to need to have multiple children!
use IPC::Open3 qw( open3 );
open(local *CHILD1_STDIN, '<', '/dev/null')
or die $!;
pipe(local *CHILD2_STDIN, local *CHILD1_STDOUT)
or die $!;
my $child1_pid = open3(
'<&CHILD1_STDIN',
'>&CHILD1_STDOUT',
'>&STDERR',
'prog1', 'arg1', 'arg2',
);
my $child2_pid = open3(
'<&CHILD2_STDIN',
'>&STDOUT',
'>&STDERR',
'prog2', 'arg1', 'arg2',
);
my #pipe_status = map { waitpid($_, 0) } $child1_pid, $child2_pid;
IPC::Open3 is rather low level. IPC::Run3 and/or IPC::Run can possibly make this easier. [Update: Indeed, IPC::Run does].
If you want to check the status, don't put them all in the same system. Open a reading pipe to the first program to get its output then open another pipe to the other program.
The operating system returns an exit status only for the last program executed and if the OS doesn't return it, perl can't report it.
I don't know of any way to get at the exit code returned by earlier programs in a pipeline other than by running each individually and using temporary files instead of pipes.
What is the way to check the status of the first (or both) programs?
There is no such way, at least, not as you have constructed things. You may have to manage sub-processes yourself via fork(), exec() and waitpid() if you must know these things.
Here is roughly what is happening in your code fragment.
perl's system() spawns a shell, and perl *wait()*s for that subprocess to terminate.
The shell sets up a pipeline:
a subshell *exec()*s grep on the read-end of the pipe
a subshell fails to locate crontab1 anywhere in $PATH, and *exit()*s 127 (on my system, that is, where 127 is the shell indicating failure to find a program to run).
grep detects end-of-file on its input and, having matched nothing, *exit()*s 1.
The shell *exit()*s with the exit code of the last process in the pipeline, which, again, is 1.
perl detects the shell's exit code of 1, which is encoded in $? as 256.
(256 >> 8 == 1)

How do you capture stderr, stdout, and the exit code all at once, in Perl?

Is it possible to run an external process from Perl, capture its stderr, stdout AND the process exit code?
I seem to be able to do combinations of these, e.g. use backticks to get stdout, IPC::Open3 to capture outputs, and system() to get exit codes.
How do you capture stderr, stdout, and the exit code all at once?
(Update: I updated the API for IO::CaptureOutput to make this even easier.)
There are several ways to do this. Here's one option, using the IO::CaptureOutput module:
use IO::CaptureOutput qw/capture_exec/;
my ($stdout, $stderr, $success, $exit_code) = capture_exec( #cmd );
This is the capture_exec() function, but IO::CaptureOutput also has a more general capture() function that can be used to capture either Perl output or output from external programs. So if some Perl module happens to use some external program, you still get the output.
It also means you only need to remember one single approach to capturing STDOUT and STDERR (or merging them) instead of using IPC::Open3 for external programs and other modules for capturing Perl output.
If you reread the documentation for IPC::Open3, you'll see a note that you should call waitpid to reap the child process. Once you do this, the status should be available in $?. The exit value is $? >> 8. See
$? in perldoc perlvar.
If you don't want the contents of STDERR, then the capture() command from IPC::System::Simple module is almost exactly what you're after:
use IPC::System::Simple qw(capture system $EXITVAL);
my $output = capture($cmd, #args);
my $exit_value = $EXITVAL;
You can use capture() with a single argument to invoke the shell, or multiple arguments to reliably avoid the shell. There's also capturex() which never calls the shell, even with a single argument.
Unlike Perl's built-in system and backticks commands, IPC::System::Simple returns the full 32-bit exit value under Windows. It also throws a detailed exception if the command can't be started, dies to a signal, or returns an unexpected exit value. This means for many programs, rather than checking the exit values yourself, you can rely upon
IPC::System::Simple to do the hard work for you:
use IPC::System::Simple qw(system capture $EXIT_ANY);
system( [0,1], "frobincate", #files); # Must return exitval 0 or 1
my #lines = capture($EXIT_ANY, "baznicate", #files); # Any exitval is OK.
foreach my $record (#lines) {
system( [0, 32], "barnicate", $record); # Must return exitval 0 or 32
}
IPC::System::Simple is pure Perl, has no dependencies, and works on both Unix and Windows systems. Unfortunately, it doesn't provide a way of capturing STDERR, so it may not be suitable for all your needs.
IPC::Run3 provides a clean and easy interface into re-plumbing all three common filehandles, but unfortunately it doesn't check to see if the command was successful, so you'll need to inspect $? manually, which is not at all fun. Providing a public interface for inspecting $? is something which is on my to-do list for IPC::System::Simple, since inspecting $? in a cross-platform fashion is not a task I'd wish on anyone.
There are other modules in the IPC:: namespace that may also provide you with assistance. YMMV.
All the best,
Paul
There are three basic ways of running external commands:
system $cmd; # using system()
$output = `$cmd`; # using backticks (``)
open (PIPE, "cmd |"); # using open()
With system(), both STDOUT and STDERR will go the same place as the script's STDOUT and STDERR, unless the system() command redirects them. Backticks and open() read only the STDOUT of your command.
You could also call something like the following with open to redirect both STDOUT and STDERR.
open(PIPE, "cmd 2>&1 |");
The return code is always stored in $? as noted by #Michael Carman.
If you're getting really complicated, you might want to try Expect.pm. But that's probably overkill if you don't need to also manage sending input to the process as well.
I found IPC:run3 to be very helpful. You can forward all child pipes to a glob or a variable; very easily! And exit code will be stored in $?.
Below is how i grabbed stderr which i knew would be a number. The cmd output informatic transformations to stdout (which i piped to a file in the args using >) and reported how many transformations to STDERR.
use IPC::Run3
my $number;
my $run = run3("cmd arg1 arg2 >output_file",\undef, \undef, \$number);
die "Command failed: $!" unless ($run && $? == 0);