wget not behaving via IPC::Open3 vs bash - perl

I'm trying to stream a file from a remote website to a local command and am running into some problems when trying to detect errors.
The code looks something like this:
use IPC::Open3;
my #cmd = ('wget','-O','-','http://10.10.1.72/index.php');#any website will do here
my ($wget_pid,$wget_in,$wget_out,$wget_err);
if (!($wget_pid = open3($wget_in,$wget_out,$wget_err,#cmd))){
print STDERR "failed to run open3\n";
exit(1)
}
close($wget_in);
my #wget_outs = <$wget_out>;
my #wget_errs = <$wget_err>;
print STDERR "wget stderr: ".join('',#wget_errs);
#page and errors outputted on the next line, seems wrong
print STDERR "wget stdout: ".join('',#wget_outs);
#clean up after this, not shown is running the filtering command, closing and waitpid'ing
When I run that wget command directly from the command-line and redirect stderr to a file, something sane happens - the stdout will be the downloaded page, the stderr will contain the info about opening the given page.
wget -O - http://10.10.1.72/index.php 2> stderr_test_file
When I run wget via open3, I'm getting both the page and the info mixed together in stdout. What I expect is the loaded page in one stream and STDERR from wget in another.
I can see I've simplified the code to the point where it's not clear why I want to use open3, but the general plan is that I wanted to stream stdout to another filtering program as I received it, and then at the end I was going to read the stderr from both wget and the filtering program to determine what, if anything went wrong.
Other important things:
I was trying to avoid writing the wget'd data to a file, then filtering that file to another file, then reading the output.
It's key that I be able to see what went wrong, not just reading $? >> 8 (i.e. I have to tell the user, hey, that IP address is wrong, or isn't the right kind of website, or whatever).
Finally, I'm choosing system/open3/exec over other perl-isms (i.e. backticks) because some of the input is provided by untrustworthy users.

You are passing an undefined value as the error handle argument to open3, and as IPC::Open3 says:
If CHLD_ERR is false, or the same file descriptor as CHLD_OUT, then STDOUT and STDERR of the child are on the same filehandle (this means that an autovivified lexical cannot be used for the STDERR filehandle, see SYNOPSIS) ...
The workaround is to initialize $wget_err to something before calling open3:
my ($wget_pid, $wget_in, $wget_out, $wget_err);
use Symbol qw(gensym);
$wget_err = gensym();
if (!$wget_pid = open3( ... ) ) { ...

Related

I have a system call that somehow is causing malformed header from script Bad header

I have a Perl CGI program. myperlcgi.pl
Within this program I have the following:
my $retval = system('extprogram')
extprogram has a print statement within.
The output from extprogram is being included within myperlcgi.pl
I tried adding
> output.txt 2>&1
to system call but did not change anything.
How do I prevent output form extprogram being used in myperlcgi.pl.
Surprised that stdout from system call is being used in myperlcgi.pl
The system command just doesn’t give you complete control over capturing STDOUT and STDERR of the executed command.
Use backticks or open to execute the command instead. That will capture the STDOUT of the command’s execution. If the command also outputs to STDERR, then you can append 2>&1 to redirect STDERR to STDOUT for capture in backticks, like so:
my $output = `$command 2>&1`;
If you really need the native return status of the executed command, you can get that information using $? or ${^CHILD_ERROR_NATIVE}. See perldoc perlvar for details.
Another option is to use the IPC::Open3 Perl library, but I find that method to be overkill for most situations.

Meaning of open( STDERR, ">&STDOUT" )

i find this in one sample script,
then i search from google and found the following words,
Note that you cannot simply open STDERR to be a dup of STDOUT in your Perl
program and avoid calling the shell to do the redirection. This doesn't
work:
open(STDERR, ">&STDOUT");
This fails because the open() makes STDERR go to where STDOUT was going at
the time of the open(). The backticks then make STDOUT go to a string, but
don't change STDERR (which still goes to the old STDOUT).
Now I am confused. What exactly is the meaning of open(STDERR, ">&STDOUT"); ?
With the & in the mode >& in the call
open STDERR, ">&STDOUT"; # or: open STDERR, ">&", \*STDOUT
the first given filehandle is made a copy of the second one. See open, and see man 2 dup2 since this goes via dup2 syscall. The notation follows the shell's I/O redirection.
Since here the first filehandle exists (STDERR)† it is first closed.
The effect is that prints to STDERR will go to where STDOUT was going before this was done, with the side effect of the original STDERR being closed.
This is legit and does not result in errors but is not a good way to redirect STDERR in general -- after that we cannot restore STDERR any more. See open for how to redirect STDERR.
The rest of the comment clearly refers to a situation where backticks (see qx), which redirect STDOUT of the executed command(s) to the program, are used after that open call. All this seems to refer to an idea of redirecting STDERR to STDOUT in this way.
Alas, the STDERR, made by that open call to go where STDOUT was going, doesn't get redirected by the backticks and thus still goes "there." In my case prints to STDERR wind up on the terminal as I see the warning (ls: cannot access...) with
perl -we'open STDERR, ">&STDOUT"; $out = qx(ls no_such)'
(unlike with perl -we'$out = qx(ls no_such 2>&1)'). Explicit prints to STDERR also go to the terminal as STDOUT (add such prints and redirect output to a file to see).
This may be expected since & made a copy of the filehandle, so the "new" one (the former STDERR) still goes where STDOUT was going, that is to the terminal. What is of course unintended in this case and thus an error.
† Every program in UNIX gets connected to standard streams stdin, stdout, and stderr, with file descriptors 0, 1, and 2 respectively. In a Perl program we then get ready filehandles for these, like the STDERR (for fd 2).
Some generally useful posts on manipulations of file descriptors in the shell:
What does “3>&1 1>&2 2>&3” do in a script?
In the shell, what does “ 2>&1 ” mean?
File descriptors & shell scripting
Order of redirections
Shell redirection i/o order
It's basically dup2(fileno(STDOUT), fileno(STDERR)). See your system's dup2 man page.
In short, it associates STDERR with the same stream as STDOUT at the system level. After the command is performed writing to either will be the same as writing to STDOUT before the change.
Unless someone's messed with STDOUT or STDERR, it's equivalent to the shell command
exec 2>&1

Perl - impdp from perl

I am not able to re-direct the output of impdp command from system () perl to a file.
#!/usr/bin/perl
$a="impdp GLS_UCELL_80TC5_DEV/comverse directory=DUMP_DIR_1028704 dumpfile=ACCESS_REGION_VALUES.dmp CONTENT=data_only";
system("$a 1>t.tmp");
However, upon changing the system command to system("$a 2>t.tmp"), t.tmp gets created.
I want to understand why the re-direction 1>t.tmp not working in my case.
1>t.tmp (or just >t.tmp) redirects STDOUT. Anything printed to STDOUT will end up in the file.
2>t.tmp redirects STDERR. Anything printed to STDERR will end up in the file.
You want to capture what your program writes to STDERR, so you need to use the latter, or >t.tmp 2>&1 which redirects both.

Perl Cleartool Accept Statement

I am using the following code to execute the ClearCase statement "Accept" from a perl script.
$acceptA = `accept $Component`;
After execution, inside my perl script, the value of $acceptA is blank.
The text displayed on the screen during execution of this line is : "ERROR You do not have permissions to ACCEPT this work."
How do I read this line? I thought it would return into the variable $acceptA as it does with the "cleartool checkin" command?
As i do not know ClearCase and how that accept works, i can only guess. Seeing how it is an error message, it might be written to STDERR instead of STDOUT and backticks only capture STDOUT of the command executed.
In that case, redirecting the commands STDERR to STDOUT would work. Try
$acceptA = `accept $Component 2>&1`
and see if that works in capturing the output in case of error as well.
I eventually redirected SYSERR to an output file which I could read/write.
open STDERR, ">/ellipse/el6.3.3_ry_sup/src/0/$logfile"
All the error messages that was displayed on the screen as part of the system command, went into $logfile.
I was also able to add to SYSERR with the following:
print STDERR "\nAccepting $Component";
Thanks for all the help.

Copy output and extract number from it in perl

I am working on a perl script in which I will run a command and get a output like : your id is <895162>. I will store this string and read the number from this string only . The problem is my main command will run in shell using the system command from perl .
like :
#ids.csh is "echo your id is <1123221>"
my $p = system ("./ids.csh 2>&1 > /dev/null");
print "$p\n";
$p =~ s/[^0-9]//g;
but the output is not copying to the $p file , Where I am going wrong ?
system runs a command but doesn't capture it. For that, you want qx/backticks:
my $p = `./ids.csh 2>/dev/null`;
As Len Jaffe said, you probably want to throw away stderr output (rather than displaying it to your screen or wherever your stderr is going), but not stdout (that contains the message you want to capture).
Note that when qx fails, it can do so for several different reasons and constructing a meaningful error message is not trivial. If you run into problems, consider using IPC::System::Simple's capture() instead.
You have redirected all of the output to /dev/null, which means that all of your output is being discarded.
I think you probably mean:
./ids.csh 2>/dev/null
Which will redirect stderr to /dev/null while leaving stdout unchanged.