I have the following command line that outputs vmstat every second with a time stamp on each line via the perl script:
vmstat 15 | /home/Beer/addtimestamp.pl > File_1
the contents of the addtimestamp.pl:
!/usr/bin/perl
while (<>) { print localtime() . ": $_"; }
So why doesn't the output get redirected to the "File_1" file?
It works perfectly when I don't, it prints out the output perfectly every second with no issues at all.
When outputting to the terminal, perl's output is line buffered, so you will see every line as it is output. When its output is a file, it will be block buffered so you will not see any output until a full block is ready to write (4k I think, but variable and system-defined).
You need to set stdout to use line buffering:
$|=1;
Search for [perl line buffered output] and you'll see plenty of results about this.
Related
I have a Perl script
for my $i (1..10) {
print "How $i\n";
sleep(1);
}
I want to run it from another perl script and capture its output. I know that I can use qx() or backticks, but they will return all the output simultaneously when the program exits.
But, what I instead want is to print the output of the first script from the second script as soon as they are available ie. for the example code the output of the first script is printed in ten steps from the second script and not in one go.
I looked at this question and wrote my second script as
my $cmd = "perl a.pl";
open my $cmd_fh, "$cmd |";
while(<$cmd_fh>) {
print "Received: $_\n";
STDOUT->flush();
}
close $cmd_fh;
However, the output is still being printed simultaneously. I wanted to know a way to get this done ?
The child sends output in chunks of 4 or KiB or 8 KiB rather than a line at a time. Perl programs, like most programs, flush STDOUT on linefeed, but only when connected to a terminal; they fully buffer the output otherwise. You can add $| = 1; to the child to disable buffering of STDOUT.
If you can't modify the child, such programs can be fooled by using pseudo-ttys instead of pipes. See IPC::Run. (Search its documentation for "pty".)
If I run a perl program and call another perl program using backquotes, print statements from the called program don't appear at the terminal.
If I call the program using 'system', the print statements are displayed.
EG:
This is ProgA.pl
print "In ProgA.pl, about to call ProgB.pl";
my $dum=`ProgB.pl`; # print output doesn't appear
### $dum=system("ProgB.pl"); # this prints OK
print"\nBack in ProgA.pl";
print "\ndum = $dum"; # ProgB's output doesn't show here either
(No warnings or errors, perl.exe found through file association)
This is ProgB.pl:
print "\nPrinting from ProgB.pl";
What is the reason for the difference?
Why isn't the backquoted call output returned in $dum (I tried both STDOUT and STDERR)? If I call dir in backquotes, I get its output in $dum.
You have a path issue.
It works as expected ($dum is assigned the value "Printing from ProgB.pl") if I change the backticks from ProgB.pl to ./ProgB.pl. Without the explicit ./ path, it searches the system path and generates an error, as you can see if you change that line to
my $dum=`ProgB.pl` or die $!;
Which generates the output
In ProgA.pl, about to call ProgB.plNo such file or directory at ./ProgA.pl line 4.
Thus illustrating once again that you should always check the return values of your system calls for error conditions.
It appears that by failing to put a newline character at the end of the print command in ProgB, I failed to flush the buffer before returning to ProgA. Thanks to Chris Turner.
Assume a pipeline with three programs:
start | middle | end
If start and end are now part of one perl script, how can I pipe data through a shell command in the perl script, in order to pass through middle?
I tried the following (apologies for lack of strict mode, it was supposed to be a simple proof of concept):
#!/usr/bin/perl -n
# Output of "start" stage
$start = "a b c d\n";
# This shell command is "middle"
open (PR, "| sed -E 's/a/-/g' |") or die 'Failed to start sed';
# Pipe data from "start" into "middle"
print PR $start;
# Read data from "middle" into "end"
$end = "";
while (<PR>) {
$end .= $_;
}
close PR;
# Apply "end" and print output
$end =~ s/b/+/g;
print $end;
Expected output:
- + c d
Actual output:
none, until I hit ENTER, then I get - b c d. The middle command is receiving data from start and processing it, but the output is going to STDOUT instead of end. Also, the attempt to read from middle seems to be reading from STDIN instead (hence the relevance of hitting ENTER).
I'm aware that this could all easily be done in one line of perl (or sed); my problem is how to do piping in perl, not how to replace chars in a string.
You can use IPC::Open2 for this.
This code creates two file handles: $to_sed, which you can print to to send input to the program, and $from_sed which you can readline (or <$from_sed>) from to read the program's output.
use IPC::Open2;
my $pid = open2(my ($from_sed, $to_sed), "sed -E 's/a/-/g'");
Most often it is simplest to involve the shell, but there is an alternative call that allows you to bypass the shell and instead run a program and populate its argv directly. It is described in the linked documentation.
The reason your code does nothing until you hit enter is because you are using perl -n.
-n causes Perl to assume the following loop around your program, which makes it iterate over filename arguments
somewhat like sed -n or awk:
LINE:
while (<>) {
... # your program goes here
}
The part in your code where you read your file again returns nothing.
If you turn on warnings you will discover that perl doesn't do bi-directional pipes.
I have a script called my_bash.sh and it calls a perl script, and appends the output of the perl script to a log file. Does the file only get written once the perl script has completed?
my_bash.sh
#!/bin/bash
echo "Starting!" > "my_log.log"
perl my_perl.pl >> "my_log.log" 2>&1
echo "Ending!" >> "my_log.log"
The issue is that as the perl script is running, I'd like to manipulate contents of the my_log.log file while it's running, but it appears the file is blank. Is there a proper way to do this? Please let me know if you'd like more information.
my_perl.pl
...
foreach $component (#arrayOfComponents)
{
print "Component Name: $component (MORE_INFO)\n";
# Do some work to gather more info (including other prints)
#...
# I want to replace "MORE_INFO" above with what I've calculated here
system("sed 's/MORE_INFO/$moreInfo/' my_log.log");
}
The sed isn't working correctly since the print statements haven't yet made it to the my_log.log.
Perl would buffer the output by default. To disable buffering set $| to a non-zero value. Add
$|++;
at the top of your perl script.
Quoting perldoc pervar:
$|
If set to nonzero, forces a flush right away and after every write or
print on the currently selected output channel. Default is 0
(regardless of whether the channel is really buffered by the system or
not; $| tells you only whether you've asked Perl explicitly to flush
after each write). STDOUT will typically be line buffered if output is
to the terminal and block buffered otherwise. Setting this variable is
useful primarily when you are outputting to a pipe or socket, such as
when you are running a Perl program under rsh and want to see the
output as it's happening. This has no effect on input buffering. See
getc for that. See select on how to select the output channel. See
also IO::Handle.
Mnemonic: when you want your pipes to be piping hot.
The answer to this question depends on how your my_perl.pl is outputting data and how much data is being output.
If you're using normal (buffered) I/O to produce your output, then my_log.log will only be written to once the STDOUT buffer fills. Generally speaking, if you're not producing a lot of output, this is when the program ends and the buffer is flushed.
If you're producing enough output to fill the output buffer, you will get output in my_log.log prior to my_perl.pl completing.
Additionally, in Perl, you can make your STDOUT unbuffered with the following code:
select STDOUT; $| = 1;
In which case, your output would be written to STDOUT (and then to my_log.log via redirection) the moment it is produced in your script.
Depending on what you need to do to the log file, who might be able to read each line of output from the perl script, do something with the line, then write it to the log yourself (or not):
#!/bin/bash
echo "Starting!" > "my_log.log"
perl my_perl.pl | \
while read line; do
# do something with the line
echo "$line" >> "my_log.log"
done
echo "Ending!" >> "my_log.log"
In between
print "Component Name: $component (MORE_INFO)\n";
and
system("sed 's/MORE_INFO/$moreInfo/' my_log.log");
do you print stuff? If not, delay the first print until you've figured out $moreInfo
my $header = "Component Name: $component (MORE_INFO)";
# ...
# now I have $moreInfo
$header =~ s/MORE_INFO/$moreInfo/;
print $header, "\n";
If you do print stuff, you could always "queue" it until you have the info you need:
my #output;
foreach my $component (...) {
#output = ("Component Name: $component (MORE_INFO)");
# ...
push #output, "something to print";
# ...
$output[0] =~ s/MORE_INFO/$moreInfo/;
print join("\n", #output), "\n";
If I have this perl app:
print `someshellscript.sh`;
that prints bunch of stuff and takes a long time to complete, how can I print that output in the middle of execution of the shell script?
Looks like Perl will only print the someshellscript.sh result when it completes, is there a way to make output flush in the middle of execution?
What you probably want to do is something like this:
open(F, "someshellscript.sh|");
while (<F>) {
print;
}
close(F);
This runs someshellscript.sh and opens a pipe that reads its output. The while loop reads each line of output generated by the script and prints it. See the open documentation page for more information.
The problem here is that escaping with backticks stores your script to a string, which you then print. For this reason, there would be no way to "flush" with print.
Using the system() command should print output continuously, but you won't be able to capture the output:
system "someshellscript.sh";