How to streamingly to read text file with Perl - perl

it can display text in file, however, after i add new text in gedit, it do not show the updated one.
sub start_thread {
my #args = #_;
print('Thread started: ', #args, "\n");
open(my $myhandle,'<',#args) or die "unable to open file"; # typical open call
for (;;) {
while (<$myhandle>) {
chomp;
print $_."\n";
}
sleep 1;
seek FH, 0, 1; # this clears the eof flag on FH
}
}
update video
https://docs.google.com/file/d/0B4hnKBXrOBqRWEdjTDFIbHJselk/edit?usp=sharing
https://docs.google.com/file/d/0B4hnKBXrOBqRcEFhU3k4dUN4cXc/edit?usp=sharing
how to print $curpos for updated data
for (;;) {
for ($curpos = tell($myhandle); $_ = <$myhandle>;
$curpos = tell($myhandle)) {
# search for some stuff and put it into files
print $curpos."\n";
}
sleep(1);
seek(FILE, $curpos, 0);
}

Like I said - it works for me. Changes to your script are minimal - just minimal cleanup.
Script: test_tail.pl
#!/usr/bin/perl
sub tail_file {
my $filename = shift;
open(my $myhandle,'<',$filename) or die "unable to open file"; # typical open call
for (;;) {
print "About to read file...\n";
while (<$myhandle>) {
chomp;
print $_."\n";
}
sleep 1;
seek $myhandle, 0, 1; # this clears the eof flag on FH
}
}
tail_file('/tmp/test_file.txt');
Then:
echo -e "aaa\nbbb\nccc\n" > /tmp/test_file.txt
# wait a bit
echo -e "ddd\neee\n" >> /tmp/test_file.txt
Meanwhile (in a different terminal);
$ perl /tmp/test_tail.pl
About to read file...
aaa
bbb
ccc
About to read file...
About to read file...
About to read file...
ddd
eee

Instead of this:
seek $myhandle, 0, 1; # this clears the eof flag on FH
Can you try something like this:
my $pos = tell $myhandle;
seek $myhandle, $pos, 0; # reset the file handle in an alternate way

The file system is trying to give you a consistent view of the file you are reading. To see the changes, you would need to reopen the file.
To see an example of this, try the following:
1.Create a file that has 100 lines of text in it, a man page, for instance:
man tail > foo
2.Print the file slowly:
cat foo | perl -ne 'print; sleep 1;'
3.While that is going on, in another shell or editor, try editing the file by deleting most lines
Result: The file will continue to print slowly, as if you never edited it. Only when you try to print it again, will you see the changes.

The following would also work:
my $TAIL = '/usr/bin/tail -f'; # Adjust accordingly
open my $fh, "$TAIL |"
or die "Unable to run $TAIL : $!";
while (<$fh>) {
# do something
}

Related

Replace the last line written to a file descritor

I am writing a small perl program where I am checking the pattern of #start and #end. The agenda is to create a separate file with the lines in between start and end patterns. This I am able to do with below script.
#!/usr/bin/perl
open(INFILE,"<","testcases") || die "Can't open file: $!";
my $binary;
my $tccounter=1;
while(<INFILE>)
{
if(/^#start/i)
{
open(OUTFILE,">",$tccounter."_case.sh") || die "Can't open file: $!";
print "start of the script\n";
next;
}
elsif(/^#end/i)
{
################################
# Want to replace the previously
# written line here with some
# addtional customized lines
################################
close(OUTFILE);
$tccounter++;
print "End of the script\n";
print "last line for this testcase is \n $binary\n";
next;
}
else
{
$binary=$_ unless(/^\s*$/);
print OUTFILE $_;
}
}
But what I additionally needed is is identify the last line that is being written to a file and then replace that additional line with some custom data.
For example, here in my case the last line for all the files is execute.
I want replace the line "execute" in all the output files.
In the current output files last line is as below:
execute
expected out files last line should be
preline
execute
postline
Input file (testcases):
#start
line1
line 2
execute
#end
#start
line3
line 4
execute
#end
#start
line5
line 6
execute
#end
#start
line7
line 8
execute
#end
I suggest that you should buffer your output
If you push each line to an array instead of printing it then, once the #end tag is seen, it is simple to locate the last non-blank line in the array and replace it
Then the output file can be opened and the contents of the array printed to it
Here's an untested example
use strict;
use warnings 'all';
open my $fh, "<", "testcases" or die "Can't open input file: $!";
my $n;
my $i;
my $print;
my #buff;
while ( <$fh> ) {
if ( /^#start/i ) {
#buff = ();
$i = undef;
$print = 1;
print "start of the script\n";
}
elsif ( /^#end/i ) {
my $file = ++$n . "_case.sh";
$print = 0;
unless ( defined $i ) {
warn "No data found in block $n";
next;
}
splice #buff, $i, 1, "preline\n", $buff[$i], "postline\n";
open my $fh, ">", $file or die qq{Can't open "$file" for output: $!};
print $fh #buff;
close $fh;
print "End of the script\n";
}
elsif ( $print ) {
push #buff, $_;
$i = $#buff if /\S/;
}
}
I think Borodins answer is the way to go (I'm just not able to comment yet).
So the general algorithm is:
collect full record, from start marker to end marker
once end marker is reached, process record content. In your case:
find last non-empty line and surround it with others
print found line
write out file for record
repeat as needed
I couldn't resist and rewrote Borodins solution using the flipflop operator:
use strict;
use warnings;
open(my $in,'<','in.file') || die "Can't open file: $!";
my ($cnt,#rec);
while( <$in> ) {
push(#rec,$_) if /^#start/i .. /^#end/i; # collect record lines (using flipflop operator)
if( /^#end/i ) { # end of record reached?
next if #rec <= 2; # ignore empty records
# determine index of last nonempty line
my ($lci) = grep {$rec[$_]=~/\S/} reverse (1..$#rec-1); # ...except markers
printf "last line for this testcase is \n%s\n", # print find
splice #rec, $lci, 1, ("preline\n",$rec[$lci],"postline\n"); # surround with pre&post
# write out result
open(my $out,'>',++$cnt.'_case.sh') || die "Can't open file: $!";
$out->print(#rec[1..$#rec-1]); # ...except markers
$out->close;
#rec=(); # empty record for next use
}
}

How do I change values with perl and regex/sed inside a file?

I'm pretty sure I am doing something stupid and I apologize for this ahead of time. I have looked at the one-liners that were suggested elsewhere on similar searches and I like the idea of them, I'm just not sure how to apply because it's not a direct swap. And if the answer is that this can't be done, then that is fine and I will script around that.
The problem: I have log files I need to send through a parser that requires the dates to be in YYYY-MM-DD. The files can be saved this way; however, some people prefer them in YYYY/MM/DD for their own viewing and send those to me. I can modify one or two dates with sed and this works beautifully; however, when there are 2-3+ years in the files, it would be nice not to have to do it manually for each date.
My code (I have left the debugging commands in place):
use strict;
use File::Copy;
use Getopt::Std;
my %ARGS = ();
getopts('f:v', \%ARGS);
my $file = $ARGS{f};
&main();
sub main($)
{
open (FIN, "<$file") || die ("Cannot open file");
print "you opened the file\n";
while (<FIN>) {
my $line = $_;
if ($line =~ /(\d*)\/(\d*)\/(\d*) /i) {
#print "you are in the if";
my $year = $1;
my $month = $2;
my $day = $3;
print $line;
print "\nyou have year $1\n";
print "you have month $2\n";
print "you have day $3\n";
s/'($1\/$2\/$3)/$1-$2-$3'/;
}
}
close FIN;
}
I can see that the regex is getting the right values into my variables but the original line is not being replaced in the file.
Questions:
1) Should this be possible to do within the same file or do I need to output it to a different file? Looking at other answers, same file should be fine.
2) Does the file need to be opened in another way or somehow set to be written to rather than merely running the replace command like I do with sed? <--I am afraid that the failure may be in here somewhere simple that I am overlooking.
Thanks!
You never write to the file. With sed, you'd use -i, and you can do exactly the same in Perl.
perl -i -pe's{(\d{4})/(\d{2})/(\d{2})}{$1-$2-$3}g' file
Or with a backup:
perl -i~ -pe's{(\d{4})/(\d{2})/(\d{2})}{$1-$2-$3}g' file
That's equivalent to
local $^I = ''; # Or for the second: local $^I = '~';
while (<>) {
s{(\d{4})/(\d{2})/(\d{2})}{$1-$2-$3}g;
print;
}
If you didn't want to rely on $^I, you'd have to replicate its behaviour.
for my $qfn (#ARGV) {
open($fh_in, '<', $qfn)
or do { warn("Can't open $ARGV: $!\n"); next; };
unlink($qfn)
or do { warn("Can't overwrite $ARGV: $!\n"); next; };
open(my $fh_out, '>', $qfn) {
or do { warn("Can't create $ARGV: $!\n"); next; };
while (<$fh_in>) {
s{(\d{4})/(\d{2})/(\d{2})}{$1-$2-$3}g;
print $fh_out $_;
}
}
perl -pi.bak -e 's|(\d{4})/(\d\d)/(\d\d)|$1-$2-$3|g;' input
Replace input with your log file name. A backup file input.bak will be created in case you ever need the original data.

How to clean the Output

I open a file and print some data on the screen , but I want to clean the screen after I output the data , I use clear; in the program but I don't see the effect of clean . It didn't clean .Does there
any command or function can let me do that?
I want to see the contain of a file only , but not to see some of the previous file on the screen ...
Here is my programs
`ls > File_List`;
open List , "<./File_List";
while(eof(List)!=1)
{
$Each = readline(*List);
chomp $Each;
print $Each;
print "\n";
`clear`;
open F , "<./$Each";
while(eof(F)!=1)
{
for($i=0;$i<20;$i++)
{
$L = readline(*F);
print $L;
}
last;
}
close(F);
sleep(3);
$Each = "";
}
close List;
Thanks
Your program uses non-idiomatic Perl. A more natural style would be
#!/usr/bin/env perl
use strict;
use warnings;
no warnings 'exec';
opendir my $dh, "." or die "$0: opendir: $!";
while (defined(my $name = readdir $dh)) {
if (-T $name) {
system("clear") == 0 or warn "$0: clear exited " . ($? >> 8);
print $name, "\n";
system("head", "-20", $name) == 0 or warn "$0: head exited " . ($? >> 8);
sleep 3;
}
}
Instead of writing a list of names to another file, read the names directly with opendir and readdir. The defined check is necessary in case you have a file named 0, which Perl considers to be a false value and would terminate the loop prematurely.
You don’t want to print everything. The directory entry may be a directory or an executable image or a tarball. The -T file test attempts to guess whether the file is a text file.
Invoke the external clear command using Perl’s system.
Finally, use the external head command to print the first 20 lines of each text file.
clear isn't working because the control sequence it outputs to clear the screen is being captured and returned to your program instead of being sent to the display.
Try
print `clear`
or
system('clear')
instead
The solution you provided doesn't work because the clear command is performed in a sub-shell. I suggest the use of a CPAN module (and multi platform supported): Term::Screen::Uni
Example:
use Term::Screen::Uni;
my $screen = Term::Screen::Uni->new;
$screen->clrscr;
Use system(), it works.
system("ls > File_List");
system("clear;");

Perl: Append to file and get new line count

Quick question, and I'm sure it's something I'm doing completely wrong with variables, however, here is the issue.
Code first:
#!/usr/bin/perl
use strict;
use warnings;
my $File = "file.txt";
my $CurrentLinesCount = `wc -l < $File` or die "wc failed: $?";
chomp($CurrentLinesCount);
sub GetStatistics() {
if (-d $dir) {
print "Current Lines In File: $CurrentLinesCount\n";
}
else {
exit;
}
}
sub EditFile() {
my $editfile = $File;
my $text = "1234\n12345\n234324\n2342\n2343";
open(MYFILE,">>$editfile") || die("Cannot Open File");
print MYFILE "$text";
close(MYFILE);
sleep 5;
}
## MAIN
GetStatistics();
EditFile();
GetStatistics();
This is the output I get:
Current Lines In File: 258
Current Lines In File: 258
I verified that the file is being written and appended to. Can someone point me in the correct direction on how to have a variable set, updated, and then called again properly?
You call subs, not variables.
Try:
sub CurrentLinesCount {
my $CurrentLinesCount = `wc -l < $File` or die "wc failed: $?";
chomp($CurrentLinesCount);
return $CurrentLinesCount;
}
...
print "Current Lines In File: ", CurrentLinesCount(), "\n";
You're only doing the call to wc once. Thus you're setting the value of $CurrentLinesCount once, and you get the same number when you print it twice.
You'll have to redo the
$CurrentLinesCount = `wc -l < $File` or die "wc failed: $?";
line after you append to the file.
Edit: Or put that line in the GetStatistics function, which would probably be a better place for it.
I would probably move the code block
my $CurrentLinesCount = `wc -l < $File` or die "wc failed: $?";
chomp($CurrentLinesCount);
to the GetStatistics subroutine, so the variable is updated whenever you call your sub.
As an optimization, you can count how many lines you added rather than recounting the whole file (unless another process may also be writing to the file).
use strict;
use warnings;
use FileHandle;
use IPC::Open2;
our $CurrentLinesCount;
our $file = "file.txt";
sub CountLines {
my $File = shift;
my $CurrentLinesCount = `wc -l < $File` or die "wc failed: $?";
$CurrentLinesCount =~ s/\s+//g;
return $CurrentLinesCount;
}
sub ShowStatistics {
my $file = shift;
if (-f $file) {
print "Current Lines In File: $CurrentLinesCount\n";
} else {
exit;
}
}
sub EditFile {
my $editfile = shift;
my $sleeptime = shift || 5;
my $text = "1234\n12345\n234324\n2342\n2343";
open(MYFILE,">>$editfile") || die("Cannot Open File");
print MYFILE "$text";
close(MYFILE);
# Look here:
my $pid = open2(*Reader, *Writer, "wc -l" );
print Writer $text;
close Writer;
$CurrentLinesCount += <Reader>;
sleep $sleeptime;
}
$CurrentLinesCount = CountLines($file);
ShowStatistics($file);
# EditFile updates $CurrentLinesCount
EditFile($file, 2);
ShowStatistics($file);
Still one too many globals for my taste, but I suppose this isn't a program of consequence. On the other hand, globals can be habit forming.
Note that wc doesn't count anything after the final "\n" when counting lines (it views "\n" as a line terminator). If you want to view "\n" as a line separator and count those trailing characters as a line, you'll need an alternate method of counting lines.

Showing progress whilst running a system() command in Perl

I have a Perl script which performs some tasks, one of which is to call a system command to "tar -cvf file.tar.....".
This can often take some time so I'd like the command line to echo back a progress indicator, something like a # echoing back to screen whilst the system call is in progress.
I've been doing some digging around and stumbled across fork. Is this the best way to go? Is it possible to fork off the system command, then create a while loop which checks on the staus of the $pid returned by the fork?
I've also seen references to waitpid.... I'm guessing I need to use this also.
fork system("tar ... ")
while ( forked process is still active) {
print #
sleep 1
}
Am I barking up the wrong tree?
Many thanks
John
Perl has a nice construction for this, called "pipe opens." You can read more about it by typing perldoc -f open at a shell prompt.
# Note the use of a list for passing the command. This avoids
# having to worry about shell quoting and related errors.
open(my $tar, '-|', 'tar', 'zxvf', 'test.tar.gz', '-C', 'wherever') or die ...;
Here's a snippet showing an example:
open(my $tar, '-|', 'tar', ...) or die "Could not run tar ... - $!";
while (<$tar>) {
print ".";
}
print "\n";
close($tar);
Replace the print "." with something that prints a hash mark every 10 to 100 lines or so to get a nice gaugebar.
An example that doesn't depend on the child process writing any kind of output, and just prints a dot about once a second as long as it's running:
use POSIX qw(:sys_wait_h);
$|++;
defined(my $pid = fork) or die "Couldn't fork: $!";
if (!$pid) { # Child
exec('long_running_command', #args)
or die "Couldn't exec: $!";
} else { # Parent
while (! waitpid($pid, WNOHANG)) {
print ".";
sleep 1;
}
print "\n";
}
Although it could probably stand to have more error-checking, and there might actually be something better already on CPAN. Proc::Background seems promising for abstracting this kind of job away but I'm not sure how reliable it is.
$|++;
open(my $tar, 'tar ... |') or die "Could not run tar ... - $!";
while ($file=<$tar>) {
print "$file";
}
print "\n";
close($tar);
This prints the filenames received from tar.
For showing progress during a long-running task, you will find Term::ProgressBar useful -- it does the "printing of # across the screen" functionality that you describe.
I would try something like this
open my $tar, "tar -cvf file.tar..... 2>&/dev/null |"
or die "can't fork: $!";
my $i = 0;
while (<$tar>) {
if( i++ % 1000 == 0 ) print;
}
close $tar or die "tar error: $! $?";
Expanding on what Hobbs provided if you would like to get the data from the child process back into the Parent process you need to have an external conduit. I ended up using the tempfs because it was simple like a file, but does not put IO hits on the disk.
** Important **
You need to exit the child process, because otherwise the "child" process will continue along the same script and you will get double print statements. So in the example below foreach (#stdoutput) would happen two times despite only being in the script once.
$shm_id = time; #get unique name for file - example "1452463743"
$shm_file = "/dev/shm/$shm_id.tmp"; #set filename in tempfs
$| = 1; #suffering from buffering
print ("Activity Indicator: "); #No new line here
defined(my $pid = fork) or die "Couldn't fork: $!";
if (!$pid) { # Child
#stdoutput=`/usr/home/script.pl -o $parameter`; #get output of external command
open (SHM, ">$shm_file");
foreach (#stdoutput) {
print SHM ("$_"); #populate file in tempfs
}
close (SHM);
exit; #quit the child process (will not kill parent script)
} else { # Parent
while (! waitpid($pid, WNOHANG)) {
print ("\#"); # prints a progress bar
sleep 5;
}
}
print ("\n"); #finish up bar and go to new line
open (SHM, "$shm_file");
#stdoutput = <SHM>; #Now open the file and read it. Now array is in parent
close (SHM);
unlink ($shm_file); #deletes the tempfs file
chomp(#stdoutput);
foreach (#stdoutput) {
print ("$_\n"); #print results of external script
}