I have a text file to parse in Perl. I parse it from the start of file and get the data that is needed.
After all that is done I want to read the last line in the file with data. The problem is that the last two lines are blank. So how do I get the last line that holds any data?
If the file is relatively short, just read on from where you finished getting the data, keeping the last non-blank line:
use autodie ':io';
open(my $fh, '<', 'file_to_read.txt');
# get the data that is needed, then:
my $last_non_blank_line;
while (my $line = readline $fh) {
# choose one of the following two lines, depending what you meant
if ( $line =~ /\S/ ) { $last_non_blank_line = $line } # line isn't all whitespace
# if ( line !~ /^$/ ) { $last_non_blank_line = $line } # line has no characters before the newline
}
If the file is longer, or you may have passed the last non-blank line in your initial data gathering step, reopen it and read from the end:
my $backwards = File::ReadBackwards->new( 'file_to_read.txt' );
my $last_non_blank_line;
do {
$last_non_blank_line = $backwards->readline;
} until ! defined $last_non_blank_line || $last_non_blank_line =~ /\S/;
perl -e 'while (<>) { if ($_) {$last = $_;} } print $last;' < my_file.txt
You can use the module File::ReadBackwards in the following way:
use File::ReadBackwards ;
$bw = File::ReadBackwards->new('filepath') or
die "can't read file";
while( defined( $log_line = $bw->readline ) ) {
print $log_line ;
exit 0;
}
If they're blank, just check $log_line for a match with \n;
If the file is small, I would store it in an array and read from the end. If its large, use File::ReadBackwards module.
Here's my variant of command line perl solution:
perl -ne 'END {print $last} $last= $_ if /\S/' file.txt
No one mentioned Path::Tiny. If the file size is relativity small you can do this:
use Path::Tiny;
my $file = path($file_name);
my ($last_line) = $file->lines({count => -1});
CPAN page.
Just remember for the large file, just as #ysth said it's better to use File::ReadBackwards. The difference can be substantial.
sometimes it is more comfortable for me to run shell commands from perl code. so I'd prefer following code to resolve the case:
$result=`tail -n 1 /path/file`;
Related
I'm trying to read contents from an input file, copy only certain lines of code from the file and print in an output file.
Certain lines of code is determined by:
Code name to determine the first line (IP1_NAME or IP2_NAME)
Pattern to determine the last line (END_OF_LIST)
Input file:
IP1_NAME
/ip1name/ip1dir/ //CLIENT_NAME/ip1name/ip1dir
/ip1testname/ip1testdir/ //CLIENT_NAME/ip1testname/ip1testdir
END_OF_LIST
IP2_NAME
/ip2name/ip2dir/ //CLIENT_NAME/ip2name/ip2dir
/ip2testname/ip2testdir/ //CLIENT_NAME/ip2testname/ip2testdir
END_OF_LIST
Output file:
(If IP1_NAME is chosen and the CLIENT_NAME should be replaced by tester_ip)
/ip1name/ip1dir/ //tester_ip/ip1name/ip1dir
/ip1testname/ip1testdir/ //tester_ip/ip1testname/ip1testdir
You could use the following one-liner to pull out the lines between the two patterns:
perl -0777 -ne 'print "$1\n" while /IP1_NAME(.*?)END_OF_LIST/gs' in.txt > out.txt
Where in.txt is your input file and out.txt is the output file.
This use case is actually described in perlfaq6: Regular Expressions.
You can then modify the output file to replace CLIENT_NAME with tester_ip:
perl -pi -e 's/CLIENT_NAME/tester_ip/' y.txt
As a script instead of a one-liner, using the scalar range operator:
#/usr/bin/env perl
use warnings;
use strict;
use autodie;
use feature qw/say/;
process('input.txt', qr/^IP1_NAME$/, qr/^END_OF_LIST$/, 'tester_ip');
sub process {
my ($filename, $startpat, $endpat, $newip) = #_;
open my $file, '<', $filename;
while (my $line = <$file>) {
chomp $line;
if ($line =~ /$startpat/ .. $line =~ /$endpat/) {
next unless $line =~ /^\s/; # Skip the start and lines.
$line =~ s/^\s+//; # Remove indentation
$line =~ s/CLIENT_NAME/$newip/g; # Replace with desired value
say $line;
}
}
}
Running this on your sample input file produces:
/ip1name/ip1dir/ //tester_ip/ip1name/ip1dir
/ip1testname/ip1testdir/ //tester_ip/ip1testname/ip1testdir
I am assuming there is additional stuff in your input file, otherwise we would not have to jump through the hoops with these start and end markers as and we could just say
perl -ne "print if /^ /"
and that would be silly, right ;-)
So, the flipflop has potential problems as I stated in my comment. And while clever, it does not buy you that much in terms of readability or verbosement (verbocity?), since you have to test again anyway in order to not process the marker lines.
As long as there is no exclusive flip flop operator, I would go for a more robust solution.
my $in;
while (<DATA>) {
$in = 1, next if /^IP\d_NAME/;
$in = 0 if /^END_OF_LIST/;
if ( $in )
{
s/CLIENT_NAME/tester_ip/;
print;
}
}
__DATA__
cruft
IP1_NAME
/ip1name/ip1dir/ //CLIENT_NAME/ip1name/ip1dir
/ip1testname/ip1testdir/ //CLIENT_NAME/ip1testname/ip1testdir
END_OF_LIST
more
cruft
IP2_NAME
/ip2name/ip2dir/ //CLIENT_NAME/ip2name/ip2dir
/ip2testname/ip2testdir/ //CLIENT_NAME/ip2testname/ip2testdir
END_OF_LIST
Lore Ipsargh!
If i have a text file and i want to run two types of operations, but each operation must read each line of the text separately from the other. The only way i know how to do it is
open out,(">>out.txt");
while (<>){
#operation one
}
while (<>){
#operation two
}
close out;
but this will run only on the first while, in which the operation runs fine, but the second one will not be complete because the second while(<>) does not actually re-read the file but tries to continue from where the first while left. Which is at the end of the file. So is there another way? Or is there a way to tell the second while to start again at the beginning?
Given you mention in a comment:
perl example.pl text.txt
The answer is - don't use <> and instead open a filehandle.
my ( $filename ) = #ARVG;
open ( my $input, "<", $filename ) or die $!;
while ( <$input> ) {
print;
}
seek ( $input, 0, 0 );
while ( <$input> ) {
#something else
}
Alternatively, you can - assuming test.txt isn't particularly large - just read the whole thing into an array.
my #input_lines = <$input>;
foreach ( #input_lines ) {
#something
}
If you want to specify multiple files on the command line, you can wrap the whole thing in a foreach loop:
foreach my $filename ( #ARVG ) {
## open; while; seek; while etc.
}
Couldn't you simply use the following?
while (<>) {
operation1($_);
operation2($_);
}
If not, then I'm assuming you need to process the content of all the files using one operation before it's process by the other.
<> reads from the files listed in #ARGV, removing them as it opens them, so the simplest solution is to backup #ARGV and repopulate it.
my #argv = #ARGV;
while (<>) { operation1($_); }
#ARGV = #argv;
while (<>) { operation2($_); }
Of course, it will fail if <> reads from something other than a plain file or a symlink to a plain file. (Same goes for any solution using seek.) The only to make that work would be to load the entire file into temporary storage (e.g. memory or a temporary file). The following is the simplest example of that:
my #lines = <>;
for (#lines) { operation1($_); }
for (#lines) { operation2($_); }
If the data fits into memory:
my #lines = <>;
for ( #lines ){
# operation one
}
for ( #lines ){
# operation two
}
You can localize #ARGV before the first pass.
#!/usr/bin/env perl
use strict;
use warnings;
{
local #ARGV = #ARGV;
while (<>){
print "Pass 1: $_";
}
}
while (<>){
print "Pass 2: $_";
}
If no file handle is used with the diamond operator, Perl will examine the #ARGV special variable. If #ARGV has no elements, then the diamond operator will read from STDIN.
This is other way of achieve your requirements:
my #stdin=<>;
foreach my $item( #stdin ) {
# ...
}
foreach my $item( #stdin ) {
# ...
}
If you need to run the operation line by line, why not try something like this
sub operation_1 {
my $line = shift;
#processing for operation 1
}
sub operation_2 {
my $line = shift;
#processing for operation 2
}
while(<>) {
my $line = $_;
chomp($line);
operation_1($line);
operation_2($line);
}
If you were reading from an actual file, you could use
seek FILEHANDLE,0,0;
However, you are using stdin and I don't think that it's possible to rewind stdin and start over.
I want to write a script that takes a CSV file, deletes its first row and creates a new output csv file.
This is my code:
use Text::CSV_XS;
use strict;
use warnings;
my $csv = Text::CSV_XS->new({sep_char => ','});
my $file = $ARGV[0];
open(my $data, '<', $file) or die "Could not open '$file'\n";
my $csvout = Text::CSV_XS->new({binary => 1, eol => $/});
open my $OUTPUT, '>', "file.csv" or die "Can't able to open file.csv\n";
my $tmp = 0;
while (my $line = <$data>) {
# if ($tmp==0)
# {
# $tmp=1;
# next;
# }
chomp $line;
if ($csv->parse($line)) {
my #fields = $csv->fields();
$csvout->print($OUTPUT, \#fields);
} else {
warn "Line could not be parsed: $line\n";
}
}
On the perl command line I write: c:\test.pl csv.csv and it doesn't create the file.csv output, but when I double click the script it creates a blank CSV file. What am I doing wrong?
Your program isn't ideally written, but I can't tell why it doesn't work if you pass the CSV file on the command line as you have described. Do you get the errors Could not open 'csv.csv' or Can't able to open file.csv? If not then the file must be created in your current directory. Perhaps you are looking in the wrong place?
If all you need to do is to drop the first line then there is no need to use a module to process the CSV data - you can handle it as a simple text file.
If the file is specified on the command line, as in c:\test.pl csv.csv, you can read from it without explicitly opening it using the <> operator.
This program reads the lines from the input file and prints them to the output only if the line counter (the $. variable) isn't equal to one).
use strict;
use warnings;
open my $out, '>', 'file.csv' or die $!;
while (my $line = <>) {
print $out $line unless $. == 1;
}
Yhm.. you don't need any modules for this task, since CSV ( comma separated value ) are simply text files - just open file, and iterate over its lines ( write to output all lines except particular number, e.g. first ). Such task ( skip first line ) is so simple, that it would be probably better to do it with command line one-liner than a dedicated script.
quick search - see e.g. this link for an example, there are numerous tutorials about perl input/output operations
http://learn.perl.org/examples/read_write_file.html
PS. Perl scripts ( programs ) usually are not "compiled" into binary file - they are of course "compiled", but, uhm, on the fly - that's why /usr/bin/perl is called rather "interpreter" than "compiler" like gcc or g++. I guess what you're looking for is some editor with syntax highlighting and other development goods - you probably could try Eclipse with perl plugin for that ( cross platform ).
http://www.eclipse.org/downloads/
http://www.epic-ide.org/download.php/
this
user#localhost:~$ cat blabla.csv | perl -ne 'print $_ if $x++; '
skips first line ( prints out only if variable incremented AFTER each use of it is more than zero )
You are missing your first (and only) argument due to Windows.
I think this question will help you: #ARGV is empty using ActivePerl in Windows 7
I'm trying to remove one line from a text file. Instead, what I have wipes out the entire file. Can someone point out the error?
removeReservation("john");
sub removeTime() {
my $name = shift;
open( FILE, "<times.txt" );
#LINES = <FILE>;
close(FILE);
open( FILE, ">times.txt" );
foreach $LINE (#LINES) {
print NEWLIST $LINE unless ( $LINE =~ m/$name/ );
}
close(FILE);
print("Reservation successfully removed.<br/>");
}
Sample times.txt file:
04/15/2012&08:00:00&bob
04/15/2012&08:00:00&john
perl -ni -e 'print unless /whatever/' filename
Oalder's answer is correct, but he should have tested whether the open statements succeeded or not. If the file times.txt doesn't exist, your program would continue on its merry way without a word of warning that something terrible has happened.
Same program as oalders' but:
Testing the results of the open.
Using the three part open statement which is more goof proof. If your file name begins with > or |, your program will fail with the old two part syntax.
Not using global file handles -- especially in subroutines. File handles are normally global in scope. Imagine if I had a file handle named FILE in my main program, and I was reading it, I called this subroutine. That would cause problems. Use locally scoped file handle names.
Variable names should be in lowercase. Constants are all uppercase. It's just a standard that developed over time. Not following it can cause confusion.
Since oalders put the program in a subroutine, you should pass the name of your file in the subroutine as well...
Here's the program:
#!/usr/bin/env perl
use strict;
use warnings;
removeTime( "john", "times.txt" );
sub removeTime {
my $name = shift;
my $time_file = shift;
if (not defined $time_file) {
#Make sure that the $time_file was passed in too.
die qq(Name of Time file not passed to subroutine "removeTime"\n);
}
# Read file into an array for processing
open( my $read_fh, "<", $time_file )
or die qq(Can't open file "$time_file" for reading: $!\n);
my #file_lines = <$read_fh>;
close( $read_fh );
# Rewrite file with the line removed
open( my $write_fh, ">", $time_file )
or die qq(Can't open file "$time_file" for writing: $!\n);
foreach my $line ( #file_lines ) {
print {$write_fh} $line unless ( $line =~ /$name/ );
}
close( $write_fh );
print( "Reservation successfully removed.<br/>" );
}
It looks like you're printing to a filehandle which you have not yet defined. At least you haven't defined it in your sample code. If you enable strict and warnings, you'll get the following message:
Name "main::NEWLIST" used only once: possible typo at remove.pl line 16.
print NEWLIST $LINE unless ($LINE =~ m/$name/);
This code should work for you:
#!/usr/bin/env perl
use strict;
use warnings;
removeTime( "john" );
sub removeTime {
my $name = shift;
open( FILE, "<times.txt" );
my #LINES = <FILE>;
close( FILE );
open( FILE, ">times.txt" );
foreach my $LINE ( #LINES ) {
print FILE $LINE unless ( $LINE =~ m/$name/ );
}
close( FILE );
print( "Reservation successfully removed.<br/>" );
}
A couple of other things to note:
1) Your sample code calls removeReservation() when you mean removeTime()
2) You don't require the round brackets in your subroutine definition unless your intention is to use prototypes. See my example above.
This is in the FAQ.
How do I change, delete, or insert a line in a file, or append to the beginning of a file?
It's always worth checking the FAQ.
Just in case someone wants to remove all lines from a file.
For example, a file (4th line is empty; 5th line has 3 spaces):
t e st1
test2 a
e
aa
bb bb
test3a
cc
To remove lines which match a pattern some might use:
# Remove all lines with a character 'a'
perl -pi -e 's/.*a.*//' fileTest && sed -i '/^$/d' fileTest;
The result:
t e st1
e
bb bb
cc
Related:
perl -h
# -p assume loop like -n but print line also, like sed
# -i[extension] edit <> files in place (makes backup if extension supplied)
# -e program one line of program (several -e's allowed, omit programfile)
sed -h
# -i[SUFFIX], --in-place[=SUFFIX]
# edit files in place (makes backup if SUFFIX supplied)
Reference 1, Reference 2
I'm trying to open a file, search for a specific string in the file to begin my search from and then performing a replacement on a string later on in the file. For example, my file looks like:
Test Old
Hello World
Old
Data
Begin_search_here
New Data
Old Data
New Data
I want to open the file, begin my search from "Begin_search_here" and then replace the next instance of the word "Old" with "New". My code is shown below and I'm correctly finding the string, but for some reason I'm not writing in the correct location.
open(FILE, "+<$filename") || die "problem opening file";
my search = 0;
while(my $line = <FILE>)
{
if($line =~ m/Begin_search_here/)
{
$search = 1;
}
if($search == 1 && $line =~m/Old/)
{
$line = s/Old/New/;
print FILE $line
}
close FILE;
Here ya go:
local $^I = '.bak';
local #ARGV = ($filename);
local $_;
my $replaced = 0;
while (<>) {
if (!$replaced && /Begin_search_here/ .. $replaced) {
$replaced = s/Old/New/;
}
print;
}
Explanation:
Setting the $^I variable enables inplace editing, just as if you had run perl with the -i flag. The original file will be saved with the same name as the original file, but with the extension ".bak"; replace ".bak" with "" if you don't want a backup made.
#ARGV is set to the list of files to do inplace editing on; here just your single file named in the variable $filename.
$_ is localized to prevent overwriting this commonly-used variable in the event this code snippet occurs in a subroutine.
The flip-flop operator .. is used to figure out what part of the file to perform substitutions in. It will be false until the first time a line matching the pattern Begin_search_here is encountered, and then will remain true until the first time a substitution occurs (as recorded in the variable $replaced), when it will turn off.
You would probably be best served by opening the input file in read mode (open( my $fh, '<', $file ) or die ...;), and writing the modified text to a temporary output file, then copying the temporary file overtop of the input file when you're done doing your processing.
You are misusing the random-access file mode. By the time you update $line and say print FILE $line, the "cursor" of your filehandle is already positioned at the beginning of the next line. So the original line is not changed and the next line is over-written, instead of overwriting the original line.
Inplace editing (see perlrun) looks like it would be well suited for this problem.
Otherwise, you need to read up on the tell function to save your file position before you read a line and seek back to that position before you rewrite the line. Oh, and the data that you write must be exactly the same size as the data you are overwriting, or you will totally fubar your file -- see this question.
I have done a number of edits like this, that I came up with a generic (yet stripped-down) strategy:
use strict;
use warnings;
use English qw<$INPLACE_EDIT>;
use Params::Util qw<_CODE>;
local $INPLACE_EDIT = '.bak';
local #ARGV = '/path/to/file';
my #line_actions
= ( qr/^Begin_search_here/
, qr/^Old Data/ => sub { s/^Old/New/ }
);
my $match = shift #line_actions;
while ( <> ) {
if ( $match and /$match/ ) {
if ( _CODE( $line_actions[0] )) {
shift( #line_actions )->( $_ );
}
$match = shift #line_actions;
}
print;
}
This works. It will, as you specified, only replaces one occurrence.
#! /usr/bin/perl -pi.bak
if (not $match_state) {
if (/Begin_search_here/) {
$match_state = "accepting";
}
}
elsif ($match_state eq "accepting") {
if (s/Old/New/) {
$match_state = "done";
}
}
Be very careful about editing a file in place. If the data you're replacing is a different length, you wreck the file. Also, if your program fails in the middle, you end up with a destroyed file.
Your best bet is to read in each line, process the line, and write each line to a new file. This will even allow you to run your program, examine the output, and if you have an error, fix it and rerun the program. Then, once everything is okay, add in the step to move your new file to the old name.
I've been using Perl since version 3.x, and I can't think of a single time I modified a file in place.
use strict;
use warnings;
open (INPUT, "$oldfile") or die qq(Can't open file "$oldFile" for reading);
open (OUTPUT, "$oldfile.$$") or die qq(Can't open file "$oldfile.$$" for writing);
my $startFlag = 0;
while (my $line = <INPUT>) {
if ($line ~= /Begin_search_here/) {
$startFlag = 1;
}
if ($startFlag) {
$line =~ s/New/Old/;
}
print OUTPUT "$line";
}
#
# Only implement these two steps once you've tested your program
#
unlink $oldfile;
rename $oldfile.$$", $oldfile;