perl manipulate with big files - perl

Question

I'm working on a 16GB file and a small file.
I tried to load both files into memory. Then, I moved on each line in the big file and validate something in the small file (for each line in the big file I iterated on the small one).
This is my code
local $/ = undef;
open my $fh1, '<', $in or die "error opening $in: $!";
my $input_file = do { local $/; <$fh1> };
local $/ = undef;
open my $fh2, '<', $handle or die "error opening $handle: $!";
my $handle_file = do { local $/; <$fh2> };
my $counter_yes = 0;
my $counter_no = 0;
my $flag = 0;
my #lines1 = split /\n/, $input_file;
foreach my $line( #lines1 ) {
my #f = split('\t', $line); # $f[0] and $f[1]
print "f0 and f1 are: $f[0] and $f[1]\n";
my #lines2 = split /\n/, $handle_file;
foreach my $input ( #lines2 ){
#print "line2 is: $input\n";
my #sp = split /:/, $input; # $sp[0] and $sp[1]
if ( $sp[0] eq $f[0] ){
my #r = split /-/, $sp[1];
if ( ($f[1] >= $r[0]) && ($f[1] <= $r[1]) ){
$flag = 1;
$counter_yes = $counter_yes;
last;
}
}
}
if ( $flag == 0 ){
$counter_no = $counter_no ;
}
}
While I running it I get the error
Split loop at script.pl line 30, <$fh2> chunk 1
What can be the reason?

Answer 1

You can run perldoc perldiag to learn what some built in errors and warnings mean.
Split loop
(P) The split was looping infinitely. (Obviously, a split
shouldn't iterate more times than there are characters of input,
which is what happened.) See "split" in perlfunc.
The string you're splitting on is so large, Perl thought it was iterating infinitely. When Perl has split a string more times than the length of the string + 10, it gives this error assuming its in an infinite loop. Unfortunately for you, it stored that number as a 32 bit integer which can only hold up to 2 billion and change. Your string is over 16 billion so the result will be unpredictable.
This was recently fixed in 5.20 along with many other related problems with working with strings over 2G in size. So if you upgrade Perl your code will "work".
However, your code is hideously inefficient and will crush the memory of most machines causing it to slow down terribly as it swaps to disk. At minimum you should only slurp in the small file and read the 16 gig file line by line.
my #small_data = <$small_fh>;
chomp #small_data;
while( my $big = <$big_fh> ) {
chomp $big;
for my $small (#small_data) {
...
}
}
But even that is going to be terribly inefficient, if your small file contains 1000 lines then that loop will run 16 trillion times!
Since it seems like you're checking to see if entries in the big file are in the small file, you're better off turning the entries in the small file into a hash table.
my %fields;
while( my $line = <$small_fh> ) {
chomp $line;
my #sp = split /:/, $line;
$fields{$sp[0]} = $sp[1];
}
Now you can iterate through the big file and just do a hash lookup.
while( my $line = <$big_fh> ) {
chomp $line;
my #f = split('\t', $line);
if( defined $fields{$f[0]} ) {
...
}
}

Answer 2

Why are you reading the whole file into one big string and splitting it into an array of lines, when you could be reading it into an array of lines to begin with? And why do you do it over and over again for the second file? You can just
chomp(my #lines1 = <$fh>);
chomp(my #lines2 = <$fh2>);
at the top of your program and eliminate $input_file and $handle_file which are otherwise unused, and all of the $/ nonsense. This could very well be the source of the problem, since the error message indicates that split is producing "too many" fields.

Answer 3

I'm working on a 16GB file and a small file.
I tried to load both files into memory.
Do you have 16GB of memory? Actually, your code requires more than 32GB of memory.
Split loop at script.pl line 30, chunk 1
I can't duplicate that error. Perl errors are usually pretty descriptive, yet that isn't even comprehensible.
Next, if you had this in your code:
my $x = 10;
#nothing changes $x
#in these
#lines
$x = 10;
What would be the purpose of the last line? Yet, you did this:
$/ = undef;
#Nothing changes $/
#in these lines
$/ = undef;
Next, all perl programs should start with the following lines:
<guess>
If you don't know, then you need to buy a beginning perl book.