Storing large file in array is not possible? - perl

Question

There is a process which stores the file in an array. Unfortunately when the file is too big (Let's say 800K lines or more than 60 MB) an error is returned like "Out of memory!". Is there any solution to this? For example the following code throw "Out of memory!".
#! /usr/bin/perl
die unless (open (INPUT, "input.txt"));
#file=<INPUT>; # It fails here
print "File stored in array\n"; # It never reaches here
$idx=0;
while ($idx < #file) {
$idx++;
}
print "The line count is = $idx\n";

Answer 1

I'd use Tie::File for that:
use Tie::File;
my #file
tie #file, 'Tie::File', "input.txt";
print "File reflected in array\n";
print "The line count is ", scalar(#file);

Answer 2

Most of the time, you don't need to read in the whole file at once. The readline operator returns only one line at a time when called in scalar context:
1 while <INPUT>; # read a line, and discard it.
say "The line count is = $.";
The $. special variable is the line number of the last read filehandle.
Edit: Line counting was just an example
Perl has no problem with large arrays, it just seems that your system doesn't have enough memory. Be aware that Perl arrays use more memory than C arrays, as a scalar allocate additional memory for flags etc, and because arrays grow in increasing steps.
If memory is an issue, you have to transform your algorithm from one that has to load a whole file into memory to one that only keeps one line at a time.
Example: Sorting a multi-gigabyte file. The normal approach print sort <$file> won't work here. Instead, we sort portions of the file, write them to tempfiles, and then switch between the tempfiles in a clever way to produce one sorted output:
use strict; use warnings; use autodie;
my $blocksize = shift #ARGV; # take lines per tempfile as command line arg
mkdir "/tmp/$$"; # $$ is the process ID variable
my $tempcounter = 0;
my #buffer;
my $save_buffer = sub {
$tempcounter++;
open my $tempfile, ">", "/tmp/$$/$tempcounter";
print $tempfile sort #buffer;
#buffer = ();
};
while (<>) {
push #buffer, $_;
$save_buffer->() if $. % $blocksize == 0;
}
$save_buffer->();
# open all files, read 1st line
my #head =
grep { defined $_->[0] }
map { open my $fh, "<", $_; [scalar(<$fh>), $fh] }
glob "/tmp/$$/*";
# sort the line-file pairs, pick least
while((my $least, #head) = sort { $a->[0] cmp $b->[0] } #head){
my ($line, $fh) = #$least; print $line;
# read next line
if (defined($line = <$fh>)){
push #head, [$line, $fh];
}
}
# clean up afterwards
END {
unlink $_ for glob "/tmp/$$/*";
rmdir "/tmp/$$";
}
Could be called like $ ./sort-large-file 10000 multi-gig-file.txt >sorted.txt.
This general approach can be applied to all kinds of problems. This is a “divide and conquer” strategy: If the problem is too big, solve a smaller problem, and then combine the pieces.