Copy data from huge files while they are open - perl

Question

I am trying to merge data from huge files to a combined file using Perl.
File will be in open condition and large amount of data is continuously being added to the files. Appending around 50,000 lines per minute.
The files are stored in a network shared folder accessed by between 10 and 30 machines.
These are JTL files generated by JMeter.
This merge runs every minute for about 6 or 7 hours, and the time taken should not be more than 30 to 40 seconds.
The process is triggered every minute by a Web Application deployed in a Linux machine.
I have written a script which stores the last line added by the individual files to the combined file in separate files.
This works fine up to 15 minutes but constantly increase the merge time.
My script
#!/usr/bin/perl
use File::Basename;
use File::Path;
$consolidatedFile = $ARGV[0];
$testEndTimestamp = $ARGV[1];
#csvFiles = #ARGV[ 2 .. $#ARGV ];
$testInProcess = 0;
$newMerge = 0;
$lastLines = "_LASTLINES";
$lastLine = "_LASTLINE";
# time() gives current time timestamp
if ( time() <= $testEndTimestamp ) {
$testInProcess = 1;
}
# File exists, has a size of zero
if ( -z $consolidatedFile ) {
mkdir $consolidatedFile . $lastLines;
$newMerge = 1;
}
open( CONSOLIDATED, ">>" . $consolidatedFile );
foreach my $file ( #csvFiles ) {
open( INPUT, "<" . $file );
#linesArray = <INPUT>;
close INPUT;
if ( $newMerge ) {
print CONSOLIDATED #linesArray[ 0 .. $#linesArray - 1 ];
open my $fh, ">", $consolidatedFile . $lastLines . "/" . basename $file . $lastLine;
print $fh $linesArray[ $#linesArray - 1 ];
close $fh;
}
else {
open( AVAILABLEFILE, "<" . $consolidatedFile . $lastLines . "/" . basename $file . $lastLine );
#lineArray = <AVAILABLEFILE>;
close AVAILABLEFILE;
$availableLastLine = $lineArray[0];
open( FILE, "<" . $file );
while ( <FILE> ) {
if ( /$availableLastLine/ ) {
last;
}
}
#grabbed = <FILE>;
close( FILE );
if ( $testInProcess ) {
if ( $#grabbed > 0 ) {
pop #grabbed;
print CONSOLIDATED #grabbed;
open( AVAILABLEFILE, ">" . $consolidatedFile . $lastLines . "/" . basename $file . $lastLine );
print AVAILABLEFILE $grabbed[ $#grabbed - 1 ];
}
close AVAILABLEFILE;
}
else {
if ( $#grabbed >= 0 ) {
print CONSOLIDATED #grabbed;
}
}
}
}
close CONSOLIDATED;
if ( !$testInProcess ) {
rmtree $consolidatedFile . $lastLines;
}
I need to optimize the script in order to reduce the time.
Is it possible to store last line in a cache?
Can anyone suggest another way for this type of merging?
Another script which stores last line in cache instead of file.
Even this does not complete merge within 1 min.
#!/usr/bin/perl
use CHI;
use File::Basename;
use File::Path;
my $cache = CHI->new(
driver => 'File',
root_dir => '/path/to/root'
);
$consolidatedFile = $ARGV[0];
$testEndTimestamp = $ARGV[1];
#csvFiles = #ARGV[ 2 .. $#ARGV ];
$testInProcess = 0;
$newMerge = 0;
$lastLines = "_LASTLINES";
$lastLine = "_LASTLINE";
# time() gives current time timestamp
if ( time() <= $testEndTimestamp ) {
$testInProcess = 1;
}
# File exists, has a size of zero
if ( -z $consolidatedFile ) {
$newMerge = 1;
}
open( CONSOLIDATED, ">>" . $consolidatedFile );
foreach my $file (#csvFiles) {
$fileLastLineKey =
$consolidatedFile . $lastLines . "_" . basename $file . $lastLine;
open( INPUT, "<" . $file );
#linesArray = <INPUT>;
close INPUT;
if ($newMerge) {
print CONSOLIDATED #linesArray[ 0 .. $#linesArray - 1 ];
$fileLastLine = $linesArray[ $#linesArray - 1 ];
$cache->set( $fileLastLineKey, $fileLastLine );
}
else {
$availableLastLine = $cache->get($fileLastLineKey);
open( FILE, "<" . $file );
while (<FILE>) {
if (/$availableLastLine/) {
last;
}
}
#grabbed = <FILE>;
close(FILE);
if ($testInProcess) {
if ( $#grabbed > 0 ) {
pop #grabbed;
print CONSOLIDATED #grabbed;
$fileLastLine = $grabbed[ $#grabbed - 1 ];
$cache->set( $fileLastLineKey, $fileLastLine );
}
}
else {
if ( $#grabbed >= 0 ) {
print CONSOLIDATED #grabbed;
$cache->remove($fileLastLineKey);
}
}
}
}
close CONSOLIDATED;
I am thinking of reading files from last line to required line and copy those lines to consolidated file.
Can anyone suggest on this???

Answer 1

You may want to try open the file in binmode and read it blockwise in a loop. This usually offers significant performance improvements. The following functions is an example, here i put at maximum $maxblocks blocks of a file on the array, from block $offset on, in an array passed as reference. Note that the last block may not contain the entire $block bytes when the file is not large enough.
sub file2binarray {
my $file=shift;
my $array=shift;
my $maxblocks=shift;
my $offset=shift;
my $block=2048;
$offset=0 if ((!defined($offset)) || ($offset !~/^\s*\d+\s*$/o));
$maxblocks="ALL"
if (!defined($maxblocks) || ($maxblocks!~/^\s*\d+\s*$/o));
my $size=(stat($file))[7];
my $mb=$size/$block;
$mb++ if ($mb*$block<$size);
$maxblocks=$mb-$offset if(($maxblocks eq "ALL")||
($maxblocks>$mb-$offset));
$offset*=$block;
open(IN,"$file") || die("Cannot open file <$file>\n");
binmode(IN);
$bytes_read=$block;
seek(IN,$offset,0);
my ($blk,$bytes_read,$buffer)=(0,0,"");
while (($bytes_read==$block)&& ($blk<$maxblocks)){
$bytes_read=sysread(IN,$buffer,$block);
push(#$array,$buffer);
$blk++;
}
close(IN);
}
To read the entire file at ones, e.g. you call it like this
my #array;
my $filename="somefile";
file2binarray ($filename,\#array,"ALL",0);
but probably you'd rather call it in a loop with some bookkeeping over the offset, and parse the array in between subsequent calls.
Hope this helps.