Read file into chunks and process the chunks in parallel - perl

Question

So, basically I want to read a file into a hash, but since the file is huge and doesn't fit into RAM I split it into chunks, process the data (the search_f2 sub) and read the next chunk of data. This seem to work, but of course it takes only one core.
Is there an easy way to fork the search_f2 sub?
I've tried a naive way with Parallel::Forkmanager but it doesn't work as far as I see.
Any hint how to achieve that? I actually don't need to return from the forked sub, it would be sufficient if it prints the result to STDOUT.
The file1 structure is the following ( basically the result of tar -tf command):
tarfile1.tar
gzip1.gz
<skip>
gzipX.gz
<skip>
tarfileX.tar
<some random number of gz files>
the file2 is just plain line break separated list of gzipX.gz files
the perl code:
#!/usr/bin/perl
use strict;
use warnings;
use feature qw(say);
use Data::Dumper;
use Parallel::ForkManager;
my $file1 = $ARGV[0] // die "Need a file as argument";
my $file2 = $ARGV[1] // die "Need a file as argument";
my $fd1 = read_f($file1);
my %hdata;
my $tarfile;
my $index = 0;
my $pm = Parallel::ForkManager->new(10);
while (my $line = <$fd1>) {
chomp $line;
if ( $line =~ m/^somepattern.*tar$/ ){
$tarfile = $line;
$index++;
}
if (++$index >= '100') {
say "\tForking a process";
my $pid = $pm->start and do {
$index = 0;
%hdata = ();
next;
};
search_f2(\%hdata);
$pm->finish;
}
push #{$hdata{$tarfile}},$line if $line =~ m/.*\.gz$/;
}
close $fd1;
#last search
search_f2(\%hdata);
sub search_f2{
my ($h) = #_;
my %hdata = %$h;
my $fd2 = read_f($file2);
while (my $ciffile = <$fd2>) {
chomp $ciffile;
foreach my $tarfile (keys %hdata) {
my $values = $hdata{$tarfile};
if (grep (/$ciffile/, #$values)) {
say "$tarfile";
delete $hdata{$tarfile};
last;
}
}
}
close $fd2;
return;
}
sub read_f {
my $file = shift;
die "Can't open file $file: $!\n" if ! -e $file;
# gzip happily parses plain files as well
open my $fh, "pigz -fdc $file|" or die "Can't open file $file: $!\n";
return $fh if $fh;
}

Answer 1

I take the quest to be the following: read a certain number of lines from a file and process each such chunk of text in its own fork. I'm not quite sure of some details in the question so here is a basic demo, hopefully to serve as a template.
Keep the number of processes to 3, and process batches ("chunks") of 2 lines in each.
use warnings;
use strict;
use feature qw(say state);
use Parallel::ForkManager;
my $file = shift // die "Usage: $0 filename\n";
my $pm = Parallel::ForkManager->new(3);
open my $fh, '<', $file or die $!;
my ($chunk, $num_lines);
while (my $line = <$fh>) {
chomp $line;
say "Processing line: |$line|";
$chunk .= $line;
if (++$num_lines >= 2) {
say "\tForking a process";
$pm->start and do {
$num_lines = 0;
$chunk = '';
next;
};
proc_chunk($chunk);
$pm->finish;
}
}
$pm->wait_all_children;
sub proc_chunk {
my ($chunk) = #_;
my $line_nos = join ' ', $chunk =~ /#([0-9]+)/g;
say "\t\tin a fork, processing chunk with lines: $line_nos";
sleep 10;
say "\t\t\t... done with fork";
}
In P::FM, that $pm->start and next; forks a process and the parent immediately jumps to the next iteration of the loop. So any needed resetting of variables need be done right there and I use a do { ... } block for it.†
The sub sleeps so that we can see the group of forks exiting practically together, what happens because the processing here is so quick. (In fact P::FM forks a new process as soon as one is finished, to keep the given number running. It doesn't wait for the whole batch to finish first, unless wait_for_available_procs is set up, see an example and a lot more detail here.)
This prints
Processing line: |This is line #1|
Processing line: |This is line #2|
Forking a process
Processing line: |This is line #3|
Processing line: |This is line #4|
Forking a process
Processing line: |This is line #5|
Processing line: |This is line #6|
Forking a process
Processing line: |This is line #7|
Processing line: |This is line #8|
Forking a process
in a fork, processing chunk with lines: 1 2
in a fork, processing chunk with lines: 3 4
in a fork, processing chunk with lines: 5 6
... done with fork
... done with fork
... done with fork
Processing line: |This is line #9|
Processing line: |This is line #10|
Forking a process
Processing line: |This is line #11|
Processing line: |This is line #12|
Forking a process
Processing line: |This is line #13|
Processing line: |This is line #14|
Forking a process
in a fork, processing chunk with lines: 7 8
in a fork, processing chunk with lines: 9 10
in a fork, processing chunk with lines: 11 12
... done with fork
... done with fork
... done with fork
Processing line: |This is line #15|
Processing line: |This is line #16|
Forking a process
Processing line: |This is line #17|
Processing line: |This is line #18|
Forking a process
Processing line: |This is line #19|
in a fork, processing chunk with lines: 13 14
Processing line: |This is line #20|
Forking a process
in a fork, processing chunk with lines: 15 16
in a fork, processing chunk with lines: 17 18
^C
[...etc...]
† The original version of the question didn't have that do { } block but had ... and next, thus with parent directly bailing out. (I see that the question was now edited to include it.)