How to optimize two-dimensional hash traversing in Perl? - perl

Question

I have a hash of hashes %signal_db. A typical element is: $signal_db{$cycle}{$key}. There are 10,000s of signals, and 10,000s of keys.
Is there any way to optimize (timewise) this piece of code:
foreach my $cycle (sort numerically keys %signal_db) {
foreach my $key (sort keys %{$signal_db{$cycle}}) {
print $signal_db{$cycle}{$key}.$key."\n";
}
}
The elements have to be printed in the same order as in my code.

Answer 1

Two micro optimizations: map inner hash instead of constant dereferencing and buffer instead of constant print. It's possible to get rid of sorting using alternative storage formats, tested two variants. Results:
Rate original try3 alternative alternative2
original 46.1/s -- -12% -21% -32%
try3 52.6/s 14% -- -10% -22%
alternative 58.6/s 27% 11% -- -13%
alternative2 67.5/s 46% 28% 15% --
Conclusion:
It's better to use presorted storage format, but without C win would probably be within 100% (on my test dataset). Provided information about data suggests that keys in outer hash are almost sequential numbers, so this cries for array.
Script:
#!/usr/bin/env perl
use strict; use warnings;
use Benchmark qw/timethese cmpthese/;
my %signal_db = map { $_ => {} } 1..1000;
%$_ = map { $_ => $_ } 'a'..'z' foreach values %signal_db;
my #signal_db = map { { cycle => $_ } } 1..1000;
$_->{'samples'} = { map { $_ => $_ } 'a'..'z' } foreach #signal_db;
my #signal_db1 = map { $_ => [] } 1..1000;
#$_ = map { $_ => $_ } 'a'..'z' foreach grep ref $_, #signal_db1;
use Sort::Key qw(nsort);
sub numerically { $a <=> $b }
my $result = cmpthese( -2, {
'original' => sub {
open my $out, '>', 'tmp.out';
foreach my $cycle (sort numerically keys %signal_db) {
foreach my $key (sort keys %{$signal_db{$cycle}}) {
print $out $signal_db{$cycle}{$key}.$key."\n";
}
}
},
'try3' => sub {
open my $out, '>', 'tmp.out';
foreach my $cycle (map $signal_db{$_}, sort numerically keys %signal_db) {
my $tmp = '';
foreach my $key (sort keys %$cycle) {
$tmp .= $cycle->{$key}.$key."\n";
}
print $out $tmp;
}
},
'alternative' => sub {
open my $out, '>', 'tmp.out';
foreach my $cycle (map $_->{'samples'}, #signal_db) {
my $tmp = '';
foreach my $key (sort keys %$cycle) {
$tmp .= $cycle->{$key}.$key."\n";
}
print $out $tmp;
}
},
'alternative2' => sub {
open my $out, '>', 'tmp.out';
foreach my $cycle (grep ref $_, #signal_db1) {
my $tmp = '';
foreach (my $i = 0; $i < #$cycle; $i+=2) {
$tmp .= $cycle->[$i+1].$cycle->[$i]."\n";
}
print $out $tmp;
}
},
} );

Answer 2

my %signal_db = map {$_ => {1 .. 1000}} 1 .. 1000;
sub numerically {$a <=> $b}
sub orig {
my $x;
foreach my $cycle (sort numerically keys %signal_db) {
foreach my $key (sort keys %{$signal_db{$cycle}}) {
$x += length $signal_db{$cycle}{$key}.$key."\n";
}
}
}
sub faster {
my $x;
our ($cycle, $key, %hash); # move allocation out of the loop
local *hash; # and use package variables which are faster to alias into
foreach $cycle (sort {$a <=> $b} # the {$a <=> $b} literal is optimized
keys %signal_db) {
*hash = $signal_db{$cycle}; # alias into %hash
foreach $key (sort keys %hash) {
$x += length $hash{$key}.$key."\n"; # simplify the lookup
}
}
}
use Benchmark 'cmpthese';
cmpthese -5 => {
orig => \&orig,
faster => \&faster,
};
which gets:
Rate orig faster
orig 2.56/s -- -15%
faster 3.03/s 18% --
Not a huge gain, but it is something. There isn't much more you can optimize without changing your data structure to use presorted arrays. (or writing the whole thing in XS)
Switching the foreach loops to use external package variables saves a little bit of time since perl does not have to create lexicals in the loop. Also package variables seem to be a bit faster to alias into. Reducing the inner lookup to a single level also helps.
I assume you are printing to STDOUT and then redirecting the output to a file? If so, using Perl to open the output file directly and then printing to that handle may allow for improvements in file IO performance. Another micro-optimization could be to experiment with different record sizes. For example, does it save any time to build an array in the inner loop, then join / print it at the bottom of the outer loop? But that is something that is fairly device dependent (and possibly pointless due to other IO caching layers), so I will leave that test up to you.

Answer 3

I'd first experiment with the Sort::Key module because sorting takes longer than simple looping and printing. Also, if the inner hashes keys are (mostly) identical, then you should simply presort them, but I'll assume this isn't the case or else you'd be doing that already.
You should obviously try assigning $signal_db{$cycle} to a reference too. You might find that each is faster than keys plus retrieval as well, especially if used with Sort::Key. I'd check if map runs faster than foreach too, probably the same, but who knows. You might find print runs faster if you pass it a list or call it multiple times.
I haven't tried this code but throwing together all these ideas except each gives :
foreach my $cycle (nsort keys %signal_db) {
my $r = $signal_db{$cycle};
map { print ($r->{$_},$_,"\n"); } (nsort keys %$r);
}
There is an article about sorting in perl here, check out the Schwartzian Transform if you wish to see how one might use each.
If your code need not be security conscious, then you could conceivably disable Perl's protection against algorithmic complexity attacks by setting PERL_HASH_SEED or related variables and/or recompile Perl with altered setting, so that perl's keys and values commands returned the keys or values in sorted order already, thus saving you considerable time sorting them. But please watch this 28C3 talk before doing so. I donno if this'll even work either, you'd need to read this part of Perl's source code, maybe easier just implementing your loop in C.