Optimize Perl script - runs too slow on 40GB+ files - perl

Question

I made the following Perl script to handle some file manipulation at work, but it's running far too slowly at the minute to be put in production.
I don't know Perl very well (not one of my languages), so can someone help me identify and replace parts of this script that would be slow given it's processing ~40 million lines?
Data being piped in is in the format:
col1|^|col2|^|col3|!|
col1|^|col2|^|col3|!|
... 40 million of these.
The date_cols array is calculated before this part of the script and basically holds the index of columns containing dates in the pre-converted format.
Here's the part of the script that will be executed for every input row. I've cleaned it up a little and added comments, but let me know if anything else is needed:
## Read from STDIN until no more lines are arailable.
while (<STDIN>)
{
## Split by field delimiter
my #fields = split('\|\^\|', $_, -1);
## Remove the terminating delimiter from the final field so it doesn't
## interfere with date processing.
$fields[-1] = (split('\|!\|', $fields[-1], -1))[0];
## Cycle through all column numbres in date_cols and convert date
## to yyyymmdd
foreach $col (#date_cols)
{
if ($fields[$col] ne "")
{
$fields[$col] = formatTime($fields[$col]);
}
}
print(join('This is an unprintable ASCII control code', #fields), "\n");
}
## Format the input time to yyyymmdd from 'Dec 26 2012 12:00AM' like format.
sub formatTime($)
{
my $col = shift;
if (substr($col, 4, 1) eq " ") {
substr($col, 4, 1) = "0";
}
return substr($col, 7, 4).$months{substr($col, 0, 3)}.substr($col, 4, 2);
}

Answer 1

If written purely for efficiency, I'd write your code like this:
sub run_loop {
local $/ = "|!|\n"; # set the record input terminator
# to the record seperator of our problem space
while (<STDIN>) {
# remove the seperator
chomp;
# Split by field delimiter
my #fields = split m/\|\^\|/, $_, -1;
# Cycle through all column numbres in date_cols and convert date
# to yyyymmdd
foreach $col (#date_cols) {
if ($fields[$col] ne "") {
# $fields[$col] = formatTime($fields[$col]);
my $temp = $fields[$col];
if (substr($temp, 4, 1) eq " ") {
substr($temp, 4, 1) = "0";
}
$fields[$col] = substr($temp, 7, 4).$months{substr($temp, 0, 3)}.substr($temp, 4, 2);
}
}
print join("\022", #fields) . "\n";
}
}
The optimizations are:
Using chomp to remove the |!|\n string at the end
Inlining the formatTime sub.
Subroutine calls are extremely expensive in Perl. If subs have to be used very efficiently, prototype checking can be disabled with the &subroutine(#args) syntax. If #args are ommited, the current arguments #_ are visible to the called sub. This can lead to bugs or additional performance. Use wisely. The goto &subroutine; syntax can be used as well, but this meddles with return (basically a tail call). Do not use.
Further optimizations could include removing the hash lookup %months, as hashing is expensive.

Answer 2

You'll have to benchmark on your data set to compare, but you can throw a regex at it. (Made all the worse by your very regex-unfriendly field and record separators!)
my $i = 0;
our %months = map { $_ => sprintf('%02d', ++$i) } qw(Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec);
while (<DATA>) {
s! \|\^\| !\022!xg; # convert field separator
s/ \| !\| $ //xg; # strip record terminator
s/\b(\w{3}) ( \d|\d\d) (\d{4}) \d\d:\d\d[AP]M\b/${3} . $months{$1} . sprintf('%02d', $2) /eg;
print;
}
Won't do what you want if one of the non-#date_cols fields matches the date regex.

Answer 3

At my work sometimes i need to grep errorlogs etc from 350+ frontends. I use script template i calling "SMP grep" ;) Its simple:
stat file, get file length
Get "chunk length" = file_length / num_processors
Andjust chunk starts and ends so they start/end at "\n". Just read(), find "\n" and calculate offsets.
fork() to make num_processor workers, each working on own chunk
This can help if you use regexps in your grep or other CPU operations(as your case i think). Admins complaining this script eats disk throughput, but its only bottleneck here if server has 8 CPUs =) Also, obviously if you need to parse 1 week data you can divide between servers.
Tomorrow i can post the code if interested.