Perl Win32::OLE Generating Access Database - perl

Question

Description: I am reading from a list of flat files and generating and loading an access database. Windows XP, Perl 5.8.8, and no access to additional modules outside the default installed.
Issue(s): Performance, Performance, Performance. It is taking ~20 minutes to load in all of the data. I am assuming that there might be a better way to load the data rather than addnew & update.
Logic: Without posting a lot of my transformations and additional logic here is what I am attempting:
Open file x
read row 0 of file x
jet->execute a Create statement from string dervied from step 2
read in rows 1 - n creating a tab delimitted string and store into an array
Open a recordset using select * from tablename
for each item in array
recordset->AddNew
split the item based on the tab
for each item in the split
rs->Fields->Item(pos)->{Value} = item_value
recordset->Update

Answer 1

One issue in slow loads is doing a commit on every update. Make sure that automatic commits are off and do one every 1000 rows or whatever. If it is not a gigantic load, don't do them at all. Also, do not create indexes during the load, create them afterwards.
Also, I'm not sure that OLE is the best way to do this. I load Access db's all of the time using DBI and Win32::ODBC. Goes pretty fast.
Per request, here is sample load program, did about 100k records per minute on WinXP, Access 2003, ActiveState Perl 5.8.8.
use strict;
use warnings;
use Win32::ODBC;
$| = 1;
my $dsn = "LinkManagerTest";
my $db = new Win32::ODBC($dsn)
or die "Connect to database $dsn failed: " . Win32::ODBC::Error();
my $rows_added = 0;
my $error_code;
while (<>) {
chomp;
print STDERR "." unless $. % 100;
print STDERR " $.\n" unless $. % 5000;
my ($source, $source_link, $url, $site_name) = split /\t/;
my $insert = qq{
insert into Links (
URL,
SiteName,
Source,
SourceLink
)
values (
'$url',
'$site_name',
'$source',
'$source_link'
)
};
$error_code = $db->Sql($insert);
if ($error_code) {
print "\nSQL update failed on line $. with error code $error_code\n";
print "SQL statement:\n$insert\n\n";
print "Error:\n" . $db->Error() . "\n\n";
}
else {
$rows_added++;
}
$db->Transact('SQL_COMMIT') unless $. % 1000;
}
$db->Transact('SQL_COMMIT');
$db->Close();
print "\n";
print "Lines Read: $.\n";
print "Rows Added: $rows_added\n";
exit 0;