CSV manipulation AWK? - perl

Question

I have two CSV files, one has a long list of reference numbers, the other a daily list of orders.
On a daily basis I need to cut & paste from the reference numbers into the daily orders. Obviously I only cut as many reference numbers as there are orders, so for example if there are 20 orders I need to get 20 reference numbers from the other file and paste into my orders file. I cut these numbers so we don't get duplicates on the next days run.
I want to automate this process but I don't know the best way. I am running windows and have used AWK for some other csv manipulation but I'm not very experienced with AWK and not sure if this is possible so I am just asking if anybody has any ideas of the best solution.

Answer 1

Parsing CSV properly is very tricky business. Most difficulty comes from mistakes in parsing quotes, double quotes, commas, spaces, etc in your content.
Rather than reinventing the wheel, I would recommend using some well tested library. I don't think awk has one, but Perl does: DBD::CSV.
On Windows, simply install ActivePerl, it already has DBD::CSV installed by default.
Then, use Perl code like this to retrieve your data and convert to some other formats inside while loop:
use DBI;
my $dbh = DBI->connect("dbi:CSV:f_ext=.csv") or die $DBI::errstr;
my $sth = $dbh->prepare("SELECT * FROM mytable"); # access mytable.csv
$sth->execute();
while (my #row = $sth->fetchrow_array()) {
print "id: $row[0], name: $row[1]\n";
}
# you can also access columns by name, like this:
# while (my $row = $sth->fetchrow_hashref()) {
# print "id: $row->{id}, name: $row->{name}\n";
# }
$sth->finish();
$dbh->disconnect();
Since you mention you have 2 input CSV files, you might be able to even use SQL join statements to get data from both tables properly joined at once.