Check for CSV file correctness in Perl - perl

Question

I have a process that reads a CSV file, and I want to make sure it's correct before I start parsing it.
I get a file name, check if it exists, then check its integrity. If it's not there or not a proper CSV file then I try the file from the previous day instead
Is there a way to check that the file is proper CSV file? I am using Text::CSV_XS to parse it.
Googling a bit I found this csv-check example code on the Text::CSV_XS Git repo. It looks like something I could use.

Answer 1

As others have noted, you have to parse the entire file to determine if it's valid. You may as well kill two birds with one stone and do your data processing and error checking at the same time.
Detecting errors
getline() returns undef when it reaches EOF or if it fails to parse a line. You can use this to parse a file, halting if there are any parse errors:
while ( my $row = $csv->getline($io) ) {
# Process row
}
$csv->eof or do_something();
You can also
use autodie;
or set the auto_diag option in Text::CSV_XS->new() to die on errors:
$csv = Text::CSV_XS->new({ auto_diag => 2 });
You can handle the errors by wrapping your parsing code in an eval block. This method will automatically call error_diag() before dieing, printing the error to stderr; this may not be what you want.
Reverting invalid files
How do you "revert" the processing you did for previous rows if you detect an error? One possibility, if your database engine supports them, are database transactions. When you start processing a file, start a transaction. If you get a parse error, simply roll back the transaction and move on to the next file; otherwise, commit the transaction.
As an aside, I haven't seen your code for inserting database records so I'm not sure if this applies, but it's not very efficient to have a separate insert statement for each row. Instead, consider either constructing a compound insert statement as you parse the file; or, for very large files, let the database do the parsing with something like MySQL's LOAD DATA INFILE (just an example since I don't know what DBMS you're using).
To use a compound insert, build the query statement in memory like Borodin suggested. If you get to the end of the file without any parse errors, execute the statement; otherwise, throw it out and move on to the next file.
For very large files, it might be fastest to let the database do the parsing, especially if you're doing minimal processing before inserting the data. MySQL's LOAD DATA INFILE, for example, will halt if it detects data interpretation or duplicate key errors. If you wrap the statement in a transaction, you can roll back if there are errors and try to load the next file. The advantage of this approach is that loading valid files will be extremely fast, much faster than if you had to parse them with Perl first.

Answer 2

You can't test the validity of a file without reading and parsing every record in it anyway.
I suggest the way to go is to process each file that you find, building in memory the data that you want to end up in the database, and if you find an error then just discard it and try with the next file.
Once you reach the end of the file and know that it is valid and complete, then you can just save your prepared data to the database, and go on to the next file.
This will work fine unless your CSV files are enormous and too large to fit into memory sensibly. In that case you should simply take two passes.

Answer 3

Here is what I did, the sub returns 1 if the file is ok and 0 if it's not ok:
sub CheckCSVFile {
my ($fileName) =#_;
my $csv = Text::CSV_XS->new();
open my $in_fh, '<:encoding(ISO-8859-1)', $fileName;
while ( <$in_fh> ) {
my $status = $csv->parse($_);
if ($status != 1){
return $status;
}
}
$csv->eof;
close $in_fh;
return 1;
}
I check for file existence beforehand, so it shouldn't error out. I also don't want to exit if something goes wrong. It's a bit crude, but worked for me.