"Unrecognized character \xEF" Error with BOM-Free File - perl

Question

I am currently extracting text from CSV files using Perl and the module, Text::CSV.
Each of the CSV files have quotation marks separating each field. The texts are being saved to independent text files with tab separation into columns. I can call and print each column from the text files no problem, but when I try to use the values in a loop, I get the error Unrecognized character \xEF.
An example of my code is as follows:
#!/usr/bin/perl
use strict;
use warnings;
use Text::CSV;
#### Match ligand data with GPCR interaction data ####
my $csv = Text::CSV->new();
my $file = $ARGV[0];
open (FILE, "<$file");
open (OUT, ">new_$file");
while (my $line2 = <FILE>)
{
binmode(STDOUT, ":utf8");
if ($line2 =~ /^(.*?)\t(.*?)\t(.*?)\t(.*?)\t(.*?)\t(.*?)\t(.*?)$/)
{
#### Data from filtered1.txt ####
my $up_fil = $1;
my $ligid_fil = $2;
my $units_fil = $3;
my $low_fil = $4;
my $median_fil = $5;
my $upper_fil = $6;
my $ref = $7;
#### Convert negative log affinity values to normal ####
my $activity = $units_fil;
$activity =~ s/p//;
my $value;
if （$median_fil ne "")
{
$value = $median_fil;
$value = (10**-$median_fil)/(10**-9);
}
elsif ($low_fil ne "" and $upper_fil ne "")
{
my $lower = $low_fil;
$lower = (10**-$low_fil)/(10**-9);
my $upper = $upper_fil;
$upper = (10**-$upper_fil)/(10**-9);
$value = "$upper - $lower";
}
else
{
$value = "n/a";
}
#### Match entries from filtered1.txt with ligands.csv ####
open (LIG, "<ligands.csv");
while (my $line3 = <LIG>)
{
$csv->parse($line3);
my #ligand_fields = $csv->fields();
if (!$ligand_fields[14]) { next; }
if ($ligand_fields[0] eq $ligid_fil)
{
#print OUT "$ligand_fields[14]\t$ligand_fields[13]\t$up_fil\t$ligid_fil\t$activity\t$value\t$ref\n";
print "$ligand_fields[14]\t$ligand_fields[13]\t$up_fil\t$ligid_fil\t$activity\t$value\t$ref\n";
next;
}
}
close LIG;
}
}
close FILE;
close OUT;
I've also tried using a regex along the lines of the following, but to no avail.
# remove BOM
${$self->{CODE}} =~ s/^(?:
\xef\xbb\xbf |
\xfe\xff |
\xff\xfe |
\x00\x00\xfe\xff |
\xff\xfe\x00\x00
)//x;
The original CSV files appear not to have any BOM, so I suspect that Text::CSV may be creating it when it is parsing and returning values. I hope this was a clear enough explanation of the problem, and if needed, I can provide more details. Thanks in advance for any advice given.

Answer 1

The documentation of Text::CSV states you should almost certainly be using the binary mode.
my $csv = Text::CSV->new ( { binary => 1 } ) # should set binary attribute.
or die "Cannot use CSV: ".Text::CSV->error_diag ();
From https://metacpan.org/pod/Text::CSV#SYNOPSIS.
You may also want to take a look at Text::CSV::Encoded.
I also see you are setting a binmode of :utf8 on STDOUT. There are a couple of problems with that:
You are setting it each time around the loop unnecessarily
The :utf8 binmode does not have good error checking, you should use :encoding(UTF-8) instead
The byte 0xEF can appear in UTF-8 byte sequences, but only in very specific circumstances, it is too high (> 0x7F) to be a single character. However in Perl \xEF or \x{ef} does not refer to the byte 0xEF, but the Unicode code point U+00EF which is represented in UTF-8 as 0xC3 0xAF. You can see this in a Unicode/UTF-8 character table such as http://www.utf8-chartable.de/.
$ perl -E 'binmode STDOUT, ":encoding(UTF-8)"; say "\xEF";'
ï
So I think that is why your regular expression to remove the BOM didn't work.
I would recommend using three argument open with either '<:encoding(UTF-8)' or '>:encoding(UTF-8)' to open all of your input and output files, and using Text::CSV in binary mode, for the best results.