Perl: Problem with changing encoding in the middle of reading a file - perl

I am using Perl to load some 'macro' files. These macros can, however, be encoded in various encodings, so there is a directive defined for users writing their macros (i.e.
#encoding iso-8859-2
at the beginning of the macro).
Every time this directive is encountered in the macro, a function setting encoding is called and looks sth like this:
sub change_encoding {
my ($file_handle, $encoding) = #_;
$file_handle->flush();
binmode($file_handle); # get rid of IO layers
binmode($file_handle,":encoding($encoding)");
}
The problem is that when I read the macro using standard
while($line = <$file_handle>){
process_macro($line);
}
I got messages saying "utf8 "\xXY" does not map to Unicode", but only if characters with diacritics is near the #encoding directive. I tried several examples and I was able to have half of the string with \xXY codes and other half of the string with correctly decoded characters, like here:
sub macro5_fn {
print "\xBElu\xBBou\xE8k\xFD k\xF9\xF2 úpěl ďábelské ódy\n";
}
If I put more comments before the function, all the characters are OK:
sub macro5_fn {
print "žluťoučký kůň úpěl ďábelské ódy\n";
}
Simply said, the number of correctly decoded characters depends on the distance of these characters from the #encoding directive, the ones that are close are not decoded correctly.
It seems to me that this is an issue of Perl and PerlIO (not) flushing the buffer. Or am I doing something wrong?
Thank you for your answers.

The problem is that <> reads more than just one line, so the next line or so is being interpreted under the old encoding before you ever see the #encoding directive for the new.
Your best bet is probably to read the file in binary mode and use the Encode module to decode each line from the current encoding.

Related

Convert unicode to HTML entities function

I have the following function that converts unicode to HTML entities, but if I run the function again over the result it will not leave the HTML entities in tact. How can I get the function to leave already converted HTML entities alone?
sub convert_unicode {
use HTML::Entities;
use Encode;
my $str = shift;
Encode::_utf8_off($str);
return encode_entities(decode('utf8',$str));
}
What you're asking for is to be able to safely double character encode. Some encodings allow this. HTML character encoding does not because it uses certain characters like & to do the encoding and it cannot tell the difference between a special character being used for encoding and one that needs to be encoded.
For example...
use HTML::Entities;
use v5.10;
say encode_entities("&foo");
That produces &foo. If we encode it again it produces &amp;foo because & is a special character which it faithfully encodes. It does not know that & is an already encoded & so it treats it as a literal & and encodes it.
You could write your own custom HTML encoding function that assumes &xxx; (and its variants) are already encoded, but that's just a guess. You can't actually tell a literal &foo; and an encoded &foo; apart. It will break with, for example, old school Perl code like &function;. Maybe you can be super clever and use an array of objects to indicate which parts are encoded and have the whole thing overload stringification so it looks like a string, and so long as everything carefully preserves that object that looks like a string it'll work...
And now we're into the lava flow anti-pattern where rather than fixing bad design, more complex and bad design is layered on top of it. Trying to "fix" that will just create more problems. The real problem lies deeper.
The real problem is that you're encoding multiple times. This probably means you've wielded your formatting and your functionality together. For example...
sub get_user_name {
my $uid = shift;
my $name = ...do a bunch of work to get the user name...
return encode_entities($name);
}
By HTML encoding the data, a function like this makes assumptions about how the data is going to be used. It limits its use to just HTML. If all your functions do this, you have a double encoding problem.
Then maybe you have something like this:
sub do_something {
my $uid = shift;
# $name is already HTML encoded.
my $name = get_user_name($uid);
my $stuff = ...something incorporating $name...
# Whoops, the user name is double encoded.
return encode_entities($stuff);
}
The answer is to leave the HTML formatting and encoding until the last minute. Ideally don't do it at all, just work with data and let an HTML template system take care of it. Template Toolkit, for example.
This also provides a clean separation between the formatting and the code, so now non-programmers can work on the formatting using a documented template system.

Unicode Juggling with Perl

I have a problem I thought to be trivial. I have to deal with Umlauts from the German alphabet (äöü). In Unicode, there seem to be several ways to display them, one of them is combining characters. I need to normalise these different ways, replace them all by the one-character code.
Such a deviant umlaut is easily found: It is a letter aou, followed by the UTF-8 char \uCC88. So I thought a regex would suffice.
This is my conversion function, employing the Encoding package.
# This sub can be extended to include more conversions
sub convert {
local $_;
$_ = shift;
$_ = encode( "utf-8", $_ );
s/u\xcc\x88/ü/g;
s/a\xcc\x88/ä/g;
s/o\xcc\x88/ö/g;
s/U\xcc\x88/Ü/g;
s/A\xcc\x88/Ä/g;
s/O\xcc\x88/Ö/g;
return $_;
}
But the resulting printed umlaut is some even more devious character (now taking 4 bytes), instead of the one on this list.
I guess the problem is this juggling with Perl's internal format, actual UTF-8 and this Encoding format.
Even changing the substitution lines to
s/u\xcc\x88/\xc3\xbc/g;
s/a\xcc\x88/\xc3\xa4/g;
s/o\xcc\x88/\xc3\xb6/g;
s/U\xcc\x88/\xc3\x9c/g;
s/A\xcc\x88/\xc3\x84/g;
s/O\xcc\x88/\xc3\x96/g;
did not help, they're converted correctly but then followed by "\xC2\xA4" in the bytes.
Any help?
You're doing it wrong: you must stop the habit of messing with characters on the representation level, i.e. do not fiddle with bytes in regex when you deal with text, not binary data.
The first step is to learn about the topic of encoding in Perl. You need this to understand the term "character strings" I am going to use in the following paragraph.
When you have character string, it might be in any of the various states of (de)composition. Use the module Unicode::Normalize to change a character string, and read the relevant chapters on equivalence and normalisation in the Unicode specification for the gory details, they are linked at the bottom of that module's documentation.
I guess you want NFC, but you have to run a sanity check against your data to see whether that's really the intended result.
use charnames qw(:full);
use Unicode::Normalize qw(NFC);
my $original_character_string = "In des Waldes tiefsten Gr\N{LATIN SMALL LETTER U WITH DIAERESIS}nden ist kein R\N{LATIN SMALL LETTER A}\N{COMBINING DIAERESIS}uber mehr zu finden.";
my $modified_character_string = NFC($original_character_string);
# "In des Waldes tiefsten Gr\x{fc}nden ist kein R\x{e4}uber mehr zu finden."

Filtering microsoft 1252 characters out of an ASCII text file opened in utf8 mode in Perl

I have a reasonable size flat file database of text documents mostly saved in 8859 format which have been collected through a web form (using Perl scripts). Up until recently I was negotiating the common 1252 characters (curly quotes, apostrophes etc.) with a simple set of regex's:
$line=~s/\x91/\&\#8216\;/g; # smart apostrophe left
$line=~s/\x92/\&\#8217\;/g; # smart apostrophe right
... etc.
However since I decided I ought to be going Unicode, and have converted all my scripts to read in and output utf8 (which works a treat for all new material), the regex for these (existing) 1252 characters no longer works and my Perl html output outputs literally the 4 characters: '\x92' and '\x93' etc. (at least that's how it appears on a browser in utf8 mode, downloading (ftp not http) and opening in a text editor (textpad) it's different, a single undefined character remains, and opening the output file in Firefox default (no content type header) 8859 mode renders the correct character).
The new utf8 pragmas at the start of the script are:
use CGI qw(-utf8);
use open IO => ':utf8';
I understand this is due to utf8 mode making the characters double byte instead of single byte and applies to those chars in the 0x80 to 0xff range, having read up the article on wikibooks relating to this, however I was non the wiser as to how to filter them. Ideally I know I ought to resave all the documents in utf8 mode (since the flat file database now contains a mixture of 8859 and utf8), however I will need some kind of filter in the first place if I'm going to do this anyway.
And I could be wrong as to the 2-byte storage internally, since it did seem to imply that Perl handles stuff very differently according to various circumstances.
If anybody could provide me with a regex solution I would be very grateful. Or some other method. I have been tearing my hair out for weeks on this with various attempts and failed hacking. There's simply about 6 1252 characters that commonly need replacing, and with a filter method I could resave the whole flippin lot in utf8 and forget there ever was a 1252...
Encoding::FixLatin was specifically written to help fix data broken in the same manner as yours.
Ikegami already mentioned the Encoding::FixLatin module.
Another way to do it, if you know that each string will be either UTF-8 or CP1252, but not a mixture of both, is to read it as a binary string and do:
unless ( utf8::decode($string) ) {
require Encode;
$string = Encode::decode(cp1252 => $string);
}
Compared to Encoding::FixLatin, this has two small advantages: a slightly lower chance of misinterpreting CP1252 text as UTF-8 (because the entire string must be valid UTF-8) and the possibility of replacing CP1252 with some other fallback encoding. A corresponding disadvantage is that this code could fall back to CP1252 on strings that are not entirely valid UTF-8 for some other reason, such as because they were truncated in the middle of a multi-byte character.
You could also use Encode.pm's support for fallback.
use Encode qw[decode];
my $octets = "\x91 Foo \xE2\x98\xBA \x92";
my $string = decode('UTF-8', $octets, sub {
my ($ordinal) = #_;
return decode('Windows-1252', pack 'C', $ordinal);
});
printf "<%s>\n",
join ' ', map { sprintf 'U+%.4X', ord $_ } split //, $string;
Output:
<U+2018 U+0020 U+0046 U+006F U+006F U+0020 U+263A U+0020 U+2019>
Did you recode the data files? If not, opening them as UTF-8 won't work. You can simply open them as
open $filehandle, '<:encoding(cp1252)', $filename or die ...;
and everything (tm) should work.
If you did recode, something seem to have gone wrong, and you need to analyze what it is, and fix it. I recommend using hexdump to find out what actually is in a file. Text consoles and editors sometimes lie to you, hexdump never lies.

Convert a UTF8 string to ASCII in Perl

I've tried everything Google and StackOverflow have recommended (that I could find) including using Encode. My code works but it just uses UTF8 and I get the wide character warnings. I know how to work around those warnings but I'm not using UTF8 for anything else so I'd like to just convert it and not have to adapt the rest of my code to deal with it. Here's my code:
my $xml = XMLin($content);
# Populate the #titles array with each item title.
my #titles;
for my $item (#{$xml->{channel}->{item}}) {
my $title = Encode::decode_utf8($item->{title});
#my $title = $item->{title};
#utf8::downgrade($title, 1);
Encode::from_to($title, 'utf8', 'iso-8859-1');
push #titles, $title;
}
return #titles;
Commented out you can see some of the other things I've tried. I'm well aware that I don't know what I'm doing here. I just want to end up with a plain old ASCII string though. Any ideas would be greatly appreciated. Thanks.
The answer depends on how you want to use the title. There are 3 basic ways to go:
Bytes that represent a UTF-8 encoded string.
This is the format that should be used if you want to store the UTF-8 encoded string outside your application, be it on disk or sending it over the network or anything outside the scope of your program.
A string of Unicode characters.
The concept of characters is internal to Perl. When you perform Encode::decode_utf8, then a bunch of bytes is attempted to be converted to a string of characters, as seen by Perl. The Perl VM (and the programmer writing Perl code) cannot externalize that concept except through decoding UTF-8 bytes on input and encoding them to UTF-8 bytes on output. For example, your program receives two bytes as input that you know they represent UTF-8 encoded character(s), let's say 0xC3 0xB6. In that case decode_utf8 returns a representation that instead of two bytes, sees one character: ö.
You can then proceed to manipulate that string in Perl. To illustrate the difference further, consider the following code:
my $bytes = "\xC3\xB6";
say length($bytes); # prints "2"
my $string = decode_utf8($bytes);
say length($string); # prints "1"
The special case of ASCII, a subset of UTF-8.
ASCII is a very small subset of Unicode, where characters in that range are represented by a single byte. Converting Unicode into ASCII is an inherently lossy operation, as most of the Unicode characters are not ASCII characters. You're either forced to drop every character in your string which is not in ASCII or try to map from a Unicode character to their closest ASCII equivalents (which isn't possible in the vast majority of cases), when trying to coerce a Unicode string to ASCII.
Since you have wide character warnings, it means that you're trying to manipulate (possibly output) Unicode characters that cannot be represented as ASCII or ISO-8859-1.
If you do not need to manipulate the title from your XML document as a string, I'd suggest you leave it as UTF-8 bytes (I'd mention that you should be careful not to mix bytes and characters in strings). If you do need to manipulate it, then decode, manipulate, and on output encode it in UTF-8.
For further reading, please use perldoc to study perlunitut, perlunifaq, perlunicode, perluniintro, and Encode.
Although this is an old question, I just spent several hours (!) trying to do more or less the same thing! That is: read data from a UTF-8 XML file, and convert that data into the Windows-1252 codepage (I could also have used Latin1, ISO-8859-1 etc.) in order to be able to create filenames with accented letters.
After much experimentation, and even more searching, I finally managed to get the conversion working. The "trick" is to use Encode::encode instead of Encode::decode.
For example, given the code in the original question, the correct (or at least one :-) way to convert from UTF-8 would be:
my $title = Encode::encode("Windows-1252", $item->{title});
or
my $title = Encode::encode("ISO-8859-1", $item->{title});
or
my $title = Encode::encode("<your-favourite-codepage-here>", $item->{title});
I hope this helps others having similar problems!
You can use the following line to simply get rid of the warning. This assumes that you want to use UTF8, which shouldn't normally be a problem.
binmode(STDOUT, ":encoding(utf8)");

How can I create a Unicode character from its bytes when they are stored in different variables in Perl?

I am trying to Convert hex representations of Unicode characters to the characters they represent. The following example works fine:
#!/usr/bin/perl
use Encode qw( encode decode );
binmode(STDOUT, ':encoding(utf-8)');
my $encoded = encode('utf8', "\x{e382}\x{af}");
eval { $encoded = decode('utf8', $encoded, Encode::FB_CROAK); 1 }
or print("coaked\n");
print "$encoded\n";
However the hex digits are stored in 3 variables.
So if i replace the encode line with this:
my $encoded = encode('utf8', "\x{${byte1}${byte2}}\x{${byte3}}");
where
my $byte1 = "e3"; my $byte2 = "82"; my $byte3 = "af";
It fails as it tries to evaluate the \x immediately and sees the $ sign and { as characters.
Does anyone know how to get around this.
Instead of
my $encoded = encode('utf8', "\x{${byte1}${byte2}}\x{${byte3}}");
You can use
my $encoded = encode('utf8', chr(hex($byte1 . $byte2)) . chr(hex($byte3)));
hex() converts from hexadecimal, and chr() returns the unicode character for a given code point.
[Edit:]
Not related to your question, but I noticed you mix utf-8 and utf8 in your program. I don't know if this is a typo, but you should be a ware that these are not the same things in Perl:
utf-8 (with hyphen, case insensitive) is what the UTF-8 standard says, whereas utf8 (no hyphen, also case insensitive) is Perls internal encoding, which is more loosely defined (it allows codepoints that are not valid unicode codepoints). In general, you should stick to utf-8 (perlunifaq has the details).
trendel's answer seems pretty good, but Encode::Escape offers an alternative solution:
use Encode::Escape::Unicode;
my $hex = '263a';
my $escaped = "\\x{" . $hex . "}\n";
print encode 'utf8', decode 'unicode-escape', $escaped;
First off, think hard about why you ended up with three variables, $byte1, $byte2, $byte3, each holding one byte's worth of data, as a two-character string, in hex. This part of your program seems hard because of a poor design decision further up. Fix that bad decision, and this part of the code will fall out naturally.
That being said, what you want to do, I think, is this:
my $byte1 = "e3"; my $byte2 = "82"; my $byte3 = "af";
my $str = chr(hex($byte1 . $byte2)) . chr(hex($byte3))
The encoding stuff is a red herring; you shouldn't be worrying about encodings in the middle of your program, only when you do IO.
I'm assuming in the above that you want to get out a two character string, U+E382 followed by U+AF. That's what you actually asked for. However, since there is no U+E382, since it's in the middle of the private use area, that's probably not what you actually wanted. Please try to reword the question? Perhaps ask a more basic question, and describe what you are trying to achieve, rather then how you are going about trying to do it?