Why does decoding "€" to "€" also turn "é" into "é" in output? - perl

Question

I'm new to Perl scripting, and I'm facing some issues in decoding a string:
use HTML::Entities;
my $string='Rémunération €';
$string=decode_entitie($string);
print "$string";
The output I get looks like Rémunération €, when it should look like Rémunération €.
Can anyone please help me with this?

Answer 1

If you run this version of your code (with the typo in decode_entities fixed, strict mode and warnings enabled, and an extra print added) at a terminal:
use strict;
use warnings;
use HTML::Entities;
my $string='Rémunération €';
print "$string\n";
$string=decode_entities($string);
print "$string\n";
you should see the following output:
Rémunération €
Wide character in print at test.pl line 7.
Rémunération €
What happens is the following chain of events:
Your code is written in UTF-8, but don't have use utf8; in it, so Perl is parsing your source code (and, in particular, any string literals in it) byte by byte. Thus, the string literal 'é' is parsed as a two-character string, because the UTF-8 encoding of é takes up two bytes.
Normally, this doesn't matter (much), because your STDOUT is also not in UTF-8 mode, and so it just takes any byte string you give it and spits it out byte by byte, and your terminal then interprets the resulting output as UTF-8 (or tries to).
So, when you do print 'é'; Perl thinks you're printing a two-character string in byte mode, and writes out two bytes, which just happen to make up the UTF-8 encoding of the single character é.
However, when you run your string through decode_entities(), it decodes the € into an actual Unicode € character, which does not fit inside a single byte.
When you try to print the resulting string, Perl notices the "wide" € character. It can't print it as a single byte, so instead, it falls back to encoding the whole string as UTF-8 (and emitting a warning, if you have those enabled, as you should). But that causes the és (which were already encoded, since Perl never decoded them while parsing your code) to get double-UTF8-encoded, producing the mojibake output you see.
A simple fix is to add use utf8; to your code, and also set all your filehandles (including STDIN / STDOUT / STDERR) to UTF-8 mode by default, e.g. like this:
use utf8;
use open qw(:std :utf8);
With those lines prepended to the test script above, the output you get should be:
Rémunération €
Rémunération €