Perl UTF8 output to a variable - perl

Question

I have the following Perl code, in which I am opening a handle to a scalar variable and writing some utf8 text to it:
use warnings;
use strict;
use 5.010;
use utf8;
use open qw( :std :encoding(utf8) );
my $output;
open my $oh, ">", \$output;
say $oh "Žluťoučký kůň.";
close $oh;
say "Žluťoučký kůň.";
print $output;
and when I run it I get the following output:
Žluťoučký kůň.
ŽluÅ¥ouÄký kůÅ.
(without any warnings or errors). So, obviously, writing an utf8 string into a variable via a handle does not work correctly here as the string seems to be double-encoded. I have tried opening $oh with >:raw, >:bytes, >:encoding(ascii), but none of it helped.
I might be doing something stupid but I cannot figure how to fix this. Any ideas?

Answer 1

First of all, :encoding(utf8) should be :encoding(utf-8).
UTF-8 is the well known encoding standard.
utf8 is a Perl-specific extension to UTF-8.
Reference
(Encoding names are case-insensitive.)
use open qw( :std :encoding(utf8) ); has two effects:
It adds :encoding(utf8) to STDIN, STDOUT and STDERR.
It sets the default layer for open in the lexical scope of the use to :encoding(utf8).
So,
use utf8;
use open qw( :std :encoding(UTF-8) );
# String of decoded text aka string of Unicode Code Points, thanks to `use utf8`.
my $text_ucp = "Žluťoučký kůň.";
# $output will contain text encoded using UTF-8 thanks to `use open`.
open my $oh, ">", \my $text_utf8;
say $oh $text_ucp;
close $oh;
# ok. Will encode the decoded text using UTF-8 thanks to `use open`.
say $text_ucp;
# XXX. Will encode the already-encoded text using UTF-8 thanks to `use open`.
print $text_utf8;
You tried to override the second effect of use open to obtain a file of Unicode Code Points, but that's futile since files can only contain bytes. Some kind of encoding or failure must occur if you try to store something other than bytes in a file.
So live with it, and decode the "file" before using it.
use utf8;
use open qw( :std :encoding(UTF-8) );
use Encode qw( decode_utf8 );
my $text_ucp = "Žluťoučký kůň.";
open my $oh, ">", \my $text_utf8;
say $oh $text_ucp;
close $oh;
my $text2_ucp = decode_utf8($text_utf8);
... Do stuff with $text_ucp and/or $text2_ucp ...
say $text_ucp;
say $text2_ucp;
It is possible to avoid the decode by working directly with UTF-8 in the second half of the program.
use utf8;
BEGIN { binmode(STDERR, ":encoding(UTF-8)"); } # We'll handle STDOUT manually.
use open qw( :encoding(UTF-8) );
use Encode qw( encode_utf8 );
my $text_ucp = "Žluťoučký kůň.";
open my $oh, ">", \my $text_utf8;
say $oh $text_ucp;
close $oh;
say encode_utf8($text_ucp);
say $text_utf8;
Of course, that means you can't use $text_utf8 anywhere that expects decoded text.