Perl + Unicode: "Wide Strings" error - perl

I am running Active Perl 5.14 on Windows 7.
I am trying to write a program that will read-in a conversion table, then work on a file and replace certain patterns by other patterns - all of the above in Unicode (UTF-8). Here is the beginning of the program:
#!/usr/local/bin/perl
# Load a conversion table from CONVTABLE to %ConvTable.
# Then find matches in a file and convert them.
use strict;
use warnings;
use Encode;
use 5.014;
use utf8;
use autodie;
use warnings qw< FATAL utf8 >;
use open qw< :std :utf8 >;
use charnames qw< :full >;
use feature qw< unicode_strings >;
my ($i,$j,$InputFile, $OutputFile,$word,$from,$to,$linetoprint);
my (#line, #lineout);
my %ConvTable; # Conversion hash
print 'Conversion table: opening file: E:\My Documents\Perl\Conversion table.txt'."\n";
my $sta= open (CONVTABLE, "<:encoding(utf8)", 'E:\My Documents\Perl\Conversion table.txt');
binmode STDOUT, ':utf8'; # output should be in UTF-8
# Load conversion hash
while (<CONVTABLE>) {
chomp;
print "$_\n"; # etc ...
# etc ...
It turns out that at this point, it says:
wide character in print at (eval 155)E:/Active Perl/lib/Perl5DB.pl:640]line 2, <CONVTABLE> line 1, etc...
Why is that? I think I've gone through and implemented all the necessary prescriptions for correct handling of Unicode strings, decoding and encoding into UTF-8?
And how to fix it?
TIA
Helen

The Perl debugger has its own output handle that is distinct from STDOUT (although it may ultimately go to the same place as STDOUT). You'll also want to do something like this near the beginning of your script:
binmode $DB::OUT, ':utf8' if $DB::OUT;

I suspect that the problem is in some part of the code that you haven't shown us. I base this suspicion on the following facts:
The error message you quote says at (eval 155). There are no evals in your code.
The code you have shown us above does not produce a "wide character" warning when I run it, even if the input contains Unicode characters. The only way I can make it produce one is to comment out both the use open line and the binmode STDOUT line.
Admittedly, my testing environment is not exactly identical to yours: I'm on Linux, and my Perl is only v5.10.1, meaning that I had to lower the version requirement and turn off the unicode_strings feature (not that you're actually using it). Still, I very much suspect that the problem is not in the code you've posted.

Related

Why do I get garbled output when I decode some HTML entities but not others?

In Perl, I am trying to decode strings which contain numeric HTML entities using HTML::Entities. Some entities work, while "newer" entities don't. For example:
decode_entities('®'); # returns ® as expected
decode_entities('Ω'); # returns Ω instead of Ω
decode_entities('★'); # returns ★ instead of ★
Is there a way to decode these "newer" HTML entities in Perl? In PHP, the html_entity_decode function seems to decode all of these entities without any problem.
The decoding works fine. It's how you're outputting them that's wrong. For example, you may have sent the strings to a terminal without encoding them for that terminal first. This is achieved through the open pragma in the following program:
$ perl -e'
use open ":std", ":encoding(UTF-8)";
use HTML::Entities qw( decode_entities );
CORE::say decode_entities($_)
for "®", "Ω", "★";
'
®
Ω
★
Make sure your terminal can handle UTF-8 encoding. It looks like it's having problems with multibyte characters. You can also try to set UTF-8 for STDOUT in case you get wide character warnings.
use strict;
use warnings;
use HTML::Entities;
binmode STDOUT, ':encoding(UTF-8)';
print decode_entities('®'); # returns ®
print decode_entities('Ω'); # returns Ω
print decode_entities('★'); # returns ★
This gives me the correct/expected results.

Could File::Find::Rule be patched to automatically handle filename character encoding/decoding?

Suppose I have a file with name æ (UNICODE : 0xE6, UTF8 : 0xC3 0xA6) in the current directory.
Then, I would like to use File::Find::Rule to locate it:
use feature qw(say);
use open qw( :std :utf8 );
use strict;
use utf8;
use warnings;
use File::Find::Rule;
my $fn = 'æ';
my #files = File::Find::Rule->new->name($fn)->in('.');
say $_ for #files;
The output is empty, so apparently this did not work.
If I try to encode the filename first:
use Encode;
my $fn = 'æ';
my $fn_utf8 = Encode::encode('UTF-8', $fn, Encode::FB_CROAK | Encode::LEAVE_SRC);
my #files = File::Find::Rule->new->name($fn_utf8)->in('.');
say $_ for #files;
The output is:
æ
So it found the file, but the returned filename is not decoded into a Perl string. To fix this, I can decode the result, replacing the last line with:
say Encode::decode('UTF-8', $_, Encode::FB_CROAK) for #files;
The question is if both the encoding and decoding could/should have been done automatically by File::Find::Rule so I could have used my original program and not have had to worry about encoding and decoding at all?
(For example, could File::Find::Rule have used I18N::Langinfo to determine that the current locale's codeset is UTF-8 ?? )
Yeah, I wish. If there's was a major Perl project I'd work on, this would be it.
The issue is that there could be badly-encoded file names, including file names encoded using a different encoding than expected. That means the first thing needed is a way of round-tripping badly-encoded file names through a decode-encode process. I think Python uses the surrogate pair code points to represent the bad bytes.
You would need a pragma to ensure backwards compatibility.

How to get a defined ${^OPEN}?

What changes are required in this code to get a defined ${^OPEN}?
#!/usr/bin/env perl
use warnings;
use strict;
use open qw( :std :utf8 );
print ${^OPEN};
Use of uninitialized value $^OPEN in print at ./perl.pl line 6.
This is quite uneasy way. May be it is better to user more readable Perl.
:utf8 outputs utf-8 charset but does not checks its validity, you should not use this one, except for one liners. Use :encoding(UTF-8) instead.
Please refer to this post How differs the open pragma with different utf8? for more information about different types of utf-8 input/outputs.
I even do not know what could possibly be ${^OPEN} variable. I advice you not to use it at all, as you should not use magic punctuation.
Hope it helps

Perl: printing Unicode strings to the Windows console

I am encountering a strange problem in printing Unicode strings to the Windows console*.
Consider this text:
אני רוצה לישון
Intermediary
היא רוצה לישון
אתם, הם
Bye
Hello, world!
test
Assume it's in a file called "file.txt".
When I go*: "type file.txt", it prints out fine. But when it's printed from a Perl program, like this:
use strict;
use warnings;
use Encode;
use 5.014;
use utf8;
use autodie;
use warnings qw< FATAL utf8 >;
use open qw< :std :utf8 >;
use feature qw< unicode_strings >;
use warnings 'all';
binmode STDOUT, ':utf8'; # output should be in UTF-8
my $word;
my #array = ( 'אני רוצה לישון', 'Intermediary',
'היא רוצה לישון', 'אתם, הם', 'Bye','Hello, world!', 'test');
foreach $word(#array) {
say $word;
}
The Unicode lines (Hebrew in this case) show up again each time, partially broken, like this:
E:\My Documents\Technical\Perl>perl "hello unicode.pl"
אני רוצה לישון
לישון
�ן
Intermediary
היא רוצה לישון
לישון
�ן
אתם, הם
�ם
Bye
Hello, world!
test
(I save everything in UTF-8).
This is mighty strange. Any suggestions?
(It's not a "Console2" problem* - the same problem shows up on a "regular" windows console, only there you don't see the Hebrew glyphs).
* Using "Console" (also called "Console2") - it's a nice little utility which enables working with Unicode with the Windows console - see, for example, here:
http://www.hanselman.com/blog/Console2ABetterWindowsCommandPrompt.aspx
** Note: at the console, you have to say, of course:
chcp 65001
Did you try the solution from perlmonk ?
It use :unix as well to avoid the console buffer.
This is the code from that link:
use Win32::API;
binmode(STDOUT, ":unix:utf8");
#Must set the console code page to UTF8
$SetConsoleOutputCP= new Win32::API( 'kernel32.dll', 'SetConsoleOutputCP', 'N','N' );
$SetConsoleOutputCP->Call(65001);
$line1="\x{2554}".("\x{2550}"x15)."\x{2557}\n";
$line2="\x{2551}".(" "x15)."\x{2551}\n";
$line3="\x{255A}".("\x{2550}"x15)."\x{255D}";
$unicode_string=$line1.$line2.$line3;
print "THIS IS THE CORRECT EXAMPLE OUTPUT IN PURE PERL: \n";
print $unicode_string;
Guys: continuing on studying that Perlmonks post, turns out that this is even neater and nicer:
replace: use Win32::API;
and:
$SetConsoleOutputCP= new Win32::API( 'kernel32.dll', 'SetConsoleOutputCP', 'N','N' );
$SetConsoleOutputCP->Call(65001);
with:
use Win32::Console;
and:
Win32::Console::OutputCP(65001);
Leaving all else intact.
This is even more in the spirit of Perl conciseness and magic.
You can also utilize Win32::Unicode::Console or Win32::Unicode::Native to achieve unicode prints on windows console.
Also, this behaviour is not present while using ConEmu, which also enables proper Unicode support in Windows' command console.

Unicode string mess in perl

I have an external module, that is returning me some strings. I am not sure how are the strings returned, exactly. I don't really know, how Unicode strings work and why.
The module should return, for example, the Czech word "být", meaning "to be". (If you cannot see the second letter - it should look like this.) If I display the string, returned by the module, with Data Dumper, I see it as b\x{fd}t.
However, if I try to print it with print $s, I got "Wide character in print" warning, and ? instead of ý.
If I try Encode::decode(whatever, $s);, the resulting string cannot be printed anyway (always with the "Wide character" warning, sometimes with mangled characters, sometimes right), no matter what I put in whatever.
If I try Encode::encode("utf-8", $s);, the resulting string CAN be printed without the problems or error message.
If I use use encoding 'utf8';, printing works without any need of encoding/decoding. However, if I use IO::CaptureOutput or Capture::Tiny module, it starts shouting "Wide character" again.
I have a few questions, mostly about what exactly happens. (I tried to read perldocs, but I was not very wise from them)
Why can't I print the string right after getting it from the module?
Why can't I print the string, decoded by "decode"? What exactly "decode" did?
What exactly "encode" did, and why there was no problem in printing it after encoding?
What exactly use encoding do? Why is the default encoding different from utf-8?
What do I have to do, if I want to print the scalars without any problems, even when I want to use one of the capturing modules?
edit: Some people tell me to use -C or binmode or PERL_UNICODE. That is a great advice. However, somehow, both the capturing modules magically destroy the UTF8-ness of STDOUT. That seems to be more a bug of the modules, but I am not really sure.
edit2: OK, the best solution was to dump the modules and write the "capturing" myself (with much less flexibility).
Because you output a string in perl's internal form (utf8) to a non-unicode filehandle.
The decode function decodes a sequence of bytes assumed to be in ENCODING into Perl's internal form (utf8). Your input seems to be already decoded,
The encode() function encodes a string from Perl's internal form into ENCODING.
The encoding pragma allows you to write your script in any encoding you like. String literals are automatically converted to perl's internal form.
Make sure perl knows which encoding your data comes in and come out.
See also perluniintro, perlunicode, Encode module, binmode() function.
I recommend reading the Unicode chapter of my book Effective Perl Programming. We put together all the docs we could find and explained Unicode in Perl much more coherently than I've seen anywhere else.
This program works fine for me:
#!perl
use utf8;
use 5.010;
binmode STDOUT, ':utf8';
my $string = return_string();
say $string;
sub return_string { 'být' }
Additionally, Capture::Tiny works just fine for me:
#!perl
use utf8;
use 5.010;
use Capture::Tiny qw(capture);
binmode STDOUT, ':utf8';
my( $stdout, $stderr ) = capture {
system( $^X, '/Users/brian/Desktop/czech.pl' );
};
say "STDOUT is [$stdout]";
IO::CaptureOutput seems to have some problems though:
#!perl
use utf8;
use 5.010;
use IO::CaptureOutput qw(capture);
binmode STDOUT, ':utf8';
capture {
system( $^X, '/Users/brian/Desktop/czech.pl' );
} \my $stdout, \my $stderr;
say "STDOUT is [$stdout]";
For this I get:
STDOUT is [být
]
However, that's easy to fix. Don't use that module. :)
You should also look at the PERL_UNICODE environment variable, which is the same as using the -C option. That allows you to set STDIN/STDOUT/STDERR (and #ARGV) to be UTF-8 without having to alter your scripts.