Related
I have read some articles online on the naming convention for Perl style which suggest using lowercase letters and separating words by underscore for functions or methods names. Some others use the first word in lowercase then capitalize the other words. Of course Windows .NET etc Capitalize every word and no underscore.
I have some packages methods many words like entriesoncurrentpage, if I follow Perl style suggested I should do it like:
sub entries_on_current_page {...}
this added four underscore letters to the method name, the other style is:
sub entriesOnCurrentPage {...}
and Windows style should be:
sub EntriesOnCurrentPage {...}
PHP sometimes uses all lowercase with underscore like mysql_real_escape_string() and sometimes uses all lowercase without underscore like htmlspecialchars, of course PHP function names are not case sensitive so this feature is not supported in Perl.
So the question is, for the long name with many words what is the best style to use for Perl coding.
Originally, most Perl developers used camel casing with the first letter lowercased. This is the standard with most programming languages. Names with first letter capitalized were used for classes and methods.
Later on, Damian Conway's book Perl Best Practices suggested using underscores rather than camel casing. Damian argued that it increased readability, and was not that much harder to type.
Damian Conway's suggestion on names became the standard because 1). He was correct. It's much more legible and isn't that much harder to type, and most importantly 2). It was incorporated into Perltidy. Perltidy is a program that helps standardize your code according to Damian's suggestions. Perltidy is much like CheckStyle in Java.
Are these arbitrary standards? Yes, all standards are somewhat arbitrary in nature. You have a few candidate suggestions for rules, and you must make a decision:
Should the curly brace in while loops and if statements be appended on the end of the line, or go under the while or if statement. In standerd C style, curly braces are cuddled. In Java, they're not suppose to be according to CheckStyle. In Kornshell, the then goes under the if. In Bash, the standard is now that the then goes on the same line even though the Bash interpreter doesn't really like it. (You have to add a semicolon before the then because it's considered a separate command.
How should variable names be done. In most languages, CamelCase rules. In .NET, you even capitalize the first character, but in Perl, we use underscores.
Should constants be all uppercase? Most languages have agreed with that. However, in shell script, you usually reserve all uppercase variable names for special environment variables such as $PWD, $PATH, etc. Therefore, in Bash and Kornshell scripts, constant variables are all camelCased like regular variables.
The idea is to follow the standard for that language. Why? Because the standard says so. Because you can't argue with The Standard as you can with your fellow programmers whether or not curly braces are cuddled or not. The main this is to realize that most standards may be somewhat arbitrary, but they don't really affect the way you program. By everyone following a standard, you make it easier to understand other people's code.
I found that is the translation for a string (msgid) is empty all gettext tools will consider the string as untranslated.
Is there a workaround for this? I do want to have an empty string as the translation for this item.
As this seems to be a big design flaw in the gettext specification, I decided to use:
Unicode Character 'ZERO WIDTH SPACE' (U+200B) inside these fields.
I realize this is an old question, but I wanted to point out an alternate approach:
msgid "This is a string"
msgstr "\0"
Since gettext uses embedded nulls to signal the end of a string, and it properly translates C escape sequences, I would guess that this might work and result in the empty string translation? It seemed to work in my program (based on GNU libintl) but I can't tell if this is actually standard / permitted by the system. As I understand gettext PO is not formally specified so there may be no authoritative answer other than looking at source code...
https://www.gnu.org/software/gettext/manual/html_node/PO-Files.html
It's often not a nice thing to do to programmers to put embedded nulls in things but it might work in your case? Arguably it's less evil than the zero-width-space trick, since it will actually result in a string whose size is zero.
Edit:
Basically, the worst thing that can happen is you get a segfault / bad behavior when running msgfmt, if it would get confused about the size of strings which it assumes don't have embedded null, and overflow a buffer somewhere.
Assuming that msgfmt can tolerate this though, libintl is going to have to do the right thing with it because the only means it has to return strings is char *, so the final application can only see up to the null character no matter what.
For what it's worth, my po-parser library spirit-po explicitly supports this :)
https://github.com/cbeck88/spirit-po
Edit: In gettext documentation, it appears that they do mention the possibility of embedded nulls in MO files and said "it was strongly debated":
https://www.gnu.org/software/gettext/manual/html_node/MO-Files.html
Nothing prevents a MO file from having embedded NULs in strings. However, the program interface currently used already presumes that strings are NUL terminated, so embedded NULs are somewhat useless. But the MO file format is general enough so other interfaces would be later possible, if for example, we ever want to implement wide characters right in MO files, where NUL bytes may accidentally appear. (No, we donโt want to have wide characters in MO files. They would make the file unnecessarily large, and the โwchar_tโ type being platform dependent, MO files would be platform dependent as well.)
This particular issue has been strongly debated in the GNU gettext development forum, and it is expectable that MO file format will evolve or change over time. It is even possible that many formats may later be supported concurrently. But surely, we have to start somewhere, and the MO file format described here is a good start. Nothing is cast in concrete, and the format may later evolve fairly easily, so we should feel comfortable with the current approach.
So, at the least it's not like they're going to say "man, embedded null in message string? We never thought of that!" Most likely it works, if msgfmt doesn't crash then I would assume it's kosher.
I have had the same problem for a long time, and I actually don't think you can at all. My best option was to insert a comment so I could mark it "translated" from there:
# No translation needed / Translated
msgid "This is a string"
msgstr ""
So far, it's been by best workaround :/ If you do end up finding a way, please post!
I wonder why most modern solutions built using Perl don't enable UTF-8 by default.
I understand there are many legacy problems for core Perl scripts, where it may break things. But, from my point of view, in the 21st century, big new projects (or projects with a big perspective) should make their software UTF-8 proof from scratch. Still I don't see it happening. For example, Moose enables strict and warnings, but not Unicode. Modern::Perl reduces boilerplate too, but no UTF-8 handling.
Why? Are there some reasons to avoid UTF-8 in modern Perl projects in the year 2011?
Commenting #tchrist got too long, so I'm adding it here.
It seems that I did not make myself clear. Let me try to add some things.
tchrist and I see situation pretty similarly, but our conclusions are completely in opposite ends. I agree, the situation with Unicode is complicated, but this is why we (Perl users and coders) need some layer (or pragma) which makes UTF-8 handling as easy as it must be nowadays.
tchrist pointed to many aspects to cover, I will read and think about them for days or even weeks. Still, this is not my point. tchrist tries to prove that there is not one single way "to enable UTF-8". I have not so much knowledge to argue with that. So, I stick to live examples.
I played around with Rakudo and UTF-8 was just there as I needed. I didn't have any problems, it just worked. Maybe there are some limitation somewhere deeper, but at start, all I tested worked as I expected.
Shouldn't that be a goal in modern Perl 5 too? I stress it more: I'm not suggesting UTF-8 as the default character set for core Perl, I suggest the possibility to trigger it with a snap for those who develop new projects.
Another example, but with a more negative tone. Frameworks should make development easier. Some years ago, I tried web frameworks, but just threw them away because "enabling UTF-8" was so obscure. I did not find how and where to hook Unicode support. It was so time-consuming that I found it easier to go the old way. Now I saw here there was a bounty to deal with the same problem with Mason 2: How to make Mason2 UTF-8 clean?. So, it is pretty new framework, but using it with UTF-8 needs deep knowledge of its internals. It is like a big red sign: STOP, don't use me!
I really like Perl. But dealing with Unicode is painful. I still find myself running against walls. Some way tchrist is right and answers my questions: new projects don't attract UTF-8 because it is too complicated in Perl 5.
๐๐๐ข๐ฅ๐ก๐๐จ๐ฉ โ: ๐ ๐ฟ๐๐จ๐๐ง๐๐ฉ๐ ๐๐๐๐ค๐ข๐ข๐๐ฃ๐๐๐ฉ๐๐ค๐ฃ๐จ
Set your PERL_UNICODE envariable to AS. This makes all Perl scripts decode #ARGV as UTFโ8 strings, and sets the encoding of all three of stdin, stdout, and stderr to UTFโ8. Both these are global effects, not lexical ones.
At the top of your source file (program, module, library, dohickey), prominently assert that you are running perl version 5.12 or better via:
use v5.12; # minimal for unicode string feature
use v5.14; # optimal for unicode string feature
Enable warnings, since the previous declaration only enables strictures and features, not warnings. I also suggest promoting Unicode warnings into exceptions, so use both these lines, not just one of them. Note however that under v5.14, the utf8 warning class comprises three other subwarnings which can all be separately enabled: nonchar, surrogate, and non_unicode. These you may wish to exert greater control over.
use warnings;
use warnings qw( FATAL utf8 );
Declare that this source unit is encoded as UTFโ8. Although once upon a time this pragma did other things, it now serves this one singular purpose alone and no other:
use utf8;
Declare that anything that opens a filehandle within this lexical scope but not elsewhere is to assume that that stream is encoded in UTFโ8 unless you tell it otherwise. That way you do not affect other moduleโs or other programโs code.
use open qw( :encoding(UTF-8) :std );
Enable named characters via \N{CHARNAME}.
use charnames qw( :full :short );
If you have a DATA handle, you must explicitly set its encoding. If you want this to be UTFโ8, then say:
binmode(DATA, ":encoding(UTF-8)");
There is of course no end of other matters with which you may eventually find yourself concerned, but these will suffice to approximate the state goal to โmake everything just work with UTFโ8โ, albeit for a somewhat weakened sense of those terms.
One other pragma, although it is not Unicode related, is:
use autodie;
It is strongly recommended.
๐ด ๐ช๐ซ๐ช ๐ ๐ฒ๐ ๐ฟ๐๐๐ ๐๐๐ ๐ฏ๐ ๐ท๐๐๐๐๐๐๐ ๐ ๐ช๐ซ๐ช ๐
๐ ๐ช ๐ญ๐๐๐๐๐โธ๐๐๐๐๐ ๐๐๐ ๐๐๐๐๐๐๐โธ๐ฌ๐๐๐๐ ๐ฎ๐๐๐ ๐ช ๐
My own boilerplate these days tends to look like this:
use 5.014;
use utf8;
use strict;
use autodie;
use warnings;
use warnings qw< FATAL utf8 >;
use open qw< :std :utf8 >;
use charnames qw< :full >;
use feature qw< unicode_strings >;
use File::Basename qw< basename >;
use Carp qw< carp croak confess cluck >;
use Encode qw< encode decode >;
use Unicode::Normalize qw< NFD NFC >;
END { close STDOUT }
if (grep /\P{ASCII}/ => #ARGV) {
#ARGV = map { decode("UTF-8", $_) } #ARGV;
}
$0 = basename($0); # shorter messages
$| = 1;
binmode(DATA, ":utf8");
# give a full stack dump on any untrapped exceptions
local $SIG{__DIE__} = sub {
confess "Uncaught exception: #_" unless $^S;
};
# now promote run-time warnings into stack-dumped
# exceptions *unless* we're in an try block, in
# which case just cluck the stack dump instead
local $SIG{__WARN__} = sub {
if ($^S) { cluck "Trapped warning: #_" }
else { confess "Deadly warning: #_" }
};
while (<>) {
chomp;
$_ = NFD($_);
...
} continue {
say NFC($_);
}
__END__
๐
๐น ๐ ๐ธ ๐ ๐ ๐ ๐ ๐ญ ๐ ๐ ๐ ๐ ๐ ๐
Saying that โPerl should [somehow!] enable Unicode by defaultโ doesnโt even start to begin to think about getting around to saying enough to be even marginally useful in some sort of rare and isolated case. Unicode is much much more than just a larger character repertoire; itโs also how those characters all interact in many, many ways.
Even the simple-minded minimal measures that (some) people seem to think they want are guaranteed to miserably break millions of lines of code, code that has no chance to โupgradeโ to your spiffy new Brave New World modernity.
It is way way way more complicated than people pretend. Iโve thought about this a huge, whole lot over the past few years. I would love to be shown that I am wrong. But I donโt think I am. Unicode is fundamentally more complex than the model that you would like to impose on it, and there is complexity here that you can never sweep under the carpet. If you try, youโll break either your own code or somebody elseโs. At some point, you simply have to break down and learn what Unicode is about. You cannot pretend it is something it is not.
๐ช goes out of its way to make Unicode easy, far more than anything else Iโve ever used. If you think this is bad, try something else for a while. Then come back to ๐ช: either you will have returned to a better world, or else you will bring knowledge of the same with you so that we can make use of your new knowledge to make ๐ช better at these things.
๐ก ๐ด๐๐๐๐ ๐๐๐ ๐ ๐๐๐๐๐๐๐ โธ ๐ฌ๐๐๐๐ ๐ช ๐ท๐๐๐๐๐๐ ๐ท๐๐๐ ๐ก
At a minimum, here are some things that would appear to be required for ๐ช to โenable Unicode by defaultโ, as you put it:
All ๐ช source code should be in UTF-8 by default. You can get that with use utf8 or export PERL5OPTS=-Mutf8.
The ๐ช DATA handle should be UTF-8. You will have to do this on a per-package basis, as in binmode(DATA, ":encoding(UTF-8)").
Program arguments to ๐ช scripts should be understood to be UTF-8 by default. export PERL_UNICODE=A, or perl -CA, or export PERL5OPTS=-CA.
The standard input, output, and error streams should default to UTF-8. export PERL_UNICODE=S for all of them, or I, O, and/or E for just some of them. This is like perl -CS.
Any other handles opened by ๐ช should be considered UTF-8 unless declared otherwise; export PERL_UNICODE=D or with i and o for particular ones of these; export PERL5OPTS=-CD would work. That makes -CSAD for all of them.
Cover both bases plus all the streams you open with export PERL5OPTS=-Mopen=:utf8,:std. See uniquote.
You donโt want to miss UTF-8 encoding errors. Try export PERL5OPTS=-Mwarnings=FATAL,utf8. And make sure your input streams are always binmoded to :encoding(UTF-8), not just to :utf8.
Code points between 128โ255 should be understood by ๐ช to be the corresponding Unicode code points, not just unpropertied binary values. use feature "unicode_strings" or export PERL5OPTS=-Mfeature=unicode_strings. That will make uc("\xDF") eq "SS" and "\xE9" =~ /\w/. A simple export PERL5OPTS=-Mv5.12 or better will also get that.
Named Unicode characters are not by default enabled, so add export PERL5OPTS=-Mcharnames=:full,:short,latin,greek or some such. See uninames and tcgrep.
You almost always need access to the functions from the standard Unicode::Normalize module various types of decompositions. export PERL5OPTS=-MUnicode::Normalize=NFD,NFKD,NFC,NFKD, and then always run incoming stuff through NFD and outbound stuff from NFC. Thereโs no I/O layer for these yet that Iโm aware of, but see nfc, nfd, nfkd, and nfkc.
String comparisons in ๐ช using eq, ne, lc, cmp, sort, &c&cc are always wrong. So instead of #a = sort #b, you need #a = Unicode::Collate->new->sort(#b). Might as well add that to your export PERL5OPTS=-MUnicode::Collate. You can cache the key for binary comparisons.
๐ช built-ins like printf and write do the wrong thing with Unicode data. You need to use the Unicode::GCString module for the former, and both that and also the Unicode::LineBreak module as well for the latter. See uwc and unifmt.
If you want them to count as integers, then you are going to have to run your \d+ captures through the Unicode::UCD::num function because ๐ชโs built-in atoi(3) isnโt currently clever enough.
You are going to have filesystem issues on ๐ฝ filesystems. Some filesystems silently enforce a conversion to NFC; others silently enforce a conversion to NFD. And others do something else still. Some even ignore the matter altogether, which leads to even greater problems. So you have to do your own NFC/NFD handling to keep sane.
All your ๐ช code involving a-z or A-Z and such MUST BE CHANGED, including m//, s///, and tr///. Itโs should stand out as a screaming red flag that your code is broken. But it is not clear how it must change. Getting the right properties, and understanding their casefolds, is harder than you might think. I use unichars and uniprops every single day.
Code that uses \p{Lu} is almost as wrong as code that uses [A-Za-z]. You need to use \p{Upper} instead, and know the reason why. Yes, \p{Lowercase} and \p{Lower} are different from \p{Ll} and \p{Lowercase_Letter}.
Code that uses [a-zA-Z] is even worse. And it canโt use \pL or \p{Letter}; it needs to use \p{Alphabetic}. Not all alphabetics are letters, you know!
If you are looking for ๐ช variables with /[\$\#\%]\w+/, then you have a problem. You need to look for /[\$\#\%]\p{IDS}\p{IDC}*/, and even that isnโt thinking about the punctuation variables or package variables.
If you are checking for whitespace, then you should choose between \h and \v, depending. And you should never use \s, since it DOES NOT MEAN [\h\v], contrary to popular belief.
If you are using \n for a line boundary, or even \r\n, then you are doing it wrong. You have to use \R, which is not the same!
If you donโt know when and whether to call Unicode::Stringprep, then you had better learn.
Case-insensitive comparisons need to check for whether two things are the same letters no matter their diacritics and such. The easiest way to do that is with the standard Unicode::Collate module. Unicode::Collate->new(level => 1)->cmp($a, $b). There are also eq methods and such, and you should probably learn about the match and substr methods, too. These are have distinct advantages over the ๐ช built-ins.
Sometimes thatโs still not enough, and you need the Unicode::Collate::Locale module instead, as in Unicode::Collate::Locale->new(locale => "de__phonebook", level => 1)->cmp($a, $b) instead. Consider that Unicode::Collate::->new(level => 1)->eq("d", "รฐ") is true, but Unicode::Collate::Locale->new(locale=>"is",level => 1)->eq("d", " รฐ") is false. Similarly, "ae" and "รฆ" are eq if you donโt use locales, or if you use the English one, but they are different in the Icelandic locale. Now what? Itโs tough, I tell you. You can play with ucsort to test some of these things out.
Consider how to match the pattern CVCV (consonsant, vowel, consonant, vowel) in the string โniรฑoโ. Its NFD form โ which you had darned well better have remembered to put it in โ becomes โnin\x{303}oโ. Now what are you going to do? Even pretending that a vowel is [aeiou] (which is wrong, by the way), you wonโt be able to do something like (?=[aeiou])\X) either, because even in NFD a code point like โรธโ does not decompose! However, it will test equal to an โoโ using the UCA comparison I just showed you. You canโt rely on NFD, you have to rely on UCA.
๐ฉ ๐ธ ๐ค ๐ค ๐ฆ ๐ ๐ ๐น ๐ฃ ๐ ๐ ๐ ๐ ๐ ๐ ๐ค ๐ค ๐ฉ
And thatโs not all. There are a million broken assumptions that people make about Unicode. Until they understand these things, their ๐ช code will be broken.
Code that assumes it can open a text file without specifying the encoding is broken.
Code that assumes the default encoding is some sort of native platform encoding is broken.
Code that assumes that web pages in Japanese or Chinese take up less space in UTFโ16 than in UTFโ8 is wrong.
Code that assumes Perl uses UTFโ8 internally is wrong.
Code that assumes that encoding errors will always raise an exception is wrong.
Code that assumes Perl code points are limited to 0x10_FFFF is wrong.
Code that assumes you can set $/ to something that will work with any valid line separator is wrong.
Code that assumes roundtrip equality on casefolding, like lc(uc($s)) eq $s or uc(lc($s)) eq $s, is completely broken and wrong. Consider that the uc("ฯ") and uc("ฯ") are both "ฮฃ", but lc("ฮฃ") cannot possibly return both of those.
Code that assumes every lowercase code point has a distinct uppercase one, or vice versa, is broken. For example, "ยช" is a lowercase letter with no uppercase; whereas both "แต" and "แดฌ" are letters, but they are not lowercase letters; however, they are both lowercase code points without corresponding uppercase versions. Got that? They are not \p{Lowercase_Letter}, despite being both \p{Letter} and \p{Lowercase}.
Code that assumes changing the case doesnโt change the length of the string is broken.
Code that assumes there are only two cases is broken. Thereโs also titlecase.
Code that assumes only letters have case is broken. Beyond just letters, it turns out that numbers, symbols, and even marks have case. In fact, changing the case can even make something change its main general category, like a \p{Mark} turning into a \p{Letter}. It can also make it switch from one script to another.
Code that assumes that case is never locale-dependent is broken.
Code that assumes Unicode gives a fig about POSIX locales is broken.
Code that assumes you can remove diacritics to get at base ASCII letters is evil, still, broken, brain-damaged, wrong, and justification for capital punishment.
Code that assumes that diacritics \p{Diacritic} and marks \p{Mark} are the same thing is broken.
Code that assumes \p{GC=Dash_Punctuation} covers as much as \p{Dash} is broken.
Code that assumes dash, hyphens, and minuses are the same thing as each other, or that there is only one of each, is broken and wrong.
Code that assumes every code point takes up no more than one print column is broken.
Code that assumes that all \p{Mark} characters take up zero print columns is broken.
Code that assumes that characters which look alike are alike is broken.
Code that assumes that characters which do not look alike are not alike is broken.
Code that assumes there is a limit to the number of code points in a row that just one \X can match is wrong.
Code that assumes \X can never start with a \p{Mark} character is wrong.
Code that assumes that \X can never hold two non-\p{Mark} characters is wrong.
Code that assumes that it cannot use "\x{FFFF}" is wrong.
Code that assumes a non-BMP code point that requires two UTF-16 (surrogate) code units will encode to two separate UTF-8 characters, one per code unit, is wrong. It doesnโt: it encodes to single code point.
Code that transcodes from UTFโ16 or UTFโ32 with leading BOMs into UTFโ8 is broken if it puts a BOM at the start of the resulting UTF-8. This is so stupid the engineer should have their eyelids removed.
Code that assumes the CESU-8 is a valid UTF encoding is wrong. Likewise, code that thinks encoding U+0000 as "\xC0\x80" is UTF-8 is broken and wrong. These guys also deserve the eyelid treatment.
Code that assumes characters like > always points to the right and < always points to the left are wrong โ because they in fact do not.
Code that assumes if you first output character X and then character Y, that those will show up as XY is wrong. Sometimes they donโt.
Code that assumes that ASCII is good enough for writing English properly is stupid, shortsighted, illiterate, broken, evil, and wrong. Off with their heads! If that seems too extreme, we can compromise: henceforth they may type only with their big toe from one foot. (The rest will be duct taped.)
Code that assumes that all \p{Math} code points are visible characters is wrong.
Code that assumes \w contains only letters, digits, and underscores is wrong.
Code that assumes that ^ and ~ are punctuation marks is wrong.
Code that assumes that รผ has an umlaut is wrong.
Code that believes things like โจ contain any letters in them is wrong.
Code that believes \p{InLatin} is the same as \p{Latin} is heinously broken.
Code that believe that \p{InLatin} is almost ever useful is almost certainly wrong.
Code that believes that given $FIRST_LETTER as the first letter in some alphabet and $LAST_LETTER as the last letter in that same alphabet, that [${FIRST_LETTER}-${LAST_LETTER}] has any meaning whatsoever is almost always complete broken and wrong and meaningless.
Code that believes someoneโs name can only contain certain characters is stupid, offensive, and wrong.
Code that tries to reduce Unicode to ASCII is not merely wrong, its perpetrator should never be allowed to work in programming again. Period. Iโm not even positive they should even be allowed to see again, since it obviously hasnโt done them much good so far.
Code that believes thereโs some way to pretend textfile encodings donโt exist is broken and dangerous. Might as well poke the other eye out, too.
Code that converts unknown characters to ? is broken, stupid, braindead, and runs contrary to the standard recommendation, which says NOT TO DO THAT! RTFM for why not.
Code that believes it can reliably guess the encoding of an unmarked textfile is guilty of a fatal mรฉlange of hubris and naรฏvetรฉ that only a lightning bolt from Zeus will fix.
Code that believes you can use ๐ช printf widths to pad and justify Unicode data is broken and wrong.
Code that believes once you successfully create a file by a given name, that when you run ls or readdir on its enclosing directory, youโll actually find that file with the name you created it under is buggy, broken, and wrong. Stop being surprised by this!
Code that believes UTF-16 is a fixed-width encoding is stupid, broken, and wrong. Revoke their programming licence.
Code that treats code points from one plane one whit differently than those from any other plane is ipso facto broken and wrong. Go back to school.
Code that believes that stuff like /s/i can only match "S" or "s" is broken and wrong. Youโd be surprised.
Code that uses \PM\pM* to find grapheme clusters instead of using \X is broken and wrong.
People who want to go back to the ASCII world should be whole-heartedly encouraged to do so, and in honor of their glorious upgrade they should be provided gratis with a pre-electric manual typewriter for all their data-entry needs. Messages sent to them should be sent via an แดสสแดแดแดs telegraph at 40 characters per line and hand-delivered by a courier. STOP.
๐ฑ ๐พ ๐ ๐ธ ๐ธ ๐ฌ ๐ฝ ๐ ๐ฑ
I donโt know how much more โdefault Unicode in ๐ชโ you can get than what Iโve written. Well, yes I do: you should be using Unicode::Collate and Unicode::LineBreak, too. And probably more.
As you see, there are far too many Unicode things that you really do have to worry about for there to ever exist any such thing as โdefault to Unicodeโ.
What youโre going to discover, just as we did back in ๐ช 5.8, that it is simply impossible to impose all these things on code that hasnโt been designed right from the beginning to account for them. Your well-meaning selfishness just broke the entire world.
And even once you do, there are still critical issues that require a great deal of thought to get right. There is no switch you can flip. Nothing but brain, and I mean real brain, will suffice here. Thereโs a heck of a lot of stuff you have to learn. Modulo the retreat to the manual typewriter, you simply cannot hope to sneak by in ignorance. This is the 21หขแต century, and you cannot wish Unicode away by willful ignorance.
You have to learn it. Period. It will never be so easy that โeverything just works,โ because that will guarantee that a lot of things donโt work โ which invalidates the assumption that there can ever be a way to โmake it all work.โ
You may be able to get a few reasonable defaults for a very few and very limited operations, but not without thinking about things a whole lot more than I think you have.
As just one example, canonical ordering is going to cause some real headaches. ๐ญ"\x{F5}" โรตโ, "o\x{303}" โรตโ, "o\x{303}\x{304}" โศญโ, and "o\x{304}\x{303}" โลฬโ should all match โรตโ, but how in the world are you going to do that? This is harder than it looks, but itโs something you need to account for. ๐ฃ
If thereโs one thing I know about Perl, it is what its Unicode bits do and do not do, and this thing I promise you: โ ฬฒแดฬฒสฬฒแดฬฒสฬฒแดฬฒ ฬฒษชฬฒsฬฒ ฬฒษดฬฒแดฬฒ ฬฒUฬฒษดฬฒษชฬฒแดฬฒแดฬฒแด
ฬฒแดฬฒ ฬฒแดฬฒแดฬฒษขฬฒษชฬฒแดฬฒ ฬฒสฬฒแดฬฒสฬฒสฬฒแดฬฒแดฬฒ ฬฒ โ ๐
You cannot just change some defaults and get smooth sailing. Itโs true that I run ๐ช with PERL_UNICODE set to "SA", but thatโs all, and even that is mostly for command-line stuff. For real work, I go through all the many steps outlined above, and I do it very, ** very** carefully.
๐ ยกฦจdlษษฅ ฦจแดษฅส ษdoษฅ puษ สปฮปษp ษษแดu ษ ษสษษฅ สปสษnl pooโ
๐
There are two stages to processing Unicode text. The first is "how can I input it and output it without losing information". The second is "how do I treat text according to local language conventions".
tchrist's post covers both, but the second part is where 99% of the text in his post comes from. Most programs don't even handle I/O correctly, so it's important to understand that before you even begin to worry about normalization and collation.
This post aims to solve that first problem
When you read data into Perl, it doesn't care what encoding it is. It allocates some memory and stashes the bytes away there. If you say print $str, it just blits those bytes out to your terminal, which is probably set to assume everything that is written to it is UTF-8, and your text shows up.
Marvelous.
Except, it's not. If you try to treat the data as text, you'll see that Something Bad is happening. You need go no further than length to see that what Perl thinks about your string and what you think about your string disagree. Write a one-liner like: perl -E 'while(<>){ chomp; say length }' and type in ๆๅญๅใ and you get 12... not the correct answer, 4.
That's because Perl assumes your string is not text. You have to tell it that it's text before it will give you the right answer.
That's easy enough; the Encode module has the functions to do that. The generic entry point is Encode::decode (or use Encode qw(decode), of course). That function takes some string from the outside world (what we'll call "octets", a fancy of way of saying "8-bit bytes"), and turns it into some text that Perl will understand. The first argument is a character encoding name, like "UTF-8" or "ASCII" or "EUC-JP". The second argument is the string. The return value is the Perl scalar containing the text.
(There is also Encode::decode_utf8, which assumes UTF-8 for the encoding.)
If we rewrite our one-liner:
perl -MEncode=decode -E 'while(<>){ chomp; say length decode("UTF-8", $_) }'
We type in ๆๅญๅใ and get "4" as the result. Success.
That, right there, is the solution to 99% of Unicode problems in Perl.
The key is, whenever any text comes into your program, you must decode it. The Internet cannot transmit characters. Files cannot store characters. There are no characters in your database. There are only octets, and you can't treat octets as characters in Perl. You must decode the encoded octets into Perl characters with the Encode module.
The other half of the problem is getting data out of your program. That's easy to; you just say use Encode qw(encode), decide what the encoding your data will be in (UTF-8 to terminals that understand UTF-8, UTF-16 for files on Windows, etc.), and then output the result of encode($encoding, $data) instead of just outputting $data.
This operation converts Perl's characters, which is what your program operates on, to octets that can be used by the outside world. It would be a lot easier if we could just send characters over the Internet or to our terminals, but we can't: octets only. So we have to convert characters to octets, otherwise the results are undefined.
To summarize: encode all outputs and decode all inputs.
Now we'll talk about three issues that make this a little challenging. The first is libraries. Do they handle text correctly? The answer is... they try. If you download a web page, LWP will give you your result back as text. If you call the right method on the result, that is (and that happens to be decoded_content, not content, which is just the octet stream that it got from the server.) Database drivers can be flaky; if you use DBD::SQLite with just Perl, it will work out, but if some other tool has put text stored as some encoding other than UTF-8 in your database... well... it's not going to be handled correctly until you write code to handle it correctly.
Outputting data is usually easier, but if you see "wide character in print", then you know you're messing up the encoding somewhere. That warning means "hey, you're trying to leak Perl characters to the outside world and that doesn't make any sense". Your program appears to work (because the other end usually handles the raw Perl characters correctly), but it is very broken and could stop working at any moment. Fix it with an explicit Encode::encode!
The second problem is UTF-8 encoded source code. Unless you say use utf8 at the top of each file, Perl will not assume that your source code is UTF-8. This means that each time you say something like my $var = 'ใปใ', you're injecting garbage into your program that will totally break everything horribly. You don't have to "use utf8", but if you don't, you must not use any non-ASCII characters in your program.
The third problem is how Perl handles The Past. A long time ago, there was no such thing as Unicode, and Perl assumed that everything was Latin-1 text or binary. So when data comes into your program and you start treating it as text, Perl treats each octet as a Latin-1 character. That's why, when we asked for the length of "ๆๅญๅใ", we got 12. Perl assumed that we were operating on the Latin-1 string "รฆรฅยญรฅรฃ" (which is 12 characters, some of which are non-printing).
This is called an "implicit upgrade", and it's a perfectly reasonable thing to do, but it's not what you want if your text is not Latin-1. That's why it's critical to explicitly decode input: if you don't do it, Perl will, and it might do it wrong.
People run into trouble where half their data is a proper character string, and some is still binary. Perl will interpret the part that's still binary as though it's Latin-1 text and then combine it with the correct character data. This will make it look like handling your characters correctly broke your program, but in reality, you just haven't fixed it enough.
Here's an example: you have a program that reads a UTF-8-encoded text file, you tack on a Unicode PILE OF POO to each line, and you print it out. You write it like:
while(<>){
chomp;
say "$_ ๐ฉ";
}
And then run on some UTF-8 encoded data, like:
perl poo.pl input-data.txt
It prints the UTF-8 data with a poo at the end of each line. Perfect, my program works!
But nope, you're just doing binary concatenation. You're reading octets from the file, removing a \n with chomp, and then tacking on the bytes in the UTF-8 representation of the PILE OF POO character. When you revise your program to decode the data from the file and encode the output, you'll notice that you get garbage ("รฐยฉ") instead of the poo. This will lead you to believe that decoding the input file is the wrong thing to do. It's not.
The problem is that the poo is being implicitly upgraded as latin-1. If you use utf8 to make the literal text instead of binary, then it will work again!
(That's the number one problem I see when helping people with Unicode. They did part right and that broke their program. That's what's sad about undefined results: you can have a working program for a long time, but when you start to repair it, it breaks. Don't worry; if you are adding encode/decode statements to your program and it breaks, it just means you have more work to do. Next time, when you design with Unicode in mind from the beginning, it will be much easier!)
That's really all you need to know about Perl and Unicode. If you tell Perl what your data is, it has the best Unicode support among all popular programming languages. If you assume it will magically know what sort of text you are feeding it, though, then you're going to trash your data irrevocably. Just because your program works today on your UTF-8 terminal doesn't mean it will work tomorrow on a UTF-16 encoded file. So make it safe now, and save yourself the headache of trashing your users' data!
The easy part of handling Unicode is encoding output and decoding input. The hard part is finding all your input and output, and determining which encoding it is. But that's why you get the big bucks :)
We're all in agreement that it is a difficult problem for many reasons,
but that's precisely the reason to try to make it easier on everybody.
There is a recent module on CPAN, utf8::all, that attempts to "turn on Unicode. All of it".
As has been pointed out, you can't magically make the entire system (outside programs, external web requests, etc.) use Unicode as well, but we can work together to make sensible tools that make doing common problems easier. That's the reason that we're programmers.
If utf8::all doesn't do something you think it should, let's improve it to make it better. Or let's make additional tools that together can suit people's varying needs as well as possible.
`
I think you misunderstand Unicode and its relationship to Perl. No matter which way you store data, Unicode, ISO-8859-1, or many other things, your program has to know how to interpret the bytes it gets as input (decoding) and how to represent the information it wants to output (encoding). Get that interpretation wrong and you garble the data. There isn't some magic default setup inside your program that's going to tell the stuff outside your program how to act.
You think it's hard, most likely, because you are used to everything being ASCII. Everything you should have been thinking about was simply ignored by the programming language and all of the things it had to interact with. If everything used nothing but UTF-8 and you had no choice, then UTF-8 would be just as easy. But not everything does use UTF-8. For instance, you don't want your input handle to think that it's getting UTF-8 octets unless it actually is, and you don't want your output handles to be UTF-8 if the thing reading from them can't handle UTF-8. Perl has no way to know those things. That's why you are the programmer.
I don't think Unicode in Perl 5 is too complicated. I think it's scary and people avoid it. There's a difference. To that end, I've put Unicode in Learning Perl, 6th Edition, and there's a lot of Unicode stuff in Effective Perl Programming. You have to spend the time to learn and understand Unicode and how it works. You're not going to be able to use it effectively otherwise.
While reading this thread, I often get the impression that people are using "UTF-8" as a synonym to "Unicode". Please make a distinction between Unicode's "Code-Points" which are an enlarged relative of the ASCII code and Unicode's various "encodings". And there are a few of them, of which UTF-8, UTF-16 and UTF-32 are the current ones and a few more are obsolete.
Please, UTF-8 (as well as all other encodings) exists and have meaning in input or in output only. Internally, since Perl 5.8.1, all strings are kept as Unicode "Code-points". True, you have to enable some features as admiringly covered previously.
There's a truly horrifying amount of ancient code out there in the wild, much of it in the form of common CPAN modules. I've found I have to be fairly careful enabling Unicode if I use external modules that might be affected by it, and am still trying to identify and fix some Unicode-related failures in several Perl scripts I use regularly (in particular, iTiVo fails badly on anything that's not 7-bit ASCII due to transcoding issues).
You should enable the unicode strings feature, and this is the default if you use v5.14;
You should not really use unicode identifiers esp. for foreign code via utf8 as they are insecure in perl5, only cperl got that right. See e.g. http://perl11.org/blog/unicode-identifiers.html
Regarding utf8 for your filehandles/streams: You need decide by yourself the encoding of your external data. A library cannot know that, and since not even libc supports utf8, proper utf8 data is rare. There's more wtf8, the windows aberration of utf8 around.
BTW: Moose is not really "Modern Perl", they just hijacked the name. Moose is perfect Larry Wall-style postmodern perl mixed with Bjarne Stroustrup-style everything goes, with an eclectic aberration of proper perl6 syntax, e.g. using strings for variable names, horrible fields syntax, and a very immature naive implementation which is 10x slower than a proper implementation.
cperl and perl6 are the true modern perls, where form follows function, and the implementation is reduced and optimized.
Quoted from in the CGI.pm docs:
When using this, the thing you must watch out for are multivalued CGI
parameters. Because a hash cannot distinguish between scalar and list
context, multivalued parameters will be returned as a packed string,
separated by the "\0" (null) character.
However, as it turns out, \0 is nothing special in Perl:
print length("test\0hi");
The output is :
7
whereas in C it should be 4.
Why does CGI.pm still use \0 as null character, when it's treated as a normal character (not the mark of end of string any more) in Perl?
It's a design mistake. I think we agree that it should not coerce the hash value to a string at all, but it probably seemed like a good idea back then and \0 simply is the least bad choice for various reasons of little importance.
Edit: People usually avoid to put NULs in their data precisely because it tends to cause breakage in C programs, so this makes this character slightly more favourable as separator.
Edit 2: hobbs comments that it goes back to Perl 4, so the mistake is not in the original design, but in carrying it over and then not trying hard enough to deprecate the feature.
Well, hindsight is always perfect. Hash::MultiValue is the smarter data structure you were thinking of.
It's a security feature.
Users of ->Vars expect a hash of key-values, where the values are strings. If one of the value happens to be a reference to an array, it would break that expectation and it could cause the program to behave badly.
If you want to support arguments with multiple values, use ->param in list context. You can use it to build your own hash, if you want.
my %hash;
for ($cgi->params) {
$hash{$_} = [ $cgi->param($_) ];
}
I strongly disagree about it being a design error. I think it's very very smart way of handling bad data (multiple instances of a parameter where at most one is expected).
I wonder why most modern solutions built using Perl don't enable UTF-8 by default.
I understand there are many legacy problems for core Perl scripts, where it may break things. But, from my point of view, in the 21st century, big new projects (or projects with a big perspective) should make their software UTF-8 proof from scratch. Still I don't see it happening. For example, Moose enables strict and warnings, but not Unicode. Modern::Perl reduces boilerplate too, but no UTF-8 handling.
Why? Are there some reasons to avoid UTF-8 in modern Perl projects in the year 2011?
Commenting #tchrist got too long, so I'm adding it here.
It seems that I did not make myself clear. Let me try to add some things.
tchrist and I see situation pretty similarly, but our conclusions are completely in opposite ends. I agree, the situation with Unicode is complicated, but this is why we (Perl users and coders) need some layer (or pragma) which makes UTF-8 handling as easy as it must be nowadays.
tchrist pointed to many aspects to cover, I will read and think about them for days or even weeks. Still, this is not my point. tchrist tries to prove that there is not one single way "to enable UTF-8". I have not so much knowledge to argue with that. So, I stick to live examples.
I played around with Rakudo and UTF-8 was just there as I needed. I didn't have any problems, it just worked. Maybe there are some limitation somewhere deeper, but at start, all I tested worked as I expected.
Shouldn't that be a goal in modern Perl 5 too? I stress it more: I'm not suggesting UTF-8 as the default character set for core Perl, I suggest the possibility to trigger it with a snap for those who develop new projects.
Another example, but with a more negative tone. Frameworks should make development easier. Some years ago, I tried web frameworks, but just threw them away because "enabling UTF-8" was so obscure. I did not find how and where to hook Unicode support. It was so time-consuming that I found it easier to go the old way. Now I saw here there was a bounty to deal with the same problem with Mason 2: How to make Mason2 UTF-8 clean?. So, it is pretty new framework, but using it with UTF-8 needs deep knowledge of its internals. It is like a big red sign: STOP, don't use me!
I really like Perl. But dealing with Unicode is painful. I still find myself running against walls. Some way tchrist is right and answers my questions: new projects don't attract UTF-8 because it is too complicated in Perl 5.
๐๐๐ข๐ฅ๐ก๐๐จ๐ฉ โ: ๐ ๐ฟ๐๐จ๐๐ง๐๐ฉ๐ ๐๐๐๐ค๐ข๐ข๐๐ฃ๐๐๐ฉ๐๐ค๐ฃ๐จ
Set your PERL_UNICODE envariable to AS. This makes all Perl scripts decode #ARGV as UTFโ8 strings, and sets the encoding of all three of stdin, stdout, and stderr to UTFโ8. Both these are global effects, not lexical ones.
At the top of your source file (program, module, library, dohickey), prominently assert that you are running perl version 5.12 or better via:
use v5.12; # minimal for unicode string feature
use v5.14; # optimal for unicode string feature
Enable warnings, since the previous declaration only enables strictures and features, not warnings. I also suggest promoting Unicode warnings into exceptions, so use both these lines, not just one of them. Note however that under v5.14, the utf8 warning class comprises three other subwarnings which can all be separately enabled: nonchar, surrogate, and non_unicode. These you may wish to exert greater control over.
use warnings;
use warnings qw( FATAL utf8 );
Declare that this source unit is encoded as UTFโ8. Although once upon a time this pragma did other things, it now serves this one singular purpose alone and no other:
use utf8;
Declare that anything that opens a filehandle within this lexical scope but not elsewhere is to assume that that stream is encoded in UTFโ8 unless you tell it otherwise. That way you do not affect other moduleโs or other programโs code.
use open qw( :encoding(UTF-8) :std );
Enable named characters via \N{CHARNAME}.
use charnames qw( :full :short );
If you have a DATA handle, you must explicitly set its encoding. If you want this to be UTFโ8, then say:
binmode(DATA, ":encoding(UTF-8)");
There is of course no end of other matters with which you may eventually find yourself concerned, but these will suffice to approximate the state goal to โmake everything just work with UTFโ8โ, albeit for a somewhat weakened sense of those terms.
One other pragma, although it is not Unicode related, is:
use autodie;
It is strongly recommended.
๐ด ๐ช๐ซ๐ช ๐ ๐ฒ๐ ๐ฟ๐๐๐ ๐๐๐ ๐ฏ๐ ๐ท๐๐๐๐๐๐๐ ๐ ๐ช๐ซ๐ช ๐
๐ ๐ช ๐ญ๐๐๐๐๐โธ๐๐๐๐๐ ๐๐๐ ๐๐๐๐๐๐๐โธ๐ฌ๐๐๐๐ ๐ฎ๐๐๐ ๐ช ๐
My own boilerplate these days tends to look like this:
use 5.014;
use utf8;
use strict;
use autodie;
use warnings;
use warnings qw< FATAL utf8 >;
use open qw< :std :utf8 >;
use charnames qw< :full >;
use feature qw< unicode_strings >;
use File::Basename qw< basename >;
use Carp qw< carp croak confess cluck >;
use Encode qw< encode decode >;
use Unicode::Normalize qw< NFD NFC >;
END { close STDOUT }
if (grep /\P{ASCII}/ => #ARGV) {
#ARGV = map { decode("UTF-8", $_) } #ARGV;
}
$0 = basename($0); # shorter messages
$| = 1;
binmode(DATA, ":utf8");
# give a full stack dump on any untrapped exceptions
local $SIG{__DIE__} = sub {
confess "Uncaught exception: #_" unless $^S;
};
# now promote run-time warnings into stack-dumped
# exceptions *unless* we're in an try block, in
# which case just cluck the stack dump instead
local $SIG{__WARN__} = sub {
if ($^S) { cluck "Trapped warning: #_" }
else { confess "Deadly warning: #_" }
};
while (<>) {
chomp;
$_ = NFD($_);
...
} continue {
say NFC($_);
}
__END__
๐
๐น ๐ ๐ธ ๐ ๐ ๐ ๐ ๐ญ ๐ ๐ ๐ ๐ ๐ ๐
Saying that โPerl should [somehow!] enable Unicode by defaultโ doesnโt even start to begin to think about getting around to saying enough to be even marginally useful in some sort of rare and isolated case. Unicode is much much more than just a larger character repertoire; itโs also how those characters all interact in many, many ways.
Even the simple-minded minimal measures that (some) people seem to think they want are guaranteed to miserably break millions of lines of code, code that has no chance to โupgradeโ to your spiffy new Brave New World modernity.
It is way way way more complicated than people pretend. Iโve thought about this a huge, whole lot over the past few years. I would love to be shown that I am wrong. But I donโt think I am. Unicode is fundamentally more complex than the model that you would like to impose on it, and there is complexity here that you can never sweep under the carpet. If you try, youโll break either your own code or somebody elseโs. At some point, you simply have to break down and learn what Unicode is about. You cannot pretend it is something it is not.
๐ช goes out of its way to make Unicode easy, far more than anything else Iโve ever used. If you think this is bad, try something else for a while. Then come back to ๐ช: either you will have returned to a better world, or else you will bring knowledge of the same with you so that we can make use of your new knowledge to make ๐ช better at these things.
๐ก ๐ด๐๐๐๐ ๐๐๐ ๐ ๐๐๐๐๐๐๐ โธ ๐ฌ๐๐๐๐ ๐ช ๐ท๐๐๐๐๐๐ ๐ท๐๐๐ ๐ก
At a minimum, here are some things that would appear to be required for ๐ช to โenable Unicode by defaultโ, as you put it:
All ๐ช source code should be in UTF-8 by default. You can get that with use utf8 or export PERL5OPTS=-Mutf8.
The ๐ช DATA handle should be UTF-8. You will have to do this on a per-package basis, as in binmode(DATA, ":encoding(UTF-8)").
Program arguments to ๐ช scripts should be understood to be UTF-8 by default. export PERL_UNICODE=A, or perl -CA, or export PERL5OPTS=-CA.
The standard input, output, and error streams should default to UTF-8. export PERL_UNICODE=S for all of them, or I, O, and/or E for just some of them. This is like perl -CS.
Any other handles opened by ๐ช should be considered UTF-8 unless declared otherwise; export PERL_UNICODE=D or with i and o for particular ones of these; export PERL5OPTS=-CD would work. That makes -CSAD for all of them.
Cover both bases plus all the streams you open with export PERL5OPTS=-Mopen=:utf8,:std. See uniquote.
You donโt want to miss UTF-8 encoding errors. Try export PERL5OPTS=-Mwarnings=FATAL,utf8. And make sure your input streams are always binmoded to :encoding(UTF-8), not just to :utf8.
Code points between 128โ255 should be understood by ๐ช to be the corresponding Unicode code points, not just unpropertied binary values. use feature "unicode_strings" or export PERL5OPTS=-Mfeature=unicode_strings. That will make uc("\xDF") eq "SS" and "\xE9" =~ /\w/. A simple export PERL5OPTS=-Mv5.12 or better will also get that.
Named Unicode characters are not by default enabled, so add export PERL5OPTS=-Mcharnames=:full,:short,latin,greek or some such. See uninames and tcgrep.
You almost always need access to the functions from the standard Unicode::Normalize module various types of decompositions. export PERL5OPTS=-MUnicode::Normalize=NFD,NFKD,NFC,NFKD, and then always run incoming stuff through NFD and outbound stuff from NFC. Thereโs no I/O layer for these yet that Iโm aware of, but see nfc, nfd, nfkd, and nfkc.
String comparisons in ๐ช using eq, ne, lc, cmp, sort, &c&cc are always wrong. So instead of #a = sort #b, you need #a = Unicode::Collate->new->sort(#b). Might as well add that to your export PERL5OPTS=-MUnicode::Collate. You can cache the key for binary comparisons.
๐ช built-ins like printf and write do the wrong thing with Unicode data. You need to use the Unicode::GCString module for the former, and both that and also the Unicode::LineBreak module as well for the latter. See uwc and unifmt.
If you want them to count as integers, then you are going to have to run your \d+ captures through the Unicode::UCD::num function because ๐ชโs built-in atoi(3) isnโt currently clever enough.
You are going to have filesystem issues on ๐ฝ filesystems. Some filesystems silently enforce a conversion to NFC; others silently enforce a conversion to NFD. And others do something else still. Some even ignore the matter altogether, which leads to even greater problems. So you have to do your own NFC/NFD handling to keep sane.
All your ๐ช code involving a-z or A-Z and such MUST BE CHANGED, including m//, s///, and tr///. Itโs should stand out as a screaming red flag that your code is broken. But it is not clear how it must change. Getting the right properties, and understanding their casefolds, is harder than you might think. I use unichars and uniprops every single day.
Code that uses \p{Lu} is almost as wrong as code that uses [A-Za-z]. You need to use \p{Upper} instead, and know the reason why. Yes, \p{Lowercase} and \p{Lower} are different from \p{Ll} and \p{Lowercase_Letter}.
Code that uses [a-zA-Z] is even worse. And it canโt use \pL or \p{Letter}; it needs to use \p{Alphabetic}. Not all alphabetics are letters, you know!
If you are looking for ๐ช variables with /[\$\#\%]\w+/, then you have a problem. You need to look for /[\$\#\%]\p{IDS}\p{IDC}*/, and even that isnโt thinking about the punctuation variables or package variables.
If you are checking for whitespace, then you should choose between \h and \v, depending. And you should never use \s, since it DOES NOT MEAN [\h\v], contrary to popular belief.
If you are using \n for a line boundary, or even \r\n, then you are doing it wrong. You have to use \R, which is not the same!
If you donโt know when and whether to call Unicode::Stringprep, then you had better learn.
Case-insensitive comparisons need to check for whether two things are the same letters no matter their diacritics and such. The easiest way to do that is with the standard Unicode::Collate module. Unicode::Collate->new(level => 1)->cmp($a, $b). There are also eq methods and such, and you should probably learn about the match and substr methods, too. These are have distinct advantages over the ๐ช built-ins.
Sometimes thatโs still not enough, and you need the Unicode::Collate::Locale module instead, as in Unicode::Collate::Locale->new(locale => "de__phonebook", level => 1)->cmp($a, $b) instead. Consider that Unicode::Collate::->new(level => 1)->eq("d", "รฐ") is true, but Unicode::Collate::Locale->new(locale=>"is",level => 1)->eq("d", " รฐ") is false. Similarly, "ae" and "รฆ" are eq if you donโt use locales, or if you use the English one, but they are different in the Icelandic locale. Now what? Itโs tough, I tell you. You can play with ucsort to test some of these things out.
Consider how to match the pattern CVCV (consonsant, vowel, consonant, vowel) in the string โniรฑoโ. Its NFD form โ which you had darned well better have remembered to put it in โ becomes โnin\x{303}oโ. Now what are you going to do? Even pretending that a vowel is [aeiou] (which is wrong, by the way), you wonโt be able to do something like (?=[aeiou])\X) either, because even in NFD a code point like โรธโ does not decompose! However, it will test equal to an โoโ using the UCA comparison I just showed you. You canโt rely on NFD, you have to rely on UCA.
๐ฉ ๐ธ ๐ค ๐ค ๐ฆ ๐ ๐ ๐น ๐ฃ ๐ ๐ ๐ ๐ ๐ ๐ ๐ค ๐ค ๐ฉ
And thatโs not all. There are a million broken assumptions that people make about Unicode. Until they understand these things, their ๐ช code will be broken.
Code that assumes it can open a text file without specifying the encoding is broken.
Code that assumes the default encoding is some sort of native platform encoding is broken.
Code that assumes that web pages in Japanese or Chinese take up less space in UTFโ16 than in UTFโ8 is wrong.
Code that assumes Perl uses UTFโ8 internally is wrong.
Code that assumes that encoding errors will always raise an exception is wrong.
Code that assumes Perl code points are limited to 0x10_FFFF is wrong.
Code that assumes you can set $/ to something that will work with any valid line separator is wrong.
Code that assumes roundtrip equality on casefolding, like lc(uc($s)) eq $s or uc(lc($s)) eq $s, is completely broken and wrong. Consider that the uc("ฯ") and uc("ฯ") are both "ฮฃ", but lc("ฮฃ") cannot possibly return both of those.
Code that assumes every lowercase code point has a distinct uppercase one, or vice versa, is broken. For example, "ยช" is a lowercase letter with no uppercase; whereas both "แต" and "แดฌ" are letters, but they are not lowercase letters; however, they are both lowercase code points without corresponding uppercase versions. Got that? They are not \p{Lowercase_Letter}, despite being both \p{Letter} and \p{Lowercase}.
Code that assumes changing the case doesnโt change the length of the string is broken.
Code that assumes there are only two cases is broken. Thereโs also titlecase.
Code that assumes only letters have case is broken. Beyond just letters, it turns out that numbers, symbols, and even marks have case. In fact, changing the case can even make something change its main general category, like a \p{Mark} turning into a \p{Letter}. It can also make it switch from one script to another.
Code that assumes that case is never locale-dependent is broken.
Code that assumes Unicode gives a fig about POSIX locales is broken.
Code that assumes you can remove diacritics to get at base ASCII letters is evil, still, broken, brain-damaged, wrong, and justification for capital punishment.
Code that assumes that diacritics \p{Diacritic} and marks \p{Mark} are the same thing is broken.
Code that assumes \p{GC=Dash_Punctuation} covers as much as \p{Dash} is broken.
Code that assumes dash, hyphens, and minuses are the same thing as each other, or that there is only one of each, is broken and wrong.
Code that assumes every code point takes up no more than one print column is broken.
Code that assumes that all \p{Mark} characters take up zero print columns is broken.
Code that assumes that characters which look alike are alike is broken.
Code that assumes that characters which do not look alike are not alike is broken.
Code that assumes there is a limit to the number of code points in a row that just one \X can match is wrong.
Code that assumes \X can never start with a \p{Mark} character is wrong.
Code that assumes that \X can never hold two non-\p{Mark} characters is wrong.
Code that assumes that it cannot use "\x{FFFF}" is wrong.
Code that assumes a non-BMP code point that requires two UTF-16 (surrogate) code units will encode to two separate UTF-8 characters, one per code unit, is wrong. It doesnโt: it encodes to single code point.
Code that transcodes from UTFโ16 or UTFโ32 with leading BOMs into UTFโ8 is broken if it puts a BOM at the start of the resulting UTF-8. This is so stupid the engineer should have their eyelids removed.
Code that assumes the CESU-8 is a valid UTF encoding is wrong. Likewise, code that thinks encoding U+0000 as "\xC0\x80" is UTF-8 is broken and wrong. These guys also deserve the eyelid treatment.
Code that assumes characters like > always points to the right and < always points to the left are wrong โ because they in fact do not.
Code that assumes if you first output character X and then character Y, that those will show up as XY is wrong. Sometimes they donโt.
Code that assumes that ASCII is good enough for writing English properly is stupid, shortsighted, illiterate, broken, evil, and wrong. Off with their heads! If that seems too extreme, we can compromise: henceforth they may type only with their big toe from one foot. (The rest will be duct taped.)
Code that assumes that all \p{Math} code points are visible characters is wrong.
Code that assumes \w contains only letters, digits, and underscores is wrong.
Code that assumes that ^ and ~ are punctuation marks is wrong.
Code that assumes that รผ has an umlaut is wrong.
Code that believes things like โจ contain any letters in them is wrong.
Code that believes \p{InLatin} is the same as \p{Latin} is heinously broken.
Code that believe that \p{InLatin} is almost ever useful is almost certainly wrong.
Code that believes that given $FIRST_LETTER as the first letter in some alphabet and $LAST_LETTER as the last letter in that same alphabet, that [${FIRST_LETTER}-${LAST_LETTER}] has any meaning whatsoever is almost always complete broken and wrong and meaningless.
Code that believes someoneโs name can only contain certain characters is stupid, offensive, and wrong.
Code that tries to reduce Unicode to ASCII is not merely wrong, its perpetrator should never be allowed to work in programming again. Period. Iโm not even positive they should even be allowed to see again, since it obviously hasnโt done them much good so far.
Code that believes thereโs some way to pretend textfile encodings donโt exist is broken and dangerous. Might as well poke the other eye out, too.
Code that converts unknown characters to ? is broken, stupid, braindead, and runs contrary to the standard recommendation, which says NOT TO DO THAT! RTFM for why not.
Code that believes it can reliably guess the encoding of an unmarked textfile is guilty of a fatal mรฉlange of hubris and naรฏvetรฉ that only a lightning bolt from Zeus will fix.
Code that believes you can use ๐ช printf widths to pad and justify Unicode data is broken and wrong.
Code that believes once you successfully create a file by a given name, that when you run ls or readdir on its enclosing directory, youโll actually find that file with the name you created it under is buggy, broken, and wrong. Stop being surprised by this!
Code that believes UTF-16 is a fixed-width encoding is stupid, broken, and wrong. Revoke their programming licence.
Code that treats code points from one plane one whit differently than those from any other plane is ipso facto broken and wrong. Go back to school.
Code that believes that stuff like /s/i can only match "S" or "s" is broken and wrong. Youโd be surprised.
Code that uses \PM\pM* to find grapheme clusters instead of using \X is broken and wrong.
People who want to go back to the ASCII world should be whole-heartedly encouraged to do so, and in honor of their glorious upgrade they should be provided gratis with a pre-electric manual typewriter for all their data-entry needs. Messages sent to them should be sent via an แดสสแดแดแดs telegraph at 40 characters per line and hand-delivered by a courier. STOP.
๐ฑ ๐พ ๐ ๐ธ ๐ธ ๐ฌ ๐ฝ ๐ ๐ฑ
I donโt know how much more โdefault Unicode in ๐ชโ you can get than what Iโve written. Well, yes I do: you should be using Unicode::Collate and Unicode::LineBreak, too. And probably more.
As you see, there are far too many Unicode things that you really do have to worry about for there to ever exist any such thing as โdefault to Unicodeโ.
What youโre going to discover, just as we did back in ๐ช 5.8, that it is simply impossible to impose all these things on code that hasnโt been designed right from the beginning to account for them. Your well-meaning selfishness just broke the entire world.
And even once you do, there are still critical issues that require a great deal of thought to get right. There is no switch you can flip. Nothing but brain, and I mean real brain, will suffice here. Thereโs a heck of a lot of stuff you have to learn. Modulo the retreat to the manual typewriter, you simply cannot hope to sneak by in ignorance. This is the 21หขแต century, and you cannot wish Unicode away by willful ignorance.
You have to learn it. Period. It will never be so easy that โeverything just works,โ because that will guarantee that a lot of things donโt work โ which invalidates the assumption that there can ever be a way to โmake it all work.โ
You may be able to get a few reasonable defaults for a very few and very limited operations, but not without thinking about things a whole lot more than I think you have.
As just one example, canonical ordering is going to cause some real headaches. ๐ญ"\x{F5}" โรตโ, "o\x{303}" โรตโ, "o\x{303}\x{304}" โศญโ, and "o\x{304}\x{303}" โลฬโ should all match โรตโ, but how in the world are you going to do that? This is harder than it looks, but itโs something you need to account for. ๐ฃ
If thereโs one thing I know about Perl, it is what its Unicode bits do and do not do, and this thing I promise you: โ ฬฒแดฬฒสฬฒแดฬฒสฬฒแดฬฒ ฬฒษชฬฒsฬฒ ฬฒษดฬฒแดฬฒ ฬฒUฬฒษดฬฒษชฬฒแดฬฒแดฬฒแด
ฬฒแดฬฒ ฬฒแดฬฒแดฬฒษขฬฒษชฬฒแดฬฒ ฬฒสฬฒแดฬฒสฬฒสฬฒแดฬฒแดฬฒ ฬฒ โ ๐
You cannot just change some defaults and get smooth sailing. Itโs true that I run ๐ช with PERL_UNICODE set to "SA", but thatโs all, and even that is mostly for command-line stuff. For real work, I go through all the many steps outlined above, and I do it very, ** very** carefully.
๐ ยกฦจdlษษฅ ฦจแดษฅส ษdoษฅ puษ สปฮปษp ษษแดu ษ ษสษษฅ สปสษnl pooโ
๐
There are two stages to processing Unicode text. The first is "how can I input it and output it without losing information". The second is "how do I treat text according to local language conventions".
tchrist's post covers both, but the second part is where 99% of the text in his post comes from. Most programs don't even handle I/O correctly, so it's important to understand that before you even begin to worry about normalization and collation.
This post aims to solve that first problem
When you read data into Perl, it doesn't care what encoding it is. It allocates some memory and stashes the bytes away there. If you say print $str, it just blits those bytes out to your terminal, which is probably set to assume everything that is written to it is UTF-8, and your text shows up.
Marvelous.
Except, it's not. If you try to treat the data as text, you'll see that Something Bad is happening. You need go no further than length to see that what Perl thinks about your string and what you think about your string disagree. Write a one-liner like: perl -E 'while(<>){ chomp; say length }' and type in ๆๅญๅใ and you get 12... not the correct answer, 4.
That's because Perl assumes your string is not text. You have to tell it that it's text before it will give you the right answer.
That's easy enough; the Encode module has the functions to do that. The generic entry point is Encode::decode (or use Encode qw(decode), of course). That function takes some string from the outside world (what we'll call "octets", a fancy of way of saying "8-bit bytes"), and turns it into some text that Perl will understand. The first argument is a character encoding name, like "UTF-8" or "ASCII" or "EUC-JP". The second argument is the string. The return value is the Perl scalar containing the text.
(There is also Encode::decode_utf8, which assumes UTF-8 for the encoding.)
If we rewrite our one-liner:
perl -MEncode=decode -E 'while(<>){ chomp; say length decode("UTF-8", $_) }'
We type in ๆๅญๅใ and get "4" as the result. Success.
That, right there, is the solution to 99% of Unicode problems in Perl.
The key is, whenever any text comes into your program, you must decode it. The Internet cannot transmit characters. Files cannot store characters. There are no characters in your database. There are only octets, and you can't treat octets as characters in Perl. You must decode the encoded octets into Perl characters with the Encode module.
The other half of the problem is getting data out of your program. That's easy to; you just say use Encode qw(encode), decide what the encoding your data will be in (UTF-8 to terminals that understand UTF-8, UTF-16 for files on Windows, etc.), and then output the result of encode($encoding, $data) instead of just outputting $data.
This operation converts Perl's characters, which is what your program operates on, to octets that can be used by the outside world. It would be a lot easier if we could just send characters over the Internet or to our terminals, but we can't: octets only. So we have to convert characters to octets, otherwise the results are undefined.
To summarize: encode all outputs and decode all inputs.
Now we'll talk about three issues that make this a little challenging. The first is libraries. Do they handle text correctly? The answer is... they try. If you download a web page, LWP will give you your result back as text. If you call the right method on the result, that is (and that happens to be decoded_content, not content, which is just the octet stream that it got from the server.) Database drivers can be flaky; if you use DBD::SQLite with just Perl, it will work out, but if some other tool has put text stored as some encoding other than UTF-8 in your database... well... it's not going to be handled correctly until you write code to handle it correctly.
Outputting data is usually easier, but if you see "wide character in print", then you know you're messing up the encoding somewhere. That warning means "hey, you're trying to leak Perl characters to the outside world and that doesn't make any sense". Your program appears to work (because the other end usually handles the raw Perl characters correctly), but it is very broken and could stop working at any moment. Fix it with an explicit Encode::encode!
The second problem is UTF-8 encoded source code. Unless you say use utf8 at the top of each file, Perl will not assume that your source code is UTF-8. This means that each time you say something like my $var = 'ใปใ', you're injecting garbage into your program that will totally break everything horribly. You don't have to "use utf8", but if you don't, you must not use any non-ASCII characters in your program.
The third problem is how Perl handles The Past. A long time ago, there was no such thing as Unicode, and Perl assumed that everything was Latin-1 text or binary. So when data comes into your program and you start treating it as text, Perl treats each octet as a Latin-1 character. That's why, when we asked for the length of "ๆๅญๅใ", we got 12. Perl assumed that we were operating on the Latin-1 string "รฆรฅยญรฅรฃ" (which is 12 characters, some of which are non-printing).
This is called an "implicit upgrade", and it's a perfectly reasonable thing to do, but it's not what you want if your text is not Latin-1. That's why it's critical to explicitly decode input: if you don't do it, Perl will, and it might do it wrong.
People run into trouble where half their data is a proper character string, and some is still binary. Perl will interpret the part that's still binary as though it's Latin-1 text and then combine it with the correct character data. This will make it look like handling your characters correctly broke your program, but in reality, you just haven't fixed it enough.
Here's an example: you have a program that reads a UTF-8-encoded text file, you tack on a Unicode PILE OF POO to each line, and you print it out. You write it like:
while(<>){
chomp;
say "$_ ๐ฉ";
}
And then run on some UTF-8 encoded data, like:
perl poo.pl input-data.txt
It prints the UTF-8 data with a poo at the end of each line. Perfect, my program works!
But nope, you're just doing binary concatenation. You're reading octets from the file, removing a \n with chomp, and then tacking on the bytes in the UTF-8 representation of the PILE OF POO character. When you revise your program to decode the data from the file and encode the output, you'll notice that you get garbage ("รฐยฉ") instead of the poo. This will lead you to believe that decoding the input file is the wrong thing to do. It's not.
The problem is that the poo is being implicitly upgraded as latin-1. If you use utf8 to make the literal text instead of binary, then it will work again!
(That's the number one problem I see when helping people with Unicode. They did part right and that broke their program. That's what's sad about undefined results: you can have a working program for a long time, but when you start to repair it, it breaks. Don't worry; if you are adding encode/decode statements to your program and it breaks, it just means you have more work to do. Next time, when you design with Unicode in mind from the beginning, it will be much easier!)
That's really all you need to know about Perl and Unicode. If you tell Perl what your data is, it has the best Unicode support among all popular programming languages. If you assume it will magically know what sort of text you are feeding it, though, then you're going to trash your data irrevocably. Just because your program works today on your UTF-8 terminal doesn't mean it will work tomorrow on a UTF-16 encoded file. So make it safe now, and save yourself the headache of trashing your users' data!
The easy part of handling Unicode is encoding output and decoding input. The hard part is finding all your input and output, and determining which encoding it is. But that's why you get the big bucks :)
We're all in agreement that it is a difficult problem for many reasons,
but that's precisely the reason to try to make it easier on everybody.
There is a recent module on CPAN, utf8::all, that attempts to "turn on Unicode. All of it".
As has been pointed out, you can't magically make the entire system (outside programs, external web requests, etc.) use Unicode as well, but we can work together to make sensible tools that make doing common problems easier. That's the reason that we're programmers.
If utf8::all doesn't do something you think it should, let's improve it to make it better. Or let's make additional tools that together can suit people's varying needs as well as possible.
`
I think you misunderstand Unicode and its relationship to Perl. No matter which way you store data, Unicode, ISO-8859-1, or many other things, your program has to know how to interpret the bytes it gets as input (decoding) and how to represent the information it wants to output (encoding). Get that interpretation wrong and you garble the data. There isn't some magic default setup inside your program that's going to tell the stuff outside your program how to act.
You think it's hard, most likely, because you are used to everything being ASCII. Everything you should have been thinking about was simply ignored by the programming language and all of the things it had to interact with. If everything used nothing but UTF-8 and you had no choice, then UTF-8 would be just as easy. But not everything does use UTF-8. For instance, you don't want your input handle to think that it's getting UTF-8 octets unless it actually is, and you don't want your output handles to be UTF-8 if the thing reading from them can't handle UTF-8. Perl has no way to know those things. That's why you are the programmer.
I don't think Unicode in Perl 5 is too complicated. I think it's scary and people avoid it. There's a difference. To that end, I've put Unicode in Learning Perl, 6th Edition, and there's a lot of Unicode stuff in Effective Perl Programming. You have to spend the time to learn and understand Unicode and how it works. You're not going to be able to use it effectively otherwise.
While reading this thread, I often get the impression that people are using "UTF-8" as a synonym to "Unicode". Please make a distinction between Unicode's "Code-Points" which are an enlarged relative of the ASCII code and Unicode's various "encodings". And there are a few of them, of which UTF-8, UTF-16 and UTF-32 are the current ones and a few more are obsolete.
Please, UTF-8 (as well as all other encodings) exists and have meaning in input or in output only. Internally, since Perl 5.8.1, all strings are kept as Unicode "Code-points". True, you have to enable some features as admiringly covered previously.
There's a truly horrifying amount of ancient code out there in the wild, much of it in the form of common CPAN modules. I've found I have to be fairly careful enabling Unicode if I use external modules that might be affected by it, and am still trying to identify and fix some Unicode-related failures in several Perl scripts I use regularly (in particular, iTiVo fails badly on anything that's not 7-bit ASCII due to transcoding issues).
You should enable the unicode strings feature, and this is the default if you use v5.14;
You should not really use unicode identifiers esp. for foreign code via utf8 as they are insecure in perl5, only cperl got that right. See e.g. http://perl11.org/blog/unicode-identifiers.html
Regarding utf8 for your filehandles/streams: You need decide by yourself the encoding of your external data. A library cannot know that, and since not even libc supports utf8, proper utf8 data is rare. There's more wtf8, the windows aberration of utf8 around.
BTW: Moose is not really "Modern Perl", they just hijacked the name. Moose is perfect Larry Wall-style postmodern perl mixed with Bjarne Stroustrup-style everything goes, with an eclectic aberration of proper perl6 syntax, e.g. using strings for variable names, horrible fields syntax, and a very immature naive implementation which is 10x slower than a proper implementation.
cperl and perl6 are the true modern perls, where form follows function, and the implementation is reduced and optimized.