Template Toolkit's somevar.substr() and UTF-8 - perl

Question

We use Template Toolkit in a Catalyst app. We configured TT to use UTF-8 and had no problems with it before.
Now I call the substr() method of a string var. Unfortunately it does split the string after n bytes instead of n chars. If the n'th and (n+1)'th byte build a unicode char it is split and only the 1st byte is part of substr() result.
How to fix or workaround that behaviour?
[% string = "fööbär";
string.length; # prints 9
string.substr(0, 5); # prints "föö" (1 ascii + 2x 2 byte unicode)
string.substr(0, 4): # prints "fö?" (1 ascii, 1x 2 byte unicode, 1 unknown char)
%]
Until now we had no problems with Unicode chars, neither ones comes from the database nor text in the templates.
Edit: This is how I configure the Catalyst::View::TT module in my Catalyst app:
__PACKAGE__->config(
# DEBUG => DEBUG_ALL,
DEFAULT_ENCODING => 'utf-8',
INCLUDE_PATH => My::App->path_to( 'root', 'templates' ),
TEMPLATE_EXTENSION => '.tt',
WRAPPER => "wrapper/default.tt",
render_die => 1,
);

Answer 1

I did a quick testing with Perl 1.12.2 for MSWin32 Template module.
It could handle all these substr operation properly.
This is my test code:
use Template;
# some useful options (see below for full list)
my $config = {
# DEFAULT_ENCODING => 'utf-8',
INCLUDE_PATH => 'd:/devel/perl', # or list ref
INTERPOLATE => 1, # expand "$var" in plain text
EVAL_PERL => 1, # evaluate Perl code blocks
};
# create Template object
my $template = Template->new($config);
# define template variables for replacement
my $vars = {
var1 => "abcdef"
};
# specify input filename, or file handle, text reference, etc.
my $input = 'ttmyfile.txt';
# process input template, substituting variables
print $template->process($input, $vars);
ttmyfile.txt
Var = [% var1 %]
[% string = "fööbär" -%]
[% string.length %] # prints 6
[% string.substr(0, 5) %] # prints "fööbä"
[% string.substr(0, 4) %] # prints "fööb"
Output:
Var = abcdef
6 # prints 6
fööbä # prints "fööbä"
fööb # prints "fööb"
1
All works fine, even without use utf8 nor DEFAULT_ENCODING. Key things here:
Make sure your template .tt files are encoded as UTF8 with BOM -- Byte Order Mark. This is a must do task! Because Template-Toolkit is detect the Unicode file encoding according to BOM.
You can use Windows Notepad to save a file with BOM, just do File --> Save --> Encoding: "UTF-8".
You can use VIM make it as well by input set fenc=utf8 and set bomb, then save the file, the file will start with BOM.
Set the NCODING paramter Template->new({NCODING => 'utf-8'}); as 'utf-8' will enforce Template to load template file as 'utf-8'.
Suggest to have use utf8 in your script, it will ensure all your inline strings are encoding as utf8 properly.
Because Catalyst::View::TT rely on Template, I believe it should be working as well! Good luck~~~

Answer 2

The Wikipedia article on UTF-8 provides a table that shows how non-ASCII characters are encoded. That table illustrates the following simple rules for UTF-8:
If the highest bit of a byte is 0, then the byte denotes an ASCII character.
If the two highest bits of a byte are 11, then this is the start of a multi-byte character, and the number of consecutive 1 bits starting from the highest order bit indicates the total number of bytes in the multi-byte character. Thus, a byte whose bit representation is 110xxxxx is the start of a 2-byte character, 1110xxxx is the start of a 3-byte character, and 11110xxx is the start of a 4-byte character. (You can ignore the hypothetical 5-byte and 6-byte characters because Unicode is limited to being a 21-bit character set rather than a 32-bit character set.)
If the two highest bits of a byte are 10, then this byte is part of a multi-byte character (but not the first byte of that character).
That information should be enough for you to write your own utility functions that are like string.length and string.substring() but work in terms of characters instead of bytes.
Update: The question did not specify the programming language being used, and I was not aware that "Template Toolkit" implied the use of Perl. Once I realised that, I did a Google search and discovered that your problem is likely to be due to the need to add a use utf8 directive to your source code. You can find a discussion about this here.

Answer 3

The answer is pretty simple (in Perl), fortunately:
use Encode qw{encode decode};
The way this works is that you decode Unicode strings into Perl strings, whereupon you can use substr() and length() the way you expect, and then you encode them again for output.
With that header:
# $unicodeString = 'fööbär';
my $perlString = decode('UTF-8', $unicodeString);
printf "%d\n", length($perlString); # should be 6
printf "%s\n", substr($perlString, 0, 3); # should be 'föö'
# whatever other processing you want here with $perlString . . .
# Then, you want to reencode that back to a proper UTF-8 string:
my $unicodeString = encode('UTF-8', $perlString);
Would that help?