Handling UTF8-encoded text in SQLite using Rose::DB::Object - perl

Question

I am using Rose::DB::Object, SQLite, and Chinese text. My classes look like this:
package My::DB;
use base qw(Rose::DB);
__PACKAGE__->use_private_registry;
__PACKAGE__->register_db(
driver => 'sqlite',
database => 'data/sqmple.db',
);
package Motorcycle;
use My::DB;
use base qw(Rose::DB::Object);
...
sub init_db { My::DB->new() };
The code used to store a record:
Motorcycle->new(
type => $self->param('type'),
brand => $self->param('brand'),
color => $self->param('color'),
)->save;
The code used to display the data (from within a Mojolicious app):
<td><%= Mojo::ByteStream->new($cycle->type)->decode("utf-8") %></td>
<td><%= Mojo::ByteStream->new($cycle->brand)->decode("utf-8") %></td>
<td><%= Mojo::ByteStream->new($cycle->color)->decode("utf-8") %></td>
How can I eliminate the decoding step? I'd like the display code to look like this instead:
<td><%= $cycle->type %></td>
<td><%= $cycle->brand %></td>
<td><%= $cycle->color %></td>

Answer 1

I think you need to get the sqlite_unicode => 1 configuration value down to SQLite, there was a similar question about UTF-8 and SQLite, setting sqlite_unicode did the trick there.
I don't think Rose::DB::SQLite supports this configuration parameter though. Based on this possibly similar issue with MySQL you might be able to patch Rose::DB::SQLite to add support for sqlite_unicode by adding this to the driver:
sub sqlite_unicode {
{
shift->dbh_attribute_boolean('sqlite_unicode', #_)
}
I'll leave a comment on John's answer so he can sanity check this.
If that works then you might want to send a patch to John Siracusa (who is not only already on this question but also the Rose::DB maintainer).

Answer 2

If you feed UTF8-encoded text into SQLite, it should give it right back to you in the same form. For example, given an SQLite database named test.db containing this schema:
CREATE TABLE things
(
id INTEGER PRIMARY KEY AUTOINCREMENT,
name VARCHAR(64) NOT NULL
);
Run this code in a script in the same directory as the test.db database:
package My::DB;
use base qw(Rose::DB);
__PACKAGE__->use_private_registry;
__PACKAGE__->register_db
(
driver => 'sqlite',
database => 'test.db',
);
package My::Thing;
use base qw(Rose::DB::Object);
__PACKAGE__->meta->setup
(
table => 'things',
columns =>
[
id => { type => 'serial', primary_key => 1, not_null => 1 },
name => { type => 'text', length => 64, not_null => 1 },
],
);
sub init_db { My::DB->new }
package main;
# Set the name to a UTF8-encoded smiley: Unicode 0x263A
my $thing = My::Thing->new(name => "\x{e2}\x{98}\x{ba}")->save;
$thing = My::Thing->new(id => $thing->id)->load;
# This will print the UTF8-encoded smiley; make sure your
# terminal can handle UTF8 output.
print $thing->name, "\n";
If this is not working for you, then perhaps your calls to get the form parameters (e.g., $self->param('type')) are returning character strings instead of UTF8-encoded strings. That is, in the case of the smiley face string, perhaps $self->param('foo') is returning "\x{263a}" and not "\x{e2}\x{98}\x{ba}". In that case, the solution would be to encode the strings as UTF8 before setting the object attributes:
Motorcycle->new(
type => utf8::encode($self->param('type')),
brand => utf8::encode($self->param('brand')),
color => utf8::encode($self->param('color')),
)->save;