Django headache with simple non-ascii string - unicode

Question

I just created the following model:
class Categoria(models.Model):
nombre=models.CharField(max_length=30)
padre=models.ForeignKey('self', blank=True, null=True)
def __unicode__(self):
return self.nombre
Then registered to the admin interface and syncdb'd
Everything ok if I just add plain ASCII chars. But if I add a "Categoria" named "á" (to say something) I get:
Environment:
Request Method: GET
Request URL: http://192.168.2.103:8000/administracion/locales/categoria/
Django Version: 1.1.1
Python Version: 2.6.4
Installed Applications:
['django.contrib.auth',
'django.contrib.contenttypes',
'django.contrib.sessions',
'django.contrib.admin',
'cruzandoelsuquiaDJ.locales']
Installed Middleware:
('django.middleware.common.CommonMiddleware',
'django.contrib.sessions.middleware.SessionMiddleware',
'django.contrib.auth.middleware.AuthenticationMiddleware')
Template error:
In template /usr/lib/pymodules/python2.6/django/contrib/admin/templates/admin/change_list.html, error at line 78
Caught an exception while rendering: ('ascii', '\xc3\xa1', 0, 1, 'ordinal not in range(128)')
68 : {% endif %}
69 : {% endblock %}
70 :
71 : <form action="" method="post"{% if cl.formset.is_multipart %} enctype="multipart/form-data"{% endif %}>
72 : {% if cl.formset %}
73 : {{ cl.formset.management_form }}
74 : {% endif %}
75 :
76 : {% block result_list %}
77 : {% if action_form and actions_on_top and cl.full_result_count %}{% admin_actions %}{% endif %}
78 : {% result_list cl %}
79 : {% if action_form and actions_on_bottom and cl.full_result_count %}{% admin_actions %}{% endif %}
80 : {% endblock %}
81 : {% block pagination %}{% pagination cl %}{% endblock %}
82 : </form>
83 : </div>
84 : </div>
85 : {% endblock %}
86 :
Traceback:
File "/usr/lib/pymodules/python2.6/django/core/handlers/base.py" in get_response
92. response = callback(request, *callback_args, **callback_kwargs)
File "/usr/lib/pymodules/python2.6/django/contrib/admin/options.py" in wrapper
226. return self.admin_site.admin_view(view)(*args, **kwargs)
File "/usr/lib/pymodules/python2.6/django/views/decorators/cache.py" in _wrapped_view_func
44. response = view_func(request, *args, **kwargs)
File "/usr/lib/pymodules/python2.6/django/contrib/admin/sites.py" in inner
186. return view(request, *args, **kwargs)
File "/usr/lib/pymodules/python2.6/django/contrib/admin/options.py" in changelist_view
986. ], context, context_instance=context_instance)
File "/usr/lib/pymodules/python2.6/django/shortcuts/__init__.py" in render_to_response
20. return HttpResponse(loader.render_to_string(*args, **kwargs), **httpresponse_kwargs)
File "/usr/lib/pymodules/python2.6/django/template/loader.py" in render_to_string
108. return t.render(context_instance)
File "/usr/lib/pymodules/python2.6/django/template/__init__.py" in render
178. return self.nodelist.render(context)
File "/usr/lib/pymodules/python2.6/django/template/__init__.py" in render
779. bits.append(self.render_node(node, context))
File "/usr/lib/pymodules/python2.6/django/template/debug.py" in render_node
71. result = node.render(context)
File "/usr/lib/pymodules/python2.6/django/template/loader_tags.py" in render
97. return compiled_parent.render(context)
File "/usr/lib/pymodules/python2.6/django/template/__init__.py" in render
178. return self.nodelist.render(context)
File "/usr/lib/pymodules/python2.6/django/template/__init__.py" in render
779. bits.append(self.render_node(node, context))
File "/usr/lib/pymodules/python2.6/django/template/debug.py" in render_node
71. result = node.render(context)
File "/usr/lib/pymodules/python2.6/django/template/loader_tags.py" in render
97. return compiled_parent.render(context)
File "/usr/lib/pymodules/python2.6/django/template/__init__.py" in render
178. return self.nodelist.render(context)
File "/usr/lib/pymodules/python2.6/django/template/__init__.py" in render
779. bits.append(self.render_node(node, context))
File "/usr/lib/pymodules/python2.6/django/template/debug.py" in render_node
71. result = node.render(context)
File "/usr/lib/pymodules/python2.6/django/template/loader_tags.py" in render
24. result = self.nodelist.render(context)
File "/usr/lib/pymodules/python2.6/django/template/__init__.py" in render
779. bits.append(self.render_node(node, context))
File "/usr/lib/pymodules/python2.6/django/template/debug.py" in render_node
71. result = node.render(context)
File "/usr/lib/pymodules/python2.6/django/template/loader_tags.py" in render
24. result = self.nodelist.render(context)
File "/usr/lib/pymodules/python2.6/django/template/__init__.py" in render
779. bits.append(self.render_node(node, context))
File "/usr/lib/pymodules/python2.6/django/template/debug.py" in render_node
81. raise wrapped
Exception Type: TemplateSyntaxError at /administracion/locales/categoria/
Exception Value: Caught an exception while rendering: ('ascii', '\xc3\xa1', 0, 1, 'ordinal not in range(128)')
My django version is 1.1 and my database is 5.1.37-1ubuntu5 with utf8 charset and the table is using a utf8_bin collation.
This problem seems too basic to be true, and I'm a django newbie so I'm sorry in advance if I'm missing something very simple :)

Answer 1

Django generally has very good Unicode support (see the Django 1.1 "Unicode data" documentation for details). In my code I find that, if I'm having a problem with simple Unicode features, the problem usually is that I'm not understanding Django's details well, not that Django has a bug in its Unicode support.
The "Unicode Data" page tells us that "All of Django’s database backends ... automatically convert strings retrieved from the database into Python Unicode strings. You don’t even need to tell Django what encoding your database uses: that is handled transparently." So your simple return self.nombre should return a Python Unicode string.
However, the Django 1.1 "Databases" page has an important note about how the MySQL backend handles the utf8_bin collation:
...if you really want case-sensitive
comparisons on a particular column or
table, you would change the column or
table to use the utf8_bin collation.
The main thing to be aware of in this
case is that if you are using MySQLdb
1.2.2, the database backend in Django will then return bytestrings (instead
of unicode strings) for any character
fields it returns receive from the
database. This is a strong variation
from Django's normal practice of
always returning unicode strings. It
is up to you, the developer, to handle
the fact that you will receive
bytestrings if you configure your
table(s) to use utf8_bin collation.
Django itself should work smoothly
with such columns, but if your code
must be prepared to call
django.utils.encoding.smart_unicode()
at times if it really wants to work
with consistent data ...
So, in your original example, the column "nombre" used utf8_bin collation. This meant that self.nombre was returning a Python byte string. When you put it in an expression that required a Python Unicode string, Python performed its default conversion. This is the equivalent of self.nombre.decode('ascii'). And of course, .decode('ascii') fails when it encounters any byte above 0x7F, such as the UTF-8 bytes which encode "á".
You discovered the two ways to solve this problem. The first is to convert the Python byte string returned by self.nombre into a Python Unicode string explicitly. I'll bet the following simpler code would have worked:
return self.nombre.decode('utf8')
The second approach is to change the MySQL collation for column "nombre", which causes Django's MySQL backend to return Python Unicode strings instead of the unusual byte strings. Then your original expression gives a Python Unicode string:
return self.nombre
Hope this helps.

Answer 2

This problem can be solved by a little bit changing of django's code.
Add code below in django/utils/encoding.py
import sys
reload(sys)
sys.setdefaultencoding('utf-8')

Answer 3

I had this issue in production and never on the development server.
Then I realized that new tables were created with utf8_bin collation instead of utf8_general_ci.
To see which tables require conversion, type
SHOW TABLE STATUS;
Then convert those with utf8_bin collation by typing
ALTER TABLE app_table CONVERT TO CHARACTER SET utf8 COLLATE utf8_general_ci;
Finally, change the default collation so this doesn't happen again:
ALTER DATABASE my_database character set utf8 collate utf8_general_ci;

Answer 4

Ok...
return u"%s"%(self.nombre.decode('utf8'),)
does the trick.
But also found that changing utf8_bin to utf8_general_ci does the trick, i.e. self.nombre works as expected.

Answer 5

I solved by problem by simply creating an Admin Model for the Model and including all variables in "list_display".

Answer 6

I had similar issue when recently changed a MySQL table to use collation utf8_bin in staging while no problem in dev (python2.7, Django1.4.2 in both environments). I found out that in dev I have MySQL-python 1.2.4c1 and in staging I have 1.2.3. Upgrading to MySQL-python 1.2.4 solved the problem for me.