convert mongodb query (unicode) to json using json_util - mongodb

Question

from bson.json_util import dumps
def json_response(response):
return {"response":dumps(response,ensure_ascii=False).encode("utf8")
,"headers":{"Content-type":"text/json"}}
This problem is making me crazy. It returns an error randomly, and I can't find the solution.
/core/handlers/wsgi.py", line 38, in __call__,
output = lookup_view(req),
File "auth/decorator.py", line 8, in wrap,
return fn(req,*args,**kwargs),
File "auth/decorator.py", line 21, in wrap,
return fn(req,*args,**kwargs),
File "contrib/admin/views.py", line 67, in submit_base_premission,
return json_response({"baseperm":baseperm,"Meta":{"gmsg":u"...","type":201}}),
File "render/render_response.py", line 85, in json_response,
return {"response":dumps(response,ensure_ascii=False).encode("utf8"),
File "/usr/local/lib/python2.7/dist-packages/bson/json_util.py", line 116, in dumps,
return json.dumps(_json_convert(obj), *args, **kwargs),
File "/usr/lib/python2.7/json/__init__.py", line 238, in dumps, referer:
**kw).encode(obj),
File "/usr/lib/python2.7/json/encoder.py", line 201, in encode,
chunks = self.iterencode(o, _one_shot=True),
File "/usr/lib/python2.7/json/encoder.py", line 264, in iterencode,
return _iterencode(o, 0),
File "/usr/lib/python2.7/json/encoder.py", line 178, in default,
raise TypeError(repr(o) + " is not JSON serializable"),
TypeError: ObjectId('51f7dcee95113b7a48e974fe') is not JSON serializable,
baseperm is a pymongo Cursor, it returns this error randomly and that is where I have the problem.
It seems that it doesn't detect objectid sometimes and doesn't convert it to str so json raises an error on dumps.

Answer 1

Check the version of the pymongo driver, if it is under version 2.4.2+ then you may need to update it. Before that version the __str__ method of ObjectId was handled incorrectly for 2.x versions of python, check the repo: github, ObjectId.__str__ should return str in 2.x.
To check the pymongo driver version, type in the python shell:
import pymongo
print(pymongo.version)
UPDATE
I suppose you have tested both environments with the same dataset, so give a try to upgrade python 2.7.3 to 2.7.5.
Else try to iterate through the cursor and construct the list before giving it to json_response() i.e.:
baseperm = list(baseperm) #now baseperm is a list of the documents
...
my_response['baseperm'] = baseperm
my_response['Meta'] = ...
...
return json_response(my_response)

Answer 2

I report this problem on mongodb issue tracker
https://jira.mongodb.org/browse/PYTHON-548
answer:
You said this only happens occasionally? The only thing I can think of that might be related is mod_wsgi spawning sub interpreters. In PyMongo that tends to cause problems with the C extensions encoding python dicts to BSON. In your case this seems to be happening after the BSON documents are decoded to python dicts. It looks like isinstance is failing to match ObjectId in json_util.default(). PYTHON-539 seemed to be a similar problem related to some package miss configuration in the user's environment.
There could be a fairly large performance hit, but could you try running PyMongo without C extensions to see if that solves the problem?
You can read about the mod_wsgi issue here:
http://api.mongodb.org/python/current/faq.html#does-pymongo-work-with-mod-wsgi