EmbeddedDocumentSerializer runs query for every ReferenceField - mongodb

Question

I have following models and serializer the target is when serializer runs to have only one query:
Models:
class Assignee(EmbeddedDocument):
id = ObjectIdField(primary_key=True)
assignee_email = EmailField(required=True)
assignee_first_name = StringField(required=True)
assignee_last_name = StringField()
assignee_time = DateTimeField(required=True, default=datetime.datetime.utcnow)
user = ReferenceField('MongoUser', required=True)
user_id = ObjectIdField(required=True)
class MongoUser(Document):
email = EmailField(required=True, unique=True)
password = StringField(required=True)
first_name = StringField(required=True)
last_name = StringField()
assignees= EmbeddedDocumentListField(Assignee)
Serializers:
class MongoUserSerializer(DocumentSerializer):
assignees = AssigneeSerializer(many=True)
class Meta:
model = MongoUser
fields = ('id', 'email', 'first_name', 'last_name', 'assignees')
depth = 2
class AssigneeSerializer(EmbeddedDocumentSerializer):
class Meta:
model = Assignee
fields = ('assignee_first_name', 'assignee_last_name', 'user')
depth = 0
When checking the mongo profiler I have 2 queries for the MongoUser Document. If I remove the assignees field from the MongoUserSerializer then there is only one query.
As a workaround I've tried to use user_id field to store only ObjectId and changed AssigneeSerializer to:
class AssigneeSerializer(EmbeddedDocumentSerializer):
class Meta:
model = Assignee
fields = ('assignee_first_name', 'assignee_last_name', 'user_id')
depth = 0
But again there are 2 queries. I think that the serializer EmbeddedDocumentSerializer fetches all the fields and queries for ReferenceField and
fields = ('assignee_first_name', 'assignee_last_name', 'user_id')
works after the queries are made.
How to use ReferenceField and not run a separate query for each reference when serializing?

Answer 1

I ended up with a workaround and not using ReferenceField. Instead I am using ObjectIdField:
#user = ReferenceField("MongoUser", required=True) # Removed now
user = ObjectIdField(required=True)
And changed value assignment as follows:
- if assignee.user == MongoUser:
+ if assignee.user == MongoUser.id:
It is not the best way - we are not using ReferenceField functionality but it is better than creating 30 queries in the serializer.
Best Regards,
Kristian

Answer 2

It's a very interesting question and I think it is related to Mongoengine's DeReference policy: https://github.com/MongoEngine/mongoengine/blob/master/mongoengine/dereference.py.
Namely, your mongoengine Documents have a method MongoUser.objects.select_related() with max_depth argument that should be large enough that Mongoengine traversed 3 levels of depth: MongoUser->assignees->Assignee->user and cached all the related MongoUser objects for current MongoUser instance. Probably, we should call this method somewhere in our DocumentSerializers in DRF-Mongoengine to prefetch the relations, but currently we don't.
See this post about classical DRF + Django ORM that explains, how to fight N+1 requests problem by doing prefetching in classical DRF. Basically, you need to override the get_queryset() method of your ModelViewSet to use select_related() method:
from rest_framework_mongoengine.viewsets import ModelViewSet
class MongoUserViewSet(ModelViewSet):
def get_queryset(self):
queryset = MongoUser.objects.all()
# Set up eager loading to avoid N+1 selects
queryset.select_related(max_depth=3)
return queryset
Unfortunately, I don't think that current implementation of ReferenceField in DRF-Mongoengine is smart enough to handle these querysets appropriately. May be ComboReferenceField will work?
Still, I've never used this feature yet and didn't have enough time to play with these settings myself, so I'd be grateful to you, if you shared your findings.