Insert field only if not null - mongodb

Question

I have a Mongodb document which I want to update with some information that I receive from a form. The original document in Mongodb has the following data:
{"id":1, "name": "James", "surname1": "Adams"} [id is univoque]
The information obtained from the form is the following (it's structure is always the same):
id = 1
name = "James"
surname1 = "Adams"
surname2 = ""
synced = 1
In order to update this document from the collection foo, I am doing the following operation:
mongoClient = MongoClient('localhost:27017').db
mongoClient["foo"].update({"id" : id}, {"$set" : {"name" : name, "surname1" : surname1, "surname2" : surname2, "synced" : synced}})
This will create an empty field for surname2. And here goes my question: which is the most pythonic way to insert in Mongodb only those fields which are not empty? Could this be done with inner Mongodb logic or this could only be performed with some Python checks of the input data?

Answer 1

The trick is to pop() the items from the record before loading. Try this as an example:
from pymongo import MongoClient
mongoClient = MongoClient('localhost:27017').db
# Setup some test data
for i in range(3):
mongoClient["foo"].insert({"id": i, "name": "James", "surname1": "Adams"})
# Simulate your incoming record
record = {"id": 1, "name": "James", "surname1": "Adams", "surname2": "", "synced": 1}
# Remove any empty items
for k, v in list(record.items()):
if v == '' or v is None:
record.pop(k)
mongoClient["foo"].update({"id": record['id']}, {"$set": record})
for item in mongoClient["foo"].find({}, {"_id": 0}):
print(item)
Result:
{'id': 0, 'name': 'James', 'surname1': 'Adams'}
{'id': 1, 'name': 'James', 'surname1': 'Adams', 'synced': 1}
{'id': 2, 'name': 'James', 'surname1': 'Adams'}

Answer 2

Wholesale document updates (where the entire database document is replaced) is expensive. If you have a way to detect the changes to a document within your application you can provide the database with targeted updates. This question was originally how to update documents in the database such that null or empty fields are removed to reduce the overall size of the document. Presumably this is to increase performance and reduce disk space consumption. Both admirable objectives. But here's the thing - not only do we want well structured documents on disk, we also must consider how we manage this data. Performing wholesale updates and replacing the entire document certainly makes development of the client easier, but it places burden on the database engine. MongoDB supports documents up to 16MB so the larger the document the more the impact of a wholesale document replacement becomes. Obviously targeted updates are preferable, but how do we do that? How do we know what fields are different between my incoming document and what is in the database? This is more of a philosophical question really. We could query the database and compare with the incoming document but then again that is an impact on the database. But consider this - for any document we want to update we must have originally sourced it from the database - right? Otherwise its a new document (an insert, not an update). If we did source it from the database then we have the ability to track what changes we made to the document. If that is true then our incoming data could be nothing more than a set of changes, rather than the whole document itself. Again this is philosophy, not science. So consider the scenario where the incoming data is changes, not the actual document. We could construct instructions to unset fields where they have been removed, and to update targeted fields where they are modified but remain. The orignal question was about how to create a 'pythonic' solution to strip blank fields but this question is really at the surface of a more in-depth problem of tracking change.