daily refresh of MongoDB Collection with Insert and update - mongodb

Question

I have a MongoDB collection where Data need to refreshed for certain fields every night. The Target collection has 3 extra custom fields which are used by end user input for respective documents.
So when daily refresh happens overnight, the data source can send new documents or updated data of existing documents. The documents can be upto 10,000.
I am using Pymongo and MongoDB to achieve this. My problem, how to identify the which record need to be updated and which record needs to be inserted with those 3 extra custom fields without impacting end user data.
For Example:
Data Source:
Manufacture Name Model Year Units
BMW 5Series 2019 10
BMW 5Series 2020 5
AUDI A4 2020 20
AUDI A7 2019 3
TOYOTA COROLLA 2020 5
TOYOTA CAMRY 2020 6
HONDA ACCORD 2020 10
HONDA PILOT 2019 15
HONDA CRV 2019 20
Once Loaded, the App table has 1 custom columns (Location) for user input
Manufacture Name Model Year Location Units
BMW 5Series 2019 London 10
BMW 5Series 2020 New York 5
AUDI A4 2020 Melbourne 20
AUDI A7 2019 London 3
TOYOTA COROLLA 2020 New York 5
TOYOTA CAMRY 2020 London 6
HONDA ACCORD 2020 Sydney 10
HONDA PILOT 2019 Tokyo 15
HONDA CRV 2019
On second day, we get new data as below
Manufacture Name Model Year Units
BMW 5Series 2019 10
BMW 5Series 2020 **35**
**BMW 7Series 2020 12**
AUDI A4 2020 20
AUDI A7 2019 3
**AUDI A6 2019 1**
TOYOTA COROLLA 2020 5
TOYOTA CAMRY 2020 6
HONDA ACCORD 2020 10
HONDA PILOT 2019 15
*HONDA CRV 2019 20* *-- deleted -- in second refresh*
The data can be 10,000 records. How to achieve this with Pymongo or MongodB? I wrote the code in PyMongo until the retrieve the source data and store the cursor in dictionary. Not sure how to proceed after this using MongoDB Upsert or bulk write and preserve/update Location column data for existing records and assign NULL values for new records.
Thanks

Answer 1

finally this is achieved as below:
import pymongo
from pymongo import UpdateOne
# Define MongoDB connection
client = pymongo.MongoClient(server, username='User', password='password', authSource='DBName', authMechanism='SCRAM-SHA-256')
#Source table
collection2 = db['cars2']
count123 = collection2.count_documents({})
#print("New Cars2 Data - Count before Insert:",count123)
source_cursor = collection2.find()
print("Cars2 - Count in Cursor:",source_cursor.count())
#Target Table
collection = db['cars']
tgt_count = collection.count_documents({})
print("Cars Collection - Count before Insert:",tgt_count)
sourcedata = []
#since this is a MongoDB Cursor object, we need push to List using list()
sourcedata = list(source_cursor)
source_cursor.close()
# ADD new columns to the Data before inserting in MongoDB
for item in sourcedata:
item['Location'] = None
item['last_refresh'] = datetime.now()
#sourcedata = source_cursor
ops = []
if tgt_count == 0:
print("Loading for First time:")
for rec in sourcedata:
#Load Data for first time with new fields
ops.append( UpdateOne({'name': rec['name'],'model':rec['model']}, {'$set': {'name':rec['name'],'model':rec['model'],'year':rec['year'],'units':rec['units'], 'Location': rec['Location'],'last_refresh':rec['last_refresh']}}, upsert=True))
#print(ops)
result = collection.bulk_write(ops)
print("Inserted Count:", result.inserted_count)
print("Matched Count:", result.matched_count)
print("Modified Count:", result.modified_count)
print("Upserted Count:", result.upserted_count)
elif tgt_count > 0:
print("Updating the Load:")
for rec in sourcedata:
#No Location field is included to avoid replacing the values of this field by NULL
ops.append( UpdateOne({'name': rec['name'],'model':rec['model']}, {'$set': {'name':rec['name'],'model':rec['model'],'year':rec['year'],'units':rec['units'], 'last_refresh':rec['last_refresh']}}, upsert=True))
result = collection.bulk_write(ops)
print("Inserted Count:", result.inserted_count)
print("Matched Count:", result.matched_count)
print("Modified Count:", result.modified_count)
print("Upserted Count:", result.upserted_count)
#because I didn’t include “Location” field above, the new UPSERT records DO NOT have “Location” field anymore. So I have to update the collection one more time to include “Location” field
nullfld_result = collection.update_many({'Location':None},{ '$set':{'Location':None}})
count2 = collection.count_documents({})
print("Count after Insert:",count2)