How to efficiently recalculate the values of large amounts of data? - mongodb

Question

I have several collections in MongoDB:
payment
{
"_id":{
"$oid":"6060ded06aa032495d640536"
},
"type":"",
"amount":10,
"createdAt":{
"date":{
"$date":"2021-03-03T16:01:14.137Z"
},
"timestamp":1614787274.137138
},
"finishedAt":{
"date":{
"$date":"2021-03-03T16:13:15.678Z"
},
"timestamp":1614787995.678263
},
"status":true,
"state":"finished",
"destination":{
"identificator":"1234"
}
}
account
{
"_id":{
"$oid":"60677a2c88b356160e415a1e"
},
"name":"",
"providerAccount":{
"identificator":"1234"
},
"targetAmount":0,
"currentAmount":0,
"status":false,
"state":false,
"priority":false,
"createdAt":{
"date":{
"$date":"2021-03-29T00:00:00.000Z"
},
"timestamp":1616976000
}
}
I need to check each payment if it matches the account in the system. If destination.identificator == providerAccount.identificator I need to change the payment type to "internal" and add the payment amount to the currentAmount in the account.
At the moment, I have a python script that does all this by iterating over each payment, but the problem is that there are more than a million such payments, and such a process can take a very long time.
Is there a more efficient way to do this?

Answer 1

You can write two different Aggregation queries which will perform their own lookup operations on the alternating collections and update the values based on logics and conditions.
Note: The execution order is very important for this to work
Note: For both the Aggregation queries, I will be making use of the $merge stage which will work only on MongoDB version >= 4.4
If you are using any earlier versions of MongoDB, loop through the records of the Aggregation results and update the documents manually using PyMongo instead of the $merge pipeline stage which will be the last stage of the Pipelines.
The first query has to be performed on payment collection, which will check if the link exists in the account collection or not.
db.payment.aggregate([
{
"$lookup": {
"from": "account",
"let": {
"invIdentifactor": "$destination.identificator"
},
"pipeline": [
{
"$match": {
"$expr": {
"$eq": [
"$providerAccount.identificator",
"$$invIdentifactor"
],
},
},
},
{
"$project": {
"_id": 1,
},
},
],
"as": "matchedAcc"
}
},
{
"$match": {
"matchedAcc": {
"$ne": []
}
},
},
{
"$project": {
"type": {
"$literal": "internal"
}
},
},
{
"$merge": {
"into": "payment",
"on": "_id",
"whenMatched": "merge",
"whenNotMatched": "discard"
},
},
])
MongoDB Playground sample Execution
Next the Aggregation query for account collection based on "type": "internal" condition added by the previous query.
Note: If there are already documents with "type": "internal" value in payment collection, change type to a different unique key name in the $project stage and update it in the below query and finally unset the key after all the process is done.
db.account.aggregate([
{
"$lookup": {
"from": "payment",
"let": {
"accIdentifactor": "$providerAccount.identificator"
},
"pipeline": [
{
"$match": {
"$expr": {
"$eq": [
"$destination.identificator",
"$$accIdentifactor"
],
},
"type": "internal",
},
},
{
"$group": {
"_id": "$providerAccount.identificator",
"totalAmount": {
"$sum": "$amount"
}
},
},
],
"as": "matchedPayment"
},
},
{
"$match": {
"$expr": {
"$gt": [
{
"$arrayElemAt": [
"$matchedPayment.totalAmount",
0
]
},
0
]
},
},
},
{
"$project": {
"currentAmount": {
"$add": [
"$currentAmount",
{
"$arrayElemAt": [
"$matchedPayment.totalAmount",
0
]
}
]
}
},
},
{
"$merge": {
"into": "account",
"on": "_id",
"whenMatched": "merge",
"whenNotMatched": "discard"
},
},
])
Mongo Playground Sample Execution
Additionally, you can pass the allowDiskUse: true option on Aggregation commands and also consider perform indexing on providerAccount.identificator and destination.identificator keys to speed this up if required and later delete those indexes.
Let me know if you want an explanation of all the stages and operators in the aggregation pipeline.