Mongoose/Mongodb, update each doc query, very slow - mongodb

Question

I have this update query in mongoose. It's 1600 posts and takes like 5 min to run.
What's the bottleneck? Am I using the wrong approach?
export const getAndStoreLatestKPI = async () => {
console.log("start kpi");
try {
const marketCaps = await getKPI();
const stocks = await mongoose.model("stock").find().exec();
for (const stock of stocks) {
const marketCap = marketCaps.find(
(marketCap) => marketCap.i === stock.insId
);
if (marketCap != null) {
const marketCapAdjustedVal =
stock.country === "Finland" ? marketCap.n * 10 : marketCap.n;
const update = {
marketCap: marketCapAdjustedVal,
};
console.log(marketCapAdjustedVal);
await mongoose
.model("stock")
.findOneAndUpdate({ insId: stock.insId }, { update });
}
}
console.log("done");
return Promise.resolve();
} catch (err) {
return Promise.reject(err);
}
};
export const getKPI = async (kpiId: number) => {
try {
const kpiFetch = await Axios.get(someurl);
return Promise.resolve(kpiFetch.data.values);
} catch (err) {
return Promise.reject(err);
}
};

Answer 1

So the main bottle neck is your for loop. for each stock item you perform several "expensive" actions such as data fetching from external API + a single update, and you're doing them 1 by 1.
What I would recommend you doing is looping on several items at once. similar to the idea multithreading.
There are several different solutions on how to do it in nodejs for example nodejs worker threads
However I personally use and recommend using bluebird which gives you this ability and many others straight out of the box.
Some sample code:
import Bluebird = require('bluebird');
const stocks = await mongoose.model("stock").find().exec();
await Bluebird.map(stocks, async (stock) => {
const marketCap = marketCaps.find(
(marketCap) => marketCap.i === stock.insId
);
if (marketCap != null) {
const marketCapAdjustedVal =
stock.country === "Finland" ? marketCap.n * 10 : marketCap.n;
const update = {
marketCap: marketCapAdjustedVal,
};
console.log(marketCapAdjustedVal);
await mongoose
.model("stock")
.findOneAndUpdate({ insId: stock.insId }, { update });
}
}, {concurrency: 25})
// concurrency details how many concurrent process run parallel. the heavier they are the less you want concurrent for obvious reasons.