How to remove duplicated documents in MongoDB 4.0 - mongodb

Question

Lets say I have the next documents in the colecction sample:
{_id: 1, comp_index1: "one", comp_index2: "AAA", field: "lots of text" }
{_id: 2, comp_index1: "two", comp_index2: "BBB", field: "mucho texto" }
{_id: 3, comp_index1: "one", comp_index2: "CCC", field: "more text" }
{_id: 4, comp_index1: "two", comp_index2: "AAA", field: "más texto" }
{_id: 5, comp_index1: "one", comp_index2: "AAA", field: "lots of text" }
I want to make comp_index1 and comp_index2 an actual unique compound index.
If I run db.sample.createIndex( { comp_index1: 1, comp_index2: 1}, { unique: true } ) it will throw E11000 duplicate key error collection so I decided to remove duplicates first (due the removal of dropDups option).
Right now I have this brute force algorithm that does the job:
db.sample.aggregate([
{
$group: {
_id: {
comp_index1: "$comp_index1",
comp_index2: "$comp_index2"
},
count: { $sum: 1 }
}
},
{
$match: { count: { $gt: 1 } }
}
], { allowDiskUse: true }).forEach(function (doc) {
for (i = 1; i < doc.count; i++) {
db.sample.remove({
comp_index1: doc._id.comp_index1,
comp_index2: doc._id.comp_index2
},
{
justOne: true
});
}
print("Removed " + (i-1) + " dups of <" + doc._id.comp_index1 + " " + doc._id.comp_index2 + ">")
})
The problem is that I have over 1.4 M documents and there are almost 200 000 dups, so this takes forever to be done, so I was wondering if there is a faster better approach.

Answer 1

After several hours I finally managed to come with a 1000 times faster solution.
var ids = [];
db.sample.aggregate([
{
$group: {
_id: {
comp_index1: "$comp_index1",
comp_index2: "$comp_index2"
},
unique_ids: { $addToSet: "$_id" }
count: { $sum: 1 }
}
},
{
$match: { count: { $gt: 1 } }
}
], { allowDiskUse: true }).forEach(function (doc) {
var i = 0;
doc.unique_ids.forEach(function (id) {
if (i++ > 0) ids.push(id);
})
})
db.sample.remove({"_id": {$in: ids}});
Despite being overall the same approach, saving in RAM all the ids to remove and then performing remove with the operator $in is way way faster. This one was took only few seconds to execute.
If you come up with another solution that does not requires using RAM, please share.

Answer 2

I recently create a code to delete duplicated documents from MongoDB, this should work:
const query = [
{
$group: {
_id: {
comp_index1: "$comp_index1",
comp_index2: "$comp_index2"
},
dups: {
$addToSet: "$_id",
},
count: {
$sum: 1,
},
},
},
{
$match: {
count: {
$gt: 1,
},
},
},
];
const cursor = collection.aggregate(query).cursor({ batchSize: 10 }).exec();
cursor.eachAsync((doc, i) => {
doc.dups.shift(); // First element skipped for deleting
doc.dups.map(async (dupId) => {
await collection.findByIdAndDelete({ _id: dupId });
});
});