Remove Duplicate character from string in Mongodb - mongodb

Question

I want to remove duplicate characters from strings in MongoDB.
Example:
Input string: xxxyzzxcdv
Output string: xyzcdv

Answer 1

Query
reduce on range(count string)
keep 2 values {"previous": [], "string": ""} (initial value of reduce)
get the cur-char {"$substrCP": ["$mystring", "$$this", 1]}
this is the current index on the string, and i take the next char
if it is in previous kep "string" as it is, else concat to add the new character
heelo
reduce on (0 1 2 3 4) `{"$range": [0, {"$strLenCP": "$mystring"}]}`
we start from `{"previous": [], "string": ""}`
- get 1 character start from index 0
`{"$substrCP": ["$mystring", "$$this", 1]}}` = "h"
- if this character is on previous don't add it
`{"$in": ["$$cur_char", "$$value.previous"]}`
- else add it on previous and on the string the 2 concats in code
Repeat for `index($$this)`= 1
- get 1 character start from index 1
`{"$substrCP": ["$mystring", "$$this", 1]}}` = "e"
.....
PlayMongo
aggregate(
[{"$set":
{"mystring":
{"$getField":
{"field": "string",
"input":
{"$reduce":
{"input": {"$range": [0, {"$strLenCP": "$mystring"}]},
"initialValue": {"previous": [], "string": ""},
"in":
{"$let":
{"vars": {"cur_char": {"$substrCP": ["$mystring", "$$this", 1]}},
"in":
{"$cond":
[{"$in": ["$$cur_char", "$$value.previous"]},
"$$value",
{"previous":
{"$concatArrays": ["$$value.previous", ["$$cur_char"]]},
"string":
{"$concat": ["$$value.string", "$$cur_char"]}}]}}}}}}}}}])
Edit
The second query removed only the duplicates we choose.
Query
removes only the characters in the array, here only ["x"]
i removed the $getField because its only for MongoDB 5 +
aggregate(
[{"$set":
{"mystring":
{"$reduce":
{"input": {"$range": [0, {"$strLenCP": "$mystring"}]},
"initialValue": {"previous": [], "string": ""},
"in":
{"$let":
{"vars": {"cur_char": {"$substrCP": ["$mystring", "$$this", 1]}},
"in":
{"$cond":
[{"$and":
[{"$in": ["$$cur_char", "$$value.previous"]},
{"$in": ["$$cur_char", ["x"]]}]},
"$$value",
{"previous":
{"$concatArrays": ["$$value.previous", ["$$cur_char"]]},
"string":
{"$concat": ["$$value.string", "$$cur_char"]}}]}}}}}}},
{"$set": {"mystring": "$mystring.string"}}])
Edit
If you need to use this aggregation for update, you can use it as pipeline update like.
update({},
[{"$set": ......])
See your driver to find how to do update with pipeline, in Java its like above, alternative run it as database command

Answer 2

Let's assume we have the following collection and the records inside of it:
db.messages.insertMany([
{
"message": "heelllo theeere"
},
{
"message": "may thhhee forrrce be wiithh yyouuu"
},
{
"message": "execute orrrrdder 66"
}
])
Due to uncertainty, I am dropping solutions for both manipulating while querying and updating the records (permanently).
If you want to remove them while running your aggregation query:
In addition to #Takis's solution, using $function pipeline operator can be another option if your MongoDB version is 4.4 or higher.
Further readings on $function operator
// Query via Aggregation Framework
db.messages.aggregate([
{
$match: {
message: {
$ne: null
}
}
},
{
$addFields: {
distinctChars: {
$function: {
body: function (message) {
return message
.split("")
.filter(function (item, pos, self) {
return self.indexOf(item) == pos;
})
.join("");
},
args: ["$message"],
lang: "js"
}
}
}
},
])
If you want to remove them via an update operation:
// Update each document using cursor
db.messages.find({ message: { $ne: null } })
.forEach(doc => {
var distinctChars = doc.message
.split("")
.filter(function (item, pos, self) {
return self.indexOf(item) == pos;
})
.join("");
db.messages.updateOne({ _id: doc._id }, [{ $set: { distinctChars: distinctChars } }]);
});
A quick reminder: Above script just shows an easy way to update the records to reach the goal without focusing on other details. It can be an expensive operation depending on your real world collection's size and configurations, sharding for instance. Consider to improve it with your own way.
Result
For both way, the result should be like the following:
[
{
"_id": {
"$oid": "618d95ccdedc26d80875b75a"
},
"message": "heelllo theeere",
"distinctChars": "helo tr"
},
{
"_id": {
"$oid": "618d95ccdedc26d80875b75b"
},
"message": "may thhhee forrrce be wiithh yyouuu",
"distinctChars": "may theforcbwiu"
},
{
"_id": {
"$oid": "618d95ccdedc26d80875b75c"
},
"message": "execute orrrrdder 66",
"distinctChars": "excut ord6"
}
]