Append a string to the end of an existing field in MongoDB - mongodb

I have a document with a field containing a very long string. I need to concatenate another string to the end of the string already contained in the field.
The way I do it now is that, from Java, I fetch the document, extract the string in the field, append the string to the end and finally update the document with the new string.
The problem: The string contained in the field is very long, which means that it takes time and resources to retrieve and work with this string in Java. Furthermore, this is an operation that is done several times per second.
My question: Is there a way to concatenate a string to an existing field, without having to fetch (db.<doc>.find()) the contents of the field first? In reality all I want is (field.contents += new_string).
I already made this work using Javascript and eval, but as I found out, MongoDB locks the database when it executes javascript, which makes the overall application even slower.

Starting Mongo 4.2, db.collection.updateMany() can accept an aggregation pipeline, finally allowing the update of a field based on its current value:
// { a: "Hello" }
db.collection.updateMany(
{},
[{ $set: { a: { $concat: [ "$a", "World" ] } } }]
)
// { a: "HelloWorld" }
The first part {} is the match query, filtering which documents to update (in this case all documents).
The second part [{ $set: { a: { $concat: [ "$a", "World" ] } } }] is the update aggregation pipeline (note the squared brackets signifying the use of an aggregation pipeline). $set (alias of $addFields) is a new aggregation operator which in this case replaces the field's value (by concatenating a itself with the suffix "World"). Note how a is modified directly based on its own value ($a).

For example (it's append to the start, the same story ):
before
{ "_id" : ObjectId("56993251e843bb7e0447829d"), "name" : "London
City", "city" : "London" }
db.airports
.find( { $text: { $search: "City" } })
.forEach(
function(e, i){
e.name='Big ' + e.name;
db.airports.save(e);
}
)
after:
{ "_id" : ObjectId("56993251e843bb7e0447829d"), "name" : "Big London
City", "city" : "London" }

Old topic but i had the same problem.
Since mongo 2.4, you can use $concat from aggregation framework.
Example
Consider these documents :
{
"_id" : ObjectId("5941003d5e785b5c0b2ac78d"),
"title" : "cov"
}
{
"_id" : ObjectId("594109b45e785b5c0b2ac97d"),
"title" : "fefe"
}
Append fefe to title field :
db.getCollection('test_append_string').aggregate(
[
{ $project: { title: { $concat: [ "$title", "fefe"] } } }
]
)
The result of aggregation will be :
{
"_id" : ObjectId("5941003d5e785b5c0b2ac78d"),
"title" : "covfefe"
}
{
"_id" : ObjectId("594109b45e785b5c0b2ac97d"),
"title" : "fefefefe"
}
You can then save the results with a bulk, see this answer for that.

this is a sample of one document i have :
{
"_id" : 1,
"s" : 1,
"ser" : 2,
"p" : "9919871172",
"d" : ISODate("2018-05-30T05:00:38.057Z"),
"per" : "10"
}
to append a string to any feild you can run a forEach loop throught all documents and then update desired field:
db.getCollection('jafar').find({}).forEach(function(el){
db.getCollection('jafar').update(
{p:el.p},
{$set:{p:'98'+el.p}})
})

This would not be possible.
One optimization you can do is create batches of updates.
i.e. fetch 10K documents, append relevant strings to each of their keys,
and then save them as single batch.
Most mongodb drivers support batch operations.

db.getCollection('<collection>').update(
// query
{},
// update
{
$set: {<field>:this.<field>+"<new string>"}
},
// options
{
"multi" : true, // update only one document
"upsert" : false // insert a new document, if no existing document match the query
});

Related

MongoDB - Updating the field type of all objects within an array from String to Int32/64

I'm fairly new to mongodb so please bear with me.
As per title, what I want to achieve is to convert a specific field in all documents within an array of a document from String to Int how do i do that?
Sample Doc :
{
reviews:[
{
snid:"1242"
},
{
snid:"8392"
}
]
}
And my objective is to convert all of the snid's from String to Int32
so far i understand that we can use something like db.collection.update() but this will update a specific field, not an array.
Another attempt is
db.collection.find({},{reviews:1,_id:0},(err,doc)=>{
//How do i push it back to the document
})
But as you can tell, I'm not entirely sure on how we should push the updated document back into the same array of sorts.
Any insights will be greatly appreciated!
1) If you're using MongoDB version >= 4.2, try below query :
db.collection.update({'reviews.snid' : {$exists : true}}, [
{
$set: {
reviews: {
$map: { input: "$reviews", in: { 'snid': { $toInt: "$$this.snid" } } }
}
}
}
],{multi :true})
Above query uses Aggregation-pipeline in .update() which was introduced in version 4.2, You can also use .updateMany() instead of .update().
It works on documents of below type :
/* 1 */
{
"_id" : ObjectId("5e810f5ec16b5679b43a2f0e"),
"reviews" : [
{
"snid" : '1242'
},
{
"snid" : '8392'
}
]
}
/* 2 */
{
"_id" : ObjectId("5e810f6ac16b5679b43a310c"),
"reviews" : [
{
"snid" : '1242232'
},
{
"snid" : '8391232'
}
]
}
/* 3 */
{
"_id" : ObjectId("5e8110b1c16b5679b43a5148"),
"abc" : 1
}
/* 4 */
{
"_id" : ObjectId("5e8110c3c16b5679b43a52f9"),
"reviews" : []
}
/* 5 */
{
"_id" : ObjectId("5e811359c16b5679b43a9229"),
"reviews" : [
{
"abc" : "1"
}
]
}
But above update query will partially work if you've a doc like below :
{
"_id" : ObjectId("5e811359c16b5679b43a9230"),
"reviews" : [
{
"abc" : "1"
},
{
"snid" : "123"
}
]
}
In that case you need to use $cond to do a conditional check in $map to see if current object has key snid then convert value or else pass on the same object as is to 'reviews' array.
2) Just in Case if your MongoDB version is < 4.2/4.0 & > 3.2 - You can use .bulkWrite() :
Cause you can not use Aggregation Pipeline in update & also $toInt. So you need to do .find() to get entire docs & write code to convert these from strings to integers & use .bulkWrite() to update docs in one update DB call (You can take _id as key for each document).
3) You can also write an aggregation query on existing collection & use $out to update entire collection or write aggregation result to new collection by running just one query. I would prefer to temporarily write it to new collection to check data is correct & rename new collection to what ever is existing by naming existing with something ends with _backup used as backup.

$Avg aggregation in Mongodb [duplicate]

For a given record id, how do I get the average of a sub document field if I have the following in MongoDB:
/* 0 */
{
"item" : "1",
"samples" : [
{
"key" : "test-key",
"value" : "1"
},
{
"key" : "test-key2",
"value" : "2"
}
]
}
/* 1 */
{
"item" : "1",
"samples" : [
{
"key" : "test-key",
"value" : "3"
},
{
"key" : "test-key2",
"value" : "4"
}
]
}
I want to get the average of the values where key = "test-key" for a given item id (in this case 1). So the average should be $avg (1 + 3) = 2
Thanks
You'll need to use the aggregation framework. The aggregation will end up looking something like this:
db.stack.aggregate([
{ $match: { "samples.key" : "test-key" } },
{ $unwind : "$samples" },
{ $match : { "samples.key" : "test-key" } },
{ $project : { "new_key" : "$samples.key", "new_value" : "$samples.value" } },
{ $group : { `_id` : "$new_key", answer : { $avg : "$new_value" } } }
])
The best way to think of the aggregation framework is like an assembly line. The query itself is an array of JSON documents, where each sub-document represents a different step in the assembly.
Step 1: $match
The first step is a basic filter, like a WHERE clause in SQL. We place this step first to filter out all documents that do not contain an array element containing test-key. Placing this at the beginning of the pipeline allows the aggregation to use indexes.
Step 2: $unwind
The second step, $unwind, is used for separating each of the elements in the "samples" array so we can perform operations across all of them. If you run the query with just that step, you'll see what I mean.
Long story short :
{ name : "bob",
children : [ {"name" : mary}, { "name" : "sue" } ]
}
becomes two documents :
{ name : "bob", children : [ { "name" : mary } ] }
{ name : "bob", children : [ { "name" : sue } ] }
Step 3: $match
The third step, $match, is an exact duplicate of the first $match stage, but has a different purpose. Since it follows $unwind, this stage filters out previous array elements, now documents, that don't match the filter criteria. In this case, we keep only documents where samples.key = "test-key"
Step 4: $project (Optional)
The fourth step, $project, restructures the document. In this case, I pulled the items out of the array so I could reference them directly. Using the example above..
{ name : "bob", children : [ { "name" : mary } ] }
becomes
{ new_name : "bob", new_child_name : mary }
Note that this step is entirely optional; later stages could be completed even without this $project after a few minor changes. In most cases $project is entirely cosmetic; aggregations have numerous optimizations under the hood such that manually including or excluding fields in a $project should not be necessary.
Step 5: $group
Finally, $group is where the magic happens. The _id value what you will be "grouping by" in the SQL world. The second field is saying to average over the value that I defined in the $project step. You can easily substitute $sum to perform a sum, but a count operation is typically done the following way: my_count : { $sum : 1 }.
The most important thing to note here is that the majority of the work being done is to format the data to a point where performing the operation is simple.
Final Note
Lastly, I wanted to note that this would not work on the example data provided since samples.value is defined as text, which can't be used in arithmetic operations. If you're interested, changing the type of a field is described here: MongoDB How to change the type of a field

db.collection.find returns multiple records after addToSet

I use addToSet command (to add UpdatedData):
MyTable.update({ "UrlLink": "http://www.someurl.com"}, {
$addToSet: {
UpdatedData: {
TheDate: ThisDate,
NewInfo: ThisInfo
}
}
},
function (err, result) {
next(err,result)
}
);
But then, when I do the query (after UpdatedData is added to my document), I see that it returns two documents. Instead of only one updated document:
db.MyTable.find( {"UrlLink": "http://www.someurl.com"})
{ "_id" : ObjectId("56485922143a886f66c2f8bb"), "UrlLink" : "http://www.someurl.com", "Stuff" : "One", "UpdatedData" : [ { "TheDate" : "11/15/2015", "NewInfo" : "Info1", "_id" : ObjectId("5648599a71efc79c660f76d3") } ] }
{ "_id" : ObjectId("5648599f71efc79c660f76d4"), "UrlLink" : "http://www.someurl.com", "Stuff": "One", "UpdatedData" : [ ] }
So it seems that addToSet creates a new document with new _id, instead of updating the old record (ObjectId("5648599f71efc79c660f76d4")). But I only see the updated document in robomongo (ObjectId("56485922143a886f66c2f8bb")). Any ideas why this happens and how I could prevent that behaviour?
update cannot create a new document, it can only update existing.
This looks like you have created two documents with the same url.. Then when you update it just updates the first one..
To prevent the creation of a document with an already existent url you can create an index and set it to unique
db.collection.createIndex({ UrlLink: 1 }, { unique: true })
This will prevent creation of new documents with the same url, and it will also make queries by UrlLink as fast as possible.

Average a Sub Document Field Across Documents in Mongo

For a given record id, how do I get the average of a sub document field if I have the following in MongoDB:
/* 0 */
{
"item" : "1",
"samples" : [
{
"key" : "test-key",
"value" : "1"
},
{
"key" : "test-key2",
"value" : "2"
}
]
}
/* 1 */
{
"item" : "1",
"samples" : [
{
"key" : "test-key",
"value" : "3"
},
{
"key" : "test-key2",
"value" : "4"
}
]
}
I want to get the average of the values where key = "test-key" for a given item id (in this case 1). So the average should be $avg (1 + 3) = 2
Thanks
You'll need to use the aggregation framework. The aggregation will end up looking something like this:
db.stack.aggregate([
{ $match: { "samples.key" : "test-key" } },
{ $unwind : "$samples" },
{ $match : { "samples.key" : "test-key" } },
{ $project : { "new_key" : "$samples.key", "new_value" : "$samples.value" } },
{ $group : { `_id` : "$new_key", answer : { $avg : "$new_value" } } }
])
The best way to think of the aggregation framework is like an assembly line. The query itself is an array of JSON documents, where each sub-document represents a different step in the assembly.
Step 1: $match
The first step is a basic filter, like a WHERE clause in SQL. We place this step first to filter out all documents that do not contain an array element containing test-key. Placing this at the beginning of the pipeline allows the aggregation to use indexes.
Step 2: $unwind
The second step, $unwind, is used for separating each of the elements in the "samples" array so we can perform operations across all of them. If you run the query with just that step, you'll see what I mean.
Long story short :
{ name : "bob",
children : [ {"name" : mary}, { "name" : "sue" } ]
}
becomes two documents :
{ name : "bob", children : [ { "name" : mary } ] }
{ name : "bob", children : [ { "name" : sue } ] }
Step 3: $match
The third step, $match, is an exact duplicate of the first $match stage, but has a different purpose. Since it follows $unwind, this stage filters out previous array elements, now documents, that don't match the filter criteria. In this case, we keep only documents where samples.key = "test-key"
Step 4: $project (Optional)
The fourth step, $project, restructures the document. In this case, I pulled the items out of the array so I could reference them directly. Using the example above..
{ name : "bob", children : [ { "name" : mary } ] }
becomes
{ new_name : "bob", new_child_name : mary }
Note that this step is entirely optional; later stages could be completed even without this $project after a few minor changes. In most cases $project is entirely cosmetic; aggregations have numerous optimizations under the hood such that manually including or excluding fields in a $project should not be necessary.
Step 5: $group
Finally, $group is where the magic happens. The _id value what you will be "grouping by" in the SQL world. The second field is saying to average over the value that I defined in the $project step. You can easily substitute $sum to perform a sum, but a count operation is typically done the following way: my_count : { $sum : 1 }.
The most important thing to note here is that the majority of the work being done is to format the data to a point where performing the operation is simple.
Final Note
Lastly, I wanted to note that this would not work on the example data provided since samples.value is defined as text, which can't be used in arithmetic operations. If you're interested, changing the type of a field is described here: MongoDB How to change the type of a field

mongodb upsert in updating an array element

Want to upsert in object properties in a array of a document
Consider a document in collection m
{ "_id" : ObjectId("524bfc39e6bed5cc5a9f3a33"),
"x" : [
{ "id":0.0, "name":"aaa"},{ "id":1.0, "name":"bbb"}
]
}
Want to add age:100 to { "id":0.0, "name":"aaa"} .
Not just age .. But but provision for upsert in the array element {}. So it can contain {age:100,"city":"amd"} (since i am getting this from the application service)
Was trying this... But did not worked as it replaced the entire array element
db.m.update({_id:ObjectId("524bfc39e6bed5cc5a9f3a33"),
"x" : {
"$elemMatch" : {
"id" : 0.0
}
}},
{
$set : {
"x.$" : {
"age": 100
}
}
},
{upsert: true}
)
Changed the document to (which i did not wanted)
{ "_id" : ObjectId("524bfc39e6bed5cc5a9f3a33"),
"x" : [
{ "age":100},{ "id":1.0, "name":"bbb"}
]
}
Is this possible without changing schema.
$set : {"x.$" : {"age": 100}}
x.$ sets the entire matched array element to {age: 100}
This should work:
db.m.update({_id:ObjectId("524bfc39e6bed5cc5a9f3a33"),
"x.id": 0.0}, {$set: {"x.$.age": 100 }});
Using elemMatch:
db.test.update({x: {$elemMatch: {id: 1}}},{$set: {"x.$.age": 44}})
Note that the upsert option here, is redundant and wouldn't work if the id isn't present in x because the positional operator $ doesn't support upserting.
This is not possible without changing schema. If you can change schema to use an object to store your items (rather than an array), you can follow the approach I outlined in this answer.