Include all existing fields and add new fields to document - mongodb

I would like to define a $project aggregation stage where I can instruct it to add a new field and include all existing fields, without having to list all the existing fields.
My document looks like this, with many fields:
{
obj: {
obj_field1: "hi",
obj_field2: "hi2"
},
field1: "a",
field2: "b",
...
field26: "z"
}
I want to make an aggregation operation like this:
[
{
$project: {
custom_field: "$obj.obj_field1",
//the next part is that I don't want to do
field1: 1,
field2: 1,
...
field26: 1
}
},
... //group, match, and whatever...
]
Is there something like an "include all fields" keyword that I can use in this case, or some other way to avoid having to list every field separately?

In 4.2+, you can use the $set aggregation pipeline operator which is nothing other than an alias to $addFieldsadded in 3.4
The $addFields stage is equivalent to a $project stage that explicitly specifies all existing fields in the input documents and adds the new fields.
db.collection.aggregate([
{ "$addFields": { "custom_field": "$obj.obj_field1" } }
])

You can use $$ROOT to references the root document. Keep all fields of this document in a field and try to get it after that (depending on your client system: Java, C++, ...)
[
{
$project: {
custom_field: "$obj.obj_field1",
document: "$$ROOT"
}
},
... //group, match, and whatever...
]

>>> There's something like "include all fields" keyword that I can use in this case or some another solution?
Unfortunaly, there is no operator to "include all fields" in aggregation operation. The only reason, why, because aggregation is mostly created to group/calculate data from collection fields (sum, avg, etc.) and return all the collection's fields is not direct purpose.

To add new fields to your document you can use $addFields
from docs
and to all the fields in your document, you can use $$ROOT
db.collection.aggregate([
{ "$addFields": { "custom_field": "$obj.obj_field1" } },
{ "$group": {
_id : "$field1",
data: { $push : "$$ROOT" }
}}
])

As of version 2.6.4, Mongo DB does not have such a feature for the $project aggregation pipeline. From the docs for $project:
Passes along the documents with only the specified fields to the next stage in the pipeline. The specified fields can be existing fields from the input documents or newly computed fields.
and
The _id field is, by default, included in the output documents. To include the other fields from the input documents in the output documents, you must explicitly specify the inclusion in $project.

according to #Deka reply, for c# mongodb driver 2.5 you can get the grouped document with all keys like below;
var group = new BsonDocument
{
{ "_id", "$groupField" },
{ "_document", new BsonDocument { { "$first", "$$ROOT" } } }
};
ProjectionDefinition<BsonDocument> projection = new BsonDocument{{ "document", "$_document"}};
var result = await col.Aggregate().Group(group).Project(projection).ToListAsync();
// For demo first record
var fistItemAsT = BsonSerializer.Deserialize<T>(result.ToArray()[0]["document"].AsBsonDocument);

Related

How to query certain elements of an array of objects? (mongodb)

say I have a mongo DB collection with records as follows:
{
email: "person1#gmail.com",
plans: [
{planName: "plan1", dataValue = 100},
{planName: "plan2", dataValue = 50}
]
},
{
email: "person2#gmail.com",
plans: [
{planName: "plan3", dataValue = 25},
{planName: "plan4", dataValue = 12.5}
]
}
and I want to query such that only the dataValue returns where the email is "person1#gmail.com" and the planName is "plan1". How would I approach this?
You can accomplish this using the Aggregation Pipeline.
The pipeline may look like this:
db.collection.aggregate([
{ $match: { "email" :"person1#gmail.com", "plans.planName": "plan1" }},
{ $unwind: "$plans" },
{ $match: { "plans.planName": "plan1" }},
{ $project: { "_id": 0, "dataValue": "$plans.dataValue" }}
])
The first $match stage will retrieve documents where the email field is equal to person1#gmail.com and any of the elements in the plans array has a planName equal to plan1.
The second $unwind stage will output one document per element in the plans array. The plans field will now be an object containing a single plan object.
In the third $match stage, the unwound documents are further matched against to only include documents with a plans.planName of plan1. Finally, the $project stage excludes the _id field and projects a single dataValue field with a value of plans.dataValue.
Note that with this approach, if the email field is not unique you may have multiple documents consist with just a dataValue field.

Can I update the exsisting record of mongodb by its id? [duplicate]

I want update an _id field of one document. I know it's not really good practice. But for some technical reason, I need to update it.
If I try to update it I get:
db.clients.update({ _id: ObjectId("123")}, { $set: { _id: ObjectId("456")}})
Performing an update on the path '_id' would modify the immutable field '_id'
And the update is rejected. How I can update it?
You cannot update it. You'll have to save the document using a new _id, and then remove the old document.
// store the document in a variable
doc = db.clients.findOne({_id: ObjectId("4cc45467c55f4d2d2a000002")})
// set a new _id on the document
doc._id = ObjectId("4c8a331bda76c559ef000004")
// insert the document, using the new _id
db.clients.insert(doc)
// remove the document with the old _id
db.clients.remove({_id: ObjectId("4cc45467c55f4d2d2a000002")})
To do it for your whole collection you can also use a loop (based on Niels example):
db.status.find().forEach(function(doc){
doc._id=doc.UserId; db.status_new.insert(doc);
});
db.status_new.renameCollection("status", true);
In this case UserId was the new ID I wanted to use
In case, you want to rename _id in same collection (for instance, if you want to prefix some _ids):
db.someCollection.find().snapshot().forEach(function(doc) {
if (doc._id.indexOf("2019:") != 0) {
print("Processing: " + doc._id);
var oldDocId = doc._id;
doc._id = "2019:" + doc._id;
db.someCollection.insert(doc);
db.someCollection.remove({_id: oldDocId});
}
});
if (doc._id.indexOf("2019:") != 0) {... needed to prevent infinite loop, since forEach picks the inserted docs, even throught .snapshot() method used.
Here I have a solution that avoid multiple requests, for loops and old document removal.
You can easily create a new idea manually using something like:_id:ObjectId()
But knowing Mongo will automatically assign an _id if missing, you can use aggregate to create a $project containing all the fields of your document, but omit the field _id. You can then save it with $out
So if your document is:
{
"_id":ObjectId("5b5ed345cfbce6787588e480"),
"title": "foo",
"description": "bar"
}
Then your query will be:
db.getCollection('myCollection').aggregate([
{$match:
{_id: ObjectId("5b5ed345cfbce6787588e480")}
}
{$project:
{
title: '$title',
description: '$description'
}
},
{$out: 'myCollection'}
])
You can also create a new document from MongoDB compass or using command and set the specific _id value that you want.
As a very small improvement to the above answers i would suggest using
let doc1 = {... doc};
then
db.dyn_user_metricFormulaDefinitions.deleteOne({_id: doc._id});
This way we don't need to create extra variable to hold old _id.
Slightly modified example of #Florent Arlandis above where we insert _id from a different field in a document:
> db.coll.insertOne({ "_id": 1, "item": { "product": { "id": 11 } }, "source": "Good Store" })
{ "acknowledged" : true, "insertedId" : 1 }
> db.coll.aggregate( [ { $set: { _id : "$item.product.id" }}, { $out: "coll" } ]) // inserting _id you want for the current collection
> db.coll.find() // check that _id is changed
{ "_id" : 11, "item" : { "product" : { "id" : 11 } }, "source" : "Good Store" }
Do not use $match filter + $out as in #Florent Arlandis's answer since $out fully remove data in collection before inserting aggregate result, so effectively you will loose all data that don't match to $match filter

Accelerate mongo update within two collections

I have a Payments collection with playerId field, which is the _id key of Person collection. I need to count once, what's the maximal payment of a person and save the value to person's document. This is how I do it now:
db.Person.find().forEach( function(person) {
var cursor = db.Payment.aggregate([
{$match: {playerId: person._id}},
{$group: {
_id:"$playerId",
maxp: {$max:"$amount"}
}}
]);
var maxPay = 0;
if (cursor.hasNext()) {
maxPay = cursor.next().maxp;
}
person.maxPay = maxPay;
db.Person.save(person);
});
I suppose seeking maxPay on Payments collection once for all Persons should be faster, but I dunno how to write that in code. Could you help me please?
You can run just a single aggregation pipeline operation which has a $lookup pipeline initially to do a "left join" on the Payment collection. This is necessary in order to get the data from the right collection (payments) embedded within the resulting documents as an array called payments.
The preceding $unwind pipeline deconstructs the embedded payments array i.e. it will generate a new record for each and every element of the payments data field. It basically flattens the data which will be useful for the next $group stage.
In this $group pipeline stage, you calculate your desired aggregates by applying the accumulator expression(s). If for instance your Person schema has other fields you wish to retain, then the $first accumulator operator should suffice in addition to the $max operator for the extra maxPay field.
UPDATE
Unfortunately, there is no operator to "include all fields" in the $group aggregation pipeline operation. This is because the $group pipeline step is mostly used to group and calculate/aggregate data from collection fields (sum, avg, etc.) and returning all the collection's fields is not the pipeline's intended purpose. The group pipeline operator is similar to the SQL's GROUP BY clause where you can't use GROUP BY unless you use any of the aggregation functions (accumulator operators in MongoDB). The same way, if you need to retain most fields, you have to use an aggregation function in MongoDB as well. In this case, you have to apply $first to each field you want to keep.
You can also use the $$ROOT system variable which references the root document. Keep all fields of this document in a field within the $group pipeline, for example:
{
"$group": {
"_id": "$_id",
"maxPay": { "$max": "$payments.amount" },
"doc": { "$first": "$$ROOT" }
}
}
The drawback with this approach is you would need a further $project pipeline to reshape the fields so that they match the original schema because the documents from the resulting pipeline will have only three fields; _id, maxPay and the embedded doc field.
The final pipeline stage, $out, writes the resulting documents of the aggregation pipeline to the same collection, akin to updating the Person collection by atomically replacing the existing collection with the new results collection. The $out operation does not change any indexes that existed on the previous collection. If the aggregation fails, the $out operation makes no changes to the pre-existing collection:
db.Person.aggregate([
{
"$lookup": {
"from": "Payment",
"localField": "_id",
"foreignField": "playerId",
"as": "payments"
}
},
{ "$unwind": {
"path": "$payments",
"preserveNullAndEmptyArrays": true
} },
{
"$group": {
"_id": "$_id",
"maxPay": { "$max": "$payments.amount" },
/* extra fields for demo purposes
"firstName": { "$first": "$firstName" },
"lastName": { "$first": "$lastName" }
*/
}
},
{ "$out": "Person" }
])

Is there any equivalent in MongoDB for MS-SQL command 'SET IDENTITY_INSERT tablename OFF'? [duplicate]

I want update an _id field of one document. I know it's not really good practice. But for some technical reason, I need to update it.
If I try to update it I get:
db.clients.update({ _id: ObjectId("123")}, { $set: { _id: ObjectId("456")}})
Performing an update on the path '_id' would modify the immutable field '_id'
And the update is rejected. How I can update it?
You cannot update it. You'll have to save the document using a new _id, and then remove the old document.
// store the document in a variable
doc = db.clients.findOne({_id: ObjectId("4cc45467c55f4d2d2a000002")})
// set a new _id on the document
doc._id = ObjectId("4c8a331bda76c559ef000004")
// insert the document, using the new _id
db.clients.insert(doc)
// remove the document with the old _id
db.clients.remove({_id: ObjectId("4cc45467c55f4d2d2a000002")})
To do it for your whole collection you can also use a loop (based on Niels example):
db.status.find().forEach(function(doc){
doc._id=doc.UserId; db.status_new.insert(doc);
});
db.status_new.renameCollection("status", true);
In this case UserId was the new ID I wanted to use
In case, you want to rename _id in same collection (for instance, if you want to prefix some _ids):
db.someCollection.find().snapshot().forEach(function(doc) {
if (doc._id.indexOf("2019:") != 0) {
print("Processing: " + doc._id);
var oldDocId = doc._id;
doc._id = "2019:" + doc._id;
db.someCollection.insert(doc);
db.someCollection.remove({_id: oldDocId});
}
});
if (doc._id.indexOf("2019:") != 0) {... needed to prevent infinite loop, since forEach picks the inserted docs, even throught .snapshot() method used.
Here I have a solution that avoid multiple requests, for loops and old document removal.
You can easily create a new idea manually using something like:_id:ObjectId()
But knowing Mongo will automatically assign an _id if missing, you can use aggregate to create a $project containing all the fields of your document, but omit the field _id. You can then save it with $out
So if your document is:
{
"_id":ObjectId("5b5ed345cfbce6787588e480"),
"title": "foo",
"description": "bar"
}
Then your query will be:
db.getCollection('myCollection').aggregate([
{$match:
{_id: ObjectId("5b5ed345cfbce6787588e480")}
}
{$project:
{
title: '$title',
description: '$description'
}
},
{$out: 'myCollection'}
])
You can also create a new document from MongoDB compass or using command and set the specific _id value that you want.
As a very small improvement to the above answers i would suggest using
let doc1 = {... doc};
then
db.dyn_user_metricFormulaDefinitions.deleteOne({_id: doc._id});
This way we don't need to create extra variable to hold old _id.
Slightly modified example of #Florent Arlandis above where we insert _id from a different field in a document:
> db.coll.insertOne({ "_id": 1, "item": { "product": { "id": 11 } }, "source": "Good Store" })
{ "acknowledged" : true, "insertedId" : 1 }
> db.coll.aggregate( [ { $set: { _id : "$item.product.id" }}, { $out: "coll" } ]) // inserting _id you want for the current collection
> db.coll.find() // check that _id is changed
{ "_id" : 11, "item" : { "product" : { "id" : 11 } }, "source" : "Good Store" }
Do not use $match filter + $out as in #Florent Arlandis's answer since $out fully remove data in collection before inserting aggregate result, so effectively you will loose all data that don't match to $match filter

How to update the _id of one MongoDB Document?

I want update an _id field of one document. I know it's not really good practice. But for some technical reason, I need to update it.
If I try to update it I get:
db.clients.update({ _id: ObjectId("123")}, { $set: { _id: ObjectId("456")}})
Performing an update on the path '_id' would modify the immutable field '_id'
And the update is rejected. How I can update it?
You cannot update it. You'll have to save the document using a new _id, and then remove the old document.
// store the document in a variable
doc = db.clients.findOne({_id: ObjectId("4cc45467c55f4d2d2a000002")})
// set a new _id on the document
doc._id = ObjectId("4c8a331bda76c559ef000004")
// insert the document, using the new _id
db.clients.insert(doc)
// remove the document with the old _id
db.clients.remove({_id: ObjectId("4cc45467c55f4d2d2a000002")})
To do it for your whole collection you can also use a loop (based on Niels example):
db.status.find().forEach(function(doc){
doc._id=doc.UserId; db.status_new.insert(doc);
});
db.status_new.renameCollection("status", true);
In this case UserId was the new ID I wanted to use
In case, you want to rename _id in same collection (for instance, if you want to prefix some _ids):
db.someCollection.find().snapshot().forEach(function(doc) {
if (doc._id.indexOf("2019:") != 0) {
print("Processing: " + doc._id);
var oldDocId = doc._id;
doc._id = "2019:" + doc._id;
db.someCollection.insert(doc);
db.someCollection.remove({_id: oldDocId});
}
});
if (doc._id.indexOf("2019:") != 0) {... needed to prevent infinite loop, since forEach picks the inserted docs, even throught .snapshot() method used.
Here I have a solution that avoid multiple requests, for loops and old document removal.
You can easily create a new idea manually using something like:_id:ObjectId()
But knowing Mongo will automatically assign an _id if missing, you can use aggregate to create a $project containing all the fields of your document, but omit the field _id. You can then save it with $out
So if your document is:
{
"_id":ObjectId("5b5ed345cfbce6787588e480"),
"title": "foo",
"description": "bar"
}
Then your query will be:
db.getCollection('myCollection').aggregate([
{$match:
{_id: ObjectId("5b5ed345cfbce6787588e480")}
}
{$project:
{
title: '$title',
description: '$description'
}
},
{$out: 'myCollection'}
])
You can also create a new document from MongoDB compass or using command and set the specific _id value that you want.
As a very small improvement to the above answers i would suggest using
let doc1 = {... doc};
then
db.dyn_user_metricFormulaDefinitions.deleteOne({_id: doc._id});
This way we don't need to create extra variable to hold old _id.
Slightly modified example of #Florent Arlandis above where we insert _id from a different field in a document:
> db.coll.insertOne({ "_id": 1, "item": { "product": { "id": 11 } }, "source": "Good Store" })
{ "acknowledged" : true, "insertedId" : 1 }
> db.coll.aggregate( [ { $set: { _id : "$item.product.id" }}, { $out: "coll" } ]) // inserting _id you want for the current collection
> db.coll.find() // check that _id is changed
{ "_id" : 11, "item" : { "product" : { "id" : 11 } }, "source" : "Good Store" }
Do not use $match filter + $out as in #Florent Arlandis's answer since $out fully remove data in collection before inserting aggregate result, so effectively you will loose all data that don't match to $match filter