Given the following document structure:
{
'_id': ObjectId("559943fcda2485a66285576e"),
'user': '70gw3',
'data': [
{
'date': ISODate("2015-06-29T00:00:00.0Z"),
'clicks': 1,
},
{
'date': ISODate("2015-06-30T00:00:00.0Z"),
'clicks': 5,
},
]
}
How can I increment the clicks value of 2015-06-30 and increase if it doesn't exists [the whole document OR the specific date], make it equals to 1?
Unfortunately it is not possible to achieve your goal within a single query. But instead you can do the following
var exists = db.yourCollection.find(
{'_id': ObjectId("559943fcda2485a66285576e"), 'data.date': <your date>}
).count()
This query will return the number of documents which have the specified _id and an object with the specified date in the data array. As the _id field is unique, you will get at most one result. If such element does not exist in the data array, you will not get any result. Thus, you have the following cases:
exists == 1: There IS an element with the specified date
exists == 0: There IS NO element with the specified date
Then this can become your condition:
if (exists) {
db.yourCollection.update({_id: <your id>, 'data.date': <your date>}, {$inc: {'data.$.clicks': 1}})
} else {
if (db.runCommand({findAndModify: 'yourCollection', query: {_id: <your id>, 'data.date': {$ne: <your date>}}, update: {$push: {data: {date: <your date>, clicks: 1}}}}).value) {
db.yourCollection.update({_id: <your id>, 'data.date': <your date>}, {$inc: {'data.$.clicks': 1}})
}
}
The correct answer is that while you need to perform two update operations in order to fulfil the logic there is no need to find and check data in a way.
You have two basic conditions
If the element is there then update it
If the element is not there then add it
The process is basically then two updates. One will succeed and the other will not. No need to check.
Probably the best way to implement this is using the "Bulk" operations API. This allows both update statements to be sent to the server at the same time, and the response it a single response so there are less trips. But the logic is the same even if you don't use this.
var bulk = db.collection.initializeOrderedBulkOp();
// Update the date where it matched
bulk.find({ "user": "70gw3", "data.date": new Date("2015-06-30") })
.updateOne({ "$inc": { "data.$.clicks": 1 } });
// Push a new element where date does not match
bulk.find({ "user": "70gw3", "data.date": { "$ne": new Date("2015-06-30") } })
.updateOne({ "$push": { "data": { "date": new Date("2015-06-30"), "clicks": 1 } }});
bulk.execute();
In short, out of two update requests sent for any document then only one of the operations will actually modify anything. This is because the conditions for testing "date" are the "inverse" of one another, in that you "update" where present, and "create" where the matching element is not present.
Both states cannot exist at the same time, as long as the "create" follows the "update" the logic is sound. So first try to update, and then try to "create" where no match is found.
Related
I want update an _id field of one document. I know it's not really good practice. But for some technical reason, I need to update it.
If I try to update it I get:
db.clients.update({ _id: ObjectId("123")}, { $set: { _id: ObjectId("456")}})
Performing an update on the path '_id' would modify the immutable field '_id'
And the update is rejected. How I can update it?
You cannot update it. You'll have to save the document using a new _id, and then remove the old document.
// store the document in a variable
doc = db.clients.findOne({_id: ObjectId("4cc45467c55f4d2d2a000002")})
// set a new _id on the document
doc._id = ObjectId("4c8a331bda76c559ef000004")
// insert the document, using the new _id
db.clients.insert(doc)
// remove the document with the old _id
db.clients.remove({_id: ObjectId("4cc45467c55f4d2d2a000002")})
To do it for your whole collection you can also use a loop (based on Niels example):
db.status.find().forEach(function(doc){
doc._id=doc.UserId; db.status_new.insert(doc);
});
db.status_new.renameCollection("status", true);
In this case UserId was the new ID I wanted to use
In case, you want to rename _id in same collection (for instance, if you want to prefix some _ids):
db.someCollection.find().snapshot().forEach(function(doc) {
if (doc._id.indexOf("2019:") != 0) {
print("Processing: " + doc._id);
var oldDocId = doc._id;
doc._id = "2019:" + doc._id;
db.someCollection.insert(doc);
db.someCollection.remove({_id: oldDocId});
}
});
if (doc._id.indexOf("2019:") != 0) {... needed to prevent infinite loop, since forEach picks the inserted docs, even throught .snapshot() method used.
Here I have a solution that avoid multiple requests, for loops and old document removal.
You can easily create a new idea manually using something like:_id:ObjectId()
But knowing Mongo will automatically assign an _id if missing, you can use aggregate to create a $project containing all the fields of your document, but omit the field _id. You can then save it with $out
So if your document is:
{
"_id":ObjectId("5b5ed345cfbce6787588e480"),
"title": "foo",
"description": "bar"
}
Then your query will be:
db.getCollection('myCollection').aggregate([
{$match:
{_id: ObjectId("5b5ed345cfbce6787588e480")}
}
{$project:
{
title: '$title',
description: '$description'
}
},
{$out: 'myCollection'}
])
You can also create a new document from MongoDB compass or using command and set the specific _id value that you want.
As a very small improvement to the above answers i would suggest using
let doc1 = {... doc};
then
db.dyn_user_metricFormulaDefinitions.deleteOne({_id: doc._id});
This way we don't need to create extra variable to hold old _id.
Slightly modified example of #Florent Arlandis above where we insert _id from a different field in a document:
> db.coll.insertOne({ "_id": 1, "item": { "product": { "id": 11 } }, "source": "Good Store" })
{ "acknowledged" : true, "insertedId" : 1 }
> db.coll.aggregate( [ { $set: { _id : "$item.product.id" }}, { $out: "coll" } ]) // inserting _id you want for the current collection
> db.coll.find() // check that _id is changed
{ "_id" : 11, "item" : { "product" : { "id" : 11 } }, "source" : "Good Store" }
Do not use $match filter + $out as in #Florent Arlandis's answer since $out fully remove data in collection before inserting aggregate result, so effectively you will loose all data that don't match to $match filter
I have two models, one is user
userSchema = new Schema({
userID: String,
age: Number
});
and the other is the score recorded several times everyday for all users
ScoreSchema = new Schema({
userID: {type: String, ref: 'User'},
score: Number,
created_date = Date,
....
})
I would like to do some query/calculation on the score for some users meeting specific requirement, say I would like to calculate the average of score for all users greater than 20 day by day.
My thought is that firstly do the populate on Scores to populate user's ages and then do the aggregate after that.
Something like
Score.
populate('userID','age').
aggregate([
{$match: {'userID.age': {$gt: 20}}},
{$group: ...},
{$group: ...}
], function(err, data){});
Is it Ok to use populate before aggregate? Or I first find all the userID meeting the requirement and save them in a array and then use $in to match the score document?
No you cannot call .populate() before .aggregate(), and there is a very good reason why you cannot. But there are different approaches you can take.
The .populate() method works "client side" where the underlying code actually performs additional queries ( or more accurately an $in query ) to "lookup" the specified element(s) from the referenced collection.
In contrast .aggregate() is a "server side" operation, so you basically cannot manipulate content "client side", and then have that data available to the aggregation pipeline stages later. It all needs to be present in the collection you are operating on.
A better approach here is available with MongoDB 3.2 and later, via the $lookup aggregation pipeline operation. Also probably best to handle from the User collection in this case in order to narrow down the selection:
User.aggregate(
[
// Filter first
{ "$match": {
"age": { "$gt": 20 }
}},
// Then join
{ "$lookup": {
"from": "scores",
"localField": "userID",
"foriegnField": "userID",
"as": "score"
}},
// More stages
],
function(err,results) {
}
)
This is basically going to include a new field "score" within the User object as an "array" of items that matched on "lookup" to the other collection:
{
"userID": "abc",
"age": 21,
"score": [{
"userID": "abc",
"score": 42,
// other fields
}]
}
The result is always an array, as the general expected usage is a "left join" of a possible "one to many" relationship. If no result is matched then it is just an empty array.
To use the content, just work with an array in any way. For instance, you can use the $arrayElemAt operator in order to just get the single first element of the array in any future operations. And then you can just use the content like any normal embedded field:
{ "$project": {
"userID": 1,
"age": 1,
"score": { "$arrayElemAt": [ "$score", 0 ] }
}}
If you don't have MongoDB 3.2 available, then your other option to process a query limited by the relations of another collection is to first get the results from that collection and then use $in to filter on the second:
// Match the user collection
User.find({ "age": { "$gt": 20 } },function(err,users) {
// Get id list
userList = users.map(function(user) {
return user.userID;
});
Score.aggregate(
[
// use the id list to select items
{ "$match": {
"userId": { "$in": userList }
}},
// more stages
],
function(err,results) {
}
);
});
So by getting the list of valid users from the other collection to the client and then feeding that to the other collection in a query is the onyl way to get this to happen in earlier releases.
I've been wrestling with the asynchronous nature of MongoDB, Mongoose and JavaScript and how to best make multiple updates to a collection.
I have an Excel sheet of client and contact data. There are some clients with multiple contacts, one per line, and the client data is the same (so the client name can be used as a unique key - in fact in the schema it's defined with unique: true).
The logic I want to achieve is:
Search the Client collection for the client with clientName as the key
If a matching clientName isn't found then create a new document for that client (not an upsert, I don't want to change anything if the client document is already in the database)
Check to see if the contact is already present in the array of contacts within the client document using firstName and lastName as the keys
If the contact isn't found then $push that contact onto the array
Of course, we could easily have a situation where the client doesn't exists (and so is created) and then immediately, the very next row of the sheet, is another contact for the same client so then I'd want to find that existing (just created) client and $push that 2nd new contact into the array.
I've tried this but it's not working:
Client.findOneAndUpdate(
{clientName: obj.client.clientname},
{$set: obj.client, $push: {contacts: obj.contact}},
{upsert: true, new: true},
function(err, client){
console.log(client)
}
)
and I've had a good look at other questions, e.g.:
create mongodb document with subdocuments atomically?
https://stackoverflow.com/questions/28026197/upserting-complex-collections-through-mongoose-via-express
but can't get a solution... I'm coming to the conclusion that maybe I have to use some app logic to do the find, then decisions in my code, then writes, rather than use a single Mongoose/Mongo statement, but then the issues of asynchronicity rear their ugly head.
Any suggestions?
The approach to handling this is not a simple one, as mixing "upserts" with adding items to "arrays" can easily lead to undesired results. It also depends on if you want logic to set other fields such as a "counter" indicating how many contacts there are within an array, which you only want to increment/decrement as items are added or removed respectively.
In the most simple case however, if the "contacts" only contained a singular value such as an ObjectId linking to another collection, then the $addToSet modifier works well, as long as there no "counters" involved:
Client.findOneAndUpdate(
{ "clientName": clientName },
{ "$addToSet": { "contacts": contact } },
{ "upsert": true, "new": true },
function(err,client) {
// handle here
}
);
And that is all fine as you are only testing to see if a doucment matches on the "clientName", if not upsert it. Whether there is a match or not, the $addToSet operator will take care of unique "singular" values, being any "object" that is truly unique.
The difficulties come in where you have something like:
{ "firstName": "John", "lastName": "Smith", "age": 37 }
Already in the contacts array, and then you want to do something like this:
{ "firstName": "John", "lastName": "Smith", "age": 38 }
Where your actual intention is that this is the "same" John Smith, and it's just that the "age" is not different. Ideally you want to just "update" that array entry end neiter create a new array or a new document.
Working this with .findOneAndUpdate() where you want the updated document to return can be difficult. So if you don't really want the modified document in response, then the Bulk Operations API of MongoDB and the core driver are of most help here.
Considering the statements:
var bulk = Client.collection.initializeOrderedBulkOP();
// First try the upsert and set the array
bulk.find({ "clientName": clientName }).upsert().updateOne({
"$setOnInsert": {
// other valid client info in here
"contacts": [contact]
}
});
// Try to set the array where it exists
bulk.find({
"clientName": clientName,
"contacts": {
"$elemMatch": {
"firstName": contact.firstName,
"lastName": contact.lastName
}
}
}).updateOne({
"$set": { "contacts.$": contact }
});
// Try to "push" the array where it does not exist
bulk.find({
"clientName": clientName,
"contacts": {
"$not": { "$elemMatch": {
"firstName": contact.firstName,
"lastName": contact.lastName
}}
}
}).updateOne({
"$push": { "contacts": contact }
});
bulk.execute(function(err,response) {
// handle in here
});
This is nice since the Bulk Operations here mean that all statements here are sent to the server at once and there is only one response. Also note here that the logic means here that at most only two operations will actually modify anything.
In the first instance, the $setOnInsert modifier makes sure that nothing is changed when the document is just a match. As the only modifications here are within that block, this only affects a document where an "upsert" occurs.
Also note on the next two statements you do not try to "upsert" again. This considers that the first statement was possibly successful where it had to be, or otherwise did not matter.
The other reason for no "upsert" there is because the condtions needed to test the presence of the element in the array would lead to the "upsert" of a new document when they were not met. That is not desired, therefore no "upsert".
What they do in fact is respectively check whether the array element is present or not, and either update the existing element or create a new one. Therefore in total, all operations mean you either modify "once" or at most "twice" in the case where an upsert occurred. The possible "twice" creates very little overhead and no real problem.
Also in the third statement the $not operator reverses the logic of the $elemMatch to determine that no array element with the query condition exists.
Translating this with .findOneAndUpdate() becomes a bit more of an issue. Not only is it the "success" that matters now, it also determines how the eventual content is returned.
So the best idea here is to run the events in "series", and then work a little magic with the result in order to return the end "updated" form.
The help we will use here is both with async.waterfall and the lodash library:
var _ = require('lodash'); // letting you know where _ is coming from
async.waterfall(
[
function(callback) {
Client.findOneAndUpdate(
{ "clientName": clientName },
{
"$setOnInsert": {
// other valid client info in here
"contacts": [contact]
}
},
{ "upsert": true, "new": true },
callback
);
},
function(client,callback) {
Client.findOneAndUpdate(
{
"clientName": clientName,
"contacts": {
"$elemMatch": {
"firstName": contact.firstName,
"lastName": contact.lastName
}
}
},
{ "$set": { "contacts.$": contact } },
{ "new": true },
function(err,newClient) {
client = client || {};
newClient = newClient || {};
client = _.merge(client,newClient);
callback(err,client);
}
);
},
function(client,callback) {
Client.findOneAndUpdate(
{
"clientName": clientName,
"contacts": {
"$not": { "$elemMatch": {
"firstName": contact.firstName,
"lastName": contact.lastName
}}
}
},
{ "$push": { "contacts": contact } },
{ "new": true },
function(err,newClient) {
newClient = newClient || {};
client = _.merge(client,newClient);
callback(err,client);
}
);
}
],
function(err,client) {
if (err) throw err;
console.log(client);
}
);
That follows the same logic as before in that only two or one of those statements is actually going to do anything with the possibility that the "new" document returned is going to be null. The "waterfall" here passes a result from each stage onto the next, including the end where also any error will immediately branch to.
In this case the null would be swapped for an empty object {} and the _.merge() method will combine the two objects into one, at each later stage. This gives you the final result which is the modified object, no matter which preceeding operations actually did anything.
Of course, there would be a differnt manipulation required for $pull, and also your question has input data as an object form in itself. But those are actually answers in themselves.
This should at least get you started on how to approach your update pattern.
I don't use mongoose so I'll post a mongo shell update; sorry for that. I think the following would do:
db.clients.update({$and:[{'clientName':'apple'},{'contacts.firstName': {$ne: 'nick'}},{'contacts.lastName': {$ne: 'white'}}]},
{$set:{'clientName':'apple'}, $push: {contacts: {'firstName': 'nick', 'lastName':'white'}}},
{upsert: true });
So:
if the client "apple" does not exists, it is created, with a contact with given first and last name. If it exists, and does not have the given contact, it has it pushed. If it exists, and already has the given contact, nothing happens.
I'm having a (hopefully) small syntax problem with $pull in Mongodb.
bulk.find({_id: new mongo.ObjectID(req.session._id)}).updateOne({$pull: {
firstArray: {id: req.params.id},
'secondArray.firstArrayIds': req.params.id
'secondArray.$.firstArrayIds': req.params.id
}});
The firstArray $pull works just fine.
But the secondArray.firstArrayIds and/or secondArray.$.firstArrayIds does not. What am I doing wrong here?
This is my data structure:
clients: {
firstArray: [
{
_id: '153'.
someField1: 'someVal1',
}
...
]
secondArray: [
{
_id: '7423'.
someField1: 'someVal1',
firstArrayIds: [153, 154, 155, ...]
}
...
]
}
EDIT What if there are more than one embedded object in secondArray which firstArrayIds can contain the id i want to delete. In other words, when deleting an object in firstdArray i want to delete references in all secondArray's firstArrayIds Not just the first match.
Your "secondArray" has a nested element structure, so you must identify the outer element you want to match in your query when using a positional $ operator in the update. You basically need something like this:
bulk.find({
"_id": new mongo.ObjectID(req.session._id),
"secondArray._id": "7423"
}).update({
"$pull": {
"firstArray": { "_id": "153" },
"secondArray.$.firstArrayIds": 153
}
});
So there are in fact "two" id values you need to pass in with your request in addition to the general document id. Even though this is nested it is okay since you are only matching on the "outer" level and only on one array. If you tried to match the position on more than one array then this is not possible with the positional operator.
I want update an _id field of one document. I know it's not really good practice. But for some technical reason, I need to update it.
If I try to update it I get:
db.clients.update({ _id: ObjectId("123")}, { $set: { _id: ObjectId("456")}})
Performing an update on the path '_id' would modify the immutable field '_id'
And the update is rejected. How I can update it?
You cannot update it. You'll have to save the document using a new _id, and then remove the old document.
// store the document in a variable
doc = db.clients.findOne({_id: ObjectId("4cc45467c55f4d2d2a000002")})
// set a new _id on the document
doc._id = ObjectId("4c8a331bda76c559ef000004")
// insert the document, using the new _id
db.clients.insert(doc)
// remove the document with the old _id
db.clients.remove({_id: ObjectId("4cc45467c55f4d2d2a000002")})
To do it for your whole collection you can also use a loop (based on Niels example):
db.status.find().forEach(function(doc){
doc._id=doc.UserId; db.status_new.insert(doc);
});
db.status_new.renameCollection("status", true);
In this case UserId was the new ID I wanted to use
In case, you want to rename _id in same collection (for instance, if you want to prefix some _ids):
db.someCollection.find().snapshot().forEach(function(doc) {
if (doc._id.indexOf("2019:") != 0) {
print("Processing: " + doc._id);
var oldDocId = doc._id;
doc._id = "2019:" + doc._id;
db.someCollection.insert(doc);
db.someCollection.remove({_id: oldDocId});
}
});
if (doc._id.indexOf("2019:") != 0) {... needed to prevent infinite loop, since forEach picks the inserted docs, even throught .snapshot() method used.
Here I have a solution that avoid multiple requests, for loops and old document removal.
You can easily create a new idea manually using something like:_id:ObjectId()
But knowing Mongo will automatically assign an _id if missing, you can use aggregate to create a $project containing all the fields of your document, but omit the field _id. You can then save it with $out
So if your document is:
{
"_id":ObjectId("5b5ed345cfbce6787588e480"),
"title": "foo",
"description": "bar"
}
Then your query will be:
db.getCollection('myCollection').aggregate([
{$match:
{_id: ObjectId("5b5ed345cfbce6787588e480")}
}
{$project:
{
title: '$title',
description: '$description'
}
},
{$out: 'myCollection'}
])
You can also create a new document from MongoDB compass or using command and set the specific _id value that you want.
As a very small improvement to the above answers i would suggest using
let doc1 = {... doc};
then
db.dyn_user_metricFormulaDefinitions.deleteOne({_id: doc._id});
This way we don't need to create extra variable to hold old _id.
Slightly modified example of #Florent Arlandis above where we insert _id from a different field in a document:
> db.coll.insertOne({ "_id": 1, "item": { "product": { "id": 11 } }, "source": "Good Store" })
{ "acknowledged" : true, "insertedId" : 1 }
> db.coll.aggregate( [ { $set: { _id : "$item.product.id" }}, { $out: "coll" } ]) // inserting _id you want for the current collection
> db.coll.find() // check that _id is changed
{ "_id" : 11, "item" : { "product" : { "id" : 11 } }, "source" : "Good Store" }
Do not use $match filter + $out as in #Florent Arlandis's answer since $out fully remove data in collection before inserting aggregate result, so effectively you will loose all data that don't match to $match filter