I want to get 5 last documents from a MongoDB collection, then keep tailing it for new documents. Can this be done at all with one query, or do I really need two queries? If two queries, what's the best way to achieve this without adding extra fields?
While answer in any language is fine, here's an example node.js code snippet of what I try to achieve (error handling omitted, and snippet edited based on first answer to the question):
MongoClient.connect("mongodb://localhost:1338/mydb", function(err, db) {
db.collection('mycollection', function(err, col) {
col.count({}, function(err, total) {
col.find({}, { tailable:true, awaitdata:true, timeout:false, skip:total-5, limit:5 }, function(err, cursor) {
cursor.each(function(err, doc) {
console.dir(doc); // print the document object to console
});
});
});
});
});
Problem: Above code prints all the documents starting from first one, and then waits for more. Options skip and limit have no effect.
Question: How to easily get 5 latest documents, then keep on tailing for more? Example in any language is fine, does not have to be node.js.
(Answer edited, it's useful to know this does not work with these versions.)
If collection was not tailable, you'd need to find out how many items there is, for that use count, and then use skip option, to skip first count-5 items.
This will NOT work, tailable and skip do not work together (MongoDB 2.4.6, node.js 0.10.18):
MongoClient.connect("mongodb://localhost:1338/mydb", function(err, db) {
db.collection('mycollection', function(err, col) {
col.count({ }, function(err, total) {
col.find({ }, { tailable: true, awaitdata: true, timeout: false, skip: total - 5, limit: 5 }, function(err, cursor) {
cursor.each(function(err, doc) {
console.dir(doc);
});
});
});
});
});
Related
Does anyone have a suggestion about how to update a field in each document in a large collection?
I use something like this:
MyModel.find().exec(function(err,data){
if(err){
return console.log(err);
}
data.forEach(function(doc){
doc.Field = doc.Field + 1;
doc.save(function (err) {
if(err) {
console.error('ERROR!');
}
});
});
});
But I get FATAL ERROR: CALL_AND_RETRY_LAST Allocation failed - process out of memory.
Is there a way to process the above update in chunks or something like that?
You can use the async.eachLimit method of the async library to limit the number of concurrent save operations (doc link is to each, scroll down to see the eachLimit variant).
For example, to limit the saves to no more than 5 outstanding at a time:
MyModel.find().exec(function(err, data){
if (err) {
return console.log(err);
}
async.eachLimit(data, 5, function(doc, callback){
doc.Field = doc.Field + 1;
doc.save(function(err) {
if (err) {
console.error('ERROR!');
}
callback(err);
});
});
});
However, in this case it would be much more efficient to use a single update with the $inc operator and the multi: true option to increment each doc's Field value by 1.
MyModel.update({}, {$inc: {Field: 1}}, {multi: true), function(err) { ... });
you need more memory: --max_new_space_size and/or --max_old_space_size, like this:
node --max-old-space-size=4096 server.js
Currently, by default v8 has a memory limit of 512MB on 32-bit
systems, and 1.4GB on 64-bit systems. The limit can be raised by
setting --max_old_space_size to a maximum of ~1024 (~1 GB) (32-bit)
and ~4096 (~4GB) (64-bit), but it is recommended that you split your
single process into several workers if you are hitting memory limits
I am trying to fetch all the records but with selected fields, I have tried the following ways but none works:
Post.find(
{
where: {},
select: ['title']
}
);
Post.find(
{},
{
fields: {
title: 1
}
}
);
As this answer points out, the fields param "WILL work as long as you pass other params with it such as limit or order."
Alternatively, if you want this throughout your application, you could define a custom toJSON function for your model, under attributes. If not, you could still define it under some other (e.g. filter) and use map to return the custom objects instead of the default model. Remember to take care of the control flow while using map though. Use async/promises/raw logic to avoid returning before all objects are processed.
The issue has been resolved in sails-mongo latest version:
https://github.com/balderdashy/waterline/issues/1098
Thanks
I've played with trying to get above answer to use limit or order to kick in the projection to no avail.
I did see this in the docs located here:
http://sailsjs.org/documentation/reference/waterline-orm/models/native
With an out of the box solution for exactly what you're doing (pasted here for ease of use).
Pet.native(function(err, collection) {
if (err) return res.serverError(err);
collection.find({}, {
name: true
}).toArray(function (err, results) {
if (err) return res.serverError(err);
return res.ok(results);
});
});
Swap out the response base things and change Pet to Post and, this ought to work in the sails console:
Post.native(function(err, collection) {
if (err) throw new Error(err);
collection.find({}, {
title: true
}).toArray(function (err, results) {
if (err) throw new Error(err);
console.log(results);
});
});
You'll still get the _id field, and if you don't want that then hit the Mongo docs on not getting those hint(title: true, _id: false)hint
Hope this helps!
All the examples I have seen for MongoDb & Mongoskin for update, have individual properties being updated, like so:
// this works when I specify the properties
db.collection('User').update({_id: mongoskin.helper.toObjectID(user._id)},
{'$set':{displayName:user.displayName}}, function(err, result) {
if (err) throw err;
if (result){ res.send(result)}
});
But what if I wanted the whole object/document to be updated instead:
// this does not appear to work
db.collection('User').update({_id: mongoskin.helper.toObjectID(user._id)}, {'$set':user},
function(err, result){
// do something
}
It returns the error:
// It appears Mongo does not like the _id as part of the update
MongoError: After applying the update to the document {_id: ObjectId('.....
To overcome this issue, this is what I had to do to make things work:
function (req, res) {
var userId = req.body.user._id
var user = req.body.user;
delete user._id;
db.collection('User').update({_id: mongoskin.helper.toObjectID(userId)},
{'$set':user}, function(err, result) {
if (err) throw err;
console.log('result: ' + result)
if (result){ res.send(result)}
});
})
It there a more elegant way of updating the whole document, instead of hacking it with:
delete user._id
If you want to update the whole object, you do not need a $set. I am not aware of mongoskin, but in shell you would do something like:
var userObj = {
_id: <something>
...
};
db.user.update({_id: user._id}, user);
Which I think can be translated in your mongoskin in the following way.
db.collection('User').update({_id: user._id}, user, function(){...})
But here is the problem. You can not update _id of the element in Mongo. And this is what your error tells you. So you can remove the _id from your user object and have it separately. Search by this separate _id and update with a user object without _id.
I'm getting a strange error in Meteor. It allows me to update a count, but when I try to reverse the count, it won't update.
Template.listItem.events({
'click .remove': function(e, template) {
e.preventDefault();
ListItems.remove(this._id);
//Router.go('listPage', {_id: template.data._id});
},
'click .listItem': function(e, template) {
e.preventDefault();
var item = ListItems.findOne(this._id);
ListItems.update(this._id, {$set: { picked: true }});
Items.update(item.itemId, {$inc: {pickedCount: 1}});
},
'click .picked': function(e, template) {
e.preventDefault();
var item = ListItems.findOne(this._id);
console.log(item.itemId);
ListItems.update(this._id, {$set: { picked: false }});
Items.update({_id: item.itemId}, {$inc: {pickedCount: -1}});
}
});
The pickedCount is what I'm trying to revert. Everything I've read said this should work. How can I display the error from MongoDB if there is one? Is this the accepted syntax for decrementing a field in Mongo?
Most Meteor.call functions (or which Collection.update is one), take a last argument as a callback. From the docs:
callback Function Optional. If present, called with an error object as
the first argument and, if no error, the number of affected documents
as the second.
Hence, you can write your call to update as:
Items.update({_id: item.itemId}, {$inc: {pickedCount: -1}}, function (err) {
console.log("Error = ", err);
});
However, the error on the server console is likely to be more informative in your case.
Hi I am trying to build an application which upserts data and fetches from the mongodb baser on the userid.This approach works fine for a single user.But when i try hitting for multiple users say 25 the data fetched seems to be null. Below is my upsert code
collection.update({'USER_ID': passVal.ID},
{'RESPONSE': Data}, { upsert: true }, function (err) {
if (err) {
console.log("Error in saving data");
}
var query = collection.findOne({'USER_ID': passVal.ID});
query.select('RESPONSE');
query.exec(function (err, data) {
if (err) return handleError(err);
console.log(data.RESPONSE);
});
})
I always get an error insome cases as data is null.I have written the read code in the call back of upsert only.I am stuck here any help regarding this will be much helpful.