I want to get all the pupils whose last mark is between 15 and 20. To do so, I perform the following query in my mongoDB using mongoose:
The models are working fine (all the other queries are ok).
Pupils.find({"marks[-1].value": {'$lt' : 20 }, "marks[-1].value" : { '$gt' : 15 }}, function(err, things){
This is not working, is there something I missed ?
* UPDATE *
I found something like:
Pupils.find({ "marks[-1].value": {$gt : 15, $lt : 20}});
But this does not work either. Is there a way to get the last mark of the marks array in this case ?
Lets consider your Pupils collection:
Pupils
{
_id,
Marks(integer),
LatestMark(int)
}
I suggest to add latest mark into Pupil document(as you can see at the document above), and update it each time when you adding new mark into nested collection.
Then you will able to query on it like this:
db.Pupils.find({ "LatestMark": {$gt : 15, $lt : 20}});
Also you can query latest mark using $where, but be care because:
Javascript executes more slowly than
the native operators, but is very flexible
I believe it's not working because the embedded collections in mongo are accessed like this:
"marks.0.value", although I haven't used mongoose.
Unfortunately for your scenario, I do not think there is a way to use negative indexing. (Mongo doesn't guarantee a preserved natural order unless you use a capped collection anyway though)
You may be able to accomplish this using Map/Reduce or a group command
http://www.mongodb.org/display/DOCS/MapReduce
This is tip than an answer.
Use double quotes for special key words like $elemMatch, $get, $lt etc while using mangoose.
In following code,$gt will not work properly.
var elmatch = { companyname: mycompany };
var condition = { company_info: { $elemMatch: elmatch } };
if(isValid(last_id)){
condition._id = { $gt: new ObjectId(last_id) };
}
console.log(condition);
User.find(condition).limit(limit).sort({_id: 1}).exec(function (err, lists) {
if (!err) {
res.send(lists);
res.end();
}else{
res.send(err);
res.end();
}
});
But this issue is solved when i'm using double quotes for special keywords
var elmatch = { companyname: mycompany };
var condition = { company_info: { "$elemMatch": elmatch } };
if(isValid(last_id)){
condition._id = { "$gt": new ObjectId(last_id) };
}
console.log(condition);
User.find(condition).limit(limit).sort({_id: 1}).exec(function (err, lists) {
if (!err) {
res.send(lists);
res.end();
}else{
res.send(err);
res.end();
}
});
I hope it will be helpful.
Related
Bit of an odd one on query performance... I need to run a query which does a total count of documents, and can also return a result set that can be limited and offset.
So, I have 57 documents in total, and the user wants 10 documents offset by 20.
I can think of 2 ways of doing this, first is query for all 57 documents (returned as an array), then using array.slice return the documents they want. The second option is to run 2 queries, the first one using mongo's native 'count' method, then run a second query using mongo's native $limit and $skip aggregators.
Which do you think would scale better? Doing it all in one query, or running two separate ones?
Edit:
// 1 query
var limit = 10;
var offset = 20;
Animals.find({}, function (err, animals) {
if (err) {
return next(err);
}
res.send({count: animals.length, animals: animals.slice(offset, limit + offset)});
});
// 2 queries
Animals.find({}, {limit:10, skip:20} function (err, animals) {
if (err) {
return next(err);
}
Animals.count({}, function (err, count) {
if (err) {
return next(err);
}
res.send({count: count, animals: animals});
});
});
I suggest you to use 2 queries:
db.collection.count() will return total number of items. This value is stored somewhere in Mongo and it is not calculated.
db.collection.find().skip(20).limit(10) here I assume you could use a sort by some field, so do not forget to add an index on this field. This query will be fast too.
I think that you shouldn't query all items and than perform skip and take, cause later when you have big data you will have problems with data transferring and processing.
Instead of using 2 separate queries, you can use aggregate() in a single query:
Aggregate "$facet" can be fetch more quickly, the Total Count and the Data with skip & limit
db.collection.aggregate([
//{$sort: {...}}
//{$match:{...}}
{$facet:{
"stage1" : [ {"$group": {_id:null, count:{$sum:1}}} ],
"stage2" : [ { "$skip": 0}, {"$limit": 2} ]
}},
{$unwind: "$stage1"},
//output projection
{$project:{
count: "$stage1.count",
data: "$stage2"
}}
]);
output as follows:-
[{
count: 50,
data: [
{...},
{...}
]
}]
Also, have a look at https://docs.mongodb.com/manual/reference/operator/aggregation/facet/
db.collection_name.aggregate([
{ '$match' : { } },
{ '$sort' : { '_id' : -1 } },
{ '$facet' : {
metadata: [ { $count: "total" } ],
data: [ { $skip: 1 }, { $limit: 10 },{ '$project' : {"_id":0} } ] // add projection here wish you re-shape the docs
} }
] )
Instead of using two queries to find the total count and skip the matched record.
$facet is the best and optimized way.
Match the record
Find total_count
skip the record
And also can reshape data according to our needs in the query.
There is a library that will do all of this for you, check out mongoose-paginate-v2
After having to tackle this issue myself, I would like to build upon user854301's answer.
Mongoose ^4.13.8 I was able to use a function called toConstructor() which allowed me to avoid building the query multiple times when filters are applied. I know this function is available in older versions too but you'll have to check the Mongoose docs to confirm this.
The following uses Bluebird promises:
let schema = Query.find({ name: 'bloggs', age: { $gt: 30 } });
// save the query as a 'template'
let query = schema.toConstructor();
return Promise.join(
schema.count().exec(),
query().limit(limit).skip(skip).exec(),
function (total, data) {
return { data: data, total: total }
}
);
Now the count query will return the total records it matched and the data returned will be a subset of the total records.
Please note the () around query() which constructs the query.
You don't have to use two queries or one complicated query with aggregate and such.
You can use one query
example:
const getNames = async (queryParams) => {
const cursor = db.collection.find(queryParams).skip(20).limit(10);
return {
count: await cursor.count(),
data: await cursor.toArray()
}
}
mongo returns a cursor that has predefined functions such as count, which will return the full count of the queried results regardless of skip and limit
So in count property, you will get the full length of the collection and in data, you will get just the chunk with offset of 20 and limit of 10 documents
Thanks Igor Igeto Mitkovski, a best solution is using native connection
document is here: https://docs.mongodb.com/manual/reference/method/cursor.count/#mongodb-method-cursor.count
and mongoose dont support it ( https://github.com/Automattic/mongoose/issues/3283 )
we have to use native connection.
const query = StudentModel.collection.find(
{
age: 13
},
{
projection:{ _id:0 }
}
).sort({ time: -1 })
const count = await query.count()
const records = await query.skip(20)
.limit(10).toArray()
I am totally new to MongoDB... I am missing a "newbie" tag, so the experts would not have to see this question.
I am trying to update all documents in a collection using an expression. The query I was expecting to solve this was:
db.QUESTIONS.update({}, { $set: { i_pp : i_up * 100 - i_down * 20 } }, false, true);
That, however, results in the following error message:
ReferenceError: i_up is not defined (shell):1
At the same time, the database did not have any problem with eating this one:
db.QUESTIONS.update({}, { $set: { i_pp : 0 } }, false, true);
Do I have to do this one document at a time or something? That just seems excessively complicated.
Update
Thank you Sergio Tulentsev for telling me that it does not work. Now, I am really struggling with how to do this. I offer 500 Profit Points to the helpful soul, who can write this in a way that MongoDB understands. If you register on our forum I can add the Profit Points to your account there.
I just came across this while searching for the MongoDB equivalent of SQL like this:
update t
set c1 = c2
where ...
Sergio is correct that you can't reference another property as a value in a straight update. However, db.c.find(...) returns a cursor and that cursor has a forEach method:
Queries to MongoDB return a cursor, which can be iterated to retrieve
results. The exact way to query will vary with language driver.
Details below focus on queries from the MongoDB shell (i.e. the
mongo process).
The shell find() method returns a cursor object which we can then iterate to retrieve specific documents from the result. We use
hasNext() and next() methods for this purpose.
for( var c = db.parts.find(); c.hasNext(); ) {
print( c.next());
}
Additionally in the shell, forEach() may be used with a cursor:
db.users.find().forEach( function(u) { print("user: " + u.name); } );
So you can say things like this:
db.QUESTIONS.find({}, {_id: true, i_up: true, i_down: true}).forEach(function(q) {
db.QUESTIONS.update(
{ _id: q._id },
{ $set: { i_pp: q.i_up * 100 - q.i_down * 20 } }
);
});
to update them one at a time without leaving MongoDB.
If you're using a driver to connect to MongoDB then there should be some way to send a string of JavaScript into MongoDB; for example, with the Ruby driver you'd use eval:
connection.eval(%q{
db.QUESTIONS.find({}, {_id: true, i_up: true, i_down: true}).forEach(function(q) {
db.QUESTIONS.update(
{ _id: q._id },
{ $set: { i_pp: q.i_up * 100 - q.i_down * 20 } }
);
});
})
Other languages should be similar.
//the only differnce is to make it look like and aggregation pipeline
db.table.updateMany({}, [{
$set: {
col3:{"$sum":["$col1","$col2"]}
},
}]
)
You can't use expressions in updates. Or, rather, you can't use expressions that depend on fields of the document. Simple self-containing math expressions are fine (e.g. 2 * 2).
If you want to set a new field for all documents that is a function of other fields, you have to loop over them and update manually. Multi-update won't help here.
Rha7 gave a good idea, but the code above is not work without defining a temporary variable.
This sample code produces an approximate calculation of the age (leap years behinds the scene) based on 'birthday' field and inserts the value into suitable field for all documents not containing such:
db.employers.find({age: {$exists: false}}).forEach(function(doc){
var new_age = parseInt((ISODate() - doc.birthday)/(3600*1000*24*365));
db.employers.update({_id: doc._id}, {$set: {age: new_age}});
});
Example to remove "00" from the beginning of a caller id:
db.call_detail_records_201312.find(
{ destination: /^001/ },
{ "destination": true }
).forEach(function(row){
db.call_detail_records_201312.update(
{ _id: row["_id"] },
{ $set: {
destination: row["destination"].replace(/^001/, '1')
}
}
)
});
I have an existing MongoDB collection containing user names. The user names contain both lower case and upper case letters.
I want to update all the user names so they only contain lower case letters.
I have tried this script, but it didn't work
db.myCollection.find().forEach(
function(e) {
e.UserName = $toLower(e.UserName);
db.myCollection.save(e);
}
)
MongoDB does not have a concept of $toLower as a command. The solution is to run a big for loop over the data and issue the updates individually.
You can do this in any driver or from the shell:
db.myCollection.find().forEach(
function(e) {
e.UserName = e.UserName.toLowerCase();
db.myCollection.save(e);
}
)
You can also replace the save with an atomic update:
db.myCollection.update({_id: e._id}, {$set: {UserName: e.UserName.toLowerCase() } })
Again, you could also do this from any of the drivers, the code will be very similar.
EDIT: Remon brings up a good point. The $toLower command does exist as part of the aggregation framework, but this has nothing to do with updating. The documentation for updating is here.
Starting Mongo 4.2, db.collection.update() can accept an aggregation pipeline, finally allowing the update of a field based on its own value:
// { username: "Hello World" }
db.collection.updateMany(
{},
[{ $set: { username: { $toLower: "$username" } } }]
)
// { username: "hello world" }
The first part {} is the match query, filtering which documents to update (in this case all documents).
The second part [{ $set: { username: { $toLower: "$username" } } }], is the update aggregation pipeline (note the squared brackets signifying the use of an aggregation pipeline):
$set is a new aggregation operator which in this case modifies the value for "username".
Using $toLower, we modify the value of "username" by its lowercase version.
Very similar solution but this worked me in new mongo 3.2
Execute the following in Mongo Shell or equivalent DB tools like MongoChef!
db.tag.find({hashtag :{ $exists:true}}).forEach(
function(e) {
e.hashtag = e.hashtag.toLowerCase();
db.tag.save(e);
});
With the accepted solution I know its very trivial to do the same for an array of elements, just in case
db.myCollection.find().forEach(
function(e) {
for(var i = 0; i < e.articles.length; i++) {
e.articles[i] = e.articles[i].toLowerCase();
}
db.myCollection.save(e);
}
)
A little late to the party but the below answer works very well with mongo 3.4 and above
First get only those records which have different case and update only those records in bulk.
The performance of this query is multifold better
var bulk = db.myCollection.initializeUnorderedBulkOp();
var count = 0
db.myCollection.find({userId:{$regex:'.*[A-Z]'}}).forEach(function(e) {
var newId = e.userId.toLowerCase();
bulk.find({_id:e._id}).updateOne({$set:{userId: newId}})
count++
if (count % 500 === 0) {
bulk.execute();
bulk = db.myCollection.initializeUnorderedBulkOp();
count = 0;
}
})
if (count > 0) bulk.execute();
Just a note to make sure the field exists for all entries in your collection. If not you will need an if statement, like the following:
if (e.UserName) e.UserName = e.UserName.toLowerCase();
I'm trying to use MongoDB to analyse Apache log files. I've created a receipts collection from the Apache access logs. Here's an abridged summary of what my models look like:
db.receipts.findOne()
{
"_id" : ObjectId("4e57908c7a044a30dc03a888"),
"path" : "/videos/1/show_invisibles.m4v",
"issued_at" : ISODate("2011-04-08T00:00:00Z"),
"status" : "200"
}
I've written a MapReduce function that groups all data by the issued_at date field. It summarizes the total number of requests, and provides a breakdown of the number of requests for each unique path. Here's an example of what the output looks like:
db.daily_hits_by_path.findOne()
{
"_id" : ISODate("2011-04-08T00:00:00Z"),
"value" : {
"count" : 6,
"paths" : {
"/videos/1/show_invisibles.m4v" : {
"count" : 2
},
"/videos/1/show_invisibles.ogv" : {
"count" : 3
},
"/videos/6/buffers_listed_and_hidden.ogv" : {
"count" : 1
}
}
}
}
How can I make the output look like this instead:
{
"_id" : ISODate("2011-04-08T00:00:00Z"),
"count" : 6,
"paths" : {
"/videos/1/show_invisibles.m4v" : {
"count" : 2
},
"/videos/1/show_invisibles.ogv" : {
"count" : 3
},
"/videos/6/buffers_listed_and_hidden.ogv" : {
"count" : 1
}
}
}
It's not currently possible, but I would suggest voting for this case: https://jira.mongodb.org/browse/SERVER-2517.
Taking the best from previous answers and comments:
db.items.find().hint({_id: 1}).forEach(function(item) {
db.items.update({_id: item._id}, item.value);
});
From http://docs.mongodb.org/manual/core/update/#replace-existing-document-with-new-document
"If the update argument contains only field and value pairs, the update() method replaces the existing document with the document in the update argument, except for the _id field."
So you need neither to $unset value, nor to list each field.
From https://docs.mongodb.com/manual/core/read-isolation-consistency-recency/#cursor-snapshot
"MongoDB cursors can return the same document more than once in some situations. ... use a unique index on this field or these fields so that the query will return each document no more than once. Query with hint() to explicitly force the query to use that index."
AFAIK, by design Mongo's map reduce will spit results out in "value tuples" and I haven't seen anything that will configure that "output format". Maybe the finalize() method can be used.
You could try running a post-process that will reshape the data using
results.find({}).forEach( function(result) {
results.update({_id: result._id}, {count: result.value.count, paths: result.value.paths})
});
Yep, that looks ugly. I know.
You can do Dan's code with a collection reference:
function clean(collection) {
collection.find().forEach( function(result) {
var value = result.value;
delete value._id;
collection.update({_id: result._id}, value);
collection.update({_id: result.id}, {$unset: {value: 1}} ) } )};
A similar approach to that of #ljonas but no need to hardcode document fields:
db.results.find().forEach( function(result) {
var value = result.value;
delete value._id;
db.results.update({_id: result._id}, value);
db.results.update({_id: result.id}, {$unset: {value: 1}} )
} );
All the proposed solutions are far from optimal. The fastest you can do so far is something like:
var flattenMRCollection=function(dbName,collectionName) {
var collection=db.getSiblingDB(dbName)[collectionName];
var i=0;
var bulk=collection.initializeUnorderedBulkOp();
collection.find({ value: { $exists: true } }).addOption(16).forEach(function(result) {
print((++i));
//collection.update({_id: result._id},result.value);
bulk.find({_id: result._id}).replaceOne(result.value);
if(i%1000==0)
{
print("Executing bulk...");
bulk.execute();
bulk=collection.initializeUnorderedBulkOp();
}
});
bulk.execute();
};
Then call it:
flattenMRCollection("MyDB","MyMRCollection")
This is WAY faster than doing sequential updates.
While experimenting with Vincent's answer, I found a couple of problems. Basically, if you perform updates within a foreach loop, this will move the document to the end of the collection and the cursor will reach that document again (example). This can be circumvented if $snapshot is used. Hence, I am providing a Java example below.
final List<WriteModel<Document>> bulkUpdate = new ArrayList<>();
// You should enable $snapshot if performing updates within foreach
collection.find(new Document().append("$query", new Document()).append("$snapshot", true)).forEach(new Block<Document>() {
#Override
public void apply(final Document document) {
// Note that I used incrementing long values for '_id'. Change to String if
// you used string '_id's
long docId = document.getLong("_id");
Document subDoc = (Document)document.get("value");
WriteModel<Document> m = new ReplaceOneModel<>(new Document().append("_id", docId), subDoc);
bulkUpdate.add(m);
// If you used non-incrementing '_id's, then you need to use a final object with a counter.
if(docId % 1000 == 0 && !bulkUpdate.isEmpty()) {
collection.bulkWrite(bulkUpdate);
bulkUpdate.removeAll(bulkUpdate);
}
}
});
// Fixing bug related to Vincent's answer.
if(!bulkUpdate.isEmpty()) {
collection.bulkWrite(bulkUpdate);
bulkUpdate.removeAll(bulkUpdate);
}
Note : This snippet takes an average of 7.4 seconds to execute on my machine with 100k records and 14 attributes (IMDB dataset). Without batching, it takes an average of 25.2 seconds.
I'm trying to 'stream' data from a node.js/MongoDB instance to the client using websockets. It is all working well.
But how to I identify the last document in the result? I'm using node-mongodb-native to connect to MongoDB from node.js.
A simplified example:
collection.find({}, {}, function(err, cursor) {
if (err) sys.puts(err.message);
cursor.each(function(err, doc) {
client.send(doc);
});
});
Since mongodb objectId contatins creation date you can sort by id, descending and then use limit(1):
db.collection.find().sort( { _id : -1 } ).limit(1);
Note: i am not familiar with node.js at all, above command is mongo shell command and i suppose you can easy rewrite it to node.js.
Say I have companies collection. Below snippet gives me last document in the collection.
db.companies.find({},{"_id":1}).skip(db.companies.find().count()-1);
Code cannot rely on _id as it may not be on a specific pattern always if it's a user defined value.
Use sort and limit, if you want to use cursor :
var last = null;
var findCursor = collection.find({}).cursor();
findCursor.on("data", function(data) {
last = data;
...
});
findCursor.on("end", function(data) {
// last result in last
....
});