MongoDB Function clarification sync insert with auto increment - mongodb

In the following link from mongoDB documentation:
https://docs.mongodb.org/manual/tutorial/create-an-auto-incrementing-field/
there is an explanation on how to build a "safe" insert with auto-incremental _id.
My questions are:
Where should i define the function at?
How can i call it later on?
I couldn't quit understand if the function is stored as a JS function is a JS file? or is it stored in the DB? or something else.
Thanks.
Here is the actual function as it is brought in the link above:
function insertDocument(doc, targetCollection) {
while (1) {
var cursor = targetCollection.find( {}, { _id: 1 } ).sort( { _id: -1 } ).limit(1);
var seq = cursor.hasNext() ? cursor.next()._id + 1 : 1;
doc._id = seq;
var results = targetCollection.insert(doc);
if( results.hasWriteError() ) {
if( results.writeError.code == 11000 /* dup key */ )
continue;
else
print( "unexpected error inserting data: " + tojson( results ) );
}
break;
}
}

Implementation approach
Here's an example of how to implement an auto-increment field with mongoose:
var CounterSchema = Schema({
_id: {type: String, required: true},
seq: { type: Number, default: 0 }
});
var counter = mongoose.model('counter', CounterSchema);
var entitySchema = mongoose.Schema({
testvalue: {type: String}
});
entitySchema.pre('save', function(next) {
var doc = this;
counter.findByIdAndUpdate({_id: 'entityId'}, {$inc: { seq: 1} }, function(error, counter) {
if(error)
return next(error);
doc.testvalue = counter.seq;
next();
});
});
Source: https://stackoverflow.com/a/30164636/236660
Please note that {_id: 'entityId'} should be set differently per collection you're using this code with. So if you're generating an ID for entitySchema, you'll have {_id: 'entityId'}. For userSchema you would use {_id: 'userId'}, etc. You need to do this so that every collection has its own auto-incremented sequence.
The code above is thread safe. The findByIdAndUpdate operation is atomic; its consistency and concurrency are handled by MongoDB.
Existing module
Also, there is a mongoose module created specifically for handling auto-incremented IDs. You may actually be better off using it:
https://www.npmjs.com/package/mongoose-auto-increment

Related

findAndModify query not executing in callback to aggregation

I have an aggregation query on a students collection that is returning two sets of results
for each student like this
{ _id: 1543,
name: 'Bill Jackson',
scores: { type: 'homework', score: 38.86823689842918 } }
{ _id: 1543,
name: 'Bill Jackson',
scores: { type: 'homework', score: 15.861613903793295 } }
That's working fine. Now in the callback I want to remove one of the scores for each student. I use ugly nested conditionals below to isolate which of the two records I want to remove, and, once that's achieved I create a find and Modify query to remove the doc but there's no evidence of it getting run. Neither the error or success callback to the findAndModify are getting run, however I am able to log that I'm inside the area where the findAndModify is getting called.
Is it possible to query the db in the callback to an aggregation? If not, how should I perform an operation that persists in the db?
//aggregation query ommitted
, function(err, result) { //callbackstarts here with result of aggregation query that returns two records for each student
for (var i=0; i<result.length; i++) {
var id = result[i]['_id'];
if (id === result[i]['_id']){
if (foo && foo === result[i]['_id']){
//if we're in here, we know we need to remove score associated with this result[i]['_id']
//create findAndModify to remove the record
var query = { '_id' : result[i]['_id']}
var sort = []
var operation = { '$pull' : { 'scores.score' : result[i]['scores']['score'] } };
var options = []
console.log('this code is getting called but findAndModify not')
db.collection('students').findAndModify(query, sort, operation, options,function(err, doc) {
if(err) throw err;
if (!doc) {
console.log("record not found");
}
else {
console.log("changed doc" + doc);
}
});
}else {
var foo = result[i]['_id'] //part of logic to isolate which of two records to remove
}

Auto increment document number in Mongo / Mongoose

My app has several users, each user has documents. Each documents needs to have a sequence number, that may look something like this: 2013-1, 2013-2 (year and sequence number), or perhaps just a simple number: 1, 2, 3...
Currently, I am assigning the sequence number from user's settings when the Mongoose docuemnt is created. Based on that sequence number and the number format from user's settings, I am generating the final document number.
What I realized is that when 2 documents are created at the same time, they will get exactly the same number, because I am incrementing the sequence number in settings just after I have saved a document. But I am assigning the sequence number when I am creating (not saving yet) the document so the sequence number will be exactly the same for both documents.
I obviously need a way to handle this sequence number auto-incrementing at the moment of saving...
How can I assure that this number is unique and automatically incremented/generated?
#emre and #WiredPraire pointed me to the right direction, but I wanted to provide a full Mongoose-compatible answer to my question. I ended up with the following solution:
var Settings = new Schema({
nextSeqNumber: { type: Number, default: 1 }
});
var Document = new Schema({
_userId: { type: Schema.Types.ObjectId, ref: "User" },
number: { type: String }
});
// Create a compound unique index over _userId and document number
Document.index({ "_userId": 1, "number": 1 }, { unique: true });
// I make sure this is the last pre-save middleware (just in case)
Document.pre('save', function(next) {
var doc = this;
// You have to know the settings_id, for me, I store it in memory: app.current.settings.id
Settings.findByIdAndUpdate( settings_id, { $inc: { nextSeqNumber: 1 } }, function (err, settings) {
if (err) next(err);
doc.number = settings.nextSeqNumber - 1; // substract 1 because I need the 'current' sequence number, not the next
next();
});
});
Please note that with this method there is no way to require the number path in the schema, and there is no point as well, because it is automatically added.
You can achieve that through:
create sequence generator, which is just another document that keeps a counter of the last number.
Use a mongoose middleware to update the auto increment the desired field.
Here is a working and tested example with the todo app.
var mongoose = require('mongoose');
mongoose.connect('mongodb://localhost/todoApp');
// Create a sequence
function sequenceGenerator(name){
var SequenceSchema, Sequence;
SequenceSchema = new mongoose.Schema({
nextSeqNumber: { type: Number, default: 1 }
});
Sequence = mongoose.model(name + 'Seq', SequenceSchema);
return {
next: function(callback){
Sequence.find(function(err, data){
if(err){ throw(err); }
if(data.length < 1){
// create if doesn't exist create and return first
Sequence.create({}, function(err, seq){
if(err) { throw(err); }
callback(seq.nextSeqNumber);
});
} else {
// update sequence and return next
Sequence.findByIdAndUpdate(data[0]._id, { $inc: { nextSeqNumber: 1 } }, function(err, seq){
if(err) { throw(err); }
callback(seq.nextSeqNumber);
});
}
});
}
};
}
// sequence instance
var sequence = sequenceGenerator('todo');
var TodoSchema = new mongoose.Schema({
name: String,
completed: Boolean,
priority: Number,
note: { type: String, default: '' },
updated_at: { type: Date, default: Date.now }
});
TodoSchema.pre('save', function(next){
var doc = this;
// get the next sequence
sequence.next(function(nextSeq){
doc.priority = nextSeq;
next();
});
});
var Todo = mongoose.model('Todo', TodoSchema);
You can test it out in the node console as follows
function cb(err, data){ console.log(err, data); }
Todo.create({name: 'hola'}, cb);
Todo.find(cb);
With every newly created object the you will see the priority increasing. Cheers!
This code is taken from MongoDB manual and it actually describes making the _id field auto increment. However, it can be applied to any field. What you want is to check whether the inserted value exists in database just after you inserted your document. If it is allready inserted, re increment the value then try to insert again. This way you can detect dublicate values and re-increment them.
while (1) {
var cursor = targetCollection.find( {}, { f: 1 } ).sort( { f: -1 } ).limit(1);
var seq = cursor.hasNext() ? cursor.next().f + 1 : 1;
doc.f = seq;
targetCollection.insert(doc);
var err = db.getLastErrorObj();
if( err && err.code ) {
if( err.code == 11000 /* dup key */ )
continue;
else
print( "unexpected error inserting data: " + tojson( err ) );
}
break;
}
In this example f is the field in your document that you want to auto increment. To make this work you need to make your field UNIQUE which can be done with indexes.
db.myCollection.ensureIndex( { "f": 1 }, { unique: true } )
You can use mongoose-auto-increment package as follows:
var mongoose = require('mongoose');
var autoIncrement = require('mongoose-auto-increment');
/* connect to your database here */
/* define your DocumentSchema here */
autoIncrement.initialize(mongoose.connection);
DocumentSchema.plugin(autoIncrement.plugin, 'Document');
var Document = mongoose.model('Document', DocumentSchema);
You only need to initialize the autoIncrement once.

cryptic mongodb error LEFT_SUBFIELD only supports Object: stats not: 6

I'm having trouble figuring out what this error means
LEFT_SUBFIELD only supports Object: stats not: 6
It seems to be happening when I am inserting into my profiles collection. I am using mongoose.js. We are inserting counts of posts in each category in the stats property, e.g.
stats: {category:count, category2: count2}.
Here is my schema
var ProfileSchema = new Schema({
uname: {
type: String,
required: true,
index: true,
unique: true
},
fname: String,
lname: String,
stats: {
type:{},
"default":{},
required:true
},
created: {
type:Date,
required:true,
"default":Date.now
}
});
I think it might be happening when I am updating the stats object $inc counts so that stats can come out to something like this update
db.status.update({_id:xyz}, {$inc: { stats.foo : 1, stats.bar:1}})
Here's my mongoose code
var tags = ["comedy", "action", "drama"];
//also adding the postId to the posts collection of profile
var updateCommand = {$push: {posts: post._id}};
var stats = {};
for (var i = tags.length - 1; i >= 0; i--){
stats["stats." + tags[i].toString()] = 1;
};
updateCommand.$inc = stats;
Profile.update(
{uname: uname},
updateCommand,
{safe:true, upsert:true},
callback
);
It also happens if you're trying to update a subdocument of a non-object.
> db.test.insert({_id: 10240292, object: 'some string'})
> db.test.update({_id: 10240292}, {$set: {'object.subkey': 'some string'}})
LEFT_SUBFIELD only supports Object: object not: 2
Maybe it's not your case, but it can help somebody who googles for this error.
You may be running into this:
https://jira.mongodb.org/browse/SERVER-2651
or
https://jira.mongodb.org/browse/SERVER-5227
Both of which are fixed in the 2.1 dev branch already but not (yet) backported to 2.0
There is a decent discussion here about a similar issue:
https://groups.google.com/forum/?fromgroups#!topic/mongodb-user/VhjhcyEdbNQ
Basically it boils down to the fact that you are likely passing an empty key as part of the update which needs to be avoided.
db.collection('fs.files').update({_id: Object_id}, {$set: {'metadata': {"foo" : "bar"}}}

Mongoose: Populate a populated field

I'm using MongoDB as a log keeper for my app to then sync mobile clients. I have this models set up in NodeJS:
var UserArticle = new Schema({
date: { type: Number, default: Math.round((new Date()).getTime() / 1000) }, //Timestamp!
user: [{type: Schema.ObjectId, ref: "User"}],
article: [{type: Schema.ObjectId, ref: "Article"}],
place: Number,
read: Number,
starred: Number,
source: String
});
mongoose.model("UserArticle",UserArticle);
var Log = new Schema({
user: [{type: Schema.ObjectId, ref: "User"}],
action: Number, // O => Insert, 1 => Update, 2 => Delete
uarticle: [{type: Schema.ObjectId, ref: "UserArticle"}],
timestamp: { type: Number, default: Math.round((new Date()).getTime() / 1000) }
});
mongoose.model("Log",Log);
When I want to retrive the log I use the follwing code:
var log = mongoose.model('Log');
log
.where("user", req.session.user)
.desc("timestamp")
.populate("uarticle")
.populate("uarticle.article")
.run(function (err, articles) {
if (err) {
console.log(err);
res.send(500);
return;
}
res.json(articles);
As you can see, I want mongoose to populate the "uarticle" field from the Log collection and, then, I want to populate the "article" field of the UserArticle ("uarticle").
But, using this code, Mongoose only populates "uarticle" using the UserArticle Model, but not the article field inside of uarticle.
Is it possible to accomplish it using Mongoose and populate() or I should do something else?
Thank you,
From what I've checked in the documentation and from what I hear from you, this cannot be achieved, but you can populate the "uarticle.article" documents yourself in the callback function.
However I want to point out another aspect which I consider more important. You have documents in collection A which reference collection B, and in collection B's documents you have another reference to documents in collection C.
You are either doing this wrong (I'm referring to the database structure), or you should be using a relational database such as MySQL here. MongoDB's power relies in the fact you can embed more information in documents, thus having to make lesser queries (having your data in a single collection). While referencing something is ok, having a reference and then another reference doesn't seem like you're taking the full advantage of MongoDB here.
Perhaps you would like to share your situation and the database structure so we could help you out more.
You can use the mongoose-deep-populate plugin to do this. Usage:
User.find({}, function (err, users) {
User.deepPopulate(users, 'uarticle.article', function (err, users) {
// now each user document includes uarticle and each uarticle includes article
})
})
Disclaimer: I'm the author of the plugin.
I faced the same problem,but after hours of efforts i find the solution.It can be without using any external plugin:)
applicantListToExport: function (query, callback) {
this
.find(query).select({'advtId': 0})
.populate({
path: 'influId',
model: 'influencer',
select: { '_id': 1,'user':1},
populate: {
path: 'userid',
model: 'User'
}
})
.populate('campaignId',{'campaignTitle':1})
.exec(callback);
}
Mongoose v5.5.5 seems to allow populate on a populated document.
You can even provide an array of multiple fields to populate on the populated document
var batch = await mstsBatchModel.findOne({_id: req.body.batchId})
.populate({path: 'loggedInUser', select: 'fname lname', model: 'userModel'})
.populate({path: 'invoiceIdArray', model: 'invoiceModel',
populate: [
{path: 'updatedBy', select: 'fname lname', model: 'userModel'},
{path: 'createdBy', select: 'fname lname', model: 'userModel'},
{path: 'aircraftId', select: 'tailNum', model: 'aircraftModel'}
]});
how about something like:
populate_deep = function(type, instance, complete, seen)
{
if (!seen)
seen = {};
if (seen[instance._id])
{
complete();
return;
}
seen[instance._id] = true;
// use meta util to get all "references" from the schema
var refs = meta.get_references(meta.schema(type));
if (!refs)
{
complete();
return;
}
var opts = [];
for (var i=0; i<refs.length; i++)
opts.push({path: refs[i].name, model: refs[i].ref});
mongoose.model(type).populate(instance, opts, function(err,o){
utils.forEach(refs, function (ref, next) {
if (ref.is_array)
utils.forEach(o[ref.name], function (v, lnext) {
populate_deep(ref.ref_type, v, lnext, seen);
}, next);
else
populate_deep(ref.ref_type, o[ref.name], next, seen);
}, complete);
});
}
meta utils is rough... want the src?
or you can simply pass an obj to the populate as:
const myFilterObj = {};
const populateObj = {
path: "parentFileds",
populate: {
path: "childFileds",
select: "childFiledsToSelect"
},
select: "parentFiledsToSelect"
};
Model.find(myFilterObj)
.populate(populateObj).exec((err, data) => console.log(data) );

Mongoose: How to model a foreign key/inverse relationship?

I am using Mongoose to model Person and Transaction collections, where each Transaction will have references to two different Person instances:
var TransactionSchema = new Schema({
, amount : { type: Number, required: true }
, from : { type: ObjectId, required: true }
, to : { type: ObjectId, required: true }
, date : Date
});
var PersonSchema = new Schema({
name : { type: String, required: true }
, transactions : [ObjectId]
});
I'd like each Person to have a collection of all the Transactions that they are either the to or from value for. So far, this is the best way I've been able to figure out how to do it:
TransactionSchema.pre('save', function(next, done) {
var transaction = this;
Person.findById(this.to, function (err, person) {
person.transactions.push(transaction);
person.save();
});
Person.findById(this.from, function (err, person) {
person.transactions.push(transaction);
person.save();
});
next();
});
This seems excessive. Is there a better way to do it, or am I trying to use MongoDB too much like a relational database? Instead of having a collection of Transactions associated with each Person instance, should I just be querying the Translation collection directly?
Thank you.
You've got to think more on the queries you are going to execute on the database when you design the MongoDB schema.
Try to duplicate data for speed and reference it for integrity. What does that mean?
Well, for example when you make a query for a Transaction, I guess you don't need all the user details from the first time no? (do you need the user's email, location when displaying info on a Transaction?)
I think you just probably need the user id and the username, so you should do something like this:
var TransactionSchema = new Schema({
, amount : { type: Number, required: true }
, from : {
user_id: {
type: ObjectId
, required: true
}
, username: {
type: String
, required: true
}
}
, to : {
user_id: {
type: ObjectId
, required: true
}
, username: {
type: String
, required: true
}
}
, date : Date
});
So instead of doing 3 queries for the page displaying the Transaction details (one for the transaction and 2 additional queries for the usernames), you'll have just one.
This is just an example, you could apply the same logic for the User schema, depending on what you're trying to achieve.
Anyway I don't think your middleware is ok, since you are not checking for errors there (you are always calling next no matter what). This is how I would write the middleware (didn't test, but the idea is important):
TransactionSchema.pre('save', function(next, done) {
var transaction = this;
Person.where('_id').in([this.to, this.from]).run(function (err, people) {
if (people.length != 2) { next(new Error("To or from doesn't exist")); return; }
Step(
function save_to() {
people[0].transactions.push(transaction);
people[0].save(this);
},
function save_from(err) {
if (err) { next(err); return; }
people[1].transactions.push(transaction);
people[1].save(this);
},
function callback(err) {
next(err);
}
);
});
});
In the code above I'm using the Step library for flow control and I'm only using one query instead of two (when searching for "to" and "from").