Sequentially numbering MongoDB subdocuments - mongodb

Question

I have a MongoDB collection of "departments" and each department has a sub-collection of "tickets". In the old MySQL-based system, we assigned a number to each ticket (per department) by counting existing records and adding one.
This gives each department a human-readable numbering system to identify tickets: 1, 2, 3, etc.
However, in Mongo, auto increment fields like this aren't used and I'm finding that because operations may be async (using Mongoose in a NodeJS app), counting existing records may not be trustworthy (same with incrementing a counter on the department collection)
I've spent time looking for a solution but am finding it difficult to sort out unrelated topics.
Is there any trusted way to make a sequential number system that relies on a custom query?
Here's an example of the models/save code I'm toying with
var OrganizationSchema = new Schema({
name: {
type: String,
required: true
},
departments: [Department.schema]
});
var DepartmentSchema = new Schema({
name: {
type: String,
required: true
}
});
// Tickets aren't stored as subdocuments in Departments because there could be
// a lot, I didn't want it to affect performance
var TicketSchema = new Schema({
project: {
type: [{ type: Schema.Types.ObjectId, ref: 'Department' }],
required: true
},
summary: {
type: String,
required: true
}
});
// in-progress save code
Organization.findOne({ 'departments._id': department }, function(err, org){
var lastTicketId = org.departments[0].lastTicketId;
console.log(lastTicketId);
var ticket = new Ticket({
department: department,
summary: req.body.summary
});
ticket.save(function(err, result) {
if (err) {
return next(err);
}
Organization.findOneAndUpdate(
{ 'departments._id': department },
{ $inc: { 'departments.$.lastTicketId': 1 } },
function(err, result) {
console.log('saving ' + result.departments[0].lastTicketId);
}
);
});
});
This save code is an API endpoint, so I'm bulk-testing 20-some API requests from a for loop. That means they're coming in very fast and I can really see the async effect on the numbering.
The console.log output is:
loading 0
loading 0
saving 0
loading 1
loading 1
loading 1
loading 1
loading 1
loading 1
loading 1
saving 1
saving 2
loading 1
loading 1
loading 1
loading 1
loading 1
loading 1
loading 1
loading 1
loading 1
loading 1
loading 1
loading 1
loading 1
loading 1
loading 1
loading 1
loading 1
loading 1
saving 3
saving 4
saving 9
saving 5
saving 10
saving 6
saving 11
saving 7
saving 12
saving 17
saving 8
saving 13
saving 18
saving 23
saving 14
saving 19
saving 24
saving 15
saving 20
saving 25
saving 16
saving 21
saving 22

Answer 1

there are 2 possible techniques with mongoDB
a counters collection
optimistic loop
see mongoDB article here:
something that is not mentioned in this article is that counter's collection guarantees a unique incremental id in a multiprocess environment but if this id is used for insertions insertion order is not 100% guaranteed to correspond to this id.
If insertion order is critical use the Optimistic Loop technique

Answer 2

If you're using Mongoose, you could try auto-increment: https://www.npmjs.com/package/mongoose-auto-increment
var OrganizationSchema = new mongoose.Schema({
...
});
var DepartmentSchema = new mongoose.Schema({
...
});
var TicketSchema = new Schema({
ticketNumber: { type: Number, required: true },
project: {
type: [{ type: Schema.Types.ObjectId, ref: 'Department' }],
required: true}],
...
});
TicketSchema.plugin(autoIncrement.plugin, {model: 'Ticket', field: 'ticketNumber'} );
This will auto increment the ticketNumber each time you insert a ticket document. It's not incremented per department though - just a unique number per ticket.
The latest ticket number is stored in a special document collection in MongoDB mongoose-auto-increments so no problems with picking the next number even if you're in a web farmed multi server situation.
Be careful if inserting Ticket documents manually though. You would need to manually update the mongoose-auto-increments collection too.

Answer 3

In the end, the question was a little incorrect because I had assumed mongo was the issue. However, in reviewing the logs I posted, I realized that eventually the numbers were being incremented correctly, so my problem had to be a javascript issue.
I figured that querying the last ID, then saving the ticket, then updating the last ID was too inefficient, but must be allowing room for async queries to overlap.
I decided to rewrite the save logic to validate the ticket without saving it, then increment the last ID in a single query, and then save the issue with the proper value.
So far it appears to work perfectly.
// Ensure validation passes before we increment an ID
// so we avoid disconnected IDs if it fails later
ticket.validate(function(err) {
if (err) {
return next(err);
}
// Find and update last ID in one transaction
// due to async issues
Organization.findOneAndUpdate(
{ 'departments._id': department },
{ $inc: { 'departments.$.lastTicketId': 1 } },
function(err, result) {
if (err) {
return next(err);
}
ticket.displayId = result.departments[0].lastTicketId;
ticket.save(function(err, user) {
if (err) {
return next(err);
}
// success
});
}
);
});