Mongoose: consistently save document order - mongodb

Suppose we have a schema like this:
const PageSchema = new mongoose.Schema({
content: String
order: Number
})
We want order to be always a unique number between 0 and n-1, where n is the total number of documents.
How can we ensure this when documents are inserted or deleted?
For inserts I currently use this hook:
PageSchema.pre('save', async function () {
if (!this.order) {
const lastPage = await this.constructor.findOne().sort({ order: -1 })
this.order = lastPage ? lastPage.order + 1 : 0
}
})
This seems to work when new documents are inserted.
When documents are removed, I would have to decrease the order of documents of higher order. However, I am not sure which hooks are called when documents are removed.
Efficiency is not an issue for me: there are not many inserts and deletes.
It would be totally ok if I could somehow just provide one function, say fix_order, that iterates over the whole collection. How can I install this function such that it gets called whenever documents are inserted or deleted?

You can use findOneAndDelete pre and post hooks to accomplish this.
As you see in the pre findOneAndDelete hook, we save a reference to the deleted document, and pass it to the postfindOneAndDelete, so that we can access the model using constructor, and use the updateMany method to be able to adjust orders.
PageSchema.pre("findOneAndDelete", async function(next) {
this.page = await this.findOne();
next();
});
PageSchema.post("findOneAndDelete", async function(doc, next) {
console.log(doc);
const result = await this.page.constructor.updateMany(
{ order: { $gt: doc.order } },
{
$inc: {
order: -1
}
}
);
console.log(result);
next();
});
Let's say you have these 3 documents:
[
{
"_id": ObjectId("5e830a6d0dec1443e82ad281"),
"content": "content1",
"order": 0,
"__v": 0
},
{
"_id": ObjectId("5e830a6d0dec1443e82ad282"),
"content": "content2",
"order": 1,
"__v": 0
},
{
"_id": ObjectId("5e830a6d0dec1443e82ad283"),
"content": "content3",
"order": 2,
"__v": 0
}
]
When you delete the content2 with "_id": ObjectId("5e830a6d0dec1443e82ad282") with findOneAndDelete method like this:
router.delete("/pages/:id", async (req, res) => {
const result = await Page.findOneAndDelete({ _id: req.params.id });
res.send(result);
});
The middlewares will run, and adjust the orders, the remaining 2 documents will look like this:
[
{
"_id": ObjectId("5e830a6d0dec1443e82ad281"),
"content": "content1",
"order": 0,
"__v": 0
},
{
"_id": ObjectId("5e830a6d0dec1443e82ad283"),
"content": "content3",
"order": 1, => DECREASED FROM 2 to 1
"__v": 0
}
]
Also you had better to include next in your pre save middleware so that other middlewares also work if you add later.
PageSchema.pre("save", async function(next) {
if (!this.order) {
const lastPage = await this.constructor.findOne().sort({ order: -1 });
this.order = lastPage ? lastPage.order + 1 : 0;
}
next();
});

Based on the answer of SuleymanSah, I wrote a mongoose plugin that does the job. This way, it can be applied to multiple schemas without unnessecary code duplication.
It has two optional arguments:
path: pathname where the ordinal number is to be stored (defaults to order)
scope: pathname or array of pathnames relative to which numbers should be given (defaults to [])
Example. Chapters should not be numbered globally, but relative to the book to which they belong:
ChapterSchema.plugin(orderPlugin, { path: 'chapterNumber', scope: 'book' })
File orderPlugin.js:
function getConditions(doc, scope) {
return Object.fromEntries([].concat(scope).map((path) => [path, doc[path]]))
}
export default (schema, options) => {
const path = (options && options.path) || 'order'
const scope = (options && options.scope) || {}
schema.add({
[path]: Number,
})
schema.pre('save', async function () {
if (!this[path]) {
const last = await this.constructor
.findOne(getConditions(this, scope))
.sort({ [path]: -1 })
this[path] = last ? last[path] + 1 : 0
}
})
schema.post('findOneAndDelete', async function (doc) {
await this.model.updateMany(
{ [path]: { $gt: doc[path] }, ...getConditions(doc, scope) },
{ $inc: { [path]: -1 } }
)
})
}

Related

How to know mongodb transaction all CRUD success?

execTransaction(session => {
return async () => {
await db.collection('table1').updateOne({ username: username }, { $push: { apps: client_id } }, { session });
await db.collection('table2').insertOne({ data: data }, { session });
};
}, data => {
console.log(data); //data.result.ok = 1
});
I use withTransaction in nodejs to run transaction.
When I run the transaction correctly, data.result.ok = 1
But when I use no exist username run updateOne, the result is still 1.
How to verify the result?
insertOne should return an Object like this if successful
{
acknowledged: true,
insertId: <object OjectId> // or the _id you specified
}
updateOne should return an Object like this if successful
{
acknowledged: true,
modifiedCount: 1, // or 0 if upserted
upsertedId: null, // or an <object OjectId>
upsertedCount: 0, // or 1 if upserted
matchedCount: 1 // or 0 if upserted
}
Now, updateOne only updates the first document that matched the query. If no document matched the query, it would simply update no document and throw no errors. You would get an object like this if that happens,
{
acknowledged: true,
modifiedCount: 0,
upsertedId: null,
upsertedCount: 0,
matchedCount: 0
}
So, checking data.result.ok would tell you nothing in that scenario. You have to manually check each method call if you want to something like that. In your case, maybe do something like this?
execTransaction(session => {
return async () => {
await Promise.all([
db.collection('table1').updateOne({ username: username }, { $push: { apps: client_id } }, { session }),
db.collection('table2').insertOne({ data: data }, { session })
])
};
}, data => {
if (data[0].matchedCount === 0) {
// do error handling ?
}
}
)
Or, if you want to know if all the CRUD operations succeeded as the title suggests, maybe this?
(await Promise.all([a list of all insertOne and updateOne operations... ])
.every(obj => obj && obj.acknowledged && (obj.insertId || obj.modifiedCount || obj.upsertedCount))
// this checks if all the properties are non zero
// thus if It's true, everything succeed
But again I would suggest you to do error handling separately for each operation when needed.

Mongoose update only fields available in request body

I am trying to update one document using findOneAndUpdate and $set but I clearly missing something very crucial here because the new request is overwriting old values.
My Device schema looks like this:
{
deviceId: {
type: String,
immutable: true,
required: true,
},
version: {
type: String,
required: true,
},
deviceStatus: {
sensors: [
{
sensorId: {
type: String,
enum: ['value1', 'value2', 'value3'],
},
status: { type: Number, min: -1, max: 2 },
},
],
},
}
And I am trying to update the document using this piece of code:
const deviceId = req.params.deviceId;
Device.findOneAndUpdate(
{ deviceId },
{ $set: req.body },
{},
(err, docs) => {
if (err) {
res.send(err);
} else {
res.send({ success: true });
}
}
);
And when I try to send a request from the postman with the body that contains one or multiple sensors, only the last request is saved in the database.
{
"deviceStatus": {
"sensors": [
{
"sensorId": "test",
"status": 1
}
]
}
}
I would like to be able to update values that are already in the database based on req.body or add new ones if needed. Any help will be appreciated.
The documentation said:
The $set operator replaces the value of a field with the specified
value.
You need the $push operator, it appends a specified value to an array.
Having this documents:
[
{
_id: 1,
"array": [
2,
4,
6
]
},
{
_id: 2,
"array": [
1,
3,
5
]
}
]
Using $set operator:
db.collection.update({
_id: 1
},
{
$set: {
array: 10
}
})
Result:
{
"_id": 1,
"array": 10
}
Using $push operator:
db.collection.update({
_id: 1
},
{
$push: {
array: 10
}
})
Result:
{
"_id": 1,
"array": [
2,
4,
6,
10
]
}
you want to using $push and $set in one findOneAndUpdate, that's impossible, I prefer use findById() and process and save() ,so just try
let result = await Device.findById(deviceId )
//implementation business logic on result
await result.save()
If you want to push new sensors every time you make request then update your code as shown below:
const deviceId = req.params.deviceId;
Device.findOneAndUpdate(
{ deviceId },
{
$push: {
"deviceStatus.sensors": { $each: req.body.sensors }
}
},
{},
(err, docs) => {
if (err) {
res.send(err);
} else {
res.send({ success: true });
}
}
);
Update to the old answer:
If you want to update sensors every time you make request then update your code as shown below:
const deviceId = req.params.deviceId;
Device.findOneAndUpdate(
{ "deviceId": deviceId },
{ "deviceStatus": req.body.sensors },
{ upsert: true },
(err, docs) => {
if (err) {
res.send(err);
} else {
res.send({ success: true });
}
}
);

Mongoose populate method not populating all fields after updating document

I change the order of an array and then populate doesn't work. It still works on the documents that I did not try to update.
I have a seccion model with preguntas inside of it
SECCION MODEL
const Schema = mongoose.Schema;
const seccionSchema = new Schema({
titulo: { type: String },
preguntas: [
{
type: Schema.Types.ObjectId,
ref: 'Pregunta',
},
],
});
PREGUNTA MODEL
const Schema = mongoose.Schema;
const preguntaSchema = new Schema({
titulo: { type: String, required: true },
});
A "Seccion" document with 2 "preguntas" looks something like this:
{
"preguntas": [
"5fb0a48ccb68c44c227a0432",
"5fb0a48ccb68c44c227a0433"
],
"_id": "5fb0a50fcb68c44c227a0436",
"titulo": "Seccion 2",
"__v": 3
}
What I want to do is change the order in the preguntas array, so that the _ids end up being [33, 32] instead of [32, 33] (last two digits of the _ids). I can update using ".save()" method or ".findOneAndUpdate()". Here is the code using "findOneAndUpdate":
router.put('/:id', async (req, res) => {
const seccionId = req.params.id;
try {
const seccion = await Seccion.findOneAndUpdate(
{ _id: seccionId },
{ preguntas: ['5fb0a48ccb68c44c227a0433', '5fb0a48ccb68c44c227a0432'] },
{ new: true }
);
res.json(seccion);
} catch (err) {
res.json({ message: err });
}
});
The result works (and the order is changed in mongo atlas):
{
"preguntas": [
"5fb0a48ccb68c44c227a0433",
"5fb0a48ccb68c44c227a0432"
],
"_id": "5fb0a50fcb68c44c227a0436",
"titulo": "Seccion 2",
"__v": 4
}
But when I try to use the "populate()" method of mongoose, only one of the "preguntas ids" or none are populated (below only the "pregunta" with id ending in 32 was populated):
{
"preguntas": [
{
"_id": "5fb0a48ccb68c44c227a0432",
"titulo": "Pregunta 3",
"__v": 0
}
],
"_id": "5fb0a50fcb68c44c227a0436",
"titulo": "Seccion 2",
"__v": 4
}
So my guess is that the populate method is not working correctly, but it does work on the other "Seccion" documents where I have not tryed to change the order of the "preguntas" inside the array. Here is the code that uses the "populate" method:
router.get('/:id', async (req, res) => {
const seccionId = req.params.id;
try {
const seccion = await Seccion.findById(seccionId).populate('preguntas');
res.json(seccion);
} catch (err) {
res.json({ message: err });
}
});
It has been really hard to make the populate work, I really appreciate anyone's help
Are you sure that pregunta of id 5fb0a48ccb68c44c227a0433 exists in the preguntas collection?
The way populate works is that it does a second find operation on the preguntas collection with the ids in the array, if a document is not found, it will be omitted from the array.
The issue is probably that the one ending with 33 does not exist in the preguntas collection.

How do I count all the documents in a collection and use the cont in a controller, with MongoDB and Express.js?

I am working on a blogging application (click the link to see the GitHub repo) with Express (version 4.17.1), EJS and MongoDB (version 4.0.10).
Trying to paginate the posts I did the following, in the controller:
exports.getPosts = (req, res, next) => {
const perPage = 5;
const currPage = req.query.page ? parseInt(req.query.page) : 1;
let postsCount = 0;
const posts = Post.find({}, (err, posts) => {
postsCount = posts.length;
let pageDecrement = currPage > 1 ? 1 : 0;
let pageIncrement = postsCount >= perPage ? 1 : 0;
if (err) {
console.log('Error: ', err);
} else {
res.render('default/index', {
moment: moment,
layout: 'default/layout',
website_name: 'MEAN Blog',
page_heading: 'XPress News',
page_subheading: 'A MEAN Stack Blogging Application',
currPage: currPage,
posts: posts,
pageDecrement: pageDecrement,
pageIncrement: pageIncrement
});
}
})
.sort({
created_at: -1
})
.populate('category')
.limit(perPage)
.skip((currPage - 1) * perPage);
};
And in the view:
<a class="btn btn-primary <%= pageDecrement == 0 ? 'disabled' : '' %>" href="/?page=<%= currPage - pageDecrement %>">← Newer Posts</a>
and
<a class="btn btn-primary <%= pageIncrement == 0 ? 'disabled' : '' %>" href="/?page=<%= currPage + pageIncrement %>">Older Posts →</a>
That works fine unless there are is a number of posts equal to perPage x N, where N is an integer, in which case the "Older Posts" button becomes disabled one page too late.
That is because postsCount = posts.length counts the posts after they are limited by .skip((currPage - 1) * perPage).
So I need to count the posts from the model/collection and bring that count variable in the controller.
My model:
const mongoose = require('mongoose');
const postSchema = new mongoose.Schema({
title: {
type: String,
required: true
},
short_description: {
type: String,
required: true
},
full_text: {
type: String,
required: true
},
category: {
type: mongoose.Schema.Types.ObjectId,
ref: 'Category'
},
post_image: {
type: String,
required: false
},
updated_at: {
type: Date,
default: Date.now()
},
created_at: {
type: Date,
default: Date.now()
}
});
module.exports = mongoose.model('Post', postSchema);
How do I count all the documents in the posts collection and use that number in the posts controller?
This can be done easier with mongodb aggregation framework.
We use $facet aggregation to get the paginated data along with the total number of documents.
In aggregation framework we use $lookup instead of mongoose populate. $lookup returns an array, to get the first item in array we use $arrayElemAt operator inside $addFields.
Playground
And here is the code to apply to your app:
(The first $match aggregation is unnecessary here, but I put in in case you may need it in the future)
exports.getPosts = async (req, res, next) => {
const perPage = 5;
const currPage = req.query.page ? parseInt(req.query.page) : 1;
const skip = (currPage - 1) * perPage;
try {
const result = await Post.aggregate([{
$match: {},
},
{
$sort: {
created_at: -1,
},
},
{
$lookup: {
from: "categories",
localField: "category",
foreignField: "_id",
as: "category",
},
},
{
$addFields: {
category: {
$arrayElemAt: ["$category", 0],
},
},
},
{
$facet: {
totalRecords: [{
$count: "total",
}, ],
data: [{
$skip: skip,
},
{
$limit: perPage,
},
],
},
},
]);
let postsCount = result[0].totalRecords[0].total;
const pageCount = Math.ceil(postsCount / perPage);
const pageDecrement = currPage > 1 ? 1 : 0;
const pageIncrement = currPage < pageCount ? 1 : 0;
const posts = result[0].data;
res.render("default/index", {
moment: moment,
layout: "default/layout",
website_name: "MEAN Blog",
page_heading: "XPress News",
page_subheading: "A MEAN Stack Blogging Application",
currPage,
posts,
pageDecrement,
pageIncrement,
});
} catch (err) {
console.log("Error: ", err);
res.status(500).send("something went wrong");
}
};
By the way, in the post schema, for date fields you use default: Date.now(), this will cause the date value always the same value, it should be in this format: default: Date.now
Read $facet.
New in version 3.4.
Processes multiple aggregation pipelines within a single stage on the
same set of input documents. Each sub-pipeline has its own field in
the output document where its results are stored as an array of
documents.
Example: See here
db.collection.aggregate([
{
$facet: {
"count": [
{ $match: {} },
{ $count: "totalCount" }
],
"data": [
{ $match: {} },
{ $sort: { _id: -1 } },
{ $skip: 1 },
{ $limit: 2 }
]
}
}
])
Mongoose Version:
Model.aggregate([
{
$facet: {
"count": [
{ $match: {} },
{ $count: "totalCount" }
],
"data": [
{ $match: {} },
{ $sort: { _id: -1 } },
{ $skip: 1 },
{ $limit: 2 }
]
}
}
]).
then(res => console.log(res)).
catch(error => console.error('error', error));
In case of Mongoose you should use this:
https://mongoosejs.com/docs/api.html#aggregate_Aggregate-facet
Official Mongodb docs:
https://docs.mongodb.com/manual/reference/operator/aggregation/facet
General idea is to perform aggregation instead of multiple calls (1 for getting needed info + 1 to get the total count of documents)
You can perform 2 separate calls of course but it will hit your performance (not much for small data volumes but still...)
So you can get all needed data with .find() and then get count like this:
https://mongoosejs.com/docs/api.html#model_Model.count
PS. btw, use async/await instead of callbacks to avoid callback hell

Can I access the positional $ operator in projection of findOneAndUpdate

I have this query that works, but I want for the doc to only display network.stations.$ instead of the entire array. If I write fields: network.stations.$, I get an error. Is there a way for the doc only to return a single element from [stations]?
Network.findOneAndUpdate({
"network.stations.id": req.params.station_Id
}, {
"network.stations.$.free_bikes": req.body.free_bikes
}, {
new: true,
fields: "network.stations"
}, (err, doc) => console.log(doc))
// I want doc to somehow point only to a single station instead of
// several stations like it currently does.
The answer is "yes", but not in the way you are expecting. As you note in the question, putting network.stations.$ in the "fields" option to positionally return the "modified" document throws a specific error:
"cannot use a positional projection and return the new document"
This however should be the "hint", because you don't really "need" the "new document" when you know what the value was you are modifying. The simple case then is to not return the "new" document, but instead return it's "found state" which was "before the atomic modification" and simply make the same modification to the returned data as you asked to apply in the statement.
As a small contained demo:
const mongoose = require('mongoose'),
Schema = mongoose.Schema;
mongoose.Promise = global.Promise;
mongoose.set('debug',true);
const uri = 'mongodb://localhost/test',
options = { useMongoClient: true };
const testSchema = new Schema({},{ strict: false });
const Test = mongoose.model('Test', testSchema, 'collection');
function log(data) {
console.log(JSON.stringify(data,undefined,2))
}
(async function() {
try {
const conn = await mongoose.connect(uri,options);
await Test.remove();
await Test.insertMany([{ a: [{ b: 1 }, { b: 2 }] }]);
for ( let i of [1,2] ) {
let result = await Test.findOneAndUpdate(
{ "a.b": { "$gte": 2 } },
{ "$inc": { "a.$.b": 1 } },
{ "fields": { "a.$": 1 } }
).lean();
console.log('returned');
log(result);
result.a[0].b = result.a[0].b + 1;
console.log('modified');
log(result);
}
} catch(e) {
console.error(e)
} finally {
mongoose.disconnect()
}
})();
Which produces:
Mongoose: collection.remove({}, {})
Mongoose: collection.insertMany([ { __v: 0, a: [ { b: 1 }, { b: 2 } ], _id: 59af214b6fb3533d274928c9 } ])
Mongoose: collection.findAndModify({ 'a.b': { '$gte': 2 } }, [], { '$inc': { 'a.$.b': 1 } }, { new: false, upsert: false, fields: { 'a.$': 1 } })
returned
{
"_id": "59af214b6fb3533d274928c9",
"a": [
{
"b": 2
}
]
}
modified
{
"_id": "59af214b6fb3533d274928c9",
"a": [
{
"b": 3
}
]
}
Mongoose: collection.findAndModify({ 'a.b': { '$gte': 2 } }, [], { '$inc': { 'a.$.b': 1 } }, { new: false, upsert: false, fields: { 'a.$': 1 } })
returned
{
"_id": "59af214b6fb3533d274928c9",
"a": [
{
"b": 3
}
]
}
modified
{
"_id": "59af214b6fb3533d274928c9",
"a": [
{
"b": 4
}
]
}
So I'm doing the modifications in a loop so you can see that the update is actually applied on the server as the next iteration increments the already incremented value.
Merely by omitting the "new" option, what you get is the document in the state which it was "matched" and it then is perfectly valid to return that document state before modification. The modification still happens.
All you need to do here is in turn make the same modification in code. Adding .lean() makes this simple, and again it's perfectly valid since you "know what you asked the server to do".
This is better than a separate query because "separately" the document can be modified by a different update in between your modification and the query to return just a projected matched field.
And it's better than returning "all" the elements and filtering later, because the potential could be a "very large array" when all you really want is the "matched element". Which of course this actually does.
Try changing fields to projection and then use the network.stations.$ like you tried before.
If your query is otherwise working then that might be enough. If it's still not working you can try changing the second argument to explicitly $set.
Network.findOneAndUpdate({
"network.stations.id": req.params.station_Id
}, {
"$set": {
"network.stations.$.free_bikes": req.body.free_bikes
}
}, {
new: true,
projection: "network.stations.$"
}, (err, doc) => console.log(doc))