Mongoose ('findOneAndUpdate') Middleware: Need access to original document - mongodb

I am attempting to use pre('findOneAndUpdate') to update the icon attribute of the Meeting document. The update is based on the pre-existing value of the yearlymeeting attribute (see below).
Because pre and post save() hooks are not executed on update(), I seem to be unable to access the original document at all. Yet this is critical for the operation I'm trying to perform. Is there any way around this?
For example, I am able to accomplish my purpose on pre('save'), like so:
meetingSchema.pre('save', function(next) {
const yearlymeetingSlug = this.yearlymeeting[0].toLowerCase().replace(/[^A-z0-9]/g, '');
this.icon = `${yearlymeetingSlug}.png`
next();
});
What I would like to be able to do is something like this:
meetingSchema.pre('findOneAndUpdate', function(next) {
const yearlymeetingSlug = originalDocument.yearlymeeting[0].toLowerCase().replace(/[^A-z0-9]/g, '');
this.icon = `${yearlymeetingSlug}.png`
next();
});
I understand that this in pre(findOneAndUpdate) refers to the query, rather than the stored document itself. Is there any way to access the document, so that I can update icon based on the stored value of yearlymeeting?

tl;dr
Not possible via middleware. Query for the doc first, and then separately update a specific version of the doc to prevent race conditions.
Can't do it the way you're trying according to this issue on the Mongoose Github (from the main dev):
By design - the document being updated might not even be in the server's memory. In order to do that, mongoose would have to do a findOne() to load the document before doing the update(), which is not acceptable.
The design is to enable you to manipulate the query object by adding or removing filters, update params, options, etc. For instance, automatically calling .populate() with find() and findOne(), setting the multi: true option by default on certain models, access control, and other possibilities.
findOneAndUpdate() is a bit of a misnomer, it uses the underlying mongodb findAndModify command, it's not the same as findOne() + update(). As a separate operation, it should have its own middleware.
Following this, there are no other suggestions in the issue thread to access the original document inside of the middleware itself.
What I've seen done (and what I've had to do many times myself), is simply have to query for the document before updating it (which, of course, could lead to a race condition depending on who is updating the doc, and when, but you can fix that by also querying for a specific version of the document -- a sort of "optimistic locking"):
let meeting = yield Meeting.findOne({}).exec()
let update = {}
// ... some conditional logic to figure out which icon to set
update.icon = // whatever
yield Meeting.update({ _id: meeting._id, version: meeting.version }, update)
This is of course assuming you have a "version" field in your schema. This sort of locking will prevent you from updating an old version of the doc. If you're gonna use this kind of versioning, you'll also probably want to add some middleware that updates the version of a doc any time the doc is updated/saved.
You can also use a more naïve implementation, where you don't use locking, which may be fine in your specific business case, as long as you're aware of the possibility of a race condition, and the risks.

This may not be the best solution, but I did find a way to make it work. I used the controller rather than schema pre hooks. Here's what my update controller looks like now:
exports.updateMeeting = async (req, res) => {
const _id = req.params.id
let meeting = await Meeting.findOneAndUpdate({ _id }, req.body, {
new: true,
runValidators: true
});
/* New Code: */
const yearlymeetingSlug = meeting.yearlymeeting[0].toLowerCase().replace(/[^A-z0-9]/g, '');
meeting.icon = `${yearlymeetingSlug}.png`;
meeting.save();
req.flash('success', 'meeting successfully updated!');
res.redirect(`/meetings/${meeting.slug}`);
};
I welcome your feedback on any problems you see with this solution.

Related

How can I "cache" a mongoDB/Mongoose result to be used in my Express.js views and routes

What I'm trying to achieve is some sort of way to cache results of a mongoDB/Mongoose query that I can use in my views and routes. I'd need to be able to update this cache whenever a new document is added to the collection. I'm not sure if this is possible and if it is then how to do it, due to how the functions are asynchronous
This is currently what I have for storing the galleries, however this is executed with every request.
app.use(function(req, res, next) {
Gallery.find(function(err, galleries) {
if (err) throw err;
res.locals.navGalleries = galleries;
next();
});
});
This is used to get gallery names, which are then displayed in the navigation bar from a dynamically generated gallery. The gallery model is setup with just a name of the gallery and a slug
and this is part of my EJS view inside of my navigation which stores the values in a dropdown menu.
<% navGalleries.forEach(function(gallery) { %>
<li>
<a href='/media/<%= gallery.slug %>'><%= gallery.name %></a>
</li>
<% }) %>
The website I'm working on is expected to get hundreds of thousands of concurrent users, so I don't want to have to query the database for every single request if not needed, and just update it whenever a new gallery is created.
Take a look at cachegoose. It will allow you to cache any query you want and invalidate that cache entry each time a new gallery is created.
You will need something like this:
const mongoose = require('mongoose');
const cachegoose = require('cachegoose');
cachegoose(mongoose); // You can specify some options here to use Redis instead of in-memory cache
app.get(function(req, res, next) {
...
Gallery
.find()
.cache(0, 'GALLERY-CACHE-KEY')
.exec(function(err, galleries) {
if (err) throw err;
res.locals.navGalleries = galleries;
next();
});
...
});
app.post(function(req, res, next) {
...
new Gallery(req.body).save(function (err) {
if (err) throw err;
// Invalidate the cache as new data has been added:
cachegoose.clearCache('GALLERY-CACHE-KEY');
});
...
});
Although you could do something simpler caching the results manually in a variable and invalidating that cache when new galleries are added, I would advise you to take a look at that package instead.
I've created modern library for handling cache clearing automatically. In short it's more advanced and faster than cachegoose, because it's also caching Mongoose Documents instances in memory.
Speedgoose has autoclearing plugin, which you can set on a given schema, So it's not only ttl-based. It works on mongoose document events to clear results related to the document. So if the result of query XYZ is cached, and it was containing the record that was edited/removed, it will be removed from the cache. Same with adding and removing documents from schema. Those events might affect every cached result in the collection. So they are clearing the cache in the scope of a given model. I should be rather enough in 90% of cases.
In near future it will have many more features. And it's pretty easy to setup!
https://www.npmjs.com/package/speedgoose
Setup
import {applySpeedGooseCacheLayer} from "speedgoose";
import mongoose from "mongoose";
applySpeedGooseCacheLayer(mongoose, {
redisUri: process.env.REDIS_URI
})
Cache something!
//Caching both - query result, and instances of related Mongoose Documents in memory. It supports sort, select,project...etc.
model.find({}).sort({fieldA : 1}).cacheQuery()
//Caching only result
model.find({}).lean().cacheQuery()
//Works also with aggregation
model.aggregate([]).cachePipeline()

Meteor.subscribe on server side

I want to create a backend service which monitors a mongodb collection for new entries. As those are being created, I wish to run processing and update them.
I thought doing so with a Meteor service/app would be a wise idea because Meteor uses 'oplog tailing' which seems ideal for this purpose (I'd rather avoid polling if possible).
As such, I figured creating a minimal server-side-only app should solve it.
So basically, I need something along these lines:
if (Meteor.isServer) {
MyCollection = new Mongo.Collection('myCollection');
Meteor.publish('myCollectionPub', function () {
return MyCollection.find({ some: criteria... });
}
// is there such a thing?
Meteor.serverSideSubscribe('MyCollectionPub',
function (newDocs) {
// process/update newDocs
});
}
According to the Meteor docs, I cannot use Meteor.subscribe() on the server (and indeed it crashes if I try).
Question is:
Are there ways of 'subscribing' to collection updates on the server?
The PeerLibrary server-autorun package (along with it's dependant, reactive-mongo) will provide you with easy server-side observation of collections.
An alternative to #tarmes suggestion is the collection-hooks package, however as pointed out by David Weldon, it will only trigger in instance it is run in:
https://github.com/matb33/meteor-collection-hooks
MyCollection.after.insert(function (userId, doc) {
// ...
});
If you need it to run even when another instance makes a change in the mongo database, you can observe a cursor that is returned from your collection:
MyCollection.find({created_at : {$gt: some_current_time}}).observe({
added: function(item) {
// Alert code
}
});

Add single record to mongo collection with meteor

I am a new user to JavaScript and the meteor framework trying to understand the basic concepts. First of all I want to add a single document to a collection without duplicate entries.
this.addRole = function(roleName){
console.log(MongoRoles.find({name: roleName}).count());
if(!MongoRoles.find({name: roleName}).count())
MongoRoles.insert({name: roleName});
}
This code is called on the server as well as on the client. The log message on the client tells me there are no entries in the collection. Even if I refresh the page several times.
On the server duplicate entries get entered into the collection. I don't know why. Probably I did not understand the key concept. Could someone point it out to me, please?
Edit-1:
No, autopublish and insecure are not installed anymore. But I already published the MongoRoles collection (server side) and subscribed to it (client side). Furthermore I created a allow rule for inserts (client side).
Edit-2:
Thanks a lot for showing me the meteor method way but I want to get the point doing it without server side only methods involved. Let us say for academic purposes. ;-)
Just wrote a small example:
Client:
Posts = new Mongo.Collection("posts");
Posts.insert({title: "title-1"});
console.log(Posts.find().count());
Server:
Posts = new Mongo.Collection("posts");
Meteor.publish(null, function () {
return Posts.find()
})
Posts.allow({
insert: function(){return true}
})
If I check the server database via 'meteor mongo' it tells me every insert of my client code is saved there.
The log on the client tells me '1 count' every time I refresh the page. But I expected both the same. What am I doing wrong?
Edit-3:
I am back on my original role example (sorry for that). Just thought I got the point but I am still clueless. If I check the variable 'roleCount', 0 is responded all the time. How can I load the correct value into my variable? What is the best way to check if a document exists before the insertion into a collection? Guess the .find() is asynchronous as well? If so, how can I do it synchronous? If I got it right I have to wait for the value (synchronous) because I really relay on it.
Shared environment (client and server):
Roles = new Mongo.Collection("jaqua_roles");
Roles.allow({
insert: function(){return true}
})
var Role = function(){
this.addRole = function(roleName){
var roleCount = Roles.find({name: roleName}).count();
console.log(roleCount);
if(roleCount === 0){
Roles.insert({name: roleName}, function(error, result){
try{
console.log("Success: " + result);
var roleCount = Roles.find({name: roleName}).count();
console.log(roleCount);
} catch(error){
}
});
}
};
this.deleteRole = function(){
};
}
role = new Role();
role.addRole('test-role');
Server only:
Meteor.publish(null, function () {
return Roles.find()
})
Meteor's insert/update/remove methods (client-side) are not a great idea to use. Too many potential security pitfalls, and it takes a lot of thought and time to really patch up any holes. Further reading here.
I'm also wondering where you're calling addRole from. Assuming it's being triggered from client-side only, I would do this:
Client-side Code:
this.addRole = function(roleName){
var roleCount = MongoRoles.find({name: roleName}).count();
console.log(roleCount);
if (roleCount === 0) {
Meteor.call('insertRole', roleName, function (error, result) {
if (error) {
// check error.error and error.reason (if I'm remembering right)
} else {
// Success!
}
});
}
}
How I've modified this code and why:
I made a roleCount variable so that you can avoid calling MongoRoles.find() twice like that, which is inefficient and consumes unneeded resources (CPU, disk I/O, etc). Store it once, then reference the variable instead, much better.
When checking numbers, try to avoid doing things like if (!count). Using if (count === 0) is clearer, and shows that you're referencing a number. Statements like if (!xyz) would make one think this is a boolean (true/false) value.
Always use === in JavaScript, unless you want to intentionally do a loose equality operation. Read more on this.
Always use open/closed curly braces for if and other blocks, even if it contains just a single line of code. This is just good practice so that if you decide to add another line later, you don't have to then wrap it in braces. Just a good practice thing.
Changed your database insert into a Meteor method (see below).
Side note: I've used JavaScript (ES5), but since you're new to JavaScript, I think you should jump right into ES6. ES is short for ECMAScript (which is what JS is based on). ES6 (or ECMAScript 2015) is the most recent stable version which includes all kinds of new awesomeness that JavaScript didn't previously have.
Server-side Code:
Meteor.method('insertRole', function (roleName) {
check(roleName, String);
try {
// Any security checks, such as logged-in user, validating roleName, etc
MongoRoles.insert({name: roleName});
} catch (error) {
// error handling. just throw an error from here and handle it on client
if (badThing) {
throw new Meteor.Error('bad-thing', 'A bad thing happened.');
}
}
});
Hope this helps. This is all off the top of my head with no testing at all. But it should give you a better idea of an improved structure when it comes to database operations.
Addressing your edits
Your code looks good, except a couple issues:
You're defining Posts twice, don't do that. Make a file, for example, /lib/collections/posts.js and put the declaration and instantiation of Mongo.Collection in there. Then it will be executed on both client and server.
Your console.log would probably return an error, or zero, because Posts.insert is asynchronous on the client side. Try the below instead:
.
Posts.insert({title: "title-1"}, function (error, result) {
console.log(Posts.find().count());
});

Navigation Property Filter

My question is this: How can you implement a default server-side "filter" for a navigation property?
In our application we seldom actually delete anything from the database. Instead, we implement "soft deletes" where each table has a Deleted bit column. If this column is true the record has been "deleted". If it is false, it has not.
This allows us to easily "undelete" records accidentally deleted by the client.
Our current ASP.NET Web API returns only "undeleted" records by default, unless a deleted argument is sent as true from the client. The idea is that the consumer of the service doesn't have to worry about specifying that they only want undeleted items.
Implementing this same functionality in Breeze is quite simple, at least for base entities. For example, here would be the implementation of the classic Todo's example, adding a "Deleted" bit field:
// Note: Will show only undeleted items by default unless you explicitly pass deleted = true.
[HttpGet]
public IQueryable<BreezeSampleTodoItem> Todos(bool deleted = false) {
return _contextProvider.Context.Todos.Where(td => td.Deleted == deleted);
}
On the client, all we need to do is...
var query = breeze.EntityQuery.from("Todos");
...to get all undeleted Todos, or...
var query = breeze.EntityQuery.from("Todos").withParameters({deleted: true})
...to get all deleted Todos.
But let's say that a BreezeSampleTodoItem has a child collection for the tools that are needed to complete that Todo. We'll call this "Tools". Tools also implements soft deletes. When we perform a query that uses expand to get a Todo with its Tools, it will return all Tools - "deleted" or not.
But how can I filter out these records by default when Todo.Tools is expanded?
It has occurred to me to have separate Web API methods for each item that may need expanded, for example:
[HttpGet]
public IQueryable<Todo> TodoAndTools(bool deletedTodos = false, bool deletedTools = false)
{
return // ...Code to get filtered Todos with filtered Tools
}
I found some example code of how to do this in another SO post, but it requires hand-coding each property of Todo. The code from the above-mentioned post also returns a List, not an IQueryable. Furthermore this requires methods to be added for every possible expansion which isn't cool.
Essentially what I'm looking for is some way to define a piece of code that gets called whenever Todos is queried, and another for whenever Tools is queried - preferably being able to pass an argument that defines if it should return Deleted items. This could be anywhere on the server-side stack - be it in the Web API method, itself, or maybe part of Entity Framework (note that filtering Include extensions is not supported in EF.)
Breeze cannot do exactly what you are asking for right now, although we have discussed the idea of allowing the filtering of "expands", but we really need more feedback as to whether the community would find this useful. Please add this to the breeze User Voice and vote for it. We take these suggestions very seriously.
Moreover, as you point out, EF does not support this.
But... what you can do is use a projection instead of an expand to do something very similar:
public IQueryable<Object> TodoAndTools(bool deleted = false
,bool deletedTools = false) {
var baseQuery = _contextProvider.Context.Todos.Where(td => td.Deleted == deleted);
return baseQuery.Select(t => new {
Todo: t,
Tools: t.Tools.Where( tool => tool.Deleted = deletedTools);
});
}
Several things to note here:
1) We are returning an IQueryable of Object instead of IQueryable of ToDo
2) Breeze will inspect the returned payload and automatically create breeze entities for any 'entityTypes' returned (even within a projection). So the result of this query will be an array of javascript objects each with two properties; 'ToDo' and 'Tools' where Tools is an array of 'Tool' entities. The nice thing is that both ToDo and Tool entities returned within the projection will be 'full' breeze entities.
3) You can still pass client side filters based on the projected property names. i.e.
var query = EntityQuery.from("TodoAndTools")
.where("Todo.Description", "startsWith", "A")
.using(em);
4) EF does support this.

when updating item with changed embedded docs in Mongoose, these embedded docs don't update

I'm doing the following using Mongoose:
that.model.update({_id: dao._id}, dao, { upsert: true }, cb);
Where dao is a mongoose representation containing (among other things) a couple of embedded documents. As a test I've deleted a couple of the embedded docs from the array before calling the update-method above.
The result is that the change to the array of embedded docs IS NOT persisted.
Anything I'm overlooking?
Hard to be certain w/o seeing more code, but if dao is a Mongoose model instance, you should be calling dao.save(cb); instead.
I solved the problem by doing something similar as proposed in the following issue: https://github.com/LearnBoost/mongoose/issues/571
For completeness some background which led to the problem.
I'm using DDD repositories which are populated on app-start. Under the hood this fetches Mongoose-objects (which are treate as DAOs in my situation) and are translated to domainobjects, which are cached in the repository. I need this separation between domainobjects and mongoose-objects, don't ask.
This means that getById, getAll and all other public interfaces of the repo work with domainobjects and not with mongoose-objects.
When doing things like add or update on the repo this internally only updates the in-mem cache (which, again, only uses domainobjects instead of mongoose-objects)
Only when doing commit on the repo does the possibly changed collection of domainobjects get persisted. This is done by creating NEW Mongoose-objects instead of fetching Existing mongoose-objects and updating those.
This is why I can't use dao.save() since, when I'm saving a different (just created) mongoose-object while a mongoose-object with the same id may possibly already exist in Mongo, it throws a duplicate id error.
Some relevant snippet from by code illustrating the solution:
var dao = that.createDAO(domainobject);
//https://github.com/LearnBoost/mongoose/issues/571
// Convert the Model instance to a simple object using Model's 'toObject' function
// to prevent weirdness like infinite looping...
var upsertData = dao.toObject();
// Delete the _id property, otherwise Mongo will return a "Mod on _id not allowed" error
delete upsertData._id;
that.model.update({_id: dao._id}, upsertData, { upsert: true }, cb);