I want to create a backend service which monitors a mongodb collection for new entries. As those are being created, I wish to run processing and update them.
I thought doing so with a Meteor service/app would be a wise idea because Meteor uses 'oplog tailing' which seems ideal for this purpose (I'd rather avoid polling if possible).
As such, I figured creating a minimal server-side-only app should solve it.
So basically, I need something along these lines:
if (Meteor.isServer) {
MyCollection = new Mongo.Collection('myCollection');
Meteor.publish('myCollectionPub', function () {
return MyCollection.find({ some: criteria... });
}
// is there such a thing?
Meteor.serverSideSubscribe('MyCollectionPub',
function (newDocs) {
// process/update newDocs
});
}
According to the Meteor docs, I cannot use Meteor.subscribe() on the server (and indeed it crashes if I try).
Question is:
Are there ways of 'subscribing' to collection updates on the server?
The PeerLibrary server-autorun package (along with it's dependant, reactive-mongo) will provide you with easy server-side observation of collections.
An alternative to #tarmes suggestion is the collection-hooks package, however as pointed out by David Weldon, it will only trigger in instance it is run in:
https://github.com/matb33/meteor-collection-hooks
MyCollection.after.insert(function (userId, doc) {
// ...
});
If you need it to run even when another instance makes a change in the mongo database, you can observe a cursor that is returned from your collection:
MyCollection.find({created_at : {$gt: some_current_time}}).observe({
added: function(item) {
// Alert code
}
});
Related
I am attempting to use pre('findOneAndUpdate') to update the icon attribute of the Meeting document. The update is based on the pre-existing value of the yearlymeeting attribute (see below).
Because pre and post save() hooks are not executed on update(), I seem to be unable to access the original document at all. Yet this is critical for the operation I'm trying to perform. Is there any way around this?
For example, I am able to accomplish my purpose on pre('save'), like so:
meetingSchema.pre('save', function(next) {
const yearlymeetingSlug = this.yearlymeeting[0].toLowerCase().replace(/[^A-z0-9]/g, '');
this.icon = `${yearlymeetingSlug}.png`
next();
});
What I would like to be able to do is something like this:
meetingSchema.pre('findOneAndUpdate', function(next) {
const yearlymeetingSlug = originalDocument.yearlymeeting[0].toLowerCase().replace(/[^A-z0-9]/g, '');
this.icon = `${yearlymeetingSlug}.png`
next();
});
I understand that this in pre(findOneAndUpdate) refers to the query, rather than the stored document itself. Is there any way to access the document, so that I can update icon based on the stored value of yearlymeeting?
tl;dr
Not possible via middleware. Query for the doc first, and then separately update a specific version of the doc to prevent race conditions.
Can't do it the way you're trying according to this issue on the Mongoose Github (from the main dev):
By design - the document being updated might not even be in the server's memory. In order to do that, mongoose would have to do a findOne() to load the document before doing the update(), which is not acceptable.
The design is to enable you to manipulate the query object by adding or removing filters, update params, options, etc. For instance, automatically calling .populate() with find() and findOne(), setting the multi: true option by default on certain models, access control, and other possibilities.
findOneAndUpdate() is a bit of a misnomer, it uses the underlying mongodb findAndModify command, it's not the same as findOne() + update(). As a separate operation, it should have its own middleware.
Following this, there are no other suggestions in the issue thread to access the original document inside of the middleware itself.
What I've seen done (and what I've had to do many times myself), is simply have to query for the document before updating it (which, of course, could lead to a race condition depending on who is updating the doc, and when, but you can fix that by also querying for a specific version of the document -- a sort of "optimistic locking"):
let meeting = yield Meeting.findOne({}).exec()
let update = {}
// ... some conditional logic to figure out which icon to set
update.icon = // whatever
yield Meeting.update({ _id: meeting._id, version: meeting.version }, update)
This is of course assuming you have a "version" field in your schema. This sort of locking will prevent you from updating an old version of the doc. If you're gonna use this kind of versioning, you'll also probably want to add some middleware that updates the version of a doc any time the doc is updated/saved.
You can also use a more naïve implementation, where you don't use locking, which may be fine in your specific business case, as long as you're aware of the possibility of a race condition, and the risks.
This may not be the best solution, but I did find a way to make it work. I used the controller rather than schema pre hooks. Here's what my update controller looks like now:
exports.updateMeeting = async (req, res) => {
const _id = req.params.id
let meeting = await Meeting.findOneAndUpdate({ _id }, req.body, {
new: true,
runValidators: true
});
/* New Code: */
const yearlymeetingSlug = meeting.yearlymeeting[0].toLowerCase().replace(/[^A-z0-9]/g, '');
meeting.icon = `${yearlymeetingSlug}.png`;
meeting.save();
req.flash('success', 'meeting successfully updated!');
res.redirect(`/meetings/${meeting.slug}`);
};
I welcome your feedback on any problems you see with this solution.
This is a very odd problem.
I am running a cursor.observe() in a Tracker.autorun(), but the changes are only occurring when I refresh the page. Here is the imporant code:
Client Side
Tracker.autorun(function (){
var initializing = true;
Links.find().observe({
added: function(doc){
var parents = Links.find({stacks: {$exists: true, $not: {$size: 0}}});
//Find the parent of this item with a stack array
if(!initializing){
_.each(parents.fetch(), function(parentDoc){
_.each(parentDoc.stacks, function (stackId){
//The troublesome server call
Meteor.call("addLinksFromDirToStack",
stackId, parentDoc.shared[0].id, parentDoc._id);
});
});
}
initializing = false;
}
});
});
Server Side
Meteor.methods({
addLinksFromDirToStack: function(stackId, userId, dirId){
var stack = Stacks.findOne(stackId);
if (stack){
var webArray = stack.webArray;
webArrayFiller(dirId, webArray);
Stacks.update(stackId, {$set: {webArray: webArray}});
Stacks.update(stackId, {$addToSet: {users: userId}});
}
// NOTE: IF I QUERY THE STACKS COLLECTION HERE, I SEE THE CHANGES
}
});
webArrayFiller = function(dirId, webArray){
//Update array
}
As commented above, If I query the Stacks collection directly after my update, the changes are reflected in the database. However, querying from the Mongo Shell, I still see the old Stack, without the update. And this is case on the client side, where the changes should be displayed.
I have tried a Meteor Reset to no avail. Any help would be much appreciated.
I figured out why it wasn't working.
I wasn't publishing every field of the Stacks collection. In particular, not the users field. Thus when I would attempt to change the collection with this line, it would fail:
Stacks.update(stackId, {$addToSet: {users: userId}});
However, Minimongo recognized the change, and so my queries would falsely show me I had succeeded. Once the server rejected the update, it would also reject the line directly above, making both changes not occur. Apparently, some find MongoDB's lack of error messages in failed transactions troublesome. It definitely made this error hard to find.
The solution was to wrap the server side code in a if (Meteor.isServer) so that it would only be run on the server side. An unfortunate consequence is that values cannot easily be returned from these calls, due to asynchronicity, but there are workarounds for that.
I am a new user to JavaScript and the meteor framework trying to understand the basic concepts. First of all I want to add a single document to a collection without duplicate entries.
this.addRole = function(roleName){
console.log(MongoRoles.find({name: roleName}).count());
if(!MongoRoles.find({name: roleName}).count())
MongoRoles.insert({name: roleName});
}
This code is called on the server as well as on the client. The log message on the client tells me there are no entries in the collection. Even if I refresh the page several times.
On the server duplicate entries get entered into the collection. I don't know why. Probably I did not understand the key concept. Could someone point it out to me, please?
Edit-1:
No, autopublish and insecure are not installed anymore. But I already published the MongoRoles collection (server side) and subscribed to it (client side). Furthermore I created a allow rule for inserts (client side).
Edit-2:
Thanks a lot for showing me the meteor method way but I want to get the point doing it without server side only methods involved. Let us say for academic purposes. ;-)
Just wrote a small example:
Client:
Posts = new Mongo.Collection("posts");
Posts.insert({title: "title-1"});
console.log(Posts.find().count());
Server:
Posts = new Mongo.Collection("posts");
Meteor.publish(null, function () {
return Posts.find()
})
Posts.allow({
insert: function(){return true}
})
If I check the server database via 'meteor mongo' it tells me every insert of my client code is saved there.
The log on the client tells me '1 count' every time I refresh the page. But I expected both the same. What am I doing wrong?
Edit-3:
I am back on my original role example (sorry for that). Just thought I got the point but I am still clueless. If I check the variable 'roleCount', 0 is responded all the time. How can I load the correct value into my variable? What is the best way to check if a document exists before the insertion into a collection? Guess the .find() is asynchronous as well? If so, how can I do it synchronous? If I got it right I have to wait for the value (synchronous) because I really relay on it.
Shared environment (client and server):
Roles = new Mongo.Collection("jaqua_roles");
Roles.allow({
insert: function(){return true}
})
var Role = function(){
this.addRole = function(roleName){
var roleCount = Roles.find({name: roleName}).count();
console.log(roleCount);
if(roleCount === 0){
Roles.insert({name: roleName}, function(error, result){
try{
console.log("Success: " + result);
var roleCount = Roles.find({name: roleName}).count();
console.log(roleCount);
} catch(error){
}
});
}
};
this.deleteRole = function(){
};
}
role = new Role();
role.addRole('test-role');
Server only:
Meteor.publish(null, function () {
return Roles.find()
})
Meteor's insert/update/remove methods (client-side) are not a great idea to use. Too many potential security pitfalls, and it takes a lot of thought and time to really patch up any holes. Further reading here.
I'm also wondering where you're calling addRole from. Assuming it's being triggered from client-side only, I would do this:
Client-side Code:
this.addRole = function(roleName){
var roleCount = MongoRoles.find({name: roleName}).count();
console.log(roleCount);
if (roleCount === 0) {
Meteor.call('insertRole', roleName, function (error, result) {
if (error) {
// check error.error and error.reason (if I'm remembering right)
} else {
// Success!
}
});
}
}
How I've modified this code and why:
I made a roleCount variable so that you can avoid calling MongoRoles.find() twice like that, which is inefficient and consumes unneeded resources (CPU, disk I/O, etc). Store it once, then reference the variable instead, much better.
When checking numbers, try to avoid doing things like if (!count). Using if (count === 0) is clearer, and shows that you're referencing a number. Statements like if (!xyz) would make one think this is a boolean (true/false) value.
Always use === in JavaScript, unless you want to intentionally do a loose equality operation. Read more on this.
Always use open/closed curly braces for if and other blocks, even if it contains just a single line of code. This is just good practice so that if you decide to add another line later, you don't have to then wrap it in braces. Just a good practice thing.
Changed your database insert into a Meteor method (see below).
Side note: I've used JavaScript (ES5), but since you're new to JavaScript, I think you should jump right into ES6. ES is short for ECMAScript (which is what JS is based on). ES6 (or ECMAScript 2015) is the most recent stable version which includes all kinds of new awesomeness that JavaScript didn't previously have.
Server-side Code:
Meteor.method('insertRole', function (roleName) {
check(roleName, String);
try {
// Any security checks, such as logged-in user, validating roleName, etc
MongoRoles.insert({name: roleName});
} catch (error) {
// error handling. just throw an error from here and handle it on client
if (badThing) {
throw new Meteor.Error('bad-thing', 'A bad thing happened.');
}
}
});
Hope this helps. This is all off the top of my head with no testing at all. But it should give you a better idea of an improved structure when it comes to database operations.
Addressing your edits
Your code looks good, except a couple issues:
You're defining Posts twice, don't do that. Make a file, for example, /lib/collections/posts.js and put the declaration and instantiation of Mongo.Collection in there. Then it will be executed on both client and server.
Your console.log would probably return an error, or zero, because Posts.insert is asynchronous on the client side. Try the below instead:
.
Posts.insert({title: "title-1"}, function (error, result) {
console.log(Posts.find().count());
});
I'm trying to backup a Lunr Index to Mongo (daily), and because it's running around 13MB, I'm triggering MongoError: document is larger than capped size errors. I'd like to use GridFS to get around the problem, but I'm having a heck of a time getting it to click.
In the simplest terms: Within Meteor, I'd like to save a 13MB JSON object to MongoDB using GridFS, and then be able to retrieve it when necessary -- all only on the server.
I've gone through the File-Collection and CollectionFS docs, and they seem far too complicated for what I'm trying to accomplish, and don't seem to address simply storing the contents of a variable. (Or, more likely, they do, and I'm just missing it.)
Here's what I'd like to do, in pseudo-code:
Backup = new GridFSCollection('backup');
var backupdata = (serialized search index data object);
Backup.insert({name:'searchindex', data:backupdata, date:new Date().getTime()});
var retrieved = Backup.findOne({name:'searchindex'});
Any suggestions?
var db = MongoInternals.defaultRemoteCollectionDriver().mongo.db;
var GridStore = MongoInternals.NpmModule.GridStore;
var largedata = new GridStore(db,'name','w'); //to write
largedata.open(function(error,gs){
if (error) return;
gs.write('somebigdata',function(error,done){
if (error) return;
largedata.close();
})
});
var largedata = new GridStore(db,'name','r'); //to read data
largedata.open(function(error,gs){
if (error) return;
gs.read(function(error,result){
if (error) return;
//then do something with the result
largedata.close();
})
});
explanation
First you need the db object, which can be exposed via the MongoInternals https://github.com/meteor/meteor/tree/devel/packages/mongo
Second, you need the GridStore object, you get it from the MongoInternals as well
Then you can create a write or read object and follow the API http://mongodb.github.io/node-mongodb-native/1.4/markdown-docs/gridfs.html
The mongo used by Meteor is 1.3.x whereas the latest node mongodb native driver is 2.x.x http://mongodb.github.io/node-mongodb-native/2.0/api-docs/
So the API may change in the future. Currently, you need to do open and close and the callback are all async, you may want to wrap them with Meteor.wrapAsync (may not work on largedata.open), Future or Fiber
I'm a couple hours new to Meteor and Mongo, coming from a Rails background and trying to understand how migrations work - or don't maybe?
I have a server/bootstrap.js file that I use to seed some data:
// if the database is empty on server start, create some sample data.
Meteor.startup(function () {
if (Users.find().count() === 0) {
var userData = [
{ name: 'Cool guy' },
{ name: 'Other dude' }
];
for (var i = 0; userData.length; i++) {
var userId = Users.insert({
name: userData[i].name
});
}
}
});
It seems like every time I want to change the database, say to add a new field, I have to run meteor reset to get it to pick up the changes.
But what happens if I create records or other data through the UI that I want to keep? In Rails, working with MySQL or PostgreSQL, I'd create a migration to create new fields without blowing away the entire database.
How does this work with Meteor and Mongo? Also thinking of the case of rolling out new changes from development to production. Thanks!
-- Update: 2013/09/24 --
Apparently, the schema-less nature of Mongo reduces or eliminates the need for migrations. In my case, modifying userData to add new fields won't work after it runs initially because of the Users count check - which is why I kept running meteor reset. I'll need to rethink my approach here and study up.
That said, there are projects out there that use migrations, like Telescope: https://github.com/SachaG/Telescope/blob/master/server/migrations.js
I also found the tutorial at http://try.mongodb.org/ useful.
First of all, your code is perfectly valid. And you know that.
mrt reset gives you a 'fresh' - empty database (as mentionned already).
If you want to reset a particular collection, you can do it so :
MyCollection.remove({});
But you have to understand the nature of NoSQL : there are no constraints on the data. It could be called NoREL (as in not a relational database, source : Wikipedia ).
MongoDB is also schema-less.
This means that you can use any field you want in your data. This is up to you (the programmer) to enforce specific constraints if you want some. In other words, there is no logic on the mongo side. It should accept any data you throw at it, just like Hubert OG demonstrated. Your code snippet could be :
// if the database is empty on server start, create some sample data.
Meteor.startup(function () {
if (Users.find().count() === 0) {
var userData = [
{ name: 'Cool guy' },
{ name: 'Other dude' },
{ nickname: 'Yet another dude' } // this line shows that mongo takes what you throw him
];
for (var i = 0; userData.length; i++) {
var userId = Users.insert({
name: userData[i].name
});
}
}
});
Source : http://www.mongodb.com/nosql
There is no need for migration there. You only have to add the logic in your application code.
Note : To import/export a database, you can have a look there : mongo import/export doc, and maybe at the db.copyDatabase(origin, destination, hostname) function.
There are no migrations in Mongo — there is no scheme! If you want to add a new field that was not there before, just do it and it will work. You can even have completely different documents in the same collection!
Items.insert({name: "keyboard", type: "input", interface: "usb"});
Items.insert({cherries: true, count: 5, unit: "buckets", taste: "awesome"});
This will just work. One of main reasons to use NoSQL (and advantages of Meteor over Rails) is that you don't have migrations to worry about.
Using mrt reset to change db model is a terrible idea. What it actually does is complete reset of db — it removes all of your data! While it's sometimes usefull in development, I bet it's not what you want in this case.