I am using sails to write a simple model which should expire after a few hours, so I need something like
createdAt: {
type: 'Date',
expires : 60,
index: true
}
But the "expireAfterSeconds" seems not working when I check my database (MongoDB), therefore I have to use
db.collection.ensureIndex( { "createdAt": 1 }, { expireAfterSeconds: 3600 } )
I wonder if it is possible to set the "expire" option in the model?
Inspecting the sails-mongo source code that is responsible for handling the index attribute property[1], it seems that it does not take into account anything like an expires property. This makes sense though, because the sails.js waterline ORM does not support database-specific functionality like expiring index data in MongoDB.
However, waterline does provide access to the MongoDB native connection (which uses node-mongodb-native) through the Collection.native() method[2]
Chances are, you only really need to make changes to the model's index once during the lifecycle of the Sails application, so the best place to do that would be in the config/bootstrap.js file. At this point in the sails.js lifecycle, all of the models have been instantiated, so you could do something like this to perform the necessary logic on your keys:
// config/bootstrap.js
exports.bootstrap = function (done) {
YourModelName.native(function (err, collection) {
// define index properties
collection.ensureIndex( { "createdAt": 1 }, { expireAfterSeconds: 3600 } );
// be sure to call the bootstrap callback to indicate completion
done();
});
}
You could take this even further and write a utility that inspects the models that sails has loaded (they are available as keys in the global object sails.models) and, if they have an expires attribute, perform the necessary native functionality. This would also need to be done in the config/bootstrap.js.
If you have native functionality that needs to be done on a per-record basis, you can use the sails.js model lifecycle hooks[3] for that.
references:
[1] https://github.com/balderdashy/sails-mongo/blob/ad7ec276af3a0e823ee5075074788ca915328db2/lib/adapter.js#L112
[2] https://github.com/balderdashy/sails-mongo/issues/21#issuecomment-20765896
[3] See the section titled "Lifecycle Callbacks" # http://sailsjs.org/#!documentation/models
UPDATE:
I created a sails hook to give advanced indexing options for models that use the sails-mongo adapter.
Supports all mongo indexing options.
https://www.npmjs.com/package/sails-hook-mongoat
OLD:
I created the below to do this automatically if the model attribute has an expires property and is of type 'date'. Just paste into to your config/bootstrap.js
Hope it helps!
_.forEach(Object.keys(sails.models), function(key) {
var model = sails.models[key];
_.forEach(Object.keys(model.attributes), function(attr) {
if (model.attributes[attr].hasOwnProperty('expires')) {
var seconds = model.attributes[attr].expires;
// delete validators from global model, otherwise sails will error
delete sails.models[key].attributes[attr].expires;
delete sails.models[key]._validator.validations[attr].expires;
if (model.attributes[attr].hasOwnProperty('type') && model.attributes[attr].type === 'date') {
var obj = {};
obj[attr] = 1;
model.native(function(err, collection) {
// define index properties
collection.ensureIndex(obj, {
expireAfterSeconds: seconds
}, function(err) {
if (err)
sails.log.error(err);
else
sails.log.info(key + " model attribute '" + attr + "' has been set to expire the document after " + seconds + " seconds.");
});
});
} else {
sails.log.warn(key + " model attribute '" + attr + "' is set to expire but is not of type 'date'. Skipping...");
}
}
});
});
Related
I'm using the request library to make calls from one sails app to another one which exposes the default blueprint endpoints. It works fine when I query by non-id fields, but I need to run some queries by passing id arrays. The problem is that the moment you provide an id, only the first id is considered, effectively not allowing this kind of query.
Is there a way to get around this? I could switch over to another attribute if all else fails but I need to know if there is a proper way around this.
Here's how I'm querying:
var idArr = [];//array of ids
var queryParams = { id: idArr };
var options: {
//headers, method and url here
json: queryParams
};
request(options, function(err, response, body){
if (err) return next(err);
return next(null, body);
});
Thanks in advance.
Sails blueprint APIs allow you to use the same waterline query langauge that you would otherwise use in code.
You can directly pass the array of id's in the get call to receive the objects as follows
GET /city?where={"id":[1, 2]}
Refer here for more.
Have fun!
Alright, I switched to a hacky solution to get moving.
For all models that needed querying by id arrays, I added a secondary attribute to the model. Let's call it code. Then, in afterCreate(), I updated code and set it equal to the id. This incurs an additional database call, but it's fine since it's called just once - when the object is created.
Here's the code.
module.exports = {
attributes: {
code: {
type: 'string'//the secondary attribute
},
// other attributes
},
afterCreate: function (newObj, next) {
Model.update({ id: newObj.id }, { code: newObj.id }, next);
}
}
Note that newObj isn't a Model object as even I was led to believe. So we cannot simply update its code and call newObj.save().
After this, in the queries having id arrays, substituting id with code makes them work as expected!
I have a model where only one record can have a 'current' property set to, for example, 1.
Is it possible for the beforeCreate or beforeUpdate to access the collection. Basically I want to do something like this:
afterUpdate: function (values, next) {
// If this value is current, reset all the others
if (values.current == 1) {
this.collection.update({
id: { '!': values.id }
}, {
current: 0
}, next);
}
}
What id don't know is what I can reliably use for this.collection in the example above.
Thanks in advance.
Yes, if you're using Sails v0.10.X you can access any model using sails global variable:
sails.models.users.find({id: 1})
Note: all model names in sails.models are in lower case
Ok, still in my toy app, I want to find out the average mileage on a group of car owners' odometers. This is pretty easy on the client but doesn't scale. Right? But on the server, I don't exactly see how to accomplish it.
Questions:
How do you implement something on the server then use it on the client?
How do you use the $avg aggregation function of mongo to leverage its optimized aggregation function?
Or alternatively to (2) how do you do a map/reduce on the server and make it available to the client?
The suggestion by #HubertOG was to use Meteor.call, which makes sense and I did this:
# Client side
Template.mileage.average_miles = ->
answer = null
Meteor.call "average_mileage", (error, result) ->
console.log "got average mileage result #{result}"
answer = result
console.log "but wait, answer = #{answer}"
answer
# Server side
Meteor.methods average_mileage: ->
console.log "server mileage called"
total = count = 0
r = Mileage.find({}).forEach (mileage) ->
total += mileage.mileage
count += 1
console.log "server about to return #{total / count}"
total / count
That would seem to work fine, but it doesn't because as near as I can tell Meteor.call is an asynchronous call and answer will always be a null return. Handling stuff on the server seems like a common enough use case that I must have just overlooked something. What would that be?
Thanks!
As of Meteor 0.6.5, the collection API doesn't support aggregation queries yet because there's no (straightforward) way to do live updates on them. However, you can still write them yourself, and make them available in a Meteor.publish, although the result will be static. In my opinion, doing it this way is still preferable because you can merge multiple aggregations and use the client-side collection API.
Meteor.publish("someAggregation", function (args) {
var sub = this;
// This works for Meteor 0.6.5
var db = MongoInternals.defaultRemoteCollectionDriver().mongo.db;
// Your arguments to Mongo's aggregation. Make these however you want.
var pipeline = [
{ $match: doSomethingWith(args) },
{ $group: {
_id: whatWeAreGroupingWith(args),
count: { $sum: 1 }
}}
];
db.collection("server_collection_name").aggregate(
pipeline,
// Need to wrap the callback so it gets called in a Fiber.
Meteor.bindEnvironment(
function(err, result) {
// Add each of the results to the subscription.
_.each(result, function(e) {
// Generate a random disposable id for aggregated documents
sub.added("client_collection_name", Random.id(), {
key: e._id.somethingOfInterest,
count: e.count
});
});
sub.ready();
},
function(error) {
Meteor._debug( "Error doing aggregation: " + error);
}
)
);
});
The above is an example grouping/count aggregation. Some things of note:
When you do this, you'll naturally be doing an aggregation on server_collection_name and pushing the results to a different collection called client_collection_name.
This subscription isn't going to be live, and will probably be updated whenever the arguments change, so we use a really simple loop that just pushes all the results out.
The results of the aggregation don't have Mongo ObjectIDs, so we generate some arbitrary ones of our own.
The callback to the aggregation needs to be wrapped in a Fiber. I use Meteor.bindEnvironment here but one can also use a Future for more low-level control.
If you start combining the results of publications like these, you'll need to carefully consider how the randomly generated ids impact the merge box. However, a straightforward implementation of this is just a standard database query, except it is more convenient to use with Meteor APIs client-side.
TL;DR version: Almost anytime you are pushing data out from the server, a publish is preferable to a method.
For more information about different ways to do aggregation, check out this post.
I did this with the 'aggregate' method. (ver 0.7.x)
if(Meteor.isServer){
Future = Npm.require('fibers/future');
Meteor.methods({
'aggregate' : function(param){
var fut = new Future();
MongoInternals.defaultRemoteCollectionDriver().mongo._getCollection(param.collection).aggregate(param.pipe,function(err, result){
fut.return(result);
});
return fut.wait();
}
,'test':function(param){
var _param = {
pipe : [
{ $unwind:'$data' },
{ $match:{
'data.y':"2031",
'data.m':'01',
'data.d':'01'
}},
{ $project : {
'_id':0
,'project_id' : "$project_id"
,'idx' : "$data.idx"
,'y' : '$data.y'
,'m' : '$data.m'
,'d' : '$data.d'
}}
],
collection:"yourCollection"
}
Meteor.call('aggregate',_param);
}
});
}
If you want reactivity, use Meteor.publish instead of Meteor.call. There's an example in the docs where they publish the number of messages in a given room (just above the documentation for this.userId), you should be able to do something similar.
You can use Meteor.methods for that.
// server
Meteor.methods({
average: function() {
...
return something;
},
});
// client
var _avg = { /* Create an object to store value and dependency */
dep: new Deps.Dependency();
};
Template.mileage.rendered = function() {
_avg.init = true;
};
Template.mileage.averageMiles = function() {
_avg.dep.depend(); /* Make the function rerun when _avg.dep is touched */
if(_avg.init) { /* Fetch the value from the server if not yet done */
_avg.init = false;
Meteor.call('average', function(error, result) {
_avg.val = result;
_avg.dep.changed(); /* Rerun the helper */
});
}
return _avg.val;
});
I'm getting ready to implement stripped down version of role based access control in my application and I'm contemplating how/what to model in my document store which happens to be mongodb with mongoose.js as my "convenience lib". But this question should apply to any document store.
It seems quite common that deciding between using embedded objects vs refs is a challenge when using a document store given the competing factors of duplication vs performance and what not. I'm trying to keep the RBAC as simple as possible and not go too crazy on nested Collections/Ref IDs which would mean a ton of loops, over using mongoose's populate, etc.
Question:
I'm already leaning toward having collections for User, Permission, and Role; but does it make sense to model Operations and Resources, or, just use key/vals for these?
See code example below or jsfiddle which should help to reason about the problem. Note it's not at all the way I want to implement this but just a way to examine to relationships!
/*
Imagine this being used in a CMS ;)
User: have a role property (e.g. role:"admin" or role:"writer")
Operation: Create,Read,Update,Delete,etc.
Resource: Page,Post,User, etc.
* For simplicity, we can represent operations and resource with simple strings.
Permission: A permission is an allowable "Operation" on a "Resource"
Role: A Role is just an abstraction of a set of possible "Permissions"
*/
// I could see this as a Permission model in mongo
var adminPerms = {
create: ['pages','posts', 'users'],
update: ['posts','pages','users'],
update_others: ['posts','pages'],
delete: ['posts','pages','users'],
read:['pages','posts','users']
};
// I could see this as a Role model in mongo
var admin = {
perms: adminPerms
};
var writerPerms = {
create: ['pages','posts'],
update: ['pages','posts'],
update_others: [],
delete: [],
read:['pages','posts']
};
var writer = {
perms: writerPerms
};
// Now we can just see if that user's perms has the operation on resource defined
function hasPerms(user, operation, resource) {
var i, len, op;
if(!user || !user.role || !operation || !resource) return false;
if(typeof rolemap[user.role] !== 'undefined' &&
typeof rolemap[user.role]['perms'] !== 'undefined' &&
typeof rolemap[user.role]['perms'][operation] !== 'undefined') {
op = rolemap[user.role]['perms'][operation];
for(i=0, len=op.length; i<len; i++) {
if(op[i] === resource) {
return true;
}
}
}
return false;
}
var rolemap = {"admin":admin, "writer":writer}
var user_admin = {name:'Rob Levin', role:'admin'}
var user_jack = {name:'Jack Black', role:'writer'}
hasPerms(user_jack, 'create', 'users')
// false
hasPerms(user_admin, 'create', 'users')
// true
hasPerms(user_admin, 'update_others', 'posts')
// true
hasPerms(user_jack, 'update_others', 'posts')
// false
EDIT: Assume that roles must be editable on a per app basis so I might want to allow admin users to control access rights; this is why I want to use a database.
Regarding doing it all inside the application that won't work given the requirement to persist and possibly change. However, one compromise in this direction is I could just the role collection:
db.role.find({name:'writer'}).pretty()
{
"_id" : ObjectId("4f4c2a510785b51c7b11bc45"),
"name" : "writer",
"perms" : {
"create" : [
"posts",
"pages"
],
"update" : [
"posts",
"pages"
],
"update_others" : [ ],
"delete" : [ ],
"read" : [
"posts",
"pages"
]
}
}
And than I could make changes like removal, etc., like the following (assuming I already have a reference to a role object retrieved from mongo at point of call):
function removePerm(role, op, resource) {
if(!role || !role.perms || !role.perms[op]) {
console.log("Something not defined!");
return false;
}
var perm = role.perms[op];
for(var i=0, len=perm.length; i<len; i++) {
if(perm[i] === resource) {
perm.splice(i,1);
break;
}
}
}
I recently used mongoosejs with a user/roles/permissions need I had with Drywall.js - the actual permissions are key/value based. They can be shared by group and also overridden granularly on the administrator level.
Although it's not exactly RBAC I'm hoping that reviewing another approach helps you get closer to achieving your goal.
Project Overview:
http://jedireza.github.com/drywall/
Mongoose Schemas:
https://github.com/jedireza/drywall/tree/master/schema
Specifically Look At:
/schema/User.js
/schema/Admin.js
/schema/AdminGroup.js
/schema/Account.js
I'm interested to see what you come up with.
Your design is almost entirely dependent on the behavior of your application. What I would recommend based on the information you've provided here is to keep the data in key/values, not in the database--CRUD operations aren't going to change, so there's no reason to put that in the db. The 'resources' are classes that you've already built into your code, so you don't need to duplicate it in the db either.
I am making a analytics system, the API call would provide a Unique User ID, but it's not in sequence and too sparse.
I need to give each Unique User ID an auto increment id to mark a analytics datapoint in a bitarray/bitset. So the first user encounters would corresponding to the first bit of the bitarray, second user would be the second bit in the bitarray, etc.
So is there a solid and fast way to generate incremental Unique User IDs in MongoDB?
As selected answer says you can use findAndModify to generate sequential IDs.
But I strongly disagree with opinion that you should not do that. It all depends on your business needs. Having 12-byte ID may be very resource consuming and cause significant scalability issues in future.
I have detailed answer here.
You can, but you should not
https://web.archive.org/web/20151009224806/http://docs.mongodb.org/manual/tutorial/create-an-auto-incrementing-field/
Each object in mongo already has an id, and they are sortable in insertion order. What is wrong with getting collection of user objects, iterating over it and use this as incremented ID? Er go for kind of map-reduce job entirely
I know this is an old question, but I shall post my answer for posterity...
It depends on the system that you are building and the particular business rules in place.
I am building a moderate to large scale CRM in MongoDb, C# (Backend API), and Angular (Frontend web app) and found ObjectId utterly terrible for use in Angular Routing for selecting particular entities. Same with API Controller routing.
The suggestion above worked perfectly for my project.
db.contacts.insert({
"id":db.contacts.find().Count()+1,
"name":"John Doe",
"emails":[
"john#doe.com",
"john.doe#business.com"
],
"phone":"555111322",
"status":"Active"
});
The reason it is perfect for my case, but not all cases is that as the above comment states, if you delete 3 records from the collection, you will get collisions.
My business rules state that due to our in house SLA's, we are not allowed to delete correspondence data or clients records for longer than the potential lifespan of the application I'm writing, and therefor, I simply mark records with an enum "Status" which is either "Active" or "Deleted". You can delete something from the UI, and it will say "Contact has been deleted" but all the application has done is change the status of the contact to "Deleted" and when the app calls the respository for a list of contacts, I filter out deleted records before pushing the data to the client app.
Therefore, db.collection.find().count() + 1 is a perfect solution for me...
It won't work for everyone, but if you will not be deleting data, it works fine.
Edit
latest versions of pymongo:
db.contacts.count() + 1
First Record should be add
"_id" = 1 in your db
$database = "demo";
$collections ="democollaction";
echo getnextid($database,$collections);
function getnextid($database,$collections){
$m = new MongoClient();
$db = $m->selectDB($database);
$cursor = $collection->find()->sort(array("_id" => -1))->limit(1);
$array = iterator_to_array($cursor);
foreach($array as $value){
return $value["_id"] + 1;
}
}
I had a similar issue, namely I was interested in generating unique numbers, which can be used as identifiers, but doesn't have to. I came up with the following solution. First to initialize the collection:
fun create(mongo: MongoTemplate) {
mongo.db.getCollection("sequence")
.insertOne(Document(mapOf("_id" to "globalCounter", "sequenceValue" to 0L)))
}
An then a service that return unique (and ascending) numbers:
#Service
class IdCounter(val mongoTemplate: MongoTemplate) {
companion object {
const val collection = "sequence"
}
private val idField = "_id"
private val idValue = "globalCounter"
private val sequence = "sequenceValue"
fun nextValue(): Long {
val filter = Document(mapOf(idField to idValue))
val update = Document("\$inc", Document(mapOf(sequence to 1)))
val updated: Document = mongoTemplate.db.getCollection(collection).findOneAndUpdate(filter, update)!!
return updated[sequence] as Long
}
}
I believe that id doesn't have the weaknesses related to concurrent environment that some of the other solutions may suffer from.
// await collection.insertOne({ autoIncrementId: 1 });
const { value: { autoIncrementId } } = await collection.findOneAndUpdate(
{ autoIncrementId: { $exists: true } },
{
$inc: { autoIncrementId: 1 },
},
);
return collection.insertOne({ id: autoIncrementId, ...data });
I used something like nested queries in MySQL to simulate auto increment, which worked for me. To get the latest id and increment one to it you can use:
lastContact = db.contacts.find().sort({$natural:-1}).limit(1)[0];
db.contacts.insert({
"id":lastContact ?lastContact ["id"] + 1 : 1,
"name":"John Doe",
"emails": ["john#doe.com", "john.doe#business.com"],
"phone":"555111322",
"status":"Active"
})
It solves the removal issue of Alex's answer. So no duplicate id will appear if any record is removed.
More explanation: I just get the id of the latest inserted document, add one to it, and then set it as the id of the new record. And ternary is for cases that we don't have any records yet or all of the records are removed.
this could be another approach
const mongoose = require("mongoose");
const contractSchema = mongoose.Schema(
{
account: {
type: mongoose.Schema.Types.ObjectId,
required: true,
},
idContract: {
type: Number,
default: 0,
},
},
{ timestamps: true }
);
contractSchema.pre("save", function (next) {
var docs = this;
mongoose
.model("contract", contractSchema)
.countDocuments({ account: docs.account }, function (error, counter) {
if (error) return next(error);
docs.idContract = counter + 1;
next();
});
});
module.exports = mongoose.model("contract", contractSchema);
// First check the table length
const data = await table.find()
if(data.length === 0){
const id = 1
// then post your query along with your id
}
else{
// find last item and then its id
const length = data.length
const lastItem = data[length-1]
const lastItemId = lastItem.id // or { id } = lastItem
const id = lastItemId + 1
// now apply new id to your new item
// even if you delete any item from middle also this work
}