I'm getting ready to implement stripped down version of role based access control in my application and I'm contemplating how/what to model in my document store which happens to be mongodb with mongoose.js as my "convenience lib". But this question should apply to any document store.
It seems quite common that deciding between using embedded objects vs refs is a challenge when using a document store given the competing factors of duplication vs performance and what not. I'm trying to keep the RBAC as simple as possible and not go too crazy on nested Collections/Ref IDs which would mean a ton of loops, over using mongoose's populate, etc.
Question:
I'm already leaning toward having collections for User, Permission, and Role; but does it make sense to model Operations and Resources, or, just use key/vals for these?
See code example below or jsfiddle which should help to reason about the problem. Note it's not at all the way I want to implement this but just a way to examine to relationships!
/*
Imagine this being used in a CMS ;)
User: have a role property (e.g. role:"admin" or role:"writer")
Operation: Create,Read,Update,Delete,etc.
Resource: Page,Post,User, etc.
* For simplicity, we can represent operations and resource with simple strings.
Permission: A permission is an allowable "Operation" on a "Resource"
Role: A Role is just an abstraction of a set of possible "Permissions"
*/
// I could see this as a Permission model in mongo
var adminPerms = {
create: ['pages','posts', 'users'],
update: ['posts','pages','users'],
update_others: ['posts','pages'],
delete: ['posts','pages','users'],
read:['pages','posts','users']
};
// I could see this as a Role model in mongo
var admin = {
perms: adminPerms
};
var writerPerms = {
create: ['pages','posts'],
update: ['pages','posts'],
update_others: [],
delete: [],
read:['pages','posts']
};
var writer = {
perms: writerPerms
};
// Now we can just see if that user's perms has the operation on resource defined
function hasPerms(user, operation, resource) {
var i, len, op;
if(!user || !user.role || !operation || !resource) return false;
if(typeof rolemap[user.role] !== 'undefined' &&
typeof rolemap[user.role]['perms'] !== 'undefined' &&
typeof rolemap[user.role]['perms'][operation] !== 'undefined') {
op = rolemap[user.role]['perms'][operation];
for(i=0, len=op.length; i<len; i++) {
if(op[i] === resource) {
return true;
}
}
}
return false;
}
var rolemap = {"admin":admin, "writer":writer}
var user_admin = {name:'Rob Levin', role:'admin'}
var user_jack = {name:'Jack Black', role:'writer'}
hasPerms(user_jack, 'create', 'users')
// false
hasPerms(user_admin, 'create', 'users')
// true
hasPerms(user_admin, 'update_others', 'posts')
// true
hasPerms(user_jack, 'update_others', 'posts')
// false
EDIT: Assume that roles must be editable on a per app basis so I might want to allow admin users to control access rights; this is why I want to use a database.
Regarding doing it all inside the application that won't work given the requirement to persist and possibly change. However, one compromise in this direction is I could just the role collection:
db.role.find({name:'writer'}).pretty()
{
"_id" : ObjectId("4f4c2a510785b51c7b11bc45"),
"name" : "writer",
"perms" : {
"create" : [
"posts",
"pages"
],
"update" : [
"posts",
"pages"
],
"update_others" : [ ],
"delete" : [ ],
"read" : [
"posts",
"pages"
]
}
}
And than I could make changes like removal, etc., like the following (assuming I already have a reference to a role object retrieved from mongo at point of call):
function removePerm(role, op, resource) {
if(!role || !role.perms || !role.perms[op]) {
console.log("Something not defined!");
return false;
}
var perm = role.perms[op];
for(var i=0, len=perm.length; i<len; i++) {
if(perm[i] === resource) {
perm.splice(i,1);
break;
}
}
}
I recently used mongoosejs with a user/roles/permissions need I had with Drywall.js - the actual permissions are key/value based. They can be shared by group and also overridden granularly on the administrator level.
Although it's not exactly RBAC I'm hoping that reviewing another approach helps you get closer to achieving your goal.
Project Overview:
http://jedireza.github.com/drywall/
Mongoose Schemas:
https://github.com/jedireza/drywall/tree/master/schema
Specifically Look At:
/schema/User.js
/schema/Admin.js
/schema/AdminGroup.js
/schema/Account.js
I'm interested to see what you come up with.
Your design is almost entirely dependent on the behavior of your application. What I would recommend based on the information you've provided here is to keep the data in key/values, not in the database--CRUD operations aren't going to change, so there's no reason to put that in the db. The 'resources' are classes that you've already built into your code, so you don't need to duplicate it in the db either.
Related
Given the data structure below in firebase, i want to run a query to retrieve the blog 'efg'. I don't know the user id at this point.
{Users :
"1234567": {
name: 'Bob',
blogs: {
'abc':{..},
'zyx':{..}
}
},
"7654321": {
name: 'Frank',
blogs: {
'efg':{..},
'hij':{..}
}
}
}
The Firebase API only allows you to filter children one level deep (or with a known path) with its orderByChild and equalTo methods.
So without modifying/expanding your current data structure that just leaves the option to retrieve all data and filter it client-side:
var ref = firebase.database().ref('Users');
ref.once('value', function(snapshot) {
snapshot.forEach(function(userSnapshot) {
var blogs = userSnapshot.val().blogs;
var daBlog = blogs['efg'];
});
});
This is of course highly inefficient and won't scale when you have a non-trivial number of users/blogs.
So the common solution to that is to a so-called index to your tree that maps the key that you are looking for to the path where it resides:
{Blogs:
"abc": "1234567",
"zyx": "1234567",
"efg": "7654321",
"hij": "7654321"
}
Then you can quickly access the blog using:
var ref = firebase.database().ref();
ref.child('Blogs/efg').once('value', function(snapshot) {
var user = snapshot.val();
ref.child('Blogs/'+user+'/blogs').once('value', function(blogSnapshot) {
var daBlog = blogSnapshot.val();
});
});
You might also want to reconsider if you can restructure your data to better fit your use-case and Firebase's limitations. They have some good documentation on structuring your data, but the most important one for people new to NoSQL/hierarchical databases seems to be "avoid building nests".
Also see my answer on Firebase query if child of child contains a value for a good example. I'd also recommend reading about many-to-many relationships in Firebase, and this article on general NoSQL data modeling.
Given your current data structure you can retrieve the User that contains the blog post you are looking for.
const db = firebase.database()
const usersRef = db.ref('users')
const query = usersRef.orderByChild('blogs/efg').limitToLast(1)
query.once('value').then((ss) => {
console.log(ss.val()) //=> { '7654321': { blogs: {...}}}
})
You need to use limitToLast since Objects are sorted last when using orderByChild docs.
It's actually super easy - just use foreslash:
db.ref('Users').child("userid/name")
db.ref('Users').child("userid/blogs")
db.ref('Users').child("userid/blogs/abc")
No need of loops or anything more.
I am using sails to write a simple model which should expire after a few hours, so I need something like
createdAt: {
type: 'Date',
expires : 60,
index: true
}
But the "expireAfterSeconds" seems not working when I check my database (MongoDB), therefore I have to use
db.collection.ensureIndex( { "createdAt": 1 }, { expireAfterSeconds: 3600 } )
I wonder if it is possible to set the "expire" option in the model?
Inspecting the sails-mongo source code that is responsible for handling the index attribute property[1], it seems that it does not take into account anything like an expires property. This makes sense though, because the sails.js waterline ORM does not support database-specific functionality like expiring index data in MongoDB.
However, waterline does provide access to the MongoDB native connection (which uses node-mongodb-native) through the Collection.native() method[2]
Chances are, you only really need to make changes to the model's index once during the lifecycle of the Sails application, so the best place to do that would be in the config/bootstrap.js file. At this point in the sails.js lifecycle, all of the models have been instantiated, so you could do something like this to perform the necessary logic on your keys:
// config/bootstrap.js
exports.bootstrap = function (done) {
YourModelName.native(function (err, collection) {
// define index properties
collection.ensureIndex( { "createdAt": 1 }, { expireAfterSeconds: 3600 } );
// be sure to call the bootstrap callback to indicate completion
done();
});
}
You could take this even further and write a utility that inspects the models that sails has loaded (they are available as keys in the global object sails.models) and, if they have an expires attribute, perform the necessary native functionality. This would also need to be done in the config/bootstrap.js.
If you have native functionality that needs to be done on a per-record basis, you can use the sails.js model lifecycle hooks[3] for that.
references:
[1] https://github.com/balderdashy/sails-mongo/blob/ad7ec276af3a0e823ee5075074788ca915328db2/lib/adapter.js#L112
[2] https://github.com/balderdashy/sails-mongo/issues/21#issuecomment-20765896
[3] See the section titled "Lifecycle Callbacks" # http://sailsjs.org/#!documentation/models
UPDATE:
I created a sails hook to give advanced indexing options for models that use the sails-mongo adapter.
Supports all mongo indexing options.
https://www.npmjs.com/package/sails-hook-mongoat
OLD:
I created the below to do this automatically if the model attribute has an expires property and is of type 'date'. Just paste into to your config/bootstrap.js
Hope it helps!
_.forEach(Object.keys(sails.models), function(key) {
var model = sails.models[key];
_.forEach(Object.keys(model.attributes), function(attr) {
if (model.attributes[attr].hasOwnProperty('expires')) {
var seconds = model.attributes[attr].expires;
// delete validators from global model, otherwise sails will error
delete sails.models[key].attributes[attr].expires;
delete sails.models[key]._validator.validations[attr].expires;
if (model.attributes[attr].hasOwnProperty('type') && model.attributes[attr].type === 'date') {
var obj = {};
obj[attr] = 1;
model.native(function(err, collection) {
// define index properties
collection.ensureIndex(obj, {
expireAfterSeconds: seconds
}, function(err) {
if (err)
sails.log.error(err);
else
sails.log.info(key + " model attribute '" + attr + "' has been set to expire the document after " + seconds + " seconds.");
});
});
} else {
sails.log.warn(key + " model attribute '" + attr + "' is set to expire but is not of type 'date'. Skipping...");
}
}
});
});
I'm using meteor 0.3.7 in Win7(32) and trying to create a simple logging system using 2 MongoDB collections to store data that are linked by DBRef.
The current pseudo schema is :
Users {
username : String,
password : String,
created : Timestamp,
}
Logs {
user_id : DBRef {$id, $ref}
message : String
}
I use server methods to insert the logs so I can do some upserts on the clients collection.
Now I want to do an old "left join" and display a list of the last n logs with the embedded User name.
I don't want to embed the Logs in Users because the most used operation is getting the last n logs. Embedding in my opinion was going to have a big impact in performance.
What is the best approach to achieve this?
Next it was great if possible to edit the User name and all items change theis name
Regards
Playing around with Cursor.observe answered my question. It may not be the most effective way of doing this, but solves my future problems of derefering DBRefs "links"
So for the server we need to publish a special collection. One that can enumerate the cursor and for each document search for the corresponding DBRef.
Bare in mind this implementation is hardcoded and should be done as a package like UnRefCollection.
Server Side
CC.Logs = new Meteor.Collection("logs");
CC.Users = new Meteor.Collection("users");
Meteor.publish('logsAndUsers', function (page, size) {
var self = this;
var startup = true;
var startupList = [], uniqArr = [];
page = page || 1;
size = size || 100;
var skip = (page - 1) * size;
var cursor = CC.Logs.find({}, {limit : size, skip : skip});
var handle = cursor.observe({
added : function(doc, idx){
var clone = _.clone(doc);
var refId = clone.user_id.oid; // showld search DBRefs
if (startup){
startupList.push(clone);
if (!_.contains(uniqArr, refId))
uniqArr.push(refId);
} else {
// Clients added logs
var deref = CC.Users.findOne({_id : refid});
clone.user = deref;
self.set('logsAndUsers', clone._id, clone);
self.flush();
}
},
removed : function(doc, idx){
self.unset('logsAndUsers', doc._id, _.keys(doc));
self.flush();
},
changed : function(new_document, idx, old_document){
var set = {};
_.each(new_document, function (v, k) {
if (!_.isEqual(v, old_document[k]))
set[k] = v;
});
self.set('logsAndUsers', new_document._id, set);
var dead_keys = _.difference(_.keys(old_document), _.keys(new_document));
self.unset('logsAndUsers', new_document._id, dead_keys);
self.flush();
},
moved : function(document, old_index, new_index){
// Not used
}
});
self.onStop(function(){
handle.stop();
});
// Deref on first Run
var derefs = CC.Users.find({_id : {$in : uniqArr} }).fetch();
_.forEach(startupList, function (item){
_.forEach(derefs, function(ditems){
if (item["user_id"].oid === ditems._id){
item.user = ditems;
return false;
}
});
self.set('logsAndUsers', item._id, item);
});
delete derefs; // Not needed anymore
startup = false;
self.complete();
self.flush();
});
For each added logs document it'll search the users collection and try to add to the logs collection the missing information.
The added function is called for each document in the logs collection in the first run I created a startupList and an array of unique users ids so for the first run it'll query the db only once. Its a good idea to put a paging mechanism to speed up things.
Client Side
On the client, subscribe to the logsAndUsers collection, if you want to make changes do it directly to the Logs collection.
LogsAndUsers = new Meteor.collection('logsAndUser');
Logs = new Meteor.colection('logs'); // Changes here are observed in the LogsAndUsers collection
Meteor.autosubscribe(function () {
var page = Session.get('page') || 1;
Meteor.subscribe('logsAndUsers', page);
});
Why not just also store the username in the logs collection as well?
Then you can query on them directly without needing any kind of "join"
If for some reason you need to be able to handle that username change, you just fetch the user object by name, then query on Logs with { user_id : user._id }
I'm using meteor 0.3.7 in Win7(32) and trying to create a simple logging system using 2 MongoDB collections to store data that are linked by DBRef.
The current pseudo schema is :
Users {
username : String,
password : String,
created : Timestamp,
}
Logs {
user_id : DBRef {$id, $ref}
message : String
}
I use server methods to insert the logs so I can do some upserts on the clients collection.
Now I want to do an old "left join" and display a list of the last n logs with the embedded User name.
I don't want to embed the Logs in Users because the most used operation is getting the last n logs. Embedding in my opinion was going to have a big impact in performance.
What is the best approach to achieve this?
Next it was great if possible to edit the User name and all items change theis name
Regards
Playing around with Cursor.observe answered my question. It may not be the most effective way of doing this, but solves my future problems of derefering DBRefs "links"
So for the server we need to publish a special collection. One that can enumerate the cursor and for each document search for the corresponding DBRef.
Bare in mind this implementation is hardcoded and should be done as a package like UnRefCollection.
Server Side
CC.Logs = new Meteor.Collection("logs");
CC.Users = new Meteor.Collection("users");
Meteor.publish('logsAndUsers', function (page, size) {
var self = this;
var startup = true;
var startupList = [], uniqArr = [];
page = page || 1;
size = size || 100;
var skip = (page - 1) * size;
var cursor = CC.Logs.find({}, {limit : size, skip : skip});
var handle = cursor.observe({
added : function(doc, idx){
var clone = _.clone(doc);
var refId = clone.user_id.oid; // showld search DBRefs
if (startup){
startupList.push(clone);
if (!_.contains(uniqArr, refId))
uniqArr.push(refId);
} else {
// Clients added logs
var deref = CC.Users.findOne({_id : refid});
clone.user = deref;
self.set('logsAndUsers', clone._id, clone);
self.flush();
}
},
removed : function(doc, idx){
self.unset('logsAndUsers', doc._id, _.keys(doc));
self.flush();
},
changed : function(new_document, idx, old_document){
var set = {};
_.each(new_document, function (v, k) {
if (!_.isEqual(v, old_document[k]))
set[k] = v;
});
self.set('logsAndUsers', new_document._id, set);
var dead_keys = _.difference(_.keys(old_document), _.keys(new_document));
self.unset('logsAndUsers', new_document._id, dead_keys);
self.flush();
},
moved : function(document, old_index, new_index){
// Not used
}
});
self.onStop(function(){
handle.stop();
});
// Deref on first Run
var derefs = CC.Users.find({_id : {$in : uniqArr} }).fetch();
_.forEach(startupList, function (item){
_.forEach(derefs, function(ditems){
if (item["user_id"].oid === ditems._id){
item.user = ditems;
return false;
}
});
self.set('logsAndUsers', item._id, item);
});
delete derefs; // Not needed anymore
startup = false;
self.complete();
self.flush();
});
For each added logs document it'll search the users collection and try to add to the logs collection the missing information.
The added function is called for each document in the logs collection in the first run I created a startupList and an array of unique users ids so for the first run it'll query the db only once. Its a good idea to put a paging mechanism to speed up things.
Client Side
On the client, subscribe to the logsAndUsers collection, if you want to make changes do it directly to the Logs collection.
LogsAndUsers = new Meteor.collection('logsAndUser');
Logs = new Meteor.colection('logs'); // Changes here are observed in the LogsAndUsers collection
Meteor.autosubscribe(function () {
var page = Session.get('page') || 1;
Meteor.subscribe('logsAndUsers', page);
});
Why not just also store the username in the logs collection as well?
Then you can query on them directly without needing any kind of "join"
If for some reason you need to be able to handle that username change, you just fetch the user object by name, then query on Logs with { user_id : user._id }
I am making a analytics system, the API call would provide a Unique User ID, but it's not in sequence and too sparse.
I need to give each Unique User ID an auto increment id to mark a analytics datapoint in a bitarray/bitset. So the first user encounters would corresponding to the first bit of the bitarray, second user would be the second bit in the bitarray, etc.
So is there a solid and fast way to generate incremental Unique User IDs in MongoDB?
As selected answer says you can use findAndModify to generate sequential IDs.
But I strongly disagree with opinion that you should not do that. It all depends on your business needs. Having 12-byte ID may be very resource consuming and cause significant scalability issues in future.
I have detailed answer here.
You can, but you should not
https://web.archive.org/web/20151009224806/http://docs.mongodb.org/manual/tutorial/create-an-auto-incrementing-field/
Each object in mongo already has an id, and they are sortable in insertion order. What is wrong with getting collection of user objects, iterating over it and use this as incremented ID? Er go for kind of map-reduce job entirely
I know this is an old question, but I shall post my answer for posterity...
It depends on the system that you are building and the particular business rules in place.
I am building a moderate to large scale CRM in MongoDb, C# (Backend API), and Angular (Frontend web app) and found ObjectId utterly terrible for use in Angular Routing for selecting particular entities. Same with API Controller routing.
The suggestion above worked perfectly for my project.
db.contacts.insert({
"id":db.contacts.find().Count()+1,
"name":"John Doe",
"emails":[
"john#doe.com",
"john.doe#business.com"
],
"phone":"555111322",
"status":"Active"
});
The reason it is perfect for my case, but not all cases is that as the above comment states, if you delete 3 records from the collection, you will get collisions.
My business rules state that due to our in house SLA's, we are not allowed to delete correspondence data or clients records for longer than the potential lifespan of the application I'm writing, and therefor, I simply mark records with an enum "Status" which is either "Active" or "Deleted". You can delete something from the UI, and it will say "Contact has been deleted" but all the application has done is change the status of the contact to "Deleted" and when the app calls the respository for a list of contacts, I filter out deleted records before pushing the data to the client app.
Therefore, db.collection.find().count() + 1 is a perfect solution for me...
It won't work for everyone, but if you will not be deleting data, it works fine.
Edit
latest versions of pymongo:
db.contacts.count() + 1
First Record should be add
"_id" = 1 in your db
$database = "demo";
$collections ="democollaction";
echo getnextid($database,$collections);
function getnextid($database,$collections){
$m = new MongoClient();
$db = $m->selectDB($database);
$cursor = $collection->find()->sort(array("_id" => -1))->limit(1);
$array = iterator_to_array($cursor);
foreach($array as $value){
return $value["_id"] + 1;
}
}
I had a similar issue, namely I was interested in generating unique numbers, which can be used as identifiers, but doesn't have to. I came up with the following solution. First to initialize the collection:
fun create(mongo: MongoTemplate) {
mongo.db.getCollection("sequence")
.insertOne(Document(mapOf("_id" to "globalCounter", "sequenceValue" to 0L)))
}
An then a service that return unique (and ascending) numbers:
#Service
class IdCounter(val mongoTemplate: MongoTemplate) {
companion object {
const val collection = "sequence"
}
private val idField = "_id"
private val idValue = "globalCounter"
private val sequence = "sequenceValue"
fun nextValue(): Long {
val filter = Document(mapOf(idField to idValue))
val update = Document("\$inc", Document(mapOf(sequence to 1)))
val updated: Document = mongoTemplate.db.getCollection(collection).findOneAndUpdate(filter, update)!!
return updated[sequence] as Long
}
}
I believe that id doesn't have the weaknesses related to concurrent environment that some of the other solutions may suffer from.
// await collection.insertOne({ autoIncrementId: 1 });
const { value: { autoIncrementId } } = await collection.findOneAndUpdate(
{ autoIncrementId: { $exists: true } },
{
$inc: { autoIncrementId: 1 },
},
);
return collection.insertOne({ id: autoIncrementId, ...data });
I used something like nested queries in MySQL to simulate auto increment, which worked for me. To get the latest id and increment one to it you can use:
lastContact = db.contacts.find().sort({$natural:-1}).limit(1)[0];
db.contacts.insert({
"id":lastContact ?lastContact ["id"] + 1 : 1,
"name":"John Doe",
"emails": ["john#doe.com", "john.doe#business.com"],
"phone":"555111322",
"status":"Active"
})
It solves the removal issue of Alex's answer. So no duplicate id will appear if any record is removed.
More explanation: I just get the id of the latest inserted document, add one to it, and then set it as the id of the new record. And ternary is for cases that we don't have any records yet or all of the records are removed.
this could be another approach
const mongoose = require("mongoose");
const contractSchema = mongoose.Schema(
{
account: {
type: mongoose.Schema.Types.ObjectId,
required: true,
},
idContract: {
type: Number,
default: 0,
},
},
{ timestamps: true }
);
contractSchema.pre("save", function (next) {
var docs = this;
mongoose
.model("contract", contractSchema)
.countDocuments({ account: docs.account }, function (error, counter) {
if (error) return next(error);
docs.idContract = counter + 1;
next();
});
});
module.exports = mongoose.model("contract", contractSchema);
// First check the table length
const data = await table.find()
if(data.length === 0){
const id = 1
// then post your query along with your id
}
else{
// find last item and then its id
const length = data.length
const lastItem = data[length-1]
const lastItemId = lastItem.id // or { id } = lastItem
const id = lastItemId + 1
// now apply new id to your new item
// even if you delete any item from middle also this work
}