Is there such possibility from database backend to force user to read only from SECONDARY members ?
I would like to restrict some users to not be able to impact performance in PRIMARY replicaset members in my on-premise deployment ( not atlas )
Issue is easy to solve if customer agree adding to the URI
readPreference=secondary
But I am checking if there is option to force from the database side without asking the customer ...
the only option I have found is to restrict by server IP address:
use admin
db.createUser(
{
user: "dbuser",
pwd: "password"
roles: [ { role: "readWrite", db: "reporting" } ],
authenticationRestrictions: [ {
clientSource: ["192.0.2.0"],
serverAddress: ["198.51.100.1","192.51.100.2"]
} ]
}
)
There are currently no supported ways to enforce this from within MongoDB itself apart from the authenticationRestrictions configurations for defining users which is noted in the question itself.
Regarding the comments - ANALYTICS tag in Atlas are a (automatic) Replica Set Tag. Replica set tags themselves can be used in on-premise deployments. But tags are used in conjunction with read preference which is set by the client application (at least in the connection string). So that approach/solution really doesn't provide any additional enforcement from read preference alone for the purposes of this question. Additional information about tags can be found here and here.
In an 'unsupported'/hacky fashion, you could create the user(s) directly and only on the SECONDARY members that you want the client to read from. This would be accomplished by taking the member out of the replica set, starting it up as a standalone, creating the user, and then joining it back to the replica set. While it would probably work, there are a number of implications that don't make this a particularly good approach. For example, elections (for high availability purposes) would change the PRIMARY (therefore where the client can read from) among other things.
Other approaches to this would be in redirecting/restricting traffic at the network layer. Again not a great approach.
Related
How can I check if meteor is using the oplog of my mongo?
I have a cluster of mongo and set two envs for my meteor.
MONGO_URL=mongodb://mongo/app?replicaSet=rs0
MONGO_OPLOG_URL=mongodb://mongo/local?authSource=app
How can I check if the opt log is actually in use. Meteor can fallback to query polling which is very inefficient but I would like to see if it's working properly with the oplog.
Any ideas?
Quoting the relevant bits from Meteor's OplogObserveDriver docs:
How to tell if your queries are using OplogObserveDriver
For now, we only have a crude way to tell how many observeChanges calls are using OplogObserveDriver, and not which calls they are.
This uses the facts package, an internal Meteor package that exposes real-time metrics for the current Meteor server. In your app, run meteor add facts, and add the {{> serverFacts}} template to your app. If you are using the autopublish package, Meteor will automatically publish all metrics to all users. If you are not using autopublish, you will have to tell Meteor which users can see your metrics by calling Facts.setUserIdFilter in server code; for example:
Facts.setUserIdFilter(function (userId) {
var user = Meteor.users.findOne(userId);
return user && user.admin;
});
(When running your app locally, Facts.setUserIdFilter(function () { return true; }); may be good enough!)
Now look at your app. The facts template will render a variety of metrics; the ones we're looking for are observe-drivers-oplog and observe-drivers-polling in the mongo-livedata section. If observe-drivers-polling is zero or not rendered at all, then all of your observeChanges calls are using OplogObserveDriver!
To set up oplog tailing, you need to set up a user on my_database, and an oplog_user on local. Then, specify the following URIs to connect to your replica set named test-shard (e.g. if there are 3 hosts named test-shard-[0-2]):
MONGO_URL="mongodb://user:PASS#test-shard-0.mongodb.net:27017,test-shard-1.mongodb.net:27017,test-shard-2.mongodb.net:27017/my_database?ssl=true&replicaSet=test-shard&authSource=admin"
MONGO_OPLOG_URL="mongodb://oplog_user:PASS#test-shard-0.mongodb.net:27017,test-shard-1.mongodb.net:27017,test-shard-2.mongodb.net:27017/local?ssl=true&replicaSet=test-shard&authSource=admin"
On MongoDB Atlas they require ssl=true, and also all users authenticate through the admin database. On another deployment you might just authenticate through my_database, in which case you'd remove the authsource=admin for MONGO_URL and write authsource=my_database for MONGO_OPLOG_URL. See this post for another example.
With MongoDB 3.6 and the Mongo node driver 3.0+, you may be able to use a succinct notation for DNS seedlist connections, e.g. on MongoDB Atlas, to specify the environment variables:
MONGO_URL="mongodb+srv://user:PASS#foo.mongodb.net/my_database"
MONGO_OPLOG_URL="mongodb+srv://oplog_user:PASS#foo.mongodb.net/local"
The link above explains how this notation fills in the ssl, replicaSet, and authSource arguments. This is a lot nicer than the long strings above, and also means you can scale your replica set up and down without needing to reconfigure anything.
As hwillson mentioned, use the facts-ui and facts-base packages (formerly facts) to see if there are any oplogObserveDrivers running in your app. If they are all pollingObserveDriver, than oplog is not set up correctly.
If you are using Kadira APM to monitor your app's performance, you can see if oplogs are working by navigating to the "Live Queries" section and having a look at the "Oplog notifications" chart.
You can see in my screenshot that oplogs are working, as values appear in the chart (bottom right). If oplogs weren't working then this chart would be empty.
This may be very late, but this is the only way that worked for me :
someCollection._driver.mongo._oplogHandle
if this is set to null then the oplog is not enabled, otherwise you can use this handle to check for more details.
I currently assign a mongodb to my meteor app using the env variable
"MONGO_URL": "mongodb://localhost:27017/dbName" when I start the meteor instance.
So all data gets written to the mongo database with the name "dbName".
I am looking for a way to individually set the dbName for each custumer upon login in order to seperate their data into different databases.
This generally unsupported as this is defined at startup. However, this thread offers a possible solution:
https://forums.meteor.com/t/switch-database-while-meteor-is-running/4361/6
var database = new MongoInternals.RemoteCollectionDriver("<mongo url>");
MyCollection = new Mongo.Collection("collection_name", { _driver: database });
This would allow you to define the database name in the mongo url but would require a fair bit of extra work to redefine your collections on a customer by customer basis.
Here's another approach that will make your life eternally easier:
Create a generic site with no accounts at mysite.com
When they login at mysite.com, figure out what site they actually belong to and redirect them to customerName.mysite.com and log them in there
Run a separate instance of Meteor configured for a different mongo at each site
nginx might help you with the above.
It is generally good practice to run separate DBs when offering a B2B
solution.
That's a matter of opinion that depends heavily on the platform. Many SaaS providers would argue that point.
I've implemented a replica set that I'm using globally. I have my master in Oregon, US and 4 secondaries. California and Virginia, Frankfurt and Sydney. I also have web servers in those same regions as well. Those web servers connect to mongo using mongoose:
var mongoose = require("mongoose");
var dbUrl = "mongodb://***.***.***.***:27017,***.***.***.***:27017,***.***.***.***:27017,***.***.***.***:27017,***.***.***.***:27017/exampleDb";
var dbOptions : {
"replSet": {
"rs_name": "exampleRepSet",
"readPreference": "ReadPreference.SECONDARY_PREFERRED",
"read_preference": "ReadPreference.SECONDARY_PREFERRED",
"w":0,
"slaveOk": true
}
}
mongoose.connect(dbUrl, dbOptions);
My problem is that my client's have higher latency to the database depending on how far away they are from the master. California get 40ms while Sydney gets 400ms. I don't understand why this is happening since they should be reading off of the secondary database in their region.
I understand that writes must be done to the primary but even if I perform a find then shouldn't it be done on the regional secondary and return pretty quick?
I realize there are some redundant options in that config but I'm getting desperate. I've also tried the option "ReadPreference.NEAREST" to no avail.
Try using the following options:
var mongoose = require("mongoose");
var dbUrl = "mongodb://***.***.***.***:27017,***.***.***.***:27017,***.***.***.***:27017,***.***.***.***:27017,***.***.***.***:27017/exampleDb";
mongoose.connect(dbUrl, {
server: {
readPreference: "nearest",
strategy: "ping"
},
replset: {
rs_name: "exampleRepSet",
readPreference: "nearest",
strategy: "ping"
}
});
Whilst the documentation specifies ping as the default strategy, it seems Mongoose mandates that you specify one when you use readPreference.
Also note that secondaryPreferred is not the same thing as nearest. secondaryPreferred prefers reads off of secondary members (as the name suggests) regardless of network latency, where nearest prioritizes reads to the member with the lowest amount of network latency.
Short of a misconfiguration in your replica set, make sure your primary is online and reachable - by default Mongoose will refuse to use a secondary if the primary is offline.
Edit
Try setting the read preference on the connection string itself with mongodb://connection/db/?readPreference=secondary and not in the dbOptions. I can't find anything in the node-mongodb-native that says a read preference can be added to the replset config. http://mongodb.github.io/node-mongodb-native/2.0/api/ReplSet.html
Old Answer
You may need to set the setting to nearest and not secondary preferred. http://docs.mongodb.org/manual/reference/read-preference/#nearest
dbOptions.db = {
readPreference: "secondaryPreferred",
};
This worked for me
I'm just getting started with mongoDB. I am trying to understand how to set up my secondary database servers so that when there is no primary, the secondaries can be used to read data. I believe the read preference I'm going for is preferredPrimary.
Now that I kinda understand which of the read preferences I want to test out, I'm trying to understand how to set up my replica set for preferredPrimary.
I've been reading through the following documentation:
http://docs.mongodb.org/manual/tutorial/configure-replica-set-tag-sets/
Questions:
Is this the right doc to follow to set up read preferences?
Assuming that it is, I want to verify that the tags names / values are anything that I come up with? So specifically, the key used in the example "dc" is NOT a keyword in mongo. Is that correct?
once I set up these tags, in my client, when I'm connecting to the mongo database, do i have to specify any settings? I'm using a php front end, and I found this:
http://php.net/manual/en/mongodb.setreadpreference.php
can you confirm that these tags replace the rs.slaveOK() method?
Environment:
mongoDB version 2.6.5
replica set with 3 members - one primary and 2 secondary servers
Yes
Yes
Yes, but the link that you provided is only for readPreference
You also need to supply custom writeConcern (extract from link in a question):
db.users.insert( { id: "xyz", status: "A" }, { writeConcern: { w: "MultipleDC" } } )
Look into php driver documentation how to do that.
Yes, you may skip call to slaveOK in this case (especially, that in 95% cases you will be reading from primary)
I'm currently using MongoDB to record application logs, and while I'm quite happy with both the performance and with being able to dump arbitrary structured data into log records, I'm troubled by the mutability of log records once stored.
In a traditional database, I would structure the grants for my log tables such that the application user had INSERT and SELECT privileges, but not UPDATE or DELETE. Similarly, in CouchDB, I could write a update validator function that rejected all attempts to modify an existing document.
However, I've been unable to find a way to restrict operations on a MongoDB database or collection beyond the three access levels (no access, read-only, "god mode") documented in the security topic on the MongoDB wiki.
Has anyone else deployed MongoDB as a document store in a setting where immutability (or at least change tracking) for documents was a requirement? What tricks or techniques did you use to ensure that poorly-written or malicious application code could not modify or destroy existing log records? Do I need to wrap my MongoDB logging in a service layer that enforces the write-only policy, or can I use some combination of configuration, query hacking, and replication to ensure a consistent, audit-able record is maintained?
I would say the best bet would be to wrap up the access to MongoDB in a service layer that enforces your specific contracts. We don't do much in the way of fine-grained access control because there are so many different cases that solving all of them correctly is tricky to get right. So for the most part it's up to the application layer to implement those kind of controls.
To add a write only (on collection level) user to the MongoDB do the following.
Say you want to have a user that can only write (insert only) to a certain collection on a certain database.
Create a file createuser.js with following contents:
function createCollectionWriter(database, username, password, rolename, collection)
{
db.getSiblingDB(database).createUser({
user : username,
pwd : password,
roles : [ ]
});
db.getSiblingDB(database).createRole({
role : rolename,
privileges : [
{
resource : { db : database, "collection" : collection },
actions : [ "insert" ]
}
],
roles : []
});
db.getSiblingDB(database).grantRolesToUser(username, [ rolename ] );
}
And execute this from command-line
$ mongo --eval="load('createuser.js'); createCollectionWriter('yourdb', 'user1', 'pass1', 'rolename', 'col1')"
This creates a user with username user1 with password pass1 and this user has write only access to database's yourdb collection col1.
A side-effect of this is that role rolename is created. If you have existing user that should have a write only access to the same collection, grant role rolename to that existing user.
Feel free to use and modify the code provided above :).
In MongoDB 1.3.2+ you can add some restriction in user :
db.addUser("guest", "passwordForGuest", true)
But it's only existing now not better. Maybe you can add some feature request
see information in MongoDB documentation : http://www.mongodb.org/display/DOCS/Security+and+Authentication