I know we can use Mongo shell to enable(or disable) change streams for Amazon Document DB. Is it possible to enable the change streams from AWS Console or MongoDB Driver?
AWS Console: No. I don't think this is controlled by DocumentDB cluster params.
MongoDB Drivers: Yes
DocumentDB Change Streams uses runCommand method to enable and disable changestreams.
In the documentation, the command used is adminCommand which is just calling runCommand on admin database. Since almost all drivers support running commands over database, you should be able to enable/disable change streams using any driver.
Following code uses pymongo to enable changestreams:
>>> from pymongo import MongoClient
>>> client = MongoClient("mongodb://<username>:<password>#xxxxxx.xxxxxx.us-east-1.docdb.amazonaws.com:27017/test_db?ssl=true&ssl_ca_certs=rds-combined-ca-bundle.pem&replicaSet=rs0&readPreference=secondaryPreferred")
>>> client['admin'].command('aggregate', 1, pipeline=[{'$listChangeStreams': 1}], cursor={})
{'waitedMS': 0, 'cursor': {'firstBatch': [], 'id': 0, 'ns': 'admin.$cmd'}, 'ok': 1.0}
>>> client['admin'].command('modifyChangeStreams', 1, database='bar', collection='foo', enable=True)
{'ok': 1.0}
>>> client['admin'].command('aggregate', 1, pipeline=[{'$listChangeStreams': 1}], cursor={})
{'waitedMS': 0, 'cursor': {'firstBatch': [{'database': 'bar', 'collection': 'foo'}], 'id': 0, 'ns': 'admin.$cmd'}, 'ok': 1.0}
You can enable changestreams on your Amazon DocumentDB cluster using the modifyChangeStream API. To enable/disable changestreams using the mongo shell -
//Enable change streams for the collection "foo" in database "bar"
db.adminCommand({modifyChangeStreams: 1,
database: "bar",
collection: "foo",
enable: true});
//Enable change streams for all collections in database "bar"
db.adminCommand({modifyChangeStreams: 1,
database: "bar",
collection: "",
enable: true});
//Enable change streams for all collections in all databases in a cluster
db.adminCommand({modifyChangeStreams: 1,
database: "",
collection: "",
enable: true});
To enable change streams from your application, you can use #krg265's suggestion.
Related
Created an AWS DMS pipeline:
Source endpoint - MongoDB
Target endpoint - RDS Postgres SQL
Successfully did all the security configuration, and both endpoints returned successful while testing it.
For the MongoDB source, I am using one of the three replicas sets with a username and a password that is not the admin username.
I also added the privilege "changeStream" in the replica set user.
But when starting the DMS migration task getting this error in cloud watch.
Encountered an error while initializing change stream: 'not authorized on admin to execute command
{ aggregate: 1, pipeline: [ { $changeStream: { fullDocument: "updateLookup", startAtOperationTime: Timestamp(1656005815, 0),
allChangesForCluster: true } }, "ok" : { "$numberDouble" : "0.0" },
"errmsg" : "not authorized on admin to execute command { aggregate: 1, pipeline: [ { $changeStream: { fullDocument:
\"updateLookup\", startAtOperationTime: Timestamp(1656005815, 0), allChangesForCluster: true } },
74f1-4aab-9ca1-f964ab655777\ (change_streams_capture.c:356)
Assuming this is due to some missing privileges in mongo replica sets USER.
I would like to know, is there a way we can replicate from one mongo replica set to another via mongo-connector? As per mongo documentation we can connect two mongo instances via mongo-connector by using a command as in the example below, but I would like to pass replica set name or use a configuration file instead of passing server:port name in command line.
Mongo Connector can replicate from one MongoDB replica set or sharded cluster to another using the Mongo DocManager. The most basic usage is like the following:
mongo-connector -m localhost:27017 -t localhost:37017 -d mongo_doc_manager
I also tried config.json option by creating below config.json file but it has failed.
{
"__comment__": "Configuration options starting with '__' are disabled",
"__comment__": "To enable them, remove the preceding '__'",
"mainAddress": "localhost:27017",
"oplogFile": "C:\Dev\mongodb\mongo-connector\oplog.timestamp",
"verbosity": 2,
"continueOnError": false,
"logging": {
"type": "file",
"filename": "C:\Dev\mongodb\mongo-connector\mongo-connector.log",
"__rotationWhen": "D",
"__rotationInterval": 1,
"__rotationBackups": 10,
"__type": "syslog"
},
"docManagers": [
{
"docManager": "mongo_doc_manager",
"targetURL": "localhost:37010",
"__autoCommitInterval": null
}
]
}
yes its possible to connect to a replica set or a shard server using mongo connector.
{
mongo-connector -m <mongodb server hostname>:<replica set port> \
-t <replication endpoint URL, e.g. http://localhost:8983/solr> \
-d <name of doc manager, e.g., solr_doc_manager>
}
you can also also pass a connection string to the mongo-connector such as
{
mongo connector -m mongodb://db1.example.net,db2.example.net:2500/?replicaSet=test&connectTimeoutMS=300000
}
to specify specifc config files you can use
{ mongo-connector -c config.json }
where config.json is your config file.
I'm able to resolve my issue by entering backslash '\' for my windows directory path.Here is my updated config file for reference. Thanks to ShaneHarveyNot able to use Configuration file for connecting to mongo-connector
{
"__comment__": "Configuration options starting with '__' are disabled",
"__comment__": "To enable them, remove the preceding '__'",
"mainAddress": "localhost:27017",
"oplogFile": "C:\\Dev\\mongodb\\mongo-connector\\oplog.timestamp",
"noDump": false,
"batchSize": -1,
"verbosity": 2,
"continueOnError": false,
"logging": {
"type": "file",
"filename": "C:\\Dev\\mongodb\\mongo-connector\\mongo-connector.log",
"__format": "%(asctime)s [%(levelname)s] %(name)s:%(lineno)d - %(message)s",
"__rotationWhen": "D",
"__rotationInterval": 1,
"__rotationBackups": 10,
"__type": "syslog",
"__host": "localhost:27017"
},
"docManagers": [
{
"docManager": "mongo_doc_manager",
"targetURL": "localhost:37017",
"__autoCommitInterval": null
}
]
}
When I execute "db.abc.find()" in mongo shell it returns the following answer:
db.abc.find()
{ "_id" : ObjectId("56a942bfec926681f17f09b6"), "name" : "foo" }
But when I execute the same command via PyMongo's eval method I receive a different answer:
>>> from pymongo import MongoClient
>>> client = MongoClient()
>>> db = client.test
>>> db.eval('db.abc.find()')
{u'_special': False, u'_options': 0.0, u'_ns': u'test.abc', u'_db': {u'_mongo': {u'slaveOk': False, u'host': u'EMBEDDED'}, u'_name': u'test'}, u'_skip': 0.0, u'_numReturned': 0.0, u'_query': {}, u'_limit': 0.0, u'_mongo': {u'slaveOk': False, u'host': u'EMBEDDED'}, u'_collection': {u'_shortName': u'abc', u'_db': {u'_mongo': {u'slaveOk': False, u'host': u'EMBEDDED'}, u'_name': u'test'}, u'_mongo': {u'slaveOk': False, u'host': u'EMBEDDED'}, u'_fullName': u'test.abc'}, u'_cursor': None, u'_fields': None, u'_batchSize': 0.0}
What this is happening?
How to fix it?
Well, eval() is definitely evil and is actually deprecated.
You should wrap the code into a function to make it work:
from bson import Code
db.eval(Code('function () { return db.abc.find(); }'))
Why don’t you just call db.abc.find() from Python and get you document(s) from cursor?
[doc for doc in db.abc.find()]
Or still:
db.abc.find_one()
I have created a user "mongo01testro" in the mongo01test database.
use mongo01test
db.addUser( "mongo01testro", "pwd01", true );
db.system.users.find();
{ "_id" : ObjectId("53xyz"), "user" : "mongo01testro", "readOnly" : true, "pwd" : "b9eel61" }
When I logged in from another session as this newly created user,
I am able to insert documents into the collection which is strange.
I am looking to do the following:
Create 2 separate users one for read only and one for read write for
each database.
Create an admin user which have sysadmin/dba access to all the
databases in MongoDB instance used for Backup/Recovery or admin
purpose.
Please kindly help.
Regards,
Parag
You forgot --auth to enable
Create Users
// ensure that we have new db, no roles, no users
use products
db.dropDatabase()
// create admin user
use products
db.createUser({
"user": "prod-admin",
"pwd": "prod-admin",
"roles": [
{"role": "clusterAdmin", "db": "admin" },
{"role": "readAnyDatabase", "db": "admin" },
"readWrite"
]},
{ "w": "majority" , "wtimeout": 5000 }
)
// login via admin acont in order to create readonly user
// mongo --username=prod-admin --password=prod-admin products
db.createUser({
"user": "prod-r",
"pwd": "prod-r",
"roles": ["read"]
})
Enable auth:
sudo vim /etc/mongod.conf # edit file
# Turn on/off security. Off is currently the default
#noauth = true
auth = true
sudo service mongod restart # reload configuiration
Check write restriction:
# check if write operation for readonly user
$ mongo --username=prod-r --password=prod-r products
MongoDB shell version: 2.6.4
connecting to: products
> db.laptop.insert({"name": "HP"})
WriteResult({
"writeError" : {
"code" : 13,
"errmsg" : "not authorized on products to execute command { insert: \"laptop\", documents: [ { _id: ObjectId('53ecb7115f0bfc61d8b1113e'), name: \"HP\" } ], ordered: true }"
}
})
# check write operation for admin user
$ mongo --username=prod-admin --password=prod-admin products
MongoDB shell version: 2.6.4
connecting to: products
> db.laptop.insert({"name": "HP"})
WriteResult({ "nInserted" : 1 })
# check read operation for readonly user
$ mongo --username=prod-r --password=prod-r products
MongoDB shell version: 2.6.4
connecting to: products
> db.laptop.findOne()
{ "_id" : ObjectId("53ecb54798da304f99625d05"), "name" : "HP" }
MongoDB changed the way it handles users in versions >= 2.2 and 2.6 and if you are updating mongodb you will have to Upgrade User Authorization Data to 2.6 Format.
In versions < 2.2 (legacy) you use the db.addUser(username, password, readOnly) function as #Hüseyin BABAL suggested. If you are using version > 2.2 or 2.6 you have a lot more control over what you can do with roles and privileges.
So assuming you use mongo v > 2.6 you can create something like:
use admin
db.createUser({
user: "myUsername",
pwd: "mypwd",
roles: [{
role: "userAdminAnyDatabase",
db: "admin"
}]
})
db.addUser({
user: "username",
pwd: "password",
roles: [{
role: "readWrite",
db: "myDBName"
}]
})
You can use the Build in Roles or you can even create custom roles.
So as you can see you clearly have a lot more options when using v > 2.6 when it comes to authentication and role management.
if the Access control is enabled on the MongoDB deployment, Then you should login by authenticating to the relevant database with admin user (or user with userAdmin role) which you want to control the access. to do that
mongo <db_name> -u <user_name> -p <user_password>
ex: mongo mongo01test -u admin -p admin_paaswrod
then execute the following query to create a read only user for current connected database,
db.createUser({user:"user_name",pwd:"password",roles:[{role:"read", db:"db_name"}]});
ex:db.createUser({user:"user_rd",pwd:"password",roles:[{role:"read", db:"mongo01test"}]});
create a user with both readWrite access,
db.createUser({user:"user_name",pwd:"password",roles:[{role:"readWrite", db:"db_name"}]});
ex:db.createUser({user:"user_rw",pwd:"pass_rw",roles:[{role:"readWrite", db:"mongo01test"}]});
to create a user with admin privileges,
use admin;
db.createUser({user:"admin_user_name",pwd:"password", roles:[{role:"dbAdmin", db:"admin"},{role:"readWriteAnyDatabase", db:"admin"},{role:"backup", db:"admin"},{role:"restore", db:"admin"}]});
Here is my scneario;
// Create admin user
use admin
db.addUser('root', 'strong_password');
// Read only user
use dbforuser1
db.addUser('user1', 'user1_pass', true);
// Read / Write access
use dbforuser2
db.addUser('user2', 'user2_pass');
If you want to login to dbforuser1
use dbforuser1
db.auth('user1', 'user1_pass')
I have a db with a bunch of collections.
Some of them have 'status' and some don't.
How can I insert 'status':'pending' into the collections that lack 'status' - but not overwrite the collections that already have a status?
Using pymongo / flask / python 2.7
I tried this:
orders = monDB.find('order')
for order in orders:
if not order['status']:
monDB.update('order', {'status':'pending'})
print 'success'
But nothing happens. What am I doing wrong?
Use $exists to check if the field exists and $set to create it if it doesn't. Assuming monDB is your collection:
monDB.update({'status': {'$exists' : False}}, {'$set' : {'status': 'pending'}}, multi=True)
In the mongo shell:
> db.myQueue.update({status: {$exists: false}}, {$set : {status: 'pending'}}, false, true)
See http://docs.mongodb.org/manual/reference/operators/ and http://docs.mongodb.org/manual/applications/update/#crud-update-update.