Mongorestore not restoring data - mongodb

I have an existing mongodump of a single collection that I am trying to restore. After running mongo restore, no errors show up and the data is not in the collection. Are there any known reasons how this could happen? I would expect if the data weren't inserted for some reason, an error would be provided in the log.
To create and attempt to restore the dump, I followed the answer provided for this question:
How to use mongodump for 1 collection
I've created a new database on a different server and it has an empty collection. I've checked the mongo log file and there are no errors, it shows the connection open and authenticate, then disconnect on the next line.
mongorestore -vvvvv -u user -p 'password' --db=MyDatabase --collection=MyCollection dump1/MyCollection.bson
2015-03-04T18:20:31.331+0000 creating new connection to:127.0.0.1:27017
2015-03-04T18:20:31.332+0000 [ConnectBG] BackgroundJob starting: ConnectBG
2015-03-04T18:20:31.332+0000 connected to server 127.0.0.1:27017 (127.0.0.1)
2015-03-04T18:20:31.332+0000 connected connection!
connected to: 127.0.0.1
2015-03-04T18:20:31.333+0000 drillDown: dump1/MyCollection.bson
2015-03-04T18:20:31.333+0000 dump1/MyCollection.bson
2015-03-04T18:20:31.333+0000 going into namespace [MyDatabase.MyCollection]
Restoring to MyDatabase.MyCollection without dropping. Restored data will be inserted without raising errors; check your server log
file size: 94876
130 objects found
2015-03-04T18:20:31.336+0000 Creating index: { key: { _id: 1 }, name: "_id_", ns: "MyDatabase.MyCollection" }
2015-03-04T18:20:31.340+0000 Creating index: { key: { geometry: "2dsphere" }, name: "geometry_2dsphere", ns: "MyDatabase.MyCollection", 2dsphereIndexVersion: 2 }
Log file:
2015-03-04T18:20:31.333+0000 [conn874] authenticate db: MyDatabase { authenticate: 1, nonce: "xxx", user: "user", key: "xxx" }
2015-03-04T18:20:31.342+0000 [conn874] end connection 127.0.0.1:59420 (25 connections now open)
The query I am using on the origin and destination is:
db.MyCollection.find()
On the origin server, the collection has 130 elements, which is what is also shown in the mongorestore output "130 objects found".
Edit:
I added the --drop option to the mongorestore command. The log file output clearly shows that it is creating the index on an empty collection.
2015-03-20T15:03:57.565+0000 [conn61965] authenticate db: MyDatabase { authenticate: 1, nonce: "xxx", user: "user", key: "xxx" }
2015-03-20T15:03:57.566+0000 [conn61965] CMD: drop MyDatabase.MyCollection
2015-03-20T15:03:57.631+0000 [conn61965] build index on: MyDatabase.MyCollection properties: { v: 1, key: { _id: 1 }, name: "_id_", ns: "MyDatabase.MyCollection" }
2015-03-20T15:03:57.631+0000 [conn61965] added index to empty collection
2015-03-20T15:03:57.652+0000 [conn61965] build index on: MyDatabase.MyCollection properties: { v: 1, key: { geometry: "2dsphere" }, name: "geometry_2dsphere", ns: "MyDatabase.MyCollection", 2dsphereIndexVersion: 2 }
2015-03-20T15:03:57.652+0000 [conn61965] added index to empty collection
2015-03-20T15:03:57.654+0000 [conn61965] end connection 127.0.0.1:59456 (21 connections now open)

So the issue ended up being that the user I was trying to do the restore with only had the read and dbAdmin roles. I had made a separate user so that the regular user used by the application did not have administrative rights. After changing my user's role from read to readWrite, it worked as expected.
To be honest, if the user didn't have the correct permissions, I really would have expected the log to show an error of some sort when it tries to run the restore without the correct permission.

Related

Ongoing replication from MongoDB to RDS PostgresSQL

Created an AWS DMS pipeline:
Source endpoint - MongoDB
Target endpoint - RDS Postgres SQL
Successfully did all the security configuration, and both endpoints returned successful while testing it.
For the MongoDB source, I am using one of the three replicas sets with a username and a password that is not the admin username.
I also added the privilege "changeStream" in the replica set user.
But when starting the DMS migration task getting this error in cloud watch.
Encountered an error while initializing change stream: 'not authorized on admin to execute command
{ aggregate: 1, pipeline: [ { $changeStream: { fullDocument: "updateLookup", startAtOperationTime: Timestamp(1656005815, 0),
allChangesForCluster: true } }, "ok" : { "$numberDouble" : "0.0" },
"errmsg" : "not authorized on admin to execute command { aggregate: 1, pipeline: [ { $changeStream: { fullDocument:
\"updateLookup\", startAtOperationTime: Timestamp(1656005815, 0), allChangesForCluster: true } },
74f1-4aab-9ca1-f964ab655777\ (change_streams_capture.c:356)
Assuming this is due to some missing privileges in mongo replica sets USER.

MongoDB - Collection-level roles

I have a trouble.
I'm trying to apply a role to user.
I want the user only find, insert, remove and update specific collection
I'm doing this.
Create role:
db.createRole({role:'user_role_col',privileges:[{resource:{db:'something_else', collection: 'col_something_else'}, actions: ['find','remove','insert','update']}], roles: []});
.
Create user with that role:
db.createUser({user: 'user_col',pwd: '1234',roles: [{role: "user_role_col", db: "something_else"}]})
.
When I created role and user, I stay on something_else database (use something_else)
0 errors got, but I can only read col_something_else, I cannot remove, update, or insert :(
What am I doing wrong?
I believe the following commands replicates what you did. Here I'm creating the role and user in the test database:
> use test
> db.createRole({role: 'testRole', privileges: [{resource: {db:'test', collection:'test'}, actions:['find','update','insert','remove']}], roles: []})
> db.createUser({user:'testUser',pwd:'password',roles:[{role:'testRole', db:'test'}]})
I would then have to quit the mongo shell and re-authenticate using the new credentials. However, the authenticationDatabase parameter must point to the test database, since that's where I created the role and the user:
$ mongo -u testUser -p password --authenticationDatabase test
> db.test2.insert({a:1}) //try inserting to a non-"test" collection
unauthorized
> db.test2.find() //try find on a non-"test" collection
unauthorized
> db.test.insert({a:1}) //insert into "test" collection
Inserted 1 record(s)
> db.test.updateMany({},{$set:{a:2}}) //update "test" collection
{
"acknowledged": true,
"matchedCount": 1,
"modifiedCount": 1
}
> db.test.find() //find on "test" collection
{
"_id": ObjectId("5a0bcfa4322032cfcc3a69c6"),
"a": 2
}
> db.test.deleteOne({a:2}) //delete from "test" collection
{
"acknowledged": true,
"deletedCount": 1
}
> db.test.drop() //try to drop "test" collection
2017-11-15T16:32:07.715+1100 E QUERY [thread1] Error: drop failed: {
"ok": 0,
"errmsg": "not authorized on test to execute command { drop: \"test\" }",
"code": 13,
"codeName": "Unauthorized"
}
I found that the new custom role (testRole) is authorized correctly. This is using MongoDB 3.4.10.

mongodb authorization exception in JMeter : code13

In MongoDB 3.2 I've setup a user with rights:
db.createUser(
{
user: "username",
pwd: "pass",
roles: [ { role: "readWrite", db: "dbname" }]
}
)
db.auth("username", "pass" )
When I use the JMeter(2.13) to connect to the database (using Jmeter's elements MongoDB Source Config , MongoDB Script) and run a query like this:
db.mycollectionname.find()
I get this error:
error: { "$err" : "not authorized on dbname to execute command { $eval: \"db.mycollectionname.find()\", args: [] }" , "code" : 13}
While I have provided all the necessary details Server Address List , Database , User , Password to Jmeter's MongoDB Source Config , MongoDB Script respectively.
Any ideas what can be happening?
I had the same issue. I had to set up a user with eval permissions even though this is not recommended (even the admin user does not have these permissions).
Try that and change you script to look at the new user and it should work.

mongorestore not working. collection is empty

i am trying to dump a mongodb collection to file, and then use that to restore to another mongodb instance.
dumping -
mongodump --host 127.0.0.1 --port 27017 --username vespauser --password <passwd> --collection vespastats --db vespa --out /archive/vespa-archive/vespa-db-backup_001
connected to: 127.0.0.1:27017
2015-04-21T16:24:07.070-0400 DATABASE: vespa to /archive/vespa-archive/vespa-db-backup_testing01/vespa
2015-04-21T16:24:07.141-0400 vespa.system.indexes to /archive/vespa-archive/vespa-db-backup_testing01/vespa/system.indexes.bson
2015-04-21T16:24:07.148-0400 4 documents
2015-04-21T16:24:07.149-0400 vespa.vespastats to /archive/vespa-archive/vespa-db-backup_testing01/vespa/vespastats.bson
2015-04-21T16:24:07.316-0400 59724 documents
2015-04-21T16:24:08.118-0400 Metadata for vespa.vespastats to /archive/vespa-archive/vespa-db-backup_testing01/vespa/vespastats.metadata.json
restoring -
mongorestore -v --drop --host 127.0.0.1 --port 27017 --username admin --password <passwd> /archive/vespa-archive/vespa-db-backup_001
2015-04-21T16:31:11.962-0400 creating new connection to:127.0.0.1:27017
2015-04-21T16:31:11.963-0400 [ConnectBG] BackgroundJob starting: ConnectBG
2015-04-21T16:31:11.963-0400 connected to server 127.0.0.1:27017 (127.0.0.1)
2015-04-21T16:31:11.963-0400 connected connection!
connected to: 127.0.0.1:27017
2015-04-21T16:31:11.966-0400 /home/amurty/vespa-db/vespa-db-backup_testing01/vespa/vespastats.bson
2015-04-21T16:31:11.966-0400 going into namespace [vespa.vespastats]
2015-04-21T16:31:11.966-0400 dropping
file size: 88808161
59724 objects found
2015-04-21T16:31:13.730-0400 Creating index: { key: { _id: 1 }, name: "_id_", ns: "vespa.vespastats" }
2015-04-21T16:31:13.848-0400 Creating index: { key: { url: 1 }, name: "url_1", ns: "vespa.vespastats", background: true }
2015-04-21T16:31:13.858-0400 Creating index: { key: { r_tstpm: 1 }, name: "r_tstpm_1", ns: "vespa.vespastats", background: true }
2015-04-21T16:31:13.859-0400 Creating index: { key: { url: 1, r_tstpm: 1 }, name: "url_1_r_tstpm_1", ns: "vespa.vespastats", background: true }
from /var/log/mongodb/mongod.log -
2015-04-21T16:31:11.963-0400 [initandlisten] connection accepted from 127.0.0.1:58444 #23 (1 connection now open)
2015-04-21T16:31:11.964-0400 [conn23] authenticate db: admin { authenticate: 1, nonce: "xxx", user: "admin", key: "xxx" }
2015-04-21T16:31:11.968-0400 [conn23] CMD: drop vespa.vespastats
2015-04-21T16:31:13.757-0400 [conn23] allocating new ns file /var/lib/mongo/vespa.ns, filling with zeroes...
2015-04-21T16:31:13.838-0400 [FileAllocator] allocating new datafile /var/lib/mongo/vespa.0, filling with zeroes...
2015-04-21T16:31:13.846-0400 [FileAllocator] done allocating datafile /var/lib/mongo/vespa.0, size: 64MB, took 0.007 secs
2015-04-21T16:31:13.847-0400 [conn23] build index on: vespa.vespastats properties: { v: 1, key: { _id: 1 }, name: "_id_", ns: "vespa.vespastats" }
2015-04-21T16:31:13.848-0400 [conn23] added index to empty collection
2015-04-21T16:31:13.857-0400 [conn23] build index on: vespa.vespastats properties: { v: 1, key: { url: 1 }, name: "url_1", ns: "vespa.vespastats", background: true }
2015-04-21T16:31:13.857-0400 [conn23] added index to empty collection
2015-04-21T16:31:13.858-0400 [conn23] build index on: vespa.vespastats properties: { v: 1, key: { r_tstpm: 1 }, name: "r_tstpm_1", ns: "vespa.vespastats", background: true }
2015-04-21T16:31:13.859-0400 [conn23] added index to empty collection
2015-04-21T16:31:13.860-0400 [conn23] build index on: vespa.vespastats properties: { v: 1, key: { url: 1, r_tstpm: 1 }, name: "url_1_r_tstpm_1", ns: "vespa.vespastats", background: true }
2015-04-21T16:31:13.860-0400 [conn23] added index to empty collection
2015-04-21T16:31:13.862-0400 [conn23] end connection 127.0.0.1:58444 (0 connections now open)
now when i login to my new mongodb instance and check collection size, i get a big 0 -
# mongo
MongoDB shell version: 2.6.9
connecting to: test
> use vespa
switched to db vespa
> db.auth('vespauser', '<paswd>')
1
> db.vespastats.find()
> db.vespastats.count()
0
>
Collection may or may not exist in the used database but the query is not returning an error, just 0.
db.vespastats.find().count()
The issue should be because it is added to database test. (doc mentions it should be automatic but I was able to reproduce this behaviour).
Therefore
use test
db.vespastats.find().count()
would have returned the actual documents in the collection vespastats.
The issue is caused by not specifying db name when using mongo binary command mongorestore. doc for mongorestore mongorestore --nsInclude=vesta.vestastats should be the updated version (even if -d still works).
To know where the collection would land, I would run 2 times the restore dump and check show dbs in mongo shell 3 times (before and after) > the db size is changing (not immediately though as it may show 8kb right after the restoration).

Create Read only user in Mongo DB Instance for a particular database

I have created a user "mongo01testro" in the mongo01test database.
use mongo01test
db.addUser( "mongo01testro", "pwd01", true );
db.system.users.find();
{ "_id" : ObjectId("53xyz"), "user" : "mongo01testro", "readOnly" : true, "pwd" : "b9eel61" }
When I logged in from another session as this newly created user,
I am able to insert documents into the collection which is strange.
I am looking to do the following:
Create 2 separate users one for read only and one for read write for
each database.
Create an admin user which have sysadmin/dba access to all the
databases in MongoDB instance used for Backup/Recovery or admin
purpose.
Please kindly help.
Regards,
Parag
You forgot --auth to enable
Create Users
// ensure that we have new db, no roles, no users
use products
db.dropDatabase()
// create admin user
use products
db.createUser({
"user": "prod-admin",
"pwd": "prod-admin",
"roles": [
{"role": "clusterAdmin", "db": "admin" },
{"role": "readAnyDatabase", "db": "admin" },
"readWrite"
]},
{ "w": "majority" , "wtimeout": 5000 }
)
// login via admin acont in order to create readonly user
// mongo --username=prod-admin --password=prod-admin products
db.createUser({
"user": "prod-r",
"pwd": "prod-r",
"roles": ["read"]
})
Enable auth:
sudo vim /etc/mongod.conf # edit file
# Turn on/off security. Off is currently the default
#noauth = true
auth = true
sudo service mongod restart # reload configuiration
Check write restriction:
# check if write operation for readonly user
$ mongo --username=prod-r --password=prod-r products
MongoDB shell version: 2.6.4
connecting to: products
> db.laptop.insert({"name": "HP"})
WriteResult({
"writeError" : {
"code" : 13,
"errmsg" : "not authorized on products to execute command { insert: \"laptop\", documents: [ { _id: ObjectId('53ecb7115f0bfc61d8b1113e'), name: \"HP\" } ], ordered: true }"
}
})
# check write operation for admin user
$ mongo --username=prod-admin --password=prod-admin products
MongoDB shell version: 2.6.4
connecting to: products
> db.laptop.insert({"name": "HP"})
WriteResult({ "nInserted" : 1 })
# check read operation for readonly user
$ mongo --username=prod-r --password=prod-r products
MongoDB shell version: 2.6.4
connecting to: products
> db.laptop.findOne()
{ "_id" : ObjectId("53ecb54798da304f99625d05"), "name" : "HP" }
MongoDB changed the way it handles users in versions >= 2.2 and 2.6 and if you are updating mongodb you will have to Upgrade User Authorization Data to 2.6 Format.
In versions < 2.2 (legacy) you use the db.addUser(username, password, readOnly) function as #Hüseyin BABAL suggested. If you are using version > 2.2 or 2.6 you have a lot more control over what you can do with roles and privileges.
So assuming you use mongo v > 2.6 you can create something like:
use admin
db.createUser({
user: "myUsername",
pwd: "mypwd",
roles: [{
role: "userAdminAnyDatabase",
db: "admin"
}]
})
db.addUser({
user: "username",
pwd: "password",
roles: [{
role: "readWrite",
db: "myDBName"
}]
})
You can use the Build in Roles or you can even create custom roles.
So as you can see you clearly have a lot more options when using v > 2.6 when it comes to authentication and role management.
if the Access control is enabled on the MongoDB deployment, Then you should login by authenticating to the relevant database with admin user (or user with userAdmin role) which you want to control the access. to do that
mongo <db_name> -u <user_name> -p <user_password>
ex: mongo mongo01test -u admin -p admin_paaswrod
then execute the following query to create a read only user for current connected database,
db.createUser({user:"user_name",pwd:"password",roles:[{role:"read", db:"db_name"}]});
ex:db.createUser({user:"user_rd",pwd:"password",roles:[{role:"read", db:"mongo01test"}]});
create a user with both readWrite access,
db.createUser({user:"user_name",pwd:"password",roles:[{role:"readWrite", db:"db_name"}]});
ex:db.createUser({user:"user_rw",pwd:"pass_rw",roles:[{role:"readWrite", db:"mongo01test"}]});
to create a user with admin privileges,
use admin;
db.createUser({user:"admin_user_name",pwd:"password", roles:[{role:"dbAdmin", db:"admin"},{role:"readWriteAnyDatabase", db:"admin"},{role:"backup", db:"admin"},{role:"restore", db:"admin"}]});
Here is my scneario;
// Create admin user
use admin
db.addUser('root', 'strong_password');
// Read only user
use dbforuser1
db.addUser('user1', 'user1_pass', true);
// Read / Write access
use dbforuser2
db.addUser('user2', 'user2_pass');
If you want to login to dbforuser1
use dbforuser1
db.auth('user1', 'user1_pass')