Mongo show collections on remote server works on windows but stucks on ubuntu - mongodb

I am in a strange situation with mongodb remote connection.
I have a Mongo DB on an ec2 instance and has been configured with password auth.
When I connect it and browse collections on a windows platform it works fine but when I try to do the same on Amazon ec2(Ubuntu) it just gets stuck.
Though I am able to connect on Ubuntu instance but when I try to run the commmand show collections.The server just hangs.
Windows-
E:\check\scheduler\trunk>mongo 54.xxx.xx.xx:27017/vdb -u vadmin -p db#dummypass
MongoDB shell version: 2.6.4
connecting to: 54.xxx.xx.xx:27017/vdb
> use vdb;
switched to db vdb
> show collections;
activity
user
adminpush
adminuserlog
friend
demo
addressbook
branddemovideo
Ubuntu(Ec2 Instance )
ubuntu#ip-10-146-147-102:~$ mongo 54.xxx.xx.xx:27017/vdb -u vadmin -p 'db#dummypass'
MongoDB shell version: 2.6.5
connecting to: 54.xxx.xx.xx:27017/vdb
> show databases;
admin 0.078125GB
local 0.078125GB
vdb 1.49951171875GB
> use vdb;
switched to db vdb
> show collections;
Its Just Stuck over here.
Finally after some time I got to see an error-
Mon Oct 20 19:52:51.949 Socket recv() errno:104 Connection reset by peer 54.xxx.xx.xx:27017
Mon Oct 20 19:52:51.954 SocketException: remote: 54.xxx.xx.xx:27017 error: 9001 socket exception [RECV_ERROR] server [54.xxx.xx.xx:27017]
Mon Oct 20 19:52:51.954 DBClientCursor::init call() failed
Mon Oct 20 19:52:51.955 Error: error doing query: failed at src/mongo/shell/query.js:78
Mon Oct 20 19:52:51.955 trying reconnect to 54.xxx.xx.xx:27017
Mon Oct 20 19:52:51.969 reconnect 54.xxx.xx.xx:27017 ok
Mongo Logs given below-
Local Server
ubuntu#ip-172-xx-x-xx:/var/log/mongodb$ tail -f mongodb.log
2014-10-20T20:34:50.621+0530 [clientcursormon] connections:0
2014-10-20T20:39:50.633+0530 [clientcursormon] mem (MB) res:36 virt:342
2014-10-20T20:39:50.633+0530 [clientcursormon] mapped (incl journal view):160
2014-10-20T20:39:50.633+0530 [clientcursormon] connections:0
2014-10-20T20:44:50.645+0530 [clientcursormon] mem (MB) res:36 virt:342
2014-10-20T20:44:50.645+0530 [clientcursormon] mapped (incl journal view):160
2014-10-20T20:44:50.645+0530 [clientcursormon] connections:0
2014-10-20T20:49:50.658+0530 [clientcursormon] mem (MB) res:36 virt:342
2014-10-20T20:49:50.658+0530 [clientcursormon] mapped (incl journal view):160
2014-10-20T20:49:50.658+0530 [clientcursormon] connections:0
Remote Server-
2014-10-20T20:55:05.688+0530 [initandlisten] connection accepted from 54.xxx.xxx.xxx:57881 #42 (11 connections now open)
2014-10-20T20:55:05.692+0530 [conn42] authenticate db: vdb { authenticate: 1, nonce: "xxx", user: "vadmin", key: "xxx" }
2014-10-20T20:55:36.039+0530 [conn13] query vdb.video query: { query: { videoType: 5, viewType: 1, status: { $nin: [ 2, 5, 3 ] }, location.regionId: 2, frameHeight: { $gt: 0 } }, orderby: { popularRating: -1 } } planSummary: IXSCAN { location.regionId: 1 }, IXSCAN { location.regionId: 1 } cursorid:86724525795 ntoreturn:3 ntoskip:0 nscanned:2892 nscannedObjects:2892 keyUpdates:0 numYields:1 locks(micros) r:232390 nreturned:3 reslen:8101 116ms
2014-10-20T20:55:36.087+0530 [conn9] query vdb.video query: { query: { videoType: 5, viewType: 1, status: { $nin: [ 2, 5, 3 ] }, location.regionId: 2, frameHeight: { $gt: 0 } }, orderby: { popularRating: -1 } } planSummary: IXSCAN { location.regionId: 1 }, IXSCAN { location.regionId: 1 } cursorid:86710924707 ntoreturn:3 ntoskip:0 nscanned:2892 nscannedObjects:2892 keyUpdates:0 numYields:1 locks(micros) r:162238 nreturned:3 reslen:8101 112ms

Finally I resolved it. I was working in a VPC and had to use Private IP to connect to remote mongo as well as mysql.

Related

data lost in mongodb replica set mode

My replica set has two nodes:
1: the master node
2: a slave node with priority:0, votes:0
The oplog size is 5000MB.
run this for loop in master shell:
for (i=0;i<1000000;i++)
{
db.getSiblingDB("ff").c.insert(
{ a:i,
d:i+".#234"+(++i)+".234546"+(++i)+".568679"+(++i)+"31234."+(++i)+".12342354"+(++i)+"5346457."+(++i)+"33543465456."+(++i)+".6346456"+(++i)+"123235434."+(++i)+".2345345345"+(++i)
}
)
}
Kill the slave node while the for loop is running: kill -9 $(pidof slave_node)
Stop the for loop after a second; then restart the slave node.
Then run db.getSiblingDB("ff").c.count() to check data in both slave and master nodes, with the results:
master:20w
slave:15w
The slave node can catch up with the primary, but there is a lot of data lost from the slave.
Why is this?
Here is the slave node's log as it restarts after being killed:
2017-11-27T05:53:53.873+0000 I NETWORK [thread1] waiting for connections on port 28006
2017-11-27T05:53:53.876+0000 I REPL [replExecDBWorker-0] New replica set config in use: { _id: "cpconfig2", version: 2, protocolVersion: 1, members: [ { _id: 0, host: "127.0.0.1:28007", arbiterOnly: false, buildIndexes: true, hidden: false, priority: 3.0, tags: {}, slaveDelay: 0, votes: 1 }, { _id: 1, host: "127.0.0.1:28006", arbiterOnly: false, buildIndexes: true, hidden: false, priority: 0.0, tags: {}, slaveDelay: 0, votes: 0 } ], settings: { chainingAllowed: true, heartbeatIntervalMillis: 2000, heartbeatTimeoutSecs: 10, electionTimeoutMillis: 10000, catchUpTimeoutMillis: 60000, getLastErrorModes: {}, getLastErrorDefaults: { w: 1, wtimeout: 0 }, replicaSetId: ObjectId('5a1ba5bbb0a652502a5f002a') } }
2017-11-27T05:53:53.876+0000 I REPL [replExecDBWorker-0] This node is 127.0.0.1:28006 in the config
2017-11-27T05:53:53.876+0000 I REPL [replExecDBWorker-0] transition to STARTUP2
2017-11-27T05:53:53.876+0000 I REPL [replExecDBWorker-0] Starting replication storage threads
2017-11-27T05:53:53.877+0000 I REPL [replExecDBWorker-0] Starting replication fetcher thread
2017-11-27T05:53:53.877+0000 I REPL [replExecDBWorker-0] Starting replication applier thread
2017-11-27T05:53:53.877+0000 I REPL [replExecDBWorker-0] Starting replication reporter thread
2017-11-27T05:53:53.877+0000 I ASIO [NetworkInterfaceASIO-Replication-0] Connecting to 127.0.0.1:28007
2017-11-27T05:53:53.877+0000 I REPL [rsSync] transition to RECOVERING
2017-11-27T05:53:53.878+0000 I REPL [rsSync] transition to SECONDARY
2017-11-27T05:53:53.879+0000 I ASIO [NetworkInterfaceASIO-Replication-0] Successfully connected to 127.0.0.1:28007, took 2ms (1 connections now open to 127.0.0.1:28007)
2017-11-27T05:53:53.879+0000 I REPL [ReplicationExecutor] Member 127.0.0.1:28007 is now in state PRIMARY
2017-11-27T05:53:54.011+0000 I FTDC [ftdc] Unclean full-time diagnostic data capture shutdown detected, found interim file, some metrics may have been lost. OK
2017-11-27T05:53:54.645+0000 I NETWORK [thread1] connection accepted from 127.0.0.1:52404 #1 (1 connection now open)
2017-11-27T05:53:54.645+0000 I NETWORK [conn1] received client metadata from 127.0.0.1:52404 conn1: { driver: { name: "NetworkInterfaceASIO-Replication", version: "3.4.9" }, os: { type: "Linux", name: "PRETTY_NAME="Debian GNU/Linux 8 (jessie)"", architecture: "x86_64", version: "Kernel 3.10.0" } }
2017-11-27T05:53:59.878+0000 I REPL [rsBackgroundSync] sync source candidate: 127.0.0.1:28007
See the page Accuracy after Unexpected Shutdown for details and information on how to recover from this situation.

mongorestore not working. collection is empty

i am trying to dump a mongodb collection to file, and then use that to restore to another mongodb instance.
dumping -
mongodump --host 127.0.0.1 --port 27017 --username vespauser --password <passwd> --collection vespastats --db vespa --out /archive/vespa-archive/vespa-db-backup_001
connected to: 127.0.0.1:27017
2015-04-21T16:24:07.070-0400 DATABASE: vespa to /archive/vespa-archive/vespa-db-backup_testing01/vespa
2015-04-21T16:24:07.141-0400 vespa.system.indexes to /archive/vespa-archive/vespa-db-backup_testing01/vespa/system.indexes.bson
2015-04-21T16:24:07.148-0400 4 documents
2015-04-21T16:24:07.149-0400 vespa.vespastats to /archive/vespa-archive/vespa-db-backup_testing01/vespa/vespastats.bson
2015-04-21T16:24:07.316-0400 59724 documents
2015-04-21T16:24:08.118-0400 Metadata for vespa.vespastats to /archive/vespa-archive/vespa-db-backup_testing01/vespa/vespastats.metadata.json
restoring -
mongorestore -v --drop --host 127.0.0.1 --port 27017 --username admin --password <passwd> /archive/vespa-archive/vespa-db-backup_001
2015-04-21T16:31:11.962-0400 creating new connection to:127.0.0.1:27017
2015-04-21T16:31:11.963-0400 [ConnectBG] BackgroundJob starting: ConnectBG
2015-04-21T16:31:11.963-0400 connected to server 127.0.0.1:27017 (127.0.0.1)
2015-04-21T16:31:11.963-0400 connected connection!
connected to: 127.0.0.1:27017
2015-04-21T16:31:11.966-0400 /home/amurty/vespa-db/vespa-db-backup_testing01/vespa/vespastats.bson
2015-04-21T16:31:11.966-0400 going into namespace [vespa.vespastats]
2015-04-21T16:31:11.966-0400 dropping
file size: 88808161
59724 objects found
2015-04-21T16:31:13.730-0400 Creating index: { key: { _id: 1 }, name: "_id_", ns: "vespa.vespastats" }
2015-04-21T16:31:13.848-0400 Creating index: { key: { url: 1 }, name: "url_1", ns: "vespa.vespastats", background: true }
2015-04-21T16:31:13.858-0400 Creating index: { key: { r_tstpm: 1 }, name: "r_tstpm_1", ns: "vespa.vespastats", background: true }
2015-04-21T16:31:13.859-0400 Creating index: { key: { url: 1, r_tstpm: 1 }, name: "url_1_r_tstpm_1", ns: "vespa.vespastats", background: true }
from /var/log/mongodb/mongod.log -
2015-04-21T16:31:11.963-0400 [initandlisten] connection accepted from 127.0.0.1:58444 #23 (1 connection now open)
2015-04-21T16:31:11.964-0400 [conn23] authenticate db: admin { authenticate: 1, nonce: "xxx", user: "admin", key: "xxx" }
2015-04-21T16:31:11.968-0400 [conn23] CMD: drop vespa.vespastats
2015-04-21T16:31:13.757-0400 [conn23] allocating new ns file /var/lib/mongo/vespa.ns, filling with zeroes...
2015-04-21T16:31:13.838-0400 [FileAllocator] allocating new datafile /var/lib/mongo/vespa.0, filling with zeroes...
2015-04-21T16:31:13.846-0400 [FileAllocator] done allocating datafile /var/lib/mongo/vespa.0, size: 64MB, took 0.007 secs
2015-04-21T16:31:13.847-0400 [conn23] build index on: vespa.vespastats properties: { v: 1, key: { _id: 1 }, name: "_id_", ns: "vespa.vespastats" }
2015-04-21T16:31:13.848-0400 [conn23] added index to empty collection
2015-04-21T16:31:13.857-0400 [conn23] build index on: vespa.vespastats properties: { v: 1, key: { url: 1 }, name: "url_1", ns: "vespa.vespastats", background: true }
2015-04-21T16:31:13.857-0400 [conn23] added index to empty collection
2015-04-21T16:31:13.858-0400 [conn23] build index on: vespa.vespastats properties: { v: 1, key: { r_tstpm: 1 }, name: "r_tstpm_1", ns: "vespa.vespastats", background: true }
2015-04-21T16:31:13.859-0400 [conn23] added index to empty collection
2015-04-21T16:31:13.860-0400 [conn23] build index on: vespa.vespastats properties: { v: 1, key: { url: 1, r_tstpm: 1 }, name: "url_1_r_tstpm_1", ns: "vespa.vespastats", background: true }
2015-04-21T16:31:13.860-0400 [conn23] added index to empty collection
2015-04-21T16:31:13.862-0400 [conn23] end connection 127.0.0.1:58444 (0 connections now open)
now when i login to my new mongodb instance and check collection size, i get a big 0 -
# mongo
MongoDB shell version: 2.6.9
connecting to: test
> use vespa
switched to db vespa
> db.auth('vespauser', '<paswd>')
1
> db.vespastats.find()
> db.vespastats.count()
0
>
Collection may or may not exist in the used database but the query is not returning an error, just 0.
db.vespastats.find().count()
The issue should be because it is added to database test. (doc mentions it should be automatic but I was able to reproduce this behaviour).
Therefore
use test
db.vespastats.find().count()
would have returned the actual documents in the collection vespastats.
The issue is caused by not specifying db name when using mongo binary command mongorestore. doc for mongorestore mongorestore --nsInclude=vesta.vestastats should be the updated version (even if -d still works).
To know where the collection would land, I would run 2 times the restore dump and check show dbs in mongo shell 3 times (before and after) > the db size is changing (not immediately though as it may show 8kb right after the restoration).

MongoDs in ReplSet won't start after trying out some MapReduce

I was practicing some MapReduce inside of my Primary's mongo shell when it suddenly became a Secondary. I SSHed into the two other VM's with the other secondaries, and discovered that the mongod's had been rendered inoperaple. I killed them and I issued the mongod --config /etc/mongod.conf to kick them off and I entered the mongo shell. After a few seconds they were interrupted with:
2014-09-14T22:29:54.142-0500 DBClientCursor::init call() failed
2014-09-14T22:29:54.143-0500 trying reconnect to 127.0.0.1:27017 (127.0.0.1) failed
2014-09-14T22:29:54.143-0500 warning: Failed to connect to 127.0.0.1:27017, reason: errno:111 Connection refused
2014-09-14T22:29:54.143-0500 reconnect 127.0.0.1:27017 (127.0.0.1) failed failed couldn't connect to server 127.0.0.1:27017 (127.0.0.1), connection attempt failed
>
This is from their (the two original secondaries in the replicaset) logs:
2014-09-14T22:09:21.879-0500 [rsBackgroundSync] replSet syncing to: vm-billing-001:27017
2014-09-14T22:09:21.880-0500 [rsSync] replSet still syncing, not yet to minValid optime 54165090:1
2014-09-14T22:09:21.882-0500 [rsBackgroundSync] replset setting syncSourceFeedback to vm-billing-001:27017
2014-09-14T22:09:21.886-0500 [rsSync] replSet SECONDARY
2014-09-14T22:09:21.886-0500 [repl writer worker 1] build index on: test.tmp.mr.CCS.nonconforming_1_inc properties: { v: 1, key: { 0: 1 }, name: "_temp_0", ns: "test.tmp.mr.CCS.nonconforming_1_inc" }
2014-09-14T22:09:21.887-0500 [repl writer worker 1] added index to empty collection
2014-09-14T22:09:21.887-0500 [repl writer worker 1] build index on: test.tmp.mr.CCS.nonconforming_1 properties: { v: 1, key: { _id: 1 }, name: "_id_", ns: "test.tmp.mr.CCS.nonconforming_1" }
2014-09-14T22:09:21.887-0500 [repl writer worker 1] added index to empty collection
2014-09-14T22:09:21.888-0500 [repl writer worker 1] build index on: test.tmp.mr.CCS.nonconforming_1 properties: { v: 1, unique: true, key: { id: 1.0 }, name: "id_1", ns: "test.tmp.mr.CCS.nonconforming_1" }
2014-09-14T22:09:21.888-0500 [repl writer worker 1] added index to empty collection
2014-09-14T22:09:21.891-0500 [repl writer worker 2] ERROR: writer worker caught exception: :: caused by :: 11000 insertDocument :: caused by :: 11000 E11000 duplicate key error index: cisco.tmp.mr.CCS.nonconforming_1.$id_1 dup key: { : null } on: { ts: Timestamp 1410748561000|46, h: 9014687153249982311, v: 2, op: "i", ns: "cisco.tmp.mr.CCS.nonconforming_1", o: { _id: 14, value: 1.0 } }
2014-09-14T22:09:21.891-0500 [repl writer worker 2] Fatal Assertion 16360
2014-09-14T22:09:21.891-0500 [repl writer worker 2]
I can issue mongo --host ... --port ... from both of the two VMs that can't start the mongo to the original primary mongo, but I do see some connection refused notes above in the error log.
My original primary mongod can still be connected to in the mongo shell, but it is a primary. I can kill it and restart it and it will start up in secondary.
How can I roll back to the last known state and restart my replica set?

Log only errors in MongoDB logs

Is there any options for only logging the errors in MongoDB log files?
With the current configuration, it seems that every request to Mongo server is logged in log files:
Wed Sep 17 08:08:07.030 [conn117] insert my_database.myCol ninserted:1 keyUpdates:0 locks(micros) w:243505 285ms
Wed Sep 17 08:08:54.447 [conn101] command anotherDatabase.$cmd command: { findandmodify: "myCol", query: { ... }, new: 0, remove: 0, upsert: 0, fields: {...}, update: {...} } update: {...} ntoreturn:1 idhack:1 nupdated:1 fastmod:1 keyUpdates:0 locks(micros) w:124172 reslen:248 124ms
Wed Sep 17 08:10:24.370 [conn95] command my_database.$cmd command: { count: "cms.myCol", query: { ... }, fields: null } ntoreturn:1 keyUpdates:0 locks(micros) r:197368 reslen:48 197ms
...
The current configuration is:
# mongodb.conf
dbpath=/var/lib/mongodb
logpath=/var/log/mongodb/mongodb.log
logappend=true
How can be the configuration updated to only log errors?
Running Mongo shell version: 2.4.10:
$ mongo --version
MongoDB shell version: 2.4.10
Appending quiet=true will reduce a lot of output.
Perhaps it is impossible to avoid any output information except error on current stage.
Appending slowms=threshold to configuration file can reduce normal log output further.
threshold is a integer value (milliseconds). It means if one operation duration doesn't exceed this value, normal log information won't output. The default value is 100.
Also, you can change this value by another way if the instance is running.
var slowms = theValueYouWant;
var level = db.getProfilingStatus().was;
db.setProfilingLevel(level, slowms);

MongoDB sharding problems

Our mongodb server deployed with 2 shards, each has 1 master server and 2 slave servers.
The four slave servers run mongo config as proxy, and two of the slave servers run arbiters.
But the mongodb coundn't be used now.
I can connect to 192.168.0.1:8000(mongos) and exec queries like 'use database' or 'show dbs', but i cann't exec queries in a choosed database such as 'db.foo.count()', 'db.foo.findOne()'
Here is the error log:
mongos> db.dev.count()
Fri Aug 16 12:55:36 uncaught exception: count failed: {
"assertion" : "DBClientBase::findN: transport error: 10.81.4.72:7100 query: { setShardVersion: \"\", init: true, configdb: \"10.81.4.72:7300,10.42.50.26:7300,10.81.51.235:7300\", serverID: ObjectId('520db0a51fa00999772612b9'), authoritative: true }",
"assertionCode" : 10276,
"errmsg" : "db assertion failure",
"ok" : 0
}
Fri Aug 16 11:23:29 [conn8431] DBClientCursor::init call() failed
Fri Aug 16 11:23:29 [conn8430] Socket recv() errno:104 Connection reset by peer 10.81.4.72:7100
Fri Aug 16 11:23:29 [conn8430] SocketException: remote: 10.81.4.72:7100 error: 9001 socket exception [1] server [10.81.4.72:7100]
Fri Aug 16 11:23:29 [conn8430] DBClientCursor::init call() failed
Fri Aug 16 11:23:29 [conn8430] DBException in process: could not initialize cursor across all shards because : DBClientBase::findN: transport error: 10.81.4.72:7100 query: { setShardVersion: "", init: true, configdb: "10.81.4.72:7300,10.42.50.26:7300,10.81.51.235:7300", serverID: ObjectId('520d99c972581e6a124d0561'), authoritative: true } # s01/10.36.31.36:7100,10.42.50.24:7100,10.81.4.72:7100
i can only start on mongos, queries wouldn't be exec if more than 1 mongos run at the same time, error log:
mongos> db.dev.count() Fri Aug 16 15:12:29 uncaught exception: count failed: { "assertion" : "DBClientBase::findN: transport error: 10.81.4.72:7100 query: { setShardVersion: \"\", init: true, configdb: \"10.81.4.72:7300,10.42.50.26:7300,10.81.51.235:7300\", serverID: ObjectId('520dd04967557902f73a9fba'), authoritative: true }", "assertionCode" : 10276, "errmsg" : "db assertion failure", "ok" : 0 }
Could you please clarify if your set-up was working before, if you are just setting it up now?
To repair your MongoDB, you might want to follow this link:
http://docs.mongodb.org/manual/tutorial/recover-data-following-unexpected-shutdown/
References
MongoDB Documentation : Deploying a Shard-Cluster
MongoDB Documentation : Add Shards to an existing cluster
Older, outdated(!) info:
YouTube Video on Setting-up Sharding for MongoDB
Corresponding Blog on blog.serverdensity.com