Mongo running in linux container, stop taking connection. - mongodb

I am running MongoDB inside the Docker, a linux container. Based on this Dockerfile.
I am using the latest mongo release, 2.6.4 installed on Ubuntu 14.04 64bits. I am able to start the mongod, I do have a /data/db folder in the container and the host machine ( even used as mounted folder, it is there in host machine. ). Do have enough memory, but I was getting errors.
From the logs below, you could clear tell that I am trying to connect the db twice, the connection won't be open for some reasons. Someone could help me?
Note: the machine only have git and mongo installed. Do mongo need anything else as dependencies?
2014-09-04T05:14:50.645+0000 [initandlisten] MongoDB starting : pid=1 port=27017 dbpath=/data/db 64-bit host=f00d8205ca65
2014-09-04T05:14:50.645+0000 [initandlisten] db version v2.6.4
2014-09-04T05:14:50.645+0000 [initandlisten] git version: 3a830be0eb92d772aa855ebb711ac91d658ee910
2014-09-04T05:14:50.645+0000 [initandlisten] build info: Linux build7.nj1.10gen.cc 2.6.32-431.3.1.el6.x86_64 #1 SMP Fri Jan 3 21:39:27 UTC 2014 x86_64 BOOST_LIB_VERSION=1_49
2014-09-04T05:14:50.645+0000 [initandlisten] allocator: tcmalloc
2014-09-04T05:14:50.646+0000 [initandlisten] options: {}
2014-09-04T05:14:50.648+0000 [initandlisten] journal dir=/data/db/journal
2014-09-04T05:14:50.648+0000 [initandlisten] recover : no journal files present, no recovery needed
2014-09-04T05:14:50.703+0000 [FileAllocator] allocating new datafile /data/db/local.ns, filling with zeroes...
2014-09-04T05:14:50.703+0000 [FileAllocator] creating directory /data/db/_tmp
2014-09-04T05:14:50.705+0000 [FileAllocator] done allocating datafile /data/db/local.ns, size: 16MB, took 0 secs
2014-09-04T05:14:50.708+0000 [FileAllocator] allocating new datafile /data/db/local.0, filling with zeroes...
2014-09-04T05:14:50.709+0000 [FileAllocator] done allocating datafile /data/db/local.0, size: 64MB, took 0.001 secs
2014-09-04T05:14:50.711+0000 [initandlisten] build index on: local.startup_log properties: { v: 1, key: { _id: 1 }, name: "_id_", ns: "local.startup_log" }
2014-09-04T05:14:50.711+0000 [initandlisten] added index to empty collection
2014-09-04T05:14:50.712+0000 [initandlisten] waiting for connections on port 27017
2014-09-04T05:15:50.721+0000 [clientcursormon] mem (MB) res:36 virt:341
2014-09-04T05:15:50.721+0000 [clientcursormon] mapped (incl journal view):160
2014-09-04T05:15:50.721+0000 [clientcursormon] connections:0
2014-09-04T05:15:57.976+0000 [initandlisten] connection accepted from 172.17.42.1:56593 #1 (1 connection now open)
2014-09-04T05:15:57.976+0000 [conn1] end connection 172.17.42.1:56593 (0 connections now open)
Someone said it could be less memory to run mongo. after running the momery check, I do still have a good amount of memory to run Mongo.
[ root#512ea0e1096b:/data ]$ free -m
total used free shared buffers cached
Mem: 2001 436 1564 0 24 276
-/+ buffers/cache: 136 1865
Swap: 2991 0 2991

Related

MongoDB unexpected shutdown and data loss

The weirdest thing happened. We have a mongodb instance with both journal and logs on separate hard drives running on a single amazon ec2 backed by ebs. The server unexpectedly shutdown this morning and when I looked at it, I noticed that mongo didn't start on startup.
I manually started mongo using the command
sudo mongod --fork --logpath /var/log/mongodb.log
It ran fine but when I looked at the database, all our our data had gone missing. I tried exiting out with the --repair option but it did nothing. This is what the logs from the initial mongod looks like:
2015-10-05T16:39:17.881+0000 [initandlisten] MongoDB starting :
pid=2757 port=27017 dbpath=/data/db 64-bit host=ip-172-31-59-166
2015-10-05T16:39:17.882+0000 [initandlisten] db version v2.6.8
2015-10-05T16:39:17.882+0000 [initandlisten] git version: 3abc04d6d4f71de00b57378e3277def8fd7a6700
2015-10-05T16:39:17.882+0000 [initandlisten] build info: Linux build5.nj1.10gen.cc 2.6.32-431.3.1.el6.x86_64 #1 SMP Fri Jan 3 21:39:27 UTC 2014 x86_64 BOOST_LIB_VERSION=1_49
2015-10-05T16:39:17.882+0000 [initandlisten] allocator: tcmalloc
2015-10-05T16:39:17.882+0000 [initandlisten] options: { processManagement: { fork: true }, systemLog: { destination: "file", path: "/var/log/mongodb.log" } }
2015-10-05T16:39:17.890+0000 [initandlisten] journal dir=/data/db/journal
2015-10-05T16:39:17.893+0000 [initandlisten] recover : no journal files present, no recovery needed
2015-10-05T16:39:17.893+0000 [initandlisten] preallocating a journal file /data/db/journal/prealloc.1
2015-10-05T16:39:28.081+0000 [initandlisten] preallocating a journal file /data/db/journal/prealloc.2
2015-10-05T16:39:40.496+0000 [initandlisten] allocating new ns file /data/db/local.ns, filling with zeroes...
2015-10-05T16:39:40.654+0000 [FileAllocator] allocating new datafile /data/db/local.0, filling with zeroes...
2015-10-05T16:39:40.654+0000 [FileAllocator] creating directory /data/db/_tmp
2015-10-05T16:39:40.659+0000 [FileAllocator] done allocating datafile /data/db/local.0, size: 64MB, took 0.002 secs
2015-10-05T16:39:40.664+0000 [initandlisten] build index on: local.startup_log properties: { v: 1, key: { _id: 1 }, name: "_id_", ns: "local.startup_log" }
2015-10-05T16:39:40.664+0000 [initandlisten] added index to empty collection
2015-10-05T16:39:40.665+0000 [initandlisten] command local.$cmd command: create { create: "startup_log", size: 10485760, capped: true } ntoreturn:1 keyUpdates:0 numYields:0 reslen:37 169ms
2015-10-05T16:39:40.669+0000 [initandlisten] waiting for connections on port 27017
This looks like there are no problems with the MongoDB initial start. But it also says that it's creating new ns file though old ones existed. What is happening here and how can I recover the data?
There's no indication anything was corrupted. The log indicates this instance was started on an empty volume.
Aside from the logPath, your MongoDB process is starting up with the default parameters. So, it's going to use /data/db as the location of the data directory. Based on how on your initial description of the problem, you may have been running the separate journal and data directories an EBS mount, i.e. your MongoDB config used --dbpath.
Check that you're using the same MongoDB configuration as was used
before the server went down
Check that your EBS volumes are mounted expected. You may use the mount option to confirm that the
path /data/db is pointing to the proper EBS volume.
And just to be sure... if the instance had used ephemeral drives the data would be lost when the instance shutdown. Sometime people confuse ephemeral drives with elastic block store, which is persistent.

MongoDB MMS replica set fails to restart

I have a MMS replica set deployed with 3 instances. It was working fine until I shut it down this morning in order to do some changes and restarted it. From logs, it feels that there was an unclean shutdown of replica set. The mongod server on primary is failing to start now.
Here are the full logs
2015-08-20T07:08:41.389+0000 W - [initandlisten] Detected unclean shutdown - /data/XXXXXXXXX/mongod.lock is not empty.
2015-08-20T07:08:41.406+0000 I JOURNAL [initandlisten] journal dir=/data/XXXXXXXXX/journal
2015-08-20T07:08:41.406+0000 I JOURNAL [initandlisten] recover begin
2015-08-20T07:08:41.406+0000 I JOURNAL [initandlisten] info no lsn file in journal/ directory
2015-08-20T07:08:41.406+0000 I JOURNAL [initandlisten] recover lsn: 0
2015-08-20T07:08:41.406+0000 I JOURNAL [initandlisten] recover /data/XXXXXXXXX/journal/j._0
2015-08-20T07:08:41.407+0000 I JOURNAL [initandlisten] recover cleaning up
2015-08-20T07:08:41.407+0000 I JOURNAL [initandlisten] removeJournalFiles
2015-08-20T07:08:41.641+0000 I JOURNAL [initandlisten] recover done
2015-08-20T07:08:41.641+0000 I JOURNAL [initandlisten] preallocating a journal file /data/XXXXXXXXXXX/journal/prealloc.0
2015-08-20T07:08:44.074+0000 I - [initandlisten] File Preallocator Progress: 744488960/1073741824 69%
2015-08-20T07:08:47.176+0000 I - [initandlisten] File Preallocator Progress: 901775360/1073741824 83%
2015-08-20T07:08:50.274+0000 I - [initandlisten] File Preallocator Progress: 1027604480/1073741824 95%
2015-08-20T07:09:09.057+0000 I JOURNAL [durability] Durability thread started
2015-08-20T07:09:09.057+0000 I JOURNAL [journal writer] Journal writer thread started
2015-08-20T07:09:09.060+0000 I CONTROL [initandlisten] MongoDB starting : pid=25507 port=27000 dbpath=/data/XXXXXXXXXXXXX 64-bit host=CH$
2015-08-20T07:09:09.060+0000 I CONTROL [initandlisten] db version v3.0.2
2015-08-20T07:09:09.060+0000 I CONTROL [initandlisten] git version: 6201872043ecbbc0a4cc169b5482dcf385fc464f
2015-08-20T07:09:09.060+0000 I CONTROL [initandlisten] OpenSSL version: OpenSSL 1.0.1f 6 Jan 2014
2015-08-20T07:09:09.060+0000 I CONTROL [initandlisten] build info: Linux ip-10-229-1-2 3.13.0-24-generic #46-Ubuntu SMP Thu Apr 10 19:11:08 U$
2015-08-20T07:09:09.060+0000 I CONTROL [initandlisten] allocator: tcmalloc
2015-08-20T07:09:09.060+0000 I CONTROL [initandlisten] options: { config: "/data/XXXXXXXXXXXX/automation-mongod.conf", net: { port: 270$
2015-08-20T07:09:09.074+0000 I - [initandlisten] Invariant failure _name == nsToDatabaseSubstring( ns ) src/mongo/db/catalog/database.c$
2015-08-20T07:09:09.091+0000 I CONTROL [initandlisten]
0xf4f859 0xef0031 0xed4b52 0x91e106 0x91e18f 0x920033 0x922cb0 0x808701 0x7d4ba4 0x7f0503489ec5 0x805d17
----- BEGIN BACKTRACE -----
{"backtrace":[{"b":"400000","o":"B4F859"},{"b":"400000","o":"AF0031"},{"b":"400000","o":"AD4B52"},{"b":"400000","o":"51E106"},{"b":"400000","o$
mongod(_ZN5mongo15printStackTraceERSo+0x29) [0xf4f859]
mongod(_ZN5mongo10logContextEPKc+0xE1) [0xef0031]
mongod(_ZN5mongo15invariantFailedEPKcS1_j+0xB2) [0xed4b52]
mongod(_ZNK5mongo8Database13getCollectionERKNS_10StringDataE+0x36) [0x91e106]
mongod(_ZN5mongo8Database30_getOrCreateCollectionInstanceEPNS_16OperationContextERKNS_10StringDataE+0x1F) [0x91e18f]
mongod(_ZN5mongo8DatabaseC1EPNS_16OperationContextERKNS_10StringDataEPNS_20DatabaseCatalogEntryE+0x1E3) [0x920033]
mongod(_ZN5mongo14DatabaseHolder6openDbEPNS_16OperationContextERKNS_10StringDataEPb+0x150) [0x922cb0]
mongod(_ZN5mongo14DatabaseHolder6openDbEPNS_16OperationContextERKNS_10StringDataEPb+0x150) [0x922cb0]
mongod(_ZN5mongo13initAndListenEi+0xC01) [0x808701]
mongod(main+0x134) [0x7d4ba4]
libc.so.6(__libc_start_main+0xF5) [0x7f0503489ec5]
mongod(+0x405D17) [0x805d17]
----- END BACKTRACE -----
2015-08-20T07:09:09.091+0000 I - [initandlisten]
***aborting after invariant() failure
Any idea how can I fix it? I have been trying to fix it since past 4 hours but nothing seems to be working.
So, it turned out that issue was in .ns file of one of the database. I had to delete that file and restart the server. Server started successfully however, the database whose .ns I deleted was lost.
For newbies in mongoDB .ns file is the namespace file that mongo creates for each database in its data (/data) directory.

MongoDB --dbpath giving access to old data

I have mongo installed on my local machine. When I start the database by running mongod I got this error (that is fairly common, and this site has solutions to workarounds):
$ mongod
mongod --help for help and startup options
2015-01-01T22:31:17.350-0600 [initandlisten] MongoDB starting : pid=2835 port=27017 dbpath=/data/db 64-bit host=hermes
2015-01-01T22:31:17.351-0600 [initandlisten] db version v2.6.4
2015-01-01T22:31:17.351-0600 [initandlisten] git version: nogitversion
2015-01-01T22:31:17.351-0600 [initandlisten] build info: Darwin minimavericks.local 13.3.0 Darwin Kernel Version 13.3.0: Tue Jun 3 21:27:35 PDT 2014; root:xnu-2422.110.17~1/RELEASE_X86_64 x86_64 BOOST_LIB_VERSION=1_49
2015-01-01T22:31:17.351-0600 [initandlisten] allocator: tcmalloc
2015-01-01T22:31:17.351-0600 [initandlisten] options: {}
2015-01-01T22:31:17.351-0600 [initandlisten] exception in initAndListen: 10296
*********************************************************************
ERROR: dbpath (/data/db) does not exist.
Create this directory or give existing directory in --dbpath.
See http://dochub.mongodb.org/core/startingandstoppingmongo
*********************************************************************
, terminating
2015-01-01T22:31:17.351-0600 [initandlisten] dbexit:
2015-01-01T22:31:17.351-0600 [initandlisten] shutdown: going to close listening sockets...
2015-01-01T22:31:17.351-0600 [initandlisten] shutdown: going to flush diaglog...
2015-01-01T22:31:17.351-0600 [initandlisten] shutdown: going to close sockets...
2015-01-01T22:31:17.351-0600 [initandlisten] shutdown: waiting for fs preallocator...
2015-01-01T22:31:17.351-0600 [initandlisten] shutdown: lock for final commit...
2015-01-01T22:31:17.351-0600 [initandlisten] shutdown: final commit...
2015-01-01T22:31:17.351-0600 [initandlisten] shutdown: closing all files...
2015-01-01T22:31:17.352-0600 [initandlisten] closeAllFiles() finished
2015-01-01T22:31:17.355-0600 [initandlisten] dbexit: really exiting now
I created a directory in my development environment at /my/curent/directory/data/db and started mongo again with $ mongod --dbpath /my/curent/directory/data/db, this time producing:
2015-01-01T22:32:31.282-0600 [initandlisten] options: { storage: { dbPath: "./data/db" } }
2015-01-01T22:32:31.319-0600 [initandlisten] journal dir=./data/db/journal
2015-01-01T22:32:31.320-0600 [initandlisten] recover : no journal files present, no recovery needed
2015-01-01T22:32:31.383-0600 [FileAllocator] allocating new datafile ./data/db/local.ns, filling with zeroes...
2015-01-01T22:32:31.383-0600 [FileAllocator] creating directory ./data/db/_tmp
2015-01-01T22:32:31.482-0600 [FileAllocator] done allocating datafile ./data/db/local.ns, size: 16MB, took 0.098 secs
2015-01-01T22:32:31.842-0600 [FileAllocator] allocating new datafile ./data/db/local.0, filling with zeroes...
2015-01-01T22:32:32.756-0600 [FileAllocator] done allocating datafile ./data/db/local.0, size: 64MB, took 0.914 secs
2015-01-01T22:32:33.215-0600 [initandlisten] build index on: local.startup_log properties: { v: 1, key: { _id: 1 }, name: "_id_", ns: "local.startup_log" }
2015-01-01T22:32:33.216-0600 [initandlisten] added index to empty collection
2015-01-01T22:32:33.216-0600 [initandlisten] command local.$cmd command: create { create: "startup_log", size: 10485760, capped: true } ntoreturn:1 keyUpdates:0 numYields:0 reslen:37 1852ms
2015-01-01T22:32:33.216-0600 [initandlisten] waiting for connections on port 27017
2015-01-01T22:33:31.402-0600 [clientcursormon] mem (MB) res:33 virt:2635
2015-01-01T22:33:31.402-0600 [clientcursormon] mapped (incl journal view):160
2015-01-01T22:33:31.402-0600 [clientcursormon] connections:0
It's great. Using the mongo shell, I can access my collections that I was working with in this directory, but I can also see all of my other databases from weeks prior (when this directory did not exist) using show dbs. What purpose does the --dbpath option serve if not to isolate where data is stored? Is something weird happening here?
MongoDB requires a data directory to store all data. MongoDB’s default data directory path is \data\db.
To use an alternate dbpath, specify the path in the configuration file (e.g. C:\Program Files\MongoDB\mongod.cfg) or on the command line with the --dbpath option.
Read more about MongoDB Installation
If you are using windows then create data\db folder under C drive.
If you are using your own directory for data then pass full path of the folder as --dbpath argument value.
If MongoDB’s default data directory path "\data\db" is correct but accessing old data means just check your Task Manager.
End all mongo command and restart your mongodb. I tried it, working well

mongodb suddenly stopped to work

I've had mongo running for a while on my dev server running Centos 6.5 and mongo 2.6.4.
But then I needed to install mysql for a different project and now suddenly mongo doesn't work.
I tried to re-install it and upgrade it to 2.6.5 at the same time. I've tried to start it and restart it:
# sudo service mongod restart
Stopping mongod: [FAILED]
Starting mongod: [FAILED]
Repairing it said I missed /data/db which I then created. The repair responds like this:
# sudo mongod --repair
2014-10-24T15:52:14.710-0400 [initandlisten] MongoDB starting : pid=1182 port=27017 dbpath=/data/db 64-bit host=myhost
2014-10-24T15:52:14.710-0400 [initandlisten] db version v2.6.5
2014-10-24T15:52:14.710-0400 [initandlisten] git version: e99d4fcb4279c0279796f237aa92fe3b64560bf6
2014-10-24T15:52:14.710-0400 [initandlisten] build info: Linux build8.nj1.10gen.cc 2.6.32-431.3.1.el6.x86_64 #1 SMP Fri Jan 3 21:39:27 UTC 2014 x86_64 BOOST_LIB_VERSION=1_49
2014-10-24T15:52:14.710-0400 [initandlisten] allocator: tcmalloc
2014-10-24T15:52:14.710-0400 [initandlisten] options: { repair: true }
2014-10-24T15:52:14.726-0400 [initandlisten] repairDatabase local
2014-10-24T15:52:14.726-0400 [initandlisten] allocating new ns file /data/db/_tmp_repairDatabase_0/local.ns, filling with zeroes...
2014-10-24T15:52:14.843-0400 [FileAllocator] allocating new datafile /data/db/_tmp_repairDatabase_0/local.0, filling with zeroes...
2014-10-24T15:52:14.843-0400 [FileAllocator] creating directory /data/db/_tmp_repairDatabase_0/_tmp
2014-10-24T15:52:14.879-0400 [FileAllocator] done allocating datafile /data/db/_tmp_repairDatabase_0/local.0, size: 64MB, took 0.017 secs
2014-10-24T15:52:14.897-0400 [initandlisten] finished checking dbs
2014-10-24T15:52:14.897-0400 [initandlisten] dbexit:
2014-10-24T15:52:14.897-0400 [initandlisten] shutdown: going to close listening sockets...
2014-10-24T15:52:14.897-0400 [initandlisten] shutdown: going to flush diaglog...
2014-10-24T15:52:14.897-0400 [initandlisten] shutdown: going to close sockets...
2014-10-24T15:52:14.897-0400 [initandlisten] shutdown: waiting for fs preallocator...
2014-10-24T15:52:14.897-0400 [initandlisten] shutdown: closing all files...
2014-10-24T15:52:14.897-0400 [initandlisten] closeAllFiles() finished
2014-10-24T15:52:14.897-0400 [initandlisten] shutdown: removing fs lock...
2014-10-24T15:52:14.897-0400 [initandlisten] dbexit: really exiting now
Starting it again, and fails.
did this:
#semanage port -a -t mongodb_port_t -p tcp 27017
Killed
Trying to enter the mongo shell responds with this:
# mongo
MongoDB shell version: 2.6.5
connecting to: test
2014-10-24T15:59:40.243-0400 warning: Failed to connect to 127.0.0.1:27017, reason: errno:111 Connection refused
2014-10-24T15:59:40.244-0400 Error: couldn't connect to server 127.0.0.1:27017 (127.0.0.1), connection attempt failed at src/mongo/shell/mongo.js:146
exception: connect failed
I'm out of ideas. Any suggestions?

MongoD Start Hangs Indefinitely at Connections:0

mongod --smallfiles
2014-06-05T13:19:15.085-0400 [initandlisten] MongoDB starting : pid=14584 port=27017 dbpath=/data/db 64-bit host=sfpop-predict06
2014-06-05T13:19:15.085-0400 [initandlisten] db version v2.6.1
2014-06-05T13:19:15.085-0400 [initandlisten] git version: 4b95b086d2374bdcfcdf2249272fb552c9c726e8
2014-06-05T13:19:15.085-0400 [initandlisten] build info: Linux build14.nj1.10gen.cc 2.6.32-431.3.1.el6.x86_64 #1 SMP Fri Jan 3 21:39:27 UTC 2014 x86_64 BOOST_LIB_VERSION=1_49
2014-06-05T13:19:15.085-0400 [initandlisten] allocator: tcmalloc
2014-06-05T13:19:15.085-0400 [initandlisten] options: { storage: { smallFiles: true } }
2014-06-05T13:19:15.091-0400 [initandlisten] journal dir=/data/db/journal
2014-06-05T13:19:15.091-0400 [initandlisten] recover : no journal files present, no recovery needed
2014-06-05T13:19:15.113-0400 [initandlisten] waiting for connections on port 27017
2014-06-05T13:20:15.137-0400 [clientcursormon] mem (MB) res:35 virt:327
2014-06-05T13:20:15.137-0400 [clientcursormon] mapped (incl journal view):64
2014-06-05T13:20:15.138-0400 [clientcursormon] connections:0
It hangs for a minute on the "waiting for connections on port 27017" line, and indefinitely on the "connections:0" line. I can run Mongo just fine, but I need to run MongoD so I can connect to the databases with PHP. How can I fix this?
CentOS 5.7
I had the same doubt as you, What James said is right, just start another terminal, and run mongo, it will be connected automatically