new to this ..so my MONGOD unexpectedly shutdown and i can't seem to repair it ..this is what i see when i try to start up ./mongod
**2018-04-21T19:59:06.237+0000 ** WARNING: --rest is specified without --httpinterface,
2018-04-21T19:59:06.238+0000 ** enabling http interface
warning: bind_ip of 0.0.0.0 is unnecessary; listens on all ips by default
2018-04-21T19:59:06.241+0000 [initandlisten] MongoDB starting : pid=2848 port=27017 dbpath=data 64-bit host=omarr11-webdevbootcamp-5900192
2018-04-21T19:59:06.241+0000 [initandlisten] db version v2.6.12
2018-04-21T19:59:06.241+0000 [initandlisten] git version: d73c92b1c85703828b55c2916a5dd4ad46535f6a
2018-04-21T19:59:06.241+0000 [initandlisten] build info: Linux build5.ny.cbi.10gen.cc 2.6.32-431.3.1.el6.x86_64 #1 SMP Fri Jan 3 21:39:27 UTC 2014 x86_64 BOOST_LIB_VERSION=1_49
2018-04-21T19:59:06.241+0000 [initandlisten] allocator: tcmalloc
2018-04-21T19:59:06.241+0000 [initandlisten] options: { net: { bindIp: "0.0.0.0", http: { RESTInterfaceEnabled: true, enabled: true } }, storage: { dbPath: "data", journal: { enabled: false } } }
**************
Unclean shutdown detected.
Please visit http://dochub.mongodb.org/core/repair for recovery instructions.
*************
2018-04-21T19:59:06.241+0000 [initandlisten] exception in initAndListen: 12596 old lock file, terminating
2018-04-21T19:59:06.241+0000 [initandlisten] dbexit:
2018-04-21T19:59:06.241+0000 [initandlisten] shutdown: going to close listening sockets...
2018-04-21T19:59:06.241+0000 [initandlisten] shutdown: going to flush diaglog...
2018-04-21T19:59:06.241+0000 [initandlisten] shutdown: going to close sockets...
2018-04-21T19:59:06.241+0000 [initandlisten] shutdown: waiting for fs preallocator...
2018-04-21T19:59:06.242+0000 [initandlisten] shutdown: closing all files...
2018-04-21T19:59:06.242+0000 [initandlisten] closeAllFiles() finished
2018-04-21T19:59:06.242+0000 [initandlisten] dbexit: really exiting now**
can someone help me repair this please?
You should look for the lock file (mongod.lock) in the data folder of
the mongodb and you need to delete it and try to start again.
In case you have some data corruption issue you can try repair option https://docs.mongodb.com/manual/tutorial/recover-data-following-unexpected-shutdown/ . But in case it is just normal course of learning deleting the mongod.lock file should start the mongod process again...
Related
Running mongodb 2.6.4 on Win2008-R2.
Full of data.... and today it went down, and won't come back up.
What to do?
2015-01-27T02:03:58.781-0600 Trying to start Windows service 'MongoDB'
2015-01-27T02:03:58.781-0600 Service running
2015-01-27T02:03:58.781-0600 [initandlisten] MongoDB starting : pid=1864 port=27017 dbpath=c:\MongoDB\Data 64-bit host=619723-MONGO-1
2015-01-27T02:03:58.781-0600 [initandlisten] targetMinOS: Windows 7/Windows Server 2008 R2
2015-01-27T02:03:58.781-0600 [initandlisten] db version v2.6.4
2015-01-27T02:03:58.781-0600 [initandlisten] git version: 3a830be0eb92d772aa855ebb711ac91d658ee910
2015-01-27T02:03:58.781-0600 [initandlisten] build info: windows sys.getwindowsversion(major=6, minor=1, build=7601, platform=2, service_pack='Service Pack 1') BOOST_LIB_VERSION=1_49
2015-01-27T02:03:58.781-0600 [initandlisten] allocator: system
2015-01-27T02:03:58.781-0600 [initandlisten] options: { config: "C:\MongoDB\mongod.cfg", service: true, storage: { dbPath: "c:\MongoDB\Data", directoryPerDB: true }, systemLog: { destination: "file", path: "c:\MongoDB\logs" } }
2015-01-27T02:03:58.890-0600 [initandlisten] journal dir=c:\MongoDB\Data\journal
2015-01-27T02:03:58.968-0600 [initandlisten] dbexception during recovery: 13532 unexpected file in journal directory c:\MongoDB\Data\journal : j._1895 : can't find its preceding file
2015-01-27T02:03:58.984-0600 [initandlisten] exception in initAndListen: 13532 unexpected file in journal directory c:\MongoDB\Data\journal : j._1895 : can't find its preceding file, terminating
2015-01-27T02:03:58.999-0600 [initandlisten] dbexit:
2015-01-27T02:03:58.999-0600 [initandlisten] shutdown: going to close listening sockets...
2015-01-27T02:03:58.999-0600 [initandlisten] shutdown: going to flush diaglog...
2015-01-27T02:03:58.999-0600 [initandlisten] shutdown: going to close sockets...
2015-01-27T02:03:58.999-0600 [initandlisten] shutdown: waiting for fs preallocator...
2015-01-27T02:03:58.999-0600 [initandlisten] shutdown: lock for final commit...
2015-01-27T02:03:58.999-0600 [initandlisten] shutdown: final commit...
2015-01-27T02:03:58.999-0600 [initandlisten] shutdown: closing all files...
2015-01-27T02:03:58.999-0600 [initandlisten] closeAllFiles() finished
2015-01-27T02:03:58.999-0600 [initandlisten] shutdown: removing fs lock...
2015-01-27T02:03:58.999-0600 [initandlisten] dbexit: really exiting now
Well, google did not seem to have anything useful on mongo error 13532
What we did:
a) Deleted all files in /journal dir
b) Deleted the mongo .lock file
c) From cmd line: mongod --repair --dbpath c:\mongodb\data\
d) Started up mongo service
Up now for several hours OK.
Not sure what the data loss was.
Able to run MongoDB locally but when trying to access remotely it failed. Havent done any firewall setup. What could be the issue? How to do troubleshoot?
Also the MongoDB status show running
service mongod status
mongod (pid 25342) is running...
But when trying to restart the service it says
service mongod restart
Stopping mongod: [ OK ]
Starting mongod: [FAILED]
But still the process is running
Update
Tried mongod --repair here is the update
2015-01-10T10:46:33.109-0500 [initandlisten] MongoDB starting : pid=29018 port=2 7017 dbpath=/data/db 64-bit host=ns4008927.ip-192-99-3.net
2015-01-10T10:46:33.109-0500 [initandlisten] db version v2.6.6
2015-01-10T10:46:33.109-0500 [initandlisten] git version: 608e8bc319627693b04cc7 da29ecc300a5f45a1f
2015-01-10T10:46:33.109-0500 [initandlisten] build info: Linux build10.nj1.10gen .cc 2.6.32-431.3.1.el6.x86_64 #1 SMP Fri Jan 3 21:39:27 UTC 2014 x86_64 BOOST_LI B_VERSION=1_49
2015-01-10T10:46:33.109-0500 [initandlisten] allocator: tcmalloc
2015-01-10T10:46:33.109-0500 [initandlisten] options: { repair: true }
2015-01-10T10:46:33.112-0500 [initandlisten] finished checking dbs
2015-01-10T10:46:33.112-0500 [initandlisten] dbexit:
2015-01-10T10:46:33.112-0500 [initandlisten] shutdown: going to close listening sockets...
2015-01-10T10:46:33.112-0500 [initandlisten] shutdown: going to flush diaglog...
2015-01-10T10:46:33.112-0500 [initandlisten] shutdown: going to close sockets...
2015-01-10T10:46:33.112-0500 [initandlisten] shutdown: waiting for fs preallocat or...
2015-01-10T10:46:33.112-0500 [initandlisten] shutdown: closing all files...
2015-01-10T10:46:33.112-0500 [initandlisten] closeAllFiles() finished
2015-01-10T10:46:33.112-0500 [initandlisten] shutdown: removing fs lock...
2015-01-10T10:46:33.112-0500 [initandlisten] dbexit: really exiting now
You need to remove this line from mongo's config file (on Ubuntu, mine is at /etc/mongod.conf):
bind_ip = 127.0.0.1
Or, if it's YAML,
bindIp: 127.0.0.1
Source: http://docs.mongodb.org/manual/reference/configuration-options/
"Changed in version 2.6.0: The deb and rpm packages include a default configuration file that sets net.bindIp to 127.0.0.1."
The default value for bindIp/bind_ip is "all interfaces" - so that's where the deletion comes in handy.
I have followed the instructions on MongoDB's website to install MongoDB on RHEL systems. Whenever I use the command sudo service mongod start I get the message Starting mongod: [FAILED]. Does anyone know what I am doing wrong? I already changed the permissions to the /var/lib/mongo and /var/log/mongodb directories.
In addition, this is what my log file reads for the last time I tried to start mongoDB
2014-11-28T16:52:42.532-0600 ***** SERVER RESTARTED *****
2014-11-28T16:52:42.535-0600 [initandlisten] MongoDB starting : pid=2846 port=27017 dbpath=/var/lib/mongo$
2014-11-28T16:52:42.535-0600 [initandlisten] db version v2.6.5
2014-11-28T16:52:42.535-0600 [initandlisten] git version: e99d4fcb4279c0279796f237aa92fe3b64560bf6
2014-11-28T16:52:42.535-0600 [initandlisten] build info: Linux build8.nj1.10gen.cc 2.6.32-431.3.1.el6.x86$
2014-11-28T16:52:42.535-0600 [initandlisten] allocator: tcmalloc
2014-11-28T16:52:42.535-0600 [initandlisten] options: { config: "/etc/mongod.conf", net: { bindIp: "127.0$
2014-11-28T16:52:42.535-0600 [initandlisten] exception in initAndListen: 10310 Unable to lock file: /var/$
2014-11-28T16:52:42.535-0600 [initandlisten] dbexit:
2014-11-28T16:52:42.535-0600 [initandlisten] shutdown: going to close listening sockets...
2014-11-28T16:52:42.535-0600 [initandlisten] shutdown: going to flush diaglog...
2014-11-28T16:52:42.535-0600 [initandlisten] shutdown: going to close sockets...
2014-11-28T16:52:42.535-0600 [initandlisten] shutdown: waiting for fs preallocator...
2014-11-28T16:52:42.535-0600 [initandlisten] shutdown: lock for final commit...
2014-11-28T16:52:42.535-0600 [initandlisten] shutdown: final commit...
2014-11-28T16:52:42.536-0600 [initandlisten] shutdown: closing all files...
2014-11-28T16:52:42.536-0600 [initandlisten] closeAllFiles() finished
2014-11-28T16:52:42.536-0600 [initandlisten] dbexit: really exiting now
that looks very similar to the output when you have a mongo already running.
probably you have a startup script installed in /etc/init.d/mongod
Your log file is clipped. Please include an unclipped version where we can see the entire line. From what's there, mongod couldn't lock something, which likely means you already have a mongod running in that directory or the permissions aren't set to allow mongod to take the lock.
I installed mongodb using the information in this link. However, when I user the service mongod start command I get the following error. Can someone help me troubleshoot?
[Piyush#localhost ~]$ service mongod start
/etc/init.d/mongod: line 54: ulimit: open files: cannot modify limit: Operation not permitted
/etc/init.d/mongod: line 56: ulimit: max user processes: cannot modify limit: Operation not permitted
Starting mongod: runuser: using restricted shell /bin/false
runuser: cannot set groups: Operation not permitted
[FAILED]
[Piyush#localhost ~]$ sudo service mongod start
[sudo] password for Piyush:
Starting mongod: [FAILED]
I tried using --repair command. I get the following output for that:
[Piyush#localhost ~]$ sudo mongod --repair
2014-09-24T00:05:06.089-0400 [initandlisten] MongoDB starting : pid=2807 port=27017 dbpath=/data/db 64-bit host=localhost.localdomain
2014-09-24T00:05:06.089-0400 [initandlisten] db version v2.6.4
2014-09-24T00:05:06.089-0400 [initandlisten] git version: 3a830be0eb92d772aa855ebb711ac91d658ee910
2014-09-24T00:05:06.089-0400 [initandlisten] build info: Linux build7.nj1.10gen.cc 2.6.32-431.3.1.el6.x86_64 #1 SMP Fri Jan 3 21:39:27 UTC 2014 x86_64 BOOST_LIB_VERSION=1_49
2014-09-24T00:05:06.089-0400 [initandlisten] allocator: tcmalloc
2014-09-24T00:05:06.090-0400 [initandlisten] options: { repair: true }
2014-09-24T00:05:06.106-0400 [initandlisten] finished checking dbs
2014-09-24T00:05:06.106-0400 [initandlisten] dbexit:
2014-09-24T00:05:06.106-0400 [initandlisten] shutdown: going to close listening sockets...
2014-09-24T00:05:06.106-0400 [initandlisten] shutdown: going to flush diaglog...
2014-09-24T00:05:06.106-0400 [initandlisten] shutdown: going to close sockets...
2014-09-24T00:05:06.106-0400 [initandlisten] shutdown: waiting for fs preallocator...
2014-09-24T00:05:06.106-0400 [initandlisten] shutdown: closing all files...
2014-09-24T00:05:06.106-0400 [initandlisten] closeAllFiles() finished
2014-09-24T00:05:06.107-0400 [initandlisten] shutdown: removing fs lock...
2014-09-24T00:05:06.107-0400 [initandlisten] dbexit: really exiting now
[EDIT 1]_____________________________________________________________________________
In the log file I see:
[root#localhost ~]# cat /var/log/mongodb/mongod.log.2014-09-24T16-00-32
2014-09-24T11:54:23.636-0400 ***** SERVER RESTARTED *****
2014-09-24T11:54:23.665-0400 [initandlisten] MongoDB starting : pid=3004 port=27017 dbpath=/var/lib/mongo 64-bit host=localhost.localdomain
2014-09-24T11:54:23.665-0400 [initandlisten] db version v2.6.4
2014-09-24T11:54:23.665-0400 [initandlisten] git version: 3a830be0eb92d772aa855ebb711ac91d658ee910
2014-09-24T11:54:23.665-0400 [initandlisten] build info: Linux build7.nj1.10gen.cc 2.6.32-431.3.1.el6.x86_64 #1 SMP Fri Jan 3 21:39:27 UTC 2014 x86_64 BOOST_LIB_VERSION=1_49
2014-09-24T11:54:23.665-0400 [initandlisten] allocator: tcmalloc
2014-09-24T11:54:23.665-0400 [initandlisten] options: { config: "/etc/mongod.conf", net: { bindIp: "127.0.0.1" }, processManagement: { fork: true, pidFilePath: "/var/run/mongodb/mongod.pid" }, storage: { dbPath: "/var/lib/mongo" }, systemLog: { destination: "file", logAppend: true, path: "/var/log/mongodb/mongod.log" } }
2014-09-24T11:54:23.680-0400 [initandlisten] journal dir=/var/lib/mongo/journal
2014-09-24T11:54:23.680-0400 [initandlisten] recover : no journal files present, no recovery needed
2014-09-24T11:54:23.680-0400 [initandlisten]
2014-09-24T11:54:23.680-0400 [initandlisten] ERROR: Insufficient free space for journal files
2014-09-24T11:54:23.680-0400 [initandlisten] Please make at least 3379MB available in /var/lib/mongo/journal or use --smallfiles
2014-09-24T11:54:23.680-0400 [initandlisten]
2014-09-24T11:54:23.681-0400 [initandlisten] exception in initAndListen: 15926 Insufficient free space for journals, terminating
2014-09-24T11:54:23.681-0400 [initandlisten] dbexit:
2014-09-24T11:54:23.681-0400 [initandlisten] shutdown: going to close listening sockets...
2014-09-24T11:54:23.681-0400 [initandlisten] shutdown: going to flush diaglog...
2014-09-24T11:54:23.681-0400 [initandlisten] shutdown: going to close sockets...
2014-09-24T11:54:23.681-0400 [initandlisten] shutdown: waiting for fs preallocator...
2014-09-24T11:54:23.681-0400 [initandlisten] shutdown: lock for final commit...
2014-09-24T11:54:23.681-0400 [initandlisten] shutdown: final commit...
2014-09-24T11:54:23.681-0400 [initandlisten] shutdown: closing all files...
2014-09-24T11:54:23.681-0400 [initandlisten] closeAllFiles() finished
2014-09-24T11:54:23.681-0400 [initandlisten] journalCleanup...
2014-09-24T11:54:23.681-0400 [initandlisten] removeJournalFiles
2014-09-24T11:54:23.684-0400 [initandlisten] shutdown: removing fs lock...
2014-09-24T11:54:23.684-0400 [initandlisten] dbexit: really exiting now
The --repair flag is meant to repair a corrupt database, not to solve any permission problems
You obviously didn't follow the instructions of the link too closely, because he did start the service as root. ;) sudo may only allow certain parts of the init scripts to be executed with superuser rights by the user calling it. Others may (though they shouldn't) silently fail. As for init scripts at least: sudo != being root.
To fix the problem: sudo chown mongod -R /var/log/mongodb/ && sudo chown mongod -R /var/run/mongodb && sudo chown mongod -R /var/lib/mongo Then become root and try to run mongo manually with sudo -u mongod mongod --dbpath /var/lib/mongo --logpath /var/log/mongodb/mongod.log. In case there are other problems left, mongodb will tell you. Fix those until starting mongodb manually as written above works flawless, then use the initscript from that point on.
Edit: corrected the --dbpath option as per comment
Because of the size of the mongodb database I'm trying to build and because I needed a filesystem that can handle parallel writes, I moved it on a cluster that has lustreFS, which fulfills these needs.
The problem is that I can't start mongod, so I can't connect anything to the database, because mongod can't get a lock on mongod.lock. I don't have root or sudo permissions but this does not seem to be the problem here because permissions are not mentioned, as you can see:
[user#host mongo]$ ./mongod --dbpath=path-to-db
2014-07-28T20:29:58.600+0300 [initandlisten] MongoDB starting : pid=54974 port=27017 dbpath=path-to-db 64-bit host=hostname
2014-07-28T20:29:58.600+0300 [initandlisten]
2014-07-28T20:29:58.600+0300 [initandlisten] ** WARNING: /proc/sys/vm/zone_reclaim_mode is 1
2014-07-28T20:29:58.600+0300 [initandlisten] ** We suggest setting it to 0
2014-07-28T20:29:58.600+0300 [initandlisten] ** http://www.kernel.org/doc/Documentation/sysctl/vm.txt
2014-07-28T20:29:58.600+0300 [initandlisten] db version v2.6.3
2014-07-28T20:29:58.601+0300 [initandlisten] git version: 255f67a66f9603c59380b2a389e386910bbb52cb
2014-07-28T20:29:58.601+0300 [initandlisten] build info: Linux build12.nj1.10gen.cc 2.6.32-431.3.1.el6.x86_64 #1 SMP Fri Jan 3 21:39:27 UTC 2014 x86_64 BOOST_LIB_VERSION=1_49
2014-07-28T20:29:58.601+0300 [initandlisten] allocator: tcmalloc
2014-07-28T20:29:58.601+0300 [initandlisten] options: { storage: { dbPath: "path-to-db" } }
2014-07-28T20:29:58.614+0300 [initandlisten] exception in initAndListen: 10310 Unable to lock file: path-to-db/mongod.lock. Is a mongod instance already running?, terminating
2014-07-28T20:29:58.615+0300 [initandlisten] dbexit:
2014-07-28T20:29:58.615+0300 [initandlisten] shutdown: going to close listening sockets...
2014-07-28T20:29:58.615+0300 [initandlisten] shutdown: going to flush diaglog...
2014-07-28T20:29:58.615+0300 [initandlisten] shutdown: going to close sockets...
2014-07-28T20:29:58.615+0300 [initandlisten] shutdown: waiting for fs preallocator...
2014-07-28T20:29:58.615+0300 [initandlisten] shutdown: lock for final commit...
2014-07-28T20:29:58.615+0300 [initandlisten] shutdown: final commit...
2014-07-28T20:29:58.615+0300 [initandlisten] shutdown: closing all files...
2014-07-28T20:29:58.615+0300 [initandlisten] closeAllFiles() finished
2014-07-28T20:29:58.615+0300 [initandlisten] dbexit: really exiting now
Could you please tell me if there's any way around that lock? And no, there weren't any other mongod instances running.
Solution: File locking was being disabled by cluster admins. After file locking was enabled, mongod started and accepted connections as it should.