mongodb terminated and won't come back up. Error 13532 - mongodb

Running mongodb 2.6.4 on Win2008-R2.
Full of data.... and today it went down, and won't come back up.
What to do?
2015-01-27T02:03:58.781-0600 Trying to start Windows service 'MongoDB'
2015-01-27T02:03:58.781-0600 Service running
2015-01-27T02:03:58.781-0600 [initandlisten] MongoDB starting : pid=1864 port=27017 dbpath=c:\MongoDB\Data 64-bit host=619723-MONGO-1
2015-01-27T02:03:58.781-0600 [initandlisten] targetMinOS: Windows 7/Windows Server 2008 R2
2015-01-27T02:03:58.781-0600 [initandlisten] db version v2.6.4
2015-01-27T02:03:58.781-0600 [initandlisten] git version: 3a830be0eb92d772aa855ebb711ac91d658ee910
2015-01-27T02:03:58.781-0600 [initandlisten] build info: windows sys.getwindowsversion(major=6, minor=1, build=7601, platform=2, service_pack='Service Pack 1') BOOST_LIB_VERSION=1_49
2015-01-27T02:03:58.781-0600 [initandlisten] allocator: system
2015-01-27T02:03:58.781-0600 [initandlisten] options: { config: "C:\MongoDB\mongod.cfg", service: true, storage: { dbPath: "c:\MongoDB\Data", directoryPerDB: true }, systemLog: { destination: "file", path: "c:\MongoDB\logs" } }
2015-01-27T02:03:58.890-0600 [initandlisten] journal dir=c:\MongoDB\Data\journal
2015-01-27T02:03:58.968-0600 [initandlisten] dbexception during recovery: 13532 unexpected file in journal directory c:\MongoDB\Data\journal : j._1895 : can't find its preceding file
2015-01-27T02:03:58.984-0600 [initandlisten] exception in initAndListen: 13532 unexpected file in journal directory c:\MongoDB\Data\journal : j._1895 : can't find its preceding file, terminating
2015-01-27T02:03:58.999-0600 [initandlisten] dbexit:
2015-01-27T02:03:58.999-0600 [initandlisten] shutdown: going to close listening sockets...
2015-01-27T02:03:58.999-0600 [initandlisten] shutdown: going to flush diaglog...
2015-01-27T02:03:58.999-0600 [initandlisten] shutdown: going to close sockets...
2015-01-27T02:03:58.999-0600 [initandlisten] shutdown: waiting for fs preallocator...
2015-01-27T02:03:58.999-0600 [initandlisten] shutdown: lock for final commit...
2015-01-27T02:03:58.999-0600 [initandlisten] shutdown: final commit...
2015-01-27T02:03:58.999-0600 [initandlisten] shutdown: closing all files...
2015-01-27T02:03:58.999-0600 [initandlisten] closeAllFiles() finished
2015-01-27T02:03:58.999-0600 [initandlisten] shutdown: removing fs lock...
2015-01-27T02:03:58.999-0600 [initandlisten] dbexit: really exiting now

Well, google did not seem to have anything useful on mongo error 13532
What we did:
a) Deleted all files in /journal dir
b) Deleted the mongo .lock file
c) From cmd line: mongod --repair --dbpath c:\mongodb\data\
d) Started up mongo service
Up now for several hours OK.
Not sure what the data loss was.

Related

Mongod unclean shutdown

new to this ..so my MONGOD unexpectedly shutdown and i can't seem to repair it ..this is what i see when i try to start up ./mongod
**2018-04-21T19:59:06.237+0000 ** WARNING: --rest is specified without --httpinterface,
2018-04-21T19:59:06.238+0000 ** enabling http interface
warning: bind_ip of 0.0.0.0 is unnecessary; listens on all ips by default
2018-04-21T19:59:06.241+0000 [initandlisten] MongoDB starting : pid=2848 port=27017 dbpath=data 64-bit host=omarr11-webdevbootcamp-5900192
2018-04-21T19:59:06.241+0000 [initandlisten] db version v2.6.12
2018-04-21T19:59:06.241+0000 [initandlisten] git version: d73c92b1c85703828b55c2916a5dd4ad46535f6a
2018-04-21T19:59:06.241+0000 [initandlisten] build info: Linux build5.ny.cbi.10gen.cc 2.6.32-431.3.1.el6.x86_64 #1 SMP Fri Jan 3 21:39:27 UTC 2014 x86_64 BOOST_LIB_VERSION=1_49
2018-04-21T19:59:06.241+0000 [initandlisten] allocator: tcmalloc
2018-04-21T19:59:06.241+0000 [initandlisten] options: { net: { bindIp: "0.0.0.0", http: { RESTInterfaceEnabled: true, enabled: true } }, storage: { dbPath: "data", journal: { enabled: false } } }
**************
Unclean shutdown detected.
Please visit http://dochub.mongodb.org/core/repair for recovery instructions.
*************
2018-04-21T19:59:06.241+0000 [initandlisten] exception in initAndListen: 12596 old lock file, terminating
2018-04-21T19:59:06.241+0000 [initandlisten] dbexit:
2018-04-21T19:59:06.241+0000 [initandlisten] shutdown: going to close listening sockets...
2018-04-21T19:59:06.241+0000 [initandlisten] shutdown: going to flush diaglog...
2018-04-21T19:59:06.241+0000 [initandlisten] shutdown: going to close sockets...
2018-04-21T19:59:06.241+0000 [initandlisten] shutdown: waiting for fs preallocator...
2018-04-21T19:59:06.242+0000 [initandlisten] shutdown: closing all files...
2018-04-21T19:59:06.242+0000 [initandlisten] closeAllFiles() finished
2018-04-21T19:59:06.242+0000 [initandlisten] dbexit: really exiting now**
can someone help me repair this please?
You should look for the lock file (mongod.lock) in the data folder of
the mongodb and you need to delete it and try to start again.
In case you have some data corruption issue you can try repair option https://docs.mongodb.com/manual/tutorial/recover-data-following-unexpected-shutdown/ . But in case it is just normal course of learning deleting the mongod.lock file should start the mongod process again...

mongodb: Journal error. Which collection is referred to?

We have two collections.
So I have to delete at least one, because mongo won't restart (losing 400gb of data in the process, but that is normal when using mongo).
From this log, I cannot tell which one to delete. Is there a clue I am missing?
2015-01-27T02:03:58.781-0600 Trying to start Windows service 'MongoDB'
2015-01-27T02:03:58.781-0600 Service running
2015-01-27T02:03:58.781-0600 [initandlisten] MongoDB starting : pid=1864 port=27017 dbpath=c:\MongoDB\Data 64-bit host=619723-MONGO-1
2015-01-27T02:03:58.781-0600 [initandlisten] targetMinOS: Windows 7/Windows Server 2008 R2
2015-01-27T02:03:58.781-0600 [initandlisten] db version v2.6.4
2015-01-27T02:03:58.781-0600 [initandlisten] git version: 3a830be0eb92d772aa855ebb711ac91d658ee910
2015-01-27T02:03:58.781-0600 [initandlisten] build info: windows sys.getwindowsversion(major=6, minor=1, build=7601, platform=2, service_pack='Service Pack 1') BOOST_LIB_VERSION=1_49
2015-01-27T02:03:58.781-0600 [initandlisten] allocator: system
2015-01-27T02:03:58.781-0600 [initandlisten] options: { config: "C:\MongoDB\mongod.cfg", service: true, storage: { dbPath: "c:\MongoDB\Data", directoryPerDB: true }, systemLog: { destination: "file", path: "c:\MongoDB\logs" } }
2015-01-27T02:03:58.890-0600 [initandlisten] journal dir=c:\MongoDB\Data\journal
2015-01-27T02:03:58.968-0600 [initandlisten] dbexception during recovery: 13532 unexpected file in journal directory c:\MongoDB\Data\journal : j._1895 : can't find its preceding file
2015-01-27T02:03:58.984-0600 [initandlisten] exception in initAndListen: 13532 unexpected file in journal directory c:\MongoDB\Data\journal : j._1895 : can't find its preceding file, terminating
2015-01-27T02:03:58.999-0600 [initandlisten] dbexit:
2015-01-27T02:03:58.999-0600 [initandlisten] shutdown: going to close listening sockets...
2015-01-27T02:03:58.999-0600 [initandlisten] shutdown: going to flush diaglog...
2015-01-27T02:03:58.999-0600 [initandlisten] shutdown: going to close sockets...
2015-01-27T02:03:58.999-0600 [initandlisten] shutdown: waiting for fs preallocator...
2015-01-27T02:03:58.999-0600 [initandlisten] shutdown: lock for final commit...
2015-01-27T02:03:58.999-0600 [initandlisten] shutdown: final commit...
2015-01-27T02:03:58.999-0600 [initandlisten] shutdown: closing all files...
2015-01-27T02:03:58.999-0600 [initandlisten] closeAllFiles() finished
2015-01-27T02:03:58.999-0600 [initandlisten] shutdown: removing fs lock...
2015-01-27T02:03:58.999-0600 [initandlisten] dbexit: really exiting now
Our plan was to reset the databases, but we did a repair instead.

MongoDB instance not starting

I have installed MongoDB on Mac OSX. When I do mongod it does not start any connections; instead it displays the output below. Can someone please help me to resolve this
$ mongod
mongod --help for help and startup options
2015-01-16T09:57:18.994-0500 [initandlisten] MongoDB starting : pid=1571 port=27017 dbpath=/data/db 64-bit host=3c15c2e72774
2015-01-16T09:57:18.995-0500 [initandlisten]
2015-01-16T09:57:18.995-0500 [initandlisten] ** WARNING: soft rlimits too low. Number of files is 256, should be at least 1000
2015-01-16T09:57:18.995-0500 [initandlisten] db version v2.6.7
2015-01-16T09:57:18.995-0500 [initandlisten] git version: nogitversion
2015-01-16T09:57:18.995-0500 [initandlisten] build info: Darwin minimavericks.local 13.4.0 Darwin Kernel Version 13.4.0: Sun Aug 17 19:50:11 PDT 2014; root:xnu-2422.115.4~1/RELEASE_X86_64 x86_64 BOOST_LIB_VERSION=1_49
2015-01-16T09:57:18.995-0500 [initandlisten] allocator: tcmalloc
2015-01-16T09:57:18.995-0500 [initandlisten] options: {}
2015-01-16T09:57:18.996-0500 [initandlisten] journal dir=/data/db/journal
2015-01-16T09:57:18.996-0500 [initandlisten] recover : no journal files present, no recovery needed
2015-01-16T09:57:19.056-0500 [initandlisten] bad .ns file: /data/db/local.ns
2015-01-16T09:57:19.056-0500 [initandlisten] warning database /data/db local could not be opened
2015-01-16T09:57:19.056-0500 [initandlisten] DBException 10079: bad .ns file length, cannot open database
2015-01-16T09:57:19.056-0500 [initandlisten] exception in initAndListen: 10079 bad .ns file length, cannot open database, terminating
2015-01-16T09:57:19.056-0500 [initandlisten] dbexit:
2015-01-16T09:57:19.056-0500 [initandlisten] shutdown: going to close listening sockets...
2015-01-16T09:57:19.056-0500 [initandlisten] shutdown: going to flush diaglog...
2015-01-16T09:57:19.056-0500 [initandlisten] shutdown: going to close sockets...
2015-01-16T09:57:19.056-0500 [initandlisten] shutdown: waiting for fs preallocator...
2015-01-16T09:57:19.056-0500 [initandlisten] shutdown: lock for final commit...
2015-01-16T09:57:19.056-0500 [initandlisten] shutdown: final commit...
2015-01-16T09:57:19.056-0500 [initandlisten] shutdown: closing all files...
2015-01-16T09:57:19.056-0500 [initandlisten] closeAllFiles() finished
2015-01-16T09:57:19.056-0500 [initandlisten] journalCleanup...
2015-01-16T09:57:19.060-0500 [initandlisten] removeJournalFiles
2015-01-16T09:57:19.060-0500 [initandlisten] shutdown: removing fs lock...
2015-01-16T09:57:19.060-0500 [initandlisten] dbexit: really exiting now
Remove any old or corrupted .ns files in /data/db and then restart mongodb
I had this problem. I solved it by increasing the memory size for the replica using Virtual Box.

Resstarting MongoDB after crash

After a crash of MongoDB I currently cannot restart the database. I cleaned up the .lock file but still get the error:
2014-12-23T19:55:41.922+0100 [initandlisten] MongoDB starting : pid=4880 port=27
017 dbpath=c:/Program Files/MongoDB 2.6 Standard/bin/asd 64-bit host=Lenovo-PC
2014-12-23T19:55:41.922+0100 [initandlisten] targetMinOS: Windows 7/Windows Serv
er 2008 R2
2014-12-23T19:55:41.922+0100 [initandlisten] db version v2.6.6
2014-12-23T19:55:41.922+0100 [initandlisten] git version: 608e8bc319627693b04cc7
da29ecc300a5f45a1f
2014-12-23T19:55:41.922+0100 [initandlisten] build info: windows sys.getwindowsv
ersion(major=6, minor=1, build=7601, platform=2, service_pack='Service Pack 1')
BOOST_LIB_VERSION=1_49
2014-12-23T19:55:41.922+0100 [initandlisten] allocator: system
2014-12-23T19:55:41.922+0100 [initandlisten] options: { storage: { dbPath: "c:/P
rogram Files/MongoDB 2.6 Standard/bin/asd" } }
2014-12-23T19:55:41.938+0100 [initandlisten] exception in initAndListen: 13627 U
nable to create/open lock file: c:/Program Files/MongoDB 2.6 Standard/bin/asd\mo
ngod.lock Zugriff verweigert. Is a mongod instance already running?, terminating
2014-12-23T19:55:41.938+0100 [initandlisten] dbexit:
2014-12-23T19:55:41.938+0100 [initandlisten] shutdown: going to close listening
sockets...
2014-12-23T19:55:41.938+0100 [initandlisten] shutdown: going to flush diaglog...
2014-12-23T19:55:41.938+0100 [initandlisten] shutdown: going to close sockets...
2014-12-23T19:55:41.938+0100 [initandlisten] shutdown: waiting for fs preallocat
or...
2014-12-23T19:55:41.938+0100 [initandlisten] shutdown: lock for final commit...
2014-12-23T19:55:41.938+0100 [initandlisten] shutdown: final commit...
2014-12-23T19:55:41.938+0100 [initandlisten] shutdown: closing all files...
2014-12-23T19:55:41.938+0100 [initandlisten] closeAllFiles() finished
2014-12-23T19:55:41.938+0100 [initandlisten] dbexit: really exiting now
when exciting mongodb with "mongod --dbpath PATH" where path is the path to the folder, where the database date shoulb be positioned.
The system runs under a virtual environment within windows.

mongodb: Is there any way of stoping mongod from trying to set a lock on mongod.lock?

Because of the size of the mongodb database I'm trying to build and because I needed a filesystem that can handle parallel writes, I moved it on a cluster that has lustreFS, which fulfills these needs.
The problem is that I can't start mongod, so I can't connect anything to the database, because mongod can't get a lock on mongod.lock. I don't have root or sudo permissions but this does not seem to be the problem here because permissions are not mentioned, as you can see:
[user#host mongo]$ ./mongod --dbpath=path-to-db
2014-07-28T20:29:58.600+0300 [initandlisten] MongoDB starting : pid=54974 port=27017 dbpath=path-to-db 64-bit host=hostname
2014-07-28T20:29:58.600+0300 [initandlisten]
2014-07-28T20:29:58.600+0300 [initandlisten] ** WARNING: /proc/sys/vm/zone_reclaim_mode is 1
2014-07-28T20:29:58.600+0300 [initandlisten] ** We suggest setting it to 0
2014-07-28T20:29:58.600+0300 [initandlisten] ** http://www.kernel.org/doc/Documentation/sysctl/vm.txt
2014-07-28T20:29:58.600+0300 [initandlisten] db version v2.6.3
2014-07-28T20:29:58.601+0300 [initandlisten] git version: 255f67a66f9603c59380b2a389e386910bbb52cb
2014-07-28T20:29:58.601+0300 [initandlisten] build info: Linux build12.nj1.10gen.cc 2.6.32-431.3.1.el6.x86_64 #1 SMP Fri Jan 3 21:39:27 UTC 2014 x86_64 BOOST_LIB_VERSION=1_49
2014-07-28T20:29:58.601+0300 [initandlisten] allocator: tcmalloc
2014-07-28T20:29:58.601+0300 [initandlisten] options: { storage: { dbPath: "path-to-db" } }
2014-07-28T20:29:58.614+0300 [initandlisten] exception in initAndListen: 10310 Unable to lock file: path-to-db/mongod.lock. Is a mongod instance already running?, terminating
2014-07-28T20:29:58.615+0300 [initandlisten] dbexit:
2014-07-28T20:29:58.615+0300 [initandlisten] shutdown: going to close listening sockets...
2014-07-28T20:29:58.615+0300 [initandlisten] shutdown: going to flush diaglog...
2014-07-28T20:29:58.615+0300 [initandlisten] shutdown: going to close sockets...
2014-07-28T20:29:58.615+0300 [initandlisten] shutdown: waiting for fs preallocator...
2014-07-28T20:29:58.615+0300 [initandlisten] shutdown: lock for final commit...
2014-07-28T20:29:58.615+0300 [initandlisten] shutdown: final commit...
2014-07-28T20:29:58.615+0300 [initandlisten] shutdown: closing all files...
2014-07-28T20:29:58.615+0300 [initandlisten] closeAllFiles() finished
2014-07-28T20:29:58.615+0300 [initandlisten] dbexit: really exiting now
Could you please tell me if there's any way around that lock? And no, there weren't any other mongod instances running.
Solution: File locking was being disabled by cluster admins. After file locking was enabled, mongod started and accepted connections as it should.