I tried to recover from a sudden shutdown of Mongo, but I'm experiencing some problems. Here is the log file:
***** SERVER RESTARTED *****
Thu Apr 25 01:09:17 [initandlisten] MongoDB starting : pid=19996 port=27017 dbpath=/var/lib/mongodb 64-bit host=Vault
Thu Apr 25 01:09:17 [initandlisten] db version v2.2.3, pdfile version 4.5
Thu Apr 25 01:09:17 [initandlisten] git version: f570771a5d8a3846eb7586eaffcf4c2f4a96bf08
Thu Apr 25 01:09:17 [initandlisten] build info: Linux ip-10-2-29-40 2.6.21.7-2.ec2.v1.2.fc8xen #1 SMP Fri Nov 20 17:48:28 EST 2009 x86_64 BOOST_LIB_VERSION=1_49
Thu Apr 25 01:09:17 [initandlisten] options: { config: "/etc/mongodb.conf", dbpath: "/var/lib/mongodb", logappend: "true", logpath: "/var/log/mongodb/mongodb.log", nohttpinterface: "true" }
Thu Apr 25 01:09:17 [initandlisten] journal dir=/var/lib/mongodb/journal
Thu Apr 25 01:09:17 [initandlisten] recover begin
Thu Apr 25 01:09:17 [initandlisten] recover lsn: 3778163987
Thu Apr 25 01:09:17 [initandlisten] recover /var/lib/mongodb/journal/j._0
Thu Apr 25 01:09:17 [initandlisten] dbexception during recovery: 13537 journal file header invalid. This could indicate corruption in a journal file, or perhaps a crash where sectors in file header were in flight written out of order at time of crash (unlikely but possible).
Thu Apr 25 01:09:17 [initandlisten] exception in initAndListen: 13537 journal file header invalid. This could indicate corruption in a journal file, or perhaps a crash where sectors in file header were in flight written out of order at time of crash (unlikely but possible)., terminating
Thu Apr 25 01:09:17 dbexit:
Thu Apr 25 01:09:17 [initandlisten] shutdown: going to close listening sockets...
Thu Apr 25 01:09:17 [initandlisten] shutdown: going to flush diaglog...
Thu Apr 25 01:09:17 [initandlisten] shutdown: going to close sockets...
Thu Apr 25 01:09:17 [initandlisten] shutdown: waiting for fs preallocator...
Thu Apr 25 01:09:17 [initandlisten] shutdown: lock for final commit...
Thu Apr 25 01:09:17 [initandlisten] shutdown: final commit...
Thu Apr 25 01:09:17 [initandlisten] shutdown: closing all files...
Thu Apr 25 01:09:17 [initandlisten] closeAllFiles() finished
Thu Apr 25 01:09:17 [initandlisten] shutdown: removing fs lock...
Thu Apr 25 01:09:17 dbexit: really exiting now
Based on your log file, you are using MongoDB version v2.2.3. This version is not officially supported by MongoDB, so I'd recommend you plan to upgrade to a newer version. The latest stable version, MongoDB 4.0.9.
As per the log message, the journal file j._0 in /var/lib/mongodb/journal/ folder is corrupted.
Try, removing the j._0 file and do the recovery process.
Disclaimer: you may lose some data that present in the journal file.
Related
When i try to run my mongod server on CentOS the result of npm start is this error:
root#server[Taxi]# npm start
> vframework#0.0.1 start /home/user/public_html/Taxi
> node server
/home/user/public_html/Taxi/node_modules/mongodb/node_modules/mongodb-core/lib/error.js:72
const MongoParseError = function(message) {
^^^^^
the version of mongod is 2.4.14 and when i run the #mongod -v command i got this error
root#server [Taxi]# mongod -v
Fri Mar 30 19:07:42.038 BackgroundJob starting: DataFileSync
Fri Mar 30 19:07:42.038 versionArrayTest passed
Fri Mar 30 19:07:42.069 shardKeyTest passed
Fri Mar 30 19:07:42.069 isInRangeTest passed
Fri Mar 30 19:07:42.069 shardObjTest passed
Fri Mar 30 19:07:42.096 [initandlisten] MongoDB starting : pid=24586 port=27017 dbpath=/data/db/ 64-bit host=server
Fri Mar 30 19:07:42.096 [initandlisten] db version v2.4.14
Fri Mar 30 19:07:42.096 [initandlisten] git version: nogitversion
Fri Mar 30 19:07:42.096 [initandlisten] build info: Linux buildvm-11.phx2.fedoraproject.org 4.7.2-201.fc24.x86_64 #1 SMP Fri Aug 26 15:58:40 UTC 2016 x86_64 BOOST_LIB_VERSION=1_41
Fri Mar 30 19:07:42.096 [initandlisten] allocator: tcmalloc
Fri Mar 30 19:07:42.096 [initandlisten] options: { verbose: true }
Fri Mar 30 19:07:42.138 [initandlisten] User Assertion: 10310:Unable to lock file: /data/db/mongod.lock. Is a mongod instance already running?
Fri Mar 30 19:07:42.176 [initandlisten] exception in initAndListen: 10310 Unable to lock file: /data/db/mongod.lock. Is a mongod instance already running?, terminating
Fri Mar 30 19:07:42.177 dbexit:
Fri Mar 30 19:07:42.177 [initandlisten] shutdown: going to close listening sockets...
Fri Mar 30 19:07:42.177 [initandlisten] shutdown: going to flush diaglog...
Fri Mar 30 19:07:42.177 [initandlisten] shutdown: going to close sockets...
Fri Mar 30 19:07:42.177 [initandlisten] shutdown: waiting for fs preallocator...
Fri Mar 30 19:07:42.177 [initandlisten] shutdown: lock for final commit...
Fri Mar 30 19:07:42.177 [initandlisten] shutdown: final commit...
Fri Mar 30 19:07:42.177 [initandlisten] shutdown: closing all files...
Fri Mar 30 19:07:42.179 [initandlisten] closeAllFiles() finished
Fri Mar 30 19:07:42.179 dbexit: really exiting now
i added a mongod.config file to path /etc/ fo specify the server adress and this is it's content
root#server [etc]# cat mongod.conf
logpath=/var/log/mongo/mongod.log
port=27027
bind_ip=0.0.0.0
dbpath=/var/lib/mongo
replSetName: rs0
My MongoDb Config file Content Is as below:
port=22000
bind_ip=127.0.0.1,192.168.1.121
auth=true
dbpath=D:\Data_For_Mongo
I am running my mongod instance using following command:
mongod -f "D:\mongoConfig.conf"
It Gives Me following Error:
ERROR: listen(): bind() failed errno:10049 The requested address is not valid in its context. for socket: 192.168.1.121:22000
The Whole Stack trace is As Below:
Fri Apr 11 09:24:18.600 [initandlisten] MongoDB starting : pid=3780 port=22000 dbpath=D:\Data_For_Mongo 64-bit host=SLK055
Fri Apr 11 09:24:18.600 [initandlisten] db version v2.4.6
Fri Apr 11 09:24:18.600 [initandlisten] git version: b9925db5eac369d77a3a5f5d98a145eaaacd9673
Fri Apr 11 09:24:18.600 [initandlisten] build info: windows sys.getwindowsversion(major=6, minor=1, build=7601, platform=2, service_pack='Service Pack 1') BOOST_LIB_VERSION=1_49
Fri Apr 11 09:24:18.615 [initandlisten] allocator: system
Fri Apr 11 09:24:18.615 [initandlisten] options: { auth: "true", bind_ip: "192.168.1.80,192.168.1.121", config: "D:\mongoConfig.conf", dbpath: "D:\Data_For_Mongo", port: 22000 }
Fri Apr 11 09:24:18.647 [initandlisten] journal dir=D:\Data_For_Mongo\journal
Fri Apr 11 09:24:18.647 [initandlisten] recover : no journal files present, no recovery needed
Fri Apr 11 09:24:18.709 [initandlisten] ERROR: listen(): bind() failed errno:10049 The requested address is not valid in its context. for socket: 192.168.1.121:22000
Fri Apr 11 09:24:18.709 [websvr] ERROR: listen(): bind() failed errno:10049 The requested address is not valid in its context. for socket: 192.168.1.121:23000
Fri Apr 11 09:24:18.709 [initandlisten] now exiting
Fri Apr 11 09:24:18.725 dbexit:
Fri Apr 11 09:24:18.725 [initandlisten] shutdown: going to close listening sockets...
Fri Apr 11 09:24:18.740 [initandlisten] closing listening socket: 528
Fri Apr 11 09:24:18.740 [initandlisten] closing listening socket: 552
Fri Apr 11 09:24:18.740 [initandlisten] shutdown: going to flush diaglog...
Fri Apr 11 09:24:18.756 [initandlisten] shutdown: going to close sockets...
Fri Apr 11 09:24:18.756 [initandlisten] shutdown: waiting for fs preallocator...
Fri Apr 11 09:24:18.772 [initandlisten] shutdown: lock for final commit...
Fri Apr 11 09:24:18.772 [initandlisten] shutdown: final commit...
Fri Apr 11 09:24:18.834 [initandlisten] shutdown: closing all files...
Fri Apr 11 09:24:18.834 [initandlisten] closeAllFiles() finished
Fri Apr 11 09:24:18.834 [initandlisten] journalCleanup...
Fri Apr 11 09:24:18.834 [initandlisten] removeJournalFiles
Fri Apr 11 09:24:18.850 [initandlisten] shutdown: removing fs lock...
Fri Apr 11 09:24:18.850 dbexit: really exiting now
Bind Multiple ips is important for me.
Can AnyOne Help Me To Solve This Error?
I Have searched about error but can't find any way to bind multiple ips in mongo.
Thats a Windows Socket error. This is normally due to the IP address you are trying to bind to not being the IP of your local machine. You should double check that your system actually has the IP of 192.168.1.121
Apologise for reopen another post with the same problem but I tried all the provided solutions and it didn't work.
My logs says:
root#marcproves:/var/log/mongodb# tail -f mongodb.log
Tue Apr 1 04:25:29.272 [initandlisten] shutdown: going to close sockets...
Tue Apr 1 04:25:29.273 [initandlisten] shutdown: waiting for fs preallocator...
Tue Apr 1 04:25:29.273 [initandlisten] shutdown: lock for final commit...
Tue Apr 1 04:25:29.273 [initandlisten] shutdown: final commit...
Tue Apr 1 04:25:29.273 [initandlisten] shutdown: closing all files...
Tue Apr 1 04:25:29.273 [initandlisten] closeAllFiles() finished
Tue Apr 1 04:25:29.273 [initandlisten] journalCleanup...
Tue Apr 1 04:25:29.273 [initandlisten] removeJournalFiles
Tue Apr 1 04:25:29.275 [initandlisten] shutdown: removing fs lock...
Tue Apr 1 04:25:29.275 dbexit: really exiting now
and my console output:
root#marcproves:/var/log/mongodb# mongo
MongoDB shell version: 2.4.9
connecting to: test
Tue Apr 1 04:55:28.679 Error: couldn't connect to server 127.0.0.1:27017 at src/mongo/shell/mongo.js:145
exception: connect failed
root#marcproves:/var/log/mongodb#
I installed mongo as it says in the mongo official webpage for ubuntu version and this happens!
FIXED: SPACE ISSUE
So I have a 2TB hard drive with 2 collections in 1 mongoDB.
Collection 1 is about 500gb and completely essential
Collection 2 is about 1500gb and needs to be deleted.
I was running a process that accidentally filled up the entire HDD, leaving 0% space. There is literally nothing on the machine apart from a Linux Distro and this mongoDB.
I would like to delete Collection 2.
In order to do this, I would normally just go into mongo and type db.collection.drop()
However, mongoDB is not currently running. In order to start it, I'm using:
[bobby#myPC bin]# ./mongod --dbpath /home/mongo &
However, since there is no space on the HDD, it returns:
[bobby#myPC bin]# Thu Feb 14 17:47:10 [initandlisten] MongoDB starting : pid=1264 port=27017 dbpath=/home/mongo 64-bit host=localhost.localdomain
Thu Feb 14 17:47:10 [initandlisten] db version v2.2.3, pdfile version 4.5
Thu Feb 14 17:47:10 [initandlisten] git version: f570771a5d8a3846eb7586eaffcf4c2f4a96bf08
Thu Feb 14 17:47:10 [initandlisten] build info: Linux ip-10-2-29-40 2.6.21.7-2.ec2.v1.2.fc8xen #1 SMP Fri Nov 20 17:48:28 EST 2009 x86_64 BOOST_LIB_VERSION=1_49
Thu Feb 14 17:47:10 [initandlisten] options: { dbpath: "/home/mongo" }
Thu Feb 14 17:47:10 [initandlisten] journal dir=/home/mongo/journal
Thu Feb 14 17:47:10 [initandlisten] recover : no journal files present, no recovery needed
Thu Feb 14 17:47:10 [initandlisten]
Thu Feb 14 17:47:10 [initandlisten] ERROR: Insufficient free space for journal files
Thu Feb 14 17:47:10 [initandlisten] Please make at least 3379MB available in /home/mongo/journal or use --smallfiles
Thu Feb 14 17:47:10 [initandlisten]
Thu Feb 14 17:47:10 [initandlisten] exception in initAndListen: 15926 Insufficient free space for journals, terminating
Thu Feb 14 17:47:10 dbexit:
Thu Feb 14 17:47:10 [initandlisten] shutdown: going to close listening sockets...
Thu Feb 14 17:47:10 [initandlisten] shutdown: going to flush diaglog...
Thu Feb 14 17:47:10 [initandlisten] shutdown: going to close sockets...
Thu Feb 14 17:47:10 [initandlisten] shutdown: waiting for fs preallocator...
Thu Feb 14 17:47:10 [initandlisten] shutdown: lock for final commit...
Thu Feb 14 17:47:10 [initandlisten] shutdown: final commit...
Thu Feb 14 17:47:10 [initandlisten] shutdown: closing all files...
Thu Feb 14 17:47:10 [initandlisten] closeAllFiles() finished
Thu Feb 14 17:47:10 [initandlisten] journalCleanup...
Thu Feb 14 17:47:10 [initandlisten] removeJournalFiles
Thu Feb 14 17:47:10 [initandlisten] shutdown: removing fs lock...
Thu Feb 14 17:47:10 dbexit: really exiting now
I can't remove this collection cause there's no space. And there's no space because I can't remove this collection.
How can I fix this? Is there some way of identifying which mongoDB files in /home/mongo I can take out?
Based on the error message, its complaining about not having enough space for the journal. Its not ideal in a production environment to run with out journaling, but you could try starting with out it while you fix the issue.
./mongod --nojournal --dbpath /home/mongo &
edit: to answer your question though, mongo creates and allocates files per database, not collection. So you wouldn't be able to target a specific file to remove a collection.
I was running MongoDB on Amazon AWS, until one day it ran out of space. Then, the database is completely inaccessible,and I tried to shutdown the database. Having restarted the server and installed larger disk, the MongoDB can't be started.
I tried to repair the database, and the error is shown like below:
Wed Aug 15 10:08:04 [initandlisten] MongoDB starting : pid=1447 port=27017 dbpath=/var/lib/mongodb/ 32-bit
** NOTE: when using MongoDB 32 bit, you are limited to about 2 gigabytes of data
** see http://blog.mongodb.org/post/137788967/32-bit-limitations
** with --dur, the limit is lower
Wed Aug 15 10:08:04 [initandlisten] db version v1.8.3, pdfile version 4.5
Wed Aug 15 10:08:04 [initandlisten] git version: c206d77e94bc3b65c76681df5a6b605f68a2de05
Wed Aug 15 10:08:04 [initandlisten] build sys info: Linux domU-12-31-39-01-70-B4 2.6.21.7-2.fc8xen #1 SMP Fri Feb 15 12:39:36 EST 2008 i686 BOOST_LIB_VERSION=1_37
Wed Aug 15 10:08:04 [initandlisten] **
Wed Aug 15 10:08:04 [initandlisten] **
Wed Aug 15 10:08:04 [initandlisten] need to upgrade database md with pdfile version 559903.1345021735, new version: 4.5
Wed Aug 15 10:08:04 [initandlisten] starting upgrade
Wed Aug 15 10:08:04 [initandlisten] md repairDatabase md
Wed Aug 15 10:08:04 [initandlisten] md.system.namespaces Assertion failure isOk() db/pdfile.h 259
0x817c27b 0x819121e 0x83cf525 0x83d6bec 0x83f5420 0x82a4fb7 0x83f8680 0x83fcece 0x8401358 0x8401e29 0x8402abd 0x828f755 0x83b3d67 0x83b931c 0x83bb3df 0x81ce25f 0x81ae714 0x8380f31 0x83831ba 0x83d4faf
mongod(_ZN5mongo12sayDbContextEPKc+0xcb) [0x817c27b]
mongod(_ZN5mongo8assertedEPKcS1_j+0x12e) [0x819121e]
mongod(_ZN5mongo11DataFileMgr7findAllEPKcRKNS_7DiskLocE+0x7a5) [0x83cf525]
mongod(_ZN5mongo13findTableScanEPKcRKNS_7BSONObjERKNS_7DiskLocE+0x13c) [0x83d6bec]
mongod(_ZNK5mongo9QueryPlan9newCursorERKNS_7DiskLocEi+0x830) [0x83f5420]
mongod(_ZN5mongo11UserQueryOp5_initEv+0x437) [0x82a4fb7]
mongod(_ZN5mongo12QueryPlanSet6Runner6initOpERNS_7QueryOpE+0x110) [0x83f8680]
mongod(_ZN5mongo12QueryPlanSet6Runner3runEv+0x2ee) [0x83fcece]
mongod(_ZN5mongo12QueryPlanSet5runOpERNS_7QueryOpE+0x2e8) [0x8401358]
mongod(_ZN5mongo16MultiPlanScanner9runOpOnceERNS_7QueryOpE+0x59) [0x8401e29]
mongod(_ZN5mongo16MultiPlanScanner5runOpERNS_7QueryOpE+0x2d) [0x8402abd]
mongod(_ZN5mongo8runQueryERNS_7MessageERNS_12QueryMessageERNS_5CurOpES1_+0x1575) [0x828f755]
mongod() [0x83b3d67]
mongod(_ZN5mongo16assembleResponseERNS_7MessageERNS_10DbResponseERKNS_8SockAddrE+0x70c) [0x83b931c]
mongod(_ZN5mongo14DBDirectClient4callERNS_7MessageES2_bPSs+0x7f) [0x83bb3df]
mongod(_ZN5mongo14DBClientCursor4initEv+0x15f) [0x81ce25f]
mongod(_ZN5mongo12DBClientBase5queryERKSsNS_5QueryEiiPKNS_7BSONObjEii+0x2a4) [0x81ae714]
mongod(_ZN5mongo6Cloner2goEPKcRSsRKSsbbbb+0x451) [0x8380f31]
mongod(_ZN5mongo9cloneFromEPKcRSsRKSsbbbb+0x5a) [0x83831ba]
mongod(_ZN5mongo14repairDatabaseESsRSsbb+0x38f) [0x83d4faf]
Wed Aug 15 10:08:04 [initandlisten] assertion 0 assertion db/pdfile.h:259 ns:md.system.namespaces query:{}
Wed Aug 15 10:08:04 [initandlisten] Assertion: 10290:bad system.namespaces object { $err: "assertion db/pdfile.h:259" }
0x819045e 0x8381181 0x83831ba 0x83d4faf 0x852a30c 0x852b217 0x852e733 0x852f59c 0x85366bb 0xb74a4ce7 0x810b551
mongod(_ZN5mongo11msgassertedEiPKc+0x15e) [0x819045e]
mongod(_ZN5mongo6Cloner2goEPKcRSsRKSsbbbb+0x6a1) [0x8381181]
mongod(_ZN5mongo9cloneFromEPKcRSsRKSsbbbb+0x5a) [0x83831ba]
mongod(_ZN5mongo14repairDatabaseESsRSsbb+0x38f) [0x83d4faf]
mongod(_ZN5mongo11doDBUpgradeERKSsSsPNS_14DataFileHeaderE+0x6c) [0x852a30c]
mongod() [0x852b217]
mongod(_ZN5mongo14_initAndListenEiPKc+0x4b3) [0x852e733]
mongod(ZN5mongo13initAndListenEiPKc+0x2c) [0x852f59c]
mongod(main+0x6bdb) [0x85366bb]
/lib/libc.so.6(_libc_start_main+0xe7) [0xb74a4ce7]
mongod() [0x810b551]
Wed Aug 15 10:08:04 [initandlisten] exception in initAndListen std::exception: bad system.namespaces object { $err: "assertion db/pdfile.h:259" }, terminating
Wed Aug 15 10:08:04 dbexit:
Wed Aug 15 10:08:04 [initandlisten] shutdown: going to close listening sockets...
Wed Aug 15 10:08:04 [initandlisten] shutdown: going to flush diaglog...
Wed Aug 15 10:08:04 [initandlisten] shutdown: going to close sockets...
Wed Aug 15 10:08:04 [initandlisten] shutdown: waiting for fs preallocator...
Wed Aug 15 10:08:04 [initandlisten] shutdown: closing all files...
Wed Aug 15 10:08:04 closeAllFiles() finished
Wed Aug 15 10:08:04 [initandlisten] shutdown: removing fs lock...
Wed Aug 15 10:08:04 dbexit: really exiting now
I have all *.number and *.ns file completely. But it seems the ns file is corrupted. Any way I could repair the database?
Thanks
If repair doesn't work then you are unfortunately out of luck.
I'm presuming that you didn't have journaling enabled or any back-ups. I would strongly encourage you to run with journaling enabled and follow one of the back-up strategies outlined here. The LVM snapshot is easy to implement, fairly quick and reliable.
You are better to run MongoDB on a 64-bit system also. 32-bit limits you to 2gb of data. You say that you "ran out of space"?