I ran db.fsyncLock() command and then the server was restarted without me running db.fsyncUnlock().
After the restart, mongo does not start with the error
2020-04-17T10:13:59.693+0300 Error: couldn't connect to server
127.0.0.1:27017 (127.0.0.1), connection attempt failed at src/mongo/shell/mongo.js:146
What should I do? Can I just delete the lock file?
Thanks a lot!
Below are the logs and the ls of the directory
1583 2020-04-17T08:08:02.481+0300 [initandlisten] shutdown: going to close sockets...
1584 2020-04-17T08:08:02.481+0300 [initandlisten] shutdown: waiting for fs preallocator...
1585 2020-04-17T08:08:02.481+0300 [initandlisten] shutdown: lock for final commit...
1586 2020-04-17T08:08:02.481+0300 [initandlisten] shutdown: final commit...
1587 2020-04-17T08:08:02.481+0300 [initandlisten] shutdown: closing all files...
1588 2020-04-17T08:08:02.481+0300 [initandlisten] closeAllFiles() finished
1589 2020-04-17T08:08:02.481+0300 [initandlisten] journalCleanup...
1590 2020-04-17T08:08:02.481+0300 [initandlisten] removeJournalFiles
1591 2020-04-17T08:08:02.486+0300 [initandlisten] shutdown: removing fs lock...
1592 2020-04-17T08:08:02.486+0300 [initandlisten] dbexit: really exiting now
1593 ~$ ls -l /var/lib/mongodb
1594 total 294932
1595 -rw------- 1 mongodb mongodb 67108864 Apr 16 21:57 ctf.0
1596 -rw------- 1 mongodb mongodb 134217728 Apr 16 21:57 ctf.1
1597 -rw------- 1 mongodb mongodb 16777216 Apr 16 21:56 ctf.ns
1598 drwxr-xr-x 2 mongodb mongodb 4096 Apr 16 22:04 journal
1599 -rw------- 1 mongodb mongodb 67108864 Feb 26 04:47 local.0
1600 -rw------- 1 mongodb mongodb 16777216 Feb 26 04:47 local.ns
1601 -rwxr-xr-x 1 mongodb mongodb 0 Apr 17 08:31 mongod.lock
Related
Previously mongodb was located at /var/lib/mongod. I've moved this to /home/mongod, and updated the /etc/mongod.conf file to point to this location. The permissions of the folder look correct:
drwxrwxr-x. 2 mongod mongod 6 Aug 6 10:37 admin
-rw-------. 1 mongod mongod 67108864 Aug 6 11:49 admin.0
-rw-------. 1 mongod mongod 16777216 Aug 6 11:49 admin.ns
drwxrwxr-x. 2 mongod mongod 6 Aug 6 10:40 journal
drwxrwxr-x. 2 mongod mongod 6 Aug 6 10:37 local
-rw-------. 1 mongod mongod 67108864 Aug 6 11:49 local.0
-rw-------. 1 mongod mongod 16777216 Aug 6 11:49 local.ns
-rwxr-xr-x. 1 mongod mongod 0 Aug 6 11:50 mongod.lock
-rwxrwxr-x. 1 mongod mongod 0 Aug 6 10:12 mongod.lock.bak
drwxrwxr-x. 2 mongod mongod 6 Aug 6 10:37 sphere
-rw-------. 1 mongod mongod 67108864 Aug 6 11:50 sphere.0
-rw-------. 1 mongod mongod 134217728 Aug 6 11:49 sphere.1
-rw-------. 1 mongod mongod 268435456 Aug 6 11:49 sphere.2
-rw-------. 1 mongod mongod 536870912 Aug 6 11:49 sphere.3
-rw-------. 1 mongod mongod 1073741824 Aug 6 11:50 sphere.4
-rw-------. 1 mongod mongod 2146435072 Aug 6 11:50 sphere.5
-rw-------. 1 mongod mongod 16777216 Aug 6 11:50 sphere.ns
drwxr-xr-x. 2 mongod mongod 6 Aug 6 11:50 _tmp
and when I run from the command line it works:
[user#localhost home]$ sudo -u mongod mongod --dbpath /home/mongod
[sudo] password for user:
2015-08-06T12:04:35.761+0100 [initandlisten] MongoDB starting : pid=10235 port=27017 dbpath=/home/mongod 64-bit host=localhost.localdomain
2015-08-06T12:04:35.762+0100 [initandlisten] db version v2.6.10
2015-08-06T12:04:35.762+0100 [initandlisten] git version: 5901dbfb49d16eaef6f2c2c50fba534d23ac7f6c
2015-08-06T12:04:35.762+0100 [initandlisten] build info: Linux build18.nj1.10gen.cc 2.6.32-431.3.1.el6.x86_64 #1 SMP Fri Jan 3 21:39:27 UTC 2014 x86_64 BOOST_LIB_VERSION=1_49
2015-08-06T12:04:35.762+0100 [initandlisten] allocator: tcmalloc
2015-08-06T12:04:35.762+0100 [initandlisten] options: { storage: { dbPath: "/home/mongod" } }
2015-08-06T12:04:35.772+0100 [initandlisten] journal dir=/home/mongod/journal
2015-08-06T12:04:35.773+0100 [initandlisten] recover : no journal files present, no recovery needed
2015-08-06T12:04:36.865+0100 [initandlisten] waiting for connections on port 27017
However running the using systemctl fails:
sudo service mongod restart
Restarting mongod (via systemctl): Job for mongod.service failed. See 'systemctl status mongod.service' and 'journalctl -xn' for details.
[FAILED]
The /etc/mongod.conf file is as follows
logpath=/var/log/mongodb/mongod.log
logappend=true
# fork and run in background
fork=true
#port=27017
dbpath=/home/mongod
# location of pidfile
pidfilepath=/var/run/mongodb/mongod.pid
# Listen to local interface only. Comment out to listen on all interfaces.
bind_ip=127.0.0.1
The output in /var/log/mongod/mongod.log is as follows:
2015-08-06T12:08:35.402+0100 ***** SERVER RESTARTED *****
2015-08-06T12:08:35.407+0100 [initandlisten] MongoDB starting : pid=10370 port=27017 dbpath=/home/mongod 64-bit host=localhost.localdomain
2015-08-06T12:08:35.407+0100 [initandlisten] db version v2.6.10
2015-08-06T12:08:35.407+0100 [initandlisten] git version: 5901dbfb49d16eaef6f2c2c50fba534d23ac7f6c
2015-08-06T12:08:35.407+0100 [initandlisten] build info: Linux build18.nj1.10gen.cc 2.6.32-431.3.1.el6.x86_64 #1 SMP Fri Jan 3 21:39:27 UTC 2014 x86_64 BOOST_LIB_VERSION=1_49
2015-08-06T12:08:35.407+0100 [initandlisten] allocator: tcmalloc
2015-08-06T12:08:35.407+0100 [initandlisten] options: { config: "/etc/mongod.conf", net: { bindIp: "127.0.0.1" }, processManagement: { fork: true, pidFilePath: "/var/run/mongodb/mongod.pid" }, storage: { dbPath: "/home/mongod" }, systemLog: { destination: "file", logAppend: true, path: "/var/log/mongodb/mongod.log" } }
2015-08-06T12:08:35.407+0100 [initandlisten] exception in initAndListen std::exception: boost::filesystem::status: Permission denied: "/home/mongod/local.ns", terminating
2015-08-06T12:08:35.407+0100 [initandlisten] dbexit:
2015-08-06T12:08:35.407+0100 [initandlisten] shutdown: going to close listening sockets...
2015-08-06T12:08:35.407+0100 [initandlisten] shutdown: going to flush diaglog...
2015-08-06T12:08:35.407+0100 [initandlisten] shutdown: going to close sockets...
2015-08-06T12:08:35.407+0100 [initandlisten] shutdown: waiting for fs preallocator...
2015-08-06T12:08:35.407+0100 [initandlisten] shutdown: lock for final commit...
2015-08-06T12:08:35.407+0100 [initandlisten] shutdown: final commit...
2015-08-06T12:08:35.407+0100 [initandlisten] shutdown: closing all files...
2015-08-06T12:08:35.407+0100 [initandlisten] closeAllFiles() finished
2015-08-06T12:08:35.407+0100 [initandlisten] dbexit: really exiting now
I've tried deleting the lock file /home/mongod/mongod.lock and running a repair with sudo -u mongod mongod --dbpath /home/mongod --repair but the error still persists. This is causing major headaches!!
So it turns out the problem was SElinux, and the solution is the same as this question.
It was permission related and is due to the SELinux security context which is set to enforced by default on CentOS.
sudo ausearch -m avc -ts today | audit2allow
On my machine I had some mongo related audits as follows:
#============= mongod_t ==============
allow mongod_t home_root_t:file getattr;
allow mongod_t user_home_dir_t:dir search;
allow mongod_t var_lib_t:lnk_file read;
allow mongod_t default_t:file getattr;
To fix the above, you do the following:
sudo chcon -Rv --type=mongod_var_lib_t /path_to_your_mongo
Where /path_to_your_mongo is where your mongod data files are located (in my case /home/mongod but often /var/lib/mongo or /data/db.
I am getting the below error when enabling text search in mongodb. Any Suggestion.
I have mongo.config filw which contains :
##store data here
dbpath=C:\mongodb\data
##all output go here
logpath=C:\mongodb\log\mongo.log
##log read and write operations
diaglog=3
Error is :
C:\mongodb-win32-x86_64-2.4.6\bin>mongod --setParameter textSearchEnabled=true
Tue Jan 14 16:52:51.336 [initandlisten] MongoDB starting : pid=976 port=27017 dbpath=\data\db\ 64-bit host=dellvostro3
Tue Jan 14 16:52:51.337 [initandlisten] db version v2.4.6
Tue Jan 14 16:52:51.337 [initandlisten] git version: b9925db5eac369d77a3a5f5d98a145eaaacd9673
Tue Jan 14 16:52:51.337 [initandlisten] build info: windows sys.getwindowsversion(major=6, minor=1, build=7601, platform=2, service_pack='Service Pack
1') BOOST_LIB_VERSION=1_49
Tue Jan 14 16:52:51.337 [initandlisten] allocator: system
Tue Jan 14 16:52:51.337 [initandlisten] options: { setParameter: [ "textSearchEnabled=true" ] }
Tue Jan 14 16:52:51.337 [initandlisten] exception in initAndListen: 10296
*********************************************************************
ERROR: dbpath (\data\db\) does not exist.
Create this directory or give existing directory in --dbpath.
See http://dochub.mongodb.org/core/startingandstoppingmongo
*********************************************************************
, terminating
Tue Jan 14 16:52:51.338 dbexit:
Tue Jan 14 16:52:51.338 [initandlisten] shutdown: going to close listening sockets...
Tue Jan 14 16:52:51.338 [initandlisten] shutdown: going to flush diaglog...
Tue Jan 14 16:52:51.338 [initandlisten] shutdown: going to close sockets...
Tue Jan 14 16:52:51.338 [initandlisten] shutdown: waiting for fs preallocator...
Tue Jan 14 16:52:51.338 [initandlisten] shutdown: lock for final commit...
Tue Jan 14 16:52:51.338 [initandlisten] shutdown: final commit...
Tue Jan 14 16:52:51.338 [initandlisten] shutdown: closing all files...
Tue Jan 14 16:52:51.338 [initandlisten] closeAllFiles() finished
Tue Jan 14 16:52:51.338 dbexit: really exiting now
You provide wrong path to your data
ERROR: dbpath (\data\db\) does not exist.
\data\db\ is default mongo data path, you need to create this directory or get provide another path when running mongod
When you run
mongod --setParameter textSearchEnabled=true
MongoDB will take \data\db\ as the dbpath. Which is not there in your machine and hence you get the error.
I see you have a config file containing the dbpath etc, but you haven't specified the config file while starting mongod.
mongod --config mongo.config --setParameter textSearchEnabled=true
See using config file here.
Also you could move the textSearchEnabled=true to your config file.
##store data here
dbpath=C:\mongodb\data
##all output go here
logpath=C:\mongodb\log\mongo.log
##log read and write operations
diaglog=3
setParameter=textSearchEnabled=true
Now you can start mongod by just using
mongod --config mongo.config
So I have a 2TB hard drive with 2 collections in 1 mongoDB.
Collection 1 is about 500gb and completely essential
Collection 2 is about 1500gb and needs to be deleted.
I was running a process that accidentally filled up the entire HDD, leaving 0% space. There is literally nothing on the machine apart from a Linux Distro and this mongoDB.
I would like to delete Collection 2.
In order to do this, I would normally just go into mongo and type db.collection.drop()
However, mongoDB is not currently running. In order to start it, I'm using:
[bobby#myPC bin]# ./mongod --dbpath /home/mongo &
However, since there is no space on the HDD, it returns:
[bobby#myPC bin]# Thu Feb 14 17:47:10 [initandlisten] MongoDB starting : pid=1264 port=27017 dbpath=/home/mongo 64-bit host=localhost.localdomain
Thu Feb 14 17:47:10 [initandlisten] db version v2.2.3, pdfile version 4.5
Thu Feb 14 17:47:10 [initandlisten] git version: f570771a5d8a3846eb7586eaffcf4c2f4a96bf08
Thu Feb 14 17:47:10 [initandlisten] build info: Linux ip-10-2-29-40 2.6.21.7-2.ec2.v1.2.fc8xen #1 SMP Fri Nov 20 17:48:28 EST 2009 x86_64 BOOST_LIB_VERSION=1_49
Thu Feb 14 17:47:10 [initandlisten] options: { dbpath: "/home/mongo" }
Thu Feb 14 17:47:10 [initandlisten] journal dir=/home/mongo/journal
Thu Feb 14 17:47:10 [initandlisten] recover : no journal files present, no recovery needed
Thu Feb 14 17:47:10 [initandlisten]
Thu Feb 14 17:47:10 [initandlisten] ERROR: Insufficient free space for journal files
Thu Feb 14 17:47:10 [initandlisten] Please make at least 3379MB available in /home/mongo/journal or use --smallfiles
Thu Feb 14 17:47:10 [initandlisten]
Thu Feb 14 17:47:10 [initandlisten] exception in initAndListen: 15926 Insufficient free space for journals, terminating
Thu Feb 14 17:47:10 dbexit:
Thu Feb 14 17:47:10 [initandlisten] shutdown: going to close listening sockets...
Thu Feb 14 17:47:10 [initandlisten] shutdown: going to flush diaglog...
Thu Feb 14 17:47:10 [initandlisten] shutdown: going to close sockets...
Thu Feb 14 17:47:10 [initandlisten] shutdown: waiting for fs preallocator...
Thu Feb 14 17:47:10 [initandlisten] shutdown: lock for final commit...
Thu Feb 14 17:47:10 [initandlisten] shutdown: final commit...
Thu Feb 14 17:47:10 [initandlisten] shutdown: closing all files...
Thu Feb 14 17:47:10 [initandlisten] closeAllFiles() finished
Thu Feb 14 17:47:10 [initandlisten] journalCleanup...
Thu Feb 14 17:47:10 [initandlisten] removeJournalFiles
Thu Feb 14 17:47:10 [initandlisten] shutdown: removing fs lock...
Thu Feb 14 17:47:10 dbexit: really exiting now
I can't remove this collection cause there's no space. And there's no space because I can't remove this collection.
How can I fix this? Is there some way of identifying which mongoDB files in /home/mongo I can take out?
Based on the error message, its complaining about not having enough space for the journal. Its not ideal in a production environment to run with out journaling, but you could try starting with out it while you fix the issue.
./mongod --nojournal --dbpath /home/mongo &
edit: to answer your question though, mongo creates and allocates files per database, not collection. So you wouldn't be able to target a specific file to remove a collection.
This question has been asked before but was never answered. Furthermore, my error logs are slightly different:
When I run 'mongo' I get this error:
MongoDB shell version: 2.2.0
connecting to: test
Fri Nov 16 10:52:05 Error: couldn't connect to server 127.0.0.1:27017 src/mongo/shell/mongo.js:91
exception: connect failed
Here are my error logs:
Fri Nov 16 10:47:44 [initandlisten] MongoDB starting : pid=46508 port=27017 dbpath=/usr/local/var/mongodb 64-bit host=Brians-MacBook-Pro-2.local
Fri Nov 16 10:47:44 [initandlisten]
Fri Nov 16 10:47:44 [initandlisten] ** WARNING: soft rlimits too low. Number of files is 256, should be at least 1000
Fri Nov 16 10:47:44 [initandlisten] db version v2.2.0, pdfile version 4.5
Fri Nov 16 10:47:44 [initandlisten] git version: f5e83eae9cfbec7fb7a071321928f00d1b0c5207
Fri Nov 16 10:47:44 [initandlisten] build info: Darwin bs-osx-106-x86-64-1.local 10.8.0 Darwin Kernel Version 10.8.0: Tue Jun 7 16:33:36 PDT 2011; root:xnu-1504.15.3~1/RELEASE_I386 i386 BOOST_LIB_VERSION=1_49
Fri Nov 16 10:47:44 [initandlisten] options: { bind_ip: "127.0.0.1", config: "/usr/local/etc/mongod.conf", dbpath: "/usr/local/var/mongodb", logappend: "true", logpath: "/usr/local/var/log/mongodb/mongo.log" }
Fri Nov 16 10:47:44 [initandlisten] journal dir=/usr/local/var/mongodb/journal
Fri Nov 16 10:47:44 [initandlisten] recover begin
Fri Nov 16 10:47:44 [initandlisten] couldn't open /usr/local/var/mongodb/journal/lsn errno:13 Permission denied
Fri Nov 16 10:47:44 [initandlisten] Assertion failure f.is_open() src/mongo/db/dur_journal.cpp 552
0x10037637b 0x1000b0196 0x100105ec2 0x10049de46 0x10049e445 0x10049e628 0x1002401ac 0x1000057ec 0x10000608d 0x10000d534 0x10000f309 0x100000ea4 0x3
0 mongod 0x000000010037637b _ZN5mongo15printStackTraceERSo + 43
1 mongod 0x00000001000b0196 _ZN5mongo12verifyFailedEPKcS1_j + 310
2 mongod 0x0000000100105ec2 _ZN5mongo3dur14journalReadLSNEv + 898
3 mongod 0x000000010049de46 _ZN5mongo3dur11RecoveryJob2goERSt6vectorIN5boost11filesystem210basic_pathISsNS4_11path_traitsEEESaIS7_EE + 86
4 mongod 0x000000010049e445 _ZN5mongo3dur8_recoverEv + 405
5 mongod 0x000000010049e628 _ZN5mongo3dur7recoverEv + 40
6 mongod 0x00000001002401ac _ZN5mongo3dur7startupEv + 124
7 mongod 0x00000001000057ec _ZN5mongo14_initAndListenEi + 1052
8 mongod 0x000000010000608d _ZN5mongo13initAndListenEi + 29
9 mongod 0x000000010000d534 _ZL11mongoDbMainiPPc + 29348
10 mongod 0x000000010000f309 main + 9
11 mongod 0x0000000100000ea4 start + 52
12 ??? 0x0000000000000003 0x0 + 3
Fri Nov 16 10:47:44 [initandlisten] dbexception during recovery: 13611 can't read lsn file in journal directory : assertion src/mongo/db/dur_journal.cpp:552
Fri Nov 16 10:47:44 [initandlisten] exception in initAndListen: 13611 can't read lsn file in journal directory : assertion src/mongo/db/dur_journal.cpp:552, terminating
Fri Nov 16 10:47:44 dbexit:
Fri Nov 16 10:47:44 [initandlisten] shutdown: going to close listening sockets...
Fri Nov 16 10:47:44 [initandlisten] shutdown: going to flush diaglog...
Fri Nov 16 10:47:44 [initandlisten] shutdown: going to close sockets...
Fri Nov 16 10:47:44 [initandlisten] shutdown: waiting for fs preallocator...
Fri Nov 16 10:47:44 [initandlisten] shutdown: lock for final commit...
Fri Nov 16 10:47:44 [initandlisten] shutdown: final commit...
Fri Nov 16 10:47:44 [initandlisten] shutdown: closing all files...
Fri Nov 16 10:47:44 [initandlisten] closeAllFiles() finished
Fri Nov 16 10:47:44 [initandlisten] shutdown: removing fs lock...
Fri Nov 16 10:47:44 dbexit: really exiting now
There is no output when I run:
netstat -f inet -a -n | grep 27101
Thanks for your help!
As #Nilam said one of your problems is the journal, however, you actually have two problems here, the one I am talking about being more of a warning:
Fri Nov 16 10:47:44 [initandlisten] ** WARNING: soft rlimits too low. Number of files is 256, should be at least 1000
This specifically relates to: https://jira.mongodb.org/browse/SERVER-4407
You should look into increasing your openfile count in Linux otherwise you could get this: http://www.mongodb.org/display/DOCS/Too+Many+Open+Files of course this page also tells you how to solve the problem.
NB: as a side note to the journal point, it looks like your MongoDB node could be trying to recover from something. I would check why.
Similar to #Hartator's answer, this solved the issue for me:
chmod -R 777 /data/db
You have
journal dir=/usr/local/var/mongodb/journal
The error shows is
[initandlisten] couldn't open /usr/local/var/mongodb/journal/lsn errno:13 Permission denied
So check if journal directory exist and proper write permissions are given to this folder.
Try to run :
chmod 777 -R /usr/local/var/mongodb
I've used mongodb a few days, it has been working fine until today.
I got a error when querying a database.
Wed Nov 2 17:56:44 [initandlisten] connection accepted from 127.0.0.1:49234 #1
Wed Nov 2 17:56:45 [conn1] command admin.$cmd command: { listDatabases: 1 } ntoreturn:1 reslen:338 813ms
Wed Nov 2 17:56:46 [conn1] admin.system.namespaces Assertion failure isOk() db/pdfile.h 300
0x10007d703 0x100087eec 0x1004165e4 0x10040774b 0x100410823 0x100437df9 0x1004b3006 0x100426693 0x100426b53 0x100429f45 0x10042d8f0 0x10043109e 0x10043353b 0x10043662a 0x1004a13ca 0x1003d5e0f 0x1003e298a 0x100603394 0x100148d8c 0x10060a554
0 mongod 0x000000010007d703 _ZN5mongo12sayDbContextEPKc + 195
1 mongod 0x0000000100087eec _ZN5mongo8assertedEPKcS1_j + 300
2 mongod 0x00000001004165e4 _ZN5mongo11DataFileMgr9getExtentERKNS_7DiskLocE + 148
3 mongod 0x000000010040774b _ZN5mongo11DataFileMgr7findAllEPKcRKNS_7DiskLocE + 75
4 mongod 0x0000000100410823 _ZN5mongo13findTableScanEPKcRKNS_7BSONObjERKNS_7DiskLocE + 131
5 mongod 0x0000000100437df9 _ZNK5mongo9QueryPlan9newCursorERKNS_7DiskLocEi + 1081
6 mongod 0x00000001004b3006 _ZN5mongo11UserQueryOp5_initEv + 214
7 mongod 0x0000000100426693 _ZN5mongo7QueryOp4initEv + 323
8 mongod 0x0000000100426b53 _ZN5mongo12QueryPlanSet6Runner6initOpERNS_7QueryOpE + 35
9 mongod 0x0000000100429f45 _ZN5mongo12QueryPlanSet6Runner4initEv + 245
10 mongod 0x000000010042d8f0 _ZN5mongo12QueryPlanSet6Runner22runUntilFirstCompletesEv + 32
11 mongod 0x000000010043109e _ZN5mongo12QueryPlanSet5runOpERNS_7QueryOpE + 334
12 mongod 0x000000010043353b _ZN5mongo16MultiPlanScanner9runOpOnceERNS_7QueryOpE + 1115
13 mongod 0x000000010043662a _ZN5mongo16MultiPlanScanner5runOpERNS_7QueryOpE + 26
14 mongod 0x00000001004a13ca _ZN5mongo8runQueryERNS_7MessageERNS_12QueryMessageERNS_5CurOpES1_ + 3194
15 mongod 0x00000001003d5e0f _ZN5mongo13receivedQueryERNS_6ClientERNS_10DbResponseERNS_7MessageE + 623
16 mongod 0x00000001003e298a _ZN5mongo16assembleResponseERNS_7MessageERNS_10DbResponseERKNS_11HostAndPortE + 1402
17 mongod 0x0000000100603394 _ZN5mongo16MyMessageHandler7processERNS_7MessageEPNS_21AbstractMessagingPortEPNS_9LastErrorE + 148
18 mongod 0x0000000100148d8c _ZN5mongo3pms9threadRunEPNS_13MessagingPortE + 700
19 mongod 0x000000010060a554 thread_proxy + 132
Wed Nov 2 17:56:46 [conn1] assertion 0 assertion db/pdfile.h:300 ns:admin.system.namespaces query:{}
Wed Nov 2 17:56:51 [conn1] end connection 127.0.0.1:49234
I restart the mongod process but no help.
I tried to repair the db but also got the error.
bin: $ ./mongod --repair --dbpath ../data/db
Wed Nov 2 18:10:51 [initandlisten] MongoDB starting : pid=678 port=27017 dbpath=../data/db 64-bit host=KTs-MacBook-Pro.local
Wed Nov 2 18:10:51 [initandlisten] db version v2.0.0, pdfile version 4.5
Wed Nov 2 18:10:51 [initandlisten] git version: 695c67dff0ffc361b8568a13366f027caa406222
Wed Nov 2 18:10:51 [initandlisten] build info: Darwin erh2.10gen.cc 9.6.0 Darwin Kernel Version 9.6.0: Mon Nov 24 17:37:00 PST 2008; root:xnu-1228.9.59~1/RELEASE_I386 i386 BOOST_LIB_VERSION=1_40
Wed Nov 2 18:10:51 [initandlisten] options: { dbpath: "../data/db", repair: true }
Wed Nov 2 18:10:51 [initandlisten] journal dir=../data/db/journal
Wed Nov 2 18:10:51 [initandlisten] recover : no journal files present, no recovery needed
Wed Nov 2 18:10:51 [initandlisten] ****
Wed Nov 2 18:10:51 [initandlisten] ****
Wed Nov 2 18:10:51 [initandlisten] need to upgrade database admin with pdfile version 4.5, new version: 4.5
Wed Nov 2 18:10:51 [initandlisten] starting upgrade
Wed Nov 2 18:10:51 [initandlisten] admin repairDatabase admin
Wed Nov 2 18:10:51 [initandlisten] removeJournalFiles
Wed Nov 2 18:10:51 [initandlisten] admin.system.namespaces Assertion failure isOk() db/pdfile.h 300
0x10007d703 0x100087eec 0x1004165e4 0x10040774b 0x100410823 0x100437df9 0x1004b3006 0x100426693 0x100426b53 0x100429f45 0x10042d8f0 0x10043109e 0x10043353b 0x10043662a 0x1004a13ca 0x1003d5e0f 0x1003e298a 0x1003e480f 0x1000ed8ec 0x1000c8b73
0 mongod 0x000000010007d703 _ZN5mongo12sayDbContextEPKc + 195
1 mongod 0x0000000100087eec _ZN5mongo8assertedEPKcS1_j + 300
2 mongod 0x00000001004165e4 _ZN5mongo11DataFileMgr9getExtentERKNS_7DiskLocE + 148
3 mongod 0x000000010040774b _ZN5mongo11DataFileMgr7findAllEPKcRKNS_7DiskLocE + 75
4 mongod 0x0000000100410823 _ZN5mongo13findTableScanEPKcRKNS_7BSONObjERKNS_7DiskLocE + 131
5 mongod 0x0000000100437df9 _ZNK5mongo9QueryPlan9newCursorERKNS_7DiskLocEi + 1081
6 mongod 0x00000001004b3006 _ZN5mongo11UserQueryOp5_initEv + 214
7 mongod 0x0000000100426693 _ZN5mongo7QueryOp4initEv + 323
8 mongod 0x0000000100426b53 _ZN5mongo12QueryPlanSet6Runner6initOpERNS_7QueryOpE + 35
9 mongod 0x0000000100429f45 _ZN5mongo12QueryPlanSet6Runner4initEv + 245
10 mongod 0x000000010042d8f0 _ZN5mongo12QueryPlanSet6Runner22runUntilFirstCompletesEv + 32
11 mongod 0x000000010043109e _ZN5mongo12QueryPlanSet5runOpERNS_7QueryOpE + 334
12 mongod 0x000000010043353b _ZN5mongo16MultiPlanScanner9runOpOnceERNS_7QueryOpE + 1115
13 mongod 0x000000010043662a _ZN5mongo16MultiPlanScanner5runOpERNS_7QueryOpE + 26
14 mongod 0x00000001004a13ca _ZN5mongo8runQueryERNS_7MessageERNS_12QueryMessageERNS_5CurOpES1_ + 3194
15 mongod 0x00000001003d5e0f _ZN5mongo13receivedQueryERNS_6ClientERNS_10DbResponseERNS_7MessageE + 623
16 mongod 0x00000001003e298a _ZN5mongo16assembleResponseERNS_7MessageERNS_10DbResponseERKNS_11HostAndPortE + 1402
17 mongod 0x00000001003e480f _ZN5mongo14DBDirectClient4callERNS_7MessageES2_bPSs + 111
18 mongod 0x00000001000ed8ec _ZN5mongo14DBClientCursor4initEv + 76
19 mongod 0x00000001000c8b73 _ZN5mongo12DBClientBase5queryERKSsNS_5QueryEiiPKNS_7BSONObjEii + 739
Wed Nov 2 18:10:51 [initandlisten] assertion 0 assertion db/pdfile.h:300 ns:admin.system.namespaces query:{}
Wed Nov 2 18:10:51 [initandlisten] admin clone failed for admin with error: query failed admin.system.namespaces
Wed Nov 2 18:10:51 [initandlisten] removeJournalFiles
Wed Nov 2 18:10:51 [initandlisten] admin Assertion failure doDBUpgrade( dbName , errmsg , h ) db/db.cpp 298
0x10007d703 0x100087eec 0x1005f4f2f 0x1005f5c3c 0x1005f6274 0x1005f7d6f 0x1000013c4
0 mongod 0x000000010007d703 _ZN5mongo12sayDbContextEPKc + 195
1 mongod 0x0000000100087eec _ZN5mongo8assertedEPKcS1_j + 300
2 mongod 0x00000001005f4f2f _ZN5mongo30repairDatabasesAndCheckVersionEv + 2063
3 mongod 0x00000001005f5c3c _ZN5mongo14_initAndListenEi + 1036
4 mongod 0x00000001005f6274 _ZN5mongo13initAndListenEi + 36
5 mongod 0x00000001005f7d6f main + 3647
6 mongod 0x00000001000013c4 start + 52
Wed Nov 2 18:10:51 [initandlisten] exception in initAndListen: 0 assertion db/db.cpp:298, terminating
Wed Nov 2 18:10:51 dbexit:
Wed Nov 2 18:10:51 [initandlisten] shutdown: going to close listening sockets...
Wed Nov 2 18:10:51 [initandlisten] shutdown: going to flush diaglog...
Wed Nov 2 18:10:51 [initandlisten] shutdown: going to close sockets...
Wed Nov 2 18:10:51 [initandlisten] shutdown: waiting for fs preallocator...
Wed Nov 2 18:10:51 [initandlisten] shutdown: lock for final commit...
Wed Nov 2 18:10:51 [initandlisten] shutdown: final commit...
Wed Nov 2 18:10:51 [initandlisten] shutdown: closing all files...
Wed Nov 2 18:10:51 [initandlisten] closeAllFiles() finished
Wed Nov 2 18:10:51 [initandlisten] journalCleanup...
Wed Nov 2 18:10:51 [initandlisten] removeJournalFiles
Wed Nov 2 18:10:51 [initandlisten] shutdown: removing fs lock...
Wed Nov 2 18:10:51 dbexit: really exiting now
The mongodb version is v2.0.0, pdfile version is 4.5, git version is 695c67dff0ffc361b8568a13366f027caa406222.
There are 5 databases in the mongodb, but only one database to be wrong.
I also tried to recover data by mongodump, but got empty bson files.
Any ideas? Thanks
I was able to resolve this issue by going to my data directory (which in the example above would be
/data/db
and deleting all the jicd.x and local.x files
If the lock file is not 0 bytes in size you may want to delete that one as well.
The mongodb JIRA ticket link is https://jira.mongodb.org/browse/SERVER-4198
As the comments in the ticket, fix the issue by removing the master=true setting then running a repair.
I have not tried this method, just as a reference here.