I've used mongodb a few days, it has been working fine until today.
I got a error when querying a database.
Wed Nov 2 17:56:44 [initandlisten] connection accepted from 127.0.0.1:49234 #1
Wed Nov 2 17:56:45 [conn1] command admin.$cmd command: { listDatabases: 1 } ntoreturn:1 reslen:338 813ms
Wed Nov 2 17:56:46 [conn1] admin.system.namespaces Assertion failure isOk() db/pdfile.h 300
0x10007d703 0x100087eec 0x1004165e4 0x10040774b 0x100410823 0x100437df9 0x1004b3006 0x100426693 0x100426b53 0x100429f45 0x10042d8f0 0x10043109e 0x10043353b 0x10043662a 0x1004a13ca 0x1003d5e0f 0x1003e298a 0x100603394 0x100148d8c 0x10060a554
0 mongod 0x000000010007d703 _ZN5mongo12sayDbContextEPKc + 195
1 mongod 0x0000000100087eec _ZN5mongo8assertedEPKcS1_j + 300
2 mongod 0x00000001004165e4 _ZN5mongo11DataFileMgr9getExtentERKNS_7DiskLocE + 148
3 mongod 0x000000010040774b _ZN5mongo11DataFileMgr7findAllEPKcRKNS_7DiskLocE + 75
4 mongod 0x0000000100410823 _ZN5mongo13findTableScanEPKcRKNS_7BSONObjERKNS_7DiskLocE + 131
5 mongod 0x0000000100437df9 _ZNK5mongo9QueryPlan9newCursorERKNS_7DiskLocEi + 1081
6 mongod 0x00000001004b3006 _ZN5mongo11UserQueryOp5_initEv + 214
7 mongod 0x0000000100426693 _ZN5mongo7QueryOp4initEv + 323
8 mongod 0x0000000100426b53 _ZN5mongo12QueryPlanSet6Runner6initOpERNS_7QueryOpE + 35
9 mongod 0x0000000100429f45 _ZN5mongo12QueryPlanSet6Runner4initEv + 245
10 mongod 0x000000010042d8f0 _ZN5mongo12QueryPlanSet6Runner22runUntilFirstCompletesEv + 32
11 mongod 0x000000010043109e _ZN5mongo12QueryPlanSet5runOpERNS_7QueryOpE + 334
12 mongod 0x000000010043353b _ZN5mongo16MultiPlanScanner9runOpOnceERNS_7QueryOpE + 1115
13 mongod 0x000000010043662a _ZN5mongo16MultiPlanScanner5runOpERNS_7QueryOpE + 26
14 mongod 0x00000001004a13ca _ZN5mongo8runQueryERNS_7MessageERNS_12QueryMessageERNS_5CurOpES1_ + 3194
15 mongod 0x00000001003d5e0f _ZN5mongo13receivedQueryERNS_6ClientERNS_10DbResponseERNS_7MessageE + 623
16 mongod 0x00000001003e298a _ZN5mongo16assembleResponseERNS_7MessageERNS_10DbResponseERKNS_11HostAndPortE + 1402
17 mongod 0x0000000100603394 _ZN5mongo16MyMessageHandler7processERNS_7MessageEPNS_21AbstractMessagingPortEPNS_9LastErrorE + 148
18 mongod 0x0000000100148d8c _ZN5mongo3pms9threadRunEPNS_13MessagingPortE + 700
19 mongod 0x000000010060a554 thread_proxy + 132
Wed Nov 2 17:56:46 [conn1] assertion 0 assertion db/pdfile.h:300 ns:admin.system.namespaces query:{}
Wed Nov 2 17:56:51 [conn1] end connection 127.0.0.1:49234
I restart the mongod process but no help.
I tried to repair the db but also got the error.
bin: $ ./mongod --repair --dbpath ../data/db
Wed Nov 2 18:10:51 [initandlisten] MongoDB starting : pid=678 port=27017 dbpath=../data/db 64-bit host=KTs-MacBook-Pro.local
Wed Nov 2 18:10:51 [initandlisten] db version v2.0.0, pdfile version 4.5
Wed Nov 2 18:10:51 [initandlisten] git version: 695c67dff0ffc361b8568a13366f027caa406222
Wed Nov 2 18:10:51 [initandlisten] build info: Darwin erh2.10gen.cc 9.6.0 Darwin Kernel Version 9.6.0: Mon Nov 24 17:37:00 PST 2008; root:xnu-1228.9.59~1/RELEASE_I386 i386 BOOST_LIB_VERSION=1_40
Wed Nov 2 18:10:51 [initandlisten] options: { dbpath: "../data/db", repair: true }
Wed Nov 2 18:10:51 [initandlisten] journal dir=../data/db/journal
Wed Nov 2 18:10:51 [initandlisten] recover : no journal files present, no recovery needed
Wed Nov 2 18:10:51 [initandlisten] ****
Wed Nov 2 18:10:51 [initandlisten] ****
Wed Nov 2 18:10:51 [initandlisten] need to upgrade database admin with pdfile version 4.5, new version: 4.5
Wed Nov 2 18:10:51 [initandlisten] starting upgrade
Wed Nov 2 18:10:51 [initandlisten] admin repairDatabase admin
Wed Nov 2 18:10:51 [initandlisten] removeJournalFiles
Wed Nov 2 18:10:51 [initandlisten] admin.system.namespaces Assertion failure isOk() db/pdfile.h 300
0x10007d703 0x100087eec 0x1004165e4 0x10040774b 0x100410823 0x100437df9 0x1004b3006 0x100426693 0x100426b53 0x100429f45 0x10042d8f0 0x10043109e 0x10043353b 0x10043662a 0x1004a13ca 0x1003d5e0f 0x1003e298a 0x1003e480f 0x1000ed8ec 0x1000c8b73
0 mongod 0x000000010007d703 _ZN5mongo12sayDbContextEPKc + 195
1 mongod 0x0000000100087eec _ZN5mongo8assertedEPKcS1_j + 300
2 mongod 0x00000001004165e4 _ZN5mongo11DataFileMgr9getExtentERKNS_7DiskLocE + 148
3 mongod 0x000000010040774b _ZN5mongo11DataFileMgr7findAllEPKcRKNS_7DiskLocE + 75
4 mongod 0x0000000100410823 _ZN5mongo13findTableScanEPKcRKNS_7BSONObjERKNS_7DiskLocE + 131
5 mongod 0x0000000100437df9 _ZNK5mongo9QueryPlan9newCursorERKNS_7DiskLocEi + 1081
6 mongod 0x00000001004b3006 _ZN5mongo11UserQueryOp5_initEv + 214
7 mongod 0x0000000100426693 _ZN5mongo7QueryOp4initEv + 323
8 mongod 0x0000000100426b53 _ZN5mongo12QueryPlanSet6Runner6initOpERNS_7QueryOpE + 35
9 mongod 0x0000000100429f45 _ZN5mongo12QueryPlanSet6Runner4initEv + 245
10 mongod 0x000000010042d8f0 _ZN5mongo12QueryPlanSet6Runner22runUntilFirstCompletesEv + 32
11 mongod 0x000000010043109e _ZN5mongo12QueryPlanSet5runOpERNS_7QueryOpE + 334
12 mongod 0x000000010043353b _ZN5mongo16MultiPlanScanner9runOpOnceERNS_7QueryOpE + 1115
13 mongod 0x000000010043662a _ZN5mongo16MultiPlanScanner5runOpERNS_7QueryOpE + 26
14 mongod 0x00000001004a13ca _ZN5mongo8runQueryERNS_7MessageERNS_12QueryMessageERNS_5CurOpES1_ + 3194
15 mongod 0x00000001003d5e0f _ZN5mongo13receivedQueryERNS_6ClientERNS_10DbResponseERNS_7MessageE + 623
16 mongod 0x00000001003e298a _ZN5mongo16assembleResponseERNS_7MessageERNS_10DbResponseERKNS_11HostAndPortE + 1402
17 mongod 0x00000001003e480f _ZN5mongo14DBDirectClient4callERNS_7MessageES2_bPSs + 111
18 mongod 0x00000001000ed8ec _ZN5mongo14DBClientCursor4initEv + 76
19 mongod 0x00000001000c8b73 _ZN5mongo12DBClientBase5queryERKSsNS_5QueryEiiPKNS_7BSONObjEii + 739
Wed Nov 2 18:10:51 [initandlisten] assertion 0 assertion db/pdfile.h:300 ns:admin.system.namespaces query:{}
Wed Nov 2 18:10:51 [initandlisten] admin clone failed for admin with error: query failed admin.system.namespaces
Wed Nov 2 18:10:51 [initandlisten] removeJournalFiles
Wed Nov 2 18:10:51 [initandlisten] admin Assertion failure doDBUpgrade( dbName , errmsg , h ) db/db.cpp 298
0x10007d703 0x100087eec 0x1005f4f2f 0x1005f5c3c 0x1005f6274 0x1005f7d6f 0x1000013c4
0 mongod 0x000000010007d703 _ZN5mongo12sayDbContextEPKc + 195
1 mongod 0x0000000100087eec _ZN5mongo8assertedEPKcS1_j + 300
2 mongod 0x00000001005f4f2f _ZN5mongo30repairDatabasesAndCheckVersionEv + 2063
3 mongod 0x00000001005f5c3c _ZN5mongo14_initAndListenEi + 1036
4 mongod 0x00000001005f6274 _ZN5mongo13initAndListenEi + 36
5 mongod 0x00000001005f7d6f main + 3647
6 mongod 0x00000001000013c4 start + 52
Wed Nov 2 18:10:51 [initandlisten] exception in initAndListen: 0 assertion db/db.cpp:298, terminating
Wed Nov 2 18:10:51 dbexit:
Wed Nov 2 18:10:51 [initandlisten] shutdown: going to close listening sockets...
Wed Nov 2 18:10:51 [initandlisten] shutdown: going to flush diaglog...
Wed Nov 2 18:10:51 [initandlisten] shutdown: going to close sockets...
Wed Nov 2 18:10:51 [initandlisten] shutdown: waiting for fs preallocator...
Wed Nov 2 18:10:51 [initandlisten] shutdown: lock for final commit...
Wed Nov 2 18:10:51 [initandlisten] shutdown: final commit...
Wed Nov 2 18:10:51 [initandlisten] shutdown: closing all files...
Wed Nov 2 18:10:51 [initandlisten] closeAllFiles() finished
Wed Nov 2 18:10:51 [initandlisten] journalCleanup...
Wed Nov 2 18:10:51 [initandlisten] removeJournalFiles
Wed Nov 2 18:10:51 [initandlisten] shutdown: removing fs lock...
Wed Nov 2 18:10:51 dbexit: really exiting now
The mongodb version is v2.0.0, pdfile version is 4.5, git version is 695c67dff0ffc361b8568a13366f027caa406222.
There are 5 databases in the mongodb, but only one database to be wrong.
I also tried to recover data by mongodump, but got empty bson files.
Any ideas? Thanks
I was able to resolve this issue by going to my data directory (which in the example above would be
/data/db
and deleting all the jicd.x and local.x files
If the lock file is not 0 bytes in size you may want to delete that one as well.
The mongodb JIRA ticket link is https://jira.mongodb.org/browse/SERVER-4198
As the comments in the ticket, fix the issue by removing the master=true setting then running a repair.
I have not tried this method, just as a reference here.
Related
I ran db.fsyncLock() command and then the server was restarted without me running db.fsyncUnlock().
After the restart, mongo does not start with the error
2020-04-17T10:13:59.693+0300 Error: couldn't connect to server
127.0.0.1:27017 (127.0.0.1), connection attempt failed at src/mongo/shell/mongo.js:146
What should I do? Can I just delete the lock file?
Thanks a lot!
Below are the logs and the ls of the directory
1583 2020-04-17T08:08:02.481+0300 [initandlisten] shutdown: going to close sockets...
1584 2020-04-17T08:08:02.481+0300 [initandlisten] shutdown: waiting for fs preallocator...
1585 2020-04-17T08:08:02.481+0300 [initandlisten] shutdown: lock for final commit...
1586 2020-04-17T08:08:02.481+0300 [initandlisten] shutdown: final commit...
1587 2020-04-17T08:08:02.481+0300 [initandlisten] shutdown: closing all files...
1588 2020-04-17T08:08:02.481+0300 [initandlisten] closeAllFiles() finished
1589 2020-04-17T08:08:02.481+0300 [initandlisten] journalCleanup...
1590 2020-04-17T08:08:02.481+0300 [initandlisten] removeJournalFiles
1591 2020-04-17T08:08:02.486+0300 [initandlisten] shutdown: removing fs lock...
1592 2020-04-17T08:08:02.486+0300 [initandlisten] dbexit: really exiting now
1593 ~$ ls -l /var/lib/mongodb
1594 total 294932
1595 -rw------- 1 mongodb mongodb 67108864 Apr 16 21:57 ctf.0
1596 -rw------- 1 mongodb mongodb 134217728 Apr 16 21:57 ctf.1
1597 -rw------- 1 mongodb mongodb 16777216 Apr 16 21:56 ctf.ns
1598 drwxr-xr-x 2 mongodb mongodb 4096 Apr 16 22:04 journal
1599 -rw------- 1 mongodb mongodb 67108864 Feb 26 04:47 local.0
1600 -rw------- 1 mongodb mongodb 16777216 Feb 26 04:47 local.ns
1601 -rwxr-xr-x 1 mongodb mongodb 0 Apr 17 08:31 mongod.lock
I have been running MongoDB on a RPI for several weeks, and suddenly it won't start. I know there's a lot of similar questions where correcting the permissions seemed to work. However, this has not worked for me.
I suspect a power off of the RPI has caused this problem, but I have no idea how to solve it.
I get the following terminal output:
$ sudo service mongodb start
[FAIL] Starting database: mongodb failed!
More details are shown with the 'mongod' command:
$ mongod
db level locking enabled: 1
mongod --help for help and startup options
Fri Aug 28 14:04:06
Fri Aug 28 14:04:06 warning: 32-bit servers don't have journaling enabled by default. Please use --journal if you want durability.
Fri Aug 28 14:04:06
Fri Aug 28 14:04:06 [initandlisten] MongoDB starting : pid=4036 port=27017 dbpath=/data/db/ 32-bit host=raspberrypi
Fri Aug 28 14:04:06 [initandlisten]
Fri Aug 28 14:04:06 [initandlisten] ** NOTE: This is a development version (2.1.1) of MongoDB.
Fri Aug 28 14:04:06 [initandlisten] ** Not recommended for production.
Fri Aug 28 14:04:06 [initandlisten]
Fri Aug 28 14:04:06 [initandlisten] ** NOTE: when using MongoDB 32 bit, you are limited to about 2 gigabytes of data
Fri Aug 28 14:04:06 [initandlisten] ** see http://blog.mongodb.org/post/137788967/32-bit-limitations
Fri Aug 28 14:04:06 [initandlisten] ** with --journal, the limit is lower
Fri Aug 28 14:04:06 [initandlisten]
Fri Aug 28 14:04:06 [initandlisten] db version v2.1.1, pdfile version 4.5
Fri Aug 28 14:04:06 [initandlisten] git version: f457ff42ec37f2562d1a5ff06b4d96a861414c94
Fri Aug 28 14:04:06 [initandlisten] build info: Linux piplusraspbian 3.12.22+ #691 PREEMPT Wed Jun 18 18:29:58 BST 2014 armv6l BOOST_LIB_VERSION=1_49
Fri Aug 28 14:04:06 [initandlisten] options: {}
Fri Aug 28 14:04:06 [initandlisten] exception in initAndListen: 10309 Unable to create/open lock file: /data/db/mongod.lock errno:13 Permission denied Is a mongod instance already running?, terminating
Fri Aug 28 14:04:06 dbexit:
Fri Aug 28 14:04:06 [initandlisten] shutdown: going to close listening sockets...
Fri Aug 28 14:04:06 [initandlisten] shutdown: going to flush diaglog...
Fri Aug 28 14:04:06 [initandlisten] shutdown: going to close sockets...
Fri Aug 28 14:04:06 [initandlisten] shutdown: waiting for fs preallocator...
Fri Aug 28 14:04:06 [initandlisten] shutdown: closing all files...
Fri Aug 28 14:04:06 [initandlisten] closeAllFiles() finished
Fri Aug 28 14:04:06 [initandlisten] shutdown: removing fs lock...
Fri Aug 28 14:04:06 [initandlisten] couldn't remove fs lock errno:9 Bad file descriptor
Fri Aug 28 14:04:06 dbexit: really exiting now
The /data/db does exist and seem to have the right permission:
$ ls -l /data
total 8
drwxr-xr-x 2 mongodb mongodb 4096 Aug 28 14:37 db
drwxr-xr-x 4 mongodb mongodb 4096 Aug 28 09:44 db_backup
The content of db has been copied to db_backup to check if that did any change, but it didn't. The db content permissions also seem fine:
$ ls -l /data/db_backup
total 131100
drwxr-xr-x 2 mongodb mongodb 4096 Aug 28 09:44 client-db
-rw------- 1 mongodb mongodb 16777216 Aug 28 09:44 client-db.0
-rw------- 1 mongodb mongodb 33554432 Aug 28 09:44 client-db.1
-rw------- 1 mongodb mongodb 16777216 Aug 28 09:44 client-db.ns
-rwxr-xr-x 1 mongodb mongodb 0 Aug 28 13:52 mongod.lock
drwxr-xr-x 2 mongodb mongodb 4096 Aug 28 09:44 myproject
-rw------- 1 mongodb mongodb 16777216 Aug 28 09:44 myproject.0
-rw------- 1 mongodb mongodb 33554432 Aug 28 09:44 myproject.1
-rw------- 1 mongodb mongodb 16777216 Aug 28 09:44 myproject.ns
I have also tried mongod --repair, but it does not help.
Does anyone have any ideas?
I don't care about the data, as it is all just temporary testing data.
There is a mongod.lock file inside the mongo's data directory, normally is /var/lib/mongodb. The content of this file is the pid of the mongod process.
If the mongod is stopped, the content of this file should be empty. Otherwise the mongod was stopped unexpectedly. To fix and start a new mongod process, assuming that mongo's data directory is /var/lib/mongodb. Just clear the file...
$ echo "" > /var/lib/mongodb/mongod.lock
and start a new mongod process.
$ mongod
I can't change port in second mongod in my cmd
1 I'm starting mongod in c:\mongodb\bin
2 I'm starting cmd and writing
cd c:\mongodb\bin
mongod --port 10003
But second mongod not work.
Help me.Thank you!
This is error what show me
C:\Windows\system32> cd c:\mongodb\bin
c:\mongodb\bin>mongod --port 10003
Wed Apr 16 22:15:56.619
Wed Apr 16 22:15:56.620 warning: 32-bit servers don't have journaling enabled by
default. Please use --journal if you want durability.
Wed Apr 16 22:15:56.622
Wed Apr 16 22:15:56.635 [initandlisten] MongoDB starting : pid=8240 port=10003 d
bpath=\data\db\ 32-bit host=Danil
Wed Apr 16 22:15:56.637 [initandlisten]
Wed Apr 16 22:15:56.638 [initandlisten] ** NOTE: This is a 32 bit MongoDB binary
.
Wed Apr 16 22:15:56.640 [initandlisten] ** 32 bit builds are limited to le
ss than 2GB of data (or less with --journal).
Wed Apr 16 22:15:56.641 [initandlisten] ** Note that journaling defaults t
o off for 32 bit and is currently off.
Wed Apr 16 22:15:56.643 [initandlisten] ** See http://dochub.mongodb.org/c
ore/32bit
Wed Apr 16 22:15:56.645 [initandlisten]
Wed Apr 16 22:15:56.646 [initandlisten] db version v2.4.10
Wed Apr 16 22:15:56.648 [initandlisten] git version: e3d78955d181e475345ebd60053
a4738a4c5268a
Wed Apr 16 22:15:56.650 [initandlisten] build info: windows sys.getwindowsversio
n(major=6, minor=0, build=6002, platform=2, service_pack='Service Pack 2') BOOST
_LIB_VERSION=1_49
Wed Apr 16 22:15:56.651 [initandlisten] allocator: system
Wed Apr 16 22:15:56.653 [initandlisten] options: { port: 10003 }
Wed Apr 16 22:15:56.655 [initandlisten] exception in initAndListen: 13627 Unable
to create/open lock file: \data\db\mongod.lock ??????? ?? ????? ???????? ??????
? ?????, ??? ??? ???? ???? ????? ?????? ?????????.. Is a mongod instance alread
y running?, terminating
Wed Apr 16 22:15:56.657 dbexit:
Wed Apr 16 22:15:56.658 [initandlisten] shutdown: going to close listening socke
ts...
Wed Apr 16 22:15:56.660 [initandlisten] shutdown: going to flush diaglog...
Wed Apr 16 22:15:56.662 [initandlisten] shutdown: going to close sockets...
Wed Apr 16 22:15:56.663 [initandlisten] shutdown: waiting for fs preallocator...
Wed Apr 16 22:15:56.665 [initandlisten] shutdown: closing all files...
Wed Apr 16 22:15:56.666 [initandlisten] closeAllFiles() finished
Wed Apr 16 22:15:56.668 dbexit: really exiting now
c:\mongodb\bin>
try to start mongod with admin option as it is not able to open the file mongod.lock
You can't start multiple MongoDB processes on the same database.
What you need to do is:
mongod --dbpath c:\database1 --port 27017
mongod --dbpath c:\database2 --port 27018
Remember that you must manually create the database folders.
Usually if you read carefully the error log you will find the problem:
"13627 Unable to create/open lock file: \data\db\mongod.lock.
Is a mongod instance already running?"
This question has been asked before but was never answered. Furthermore, my error logs are slightly different:
When I run 'mongo' I get this error:
MongoDB shell version: 2.2.0
connecting to: test
Fri Nov 16 10:52:05 Error: couldn't connect to server 127.0.0.1:27017 src/mongo/shell/mongo.js:91
exception: connect failed
Here are my error logs:
Fri Nov 16 10:47:44 [initandlisten] MongoDB starting : pid=46508 port=27017 dbpath=/usr/local/var/mongodb 64-bit host=Brians-MacBook-Pro-2.local
Fri Nov 16 10:47:44 [initandlisten]
Fri Nov 16 10:47:44 [initandlisten] ** WARNING: soft rlimits too low. Number of files is 256, should be at least 1000
Fri Nov 16 10:47:44 [initandlisten] db version v2.2.0, pdfile version 4.5
Fri Nov 16 10:47:44 [initandlisten] git version: f5e83eae9cfbec7fb7a071321928f00d1b0c5207
Fri Nov 16 10:47:44 [initandlisten] build info: Darwin bs-osx-106-x86-64-1.local 10.8.0 Darwin Kernel Version 10.8.0: Tue Jun 7 16:33:36 PDT 2011; root:xnu-1504.15.3~1/RELEASE_I386 i386 BOOST_LIB_VERSION=1_49
Fri Nov 16 10:47:44 [initandlisten] options: { bind_ip: "127.0.0.1", config: "/usr/local/etc/mongod.conf", dbpath: "/usr/local/var/mongodb", logappend: "true", logpath: "/usr/local/var/log/mongodb/mongo.log" }
Fri Nov 16 10:47:44 [initandlisten] journal dir=/usr/local/var/mongodb/journal
Fri Nov 16 10:47:44 [initandlisten] recover begin
Fri Nov 16 10:47:44 [initandlisten] couldn't open /usr/local/var/mongodb/journal/lsn errno:13 Permission denied
Fri Nov 16 10:47:44 [initandlisten] Assertion failure f.is_open() src/mongo/db/dur_journal.cpp 552
0x10037637b 0x1000b0196 0x100105ec2 0x10049de46 0x10049e445 0x10049e628 0x1002401ac 0x1000057ec 0x10000608d 0x10000d534 0x10000f309 0x100000ea4 0x3
0 mongod 0x000000010037637b _ZN5mongo15printStackTraceERSo + 43
1 mongod 0x00000001000b0196 _ZN5mongo12verifyFailedEPKcS1_j + 310
2 mongod 0x0000000100105ec2 _ZN5mongo3dur14journalReadLSNEv + 898
3 mongod 0x000000010049de46 _ZN5mongo3dur11RecoveryJob2goERSt6vectorIN5boost11filesystem210basic_pathISsNS4_11path_traitsEEESaIS7_EE + 86
4 mongod 0x000000010049e445 _ZN5mongo3dur8_recoverEv + 405
5 mongod 0x000000010049e628 _ZN5mongo3dur7recoverEv + 40
6 mongod 0x00000001002401ac _ZN5mongo3dur7startupEv + 124
7 mongod 0x00000001000057ec _ZN5mongo14_initAndListenEi + 1052
8 mongod 0x000000010000608d _ZN5mongo13initAndListenEi + 29
9 mongod 0x000000010000d534 _ZL11mongoDbMainiPPc + 29348
10 mongod 0x000000010000f309 main + 9
11 mongod 0x0000000100000ea4 start + 52
12 ??? 0x0000000000000003 0x0 + 3
Fri Nov 16 10:47:44 [initandlisten] dbexception during recovery: 13611 can't read lsn file in journal directory : assertion src/mongo/db/dur_journal.cpp:552
Fri Nov 16 10:47:44 [initandlisten] exception in initAndListen: 13611 can't read lsn file in journal directory : assertion src/mongo/db/dur_journal.cpp:552, terminating
Fri Nov 16 10:47:44 dbexit:
Fri Nov 16 10:47:44 [initandlisten] shutdown: going to close listening sockets...
Fri Nov 16 10:47:44 [initandlisten] shutdown: going to flush diaglog...
Fri Nov 16 10:47:44 [initandlisten] shutdown: going to close sockets...
Fri Nov 16 10:47:44 [initandlisten] shutdown: waiting for fs preallocator...
Fri Nov 16 10:47:44 [initandlisten] shutdown: lock for final commit...
Fri Nov 16 10:47:44 [initandlisten] shutdown: final commit...
Fri Nov 16 10:47:44 [initandlisten] shutdown: closing all files...
Fri Nov 16 10:47:44 [initandlisten] closeAllFiles() finished
Fri Nov 16 10:47:44 [initandlisten] shutdown: removing fs lock...
Fri Nov 16 10:47:44 dbexit: really exiting now
There is no output when I run:
netstat -f inet -a -n | grep 27101
Thanks for your help!
As #Nilam said one of your problems is the journal, however, you actually have two problems here, the one I am talking about being more of a warning:
Fri Nov 16 10:47:44 [initandlisten] ** WARNING: soft rlimits too low. Number of files is 256, should be at least 1000
This specifically relates to: https://jira.mongodb.org/browse/SERVER-4407
You should look into increasing your openfile count in Linux otherwise you could get this: http://www.mongodb.org/display/DOCS/Too+Many+Open+Files of course this page also tells you how to solve the problem.
NB: as a side note to the journal point, it looks like your MongoDB node could be trying to recover from something. I would check why.
Similar to #Hartator's answer, this solved the issue for me:
chmod -R 777 /data/db
You have
journal dir=/usr/local/var/mongodb/journal
The error shows is
[initandlisten] couldn't open /usr/local/var/mongodb/journal/lsn errno:13 Permission denied
So check if journal directory exist and proper write permissions are given to this folder.
Try to run :
chmod 777 -R /usr/local/var/mongodb
I was running MongoDB on Amazon AWS, until one day it ran out of space. Then, the database is completely inaccessible,and I tried to shutdown the database. Having restarted the server and installed larger disk, the MongoDB can't be started.
I tried to repair the database, and the error is shown like below:
Wed Aug 15 10:08:04 [initandlisten] MongoDB starting : pid=1447 port=27017 dbpath=/var/lib/mongodb/ 32-bit
** NOTE: when using MongoDB 32 bit, you are limited to about 2 gigabytes of data
** see http://blog.mongodb.org/post/137788967/32-bit-limitations
** with --dur, the limit is lower
Wed Aug 15 10:08:04 [initandlisten] db version v1.8.3, pdfile version 4.5
Wed Aug 15 10:08:04 [initandlisten] git version: c206d77e94bc3b65c76681df5a6b605f68a2de05
Wed Aug 15 10:08:04 [initandlisten] build sys info: Linux domU-12-31-39-01-70-B4 2.6.21.7-2.fc8xen #1 SMP Fri Feb 15 12:39:36 EST 2008 i686 BOOST_LIB_VERSION=1_37
Wed Aug 15 10:08:04 [initandlisten] **
Wed Aug 15 10:08:04 [initandlisten] **
Wed Aug 15 10:08:04 [initandlisten] need to upgrade database md with pdfile version 559903.1345021735, new version: 4.5
Wed Aug 15 10:08:04 [initandlisten] starting upgrade
Wed Aug 15 10:08:04 [initandlisten] md repairDatabase md
Wed Aug 15 10:08:04 [initandlisten] md.system.namespaces Assertion failure isOk() db/pdfile.h 259
0x817c27b 0x819121e 0x83cf525 0x83d6bec 0x83f5420 0x82a4fb7 0x83f8680 0x83fcece 0x8401358 0x8401e29 0x8402abd 0x828f755 0x83b3d67 0x83b931c 0x83bb3df 0x81ce25f 0x81ae714 0x8380f31 0x83831ba 0x83d4faf
mongod(_ZN5mongo12sayDbContextEPKc+0xcb) [0x817c27b]
mongod(_ZN5mongo8assertedEPKcS1_j+0x12e) [0x819121e]
mongod(_ZN5mongo11DataFileMgr7findAllEPKcRKNS_7DiskLocE+0x7a5) [0x83cf525]
mongod(_ZN5mongo13findTableScanEPKcRKNS_7BSONObjERKNS_7DiskLocE+0x13c) [0x83d6bec]
mongod(_ZNK5mongo9QueryPlan9newCursorERKNS_7DiskLocEi+0x830) [0x83f5420]
mongod(_ZN5mongo11UserQueryOp5_initEv+0x437) [0x82a4fb7]
mongod(_ZN5mongo12QueryPlanSet6Runner6initOpERNS_7QueryOpE+0x110) [0x83f8680]
mongod(_ZN5mongo12QueryPlanSet6Runner3runEv+0x2ee) [0x83fcece]
mongod(_ZN5mongo12QueryPlanSet5runOpERNS_7QueryOpE+0x2e8) [0x8401358]
mongod(_ZN5mongo16MultiPlanScanner9runOpOnceERNS_7QueryOpE+0x59) [0x8401e29]
mongod(_ZN5mongo16MultiPlanScanner5runOpERNS_7QueryOpE+0x2d) [0x8402abd]
mongod(_ZN5mongo8runQueryERNS_7MessageERNS_12QueryMessageERNS_5CurOpES1_+0x1575) [0x828f755]
mongod() [0x83b3d67]
mongod(_ZN5mongo16assembleResponseERNS_7MessageERNS_10DbResponseERKNS_8SockAddrE+0x70c) [0x83b931c]
mongod(_ZN5mongo14DBDirectClient4callERNS_7MessageES2_bPSs+0x7f) [0x83bb3df]
mongod(_ZN5mongo14DBClientCursor4initEv+0x15f) [0x81ce25f]
mongod(_ZN5mongo12DBClientBase5queryERKSsNS_5QueryEiiPKNS_7BSONObjEii+0x2a4) [0x81ae714]
mongod(_ZN5mongo6Cloner2goEPKcRSsRKSsbbbb+0x451) [0x8380f31]
mongod(_ZN5mongo9cloneFromEPKcRSsRKSsbbbb+0x5a) [0x83831ba]
mongod(_ZN5mongo14repairDatabaseESsRSsbb+0x38f) [0x83d4faf]
Wed Aug 15 10:08:04 [initandlisten] assertion 0 assertion db/pdfile.h:259 ns:md.system.namespaces query:{}
Wed Aug 15 10:08:04 [initandlisten] Assertion: 10290:bad system.namespaces object { $err: "assertion db/pdfile.h:259" }
0x819045e 0x8381181 0x83831ba 0x83d4faf 0x852a30c 0x852b217 0x852e733 0x852f59c 0x85366bb 0xb74a4ce7 0x810b551
mongod(_ZN5mongo11msgassertedEiPKc+0x15e) [0x819045e]
mongod(_ZN5mongo6Cloner2goEPKcRSsRKSsbbbb+0x6a1) [0x8381181]
mongod(_ZN5mongo9cloneFromEPKcRSsRKSsbbbb+0x5a) [0x83831ba]
mongod(_ZN5mongo14repairDatabaseESsRSsbb+0x38f) [0x83d4faf]
mongod(_ZN5mongo11doDBUpgradeERKSsSsPNS_14DataFileHeaderE+0x6c) [0x852a30c]
mongod() [0x852b217]
mongod(_ZN5mongo14_initAndListenEiPKc+0x4b3) [0x852e733]
mongod(ZN5mongo13initAndListenEiPKc+0x2c) [0x852f59c]
mongod(main+0x6bdb) [0x85366bb]
/lib/libc.so.6(_libc_start_main+0xe7) [0xb74a4ce7]
mongod() [0x810b551]
Wed Aug 15 10:08:04 [initandlisten] exception in initAndListen std::exception: bad system.namespaces object { $err: "assertion db/pdfile.h:259" }, terminating
Wed Aug 15 10:08:04 dbexit:
Wed Aug 15 10:08:04 [initandlisten] shutdown: going to close listening sockets...
Wed Aug 15 10:08:04 [initandlisten] shutdown: going to flush diaglog...
Wed Aug 15 10:08:04 [initandlisten] shutdown: going to close sockets...
Wed Aug 15 10:08:04 [initandlisten] shutdown: waiting for fs preallocator...
Wed Aug 15 10:08:04 [initandlisten] shutdown: closing all files...
Wed Aug 15 10:08:04 closeAllFiles() finished
Wed Aug 15 10:08:04 [initandlisten] shutdown: removing fs lock...
Wed Aug 15 10:08:04 dbexit: really exiting now
I have all *.number and *.ns file completely. But it seems the ns file is corrupted. Any way I could repair the database?
Thanks
If repair doesn't work then you are unfortunately out of luck.
I'm presuming that you didn't have journaling enabled or any back-ups. I would strongly encourage you to run with journaling enabled and follow one of the back-up strategies outlined here. The LVM snapshot is easy to implement, fairly quick and reliable.
You are better to run MongoDB on a 64-bit system also. 32-bit limits you to 2gb of data. You say that you "ran out of space"?