Mongodb crashes after trying to sort - mongodb

I'm running mongodb 2.4.10 on a raspberry pi 3b. After few days it started to randomly crash and doesn't start without --journal. I created new database, but again it started to crash. After each crash it requires --repair.
What I noticed is it crashes after trying to sort
User.find({}).limit(10).sort({
messages: -1
}).exec(function(err, docs) {
...
});
But it works fines with find
User.find({}, function(err, docs) {
...
});
Here's full log of mongodb
Sat Feb 11 05:56:02.356
Sat Feb 11 05:56:02.356 warning: 32-bit servers don't have journaling enabled by default. Please use --journal if you want durability.
Sat Feb 11 05:56:02.356
Sat Feb 11 05:56:02.386 [initandlisten] MongoDB starting : pid=754 port=27017 dbpath=db 32-bit host=raspberrypi
Sat Feb 11 05:56:02.386 [initandlisten]
Sat Feb 11 05:56:02.386 [initandlisten] ** NOTE: This is a 32 bit MongoDB binary.
Sat Feb 11 05:56:02.386 [initandlisten] ** 32 bit builds are limited to less than 2GB of data (or less with --journal).
Sat Feb 11 05:56:02.386 [initandlisten] ** Note that journaling defaults to off for 32 bit and is currently off.
Sat Feb 11 05:56:02.386 [initandlisten] ** See http://dochub.mongodb.org/core/32bit
Sat Feb 11 05:56:02.386 [initandlisten]
Sat Feb 11 05:56:02.386 [initandlisten] db version v2.4.10
Sat Feb 11 05:56:02.386 [initandlisten] git version: nogitversion
Sat Feb 11 05:56:02.386 [initandlisten] build info: Linux bm-wb-03 3.19.0-trunk-armmp #1 SMP Debian 3.19.1-1~exp1+plugwash1 (2015-03-28) armv7l BOOST_LIB_VERSION=1_55
Sat Feb 11 05:56:02.386 [initandlisten] allocator: system
Sat Feb 11 05:56:02.386 [initandlisten] options: { dbpath: "db" }
Sat Feb 11 05:56:02.441 [websvr] admin web console waiting for connections on port 28017
Sat Feb 11 05:56:02.442 [initandlisten] waiting for connections on port 27017
Sat Feb 11 05:56:03.724 [initandlisten] connection accepted from 127.0.0.1:43800 #1 (1 connection now open)
Sat Feb 11 05:56:39.087 [conn1] assertion 13097 Unsupported projection option: messages ns:test.users query:{ id: 378418036.0 }
Sat Feb 11 05:56:53.794 [conn1] end connection 127.0.0.1:43800 (0 connections now open)
Sat Feb 11 05:57:01.512 [initandlisten] connection accepted from 127.0.0.1:43960 #2 (1 connection now open)
Sat Feb 11 05:57:01.786 [initandlisten] connection accepted from 127.0.0.1:43962 #3 (2 connections now open)
Sat Feb 11 05:57:01.787 [initandlisten] connection accepted from 127.0.0.1:43964 #4 (3 connections now open)
Sat Feb 11 05:57:01.787 [initandlisten] connection accepted from 127.0.0.1:43966 #5 (4 connections now open)
Sat Feb 11 05:57:08.329 [initandlisten] connection accepted from 127.0.0.1:43974 #6 (5 connections now open)
Sat Feb 11 05:57:09.474 Invalid access at address: 0x6b70f6c7 from thread: conn3
Sat Feb 11 05:57:09.474 Got signal: 7 (Bus error).
Sat Feb 11 05:57:09.479 Backtrace:
0x664160 0x16e370 0x16e6c0 0x766031a0 0x3bfa40 0x433a5c 0x430844 0x431428 0x3c5dc8 0x3c81a4 0x182474 0x64aa08
mongod(_ZN5mongo15printStackTraceERSo+0x1c) [0x664160]
mongod(_ZN5mongo10abruptQuitEi+0x2a0) [0x16e370]
mongod(_ZN5mongo24abruptQuitWithAddrSignalEiP9siginfo_tPv+0x1fc) [0x16e6c0]
/lib/arm-linux-gnueabihf/libc.so.6(__default_rt_sa_restorer_v2+0) [0x766031a0]
mongod(_ZN5mongo22BSONElementManipulator9SetNumberEd+0x98) [0x3bfa40]
mongod(_ZN5mongo11ModSetState16applyModsInPlaceEb+0x3d4) [0x433a5c]
mongod(_ZN5mongo14_updateObjectsEbPKcRKNS_7BSONObjES4_bbbRNS_7OpDebugEPNS_11RemoveSaverEbRKNS_24QueryPlanSelectionPolicyEb+0x2788) [0x430844]
mongod(_ZN5mongo13updateObjectsEPKcRKNS_7BSONObjES4_bbbRNS_7OpDebugEbRKNS_24QueryPlanSelectionPolicyE+0x70) [0x431428]
mongod(_ZN5mongo14receivedUpdateERNS_7MessageERNS_5CurOpE+0x35c) [0x3c5dc8]
mongod(_ZN5mongo16assembleResponseERNS_7MessageERNS_10DbResponseERKNS_11HostAndPortE+0x14b0) [0x3c81a4]
mongod(_ZN5mongo16MyMessageHandler7processERNS_7MessageEPNS_21AbstractMessagingPortEPNS_9LastErrorE+0x9c) [0x182474]
mongod(_ZN5mongo17PortMessageServer17handleIncomingMsgEPv+0x420) [0x64aa08]

Related

Mongo service doesn't connect

I just did a manual installation of mongoDB like the documentation says and when I run ./mongocommand this is what I get.
./mongo
MongoDB shell version: 2.4.8
connecting to: test
It doesn't say anymore and doesn't connect to testdatabase.
Anyone know how can I solve this problem?
Thank's in advance
EDIT: I uninstall all and re-install using Brew. Now this is what I get when I run mongod
all output going to: /usr/local/var/log/mongodb/mongo.log
And this when i run mongo
MongoDB shell version: 2.4.8
connecting to: test
Wed Jan 8 13:35:04.634 Error: couldn't connect to server 127.0.0.1:27017 at src/mongo/shell/mongo.js:145
exception: connect failed
And this is what /usr/local/var/log/mongodb/mongo.logshows
***** SERVER RESTARTED *****
Wed Jan 8 13:34:11.749 [initandlisten] MongoDB starting : pid=745 port=27017 dbpath=/usr/local/var/mongodb 64-bit host=MacBook-Pro-de-Kemical.local
Wed Jan 8 13:34:11.749 [initandlisten]
Wed Jan 8 13:34:11.749 [initandlisten] ** WARNING: soft rlimits too low. Number of files is 256, should be at least 1000
Wed Jan 8 13:34:11.749 [initandlisten] db version v2.4.8
Wed Jan 8 13:34:11.749 [initandlisten] git version: nogitversion
Wed Jan 8 13:34:11.749 [initandlisten] build info: Darwin minimountain.local 12.5.0 Darwin Kernel Version 12.5.0: Sun Sep 29 13:33:47 PDT 2013; root:xnu-2050.48.12~1/RELEASE_X86_64 x86_64 BOOST_LIB_VERSION=1_49
Wed Jan 8 13:34:11.749 [initandlisten] allocator: tcmalloc
Wed Jan 8 13:34:11.749 [initandlisten] options: { bind_ip: "127.0.0.1", config: "/usr/local/etc/mongod.conf", dbpath: "/usr/local/var/mongodb", logappend: "true", logpath: "/usr/local/var/log/mongodb/mongo.log" }
Wed Jan 8 13:34:11.750 [initandlisten] exception in initAndListen: 10310 Unable to lock file: /usr/local/var/mongodb/mongod.lock. Is a mongod instance already running?, terminating
Wed Jan 8 13:34:11.750 dbexit:
Wed Jan 8 13:34:11.750 [initandlisten] shutdown: going to close listening sockets...
Wed Jan 8 13:34:11.750 [initandlisten] shutdown: going to flush diaglog...
Wed Jan 8 13:34:11.750 [initandlisten] shutdown: going to close sockets...
Wed Jan 8 13:34:11.750 [initandlisten] shutdown: waiting for fs preallocator...
Wed Jan 8 13:34:11.750 [initandlisten] shutdown: lock for final commit...
Wed Jan 8 13:34:11.750 [initandlisten] shutdown: final commit...
Wed Jan 8 13:34:11.750 [initandlisten] shutdown: closing all files...
Wed Jan 8 13:34:11.750 [initandlisten] closeAllFiles() finished
Wed Jan 8 13:34:11.750 dbexit: really exiting now
First of all, have you actually started the "mongod" server process?
If you have, you need to find out what is happening and one way to do this is with the "strace" tool. Running it like follows creates a log file in /tmp/strace.log and that should tell you system level wise what's going on:
strace -o /tmp/strace.log ./mongo

mongod shell doesn't start, data/db doesn't exsist

When I start mongod trowed error. What is the problem?
root#cs12867:~# mongod
mongod --help for help and startup options
Sun Dec 22 15:37:44.081 [initandlisten] MongoDB starting : pid=18415 port=27017 dbpath=/data/db/ 64-bit host=cs12867
Sun Dec 22 15:37:44.081 [initandlisten] db version v2.4.6
Sun Dec 22 15:37:44.081 [initandlisten] git version: b9925db5eac369d77a3a5f5d98a145eaaacd9673
Sun Dec 22 15:37:44.081 [initandlisten] build info: Linux ip-10-2-29-40 2.6.21.7-2.ec2.v1.2.fc8xen #1 SMP Fri Nov 20 17:48:28 EST 2009 x86_64 BOOST_LIB_VERSION=1_49
Sun Dec 22 15:37:44.081 [initandlisten] allocator: tcmalloc
Sun Dec 22 15:37:44.081 [initandlisten] options: {}
Sun Dec 22 15:37:44.081 [initandlisten] exception in initAndListen: 10296
*********************************************************************
ERROR: dbpath (/data/db/) does not exist.
Create this directory or give existing directory in --dbpath.
See http://dochub.mongodb.org/core/startingandstoppingmongo
*********************************************************************
, terminating
Sun Dec 22 15:37:44.081 dbexit:
Sun Dec 22 15:37:44.081 [initandlisten] shutdown: going to close listening sockets...
Sun Dec 22 15:37:44.081 [initandlisten] shutdown: going to flush diaglog...
Sun Dec 22 15:37:44.081 [initandlisten] shutdown: going to close sockets...
Sun Dec 22 15:37:44.081 [initandlisten] shutdown: waiting for fs preallocator...
Sun Dec 22 15:37:44.081 [initandlisten] shutdown: lock for final commit...
Sun Dec 22 15:37:44.081 [initandlisten] shutdown: final commit...
Sun Dec 22 15:37:44.081 [initandlisten] shutdown: closing all files...
Sun Dec 22 15:37:44.081 [initandlisten] closeAllFiles() finished
Sun Dec 22 15:37:44.081 dbexit: really exiting now
I set --dbpath:
root#cs12867:~# mongod --dbpath /var/lib/mongodb/data/db
Sun Dec 22 15:29:25.121 [initandlisten] MongoDB starting : pid=17739 port=27017 dbpath=/var/lib/mongodb/data/db 64-bit host=cs12867
Sun Dec 22 15:29:25.121 [initandlisten] db version v2.4.6
Sun Dec 22 15:29:25.121 [initandlisten] git version: b9925db5eac369d77a3a5f5d98a145eaaacd9673
Sun Dec 22 15:29:25.121 [initandlisten] build info: Linux ip-10-2-29-40 2.6.21.7-2.ec2.v1.2.fc8xen #1 SMP Fri Nov 20 17:48:28 EST 2009 x86_64 BOOST_LIB_VERSION=1_49
Sun Dec 22 15:29:25.121 [initandlisten] allocator: tcmalloc
Sun Dec 22 15:29:25.121 [initandlisten] options: { dbpath: "/var/lib/mongodb/data/db" }
Sun Dec 22 15:29:25.121 [initandlisten]
Sun Dec 22 15:29:25.121 [initandlisten] ** WARNING: Readahead for /var/lib/mongodb/data/db is set to 2048KB
Sun Dec 22 15:29:25.121 [initandlisten] ** We suggest setting it to 256KB (512 sectors) or less
Sun Dec 22 15:29:25.121 [initandlisten] ** http://dochub.mongodb.org/core/readahead
Sun Dec 22 15:29:25.123 [initandlisten] journal dir=/var/lib/mongodb/data/db/journal
Sun Dec 22 15:29:25.123 [initandlisten] recover : no journal files present, no recovery needed
Sun Dec 22 15:29:25.134 [initandlisten] waiting for connections on port 27017
Sun Dec 22 15:29:25.134 [websvr] admin web console waiting for connections on port 28017
Sun Dec 22 15:29:26.179 [initandlisten] connection accepted from 127.0.0.1:37164 #1 (1 connection now open)
Sun Dec 22 15:29:26.179 [FileAllocator] allocating new datafile /var/lib/mongodb/data/db/Tester.ns, filling with zeroes...
Sun Dec 22 15:29:26.179 [FileAllocator] creating directory /var/lib/mongodb/data/db/_tmp
Sun Dec 22 15:29:26.181 [FileAllocator] done allocating datafile /var/lib/mongodb/data/db/Tester.ns, size: 16MB, took 0 secs
Sun Dec 22 15:29:26.181 [FileAllocator] allocating new datafile /var/lib/mongodb/data/db/Tester.0, filling with zeroes...
Sun Dec 22 15:29:26.181 [FileAllocator] done allocating datafile /var/lib/mongodb/data/db/Tester.0, size: 64MB, took 0 secs
Sun Dec 22 15:29:26.182 [FileAllocator] allocating new datafile /var/lib/mongodb/data/db/Tester.1, filling with zeroes...
Sun Dec 22 15:29:26.182 [FileAllocator] done allocating datafile /var/lib/mongodb/data/db/Tester.1, size: 128MB, took 0 secs
Sun Dec 22 15:29:26.184 [conn1] build index Tester.user { _id: 1 }
Sun Dec 22 15:29:26.184 [conn1] build index done. scanned 0 total records. 0 secs
Sun Dec 22 15:29:26.184 [conn1] end connection 127.0.0.1:37164 (0 connections now open)
Sun Dec 22 15:29:26.192 [initandlisten] connection accepted from 127.0.0.1:37165 #2 (1 connection now open)
Sun Dec 22 15:29:26.732 [initandlisten] connection accepted from 127.0.0.1:37166 #3 (2 connections now open)
Sun Dec 22 15:29:26.733 [conn3] end connection 127.0.0.1:37166 (1 connection now open)
Sun Dec 22 15:29:26.740 [initandlisten] connection accepted from 127.0.0.1:37167 #4 (2 connections now open)
Sun Dec 22 15:29:26.740 [conn4] end connection 127.0.0.1:37167 (1 connection now open)
Sun Dec 22 15:29:26.834 [initandlisten] connection accepted from 127.0.0.1:37168 #5 (2 connections now open)
Sun Dec 22 15:29:26.835 [conn5] end connection 127.0.0.1:37168 (1 connection now open)
Sun Dec 22 15:29:26.843 [initandlisten] connection accepted from 127.0.0.1:37169 #6 (2 connections now open)
Sun Dec 22 15:29:27.252 [initandlisten] connection accepted from 127.0.0.1:37170 #7 (3 connections now open)
Sun Dec 22 15:29:27.253 [conn7] end connection 127.0.0.1:37170 (2 connections now open)
Sun Dec 22 15:29:27.304 [initandlisten] connection accepted from 127.0.0.1:37171 #8 (3 connections now open)
Sun Dec 22 15:29:27.304 [conn8] end connection 127.0.0.1:37171 (2 connections now open)
Sun Dec 22 15:29:28.332 [initandlisten] connection accepted from 127.0.0.1:37173 #9 (3 connections now open)
Sun Dec 22 15:29:28.333 [conn9] end connection 127.0.0.1:37173 (2 connections now open)
Sun Dec 22 15:29:28.352 [initandlisten] connection accepted from 127.0.0.1:37174 #10 (3 connections now open)
Sun Dec 22 15:29:28.842 [conn10] end connection 127.0.0.1:37174 (2 connections now open)
Sun Dec 22 15:29:28.862 [initandlisten] connection accepted from 127.0.0.1:37176 #11 (3 connections now open)
Sun Dec 22 15:29:28.863 [conn11] end connection 127.0.0.1:37176 (2 connections now open)
Sun Dec 22 15:29:30.362 [initandlisten] connection accepted from 127.0.0.1:37178 #12 (3 connections now open)
Sun Dec 22 15:29:30.363 [initandlisten] connection accepted from 127.0.0.1:37179 #13 (4 connections now open)
Sun Dec 22 15:29:30.364 [conn13] end connection 127.0.0.1:37179 (3 connections now open)
Sun Dec 22 15:37:36.523 [signalProcessingThread] shutdown: closing all files...
Sun Dec 22 15:37:36.523 [signalProcessingThread] closeAllFiles() finished
Sun Dec 22 15:37:36.523 [signalProcessingThread] journalCleanup...
Sun Dec 22 15:37:36.523 [signalProcessingThread] removeJournalFiles
Sun Dec 22 15:37:36.525 [signalProcessingThread] shutdown: removing fs lock...
Sun Dec 22 15:37:36.525 dbexit: really exiting now
Then again:
root#cs12867:~# mongod
mongod --help for help and startup options
Sun Dec 22 15:37:44.081 [initandlisten] MongoDB starting : pid=18415 port=27017 dbpath=/data/db/ 64-bit host=cs12867
Sun Dec 22 15:37:44.081 [initandlisten] db version v2.4.6
Sun Dec 22 15:37:44.081 [initandlisten] git version: b9925db5eac369d77a3a5f5d98a145eaaacd9673
Sun Dec 22 15:37:44.081 [initandlisten] build info: Linux ip-10-2-29-40 2.6.21.7-2.ec2.v1.2.fc8xen #1 SMP Fri Nov 20 17:48:28 EST 2009 x86_64 BOOST_LIB_VERSION=1_49
Sun Dec 22 15:37:44.081 [initandlisten] allocator: tcmalloc
Sun Dec 22 15:37:44.081 [initandlisten] options: {}
Sun Dec 22 15:37:44.081 [initandlisten] exception in initAndListen: 10296
*********************************************************************
ERROR: dbpath (/data/db/) does not exist.
Create this directory or give existing directory in --dbpath.
See http://dochub.mongodb.org/core/startingandstoppingmongo
*********************************************************************
, terminating
Sun Dec 22 15:37:44.081 dbexit:
Sun Dec 22 15:37:44.081 [initandlisten] shutdown: going to close listening sockets...
Sun Dec 22 15:37:44.081 [initandlisten] shutdown: going to flush diaglog...
Sun Dec 22 15:37:44.081 [initandlisten] shutdown: going to close sockets...
Sun Dec 22 15:37:44.081 [initandlisten] shutdown: waiting for fs preallocator...
Sun Dec 22 15:37:44.081 [initandlisten] shutdown: lock for final commit...
Sun Dec 22 15:37:44.081 [initandlisten] shutdown: final commit...
Sun Dec 22 15:37:44.081 [initandlisten] shutdown: closing all files...
Sun Dec 22 15:37:44.081 [initandlisten] closeAllFiles() finished
Sun Dec 22 15:37:44.081 dbexit: really exiting now
The default db path for mongodb is /data/db/.
So when you run:
mongod
MongoDB checks if /data/db/ is present and if the user has access to it. In your case, there is no such directory and hence the error.
However you can override the default db path using the --dbpath argument of mongod( or use a config file).
mongod --dbpath /var/lib/mongodb/data/db
In this case instead of checking for /data/db/ mongoDB check for /var/lib/mongodb/data/db. In your case, you have the specified directory and you have access to it and hence it runs.
And doing this doesn't start the mongo shell. To start the shell, start mongod first as mentioned above and leave it running. Then in another terminal execute
mongo
This will start the mongo shell.
As I see from the problem that creating the data/db directory will fix your problem.
Create that and then add like this if you have directory path in c drive.
mongod --dbpath "c://data/db"
After setting the dbpath you should start the mongodb as service.
sudo service mongodb start
Another solution to that problem could be:
.
Delete the files inside of /data/journal/ or /data/db/journal/
.
Especially if your database stopped working after an unclean restart / crash. It seems that the journal files can get corrupted and block the mongo service from starting.
Remember to backup the files first just in case.
Following these posts:
https://stackoverflow.com/a/16323101
https://jira.mongodb.org/browse/SERVER-5380

Mongodb: kern.sched unavailable

I am quite new to mongodb. After install it via: brew intall mongodb then type mongod in the terminal, error throws out: kern.sched unavailable.
I am using mac os verion 10.6.8.
The log:
Sat Apr 13 16:40:56.374 [initandlisten] MongoDB starting : pid=6387 port=27017 dbpath=/usr/local/var/mongodb 64-bit host=fatbizkit
Sat Apr 13 16:40:56.374 [initandlisten]
Sat Apr 13 16:40:56.374 [initandlisten] ** WARNING: soft rlimits too low. Number of files is 256, should be at least 1000
Sat Apr 13 16:40:56.374 [initandlisten] db version v2.4.1
Sat Apr 13 16:40:56.374 [initandlisten] git version: 1560959e9ce11a693be8b4d0d160d633eee75110
Sat Apr 13 16:40:56.374 [initandlisten] build info: Darwin bs-osx-106-x86-64-1.local 10.8.0 Darwin Kernel Version 10.8.0: Tue Jun 7 16:33:$
Sat Apr 13 16:40:56.374 [initandlisten] allocator: system
Sat Apr 13 16:40:56.374 [initandlisten] options: { bind_ip: "127.0.0.1", config: "/usr/local/etc/mongod.conf", dbpath: "/usr/local/var/mong$
Sat Apr 13 16:40:56.375 [initandlisten] journal dir=/usr/local/var/mongodb/journal
Sat Apr 13 16:40:56.375 [initandlisten] recover : no journal files present, no recovery needed
Sat Apr 13 16:40:56.441 [websvr] admin web console waiting for connections on port 28017
Sat Apr 13 16:40:56.442 [initandlisten] waiting for connections on port 27017
Sat Apr 13 16:41:29.416 got signal 2 (Interrupt), will terminate after current cmd ends
Sat Apr 13 16:41:29.416 [interruptThread] now exiting
Sat Apr 13 16:41:29.416 dbexit:
Sat Apr 13 16:41:29.416 [interruptThread] shutdown: going to close listening sockets...
Sat Apr 13 16:41:29.416 [interruptThread] closing listening socket: 9
Sat Apr 13 16:41:29.416 [interruptThread] closing listening socket: 10
Sat Apr 13 16:41:29.416 [interruptThread] closing listening socket: 11
Sat Apr 13 16:41:29.416 [interruptThread] removing socket file: /tmp/mongodb-27017.sock
Sat Apr 13 16:41:29.417 [interruptThread] shutdown: going to flush diaglog...
Sat Apr 13 16:41:29.417 [interruptThread] shutdown: going to close sockets...
Sat Apr 13 16:41:29.417 [interruptThread] shutdown: waiting for fs preallocator...
Sat Apr 13 16:41:29.417 [interruptThread] shutdown: lock for final commit...
Sat Apr 13 16:41:29.417 [interruptThread] shutdown: final commit...
Sat Apr 13 16:41:29.432 [interruptThread] shutdown: closing all files...
Sat Apr 13 16:41:29.442 [interruptThread] closeAllFiles() finished
Sat Apr 13 16:41:29.442 [interruptThread] journalCleanup...
Sat Apr 13 16:41:29.442 [interruptThread] removeJournalFiles
Sat Apr 13 16:41:29.442 [interruptThread] shutdown: removing fs lock...
Sat Apr 13 16:41:29.442 dbexit: really exiting now
I tried google, but no luck. Anyone any idea?
After installing mongodb with brew and typing the command mongod I had the same error, and the same logs. I left the terminal opened.
I opened another terminal window, and typed the command mongo and i am now on mongodb command line.
I could also see that mongodb was running in a browser on the following url : http://localhost:28017/
As it work i left the error for now, I'll try to find an answer later.
Hope it will help you.
what mongodb version are you using?
log statements you post did not show any info about the error, can you try to reproduce again and send detailed logs?
Did you try to re-install or install from other source?
Regards,
Moacy
I had this problem after using a startup file from http://hunterford.me/mongodb-startup-item/ . I do not know if this is the only possible cause of the error you describe, but it was in my case.
The cause of the error seemed to be that in the startup file there was the segment:
<string>/usr/local/bin/mongod</string>
Which may or may not be where mongod is actually installed. I installed mongo using macports, so I found mongod to be in opt/local/bin - YMMV
Once I had changed the directory value in the startup item to the correct one, I no longer had this problem, though I have had further problems with my "soft rlimits being too low", Solutions to this exist but since you are using mac os x for mongo, I suspect you're developing/demonstrating and so this probably won't be a problem.
I hope this helps.

Mongodb gives error during startup

Whenever I try to play with mongo's interactive shell, it dies:
somekittens#DLserver01:~$ mongo
MongoDB shell version: 2.2.2
connecting to: test
Mon Dec 17 13:14:16 DBClientCursor::init call() failed
Mon Dec 17 13:14:16 Error: Error during mongo startup. :: caused by :: 10276 DBClientBase::findN: transport error: 127.0.0.1:27017 ns: admin.$cmd query: { whatsmyuri: 1 } src/mongo/shell/mongo.js:91
exception: connect failed
I'm able to repair the install (deleting mongodb.lock, etc) and get back to this point, but it'll only die again.
/var/log/mongodb/mongodb.log
Mon Dec 17 13:14:03
Mon Dec 17 13:14:03 warning: 32-bit servers don't have journaling enabled by default. Please use --journal if you want durability.
Mon Dec 17 13:14:03
Mon Dec 17 13:14:03 [initandlisten] MongoDB starting : pid=2674 port=27017 dbpath=/var/lib/mongodb 32-bit host=DLserver01
Mon Dec 17 13:14:03 [initandlisten]
Mon Dec 17 13:14:03 [initandlisten] ** NOTE: when using MongoDB 32 bit, you are limited to about 2 gigabytes of data
Mon Dec 17 13:14:03 [initandlisten] ** see http://blog.mongodb.org/post/137788967/32-bit-limitations
Mon Dec 17 13:14:03 [initandlisten] ** with --journal, the limit is lower
Mon Dec 17 13:14:03 [initandlisten]
Mon Dec 17 13:14:03 [initandlisten] db version v2.2.2, pdfile version 4.5
Mon Dec 17 13:14:03 [initandlisten] git version: d1b43b61a5308c4ad0679d34b262c5af9d664267
Mon Dec 17 13:14:03 [initandlisten] build info: Linux domU-12-31-39-01-70-B4 2.6.21.7-2.fc8xen #1 SMP Fri Feb 15 12:39:36 EST 2008 i686 BOOST_LIB_VERSION=1_49
Mon Dec 17 13:14:03 [initandlisten] options: { config: "/etc/mongodb.conf", dbpath: "/var/lib/mongodb", logappend: "true", logpath: "/var/log/mongodb/mongodb.log" }
Mon Dec 17 13:14:03 [initandlisten] Unable to check for journal files due to: boost::filesystem::basic_directory_iterator constructor: No such file or directory: "/var/lib/mongodb/journal"
Mon Dec 17 13:14:03 [initandlisten] couldn't unlink socket file /tmp/mongodb-27017.sockerrno:1 Operation not permitted skipping
Mon Dec 17 13:14:03 [initandlisten] waiting for connections on port 27017
Mon Dec 17 13:14:03 [websvr] admin web console waiting for connections on port 28017
Mon Dec 17 13:14:16 [initandlisten] connection accepted from 127.0.0.1:57631 #1 (1 connection now open)
Mon Dec 17 13:14:16 Invalid operation at address: 0x819bb23 from thread: conn1
Mon Dec 17 13:14:16 Got signal: 4 (Illegal instruction).
Mon Dec 17 13:14:16 Backtrace:
0x8759eaa 0x817033a 0x81709ff 0x20e40c 0x819bb23 0x854cd54 0x85377d1 0x846b594 0x83e5591 0x83e6c15 0x81902b4 0x8746731 0x49ad4c 0x34ed3e
/usr/bin/mongod(_ZN5mongo15printStackTraceERSo+0x2a) [0x8759eaa]
/usr/bin/mongod(_ZN5mongo10abruptQuitEi+0x3ba) [0x817033a]
/usr/bin/mongod(_ZN5mongo24abruptQuitWithAddrSignalEiP7siginfoPv+0x2af) [0x81709ff]
[0x20e40c]
/usr/bin/mongod(_ZNK5mongo7BSONObj4copyEv+0x33) [0x819bb23]
/usr/bin/mongod(_ZN5mongo11ParsedQuery4initERKNS_7BSONObjE+0x494) [0x854cd54]
/usr/bin/mongod(_ZN5mongo11ParsedQueryC1ERNS_12QueryMessageE+0x91) [0x85377d1]
/usr/bin/mongod(_ZN5mongo8runQueryERNS_7MessageERNS_12QueryMessageERNS_5CurOpES1_+0x34) [0x846b594]
/usr/bin/mongod() [0x83e5591]
/usr/bin/mongod(_ZN5mongo16assembleResponseERNS_7MessageERNS_10DbResponseERKNS_11HostAndPortE+0x3d5) [0x83e6c15]
/usr/bin/mongod(_ZN5mongo16MyMessageHandler7processERNS_7MessageEPNS_21AbstractMessagingPortEPNS_9LastErrorE+0x84) [0x81902b4]
/usr/bin/mongod(_ZN5mongo3pms9threadRunEPNS_13MessagingPortE+0x421) [0x8746731]
/lib/i386-linux-gnu/libpthread.so.0(+0x6d4c) [0x49ad4c]
/lib/i386-linux-gnu/libc.so.6(clone+0x5e) [0x34ed3e]
Connecting using node's shell:
> mdb.open(function(err, db) { console.log(err) });
[Error: failed to connect to [localhost:27017]]
I've searched around for this error and found nothing of use. This is running on a fairly old server (Ubuntu 12.04 32-bit, 640MB RAM, 500MHz P2). How can I fix this?
There is an issue Invalid operation at address: 0x819b263 from thread: TTLMonitor in mongodb jira list. I think that is about your case.
A new server may be the easiest solution, otherwise you have to download the source code, make some modification and compile it youself.

MongoDB Unclean Shutdown - Fail to repair

I was running MongoDB on Amazon AWS, until one day it ran out of space. Then, the database is completely inaccessible,and I tried to shutdown the database. Having restarted the server and installed larger disk, the MongoDB can't be started.
I tried to repair the database, and the error is shown like below:
Wed Aug 15 10:08:04 [initandlisten] MongoDB starting : pid=1447 port=27017 dbpath=/var/lib/mongodb/ 32-bit
** NOTE: when using MongoDB 32 bit, you are limited to about 2 gigabytes of data
** see http://blog.mongodb.org/post/137788967/32-bit-limitations
** with --dur, the limit is lower
Wed Aug 15 10:08:04 [initandlisten] db version v1.8.3, pdfile version 4.5
Wed Aug 15 10:08:04 [initandlisten] git version: c206d77e94bc3b65c76681df5a6b605f68a2de05
Wed Aug 15 10:08:04 [initandlisten] build sys info: Linux domU-12-31-39-01-70-B4 2.6.21.7-2.fc8xen #1 SMP Fri Feb 15 12:39:36 EST 2008 i686 BOOST_LIB_VERSION=1_37
Wed Aug 15 10:08:04 [initandlisten] **
Wed Aug 15 10:08:04 [initandlisten] **
Wed Aug 15 10:08:04 [initandlisten] need to upgrade database md with pdfile version 559903.1345021735, new version: 4.5
Wed Aug 15 10:08:04 [initandlisten] starting upgrade
Wed Aug 15 10:08:04 [initandlisten] md repairDatabase md
Wed Aug 15 10:08:04 [initandlisten] md.system.namespaces Assertion failure isOk() db/pdfile.h 259
0x817c27b 0x819121e 0x83cf525 0x83d6bec 0x83f5420 0x82a4fb7 0x83f8680 0x83fcece 0x8401358 0x8401e29 0x8402abd 0x828f755 0x83b3d67 0x83b931c 0x83bb3df 0x81ce25f 0x81ae714 0x8380f31 0x83831ba 0x83d4faf
mongod(_ZN5mongo12sayDbContextEPKc+0xcb) [0x817c27b]
mongod(_ZN5mongo8assertedEPKcS1_j+0x12e) [0x819121e]
mongod(_ZN5mongo11DataFileMgr7findAllEPKcRKNS_7DiskLocE+0x7a5) [0x83cf525]
mongod(_ZN5mongo13findTableScanEPKcRKNS_7BSONObjERKNS_7DiskLocE+0x13c) [0x83d6bec]
mongod(_ZNK5mongo9QueryPlan9newCursorERKNS_7DiskLocEi+0x830) [0x83f5420]
mongod(_ZN5mongo11UserQueryOp5_initEv+0x437) [0x82a4fb7]
mongod(_ZN5mongo12QueryPlanSet6Runner6initOpERNS_7QueryOpE+0x110) [0x83f8680]
mongod(_ZN5mongo12QueryPlanSet6Runner3runEv+0x2ee) [0x83fcece]
mongod(_ZN5mongo12QueryPlanSet5runOpERNS_7QueryOpE+0x2e8) [0x8401358]
mongod(_ZN5mongo16MultiPlanScanner9runOpOnceERNS_7QueryOpE+0x59) [0x8401e29]
mongod(_ZN5mongo16MultiPlanScanner5runOpERNS_7QueryOpE+0x2d) [0x8402abd]
mongod(_ZN5mongo8runQueryERNS_7MessageERNS_12QueryMessageERNS_5CurOpES1_+0x1575) [0x828f755]
mongod() [0x83b3d67]
mongod(_ZN5mongo16assembleResponseERNS_7MessageERNS_10DbResponseERKNS_8SockAddrE+0x70c) [0x83b931c]
mongod(_ZN5mongo14DBDirectClient4callERNS_7MessageES2_bPSs+0x7f) [0x83bb3df]
mongod(_ZN5mongo14DBClientCursor4initEv+0x15f) [0x81ce25f]
mongod(_ZN5mongo12DBClientBase5queryERKSsNS_5QueryEiiPKNS_7BSONObjEii+0x2a4) [0x81ae714]
mongod(_ZN5mongo6Cloner2goEPKcRSsRKSsbbbb+0x451) [0x8380f31]
mongod(_ZN5mongo9cloneFromEPKcRSsRKSsbbbb+0x5a) [0x83831ba]
mongod(_ZN5mongo14repairDatabaseESsRSsbb+0x38f) [0x83d4faf]
Wed Aug 15 10:08:04 [initandlisten] assertion 0 assertion db/pdfile.h:259 ns:md.system.namespaces query:{}
Wed Aug 15 10:08:04 [initandlisten] Assertion: 10290:bad system.namespaces object { $err: "assertion db/pdfile.h:259" }
0x819045e 0x8381181 0x83831ba 0x83d4faf 0x852a30c 0x852b217 0x852e733 0x852f59c 0x85366bb 0xb74a4ce7 0x810b551
mongod(_ZN5mongo11msgassertedEiPKc+0x15e) [0x819045e]
mongod(_ZN5mongo6Cloner2goEPKcRSsRKSsbbbb+0x6a1) [0x8381181]
mongod(_ZN5mongo9cloneFromEPKcRSsRKSsbbbb+0x5a) [0x83831ba]
mongod(_ZN5mongo14repairDatabaseESsRSsbb+0x38f) [0x83d4faf]
mongod(_ZN5mongo11doDBUpgradeERKSsSsPNS_14DataFileHeaderE+0x6c) [0x852a30c]
mongod() [0x852b217]
mongod(_ZN5mongo14_initAndListenEiPKc+0x4b3) [0x852e733]
mongod(ZN5mongo13initAndListenEiPKc+0x2c) [0x852f59c]
mongod(main+0x6bdb) [0x85366bb]
/lib/libc.so.6(_libc_start_main+0xe7) [0xb74a4ce7]
mongod() [0x810b551]
Wed Aug 15 10:08:04 [initandlisten] exception in initAndListen std::exception: bad system.namespaces object { $err: "assertion db/pdfile.h:259" }, terminating
Wed Aug 15 10:08:04 dbexit:
Wed Aug 15 10:08:04 [initandlisten] shutdown: going to close listening sockets...
Wed Aug 15 10:08:04 [initandlisten] shutdown: going to flush diaglog...
Wed Aug 15 10:08:04 [initandlisten] shutdown: going to close sockets...
Wed Aug 15 10:08:04 [initandlisten] shutdown: waiting for fs preallocator...
Wed Aug 15 10:08:04 [initandlisten] shutdown: closing all files...
Wed Aug 15 10:08:04 closeAllFiles() finished
Wed Aug 15 10:08:04 [initandlisten] shutdown: removing fs lock...
Wed Aug 15 10:08:04 dbexit: really exiting now
I have all *.number and *.ns file completely. But it seems the ns file is corrupted. Any way I could repair the database?
Thanks
If repair doesn't work then you are unfortunately out of luck.
I'm presuming that you didn't have journaling enabled or any back-ups. I would strongly encourage you to run with journaling enabled and follow one of the back-up strategies outlined here. The LVM snapshot is easy to implement, fairly quick and reliable.
You are better to run MongoDB on a 64-bit system also. 32-bit limits you to 2gb of data. You say that you "ran out of space"?