Cant initialise replica set on debian (open/create failed in createPrivateMap) - mongodb

I try to setup MongoDB on my new virtual Server running with Debian 7.3. If a try to configurate the replica set with
hosts = {
"_id" : "rs0",
"version" : 1,
"members" : [
{
"_id" : 0,
"host" : "localhost:27017"
}
]
}
rs.initiate(hosts);
MongoDB crashes with following exceptions:
Tue Jan 21 00:10:24.599 [initandlisten] MongoDB starting : pid=3616 port=27017 dbpath=/var/lib/mongodb 64-bit host=lvps176-28-17-95.dedicated.hosteurope.de
Tue Jan 21 00:10:24.599 [initandlisten]
Tue Jan 21 00:10:24.600 [initandlisten] ** WARNING: You are running in OpenVZ. This is known to be broken!!!
Tue Jan 21 00:10:24.600 [initandlisten]
Tue Jan 21 00:10:24.600 [initandlisten] db version v2.4.7
Tue Jan 21 00:10:24.600 [initandlisten] git version: 0161738abf06c1f067b56a465b706efd6f4bf2aa
Tue Jan 21 00:10:24.600 [initandlisten] build info: Linux ip-10-2-29-40 2.6.21.7-2.ec2.v1.2.fc8xen #1 SMP Fri Nov 20 17:48:28 EST 2009 x86_64 BOOST_LIB_VERSION=1_49
Tue Jan 21 00:10:24.600 [initandlisten] allocator: tcmalloc
Tue Jan 21 00:10:24.600 [initandlisten] options: { config: "/etc/mongodb.conf", dbpath: "/var/lib/mongodb", logappend: "true", logpath: "/var/log/mongodb/mongodb.log", replSet: "rs0" }
Tue Jan 21 00:10:24.609 [initandlisten] journal dir=/var/lib/mongodb/journal
Tue Jan 21 00:10:24.609 [initandlisten] recover : no journal files present, no recovery needed
Tue Jan 21 00:10:24.740 [initandlisten] preallocateIsFaster=true 2.38
Tue Jan 21 00:10:24.780 [initandlisten] waiting for connections on port 27017
Tue Jan 21 00:10:24.780 [websvr] admin web console waiting for connections on port 28017
Tue Jan 21 00:10:24.786 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG)
Tue Jan 21 00:10:24.786 [rsStart] replSet info you may need to run replSetInitiate -- rs.initiate() in the shell -- if that is not already done
Tue Jan 21 00:10:27.429 [initandlisten] connection accepted from 127.0.0.1:50602 #1 (1 connection now open)
Tue Jan 21 00:10:34.786 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG)
Tue Jan 21 00:10:40.160 [conn1] replSet replSetInitiate admin command received from client
Tue Jan 21 00:10:40.163 [conn1] replSet replSetInitiate config object parses ok, 1 members specified
Tue Jan 21 00:10:40.164 [conn1] replSet replSetInitiate all members seem up
Tue Jan 21 00:10:40.164 [conn1] ******
Tue Jan 21 00:10:40.164 [conn1] creating replication oplog of size: 24630MB...
Tue Jan 21 00:10:40.165 [FileAllocator] allocating new datafile /var/lib/mongodb/local.1, filling with zeroes...
Tue Jan 21 00:10:40.165 [FileAllocator] creating directory /var/lib/mongodb/_tmp
Tue Jan 21 00:10:40.205 [FileAllocator] done allocating datafile /var/lib/mongodb/local.1, size: 2047MB, took 0.036 secs
Tue Jan 21 00:10:40.206 [FileAllocator] allocating new datafile /var/lib/mongodb/local.2, filling with zeroes...
Tue Jan 21 00:10:40.233 [FileAllocator] done allocating datafile /var/lib/mongodb/local.2, size: 2047MB, took 0.027 secs
Tue Jan 21 00:10:40.234 [FileAllocator] allocating new datafile /var/lib/mongodb/local.3, filling with zeroes...
Tue Jan 21 00:10:40.255 [FileAllocator] done allocating datafile /var/lib/mongodb/local.3, size: 2047MB, took 0.02 secs
Tue Jan 21 00:10:40.256 [FileAllocator] allocating new datafile /var/lib/mongodb/local.4, filling with zeroes...
Tue Jan 21 00:10:40.275 [FileAllocator] done allocating datafile /var/lib/mongodb/local.4, size: 2047MB, took 0.019 secs
Tue Jan 21 00:10:40.276 [FileAllocator] allocating new datafile /var/lib/mongodb/local.5, filling with zeroes...
Tue Jan 21 00:10:40.355 [FileAllocator] done allocating datafile /var/lib/mongodb/local.5, size: 2047MB, took 0.079 secs
Tue Jan 21 00:10:40.356 [FileAllocator] allocating new datafile /var/lib/mongodb/local.6, filling with zeroes...
Tue Jan 21 00:10:40.372 [FileAllocator] done allocating datafile /var/lib/mongodb/local.6, size: 2047MB, took 0.014 secs
Tue Jan 21 00:10:40.372 [FileAllocator] allocating new datafile /var/lib/mongodb/local.7, filling with zeroes...
Tue Jan 21 00:10:40.498 [FileAllocator] done allocating datafile /var/lib/mongodb/local.7, size: 2047MB, took 0.121 secs
Tue Jan 21 00:10:40.499 [FileAllocator] allocating new datafile /var/lib/mongodb/local.8, filling with zeroes...
Tue Jan 21 00:10:40.546 [FileAllocator] done allocating datafile /var/lib/mongodb/local.8, size: 2047MB, took 0.046 secs
Tue Jan 21 00:10:40.546 [conn1] ERROR: mmap private failed with out of memory. (64 bit build)
Tue Jan 21 00:10:40.546 [conn1] Assertion: 13636:file /var/lib/mongodb/local.8 open/create failed in createPrivateMap (look in log for more information)
0xde0151 0xda188b 0xda1dcc 0xa5a63b 0xa5af9a 0xaba3b1 0x8d518d 0x8d5698 0x8d577f 0x8d5a1e 0xabbb00 0xac1429 0xa75908 0xc10af1 0x8dd4da 0x8de04d 0x8df582 0xa81f00 0xa867cc 0x9fa469
/usr/bin/mongod(_ZN5mongo15printStackTraceERSo+0x21) [0xde0151]
/usr/bin/mongod(_ZN5mongo11msgassertedEiPKc+0x9b) [0xda188b]
/usr/bin/mongod() [0xda1dcc]
/usr/bin/mongod(_ZN5mongo8MongoMMF13finishOpeningEv+0x1fb) [0xa5a63b]
/usr/bin/mongod(_ZN5mongo8MongoMMF6createERKSsRyb+0x5a) [0xa5af9a]
/usr/bin/mongod(_ZN5mongo13MongoDataFile4openEPKcib+0x141) [0xaba3b1]
/usr/bin/mongod(_ZN5mongo8Database7getFileEiib+0xbd) [0x8d518d]
/usr/bin/mongod(_ZN5mongo8Database8addAFileEib+0x38) [0x8d5698]
/usr/bin/mongod(_ZN5mongo8Database12suitableFileEPKcibb+0xaf) [0x8d577f]
/usr/bin/mongod(_ZN5mongo8Database11allocExtentEPKcibb+0x9e) [0x8d5a1e]
/usr/bin/mongod(_ZN5mongo13_userCreateNSEPKcRKNS_7BSONObjERSsPb+0x7a0) [0xabbb00]
/usr/bin/mongod(_ZN5mongo12userCreateNSEPKcNS_7BSONObjERSsbPb+0x2b9) [0xac1429]
/usr/bin/mongod(_ZN5mongo11createOplogEv+0xa78) [0xa75908]
/usr/bin/mongod(_ZN5mongo18CmdReplSetInitiate3runERKSsRNS_7BSONObjEiRSsRNS_14BSONObjBuilderEb+0x1da1) [0xc10af1]
/usr/bin/mongod(_ZN5mongo12_execCommandEPNS_7CommandERKSsRNS_7BSONObjEiRSsRNS_14BSONObjBuilderEb+0x3a) [0x8dd4da]
/usr/bin/mongod(_ZN5mongo7Command11execCommandEPS0_RNS_6ClientEiPKcRNS_7BSONObjERNS_14BSONObjBuilderEb+0x71d) [0x8de04d]
/usr/bin/mongod(_ZN5mongo12_runCommandsEPKcRNS_7BSONObjERNS_11_BufBuilderINS_16TrivialAllocatorEEERNS_14BSONObjBuilderEbi+0x5f2) [0x8df582]
/usr/bin/mongod(_ZN5mongo11runCommandsEPKcRNS_7BSONObjERNS_5CurOpERNS_11_BufBuilderINS_16TrivialAllocatorEEERNS_14BSONObjBuilderEbi+0x40) [0xa81f00]
/usr/bin/mongod(_ZN5mongo8runQueryERNS_7MessageERNS_12QueryMessageERNS_5CurOpES1_+0xd7c) [0xa867cc]
/usr/bin/mongod() [0x9fa469]
Tue Jan 21 00:10:40.563 [conn1] replSet replSetInitiate exception: file /var/lib/mongodb/local.8 open/create failed in createPrivateMap (look in log for more information)
Tue Jan 21 00:10:40.563 [conn1] command admin.$cmd command: { replSetInitiate: { _id: "rs0", version: 1.0, members: [ { _id: 0.0, host: "localhost:27017" } ] } } ntoreturn:1 keyUpdates:0 locks(micros) W:401$
Tue Jan 21 00:10:44.787 [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG)
Tue Jan 21 00:10:53.835 [conn1] replSet replSetInitiate admin command received from client
Tue Jan 21 00:10:53.835 [conn1] replSet replSetInitiate config object parses ok, 1 members specified
Tue Jan 21 00:10:53.835 [conn1] replSet replSetInitiate all members seem up
Tue Jan 21 00:10:53.835 [conn1] replSet info saving a newer config version to local.system.replset
Tue Jan 21 00:10:54.564 Invalid access at address: 0x18 from thread: conn1
Tue Jan 21 00:10:54.564 Got signal: 11 (Segmentation fault).
Why MongoDB tries to create 8 files a 2GB? My machine run with 16GB ram, may that is the problem? Because in one line the error "[conn1] ERROR: mmap private failed with out of memory. (64 bit build)" looks like my machine run out of memory. But i only create a replica set with one member and an empty database. Sombody know that bug?

It's not RAM but disk space that is the problem, it's not a bug either.
Tue Jan 21 00:10:40.164 [conn1] creating replication oplog of size: 24630MB...
From MongoDB docs:
The oplog (operations log) is a special capped collection that keeps a
rolling record of all operations that modify the data stored in your
databases. MongoDB applies database operations on the primary and then
records the operations on the primary’s oplog. The secondary members
then copy and apply these operations in an asynchronous process. All
replica set members contain a copy of the oplog, allowing them to
maintain the current state of the database.
For 64-bit Linux, Solaris, FreeBSD, and Windows systems, MongoDB
allocates 5% of the available free disk space to the oplog. If this
amount is smaller than a gigabyte, then MongoDB allocates 1 gigabyte
of space.
Above via http://docs.mongodb.org/manual/core/replica-set-oplog/
Oplog is needed for replication (It's a capped collection of a fixed size) and this is created automatically when you create replica set. oplogSize can be set via configuration options (if you would like to just experiment with the setup and can't free up some more disk space).
Here's a doc on it: http://docs.mongodb.org/manual/reference/configuration-options/#oplogSize
However:
Once the mongod has created the oplog for the first time, changing
oplogSize will not affect the size of the oplog.
via http://docs.mongodb.org/manual/reference/configuration-options/#oplogSize
If you would like to change oplogSize after it has been already created you could use this tutorial: http://docs.mongodb.org/manual/tutorial/change-oplog-size/
However if this is your "playground" installation, it better to delete content of your old MongoDB data directory (/var/lib/mongodb), change config file /etc/mongodb.conf (or pass --oplogSize param to mongo when it starts) and just have a "fresh start" with smaller oplog or point your MongoDB dbpath directory to a place where it has more disk space.

Related

MongoDB suddenly won't start

I have been running MongoDB on a RPI for several weeks, and suddenly it won't start. I know there's a lot of similar questions where correcting the permissions seemed to work. However, this has not worked for me.
I suspect a power off of the RPI has caused this problem, but I have no idea how to solve it.
I get the following terminal output:
$ sudo service mongodb start
[FAIL] Starting database: mongodb failed!
More details are shown with the 'mongod' command:
$ mongod
db level locking enabled: 1
mongod --help for help and startup options
Fri Aug 28 14:04:06
Fri Aug 28 14:04:06 warning: 32-bit servers don't have journaling enabled by default. Please use --journal if you want durability.
Fri Aug 28 14:04:06
Fri Aug 28 14:04:06 [initandlisten] MongoDB starting : pid=4036 port=27017 dbpath=/data/db/ 32-bit host=raspberrypi
Fri Aug 28 14:04:06 [initandlisten]
Fri Aug 28 14:04:06 [initandlisten] ** NOTE: This is a development version (2.1.1) of MongoDB.
Fri Aug 28 14:04:06 [initandlisten] ** Not recommended for production.
Fri Aug 28 14:04:06 [initandlisten]
Fri Aug 28 14:04:06 [initandlisten] ** NOTE: when using MongoDB 32 bit, you are limited to about 2 gigabytes of data
Fri Aug 28 14:04:06 [initandlisten] ** see http://blog.mongodb.org/post/137788967/32-bit-limitations
Fri Aug 28 14:04:06 [initandlisten] ** with --journal, the limit is lower
Fri Aug 28 14:04:06 [initandlisten]
Fri Aug 28 14:04:06 [initandlisten] db version v2.1.1, pdfile version 4.5
Fri Aug 28 14:04:06 [initandlisten] git version: f457ff42ec37f2562d1a5ff06b4d96a861414c94
Fri Aug 28 14:04:06 [initandlisten] build info: Linux piplusraspbian 3.12.22+ #691 PREEMPT Wed Jun 18 18:29:58 BST 2014 armv6l BOOST_LIB_VERSION=1_49
Fri Aug 28 14:04:06 [initandlisten] options: {}
Fri Aug 28 14:04:06 [initandlisten] exception in initAndListen: 10309 Unable to create/open lock file: /data/db/mongod.lock errno:13 Permission denied Is a mongod instance already running?, terminating
Fri Aug 28 14:04:06 dbexit:
Fri Aug 28 14:04:06 [initandlisten] shutdown: going to close listening sockets...
Fri Aug 28 14:04:06 [initandlisten] shutdown: going to flush diaglog...
Fri Aug 28 14:04:06 [initandlisten] shutdown: going to close sockets...
Fri Aug 28 14:04:06 [initandlisten] shutdown: waiting for fs preallocator...
Fri Aug 28 14:04:06 [initandlisten] shutdown: closing all files...
Fri Aug 28 14:04:06 [initandlisten] closeAllFiles() finished
Fri Aug 28 14:04:06 [initandlisten] shutdown: removing fs lock...
Fri Aug 28 14:04:06 [initandlisten] couldn't remove fs lock errno:9 Bad file descriptor
Fri Aug 28 14:04:06 dbexit: really exiting now
The /data/db does exist and seem to have the right permission:
$ ls -l /data
total 8
drwxr-xr-x 2 mongodb mongodb 4096 Aug 28 14:37 db
drwxr-xr-x 4 mongodb mongodb 4096 Aug 28 09:44 db_backup
The content of db has been copied to db_backup to check if that did any change, but it didn't. The db content permissions also seem fine:
$ ls -l /data/db_backup
total 131100
drwxr-xr-x 2 mongodb mongodb 4096 Aug 28 09:44 client-db
-rw------- 1 mongodb mongodb 16777216 Aug 28 09:44 client-db.0
-rw------- 1 mongodb mongodb 33554432 Aug 28 09:44 client-db.1
-rw------- 1 mongodb mongodb 16777216 Aug 28 09:44 client-db.ns
-rwxr-xr-x 1 mongodb mongodb 0 Aug 28 13:52 mongod.lock
drwxr-xr-x 2 mongodb mongodb 4096 Aug 28 09:44 myproject
-rw------- 1 mongodb mongodb 16777216 Aug 28 09:44 myproject.0
-rw------- 1 mongodb mongodb 33554432 Aug 28 09:44 myproject.1
-rw------- 1 mongodb mongodb 16777216 Aug 28 09:44 myproject.ns
I have also tried mongod --repair, but it does not help.
Does anyone have any ideas?
I don't care about the data, as it is all just temporary testing data.
There is a mongod.lock file inside the mongo's data directory, normally is /var/lib/mongodb. The content of this file is the pid of the mongod process.
If the mongod is stopped, the content of this file should be empty. Otherwise the mongod was stopped unexpectedly. To fix and start a new mongod process, assuming that mongo's data directory is /var/lib/mongodb. Just clear the file...
$ echo "" > /var/lib/mongodb/mongod.lock
and start a new mongod process.
$ mongod

Assertion: 13524:out of memory AlignedBuilder

I start MongoDB, using commang: sudo service mongodb start.
OS: Ubuntu 11.04
My log is here:
***** SERVER RESTARTED *****
Mon Apr 29 21:55:23.697 [initandlisten] MongoDB starting : pid=3995 port=27017 dbpath=/srv/mongodb/ 64-bit host=myhost
Mon Apr 29 21:55:23.697 [initandlisten]
Mon Apr 29 21:55:23.697 [initandlisten] ** WARNING: You are running in OpenVZ. This is known to be broken!!!
Mon Apr 29 21:55:23.697 [initandlisten]
Mon Apr 29 21:55:23.697 [initandlisten] db version v2.4.3
Mon Apr 29 21:55:23.697 [initandlisten] git version: fe1743177a5ea03e91e0052fb5e2cb2945f6d95f
Mon Apr 29 21:55:23.697 [initandlisten] build info: Linux ip-10-2-29-40 2.6.21.7-2.ec2.v1.2.fc8xen #1 SMP Fri Nov 20 17:48:28 EST 2009 x86_64 BOOST_LIB_VERSION=1_49
Mon Apr 29 21:55:23.697 [initandlisten] allocator: tcmalloc
Mon Apr 29 21:55:23.697 [initandlisten] options: { config: "/etc/mongodb.conf", dbpath: "/srv/mongodb/", logappend: "true", logpath: "/var/log/mongodb/mongodb.log" }
Mon Apr 29 21:55:23.701 [initandlisten] journal dir=/srv/mongodb/journal
Mon Apr 29 21:55:23.706 [initandlisten] recover : no journal files present, no recovery needed
Mon Apr 29 21:55:24.465 [initandlisten] preallocateIsFaster=true 14.5
Mon Apr 29 21:55:26.100 [initandlisten] preallocateIsFaster=true 32.4
Mon Apr 29 21:55:27.419 [initandlisten] preallocateIsFaster=true 6.1
Mon Apr 29 21:55:27.419 [initandlisten] preallocateIsFaster check took 3.71 secs
Mon Apr 29 21:55:27.419 [initandlisten] preallocating a journal file /srv/mongodb/journal/prealloc.0
Mon Apr 29 21:55:30.011 [initandlisten] File Preallocator Progress: 377487360/1073741824 35%
Mon Apr 29 21:55:48.672 [initandlisten] preallocating a journal file /srv/mongodb/journal/prealloc.1
Mon Apr 29 21:55:51.008 [initandlisten] File Preallocator Progress: 283115520/1073741824 26%
Mon Apr 29 21:55:54.032 [initandlisten] File Preallocator Progress: 754974720/1073741824 70%
Mon Apr 29 21:56:12.230 [initandlisten] preallocating a journal file /srv/mongodb/journal/prealloc.2
Mon Apr 29 21:56:15.006 [initandlisten] File Preallocator Progress: 765460480/1073741824 71%
Mon Apr 29 21:56:37.587 [FileAllocator] allocating new datafile /srv/mongodb/local.ns, filling with zeroes...
Mon Apr 29 21:56:37.587 [FileAllocator] creating directory /srv/mongodb/_tmp
Mon Apr 29 21:56:39.362 [FileAllocator] done allocating datafile /srv/mongodb/local.ns, size: 16MB, took 1.19 secs
Mon Apr 29 21:56:39.362 [FileAllocator] allocating new datafile /srv/mongodb/local.0, filling with zeroes...
Mon Apr 29 21:56:43.413 [FileAllocator] done allocating datafile /srv/mongodb/local.0, size: 64MB, took 4.051 secs
Mon Apr 29 21:56:43.426 [initandlisten] command local.$cmd command: { create: "startup_log", size: 10485760, capped: true } ntoreturn:1 keyUpdates:0 reslen:37 5839ms
Mon Apr 29 21:56:43.426 [initandlisten] waiting for connections on port 27017
Mon Apr 29 21:56:43.426 [journal] Assertion: 13524:out of memory AlignedBuilder
Mon Apr 29 21:56:43.426 [websvr] admin web console waiting for connections on port 28017
0xdcf361 0xd90a1b 0xd8e9ec 0xd8ec80 0x924cd1 0x924ec2 0x91a0b1 0x91bcf9 0x91c0bb 0xe17cb9 0x2ab868e0bd8c 0x2ab869ab6fdd
/usr/bin/mongod(_ZN5mongo15printStackTraceERSo+0x21) [0xdcf361]
/usr/bin/mongod(_ZN5mongo11msgassertedEiPKc+0x9b) [0xd90a1b]
/usr/bin/mongod() [0xd8e9ec]
/usr/bin/mongod(_ZN5mongo14AlignedBuilderC1Ej+0x10) [0xd8ec80]
/usr/bin/mongod(_ZN5mongo3dur7Journal7journalERKNS0_11JSectHeaderERKNS_14AlignedBuilderE+0x221) [0x924cd1]
/usr/bin/mongod(_ZN5mongo3dur14WRITETOJOURNALENS0_11JSectHeaderERNS_14AlignedBuilderE+0x32) [0x924ec2]
/usr/bin/mongod(_ZN5mongo3dur27groupCommitWithLimitedLocksEv+0x141) [0x91a0b1]
/usr/bin/mongod() [0x91bcf9]
/usr/bin/mongod(_ZN5mongo3dur9durThreadEv+0x2eb) [0x91c0bb]
/usr/bin/mongod() [0xe17cb9]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x6d8c) [0x2ab868e0bd8c]
/lib/x86_64-linux-gnu/libc.so.6(clone+0x6d) [0x2ab869ab6fdd]
Mon Apr 29 21:56:43.880 [journal] dbexception in groupCommitLL causing immediate shutdown: 13524 out of memory AlignedBuilder
Mon Apr 29 21:56:43.880 dur1
Mon Apr 29 21:56:43.880 Got signal: 6 (Aborted).
Mon Apr 29 21:56:43.882 Backtrace:
0xdcf361 0x6cf729 0x2ab869a01d80 0x2ab869a01d05 0x2ab869a05ab6 0x9e8de7 0x91a4df 0x91bcf9 0x91c0bb 0xe17cb9 0x2ab868e0bd8c 0x2ab869ab6fdd
/usr/bin/mongod(_ZN5mongo15printStackTraceERSo+0x21) [0xdcf361]
/usr/bin/mongod(_ZN5mongo10abruptQuitEi+0x399) [0x6cf729]
/lib/x86_64-linux-gnu/libc.so.6(+0x33d80) [0x2ab869a01d80]
/lib/x86_64-linux-gnu/libc.so.6(gsignal+0x35) [0x2ab869a01d05]
/lib/x86_64-linux-gnu/libc.so.6(abort+0x186) [0x2ab869a05ab6]
/usr/bin/mongod(_ZN5mongo10mongoAbortEPKc+0x47) [0x9e8de7]
/usr/bin/mongod(_ZN5mongo3dur27groupCommitWithLimitedLocksEv+0x56f) [0x91a4df]
/usr/bin/mongod() [0x91bcf9]
/usr/bin/mongod(_ZN5mongo3dur9durThreadEv+0x2eb) [0x91c0bb]
/usr/bin/mongod() [0xe17cb9]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x6d8c) [0x2ab868e0bd8c]
/lib/x86_64-linux-gnu/libc.so.6(clone+0x6d) [0x2ab869ab6fdd]
How resolve this problem? I tryed reinstall MongoDB, but my problem is stayed.
PS: Why dbpath directory is so big??
root#myhost:~# du -hs /srv/mongodb/
3.1G /srv/mongodb/
You are using OpenVZ as your VM hypervisor. The way OpenVZ manages memory means that MongoDB's memory mapped file structure will NOT work.
You should try getting a VM that uses KVM as it's hypervisor, if possible.
I had this problem when filesystem went to 100% (due verbose logs that where not in /var).
it ended up in bad journal files.
impossible to repair the database (even with --repair)
The only way to recover seems to restart from a backup, and re-importing the proper tables.
to be able to re-start the DB in that situation (for mongodump or because it is prod) is as follow.
You have to start Mongo without journaling (very dangerous, you should leave this mode asap)
set nojournal = true in mongod.conf
then:
rm /mnt/pathto/mongodb/journal/*
rm /mnt/pathto/mongodb/mongod.lock
chown -R mongodb:mongodb /mnt/pathto/mongodb (in case you tried as root "mongod -f /etc/mongodb.conf --repair" as I did)
service mongodb start
DB is up & running, you can do your dumps a restart it WITH journaling (after restoring a backup then the mongodumps)

Mongodb gives error during startup

Whenever I try to play with mongo's interactive shell, it dies:
somekittens#DLserver01:~$ mongo
MongoDB shell version: 2.2.2
connecting to: test
Mon Dec 17 13:14:16 DBClientCursor::init call() failed
Mon Dec 17 13:14:16 Error: Error during mongo startup. :: caused by :: 10276 DBClientBase::findN: transport error: 127.0.0.1:27017 ns: admin.$cmd query: { whatsmyuri: 1 } src/mongo/shell/mongo.js:91
exception: connect failed
I'm able to repair the install (deleting mongodb.lock, etc) and get back to this point, but it'll only die again.
/var/log/mongodb/mongodb.log
Mon Dec 17 13:14:03
Mon Dec 17 13:14:03 warning: 32-bit servers don't have journaling enabled by default. Please use --journal if you want durability.
Mon Dec 17 13:14:03
Mon Dec 17 13:14:03 [initandlisten] MongoDB starting : pid=2674 port=27017 dbpath=/var/lib/mongodb 32-bit host=DLserver01
Mon Dec 17 13:14:03 [initandlisten]
Mon Dec 17 13:14:03 [initandlisten] ** NOTE: when using MongoDB 32 bit, you are limited to about 2 gigabytes of data
Mon Dec 17 13:14:03 [initandlisten] ** see http://blog.mongodb.org/post/137788967/32-bit-limitations
Mon Dec 17 13:14:03 [initandlisten] ** with --journal, the limit is lower
Mon Dec 17 13:14:03 [initandlisten]
Mon Dec 17 13:14:03 [initandlisten] db version v2.2.2, pdfile version 4.5
Mon Dec 17 13:14:03 [initandlisten] git version: d1b43b61a5308c4ad0679d34b262c5af9d664267
Mon Dec 17 13:14:03 [initandlisten] build info: Linux domU-12-31-39-01-70-B4 2.6.21.7-2.fc8xen #1 SMP Fri Feb 15 12:39:36 EST 2008 i686 BOOST_LIB_VERSION=1_49
Mon Dec 17 13:14:03 [initandlisten] options: { config: "/etc/mongodb.conf", dbpath: "/var/lib/mongodb", logappend: "true", logpath: "/var/log/mongodb/mongodb.log" }
Mon Dec 17 13:14:03 [initandlisten] Unable to check for journal files due to: boost::filesystem::basic_directory_iterator constructor: No such file or directory: "/var/lib/mongodb/journal"
Mon Dec 17 13:14:03 [initandlisten] couldn't unlink socket file /tmp/mongodb-27017.sockerrno:1 Operation not permitted skipping
Mon Dec 17 13:14:03 [initandlisten] waiting for connections on port 27017
Mon Dec 17 13:14:03 [websvr] admin web console waiting for connections on port 28017
Mon Dec 17 13:14:16 [initandlisten] connection accepted from 127.0.0.1:57631 #1 (1 connection now open)
Mon Dec 17 13:14:16 Invalid operation at address: 0x819bb23 from thread: conn1
Mon Dec 17 13:14:16 Got signal: 4 (Illegal instruction).
Mon Dec 17 13:14:16 Backtrace:
0x8759eaa 0x817033a 0x81709ff 0x20e40c 0x819bb23 0x854cd54 0x85377d1 0x846b594 0x83e5591 0x83e6c15 0x81902b4 0x8746731 0x49ad4c 0x34ed3e
/usr/bin/mongod(_ZN5mongo15printStackTraceERSo+0x2a) [0x8759eaa]
/usr/bin/mongod(_ZN5mongo10abruptQuitEi+0x3ba) [0x817033a]
/usr/bin/mongod(_ZN5mongo24abruptQuitWithAddrSignalEiP7siginfoPv+0x2af) [0x81709ff]
[0x20e40c]
/usr/bin/mongod(_ZNK5mongo7BSONObj4copyEv+0x33) [0x819bb23]
/usr/bin/mongod(_ZN5mongo11ParsedQuery4initERKNS_7BSONObjE+0x494) [0x854cd54]
/usr/bin/mongod(_ZN5mongo11ParsedQueryC1ERNS_12QueryMessageE+0x91) [0x85377d1]
/usr/bin/mongod(_ZN5mongo8runQueryERNS_7MessageERNS_12QueryMessageERNS_5CurOpES1_+0x34) [0x846b594]
/usr/bin/mongod() [0x83e5591]
/usr/bin/mongod(_ZN5mongo16assembleResponseERNS_7MessageERNS_10DbResponseERKNS_11HostAndPortE+0x3d5) [0x83e6c15]
/usr/bin/mongod(_ZN5mongo16MyMessageHandler7processERNS_7MessageEPNS_21AbstractMessagingPortEPNS_9LastErrorE+0x84) [0x81902b4]
/usr/bin/mongod(_ZN5mongo3pms9threadRunEPNS_13MessagingPortE+0x421) [0x8746731]
/lib/i386-linux-gnu/libpthread.so.0(+0x6d4c) [0x49ad4c]
/lib/i386-linux-gnu/libc.so.6(clone+0x5e) [0x34ed3e]
Connecting using node's shell:
> mdb.open(function(err, db) { console.log(err) });
[Error: failed to connect to [localhost:27017]]
I've searched around for this error and found nothing of use. This is running on a fairly old server (Ubuntu 12.04 32-bit, 640MB RAM, 500MHz P2). How can I fix this?
There is an issue Invalid operation at address: 0x819b263 from thread: TTLMonitor in mongodb jira list. I think that is about your case.
A new server may be the easiest solution, otherwise you have to download the source code, make some modification and compile it youself.

After running mongo --repair database is empy

On my developer machine I've reinstalled ubuntu and just copied files to /data/db from my old install.
After mongo --repair which seemingly went normal (no error messages or such) I see no databases I'm expecting to see.
How can I repair my database?
Here the first part of the log file:
Thu Dec 6 17:55:21 [initandlisten] MongoDB starting : pid=2123 port=27017 dbpath=/data/db/ 64-bit host=sog
Thu Dec 6 17:55:21 [initandlisten] db version v2.2.2, pdfile version 4.5
Thu Dec 6 17:55:21 [initandlisten] git version: d1b43b61a5308c4ad0679d34b262c5af9d664267
Thu Dec 6 17:55:21 [initandlisten] build info: Linux ip-10-2-29-40 2.6.21.7-2.ec2.v1.2.fc8xen #1 SMP Fri Nov 20 17:48:28 EST 2009 x86_64 BOOST_LIB_VERSION=1_49
Thu Dec 6 17:55:21 [initandlisten] options: { repair: true }
Thu Dec 6 17:55:21 [initandlisten] Unable to check for journal files due to: boost::filesystem::basic_directory_iterator constructor: No such file or directory: "/data/db/journal"
Thu Dec 6 17:55:21 [initandlisten] ****
Thu Dec 6 17:55:21 [initandlisten] ****
Thu Dec 6 17:55:21 [initandlisten] need to upgrade database fan-fiction with pdfile version 4.5, new version: 4.5
Thu Dec 6 17:55:21 [initandlisten] starting upgrade
Thu Dec 6 17:55:21 [initandlisten] fan-fiction repairDatabase fan-fiction
Thu Dec 6 17:55:21 [FileAllocator] allocating new datafile /data/db/_tmp_repairDatabase_1/fan-fiction.ns, filling with zeroes...
Thu Dec 6 17:55:21 [FileAllocator] creating directory /data/db/_tmp_repairDatabase_1/_tmp
Thu Dec 6 17:55:21 [FileAllocator] done allocating datafile /data/db/_tmp_repairDatabase_1/fan-fiction.ns, size: 16MB, took 0.001 secs
Thu Dec 6 17:55:21 [FileAllocator] allocating new datafile /data/db/_tmp_repairDatabase_1/fan-fiction.0, filling with zeroes...
Ok, I've copied files under root and forgot to chown them. Now everything works like charm.

MongoDB repair command failed

Previously i was running out of disk space, and mongodb stopped working. Then I have increased disk size but mongodb does not start working.
Though i have journaling enabled, i have execute following command
sudo -u mongodb mongod --dbpath /var/lib/mongodb/ --repair
But this repair command got an exception and stop repairing and then exit.
Fri Nov 30 13:29:36 [initandlisten] build index bd_production.news { _id: 1 }
Fri Nov 30 13:29:36 [initandlisten] fastBuildIndex dupsToDrop:0
Fri Nov 30 13:29:36 [initandlisten] build index done. scanned 2549 total
records. 0.008 secs
Fri Nov 30 13:29:36 [initandlisten] bd_production.change_sets
Assertion failure isOk() src/mongo/db/pdfile.h 360
0x879d86a 0x85a9835 0x85e441e 0x84caa02 0x84c7d19 0x8229b5a 0x822bfd8
0x875bd51 0x875f0c7 0x8760df4 0x83e6523 0x83b6c3b 0x8753b07 0x83b92bf
0x8827ab7 0x882a53b 0x882d4bf 0x882d691 0x85ed280 0x81719dc
mongod(_ZN5mongo15printStackTraceERSo+0x2a) [0x879d86a]
mongod(_ZN5mongo10logContextEPKc+0xa5) [0x85a9835]
...
...
...
... some error msg
Fri Nov 30 13:29:36 [initandlisten] assertion 0 assertion
src/mongo/db/pdfile.h:360 ns:bd_production.change_sets
query:{}
Fri Nov 30 13:29:36 [initandlisten] problem detected during query over
bd_production.change_sets : { $err: "assertion
src/mongo/db/pdfile.h:360" }
Fri Nov 30 13:29:36 [initandlisten] query
bd_production.change_sets ntoreturn:0 keyUpdates:0 exception:
assertion src/mongo/db/pdfile.h:360 reslen:71 197ms
Fri Nov 30 13:29:36 [initandlisten] exception in initAndListen: 13106
nextSafe(): { $err: "assertion src/mongo/db/pdfile.h:360" }, terminating
Fri Nov 30 13:29:36 dbexit:
...
...
news collection is repaired successfully but the 'change_set' does not repair successfully.
How can i repair that particular collection (change_set) or database ?
UPDATE:
When I run mongodump with --repair for that change_set collection i got following error message:
Tue Dec 4 10:45:21 [tools] backwards extent pass
Tue Dec 4 10:45:21 [tools] extent loc: 5:1181e000
Tue Dec 4 10:45:21 [FileAllocator] allocating new datafile /home/suvankar/dd/bd_production.5, filling with zeroes...
Tue Dec 4 10:45:21 [FileAllocator] creating directory /home/suvankar/dd/_tmp
Tue Dec 4 10:45:21 [FileAllocator] done allocating datafile /home/suvankar/dd/bd_production.5, size: 511MB, took 0.042 secs
Tue Dec 4 10:45:21 [tools] warning: Extent not ok magic: 0 going to try to continue
Tue Dec 4 10:45:21 [tools] length:0
Tue Dec 4 10:45:21 [tools] ERROR: offset is 0 for record which should be impossible
Tue Dec 4 10:45:21 [tools] wrote 1 documents
Tue Dec 4 10:45:21 [tools] extent loc: 0:0
Tue Dec 4 10:45:21 [tools] ERROR: invalid extent ofs: 0
Tue Dec 4 10:45:21 [tools] 5 objects
Tue Dec 4 10:45:21 dbexit:
Tue Dec 4 10:45:21 [tools] shutdown: going to close listening sockets...
Tue Dec 4 10:45:21 [tools] shutdown: going to flush diaglog...
Tue Dec 4 10:45:21 [tools] shutdown: going to close sockets...
Tue Dec 4 10:45:21 [tools] shutdown: waiting for fs preallocator...
Tue Dec 4 10:45:21 [tools] shutdown: lock for final commit...
If mongod with repair is not doing it, then it is running into a level of corruption that it can't fix or work around in terms of having a valid and correct set of database files to start up.
You can run mongodump with repair, which is more aggressive in terms of trying to get around the corruption, and is not starting a mongod instance (hence does not require the files to be correct in order to proceed).
mongodump --repair --dbpath /var/lib/mongodb/ <other options here>
Be aware though, that because of the way it attempts to route around the corruption, you may end up with multiple copies of a document. With how mongorestore works this is not an issue, but depending on the level of corruption you can end up with dump files far larger than you would expect. In a very extreme case, I once saw 10x data produced, though that was the exception rather than the rule.
Once you have dumped everything out to your satisfaction, start mongod clean and re-import to get back to a good state.