Assertion: 13524:out of memory AlignedBuilder - mongodb

Question

I start MongoDB, using commang: sudo service mongodb start.
OS: Ubuntu 11.04
My log is here:
***** SERVER RESTARTED *****
Mon Apr 29 21:55:23.697 [initandlisten] MongoDB starting : pid=3995 port=27017 dbpath=/srv/mongodb/ 64-bit host=myhost
Mon Apr 29 21:55:23.697 [initandlisten]
Mon Apr 29 21:55:23.697 [initandlisten] ** WARNING: You are running in OpenVZ. This is known to be broken!!!
Mon Apr 29 21:55:23.697 [initandlisten]
Mon Apr 29 21:55:23.697 [initandlisten] db version v2.4.3
Mon Apr 29 21:55:23.697 [initandlisten] git version: fe1743177a5ea03e91e0052fb5e2cb2945f6d95f
Mon Apr 29 21:55:23.697 [initandlisten] build info: Linux ip-10-2-29-40 2.6.21.7-2.ec2.v1.2.fc8xen #1 SMP Fri Nov 20 17:48:28 EST 2009 x86_64 BOOST_LIB_VERSION=1_49
Mon Apr 29 21:55:23.697 [initandlisten] allocator: tcmalloc
Mon Apr 29 21:55:23.697 [initandlisten] options: { config: "/etc/mongodb.conf", dbpath: "/srv/mongodb/", logappend: "true", logpath: "/var/log/mongodb/mongodb.log" }
Mon Apr 29 21:55:23.701 [initandlisten] journal dir=/srv/mongodb/journal
Mon Apr 29 21:55:23.706 [initandlisten] recover : no journal files present, no recovery needed
Mon Apr 29 21:55:24.465 [initandlisten] preallocateIsFaster=true 14.5
Mon Apr 29 21:55:26.100 [initandlisten] preallocateIsFaster=true 32.4
Mon Apr 29 21:55:27.419 [initandlisten] preallocateIsFaster=true 6.1
Mon Apr 29 21:55:27.419 [initandlisten] preallocateIsFaster check took 3.71 secs
Mon Apr 29 21:55:27.419 [initandlisten] preallocating a journal file /srv/mongodb/journal/prealloc.0
Mon Apr 29 21:55:30.011 [initandlisten] File Preallocator Progress: 377487360/1073741824 35%
Mon Apr 29 21:55:48.672 [initandlisten] preallocating a journal file /srv/mongodb/journal/prealloc.1
Mon Apr 29 21:55:51.008 [initandlisten] File Preallocator Progress: 283115520/1073741824 26%
Mon Apr 29 21:55:54.032 [initandlisten] File Preallocator Progress: 754974720/1073741824 70%
Mon Apr 29 21:56:12.230 [initandlisten] preallocating a journal file /srv/mongodb/journal/prealloc.2
Mon Apr 29 21:56:15.006 [initandlisten] File Preallocator Progress: 765460480/1073741824 71%
Mon Apr 29 21:56:37.587 [FileAllocator] allocating new datafile /srv/mongodb/local.ns, filling with zeroes...
Mon Apr 29 21:56:37.587 [FileAllocator] creating directory /srv/mongodb/_tmp
Mon Apr 29 21:56:39.362 [FileAllocator] done allocating datafile /srv/mongodb/local.ns, size: 16MB, took 1.19 secs
Mon Apr 29 21:56:39.362 [FileAllocator] allocating new datafile /srv/mongodb/local.0, filling with zeroes...
Mon Apr 29 21:56:43.413 [FileAllocator] done allocating datafile /srv/mongodb/local.0, size: 64MB, took 4.051 secs
Mon Apr 29 21:56:43.426 [initandlisten] command local.$cmd command: { create: "startup_log", size: 10485760, capped: true } ntoreturn:1 keyUpdates:0 reslen:37 5839ms
Mon Apr 29 21:56:43.426 [initandlisten] waiting for connections on port 27017
Mon Apr 29 21:56:43.426 [journal] Assertion: 13524:out of memory AlignedBuilder
Mon Apr 29 21:56:43.426 [websvr] admin web console waiting for connections on port 28017
0xdcf361 0xd90a1b 0xd8e9ec 0xd8ec80 0x924cd1 0x924ec2 0x91a0b1 0x91bcf9 0x91c0bb 0xe17cb9 0x2ab868e0bd8c 0x2ab869ab6fdd
/usr/bin/mongod(_ZN5mongo15printStackTraceERSo+0x21) [0xdcf361]
/usr/bin/mongod(_ZN5mongo11msgassertedEiPKc+0x9b) [0xd90a1b]
/usr/bin/mongod() [0xd8e9ec]
/usr/bin/mongod(_ZN5mongo14AlignedBuilderC1Ej+0x10) [0xd8ec80]
/usr/bin/mongod(_ZN5mongo3dur7Journal7journalERKNS0_11JSectHeaderERKNS_14AlignedBuilderE+0x221) [0x924cd1]
/usr/bin/mongod(_ZN5mongo3dur14WRITETOJOURNALENS0_11JSectHeaderERNS_14AlignedBuilderE+0x32) [0x924ec2]
/usr/bin/mongod(_ZN5mongo3dur27groupCommitWithLimitedLocksEv+0x141) [0x91a0b1]
/usr/bin/mongod() [0x91bcf9]
/usr/bin/mongod(_ZN5mongo3dur9durThreadEv+0x2eb) [0x91c0bb]
/usr/bin/mongod() [0xe17cb9]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x6d8c) [0x2ab868e0bd8c]
/lib/x86_64-linux-gnu/libc.so.6(clone+0x6d) [0x2ab869ab6fdd]
Mon Apr 29 21:56:43.880 [journal] dbexception in groupCommitLL causing immediate shutdown: 13524 out of memory AlignedBuilder
Mon Apr 29 21:56:43.880 dur1
Mon Apr 29 21:56:43.880 Got signal: 6 (Aborted).
Mon Apr 29 21:56:43.882 Backtrace:
0xdcf361 0x6cf729 0x2ab869a01d80 0x2ab869a01d05 0x2ab869a05ab6 0x9e8de7 0x91a4df 0x91bcf9 0x91c0bb 0xe17cb9 0x2ab868e0bd8c 0x2ab869ab6fdd
/usr/bin/mongod(_ZN5mongo15printStackTraceERSo+0x21) [0xdcf361]
/usr/bin/mongod(_ZN5mongo10abruptQuitEi+0x399) [0x6cf729]
/lib/x86_64-linux-gnu/libc.so.6(+0x33d80) [0x2ab869a01d80]
/lib/x86_64-linux-gnu/libc.so.6(gsignal+0x35) [0x2ab869a01d05]
/lib/x86_64-linux-gnu/libc.so.6(abort+0x186) [0x2ab869a05ab6]
/usr/bin/mongod(_ZN5mongo10mongoAbortEPKc+0x47) [0x9e8de7]
/usr/bin/mongod(_ZN5mongo3dur27groupCommitWithLimitedLocksEv+0x56f) [0x91a4df]
/usr/bin/mongod() [0x91bcf9]
/usr/bin/mongod(_ZN5mongo3dur9durThreadEv+0x2eb) [0x91c0bb]
/usr/bin/mongod() [0xe17cb9]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x6d8c) [0x2ab868e0bd8c]
/lib/x86_64-linux-gnu/libc.so.6(clone+0x6d) [0x2ab869ab6fdd]
How resolve this problem? I tryed reinstall MongoDB, but my problem is stayed.
PS: Why dbpath directory is so big??
root#myhost:~# du -hs /srv/mongodb/
3.1G /srv/mongodb/

Answer 1

You are using OpenVZ as your VM hypervisor. The way OpenVZ manages memory means that MongoDB's memory mapped file structure will NOT work.
You should try getting a VM that uses KVM as it's hypervisor, if possible.

Answer 2

I had this problem when filesystem went to 100% (due verbose logs that where not in /var).
it ended up in bad journal files.
impossible to repair the database (even with --repair)
The only way to recover seems to restart from a backup, and re-importing the proper tables.
to be able to re-start the DB in that situation (for mongodump or because it is prod) is as follow.
You have to start Mongo without journaling (very dangerous, you should leave this mode asap)
set nojournal = true in mongod.conf
then:
rm /mnt/pathto/mongodb/journal/*
rm /mnt/pathto/mongodb/mongod.lock
chown -R mongodb:mongodb /mnt/pathto/mongodb (in case you tried as root "mongod -f /etc/mongodb.conf --repair" as I did)
service mongodb start
DB is up & running, you can do your dumps a restart it WITH journaling (after restoring a backup then the mongodumps)