We experienced some strange thing in our mongodb gridfs platform.
The platform actually is a bi Xeon E5 (bi quad core) with 128GB of memory, running on freebsd 9 with a zfs pool dedicated for mongodb.
[root#mongofile1 ~]# uname -sr
FreeBSD 9.1-RELEASE
our /boot/loader.conf
vfs.zfs.arc_min="2048M"
vfs.zfs.arc_max="7680M"
vm.kmem_size_max="16G"
vm.kmem_size="12G"
vfs.zfs.prefetch_disable="1"
kern.ipc.nmbclusters="32768"
/etc/sysctl.conf
net.inet.tcp.msl=15000
net.inet.tcp.keepidle=300000
kern.ipc.nmbclusters=32768
kern.ipc.maxsockbuf=2097152
kern.ipc.somaxconn=8192
kern.maxfiles=65536
kern.maxfilesperproc=32768
net.inet.tcp.delayed_ack=0
net.inet.tcp.sendspace=65535
net.inet.udp.recvspace=65535
net.inet.udp.maxdgram=57344
net.local.stream.recvspace=65535
net.local.stream.sendspace=65535
we follow the recommandation for the ulimit :
[root#mongofile1 ~]# su - mongodb
$ ulimit -a
cpu time (seconds, -t) unlimited
file size (512-blocks, -f) unlimited
data seg size (kbytes, -d) 33554432
stack size (kbytes, -s) 524288
core file size (512-blocks, -c) unlimited
max memory size (kbytes, -m) unlimited
locked memory (kbytes, -l) unlimited
max user processes (-u) 5547
open files (-n) 32768
virtual mem size (kbytes, -v) unlimited
swap limit (kbytes, -w) unlimited
sbsize (bytes, -b) unlimited
pseudo-terminals (-p) unlimited
This server have a twin( same config exactly) for ReplSet in other datacenter and we have a virtualized arbiter.
Some time, almost 3 days, the process of mongodb exit
The problem begin with
Fri Nov 8 11:27:31.741 [conn774697] end connection 192.168.10.162:47963 (23 connections now open)
Fri Nov 8 11:27:31.770 [initandlisten] can't create new thread, closing connection
Fri Nov 8 11:27:31.771 [rsHealthPoll] replSet member mongofile2:27017 is now in state DOWN
Fri Nov 8 11:27:31.774 [initandlisten] connection accepted from 192.168.10.162:47968 #774702 (20 connections now open)
Fri Nov 8 11:27:31.774 [initandlisten] connection accepted from 192.168.10.161:28522 #774703 (21 connections now open)
Fri Nov 8 11:27:31.774 [initandlisten] connection accepted from 192.168.10.164:15406 #774704 (22 connections now open)
Fri Nov 8 11:27:31.774 [initandlisten] connection accepted from 192.168.10.163:25750 #774705 (23 connections now open)
Fri Nov 8 11:27:31.810 [initandlisten] connection accepted from 192.168.10.182:20779 #774706 (24 connections now open)
Fri Nov 8 11:27:31.855 [initandlisten] connection accepted from 192.168.10.161:28524 #774707 (25 connections now open)
Fri Nov 8 11:27:31.869 [initandlisten] connection accepted from 192.168.10.182:20786 #774708 (26 connections now open)
and after many "can create new thread"
[root#mongofile1 /usr/mongodb]# tail -n 15000 mongod.log.old |grep "create new thread"|wc
5020 55220 421680
and finish by a magnificent
Fri Nov 8 11:30:22.333 [rsMgr] replSet warning caught unexpected exception in electSelf()
pure virtual method called
Fri Nov 8 11:30:22.333 Got signal: 6 (Abort trap: 6).
Fri Nov 8 11:30:22.337 Backtrace:
0x599efc 0x8035cb516
0x599efc <_ZN5mongo10abruptQuitEi+988> at /usr/local/bin/mongod
0x8035cb516 <_pthread_sigmask+918> at /lib/libthr.so.3
Extract of mongodb from top
78126 mongodb 77 20 0 1253G 1449M sbwait 0 0:20 0.00% mongod
If i restart the process when it crash, the problem is fixed for almost 3 days ;-) ( Windows Method is not the solution ...)
Do you already see that ?
The fact that the error message says "pure virtual method called" would indicate that this is at least a bug in the mongodb code. The code that generates that message is here, I think.
Related
I am running MongoDB inside the Docker, a linux container. Based on this Dockerfile.
I am using the latest mongo release, 2.6.4 installed on Ubuntu 14.04 64bits. I am able to start the mongod, I do have a /data/db folder in the container and the host machine ( even used as mounted folder, it is there in host machine. ). Do have enough memory, but I was getting errors.
From the logs below, you could clear tell that I am trying to connect the db twice, the connection won't be open for some reasons. Someone could help me?
Note: the machine only have git and mongo installed. Do mongo need anything else as dependencies?
2014-09-04T05:14:50.645+0000 [initandlisten] MongoDB starting : pid=1 port=27017 dbpath=/data/db 64-bit host=f00d8205ca65
2014-09-04T05:14:50.645+0000 [initandlisten] db version v2.6.4
2014-09-04T05:14:50.645+0000 [initandlisten] git version: 3a830be0eb92d772aa855ebb711ac91d658ee910
2014-09-04T05:14:50.645+0000 [initandlisten] build info: Linux build7.nj1.10gen.cc 2.6.32-431.3.1.el6.x86_64 #1 SMP Fri Jan 3 21:39:27 UTC 2014 x86_64 BOOST_LIB_VERSION=1_49
2014-09-04T05:14:50.645+0000 [initandlisten] allocator: tcmalloc
2014-09-04T05:14:50.646+0000 [initandlisten] options: {}
2014-09-04T05:14:50.648+0000 [initandlisten] journal dir=/data/db/journal
2014-09-04T05:14:50.648+0000 [initandlisten] recover : no journal files present, no recovery needed
2014-09-04T05:14:50.703+0000 [FileAllocator] allocating new datafile /data/db/local.ns, filling with zeroes...
2014-09-04T05:14:50.703+0000 [FileAllocator] creating directory /data/db/_tmp
2014-09-04T05:14:50.705+0000 [FileAllocator] done allocating datafile /data/db/local.ns, size: 16MB, took 0 secs
2014-09-04T05:14:50.708+0000 [FileAllocator] allocating new datafile /data/db/local.0, filling with zeroes...
2014-09-04T05:14:50.709+0000 [FileAllocator] done allocating datafile /data/db/local.0, size: 64MB, took 0.001 secs
2014-09-04T05:14:50.711+0000 [initandlisten] build index on: local.startup_log properties: { v: 1, key: { _id: 1 }, name: "_id_", ns: "local.startup_log" }
2014-09-04T05:14:50.711+0000 [initandlisten] added index to empty collection
2014-09-04T05:14:50.712+0000 [initandlisten] waiting for connections on port 27017
2014-09-04T05:15:50.721+0000 [clientcursormon] mem (MB) res:36 virt:341
2014-09-04T05:15:50.721+0000 [clientcursormon] mapped (incl journal view):160
2014-09-04T05:15:50.721+0000 [clientcursormon] connections:0
2014-09-04T05:15:57.976+0000 [initandlisten] connection accepted from 172.17.42.1:56593 #1 (1 connection now open)
2014-09-04T05:15:57.976+0000 [conn1] end connection 172.17.42.1:56593 (0 connections now open)
Someone said it could be less memory to run mongo. after running the momery check, I do still have a good amount of memory to run Mongo.
[ root#512ea0e1096b:/data ]$ free -m
total used free shared buffers cached
Mem: 2001 436 1564 0 24 276
-/+ buffers/cache: 136 1865
Swap: 2991 0 2991
I'm having issues connecting to my mongo shell. I can connect successfully when I remove all directives from my IPTables. CentOS 6.5, mongod is running. Here is the error I'm getting:
[samir#core ~]$ mongo 127.0.0.1:27017
MongoDB shell version: 2.4.9
connecting to: 127.0.0.1:27017/test
Wed Mar 12 00:42:12.193 Error: couldn't connect to server 127.0.0.1:27017 at src/mongo/shell/mongo.js:147
exception: connect failed
I also tried to start with just mongo.
Mongod is running:
[samir#core ~]$ sudo service mongod start
Starting mongod: about to fork child process, waiting until server is ready for connections.
forked process: 2104
all output going to: /var/log/mongo/mongod.log
child process started successfully, parent exiting
[ OK ]
The mongo log does not give any pointed either:
***** SERVER RESTARTED *****
Wed Mar 12 00:44:30.634 [initandlisten] MongoDB starting : pid=2104 port=27017 dbpath=/var/lib/mongo 64-bit host=core.datafit.io
Wed Mar 12 00:44:30.634 [initandlisten] db version v2.4.9
Wed Mar 12 00:44:30.634 [initandlisten] git version: 52fe0d21959e32a5bdbecdc62057db386e4e029c
Wed Mar 12 00:44:30.634 [initandlisten] build info: Linux ip-10-2-29-40 2.6.21.7-2.ec2.v1.2.fc8xen #1 SMP Fri Nov 20 17:48:28 EST 2009 x86_64 BOOST_LIB_VERSION=1_49
Wed Mar 12 00:44:30.634 [initandlisten] allocator: tcmalloc
Wed Mar 12 00:44:30.634 [initandlisten] options: { config: "/etc/mongod.conf", dbpath: "/var/lib/mongo", fork: "true", logappend: "true", logpath: "/var/log/mongo/mongod.log", pidfilepath: "/var/run/mongodb/mongod.pid" }
Wed Mar 12 00:44:30.639 [initandlisten] journal dir=/var/lib/mongo/journal
Wed Mar 12 00:44:30.640 [initandlisten] recover : no journal files present, no recovery needed
Wed Mar 12 00:44:30.725 [FileAllocator] allocating new datafile /var/lib/mongo/local.ns, filling with zeroes...
Wed Mar 12 00:44:30.725 [FileAllocator] creating directory /var/lib/mongo/_tmp
Wed Mar 12 00:44:30.738 [FileAllocator] done allocating datafile /var/lib/mongo/local.ns, size: 16MB, took 0.011 secs
Wed Mar 12 00:44:30.738 [FileAllocator] allocating new datafile /var/lib/mongo/local.0, filling with zeroes...
Wed Mar 12 00:44:30.753 [FileAllocator] done allocating datafile /var/lib/mongo/local.0, size: 64MB, took 0.014 secs
Wed Mar 12 00:44:30.755 [initandlisten] waiting for connections on port 27017
Wed Mar 12 00:44:30.756 [websvr] admin web console waiting for connections on port 28017
My IPTables looks like this:
Chain INPUT (policy DROP 49 packets, 3315 bytes)
pkts bytes target prot opt in out source destination
1665 130K ACCEPT all -- eth0 any anywhere anywhere state RELATED,ESTABLISHED
2 128 ACCEPT tcp -- any any anywhere anywhere tcp dpt:ssh
0 0 ACCEPT tcp -- any any anywhere anywhere tcp dpt:http
0 0 ACCEPT tcp -- any any anywhere anywhere tcp dpt:https
18 1080 ACCEPT tcp -- any any anywhere anywhere tcp dpt:27017
0 0 ACCEPT tcp -- any any anywhere anywhere tcp dpt:28017
Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
pkts bytes target prot opt in out source destination
Chain OUTPUT (policy ACCEPT 987 packets, 156K bytes)
pkts bytes target prot opt in out source destination
If I set the INPUT directive to ACCEPT, mongo shell works:
[samir#core db]$ sudo mongo
MongoDB shell version: 2.4.9
connecting to: test
Welcome to the MongoDB shell.
For interactive help, type "help".
For more comprehensive documentation, see
http://docs.mongodb.org/
Questions? Try the support group
http://groups.google.com/group/mongodb-user
>
Netstat -pln | rep 27017:
[samir#core db]$ sudo netstat -pln | grep 27017
tcp 0 0 0.0.0.0:27017 0.0.0.0:* LISTEN 2178/mongod
unix 2 [ ACC ] STREAM LISTENING 12125 2178/mongod /tmp/mongodb-27017.sock
Something must be wrong with my iptables input policies. Any ideas?
You migth want to try, substituting the line
iptables -A INPUT -s 127.0.0.1 -j ACCEPT
With
iptables -A INPUT -i lo -j ACCEPT
I am getting this error while installing the mongodb on my windows7 system.
C:>mongodb\bin\mongo.exe
MongoDB shell version: 2.4.3
connecting to: test
Server has startup warnings:
Sun May 12 16:19:51.162 [initandlisten]
Sun May 12 16:19:51.162 [initandlisten] ** NOTE: This is a 32 bit MongoDB binary
.
Sun May 12 16:19:51.163 [initandlisten] ** 32 bit builds are limited to le
ss than 2GB of data (or less with --journal).
Sun May 12 16:19:51.163 [initandlisten] ** Note that journaling defaults t
o off for 32 bit and is currently off.
Sun May 12 16:19:51.163 [initandlisten] ** See http://dochub.mongodb.org/core/32bit
Sun May 12 16:19:51.164 [initandlisten]
I didn't find any solution for this. Any help? Thank you
Where is your error?It's just a warning that you shouldn't use Mongo on 32-bit machines because then the limit of storage size is 2 GB. Just read the article in the link you already have.
I am wondering about the MongoDB memory consumption. I have read the corresponding manual sections and the other questions on the topic, but I think this situation is different. May I ask you for your advice?
This is the error from the DB log file:
Fri Oct 26 20:34:00 [conn1] ERROR: mmap private failed with out of memory. (64 bit build)
Fri Oct 26 20:34:00 [conn1] Assertion: 13636:file /docdata/mongodb/data/xxx_letters.5 open/create failed in createPrivateMap (look in log for more information)
These are the data files:
total 4.0G
drwxr-xr-x 2 mongodb mongodb 4.0K 2012-10-26 20:21 journal
-rw------- 1 mongodb mongodb 64M 2012-10-25 19:34 xxx_letters.0
-rw------- 1 mongodb mongodb 128M 2012-10-20 22:10 xxx_letters.1
-rw------- 1 mongodb mongodb 256M 2012-10-24 09:10 xxx_letters.2
-rw------- 1 mongodb mongodb 512M 2012-10-26 10:04 xxx_letters.3
-rw------- 1 mongodb mongodb 1.0G 2012-10-26 19:56 xxx_letters.4
-rw------- 1 mongodb mongodb 2.0G 2012-10-03 11:32 xxx_letters.5
-rw------- 1 mongodb mongodb 16M 2012-10-26 19:56 xxx_letters.ns
This is the output of free -tm:
total used free shared buffers cached
Mem: 3836 3804 31 0 65 2722
-/+ buffers/cache: 1016 2819
Swap: 4094 513 3581
Total: 7930 4317 3612
Is it really necessary to have enough system memory so that the largest data files fit in? Why grow the files that much? (From the sequence shown above, I expect the next file to be 4GB.) I'll try to extend the RAM, but data will eventually grow even more. Or maybe this is not a memory problem at all?
I have got a 64 bit Linux system and use the 64 bit MongoDB 2.0.7-rc1. There is plenty of disk space, the CPU load is 0.0. This is uname -a:
Linux xxx 2.6.32.54-0.3-default #1 SMP 2012-01-27 17:38:56 +0100 x86_64 x86_64 x86_64 GNU/Linux
ulimit -a solved the mystery:
core file size (blocks, -c) 1
data seg size (kbytes, -d) unlimited
scheduling priority (-e) 0
file size (blocks, -f) unlimited
pending signals (-i) 30619
max locked memory (kbytes, -l) 64
max memory size (kbytes, -m) 3338968
open files (-n) 1024
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority (-r) 0
stack size (kbytes, -s) 8192
cpu time (seconds, -t) unlimited
max user processes (-u) 30619
virtual memory (kbytes, -v) 6496960
file locks (-x) unlimited
It worked after setting max memory size and virtual memory to unlimited and restarting everything. BTW, the next file had again 2GB.
Sorry for bothering you, but I was desperate. Maybe this helps somebody "googling" with a similar problem.
I'm running an Ubuntu 12.04 server instance on VirtualBox on a OS X 10.7 host (MacBook Air 13"). The instance is configured with 512mb ram. The disk image is set to 8GB (dynamically allocated if that makes any difference) and is using 2.6GB.
I've just installed MongoDB from the 10gen repo as described here: http://docs.mongodb.org/manual/tutorial/install-mongodb-on-debian-or-ubuntu-linux/
I haven't changed any of the default settings as yet.
The mongod process is now running at 15-20% cpu constantly, before I've connected anything to it. I wouldn't normally have checked but it's now causing the fans to stay on whenever the VM is running.
Restarting mongodb makes no difference. Restarting the VM makes no difference.
The log is as follows:
Thu Jun 21 16:52:14 [initandlisten] MongoDB starting : pid=1737 port=27017 dbpath=/var/lib/mongodb 64-bit host=devbox
Thu Jun 21 16:52:14 [initandlisten] db version v2.0.6, pdfile version 4.5
Thu Jun 21 16:52:14 [initandlisten] git version: e1c0cbc25863f6356aa4e31375add7bb49fb05bc
Thu Jun 21 16:52:14 [initandlisten] build info: Linux ip-10-110-9-236 2.6.21.7-2.ec2.v1.2.fc8xen #1 SMP Fri Nov 20 17:48:28 EST 2009 x86_64 BOOST_LIB_VERSION=1_41
Thu Jun 21 16:52:14 [initandlisten] options: { config: "/etc/mongodb.conf", dbpath: "/var/lib/mongodb", logappend: "true", logpath: "/var/log/mongodb/mongodb.log" }
Thu Jun 21 16:52:14 [initandlisten] journal dir=/var/lib/mongodb/journal
Thu Jun 21 16:52:14 [initandlisten] recover : no journal files present, no recovery needed
Thu Jun 21 16:52:14 [initandlisten] waiting for connections on port 27017
Thu Jun 21 16:52:14 [websvr] admin web console waiting for connections on port 28017
I'm assuming 15-20% cpu usage at idle isn't normal, so I'd like to know if anyone has any suggestions on what to try to fix it?
edit
I've now converted the disk image to a fixed size VDI and it's made no difference, still at 15-20% CPU usage on the guest.
This is a bug in VirtualBox on OSX as far as I can tell, I switched to Parallels and haven't had an issue since.