We have a 3-node replicaSet that periodically crashes and is unable to recover. Looking through our PRIMARY server's mongod.log file, I see multiple errors. I'm not sure where to begin or even what to include in this post but I'll start with the errors I am receiving. If I'm missing something, please let me know and I'll edit the post and include it. Can anyone shed any light on why this is happening?
Thu Feb 27 14:09:47.790 [rsSyncNotifier] replset tracking exception: exception: 10278 dbclient error communicating with server: mongos2i.hostname.com:27017
Thu Feb 27 14:09:47.790 [rsBackgroundSync] replSet sync source problem: 10278 dbclient error communicating with server: mongos2i.hostname.com:27017
Thu Feb 27 14:09:47.790 [rsBackgroundSync] replSet syncing to: mongos2i.hostname.com:27017
Thu Feb 27 14:09:47.791 [rsBackgroundSync] repl: couldn't connect to server mongos2i.hostname.com:27017
Thu Feb 27 14:09:47.792 [conn152] end connection xx.xxx.xxx.107:43904 (71 connections now open)
Thu Feb 27 14:09:48.077 [rsHealthPoll] DBClientCursor::init call() failed
Thu Feb 27 14:09:48.077 [rsHealthPoll] replset info mongos2i.hostname.com:27017 heartbeat failed, retrying
Thu Feb 27 14:09:48.078 [rsHealthPoll] replSet info mongos2i.hostname.com:27017 is down (or slow to respond):
Thu Feb 27 14:09:48.078 [rsHealthPoll] replSet member mongos2i.hostname.com:27017 is now in state DOWN
Thu Feb 27 14:09:48.080 [rsMgr] not electing self, mongos1i.hostname.com:27017 would veto with 'mongom1i.hostname.com:27017 is trying to elect itself but mongos2i.hostname.com:27017 is already primary and more up-to-date'
Thu Feb 27 14:09:49.079 [conn153] replSet info voting yea for mongos1i.hostname.com:27017 (1)
Thu Feb 27 14:09:50.080 [rsHealthPoll] replSet member mongos1i.hostname.com:27017 is now in state PRIMARY
Thu Feb 27 14:09:50.081 [rsHealthPoll] replSet member mongos2i.hostname.com:27017 is up
Thu Feb 27 14:09:50.082 [initandlisten] connection accepted from xx.xxx.xxx.107:43907 #154 (72 connections now open)
Thu Feb 27 14:09:50.082 [conn154] end connection xx.xxx.xxx.107:43907 (71 connections now open)
Thu Feb 27 14:09:50.086 [initandlisten] connection accepted from xx.xxx.xxx.107:43909 #155 (72 connections now open)
Thu Feb 27 14:09:50.792 [rsBackgroundSync] replSet syncing to: mongos1i.hostname.com:27017
Thu Feb 27 14:09:52.082 [rsHealthPoll] replSet member mongos2i.hostname.com:27017 is now in state SECONDARY
Thu Feb 27 14:10:04.090 [conn155] end connection xx.xxx.xxx.107:43909 (71 connections now open)
Thu Feb 27 14:10:04.091 [initandlisten] connection accepted from xx.xxx.xxx.107:43913 #156 (72 connections now open)
Thu Feb 27 14:10:10.731 [conn153] end connection xx.xxx.xxx.97:52297 (71 connections now open)
Thu Feb 27 14:10:10.732 [initandlisten] connection accepted from xx.xxx.xxx.97:52302 #157 (72 connections now open)
Thu Feb 27 14:10:29.706 [initandlisten] connection accepted from 127.0.0.1:56436 #158 (73 connections now open)
Thu Feb 27 14:10:34.100 [conn156] end connection xx.xxx.xxx.107:43913 (72 connections now open)
Thu Feb 27 14:10:34.101 [initandlisten] connection accepted from xx.xxx.xxx.107:43916 #159 (73 connections now open)
Thu Feb 27 14:10:40.743 [conn157] end connection xx.xxx.xxx.97:52302 (72 connections now open)
Thu Feb 27 14:10:40.744 [initandlisten] connection accepted from xx.xxx.xxx.97:52309 #160 (73 connections now open)
Thu Feb 27 14:11:04.110 [conn159] end connection xx.xxx.xxx.107:43916 (72 connections now open)
Thu Feb 27 14:11:04.111 [initandlisten] connection accepted from xx.xxx.xxx.107:43918 #161 (73 connections now open)
Thu Feb 27 14:11:09.191 [conn161] end connection xx.xxx.xxx.107:43918 (72 connections now open)
Thu Feb 27 14:11:09.452 [initandlisten] connection accepted from xx.xxx.xxx.107:43919 #162 (73 connections now open)
Thu Feb 27 14:11:09.453 [conn162] end connection xx.xxx.xxx.107:43919 (72 connections now open)
Thu Feb 27 14:11:09.456 [initandlisten] connection accepted from xx.xxx.xxx.107:43921 #163 (73 connections now open)
Thu Feb 27 14:11:10.111 [rsHealthPoll] DBClientCursor::init call() failed
Thu Feb 27 14:11:10.111 [rsHealthPoll] replset info mongos2i.hostname.com:27017 heartbeat failed, retrying
Thu Feb 27 14:11:10.113 [rsHealthPoll] replSet member mongos2i.hostname.com:27017 is now in state STARTUP2
Thu Feb 27 14:11:10.755 [conn160] end connection xx.xxx.xxx.97:52309 (72 connections now open)
Thu Feb 27 14:11:10.757 [initandlisten] connection accepted from xx.xxx.xxx.97:52311 #164 (73 connections now open)
Thu Feb 27 14:11:12.113 [rsHealthPoll] replSet member mongos2i.hostname.com:27017 is now in state SECONDARY
Thu Feb 27 14:11:23.462 [conn163] end connection xx.xxx.xxx.107:43921 (72 connections now open)
Thu Feb 27 14:11:23.463 [initandlisten] connection accepted from xx.xxx.xxx.107:43925 #165 (73 connections now open)
Thu Feb 27 14:11:31.831 [conn158] end connection 127.0.0.1:56436 (72 connections now open)
Thu Feb 27 14:11:40.768 [conn164] end connection xx.xxx.xxx.97:52311 (71 connections now open)
Thu Feb 27 14:11:40.769 [initandlisten] connection accepted from xx.xxx.xxx.97:52315 #166 (72 connections now open)
Thu Feb 27 14:11:53.082 [signalProcessingThread] got signal 15 (Terminated), will terminate after current cmd ends
Thu Feb 27 14:11:53.082 [signalProcessingThread] now exiting
Thu Feb 27 14:11:53.082 dbexit:
We are using CentOS and Mongo 2.4.9.
Thanks in advance for the help.
The log output you have posted shows that your MongoDB instance did not crash. It exited normally.
Consider the following lines:
Thu Feb 27 14:11:53.082 [signalProcessingThread] got signal 15 (Terminated), will terminate after current cmd ends
Thu Feb 27 14:11:53.082 [signalProcessingThread] now exiting
Thu Feb 27 14:11:53.082 dbexit:
The first line above indicates that your MongoDB instancce recieved signal 15 from your OS (SIGTERM). This lead to MongoDB terminating. SIGTERM is the default level for the kill command and for stop portion of an init script in most Linux distros.
Related
I'm running mongodb 2.4.10 on a raspberry pi 3b. After few days it started to randomly crash and doesn't start without --journal. I created new database, but again it started to crash. After each crash it requires --repair.
What I noticed is it crashes after trying to sort
User.find({}).limit(10).sort({
messages: -1
}).exec(function(err, docs) {
...
});
But it works fines with find
User.find({}, function(err, docs) {
...
});
Here's full log of mongodb
Sat Feb 11 05:56:02.356
Sat Feb 11 05:56:02.356 warning: 32-bit servers don't have journaling enabled by default. Please use --journal if you want durability.
Sat Feb 11 05:56:02.356
Sat Feb 11 05:56:02.386 [initandlisten] MongoDB starting : pid=754 port=27017 dbpath=db 32-bit host=raspberrypi
Sat Feb 11 05:56:02.386 [initandlisten]
Sat Feb 11 05:56:02.386 [initandlisten] ** NOTE: This is a 32 bit MongoDB binary.
Sat Feb 11 05:56:02.386 [initandlisten] ** 32 bit builds are limited to less than 2GB of data (or less with --journal).
Sat Feb 11 05:56:02.386 [initandlisten] ** Note that journaling defaults to off for 32 bit and is currently off.
Sat Feb 11 05:56:02.386 [initandlisten] ** See http://dochub.mongodb.org/core/32bit
Sat Feb 11 05:56:02.386 [initandlisten]
Sat Feb 11 05:56:02.386 [initandlisten] db version v2.4.10
Sat Feb 11 05:56:02.386 [initandlisten] git version: nogitversion
Sat Feb 11 05:56:02.386 [initandlisten] build info: Linux bm-wb-03 3.19.0-trunk-armmp #1 SMP Debian 3.19.1-1~exp1+plugwash1 (2015-03-28) armv7l BOOST_LIB_VERSION=1_55
Sat Feb 11 05:56:02.386 [initandlisten] allocator: system
Sat Feb 11 05:56:02.386 [initandlisten] options: { dbpath: "db" }
Sat Feb 11 05:56:02.441 [websvr] admin web console waiting for connections on port 28017
Sat Feb 11 05:56:02.442 [initandlisten] waiting for connections on port 27017
Sat Feb 11 05:56:03.724 [initandlisten] connection accepted from 127.0.0.1:43800 #1 (1 connection now open)
Sat Feb 11 05:56:39.087 [conn1] assertion 13097 Unsupported projection option: messages ns:test.users query:{ id: 378418036.0 }
Sat Feb 11 05:56:53.794 [conn1] end connection 127.0.0.1:43800 (0 connections now open)
Sat Feb 11 05:57:01.512 [initandlisten] connection accepted from 127.0.0.1:43960 #2 (1 connection now open)
Sat Feb 11 05:57:01.786 [initandlisten] connection accepted from 127.0.0.1:43962 #3 (2 connections now open)
Sat Feb 11 05:57:01.787 [initandlisten] connection accepted from 127.0.0.1:43964 #4 (3 connections now open)
Sat Feb 11 05:57:01.787 [initandlisten] connection accepted from 127.0.0.1:43966 #5 (4 connections now open)
Sat Feb 11 05:57:08.329 [initandlisten] connection accepted from 127.0.0.1:43974 #6 (5 connections now open)
Sat Feb 11 05:57:09.474 Invalid access at address: 0x6b70f6c7 from thread: conn3
Sat Feb 11 05:57:09.474 Got signal: 7 (Bus error).
Sat Feb 11 05:57:09.479 Backtrace:
0x664160 0x16e370 0x16e6c0 0x766031a0 0x3bfa40 0x433a5c 0x430844 0x431428 0x3c5dc8 0x3c81a4 0x182474 0x64aa08
mongod(_ZN5mongo15printStackTraceERSo+0x1c) [0x664160]
mongod(_ZN5mongo10abruptQuitEi+0x2a0) [0x16e370]
mongod(_ZN5mongo24abruptQuitWithAddrSignalEiP9siginfo_tPv+0x1fc) [0x16e6c0]
/lib/arm-linux-gnueabihf/libc.so.6(__default_rt_sa_restorer_v2+0) [0x766031a0]
mongod(_ZN5mongo22BSONElementManipulator9SetNumberEd+0x98) [0x3bfa40]
mongod(_ZN5mongo11ModSetState16applyModsInPlaceEb+0x3d4) [0x433a5c]
mongod(_ZN5mongo14_updateObjectsEbPKcRKNS_7BSONObjES4_bbbRNS_7OpDebugEPNS_11RemoveSaverEbRKNS_24QueryPlanSelectionPolicyEb+0x2788) [0x430844]
mongod(_ZN5mongo13updateObjectsEPKcRKNS_7BSONObjES4_bbbRNS_7OpDebugEbRKNS_24QueryPlanSelectionPolicyE+0x70) [0x431428]
mongod(_ZN5mongo14receivedUpdateERNS_7MessageERNS_5CurOpE+0x35c) [0x3c5dc8]
mongod(_ZN5mongo16assembleResponseERNS_7MessageERNS_10DbResponseERKNS_11HostAndPortE+0x14b0) [0x3c81a4]
mongod(_ZN5mongo16MyMessageHandler7processERNS_7MessageEPNS_21AbstractMessagingPortEPNS_9LastErrorE+0x9c) [0x182474]
mongod(_ZN5mongo17PortMessageServer17handleIncomingMsgEPv+0x420) [0x64aa08]
I am interacting with many db collections of MongoDB using Java driver. The queries are basically INSERT,DELETE and FETCH documents. Everything is working fine except the connections are open in the logs.
Below is the log file
Fri Dec 27 17:27:28.484 [initandlisten] connection accepted from 127.0.0.1:51018 #125 (41 connections now open)
Fri Dec 27 17:27:29.209 [initandlisten] connection accepted from 127.0.0.1:51020 #126 (42 connections now open)
Fri Dec 27 17:27:30.222 [initandlisten] connection accepted from 127.0.0.1:51021 #127 (43 connections now open)
Fri Dec 27 17:27:30.254 [conn126] end connection 127.0.0.1:51020 (42 connections now open)
Fri Dec 27 17:27:31.492 [conn125] info DFM::findAll(): extent 0:9f000 was empty, skipping ahead. ns:pingpong.SearchResult
Fri Dec 27 17:27:31.516 [conn125] end connection 127.0.0.1:51018 (41 connections now open)
Fri Dec 27 17:27:40.038 [initandlisten] connection accepted from 127.0.0.1:51032 #128 (42 connections now open)
Fri Dec 27 17:27:40.625 [initandlisten] connection accepted from 127.0.0.1:51033 #129 (43 connections now open)
Fri Dec 27 17:27:41.640 [initandlisten] connection accepted from 127.0.0.1:51034 #130 (44 connections now open)
Fri Dec 27 17:27:41.675 [conn129] end connection 127.0.0.1:51033 (43 connections now open)
Fri Dec 27 17:27:43.062 [conn128] end connection 127.0.0.1:51032 (42 connections now open)
Fri Dec 27 17:27:52.595 [initandlisten] connection accepted from 127.0.0.1:51037 #131 (43 connections now open)
Fri Dec 27 17:27:53.275 [initandlisten] connection accepted from 127.0.0.1:51038 #132 (44 connections now open)
Fri Dec 27 17:27:53.601 [conn131] info DFM::findAll(): extent 0:9f000 was empty, skipping ahead. ns:pingpong.SearchResult
Fri Dec 27 17:27:54.330 [initandlisten] connection accepted from 127.0.0.1:51039 #133 (45 connections now open)
Fri Dec 27 17:27:54.375 [conn132] end connection 127.0.0.1:51038 (44 connections now open)
Fri Dec 27 17:27:55.619 [conn131] end connection 127.0.0.1:51037 (43 connections now open)
Fri Dec 27 17:28:07.823 [initandlisten] connection accepted from 127.0.0.1:51042 #134 (44 connections now open)
Fri Dec 27 17:28:08.439 [initandlisten] connection accepted from 127.0.0.1:51043 #135 (45 connections now open)
Fri Dec 27 17:28:09.475 [initandlisten] connection accepted from 127.0.0.1:51044 #136 (46 connections now open)
Fri Dec 27 17:28:09.496 [conn135] info DFM::findAll(): extent 0:24000 was empty, skipping ahead. ns:pingpong.Messages
Fri Dec 27 17:28:09.509 [conn135] end connection 127.0.0.1:51043 (45 connections now open)
Fri Dec 27 17:28:10.838 [conn134] end connection 127.0.0.1:51042 (44 connections now open)
Sat Dec 28 04:10:58.018 [initandlisten] connection accepted from 127.0.0.1:53778 #137 (45 connections now open)
Sat Dec 28 04:11:04.501 [initandlisten] connection accepted from 127.0.0.1:53779 #138 (46 connections now open)
Sat Dec 28 04:11:05.561 [initandlisten] connection accepted from 127.0.0.1:53780 #139 (47 connections now open)
Sat Dec 28 04:11:05.621 [conn138] info DFM::findAll(): extent 0:24000 was empty, skipping ahead. ns:pingpong.Messages
Sat Dec 28 04:11:05.658 [conn138] info DFM::findAll(): extent 0:24000 was empty, skipping ahead. ns:pingpong.Messages
Sat Dec 28 04:11:05.699 [conn138] info DFM::findAll(): extent 0:24000 was empty, Sat Dec 28 04:11:05.737 [conn138] end connection 127.0.0.1:53779 (46 connections now open)
I am using mongoclient.close() after each of the INSERT,DELETE and FETCH commands. But still the connections are open. Am I missing something. The below is the delete code
MongoClient mongoclient;
String mobileID=getMobileID(messageid);
DBUtils dbUtils=new DBUtils();
mongoclient=dbUtils.connectToDB("pingpong");
dbUtils.deleteMessage(mobileID,messageid);
mongoclient.close();
Am I missing something. Please let me know.
As JohnnyHK mentioned that in the comments, it is not the best approach.
In the case you're building a web application (example: REST structure), and using a container for your app' (example: Tomcat 8) what you should do is declaring a shared MongoClient instance like this: private static MongoClient mongoClient;
Then you have to set up a servletContextListener and implement the basic two methods that are called when the context is initialized and/or stopped.
See below example. I'm using the mongo java driver 3.0.3, tomcat and the servlet-api 3.1.0, which is added as dependency with Maven
Dependency:
<dependency>
<groupId>javax.servlet</groupId>
<artifactId>javax.servlet-api</artifactId>
<version>3.1.0</version>
</dependency>
Example:
#Path("/rest")
#WebListener
public class RESTInterface implements ServletContextListener{
private static MongoClient mongoClient;
#Override
public void contextInitialized(ServletContextEvent servletContextEvent) {
System.out.println("initialized");
try
{
ServerAddress serverAddress = new ServerAddress(IP_connectionString,27017); //default port 27017
mongoClient = new MongoClient(serverAddress,options);
}
catch (Exception e)
{
...
}
}
#Override
public void contextDestroyed(ServletContextEvent servletContextEvent) {
System.out.println("MongoClient closed");
mongoClient.close();
}
//whatever you want with the mongoClient !
}
When I start mongod trowed error. What is the problem?
root#cs12867:~# mongod
mongod --help for help and startup options
Sun Dec 22 15:37:44.081 [initandlisten] MongoDB starting : pid=18415 port=27017 dbpath=/data/db/ 64-bit host=cs12867
Sun Dec 22 15:37:44.081 [initandlisten] db version v2.4.6
Sun Dec 22 15:37:44.081 [initandlisten] git version: b9925db5eac369d77a3a5f5d98a145eaaacd9673
Sun Dec 22 15:37:44.081 [initandlisten] build info: Linux ip-10-2-29-40 2.6.21.7-2.ec2.v1.2.fc8xen #1 SMP Fri Nov 20 17:48:28 EST 2009 x86_64 BOOST_LIB_VERSION=1_49
Sun Dec 22 15:37:44.081 [initandlisten] allocator: tcmalloc
Sun Dec 22 15:37:44.081 [initandlisten] options: {}
Sun Dec 22 15:37:44.081 [initandlisten] exception in initAndListen: 10296
*********************************************************************
ERROR: dbpath (/data/db/) does not exist.
Create this directory or give existing directory in --dbpath.
See http://dochub.mongodb.org/core/startingandstoppingmongo
*********************************************************************
, terminating
Sun Dec 22 15:37:44.081 dbexit:
Sun Dec 22 15:37:44.081 [initandlisten] shutdown: going to close listening sockets...
Sun Dec 22 15:37:44.081 [initandlisten] shutdown: going to flush diaglog...
Sun Dec 22 15:37:44.081 [initandlisten] shutdown: going to close sockets...
Sun Dec 22 15:37:44.081 [initandlisten] shutdown: waiting for fs preallocator...
Sun Dec 22 15:37:44.081 [initandlisten] shutdown: lock for final commit...
Sun Dec 22 15:37:44.081 [initandlisten] shutdown: final commit...
Sun Dec 22 15:37:44.081 [initandlisten] shutdown: closing all files...
Sun Dec 22 15:37:44.081 [initandlisten] closeAllFiles() finished
Sun Dec 22 15:37:44.081 dbexit: really exiting now
I set --dbpath:
root#cs12867:~# mongod --dbpath /var/lib/mongodb/data/db
Sun Dec 22 15:29:25.121 [initandlisten] MongoDB starting : pid=17739 port=27017 dbpath=/var/lib/mongodb/data/db 64-bit host=cs12867
Sun Dec 22 15:29:25.121 [initandlisten] db version v2.4.6
Sun Dec 22 15:29:25.121 [initandlisten] git version: b9925db5eac369d77a3a5f5d98a145eaaacd9673
Sun Dec 22 15:29:25.121 [initandlisten] build info: Linux ip-10-2-29-40 2.6.21.7-2.ec2.v1.2.fc8xen #1 SMP Fri Nov 20 17:48:28 EST 2009 x86_64 BOOST_LIB_VERSION=1_49
Sun Dec 22 15:29:25.121 [initandlisten] allocator: tcmalloc
Sun Dec 22 15:29:25.121 [initandlisten] options: { dbpath: "/var/lib/mongodb/data/db" }
Sun Dec 22 15:29:25.121 [initandlisten]
Sun Dec 22 15:29:25.121 [initandlisten] ** WARNING: Readahead for /var/lib/mongodb/data/db is set to 2048KB
Sun Dec 22 15:29:25.121 [initandlisten] ** We suggest setting it to 256KB (512 sectors) or less
Sun Dec 22 15:29:25.121 [initandlisten] ** http://dochub.mongodb.org/core/readahead
Sun Dec 22 15:29:25.123 [initandlisten] journal dir=/var/lib/mongodb/data/db/journal
Sun Dec 22 15:29:25.123 [initandlisten] recover : no journal files present, no recovery needed
Sun Dec 22 15:29:25.134 [initandlisten] waiting for connections on port 27017
Sun Dec 22 15:29:25.134 [websvr] admin web console waiting for connections on port 28017
Sun Dec 22 15:29:26.179 [initandlisten] connection accepted from 127.0.0.1:37164 #1 (1 connection now open)
Sun Dec 22 15:29:26.179 [FileAllocator] allocating new datafile /var/lib/mongodb/data/db/Tester.ns, filling with zeroes...
Sun Dec 22 15:29:26.179 [FileAllocator] creating directory /var/lib/mongodb/data/db/_tmp
Sun Dec 22 15:29:26.181 [FileAllocator] done allocating datafile /var/lib/mongodb/data/db/Tester.ns, size: 16MB, took 0 secs
Sun Dec 22 15:29:26.181 [FileAllocator] allocating new datafile /var/lib/mongodb/data/db/Tester.0, filling with zeroes...
Sun Dec 22 15:29:26.181 [FileAllocator] done allocating datafile /var/lib/mongodb/data/db/Tester.0, size: 64MB, took 0 secs
Sun Dec 22 15:29:26.182 [FileAllocator] allocating new datafile /var/lib/mongodb/data/db/Tester.1, filling with zeroes...
Sun Dec 22 15:29:26.182 [FileAllocator] done allocating datafile /var/lib/mongodb/data/db/Tester.1, size: 128MB, took 0 secs
Sun Dec 22 15:29:26.184 [conn1] build index Tester.user { _id: 1 }
Sun Dec 22 15:29:26.184 [conn1] build index done. scanned 0 total records. 0 secs
Sun Dec 22 15:29:26.184 [conn1] end connection 127.0.0.1:37164 (0 connections now open)
Sun Dec 22 15:29:26.192 [initandlisten] connection accepted from 127.0.0.1:37165 #2 (1 connection now open)
Sun Dec 22 15:29:26.732 [initandlisten] connection accepted from 127.0.0.1:37166 #3 (2 connections now open)
Sun Dec 22 15:29:26.733 [conn3] end connection 127.0.0.1:37166 (1 connection now open)
Sun Dec 22 15:29:26.740 [initandlisten] connection accepted from 127.0.0.1:37167 #4 (2 connections now open)
Sun Dec 22 15:29:26.740 [conn4] end connection 127.0.0.1:37167 (1 connection now open)
Sun Dec 22 15:29:26.834 [initandlisten] connection accepted from 127.0.0.1:37168 #5 (2 connections now open)
Sun Dec 22 15:29:26.835 [conn5] end connection 127.0.0.1:37168 (1 connection now open)
Sun Dec 22 15:29:26.843 [initandlisten] connection accepted from 127.0.0.1:37169 #6 (2 connections now open)
Sun Dec 22 15:29:27.252 [initandlisten] connection accepted from 127.0.0.1:37170 #7 (3 connections now open)
Sun Dec 22 15:29:27.253 [conn7] end connection 127.0.0.1:37170 (2 connections now open)
Sun Dec 22 15:29:27.304 [initandlisten] connection accepted from 127.0.0.1:37171 #8 (3 connections now open)
Sun Dec 22 15:29:27.304 [conn8] end connection 127.0.0.1:37171 (2 connections now open)
Sun Dec 22 15:29:28.332 [initandlisten] connection accepted from 127.0.0.1:37173 #9 (3 connections now open)
Sun Dec 22 15:29:28.333 [conn9] end connection 127.0.0.1:37173 (2 connections now open)
Sun Dec 22 15:29:28.352 [initandlisten] connection accepted from 127.0.0.1:37174 #10 (3 connections now open)
Sun Dec 22 15:29:28.842 [conn10] end connection 127.0.0.1:37174 (2 connections now open)
Sun Dec 22 15:29:28.862 [initandlisten] connection accepted from 127.0.0.1:37176 #11 (3 connections now open)
Sun Dec 22 15:29:28.863 [conn11] end connection 127.0.0.1:37176 (2 connections now open)
Sun Dec 22 15:29:30.362 [initandlisten] connection accepted from 127.0.0.1:37178 #12 (3 connections now open)
Sun Dec 22 15:29:30.363 [initandlisten] connection accepted from 127.0.0.1:37179 #13 (4 connections now open)
Sun Dec 22 15:29:30.364 [conn13] end connection 127.0.0.1:37179 (3 connections now open)
Sun Dec 22 15:37:36.523 [signalProcessingThread] shutdown: closing all files...
Sun Dec 22 15:37:36.523 [signalProcessingThread] closeAllFiles() finished
Sun Dec 22 15:37:36.523 [signalProcessingThread] journalCleanup...
Sun Dec 22 15:37:36.523 [signalProcessingThread] removeJournalFiles
Sun Dec 22 15:37:36.525 [signalProcessingThread] shutdown: removing fs lock...
Sun Dec 22 15:37:36.525 dbexit: really exiting now
Then again:
root#cs12867:~# mongod
mongod --help for help and startup options
Sun Dec 22 15:37:44.081 [initandlisten] MongoDB starting : pid=18415 port=27017 dbpath=/data/db/ 64-bit host=cs12867
Sun Dec 22 15:37:44.081 [initandlisten] db version v2.4.6
Sun Dec 22 15:37:44.081 [initandlisten] git version: b9925db5eac369d77a3a5f5d98a145eaaacd9673
Sun Dec 22 15:37:44.081 [initandlisten] build info: Linux ip-10-2-29-40 2.6.21.7-2.ec2.v1.2.fc8xen #1 SMP Fri Nov 20 17:48:28 EST 2009 x86_64 BOOST_LIB_VERSION=1_49
Sun Dec 22 15:37:44.081 [initandlisten] allocator: tcmalloc
Sun Dec 22 15:37:44.081 [initandlisten] options: {}
Sun Dec 22 15:37:44.081 [initandlisten] exception in initAndListen: 10296
*********************************************************************
ERROR: dbpath (/data/db/) does not exist.
Create this directory or give existing directory in --dbpath.
See http://dochub.mongodb.org/core/startingandstoppingmongo
*********************************************************************
, terminating
Sun Dec 22 15:37:44.081 dbexit:
Sun Dec 22 15:37:44.081 [initandlisten] shutdown: going to close listening sockets...
Sun Dec 22 15:37:44.081 [initandlisten] shutdown: going to flush diaglog...
Sun Dec 22 15:37:44.081 [initandlisten] shutdown: going to close sockets...
Sun Dec 22 15:37:44.081 [initandlisten] shutdown: waiting for fs preallocator...
Sun Dec 22 15:37:44.081 [initandlisten] shutdown: lock for final commit...
Sun Dec 22 15:37:44.081 [initandlisten] shutdown: final commit...
Sun Dec 22 15:37:44.081 [initandlisten] shutdown: closing all files...
Sun Dec 22 15:37:44.081 [initandlisten] closeAllFiles() finished
Sun Dec 22 15:37:44.081 dbexit: really exiting now
The default db path for mongodb is /data/db/.
So when you run:
mongod
MongoDB checks if /data/db/ is present and if the user has access to it. In your case, there is no such directory and hence the error.
However you can override the default db path using the --dbpath argument of mongod( or use a config file).
mongod --dbpath /var/lib/mongodb/data/db
In this case instead of checking for /data/db/ mongoDB check for /var/lib/mongodb/data/db. In your case, you have the specified directory and you have access to it and hence it runs.
And doing this doesn't start the mongo shell. To start the shell, start mongod first as mentioned above and leave it running. Then in another terminal execute
mongo
This will start the mongo shell.
As I see from the problem that creating the data/db directory will fix your problem.
Create that and then add like this if you have directory path in c drive.
mongod --dbpath "c://data/db"
After setting the dbpath you should start the mongodb as service.
sudo service mongodb start
Another solution to that problem could be:
.
Delete the files inside of /data/journal/ or /data/db/journal/
.
Especially if your database stopped working after an unclean restart / crash. It seems that the journal files can get corrupted and block the mongo service from starting.
Remember to backup the files first just in case.
Following these posts:
https://stackoverflow.com/a/16323101
https://jira.mongodb.org/browse/SERVER-5380
Whenever I try to play with mongo's interactive shell, it dies:
somekittens#DLserver01:~$ mongo
MongoDB shell version: 2.2.2
connecting to: test
Mon Dec 17 13:14:16 DBClientCursor::init call() failed
Mon Dec 17 13:14:16 Error: Error during mongo startup. :: caused by :: 10276 DBClientBase::findN: transport error: 127.0.0.1:27017 ns: admin.$cmd query: { whatsmyuri: 1 } src/mongo/shell/mongo.js:91
exception: connect failed
I'm able to repair the install (deleting mongodb.lock, etc) and get back to this point, but it'll only die again.
/var/log/mongodb/mongodb.log
Mon Dec 17 13:14:03
Mon Dec 17 13:14:03 warning: 32-bit servers don't have journaling enabled by default. Please use --journal if you want durability.
Mon Dec 17 13:14:03
Mon Dec 17 13:14:03 [initandlisten] MongoDB starting : pid=2674 port=27017 dbpath=/var/lib/mongodb 32-bit host=DLserver01
Mon Dec 17 13:14:03 [initandlisten]
Mon Dec 17 13:14:03 [initandlisten] ** NOTE: when using MongoDB 32 bit, you are limited to about 2 gigabytes of data
Mon Dec 17 13:14:03 [initandlisten] ** see http://blog.mongodb.org/post/137788967/32-bit-limitations
Mon Dec 17 13:14:03 [initandlisten] ** with --journal, the limit is lower
Mon Dec 17 13:14:03 [initandlisten]
Mon Dec 17 13:14:03 [initandlisten] db version v2.2.2, pdfile version 4.5
Mon Dec 17 13:14:03 [initandlisten] git version: d1b43b61a5308c4ad0679d34b262c5af9d664267
Mon Dec 17 13:14:03 [initandlisten] build info: Linux domU-12-31-39-01-70-B4 2.6.21.7-2.fc8xen #1 SMP Fri Feb 15 12:39:36 EST 2008 i686 BOOST_LIB_VERSION=1_49
Mon Dec 17 13:14:03 [initandlisten] options: { config: "/etc/mongodb.conf", dbpath: "/var/lib/mongodb", logappend: "true", logpath: "/var/log/mongodb/mongodb.log" }
Mon Dec 17 13:14:03 [initandlisten] Unable to check for journal files due to: boost::filesystem::basic_directory_iterator constructor: No such file or directory: "/var/lib/mongodb/journal"
Mon Dec 17 13:14:03 [initandlisten] couldn't unlink socket file /tmp/mongodb-27017.sockerrno:1 Operation not permitted skipping
Mon Dec 17 13:14:03 [initandlisten] waiting for connections on port 27017
Mon Dec 17 13:14:03 [websvr] admin web console waiting for connections on port 28017
Mon Dec 17 13:14:16 [initandlisten] connection accepted from 127.0.0.1:57631 #1 (1 connection now open)
Mon Dec 17 13:14:16 Invalid operation at address: 0x819bb23 from thread: conn1
Mon Dec 17 13:14:16 Got signal: 4 (Illegal instruction).
Mon Dec 17 13:14:16 Backtrace:
0x8759eaa 0x817033a 0x81709ff 0x20e40c 0x819bb23 0x854cd54 0x85377d1 0x846b594 0x83e5591 0x83e6c15 0x81902b4 0x8746731 0x49ad4c 0x34ed3e
/usr/bin/mongod(_ZN5mongo15printStackTraceERSo+0x2a) [0x8759eaa]
/usr/bin/mongod(_ZN5mongo10abruptQuitEi+0x3ba) [0x817033a]
/usr/bin/mongod(_ZN5mongo24abruptQuitWithAddrSignalEiP7siginfoPv+0x2af) [0x81709ff]
[0x20e40c]
/usr/bin/mongod(_ZNK5mongo7BSONObj4copyEv+0x33) [0x819bb23]
/usr/bin/mongod(_ZN5mongo11ParsedQuery4initERKNS_7BSONObjE+0x494) [0x854cd54]
/usr/bin/mongod(_ZN5mongo11ParsedQueryC1ERNS_12QueryMessageE+0x91) [0x85377d1]
/usr/bin/mongod(_ZN5mongo8runQueryERNS_7MessageERNS_12QueryMessageERNS_5CurOpES1_+0x34) [0x846b594]
/usr/bin/mongod() [0x83e5591]
/usr/bin/mongod(_ZN5mongo16assembleResponseERNS_7MessageERNS_10DbResponseERKNS_11HostAndPortE+0x3d5) [0x83e6c15]
/usr/bin/mongod(_ZN5mongo16MyMessageHandler7processERNS_7MessageEPNS_21AbstractMessagingPortEPNS_9LastErrorE+0x84) [0x81902b4]
/usr/bin/mongod(_ZN5mongo3pms9threadRunEPNS_13MessagingPortE+0x421) [0x8746731]
/lib/i386-linux-gnu/libpthread.so.0(+0x6d4c) [0x49ad4c]
/lib/i386-linux-gnu/libc.so.6(clone+0x5e) [0x34ed3e]
Connecting using node's shell:
> mdb.open(function(err, db) { console.log(err) });
[Error: failed to connect to [localhost:27017]]
I've searched around for this error and found nothing of use. This is running on a fairly old server (Ubuntu 12.04 32-bit, 640MB RAM, 500MHz P2). How can I fix this?
There is an issue Invalid operation at address: 0x819b263 from thread: TTLMonitor in mongodb jira list. I think that is about your case.
A new server may be the easiest solution, otherwise you have to download the source code, make some modification and compile it youself.
I'm using node.js and MongoDB for my application. Whenever I use localhost, I have no problems and the application works fine. However, on the server, the database is limiting me to 50 connections. Here's an example of the log:
Wed Jul 27 13:33:29 [initandlisten] waiting for connections on port 27017
Wed Jul 27 13:33:29 [websvr] web admin interface listening on port 28017
Wed Jul 27 13:34:50 [initandlisten] connection accepted from 127.0.0.1:42035 #1
Wed Jul 27 13:35:16 [initandlisten] connection accepted from 127.0.0.1:42181 #2
Wed Jul 27 13:35:16 [initandlisten] connection accepted from 127.0.0.1:42182 #3
Wed Jul 27 13:35:25 [initandlisten] connection accepted from 127.0.0.1:42249 #4
...
Wed Jul 27 13:36:09 [initandlisten] connection accepted from 127.0.0.1:42518 #50
Wed Jul 27 13:36:10 [initandlisten] connection accepted from 127.0.0.1:42524 #51
Wed Jul 27 13:36:10 [initandlisten] can't create new thread, closing connection
I'm launching the process with the command mongod --maxConns=5000. Does anyone know what could be causing this connection limit?
Can you post the code you're using to connect? If you're connecting to the DB on each request then you'll quickly run out of connections. In most cases it's best to share the DB connection among requests, for example by connecting on app startup.