Tue Apr 1 17:21:38.898 [initandlisten] connection accepted from 127.0.0.1:58349 #42 (5 connections now open)
Tue Apr 1 17:21:50.855 [journal] exception in dur::groupCommitLL causing immediate shutdown: boost::filesystem::current_path: Too many open files in system
Tue Apr 1 17:21:50.855 dur4
Tue Apr 1 17:21:50.855 Got signal: 6 (Abort trap: 6).
Tue Apr 1 17:21:50.858 Backtrace:
0x100447d8b 0x100001531 0x7fff9225990a 0x104e80818 0x7fff922b0f61 0x1001e462a 0x10016dc3c 0x10016d537 0x10047a8b5 0x7fff9226b772 0x7fff922581a1
0 mongod 0x0000000100447d8b _ZN5mongo15printStackTraceERSo + 43
1 mongod 0x0000000100001531 _ZN5mongo10abruptQuitEi + 225
2 libsystem_c.dylib 0x00007fff9225990a _sigtramp + 26
3 ??? 0x0000000104e80818 0x0 + 4377282584
4 libsystem_c.dylib 0x00007fff922b0f61 abort + 143
5 mongod 0x00000001001e462a _ZN5mongo10mongoAbortEPKc + 90
6 mongod 0x000000010016dc3c _ZN5mongo3dur27groupCommitWithLimitedLocksEv + 1052
7 mongod 0x000000010016d537 _ZN5mongo3dur9durThreadEv + 583
8 mongod 0x000000010047a8b5 thread_proxy + 229
9 libsystem_c.dylib 0x00007fff9226b772 _pthread_start + 327
10 libsystem_c.dylib 0x00007fff922581a1 thread_start + 13
Would anyone know what is causing mongo to crash? If you need more context please comment - I'll see what else I can dig up.
Change the ulimit value works for me (previously it was 256):
ulimit -n 1000
In this case I'm setting ulimit to 1.000, but you can use whatever value that you want.
According to documentation:
ulimit refers to the per-user limitations for various resources.
Therefore, if your mongod instance executes as a user that is also
running multiple processes, or multiple mongod processes, you might
see contention for these resources. Also, be aware that the processes
value (i.e. -u) refers to the combined number of distinct processes
and sub-process threads.
Related
I've used MAMP Pro for years and never experienced this issue, until the last week or so. I haven't installed any updates for MacOS or MAMP Pro in that time.
On starting MAMP Pro the servers start to boot, MySQL fires up fine and before Apache can start properly the whole MAMP App crashes.
I'll not post the full crash report for fear of sending readers to sleep, however here is what I have worked out as the failing thread...
Crashed Thread: 0 Dispatch queue: com.apple.main-thread
Exception Type: EXC_CRASH (SIGABRT)
Exception Codes: 0x0000000000000000, 0x0000000000000000
Exception Note: EXC_CORPSE_NOTIFY
Application Specific Information:
abort() called
Thread 0 Crashed:: Dispatch queue: com.apple.main-thread
0 libsystem_kernel.dylib 0x00007fff6d9fefce __pthread_kill + 10
1 libsystem_pthread.dylib 0x00007fff6db3c150 pthread_kill + 333
2 libsystem_c.dylib 0x00007fff6d95b30a abort + 127
3 net.hockeyapp.sdk.mac 0x000000010b5a3c02 uncaught_exception_handler + 27
4 com.apple.CoreFoundation 0x00007fff4633fbd2 __handleUncaughtException + 770
5 libobjc.A.dylib 0x00007fff6ccc4a41 _objc_terminate() + 91
6 net.hockeyapp.sdk.mac 0x000000010b591417 BITCrashUncaughtCXXTerminateHandler() + 766
7 libc++abi.dylib 0x00007fff6b95b7c9 std::__terminate(void (*)()) + 8
8 libc++abi.dylib 0x00007fff6b95b843 std::terminate() + 51
9 libdispatch.dylib 0x00007fff6d875e9c _dispatch_client_callout + 28
10 libdispatch.dylib 0x00007fff6d881471 _dispatch_main_queue_callback_4CF + 1148
11 com.apple.CoreFoundation 0x00007fff46268b99 __CFRUNLOOP_IS_SERVICING_THE_MAIN_DISPATCH_QUEUE__ + 9
12 com.apple.CoreFoundation 0x00007fff4622b44a __CFRunLoopRun + 2586
13 com.apple.CoreFoundation 0x00007fff4622a797 CFRunLoopRunSpecific + 487
14 com.apple.HIToolbox 0x00007fff4553f866 RunCurrentEventLoopInMode + 286
15 com.apple.HIToolbox 0x00007fff4553f5d6 ReceiveNextEventCommon + 613
16 com.apple.HIToolbox 0x00007fff4553f354 _BlockUntilNextEventMatchingListInModeWithFilter + 64
17 com.apple.AppKit 0x00007fff4383ca23 _DPSNextEvent + 2085
18 com.apple.AppKit 0x00007fff43fd1e6c -[NSApplication(NSEvent) _nextEventMatchingEventMask:untilDate:inMode:dequeue:] + 3044
19 com.apple.AppKit 0x00007fff43831831 -[NSApplication run] + 764
20 com.apple.AppKit 0x00007fff438009d2 NSApplicationMain + 804
21 libdyld.dylib 0x00007fff6d8af145 start + 1
Has anyone else experienced this, or can offer an explanation of how I might be able to fix it? I work offline a lot so not being able to fire up MAMP is a game changer.
Cheers.
My issue is this: When connected to the web, Mamp freezes and takes several minutes just to load pages. As soon as I disconnect from the web- it loads pages immediately.
I've Flushed DNS, cleared caches etc all to no avail. I'm on a Macbook, tried editing etc/hosts and still nothing. Didn't change anything yet it all just came out of nowhere.
Frustrating and can't work properly like this.
I have 2 mongod instances running with the following parameters
--noprealloc --smallfiles --replSet mongors1 --dbpath /data/db --nojournal
The goal of the exercise is to create a replicated environment with a minimal disk footprint for local development purposes.
At this point in time, all is good with each respective data system being around ~32M and having the following
ls -o data/db
total 32784
-rw------- 1 999 16777216 Sep 22 11:38 local.0
-rw------- 1 999 16777216 Sep 22 11:38 local.ns
-rwxr-xr-x 1 999 2 Sep 22 11:38 mongod.lock
-rw-r--r-- 1 999 69 Sep 22 11:38 storage.bson
drwxr-xr-x 2 999 4096 Sep 22 11:38 _tmp
After logging on to the first member and running rs.initiate(); an additional 1G of disk space is utilized.
ls -o data/db
total 1080856
-rw------- 1 999 16777216 Sep 22 11:39 local.0
-rw------- 1 999 536608768 Sep 22 11:39 local.1
-rw------- 1 999 536608768 Sep 22 11:39 local.2
-rw------- 1 999 16777216 Sep 22 11:39 local.ns
-rwxr-xr-x 1 999 2 Sep 22 11:38 mongod.lock
-rw-r--r-- 1 999 69 Sep 22 11:38 storage.bson
drwxr-xr-x 2 999 4096 Sep 22 11:39 _tmp
This seems excessive given the properties of the nodes being replicated and the configuration they are running.
Mongo 3.0.6 is the version in use.
Eventually this will be scaled up to replica sets with 3 members across 2+ shards. A minimal disk requirement of 6Gb to store zero data initially seems sub-optimal.
Is there a way to reduce this to something more representative of the nodes needs?
Any help is appreciated. Thanks in advance
The local database contains the oplog, and I'll leave you to research yourself as to what size this should be for a given node. To address the question at hand, from the docs:
For 64-bit Linux, Solaris, FreeBSD, and Windows systems, MongoDB
allocates 5% of the available free disk space, but will always
allocate at least 1 gigabyte and never more than 50 gigabytes.
That's where your usage is coming from - to alter that allocation you will either need to resize the oplog or, if starting from scratch, look at the oplogSizeMB option (or for the CLI equivalent see here).
In addition to what Adam said, add the
--oplogSize X
to your parameters and replace X with the amount of MB you want the oplog to be.
I was running mongodb without any issues until now. I don't really know what happened, as don't remember doing nothing weird. Still, now I get the following error when I try to launch the mongo server.
$ mongod --config /usr/local/etc/mongod.conf
2015-01-22T13:58:04.153+0100 Assertion failure getcwd(buffer, 1000) src/mongo/db/server_options_helpers.cpp 355
2015-01-22T13:58:04.169+0100 0x1017ae5aa 0x101759135 0x101745bf2 0x1015509ed 0x10132b7a3 0x100fd2be6 0x100fc5291 0x100fd60be 0x100fd68ae 0x100fc41eb 0x7fff8aff35c9
0 mongod 0x00000001017ae5aa _ZN5mongo15printStackTraceERNSt3__113basic_ostreamIcNS0_11char_traitsIcEEEE + 58
1 mongod 0x0000000101759135 _ZN5mongo10logContextEPKc + 453
2 mongod 0x0000000101745bf2 _ZN5mongo12verifyFailedEPKcS1_j + 626
3 mongod 0x00000001015509ed _ZN5mongo18storeServerOptionsERKNS_17optionenvironment11EnvironmentERKNSt3__16vectorINS4_12basic_stringIcNS4_11char_traitsIcEENS4_9allocatorIcEEEENS9_ISB_EEEE + 10237
4 mongod 0x000000010132b7a3 _ZN5mongo18storeMongodOptionsERKNS_17optionenvironment11EnvironmentERKNSt3__16vectorINS4_12basic_stringIcNS4_11char_traitsIcEENS4_9allocatorIcEEEENS9_ISB_EEEE + 35
5 mongod 0x0000000100fd2be6 _ZN5mongo45_mongoInitializerFunction_MongodOptions_StoreEPNS_18InitializerContextE + 38
6 mongod 0x0000000100fc5291 _ZN5boost6detail8function17function_invoker1IPFN5mongo6StatusEPNS3_18InitializerContextEES4_S6_E6invokeERNS1_15function_bufferES6_ + 17
7 mongod 0x0000000100fd60be _ZNK5mongo11Initializer7executeERKNSt3__16vectorINS1_12basic_stringIcNS1_11char_traitsIcEENS1_9allocatorIcEEEENS6_IS8_EEEERKNS1_3mapIS8_S8_NS1_4lessIS8_EENS6_INS1_4pairIKS8_S8_EEEEEE + 270
8 mongod 0x0000000100fd68ae _ZN5mongo21runGlobalInitializersEiPKPKcS3_ + 526
9 mongod 0x0000000100fc41eb main + 331
10 libdyld.dylib 0x00007fff8aff35c9 start + 1
2015-01-22T13:58:04.177+0100 SEVERE: Failed global initialization: UnknownError assertion src/mongo/db/server_options_helpers.cpp:355
I've tried running mongo --repair too, but got the same error.
This is weird!
I'm on OSX, using iTerm where I've tried to run the above process. Once I've quite iTerm, and restarted it, the created shell could run the process without any problems.
It seems as the process could not stop previously. Before, I even tried running mongo to see if I can connect, but it failed.
We experienced some strange thing in our mongodb gridfs platform.
The platform actually is a bi Xeon E5 (bi quad core) with 128GB of memory, running on freebsd 9 with a zfs pool dedicated for mongodb.
[root#mongofile1 ~]# uname -sr
FreeBSD 9.1-RELEASE
our /boot/loader.conf
vfs.zfs.arc_min="2048M"
vfs.zfs.arc_max="7680M"
vm.kmem_size_max="16G"
vm.kmem_size="12G"
vfs.zfs.prefetch_disable="1"
kern.ipc.nmbclusters="32768"
/etc/sysctl.conf
net.inet.tcp.msl=15000
net.inet.tcp.keepidle=300000
kern.ipc.nmbclusters=32768
kern.ipc.maxsockbuf=2097152
kern.ipc.somaxconn=8192
kern.maxfiles=65536
kern.maxfilesperproc=32768
net.inet.tcp.delayed_ack=0
net.inet.tcp.sendspace=65535
net.inet.udp.recvspace=65535
net.inet.udp.maxdgram=57344
net.local.stream.recvspace=65535
net.local.stream.sendspace=65535
we follow the recommandation for the ulimit :
[root#mongofile1 ~]# su - mongodb
$ ulimit -a
cpu time (seconds, -t) unlimited
file size (512-blocks, -f) unlimited
data seg size (kbytes, -d) 33554432
stack size (kbytes, -s) 524288
core file size (512-blocks, -c) unlimited
max memory size (kbytes, -m) unlimited
locked memory (kbytes, -l) unlimited
max user processes (-u) 5547
open files (-n) 32768
virtual mem size (kbytes, -v) unlimited
swap limit (kbytes, -w) unlimited
sbsize (bytes, -b) unlimited
pseudo-terminals (-p) unlimited
This server have a twin( same config exactly) for ReplSet in other datacenter and we have a virtualized arbiter.
Some time, almost 3 days, the process of mongodb exit
The problem begin with
Fri Nov 8 11:27:31.741 [conn774697] end connection 192.168.10.162:47963 (23 connections now open)
Fri Nov 8 11:27:31.770 [initandlisten] can't create new thread, closing connection
Fri Nov 8 11:27:31.771 [rsHealthPoll] replSet member mongofile2:27017 is now in state DOWN
Fri Nov 8 11:27:31.774 [initandlisten] connection accepted from 192.168.10.162:47968 #774702 (20 connections now open)
Fri Nov 8 11:27:31.774 [initandlisten] connection accepted from 192.168.10.161:28522 #774703 (21 connections now open)
Fri Nov 8 11:27:31.774 [initandlisten] connection accepted from 192.168.10.164:15406 #774704 (22 connections now open)
Fri Nov 8 11:27:31.774 [initandlisten] connection accepted from 192.168.10.163:25750 #774705 (23 connections now open)
Fri Nov 8 11:27:31.810 [initandlisten] connection accepted from 192.168.10.182:20779 #774706 (24 connections now open)
Fri Nov 8 11:27:31.855 [initandlisten] connection accepted from 192.168.10.161:28524 #774707 (25 connections now open)
Fri Nov 8 11:27:31.869 [initandlisten] connection accepted from 192.168.10.182:20786 #774708 (26 connections now open)
and after many "can create new thread"
[root#mongofile1 /usr/mongodb]# tail -n 15000 mongod.log.old |grep "create new thread"|wc
5020 55220 421680
and finish by a magnificent
Fri Nov 8 11:30:22.333 [rsMgr] replSet warning caught unexpected exception in electSelf()
pure virtual method called
Fri Nov 8 11:30:22.333 Got signal: 6 (Abort trap: 6).
Fri Nov 8 11:30:22.337 Backtrace:
0x599efc 0x8035cb516
0x599efc <_ZN5mongo10abruptQuitEi+988> at /usr/local/bin/mongod
0x8035cb516 <_pthread_sigmask+918> at /lib/libthr.so.3
Extract of mongodb from top
78126 mongodb 77 20 0 1253G 1449M sbwait 0 0:20 0.00% mongod
If i restart the process when it crash, the problem is fixed for almost 3 days ;-) ( Windows Method is not the solution ...)
Do you already see that ?
The fact that the error message says "pure virtual method called" would indicate that this is at least a bug in the mongodb code. The code that generates that message is here, I think.
I've used mongodb a few days, it has been working fine until today.
I got a error when querying a database.
Wed Nov 2 17:56:44 [initandlisten] connection accepted from 127.0.0.1:49234 #1
Wed Nov 2 17:56:45 [conn1] command admin.$cmd command: { listDatabases: 1 } ntoreturn:1 reslen:338 813ms
Wed Nov 2 17:56:46 [conn1] admin.system.namespaces Assertion failure isOk() db/pdfile.h 300
0x10007d703 0x100087eec 0x1004165e4 0x10040774b 0x100410823 0x100437df9 0x1004b3006 0x100426693 0x100426b53 0x100429f45 0x10042d8f0 0x10043109e 0x10043353b 0x10043662a 0x1004a13ca 0x1003d5e0f 0x1003e298a 0x100603394 0x100148d8c 0x10060a554
0 mongod 0x000000010007d703 _ZN5mongo12sayDbContextEPKc + 195
1 mongod 0x0000000100087eec _ZN5mongo8assertedEPKcS1_j + 300
2 mongod 0x00000001004165e4 _ZN5mongo11DataFileMgr9getExtentERKNS_7DiskLocE + 148
3 mongod 0x000000010040774b _ZN5mongo11DataFileMgr7findAllEPKcRKNS_7DiskLocE + 75
4 mongod 0x0000000100410823 _ZN5mongo13findTableScanEPKcRKNS_7BSONObjERKNS_7DiskLocE + 131
5 mongod 0x0000000100437df9 _ZNK5mongo9QueryPlan9newCursorERKNS_7DiskLocEi + 1081
6 mongod 0x00000001004b3006 _ZN5mongo11UserQueryOp5_initEv + 214
7 mongod 0x0000000100426693 _ZN5mongo7QueryOp4initEv + 323
8 mongod 0x0000000100426b53 _ZN5mongo12QueryPlanSet6Runner6initOpERNS_7QueryOpE + 35
9 mongod 0x0000000100429f45 _ZN5mongo12QueryPlanSet6Runner4initEv + 245
10 mongod 0x000000010042d8f0 _ZN5mongo12QueryPlanSet6Runner22runUntilFirstCompletesEv + 32
11 mongod 0x000000010043109e _ZN5mongo12QueryPlanSet5runOpERNS_7QueryOpE + 334
12 mongod 0x000000010043353b _ZN5mongo16MultiPlanScanner9runOpOnceERNS_7QueryOpE + 1115
13 mongod 0x000000010043662a _ZN5mongo16MultiPlanScanner5runOpERNS_7QueryOpE + 26
14 mongod 0x00000001004a13ca _ZN5mongo8runQueryERNS_7MessageERNS_12QueryMessageERNS_5CurOpES1_ + 3194
15 mongod 0x00000001003d5e0f _ZN5mongo13receivedQueryERNS_6ClientERNS_10DbResponseERNS_7MessageE + 623
16 mongod 0x00000001003e298a _ZN5mongo16assembleResponseERNS_7MessageERNS_10DbResponseERKNS_11HostAndPortE + 1402
17 mongod 0x0000000100603394 _ZN5mongo16MyMessageHandler7processERNS_7MessageEPNS_21AbstractMessagingPortEPNS_9LastErrorE + 148
18 mongod 0x0000000100148d8c _ZN5mongo3pms9threadRunEPNS_13MessagingPortE + 700
19 mongod 0x000000010060a554 thread_proxy + 132
Wed Nov 2 17:56:46 [conn1] assertion 0 assertion db/pdfile.h:300 ns:admin.system.namespaces query:{}
Wed Nov 2 17:56:51 [conn1] end connection 127.0.0.1:49234
I restart the mongod process but no help.
I tried to repair the db but also got the error.
bin: $ ./mongod --repair --dbpath ../data/db
Wed Nov 2 18:10:51 [initandlisten] MongoDB starting : pid=678 port=27017 dbpath=../data/db 64-bit host=KTs-MacBook-Pro.local
Wed Nov 2 18:10:51 [initandlisten] db version v2.0.0, pdfile version 4.5
Wed Nov 2 18:10:51 [initandlisten] git version: 695c67dff0ffc361b8568a13366f027caa406222
Wed Nov 2 18:10:51 [initandlisten] build info: Darwin erh2.10gen.cc 9.6.0 Darwin Kernel Version 9.6.0: Mon Nov 24 17:37:00 PST 2008; root:xnu-1228.9.59~1/RELEASE_I386 i386 BOOST_LIB_VERSION=1_40
Wed Nov 2 18:10:51 [initandlisten] options: { dbpath: "../data/db", repair: true }
Wed Nov 2 18:10:51 [initandlisten] journal dir=../data/db/journal
Wed Nov 2 18:10:51 [initandlisten] recover : no journal files present, no recovery needed
Wed Nov 2 18:10:51 [initandlisten] ****
Wed Nov 2 18:10:51 [initandlisten] ****
Wed Nov 2 18:10:51 [initandlisten] need to upgrade database admin with pdfile version 4.5, new version: 4.5
Wed Nov 2 18:10:51 [initandlisten] starting upgrade
Wed Nov 2 18:10:51 [initandlisten] admin repairDatabase admin
Wed Nov 2 18:10:51 [initandlisten] removeJournalFiles
Wed Nov 2 18:10:51 [initandlisten] admin.system.namespaces Assertion failure isOk() db/pdfile.h 300
0x10007d703 0x100087eec 0x1004165e4 0x10040774b 0x100410823 0x100437df9 0x1004b3006 0x100426693 0x100426b53 0x100429f45 0x10042d8f0 0x10043109e 0x10043353b 0x10043662a 0x1004a13ca 0x1003d5e0f 0x1003e298a 0x1003e480f 0x1000ed8ec 0x1000c8b73
0 mongod 0x000000010007d703 _ZN5mongo12sayDbContextEPKc + 195
1 mongod 0x0000000100087eec _ZN5mongo8assertedEPKcS1_j + 300
2 mongod 0x00000001004165e4 _ZN5mongo11DataFileMgr9getExtentERKNS_7DiskLocE + 148
3 mongod 0x000000010040774b _ZN5mongo11DataFileMgr7findAllEPKcRKNS_7DiskLocE + 75
4 mongod 0x0000000100410823 _ZN5mongo13findTableScanEPKcRKNS_7BSONObjERKNS_7DiskLocE + 131
5 mongod 0x0000000100437df9 _ZNK5mongo9QueryPlan9newCursorERKNS_7DiskLocEi + 1081
6 mongod 0x00000001004b3006 _ZN5mongo11UserQueryOp5_initEv + 214
7 mongod 0x0000000100426693 _ZN5mongo7QueryOp4initEv + 323
8 mongod 0x0000000100426b53 _ZN5mongo12QueryPlanSet6Runner6initOpERNS_7QueryOpE + 35
9 mongod 0x0000000100429f45 _ZN5mongo12QueryPlanSet6Runner4initEv + 245
10 mongod 0x000000010042d8f0 _ZN5mongo12QueryPlanSet6Runner22runUntilFirstCompletesEv + 32
11 mongod 0x000000010043109e _ZN5mongo12QueryPlanSet5runOpERNS_7QueryOpE + 334
12 mongod 0x000000010043353b _ZN5mongo16MultiPlanScanner9runOpOnceERNS_7QueryOpE + 1115
13 mongod 0x000000010043662a _ZN5mongo16MultiPlanScanner5runOpERNS_7QueryOpE + 26
14 mongod 0x00000001004a13ca _ZN5mongo8runQueryERNS_7MessageERNS_12QueryMessageERNS_5CurOpES1_ + 3194
15 mongod 0x00000001003d5e0f _ZN5mongo13receivedQueryERNS_6ClientERNS_10DbResponseERNS_7MessageE + 623
16 mongod 0x00000001003e298a _ZN5mongo16assembleResponseERNS_7MessageERNS_10DbResponseERKNS_11HostAndPortE + 1402
17 mongod 0x00000001003e480f _ZN5mongo14DBDirectClient4callERNS_7MessageES2_bPSs + 111
18 mongod 0x00000001000ed8ec _ZN5mongo14DBClientCursor4initEv + 76
19 mongod 0x00000001000c8b73 _ZN5mongo12DBClientBase5queryERKSsNS_5QueryEiiPKNS_7BSONObjEii + 739
Wed Nov 2 18:10:51 [initandlisten] assertion 0 assertion db/pdfile.h:300 ns:admin.system.namespaces query:{}
Wed Nov 2 18:10:51 [initandlisten] admin clone failed for admin with error: query failed admin.system.namespaces
Wed Nov 2 18:10:51 [initandlisten] removeJournalFiles
Wed Nov 2 18:10:51 [initandlisten] admin Assertion failure doDBUpgrade( dbName , errmsg , h ) db/db.cpp 298
0x10007d703 0x100087eec 0x1005f4f2f 0x1005f5c3c 0x1005f6274 0x1005f7d6f 0x1000013c4
0 mongod 0x000000010007d703 _ZN5mongo12sayDbContextEPKc + 195
1 mongod 0x0000000100087eec _ZN5mongo8assertedEPKcS1_j + 300
2 mongod 0x00000001005f4f2f _ZN5mongo30repairDatabasesAndCheckVersionEv + 2063
3 mongod 0x00000001005f5c3c _ZN5mongo14_initAndListenEi + 1036
4 mongod 0x00000001005f6274 _ZN5mongo13initAndListenEi + 36
5 mongod 0x00000001005f7d6f main + 3647
6 mongod 0x00000001000013c4 start + 52
Wed Nov 2 18:10:51 [initandlisten] exception in initAndListen: 0 assertion db/db.cpp:298, terminating
Wed Nov 2 18:10:51 dbexit:
Wed Nov 2 18:10:51 [initandlisten] shutdown: going to close listening sockets...
Wed Nov 2 18:10:51 [initandlisten] shutdown: going to flush diaglog...
Wed Nov 2 18:10:51 [initandlisten] shutdown: going to close sockets...
Wed Nov 2 18:10:51 [initandlisten] shutdown: waiting for fs preallocator...
Wed Nov 2 18:10:51 [initandlisten] shutdown: lock for final commit...
Wed Nov 2 18:10:51 [initandlisten] shutdown: final commit...
Wed Nov 2 18:10:51 [initandlisten] shutdown: closing all files...
Wed Nov 2 18:10:51 [initandlisten] closeAllFiles() finished
Wed Nov 2 18:10:51 [initandlisten] journalCleanup...
Wed Nov 2 18:10:51 [initandlisten] removeJournalFiles
Wed Nov 2 18:10:51 [initandlisten] shutdown: removing fs lock...
Wed Nov 2 18:10:51 dbexit: really exiting now
The mongodb version is v2.0.0, pdfile version is 4.5, git version is 695c67dff0ffc361b8568a13366f027caa406222.
There are 5 databases in the mongodb, but only one database to be wrong.
I also tried to recover data by mongodump, but got empty bson files.
Any ideas? Thanks
I was able to resolve this issue by going to my data directory (which in the example above would be
/data/db
and deleting all the jicd.x and local.x files
If the lock file is not 0 bytes in size you may want to delete that one as well.
The mongodb JIRA ticket link is https://jira.mongodb.org/browse/SERVER-4198
As the comments in the ticket, fix the issue by removing the master=true setting then running a repair.
I have not tried this method, just as a reference here.