I try to start mongod.exe but I have and I get the following error:
C:\MongoDB\Server\30\bin>mongod.exe
2015-12-16T19:12:17.108+0100 I CONTROL 2015-12-16T19:12:17.110+0100 W CONTROL 32-bit servers don't have journaling enabled by default.
Please use --journal if you want durability.
2015-12-16T19:12:17.110+0100 I CONTROL
2015-12-16T19:12:17.120+0100 I CONTROL Hotfix KB2731284 or later update is not installed, will zero-out data files
2015-12-16T19:12:17.132+0100 I STORAGE [initandlisten] **************
2015-12-16T19:12:17.132+0100 I STORAGE [initandlisten] Error: journal files are present in journal directory, yet starting without journaling enabled.
2015-12-16T19:12:17.133+0100 I STORAGE [initandlisten] It is recommended that you start with journaling enabled so that recovery may occur.
2015-12-16T19:12:17.133+0100 I STORAGE [initandlisten] **************
2015-12-16T19:12:17.135+0100 I STORAGE [initandlisten] exception in initAndListen: 13597 can't start without --journal enabled when journal/ files are present, terminating
2015-12-16T19:12:17.135+0100 I CONTROL [initandlisten] dbexit: rc: 100
I also tried to run it with --repair but then I get the same error.
Finally, I tried to delete the mongod.lock file but I still get the error.
How should I fix the unclean shutdown?
The solution to this problem is mongod --repair. This command automatically shuts down the all processes and repairs Mongodb issues. You can find more details in the official documentation.
Ok, to get some confusion right here. Journal files are not there to annoy you. They hold data not yet applied to the datafiles, but already received and acknowledged by the server. The mongod process finished a request after applying the data to the journal, but before applying them to the data files.
This behavior is configured by the chosen write concern.
Bottom line: special measurements were taken to make the data in the journal durable, you should not ignore that.
So you should create a configuration file containing this (among other things, if one already exists):
storage:
journal:
enabled: true
Please follow the documentation on running MongoDB on windows to the letter. Adjust the configuration file with options according to your needs.
If you are absolutely, positively sure that you do not need journaling, you can start mongodb with the --journal command line option just once, shut the instance down after the journal was successfully applied and remove the journal files then. Expect any write with a write concern involving the journal to fail, however.
Note 1 You are using the 32-bit version of MongoDB, which is only suitable for testing. Note that the 32-bit version only supports up to 2Gb of data.
Note 2 MongoDB is VERY well documented. You really should read the manual from top to bottom – it get's you started fast enough with providing a lot of information on the internals.
start cmd shell as admin and call start_mongo. This should fix it
Same error.
It' permission issue. If you get this error on Windows platform you should do all operations with administrator privilegies.
On Linux run
mongod --repair
but you should run it with sudo or under root. If under root you should change ownership of the files in data DB directory. Do it carefully or another error will appear.
Try removing the lock file, then running with --repair.
Here's what the Mongo Docs say about recovering data / restarting after an unexpected shutdown.
Related
MongoDB was working beautifully for me for several months until I had an unexpected shutdown a week or two ago. Since then, I've been getting the error in the title that snowballs into an invalid argument, then a library panic, then some fatal assertions which cause MongoDB to crash.
Now, I've done my research: the normal answers are to run the repair function and to make sure SELinux isn't screwing up the process. Neither of those have worked. The error gets thrown during WiredTiger's checkpoint process, so reads/writes to the database aren't the issue, and because it's during the checkpoint process, it guarantees that MongoDB won't stay up for more than a day.
To be clear: all the files in the database are owned by mongod:mongod, have permissions set to 600 (default, and I tried setting them to 755 to see if that fixed it, and it didn't). I'm running mongodb as a service on a CentOS 7 box, and the service file specifies that it should run as user mongod. The mongod.conf file specifies a mounted filesystem as the database, and it was happy with that until the unexpected shutdown. I'm running MongoDB version 4.0.1, so WiredTiger really doesn't like it if I disable Journaling either (disregarding the fact that I shouldn't disable it in the first place).
I feel like I've exhausted all my options, and that the only thing I can do is backup my data and reinstall MongoDB. Are there any that I've missed?
After creating a backup of my data via mongodump, shutting down mongo, removing the entire database with rm -rf 'path-to-database', rebooting mongo (without the replication config), and restoring the data with mongorestore, mongodb still crashes. This time, however, it's with an Invariant failure after the open: operation not permitted. The only conclusion I can think of is that the data itself has become corrupted in some way. Thankfully, this isn't "mission critical" data, so to speak, and I can easily obtain new data.
Unfortunately, this doesn't answer my original question of "what other options do I have?". However, I'm still posting this in case others run into this same kind of issue.
EDIT: invariant issue was caused by me forgetting to re-initialize my replication set. After fixing that, it's clean. Because of this, I no longer believe it was a data corruption issue, but a checkpoint corruption issue.
EDIT 2: So the issue arose again after about a week, and after another week of trying various debugging methods, I tried simply moving the mongo process to another server. So far, that's been working. The previous server was acting up (I couldn't even run top at one point - another process had a lock on a necessary library file to run it), so here's to hoping that the current server doesn't follow suite.
My MongoDB had crashed due to out of memory error that occurred when it tried appending to a journal file.
At that instance, my mongod.lock file was empty. I restarted mongod without any options. It was accepting connections normally. Then I ran mongo.exe, but was unable to connect to db. It got stuck to "connecting to test" but never connected successfully.
I ended that process and I restarted mongod with --nojournal option. But that didnt help either.
But now I see mongod.lock file non empty. Also,all my journal entries are deleted.
The question is, does --noJournal option deletes existing journal entries? Also, is there a way to recover the journal entries?
Recovering after a crash
First, please read this article:
Recover Data after an Unexpected Shutdown
After a crash, you have two options:
if it is a standalone instance, run mongod with the --repair option;
if the instance is a part of a replica set, wipe all data and either restore from a backup or perform an initial sync from another replica set member.
The --nojournal option
Running mongod --nojournal will not remove journal files. In fact, mongod will not even start if there are journal files present. It will give you the following message and shut down.
Error: journal files are present in journal directory, yet starting without journaling enabled.
It is recommended that you start with journaling enabled so that recovery may occur.
exception in initAndListen: 13597 can't start without --journal enabled when journal/ files are present, terminating
If you then run mongod without the --nojournal option, it will apply all changes saved in journal files and remove the files. Only then can you restart it with --nojournal.
I believe this is what happened in your case. You don't need to attempt to restore your journal files, as they are already applied to your data.
I am running mongoDB on a local appliance machine, with journaling enabled.
Is it always guaranteed that mongo will recover itself automatically even on power outage(meaning that the database was not closed properly) when journaling is enabled?
On what scenarios MongoDB will be corrupted even if journaling is enabled(besides filesystem corruption)?
Yes, it is guaranteed (assuming no filesystem corruption):
With journaling enabled, MongoDB creates a journal subdirectory within the directory defined by dbPath, which is /data/db by default. The journal directory holds journal files, which contain write-ahead redo logs. The directory also holds a last-sequence-number file. A clean shutdown removes all the files in the journal directory. A dirty shutdown (crash) leaves files in the journal directory; these are used to automatically recover the database to a consistent state when the mongod process is restarted.
(Journaling /core/journaling in the manual)
This is a big point for journaling in the first place and one of the primary reason journaling is used. Note data will still likely be lost (from the last 100ms or so) but the DB will be in a consistent state.
I am currently experimenting with MongoDB replica set mechanism.
I already have a working standalone Mongo server with a main database of about 20GB of data.
I decided to convert this mongo server to a primary replica set server, then added a 2nd machine with a similar configuration (but a newer mongo version), as a secondary replica set server.
This works fine, all data is replicated to the secondary as expected.
But I would like to perform some alteration operations on the data (because somehow, my data model has changed and I need to, for example rename some properties, or convert references to a simple ObjectId, some things like that). By the same time I would like to update the first server which has an old version (2.4) to the last version available (2.6).
So I decided to follow the instructions on the MongoDB website to perform maintenance on replica set members.
shut down the secondary server. (ok)
restart server as standalone on another port (both servers usually run on 27017)
mongod --dbpath /my/database/path --port 37017
And then, the server never restarts correctly and I get this:
2014-10-03T08:20:58.716+0200 [initandlisten] opening db: myawesomedb
2014-10-03T08:20:58.735+0200 [initandlisten] myawesomedb Assertion failure _name == nsToDatabaseSubstring( ns ) src/mongo/db/catalog/database.cpp 472
2014-10-03T08:20:58.740+0200 [initandlisten] myawesomedb 0x11e6111 0x1187e49 0x116c15e 0x8c2208 0x765f0e 0x76ab3f 0x76c62f 0x76cedb 0x76d475 0x76d699 0x7fd958c3eec5 0x764329
/usr/bin/mongod(_ZN5mongo15printStackTraceERSo+0x21) [0x11e6111]
/usr/bin/mongod(_ZN5mongo10logContextEPKc+0x159) [0x1187e49]
/usr/bin/mongod(_ZN5mongo12verifyFailedEPKcS1_j+0x17e) [0x116c15e]
/usr/bin/mongod(_ZN5mongo8Database13getCollectionERKNS_10StringDataE+0x288) [0x8c2208]
/usr/bin/mongod(_ZN5mongo17checkForIdIndexesEPNS_8DatabaseE+0x19e) [0x765f0e]
/usr/bin/mongod() [0x76ab3f]
/usr/bin/mongod(_ZN5mongo14_initAndListenEi+0x5df) [0x76c62f]
/usr/bin/mongod(_ZN5mongo13initAndListenEi+0x1b) [0x76cedb]
/usr/bin/mongod() [0x76d475]
/usr/bin/mongod(main+0x9) [0x76d699]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf5) [0x7fd958c3eec5]
/usr/bin/mongod() [0x764329]
2014-10-03T08:20:58.756+0200 [initandlisten] exception in initAndListen: 0 assertion src/mongo/db/catalog/database.cpp:472, terminating
2014-10-03T08:20:58.757+0200 [initandlisten] dbexit:
What am I doing wrong ?
Note that at this time, the first server is still running as primary member.
Thanks in advance!
I believe you are hitting a bug in VMWare here (can you confirm you are using VMWare VMs? confirmed) - I have seen it confirmed on Ubuntu and Fedora so far. The bug causes pieces of previous data to not be zero'ed out when creating the MongoDB namespace files (not always, but sometimes). That previous data essentially manifests as corruption in the namespace files and leads to the assertion you saw.
To work around the issue, there will be a fix released in MongoDB versions 2.4.12 and 2.6.5+ as part of SERVER-15369. The OS/Kernel level fix will eventually percolate down from the kernel bug and the Ubuntu patch, but that may take some time to actually be available as an official update (hence the need for the workaround change in MongoDB itself in the interim).
The issue will only become apparent when you upgrade to 2.6 because of additional checking added to that version that was not present in 2.4, however the corruption is still present, just not reported on version 2.4
If you still have your primary running, and it does not have the corruption, I would recommend syncing a secondary that is not on a VMWare VM and/or taking a backup of your files as soon as possible for safety - there is no automatic way to fix this corruption right now.
You can also look at using version 2.6.5 once it is released (2.6.5 rc4 is available as of writing this which includes the fix). You will still need to resync with that version off your good source to create a working secondary, but at least there will then be no corruption of the ns files.
Updates:
Version 2.6.5 which includes the fix mentioned was released on October 9th
Version 2.4.12 which includes the fix was released on October 16th
Official MongoDB Advisory: https://groups.google.com/forum/#!topic/mongodb-announce/gPjazaAePoo
I have a single machine that has MongoDB and its data is at /data/db as usual.
When my machine crashed due to an electric power trip, my MongoDB refuse to start at launch (Mac OS X Server via LaunchAgent) and also /data/db mysteriously disappear!
Also all log file are wipe out. This happen on my development SSD MBA and I thought is just a weird SSD case. But my XServe server is getting it as well when the power trip.
Am I missing some data protection articles somewhere? For sure it can't be this unreliable by just deleting /data/db!!??
MongoDB will never ever remove your database files!
In case of a crash you have to start mongod using the --repair option.
In addition: using the new journaling option of MongoDB in V 1.8+ that should help a lot when you run MongoDB as standalone service.
No that is not normal.
If it won't start, it's likely mongodb is indicating that you need to run a repair because mongod.lock is present and has a certain state in /data/db. But that would mean /data/db exists.
If /data/db exists but were empty (which in this case would be bad obviously), it would start right up.
If you log(s) are missing, sounds like a more general disk issue.
So check the startup message if about mongod.lock there is data there. Also with v1.8+ use journaling. (albeit you wouldn't lose all datafiles even without journaling)