My MongoDB had crashed due to out of memory error that occurred when it tried appending to a journal file.
At that instance, my mongod.lock file was empty. I restarted mongod without any options. It was accepting connections normally. Then I ran mongo.exe, but was unable to connect to db. It got stuck to "connecting to test" but never connected successfully.
I ended that process and I restarted mongod with --nojournal option. But that didnt help either.
But now I see mongod.lock file non empty. Also,all my journal entries are deleted.
The question is, does --noJournal option deletes existing journal entries? Also, is there a way to recover the journal entries?
Recovering after a crash
First, please read this article:
Recover Data after an Unexpected Shutdown
After a crash, you have two options:
if it is a standalone instance, run mongod with the --repair option;
if the instance is a part of a replica set, wipe all data and either restore from a backup or perform an initial sync from another replica set member.
The --nojournal option
Running mongod --nojournal will not remove journal files. In fact, mongod will not even start if there are journal files present. It will give you the following message and shut down.
Error: journal files are present in journal directory, yet starting without journaling enabled.
It is recommended that you start with journaling enabled so that recovery may occur.
exception in initAndListen: 13597 can't start without --journal enabled when journal/ files are present, terminating
If you then run mongod without the --nojournal option, it will apply all changes saved in journal files and remove the files. Only then can you restart it with --nojournal.
I believe this is what happened in your case. You don't need to attempt to restore your journal files, as they are already applied to your data.
Related
My project is an app for a service-based company, as well as a general-purpose full stack boilerplate/template. The app has been up and running for months in development. This issue occurred initially 2 days ago.
In order to start my replica set, I first open 6 separate command prompt windows as admin. I then set the dbpath etc. for the primary:
mongod --dbpath "C:\Program Files\MongoDB\Server\5.0\replicaComplex2\rep01\data" --logpath "C:\Program Files\MongoDB\Server\5.0\replicaComplex1\rep01\log\mongod.log" --port 30000 --storageEngine=wiredTiger --journal --replSet jupiter_rep1
Then in a new terminal I set the config:
mongo --port 30000
rsconfig={_id:"jupiter_rep1",members:[{_id:0,host:"localhost:30000"}]}
I usually have to reconf in order to set the primary:
----reconf
rsconf = rs.conf()
rsconf.members = [{_id: 0, host: "localhost:30000"}]
rs.reconfig(rsconf, {force: true})
I then initialize the replica set:
rs.initiate(rsconfig)
Then I go to a new terminal/prompt and set the dbpath for the other two nodes:
mongod --dbpath "C:\Program Files\MongoDB\Server\5.0\replicaComplex2\rep02\data" --logpath "C:\Program Files\MongoDB\Server\5.0\replicaComplex1\rep02\log\mongod.log" --port 30001 --storageEngine=wiredTiger --journal --replSet jupiter_rep1
And the same for the third node.
However, this is where I am running into the problem. At this point the secondary nodes close out before I can actually set them as secondary.
I first tried to close all the windows and restart the actual machine. Nope.
Second, I uninstalled mongo dB and reinstalled it. Nope.
Third, I started the two secondary nodes prior to the primary or initialization. When i start the primary and initialize the replica set the secondary nodes shut down.
I'm on windows...
I also have the MongoDB Server stopped.
Input is appreciated!
Update*
I didn't include the log file error in my original question. They are separated for easier reading.
{"t":{"$date":"2022-04-15T16:05:44.353-05:00"},"s":"I",
"c":"ROLLBACK", "id":21606, "ctx":"BackgroundSync","msg":"Finding
common point"}
{"t":{"$date":"2022-04-15T16:05:44.353-05:00"},"s":"I", "c":"-",
"id":4939300, "ctx":"monitoring-keys-for-HMAC","msg":"Failed to
refresh key
cache","attr":{"error":"ReadConcernMajorityNotAvailableYet: Read
concern majority reads are currently not
possible.","nextWakeupMillis":800}}
{"t":{"$date":"2022-04-15T16:05:44.377-05:00"},"s":"I",
"c":"ROLLBACK", "id":21607, "ctx":"BackgroundSync","msg":"Rollback
common
point","attr":{"commonPointOpTime":{"ts":{"$timestamp":
{"t":1649857370,"i":1}},"t":149}}}
{"t":{"$date":"2022-04-15T16:05:44.378-05:00"},"s":"F",
"c":"ROLLBACK", "id":51121, "ctx":"BackgroundSync","msg":"Common
point must be at least stable
timestamp","attr":{"commonPoint":{"$timestamp":
{"t":1649857370,"i":1}},"stableTimestamp":{"$timestamp":
{"t":1649857964,"i":1}}}}
{"t":{"$date":"2022-04-15T16:05:44.378-05:00"},"s":"F", "c":"-",
"id":23091, "ctx":"BackgroundSync","msg":"Fatal
assertion","attr":{"msgid":51121,"file":"src\mongo\db\repl\rollback_impl.cpp","line":1146}}
{"t":{"$date":"2022-04-15T16:05:44.378-05:00"},"s":"F", "c":"-",
"id":23092, "ctx":"BackgroundSync","msg":"\n\n***aborting after
fassert() failure\n\n"}
Thanks!
There are several issues in your setup, thus I put them all in an answer, although it would be rather a comment.
Have a look at the logfiles C:\Program Files\MongoDB\Server\5.0\replicaComplex1\rep01\log\mongod.log and C:\Program Files\MongoDB\Server\5.0\replicaComplex1\rep02\log\mongod.log
They should show you the error message, I don't see any reason why the processes should stop.
Why do you have to have to reconf in order to set the primary? Does not make any sense. You should follow the Deploy a Replica Set tutorial.
I suggest to use a configuration file instead of command line options. Pay attention to the Windows Service Options
You should install MongoDB as a service, see mongod.exe, for example mongod --install --config C:\ProgramData\MongoDB\rep01\mongod.conf. The the services will start automatically at boot time.
Storage engine wiredTiger and enabled journal are default, you can skip these options.
As the name implies C:\Program Files\... is typically not the place where you store application data and logfiles. Usually in C:\Program Files\... you find only the binaries. Consider a different location, e.g. C:\ProgramData\MongoDB\... of C:\MongoDB\...
At the suggestion of #Wernfried Domscheit, I created a mongo replica set service and, in the process, I did find the issue.
To create the replica service, I used the following guide.
Apparently, the log and dB path in my three config files all had the same link to the first node of the set. This is how it was months ago when I first copied and pasted the first config file to create the other two. So, I am thinking my external hard drive disconnected and some memory issue occurred. So due to the log paths being the same I got the log path error.
Thanks.
*Will switch to a Bluetooth set-up.
When I am saving documents in monogdb, getting the following error
"{WriteConcern detected an error 'bad offset:0 accessing file:xxxx - consider
repairing database'. (Response was { \"err\" : \"bad offset:0 accessing file:
xxxxx - consider repairing database\", \"code\" : 13440, \"n\" : 0,
\"connectionId\" : 13, \"ok\" : 1.0 }).}"
any help please
As the error states, the database requires repairing because it is in an inconsistent state due to whatever operation that caused unexpected shutdown.
In MongoDB WriteConcern provides feedback/acknowledgement in response to a write operation. There are multiple levels of WriteConcerns or guarantees that on whether the write operation was successful or not. The WriteConcerns levels are as follow:
Unacknowledged (the client does not wait for acknowledgement of write op)
Acknowledged (client waits for success or exception ack. of write op)
Journaled (mongodb sends acknowledgment only after committing the write to journal)
Replica Acknowledged (ack. of write op on primary and/or other members of replica set)
You can set the levels of WriteConcerns to any level depending on importance of write operation.
The WriteConcern in your case returns an exception stating that database is in an inconsistent state may/may not data might have been corrupted. In order to get back to operational mode, you need to repair the database in one of many ways.
Via Mongodb shell (if you can connect to MongoDb)
Via command line with mongod see below
If you can get into MongoDB shell, then you need to find the database and repair it as follow
use dbName
db.repairDatabase() //repairs the above database
If you cannot get into MongoDB shell or wish to use other method then you can use the following methods
Repair Data files and preserve original files
mongod --dbpath /data/db --repair --repairpath /data/db0
mongod --dbpath /data/db0
Once the first command above complete, the newly repaired data will be under /data/db0 directory and you can use the second command to start mongodb using repaired data. The original data files are preserved in the default /data/db or a custom location (if not default)
Repair Data files WITHOUT preserve original files
This method you only supply --repair and no --repathpath and it will attempt to repair the original data files. First you have to delete mongod.lock file, steps are as follow:
rm /data/db/mongod.lock
Assuming your data is in /data/db directory, if not specify the location.
mongod --dbpath /data/db --repair
Start mongod with only --repair option, you don't have to pass --repairpath option. This will attempt to repair the data. Then finally start the database
mongod --dbpath /data/db
If you follow the first method of repair then you can delete the mongod.lock after the repair is successful. If you follow the second method then you can delete he mongod.lock before performing the repair operation.
Read the documentation how to safely shutdown the database, what is in mongod.lock? How you should do the repair (for example running it with same user in order to preserve the file ownership). It is also possible that the permissions to your data files has changed, you can chown them back, see this question for more details.
I try to start mongod.exe but I have and I get the following error:
C:\MongoDB\Server\30\bin>mongod.exe
2015-12-16T19:12:17.108+0100 I CONTROL 2015-12-16T19:12:17.110+0100 W CONTROL 32-bit servers don't have journaling enabled by default.
Please use --journal if you want durability.
2015-12-16T19:12:17.110+0100 I CONTROL
2015-12-16T19:12:17.120+0100 I CONTROL Hotfix KB2731284 or later update is not installed, will zero-out data files
2015-12-16T19:12:17.132+0100 I STORAGE [initandlisten] **************
2015-12-16T19:12:17.132+0100 I STORAGE [initandlisten] Error: journal files are present in journal directory, yet starting without journaling enabled.
2015-12-16T19:12:17.133+0100 I STORAGE [initandlisten] It is recommended that you start with journaling enabled so that recovery may occur.
2015-12-16T19:12:17.133+0100 I STORAGE [initandlisten] **************
2015-12-16T19:12:17.135+0100 I STORAGE [initandlisten] exception in initAndListen: 13597 can't start without --journal enabled when journal/ files are present, terminating
2015-12-16T19:12:17.135+0100 I CONTROL [initandlisten] dbexit: rc: 100
I also tried to run it with --repair but then I get the same error.
Finally, I tried to delete the mongod.lock file but I still get the error.
How should I fix the unclean shutdown?
The solution to this problem is mongod --repair. This command automatically shuts down the all processes and repairs Mongodb issues. You can find more details in the official documentation.
Ok, to get some confusion right here. Journal files are not there to annoy you. They hold data not yet applied to the datafiles, but already received and acknowledged by the server. The mongod process finished a request after applying the data to the journal, but before applying them to the data files.
This behavior is configured by the chosen write concern.
Bottom line: special measurements were taken to make the data in the journal durable, you should not ignore that.
So you should create a configuration file containing this (among other things, if one already exists):
storage:
journal:
enabled: true
Please follow the documentation on running MongoDB on windows to the letter. Adjust the configuration file with options according to your needs.
If you are absolutely, positively sure that you do not need journaling, you can start mongodb with the --journal command line option just once, shut the instance down after the journal was successfully applied and remove the journal files then. Expect any write with a write concern involving the journal to fail, however.
Note 1 You are using the 32-bit version of MongoDB, which is only suitable for testing. Note that the 32-bit version only supports up to 2Gb of data.
Note 2 MongoDB is VERY well documented. You really should read the manual from top to bottom – it get's you started fast enough with providing a lot of information on the internals.
start cmd shell as admin and call start_mongo. This should fix it
Same error.
It' permission issue. If you get this error on Windows platform you should do all operations with administrator privilegies.
On Linux run
mongod --repair
but you should run it with sudo or under root. If under root you should change ownership of the files in data DB directory. Do it carefully or another error will appear.
Try removing the lock file, then running with --repair.
Here's what the Mongo Docs say about recovering data / restarting after an unexpected shutdown.
I am running mongoDB on a local appliance machine, with journaling enabled.
Is it always guaranteed that mongo will recover itself automatically even on power outage(meaning that the database was not closed properly) when journaling is enabled?
On what scenarios MongoDB will be corrupted even if journaling is enabled(besides filesystem corruption)?
Yes, it is guaranteed (assuming no filesystem corruption):
With journaling enabled, MongoDB creates a journal subdirectory within the directory defined by dbPath, which is /data/db by default. The journal directory holds journal files, which contain write-ahead redo logs. The directory also holds a last-sequence-number file. A clean shutdown removes all the files in the journal directory. A dirty shutdown (crash) leaves files in the journal directory; these are used to automatically recover the database to a consistent state when the mongod process is restarted.
(Journaling /core/journaling in the manual)
This is a big point for journaling in the first place and one of the primary reason journaling is used. Note data will still likely be lost (from the last 100ms or so) but the DB will be in a consistent state.
I have a single machine that has MongoDB and its data is at /data/db as usual.
When my machine crashed due to an electric power trip, my MongoDB refuse to start at launch (Mac OS X Server via LaunchAgent) and also /data/db mysteriously disappear!
Also all log file are wipe out. This happen on my development SSD MBA and I thought is just a weird SSD case. But my XServe server is getting it as well when the power trip.
Am I missing some data protection articles somewhere? For sure it can't be this unreliable by just deleting /data/db!!??
MongoDB will never ever remove your database files!
In case of a crash you have to start mongod using the --repair option.
In addition: using the new journaling option of MongoDB in V 1.8+ that should help a lot when you run MongoDB as standalone service.
No that is not normal.
If it won't start, it's likely mongodb is indicating that you need to run a repair because mongod.lock is present and has a certain state in /data/db. But that would mean /data/db exists.
If /data/db exists but were empty (which in this case would be bad obviously), it would start right up.
If you log(s) are missing, sounds like a more general disk issue.
So check the startup message if about mongod.lock there is data there. Also with v1.8+ use journaling. (albeit you wouldn't lose all datafiles even without journaling)