My project is an app for a service-based company, as well as a general-purpose full stack boilerplate/template. The app has been up and running for months in development. This issue occurred initially 2 days ago.
In order to start my replica set, I first open 6 separate command prompt windows as admin. I then set the dbpath etc. for the primary:
mongod --dbpath "C:\Program Files\MongoDB\Server\5.0\replicaComplex2\rep01\data" --logpath "C:\Program Files\MongoDB\Server\5.0\replicaComplex1\rep01\log\mongod.log" --port 30000 --storageEngine=wiredTiger --journal --replSet jupiter_rep1
Then in a new terminal I set the config:
mongo --port 30000
rsconfig={_id:"jupiter_rep1",members:[{_id:0,host:"localhost:30000"}]}
I usually have to reconf in order to set the primary:
----reconf
rsconf = rs.conf()
rsconf.members = [{_id: 0, host: "localhost:30000"}]
rs.reconfig(rsconf, {force: true})
I then initialize the replica set:
rs.initiate(rsconfig)
Then I go to a new terminal/prompt and set the dbpath for the other two nodes:
mongod --dbpath "C:\Program Files\MongoDB\Server\5.0\replicaComplex2\rep02\data" --logpath "C:\Program Files\MongoDB\Server\5.0\replicaComplex1\rep02\log\mongod.log" --port 30001 --storageEngine=wiredTiger --journal --replSet jupiter_rep1
And the same for the third node.
However, this is where I am running into the problem. At this point the secondary nodes close out before I can actually set them as secondary.
I first tried to close all the windows and restart the actual machine. Nope.
Second, I uninstalled mongo dB and reinstalled it. Nope.
Third, I started the two secondary nodes prior to the primary or initialization. When i start the primary and initialize the replica set the secondary nodes shut down.
I'm on windows...
I also have the MongoDB Server stopped.
Input is appreciated!
Update*
I didn't include the log file error in my original question. They are separated for easier reading.
{"t":{"$date":"2022-04-15T16:05:44.353-05:00"},"s":"I",
"c":"ROLLBACK", "id":21606, "ctx":"BackgroundSync","msg":"Finding
common point"}
{"t":{"$date":"2022-04-15T16:05:44.353-05:00"},"s":"I", "c":"-",
"id":4939300, "ctx":"monitoring-keys-for-HMAC","msg":"Failed to
refresh key
cache","attr":{"error":"ReadConcernMajorityNotAvailableYet: Read
concern majority reads are currently not
possible.","nextWakeupMillis":800}}
{"t":{"$date":"2022-04-15T16:05:44.377-05:00"},"s":"I",
"c":"ROLLBACK", "id":21607, "ctx":"BackgroundSync","msg":"Rollback
common
point","attr":{"commonPointOpTime":{"ts":{"$timestamp":
{"t":1649857370,"i":1}},"t":149}}}
{"t":{"$date":"2022-04-15T16:05:44.378-05:00"},"s":"F",
"c":"ROLLBACK", "id":51121, "ctx":"BackgroundSync","msg":"Common
point must be at least stable
timestamp","attr":{"commonPoint":{"$timestamp":
{"t":1649857370,"i":1}},"stableTimestamp":{"$timestamp":
{"t":1649857964,"i":1}}}}
{"t":{"$date":"2022-04-15T16:05:44.378-05:00"},"s":"F", "c":"-",
"id":23091, "ctx":"BackgroundSync","msg":"Fatal
assertion","attr":{"msgid":51121,"file":"src\mongo\db\repl\rollback_impl.cpp","line":1146}}
{"t":{"$date":"2022-04-15T16:05:44.378-05:00"},"s":"F", "c":"-",
"id":23092, "ctx":"BackgroundSync","msg":"\n\n***aborting after
fassert() failure\n\n"}
Thanks!
There are several issues in your setup, thus I put them all in an answer, although it would be rather a comment.
Have a look at the logfiles C:\Program Files\MongoDB\Server\5.0\replicaComplex1\rep01\log\mongod.log and C:\Program Files\MongoDB\Server\5.0\replicaComplex1\rep02\log\mongod.log
They should show you the error message, I don't see any reason why the processes should stop.
Why do you have to have to reconf in order to set the primary? Does not make any sense. You should follow the Deploy a Replica Set tutorial.
I suggest to use a configuration file instead of command line options. Pay attention to the Windows Service Options
You should install MongoDB as a service, see mongod.exe, for example mongod --install --config C:\ProgramData\MongoDB\rep01\mongod.conf. The the services will start automatically at boot time.
Storage engine wiredTiger and enabled journal are default, you can skip these options.
As the name implies C:\Program Files\... is typically not the place where you store application data and logfiles. Usually in C:\Program Files\... you find only the binaries. Consider a different location, e.g. C:\ProgramData\MongoDB\... of C:\MongoDB\...
At the suggestion of #Wernfried Domscheit, I created a mongo replica set service and, in the process, I did find the issue.
To create the replica service, I used the following guide.
Apparently, the log and dB path in my three config files all had the same link to the first node of the set. This is how it was months ago when I first copied and pasted the first config file to create the other two. So, I am thinking my external hard drive disconnected and some memory issue occurred. So due to the log paths being the same I got the log path error.
Thanks.
*Will switch to a Bluetooth set-up.
I have upgraded MongodDB sharded cluster having two replica sets from 3.2 to 3.4. Current storage engine is MMAPv1. After successfully upgrading all the secondary, primary, config server and mongos to 3.4, when I run config server using following command.
sudo mongod --configsvr
I keep getting following Error.
SHARDING [shard registry reload] Periodic reload of shard registry failed :: caused by :: 115 could not get updated shard list from config server due to Current storage engine does not support majority readConcerns; will retry after 30s
And also I am unable to connect mongos with config server. When I try to connect it using following command
sudo mongos --configdb [ip-of-my-config-server]:27019
It gives me following error.
BadValue: configdb supports only replica set connection string
I suppose mongos is unable to connect to config server due to the majority readConcerns error on config server.
MongoDB manual says
"When reading from the replica set config servers, MongoDB 3.4 uses a Read Concern level of "majority"."
And to use a read concern level of "majority", WiredTiger must be used as storage engine.
So it seems I have to switch to WiredTiger storage engine to make it work. But when I was going to switch to WiredTiger storage engine of a secondary replica set member, according to manual
"This procedure completely removes a secondary replica set member’s data"
So I am stuck halfway. Situation is
Config server is giving error regarding majority readConcerns.
I have to switch to WiredTiger to get rid of it.
Switching to WiredTiger will remove data from secondary members.
Data will not be replicated back to secondary members during this switching to WiredTiger procedure because of config server error and eventually I will be ended up losing all the data (Please correct if I am wrong).
Questions are:
Can I make MongoDB 3.4 to use a Read Concern level of "local" when reading from the replica set config servers?
How can I switch to WiredTiger engine without losing data in my scenario?
You could migrate each node in the replica set as if it was a standalone, by using mongodump to back up the data, restarting with WiredTiger and a blank data directory, then using mongorestore to populate the new database.
This isn't normally recommended for replica set nodes, but only because it's just easier to wipe the data on a secondary and let it resync from the other nodes. Doing it this way will work just fine, but involves a little more fiddly work for you with the dump and restore tools.
What's the best/fastest/safest way to recover deleted files from ext4 ?
Specs:
The disk is 1TB SSHD (hibrid HDD + SSD), also the partition is encrypted with LUKS Encryption (version 1)
Mongodb is using WiredTiger as a storage engine.
Also if I manage a partial recovery of files, could I do a partial recovery of mongo's collections?
Step 1: File recovery
Fast Recovery of files using extundelete:
sudo umount /path/to/disk &&
sudo extundelete /path/to/disk --restore-directory /path/to/dir -o /restored/path/
/path/to/disk represents the disk path, e.g. /dev/sdd , /dev/mapping/label
/path/to/dir represents the path that you want recovered relative to disk mounting point, e.g. if /dev/ssd would be mounted at /mnt/label/ the full path would be /mnt/label/path/to/dir and the relative path is /path/to/dir
pros of recovery with extundelete:
it's lightweight
can work if the disk is mounted or encrypted
pretty fast, it gave answers if recovery is possible in seconds and it writes the recovered files with over 100 MB/s
cons for data recovery in general
no guarantee for success
won't work if new data was written in the deleted sectors (so unmount the disk as soon as possible and make an image of the broken disk before any recovery)
Step 2 : repair mongodb if missing data
Backup before this step, mongod --repair could delete good data
Untested, but from my understanding mongod --repair should help repair the database if incomplete otherwise you can continue recovery for WiredTiger with :
Recovering a WiredTiger collection from a corrupt mongodb installation
My MongoDB had crashed due to out of memory error that occurred when it tried appending to a journal file.
At that instance, my mongod.lock file was empty. I restarted mongod without any options. It was accepting connections normally. Then I ran mongo.exe, but was unable to connect to db. It got stuck to "connecting to test" but never connected successfully.
I ended that process and I restarted mongod with --nojournal option. But that didnt help either.
But now I see mongod.lock file non empty. Also,all my journal entries are deleted.
The question is, does --noJournal option deletes existing journal entries? Also, is there a way to recover the journal entries?
Recovering after a crash
First, please read this article:
Recover Data after an Unexpected Shutdown
After a crash, you have two options:
if it is a standalone instance, run mongod with the --repair option;
if the instance is a part of a replica set, wipe all data and either restore from a backup or perform an initial sync from another replica set member.
The --nojournal option
Running mongod --nojournal will not remove journal files. In fact, mongod will not even start if there are journal files present. It will give you the following message and shut down.
Error: journal files are present in journal directory, yet starting without journaling enabled.
It is recommended that you start with journaling enabled so that recovery may occur.
exception in initAndListen: 13597 can't start without --journal enabled when journal/ files are present, terminating
If you then run mongod without the --nojournal option, it will apply all changes saved in journal files and remove the files. Only then can you restart it with --nojournal.
I believe this is what happened in your case. You don't need to attempt to restore your journal files, as they are already applied to your data.
I am running mongoDB on a local appliance machine, with journaling enabled.
Is it always guaranteed that mongo will recover itself automatically even on power outage(meaning that the database was not closed properly) when journaling is enabled?
On what scenarios MongoDB will be corrupted even if journaling is enabled(besides filesystem corruption)?
Yes, it is guaranteed (assuming no filesystem corruption):
With journaling enabled, MongoDB creates a journal subdirectory within the directory defined by dbPath, which is /data/db by default. The journal directory holds journal files, which contain write-ahead redo logs. The directory also holds a last-sequence-number file. A clean shutdown removes all the files in the journal directory. A dirty shutdown (crash) leaves files in the journal directory; these are used to automatically recover the database to a consistent state when the mongod process is restarted.
(Journaling /core/journaling in the manual)
This is a big point for journaling in the first place and one of the primary reason journaling is used. Note data will still likely be lost (from the last 100ms or so) but the DB will be in a consistent state.