I have 3 member replicaSet in MongoDB which fell apart when re-configuring the host names of the sever instances. I had to reconfigure the replicaSet, however I am curious how MongoDB handles records that are not synced across all the members.
Case 1) There is a new record on the MongoDB server that I access to reconfigure the set.
Case 2) There is a new record on another MongoDB server that is added later to the replica set.
Each replica-set has one primary node and one or more secondary nodes.
All writes happen on the primary. The primary then sends these changes to the secondaries (the list of changes is referred to as "the oplog"). That means the primary is always the member with the most up-to-date data.
When the primary is suddenly unreachable, the replica-set is put into read-only mode and an election takes place to find a new primary. Usually the secondary which is most up-to-date is selected (more details on replica-set election). Any writes which were not propagated to that secondary yet are lost.
When the old primary goes back online, it re-joins the replica-set as a secondary. Its data gets synchronized to the state of the new primary. Any writes which only happened on the old primary which weren't propagated to the new primary before the crash are rolled back.
The rolled-back writes are backed up as bson-files in the directory /rollback and can be re-added to the replica-set using bsondump and mongorestore. Details about this procedure can be found in the article Rollbacks During Replica Set Failover
Related
I have a mongodb replica set with one primary and two secondary, primary were down for one day, due to some os issues. Then one of the secondary elected as primary. Used that secondary as primary and used in application. After one day, old primary turned up and currently it is using as secondary. It seems only current data are updating in this secondary not synching the data of the downtime day. Is there any way to know, it is synching in background? if not synching what i have to do? I need to convert old primary again as primary, it can be done only if all data get synched.
If your old PRIMARY become SECONDARY this mean it successfully replicated the missed data from your new PRIMARY ... , exception is made only for the writes that exist in your old PRIMARY immediately before the election not able to replicate to your new PRIMARY. Those writes you can locate in your "rollback" folder in your old PRIMARY in bson format , you will need to review those bson files and check manually if you need to re-insert or consider them again.
Troubleshooting steps to follow:
Check the current replicaSet status with:
rs.status()
If you have 1x PRIMARY + 2x SECONDARY it seems everything to be fine , your members are in sync to the PRIMARY.
If some of the members are in different state they maybe init syncing and you will need to wait for some time until the process finish or there is other problem so you may need to wait abit.
If for some reason the members do not init sync succesfully , you can force fresh init sync so they try from the scratch.
Here are simple steps to start init sync:
4.1. Stop the member:
mongo --p XXX
use admin
db.shutdownServer()
4.2. Go the data folder and remove anything inside:
cd /my_member_data_folder/
rm -rf *
4.3. Start the member again and wait until init sync and successfully move to SECONDARY state.
mongod --config my_config_file.conf
In case everything is fine and you just need to switch the replicaSet to your old PRIMARY you may do so by just re-configure the priority of this member as follow:
newPRIMARY>var x=rs.conf()
newPRIMARY>x.members[0].priority=10
newPRIMARY>rs.reconfig(x)
Supposing members[0] is the old PRIMARY , and the rest of member they have the priority < 10
Consider the below diagram in MongoDB
I have two scenarios
Scenario 1 :-
Router directs the write call to master.Its writen to master but then it goes down before it gets replicted to slaves(i am using
synch replication mode)
Will router select one slave as master and also write the above request to both slaves ?
Scenario 2 :-
Router directs the write call to master. Its writen to master but then network link b/w it and one slave is broken(using
synch replication mode)
Will router select another slave(which is connected to all other nodes) as master and also write the above request to slave ?
Let's first use MongoDB terminologies: Primary instead of master and Secondary instead of slave.
Scenario 1: Will router select one slave as master and also write the above request to both slaves ?
A secondary can become a primary. If the current primary becomes unavailable, the replica set holds an election to choose which of the secondaries becomes the new primary. See also Replica Set Elections.
In scenario 1, if the primary had accepted write operations that the secondaries had not successfully replicated before the primary stepped down, then a rollback will revert the write operations on a former primary when the node rejoins the replica set. See also RollBacks During Replica Set Failover.
You can run all voting members with journaling enabled and use writeConcern majority to prevent rollbacks. See also Avoid Replica Set Rollbacks.
Scenario 2: Will router select another slave(which is connected to all other nodes) as master and also write the above request to slave ?
There are two parts here, the first part is replica set election. In this case because the primary and one of the secondaries are still majority, no election will be held. The primary will still be primary and replicating to one of the secondaries.
The second part is about replication of data. Secondary members copy oplog from their sync source and apply these operations in an asynchronous process. A secondary sync source may automatically change as needed based on changes in the ping time and state of other members’ replication. See also Replica Set Data Synchronization
In scenario 2, the secondary may change its sync source to the other secondary.
You may also found the following useful:
Replica Set High Availability
Replica Set Deployment Architectures
Replica Set Distributed Across Two or More Data Centers
I would like to have Mongo on a server, then to replicate this onto a laptop.
The laptop is needed to leave the network and still be able to read/write, and once back on the network sync these changes with the primary.
At the same time I need the VM (Primary), to still be accessible (read/write).
So when each device is not talking to each other then for them to make themselves primary.
I have set up a very basic replica, Primary on a VM and the Secondary on the machine running the VM. In all examples I have seen it recommends having 3 servers for the replica, but I only need 2!
A couple of questions:
Is this possible with Mongo? If not then any suggestions!
When I turn off the Network adapter on the VM(Primary), the secondary doesn't seem to want to become the primary.
Is it possible to run 2 instance of Mongo, and then use the other instance as the 3 member of the replica.
Any advice would be great, thanks.
MongoDB always needs even number servers. in your case it should be like one primary one secondary and one arbiter. arbiter server instance is responsible for electing new server as primary when current primary goes down.
We have sharded cluster with 2 secondaries on each shard.
Due to space problem one of the secondaries got corrupted.
In order to add new node to the existing shard we have removed data directories on the problematic secondary data node.
Then added new data node using rs.config into existing replica set.
We have around 1.2TB data.
I can see the data folder size is increasing so it proves that its synchronizing from the primary shard.
When I do rs.status() the replica set member shows that the new node is in STARTUP2 mode
Also it shows otptitime as
"optimeDate" : ISODate("1970-01-01T00:00:00Z"),
Data node is able to see primary node as checked from "lastHeartbeatRecv"
We are using Amazon AWS.
Please advise if there are any different method to add new data node with fast sync from Primary as the data is 1.2TB and we have kicked of the sync process 7 days before.
Copy the recent snapshot of the good secondary one into the problematic secondary node after cleaning the data directories. Then add this node into replica set. Let the oplog be applied automatically and synch with primary node. This way synchronization time will be reduced as secondary has to just catch up the backlog from the day when snapshot of the good secondary node is taken.
I am experimenting the mongodb replica set, in which I started localhost:27017 as secondary and localhost:27018 as primary.
then I disconnect the localhost:27018, and I expected localhost:27017 to automatically become a primary, but it doesn't work as expected. In the shell, using command like rs.add() or rs.remove() gives out a error like ' "errmsg" : "replSetReconfig command must be sent to the current replica set primary.",'
I know this error is because the command is now running in the secondary, but since the primary is closed down already. what steps I should do now?
And also why closing down the primary doesn't allow the replica set to vote for a new primary ? what is the flexible way to make it automatic vote for a new primary
To elect a new primary in MongoDB replica set the majority of the members must vote for the new primary. That's majority of the original (or configured) members.
You configured two members - one is not a majority. You need to add a third member, either a regular node or an arbiter, if you want your failover to happen automatically.