I have a three-node replica set (1 Primary, 1 Secondary, 1 Arbiter), on three different Amazon server instances. The servers where they are hosted required a memory upgrade so I needed to shut down the MongoDB instances as well.
I shut down the MongoDB instances in this order:
Secondary
Arbiter
Primary
I used the process below for shutting down each server
use admin
db.shutdownServer()
All MongoDB instances did shut down properly without any problems. So far everything is fine.
After the Amazon server upgrade, I started the MongoDB instances in the following order:
Arbiter
Secondary
Primary
The arbiter is in arbiter mode and secondary is in secondary mode, but to my surprise the primary machine went to "RECOVERING" mode.
I don't know the reason, why the primary machine went to "RECOVERING".
I have examined logs. It is showing no member to sync...something stuff like that
My basic doubt is "PRIMARY has to be in PRIMARY until there is reconfig happens in replica set".
Am I missing a step during the shutdown of servers? Or am I missing a step during the restart of servers?
Please shed some light on this so that how can I overcome this problem. I need to shut down the MongoDB servers frequently since there is a lot of upgrades happening in Amazon servers.
After you started you replica set, you "SECONDARY" became "PRIMARY" and you "PRIMARY" was probably at secondary state after short while. To keep primary status at your "PRIMARY", you must give it higher priority than what your "SECONDARY" have.
Check with rs.conf() command.
Check here how to force node to be primary
Related
I would like to have Mongo on a server, then to replicate this onto a laptop.
The laptop is needed to leave the network and still be able to read/write, and once back on the network sync these changes with the primary.
At the same time I need the VM (Primary), to still be accessible (read/write).
So when each device is not talking to each other then for them to make themselves primary.
I have set up a very basic replica, Primary on a VM and the Secondary on the machine running the VM. In all examples I have seen it recommends having 3 servers for the replica, but I only need 2!
A couple of questions:
Is this possible with Mongo? If not then any suggestions!
When I turn off the Network adapter on the VM(Primary), the secondary doesn't seem to want to become the primary.
Is it possible to run 2 instance of Mongo, and then use the other instance as the 3 member of the replica.
Any advice would be great, thanks.
MongoDB always needs even number servers. in your case it should be like one primary one secondary and one arbiter. arbiter server instance is responsible for electing new server as primary when current primary goes down.
Mongodb docs state
Do not run an arbiter on systems that also host the primary or the secondary members of the replica set.
However I could not find any explanation for this. Is it for preventing the arbiter go down together with a secondary or primary when a failure occurs ?
Technical it is possible to run a setup like this, but you lost redundancy.
Let's say you have a server with AB and C where B is an arbiter running on the same server as A. If this server goes down, you've lost your majority and B can't elect a new primary. So if the wrong server goes down you have no redundancy.
Fortunately arbiters don't save any data, so a small and cheap server instance is enough to run them.
I've set up 2 machines running MongoDB ( O.S. Windows ) and i've set the replication. I was wondering if it's possible to kill the process that is running as primary to check if the secondary will turn as primary.
Is it possible? How? I just killed the primary process and on my mongo client ( that is connected to primary ), it didn't turned to secondary server.
Thank in advance.
You will need to set up a third server as an arbitrer.
The arbitrer is the one that determinates which of the other servers will be primary.
Usually, it is recomended that each replica set have an odd number of servers(the miniumun would be 1 primary, 1 secondary, 1 arbitrer)
This might help you.
http://docs.mongodb.org/manual/tutorial/add-replica-set-arbiter/
The arbitrer can be in a very tiny server, since it does not stores data, its only function is to decide.
You should have at least a 3 members configuration.
Replica documentation
Set a correct replica, connect your mongoshell to the primary instance and shut it down (Shutdown), connect your mongoshell with a secondary and check replica set's status (RS Status).
That's all !
The easiest way is to shutdown your primary (assuming that primarydb is your primary and secondaryDB is your secondary. The process would be as follow
Connect to your primary and secondaries.
Stop the primary.
Look at your secondaries if a new primary is elected.
Connect to your primary in the shell
conn1 = new Mongo("localhost:31000")
primaryDB = conn1.getDB("test")
Connect to your secondaries
conn2 = new Mongo("localhost:31001")
secondaryDB = conn2.getDB("test")
...
Shutdown your primary
primaryDB.adminCommand({"shutdown" : 1})
Check if one of the secondaries is elected as the primary
secondaryDB.isMaster()
In the result of the last operation you will see which server is now the new primary (look out for the primary key).
As mentioned in the comments, you should have at least three servers (one primary, two secondaries) to get the election process working.
I am experimenting the mongodb replica set, in which I started localhost:27017 as secondary and localhost:27018 as primary.
then I disconnect the localhost:27018, and I expected localhost:27017 to automatically become a primary, but it doesn't work as expected. In the shell, using command like rs.add() or rs.remove() gives out a error like ' "errmsg" : "replSetReconfig command must be sent to the current replica set primary.",'
I know this error is because the command is now running in the secondary, but since the primary is closed down already. what steps I should do now?
And also why closing down the primary doesn't allow the replica set to vote for a new primary ? what is the flexible way to make it automatic vote for a new primary
To elect a new primary in MongoDB replica set the majority of the members must vote for the new primary. That's majority of the original (or configured) members.
You configured two members - one is not a majority. You need to add a third member, either a regular node or an arbiter, if you want your failover to happen automatically.
My test system (due to lack of resources) has a dual mongodb replicaset. There is no arbiter.
During some system changes one of the servers got put out of action and will not be coming back. This server happened to host the primary mongo node. This left the only other member of the set as a secondary.
I know I should have had at least three nodes for the cluster (our prod setup does).
Is there a way I can make the primary that is now offline to step down? I haven't been able to change any of the rs.conf() settings because the only working node is secondary. starting an arbiter doesn't seem to work because I cannot add it to the replset as the primary is down.
Has anyone encountered this before and managed to resolve it?
To recap:
SERVER A (PRIMARY) - OFFLINE
SERVER B (SECONDARY) - ONLINE
A + B = REPLSET
Any help would be greatly appreciated.
The mongodb website has documentation for what to do (in an emergency only) when you need to reconfigure a replica set when members are down. This sounds like the situation you are in.
Basically, if you're on version >= 2.0, and it's an emergency, you can add force: true to the replica set configuration command.