I've read quite a bit about mongodb replica set and how the elections work on a failover. My curiosity is assuming the client will use readPreference set to primary only, is there any advantage to having odd number of members against having even number of members + 1 arbiter?
For example if you have a 3 member replica set you can set all 3 members to be replicas or you can have only 2 replicas and an arbiter (which you can install on a smaller machine). The safety is basically the same, any machine can go down and the replica set is still ok, but if two of them go down then the replica set is in stalemate (it cannot elect a new primary).
The only difference is that in the second case you can use a way smaller machine for the arbiter.
It's actually not true that three data holding nodes provide the same "safety" net as two data holding nodes plus an arbiter.
Consider these cases:
1) one of your nodes loses its disk and you need to fully resync it. If you had three data holding nodes you can resync off of the other secondary, instead of the primary (which would reduce the load on the primary).
2) one of your nodes loses its disk and it takes you a while to located a new one. While that secondary is down you are running with ZERO safety net if you had two nodes and an arbiter since you only have one node with data left, if anything happens to it, you are toast.
Related
I want to use MongoDB with replication; I created a VM with 2 secondary nodes and 1 arbiter:
1 Primary
2 Secondary
1 Arbiter
I'm trying to understand how this system works, so I have some questions:
1) According to information "If a replica set has an even number of members, add an arbiter." I added an arbiter. But I'm not sure if I have done it correctly. Does this even number apply to secondaries or to all members in total?
2) What does this arbiter doing? I actually don't understand its job.
3) I created public IP addresses for each VM, in order to connect to them from outside. I successfully connected from my application, using this connection string:
mongodb://username:password#vm0:27017,vm1:27017,vm2:27017/dbname?replicaSet=xxx&readPreference=primaryPreferred
I didn't add the arbiter in this connection string but Should I add it or not?
4) When I shut down the primary machine, one of the secondary machine successfully became primary as I expect. There is no problem in this case; but when I shut down the second primary machine my application throws an error. The second secondary node has not become primary - why is this happening?
5) If all VMs are working but I shut down the arbiter, my application again throws an error and I cannot connect to the db. I'm trying this because I'm thinking the case of if there will be something wrong on arbiter machine and it may be shut down in the future because of the maintenance or any other problems.
Maybe because I didn't understand the role of an arbiter; I'm thinking this is wrong but why it is not converting any secondary machine to arbiter? And why when I shut down the arbiter does the whole system not work?
Thanks.
1) If you have 1 Primary and 2 Secondaries, you have 3 members in your replica set. Therefore you should not be adding an arbiter. You already have an odd number of nodes.
2) An arbiter is a node which doesn't hold data and can't be elected as Primary. It is only used to elect a new Primary if the current Primary goes down.
For example, say you have 1 Primary and 1 Secondary. The replica set has 2 members. If the primary goes down, the replica set will attempt to vote to elect a new Primary. In order for a node to be elected, it needs to win over half the votes. But if the Secondary votes for itself, it will only get 1 out of 2 votes. That's not more than half so it will not be elected. Thus the replica set will not be able to elect a new Primary and your whole replica set will go down.
To fix this, you can add an arbiter to the replica set. This is usually a much smaller machine since it doesn't need to hold data. It just has one job, voting for the Secondary to be the new Primary in the case of elections.
But, since you already have 3 data-bearing nodes, you won't want to add an arbiter. You can read more about arbiters here.
3) You can add arbiters to connection strings but in general you won't need to. Adding the data-bearing nodes is just fine. That's what people usually do.
4) You have 4 members in the replica set. You took down 2 of them. That means there are only 2 votes left. The final secondary won't be able to get more than 50% of the votes so no Primary will be elected.
In general, testing two nodes going down is overkill. You probably want a 3 member replica set. Each member should be in a different availability zone (Availability Set in Azure). If two nodes go down your replica set will be unavailable. But two nodes going down at the same time is very unlikely if all nodes are in different availability zones. So don't worry too much about more than one node going down. If that's a real concern (in most applications it really isn't), you want to make a 5 member replica set.
5) That's weird. This sounds like your replica set might be configured incorrectly. As I said, you don't need an arbiter anyway. So you could just try setting it up again without the arbiter and see if it works. Open a new question if you're still having issues. Make sure to include the output of running rs.status() in your question.
I came across MongoDB official site explaining on having odd number of members replica set up. I also heard of the term Arbiter from the same site, which based on my understanding, it will not be elected as primary and it does participate on election (from https://docs.mongodb.com/manual/core/replica-set-arbiter/).
There is also a post related to Arbiter in Why do we need an 'arbiter' in MongoDB replication? which then relates to CAP theorem, which further gets things more complicated.
First of all, why do we need to make the number of members odd? Also, can someone explain to me what this Arbiter is and what is its role in a given replica set in simple layman English??
Thanks in advance.
In short: it is to stop the two normal nodes of the replica set getting into a split-brain situation if they lose contact with each other.
MongoDB replica sets are designed so that, if one or more members goes down or loses contact, the other members are able to keep going as long as between them they have a majority. The majority clause is important: without that, you might have a situation where the network is split in two, and the nodes on each side of the partition think that they're still carrying on the replica set, and end up with different sets of data.
So to avoid the split brain problem, the nodes of a replica set will not continue if they can't command an absolute majority. An example of this is if you have two nodes, in a replica set like this:
If they lose communication, the outcome is symmetrical:
Each one will reason the same way:
realise it has lost communication with the other
assess whether it is possible to keep the replica set going
realise that 1 node (out of 2) does not constitute a majority
revert to Secondary mode
The difference an Arbiter makes
If there is a third node, then even if the two main nodes lose contact with each other then there will still be one of them in contact with the arbiter. This allows the two main nodes to make different decisions, and keep the replica set going while avoiding the split-brain problem.
Consider the following example of a 3-node replica set:
Whichever way the network partition goes, one node will still be in contact with the arbiter; for example like this:
Node A will:
realise it can contact neither node B nor the arbiter
assess whether it is possible to keep the replica set going
realise that 1 node (out of 3) does not constitute a majority
revert to Secondary mode
Whereas node B is able to react differently:
realise it cannot contact node A, but still has contact with the arbiter
assess whether it is possible to keep the replica set going
realise that 2 nodes (out of 3) do constitute a majority
take over as Primary
This also illustrates how you should deploy an arbiter to get that benefit:
try to put the arbiter on a system independent of both the data-bearing nodes, to maximise the chance of it still being able to communicate with either throughout network problems
it doesn't need to store data, so you don't need high-spec hardware
Just 1 arbiter is enough to break the deadlock; you don't get any benefit from multiple arbiters
Take the example of a 2-member replica set: in the event of a network-partitioning, i.e., the 2 members lost touch of each other, who gets to become the primary? There will be a tie and a need for a tie-breaker. That would not be the case if we have a 3-member replica set: the group that contains two nodes will win and one of them will become primary. That is the basis of the requirement for an odd number of nodes in a replica set. As for an arbiter, it happens to be light weight so that I guess one can save money by having in place a smaller machine, since we do not expect it to hold any data, and that we just need it to be present to vote for primary.
In MongoDB tutorial of deploying geographically distributed replica set it is said that
Ensure that a majority of the voting members are within a primary facility, “Site A”. This includes priority 0 members and arbiters.
I am confused by arbiters there since in other place in documentation I found that
There should only be at most one arbiter configured in any replica set.
So how many arbiters at most can be in a replica set? If more that one arbiter allowed, then what is the point to have more than one arbiter in replica set?
Introduction
The fact that "arbiters" is written in plural in the first sentence has style reasons, not technical reasons.
You really should have at most 1 arbiter. Iirc, you technically could have more, but to be honest with you, I never tried it. But let's assume you could for the sake of the explanation below.
You seem to be a bit unsure here, but correctly assume that it does not make any sense to have more than one arbiter.
Recap: What are arbiters there for?
An arbiter exists to provide a quorum in elections.
Take a replica set with two data bearing nodes. That setup will run as expected as long as both instances are up – they form a quorum of 2 votes of 2 original members of a replica set. If one machine goes down, however, we only have 1 vote of 2 originally present, which is not a qualified majority, and the data bearing node still running will subsequently revert to secondary state, making writes impossible.
To prevent that, an arbiter is added to the mix. An arbiter does nothing more than to track which of the available data bearing nodes has the most current data set available and vote for that member in case of an election. So with our replica set with two data bearing nodes, in order to get a qualified majority of votes in case 1 of the nodes forming the replica set goes down, we only need 1 arbiter, since 2/3 votes provides a qualified majority.
Arbiters beyond 2 data bearing nodes
If we had a replica set with 3 data bearing nodes, we would not need an arbiter, since we have 3 voting members, and if 1 member goes down, the others still form a qualified majority needed to hold an election.
A bit more abstract, we can find out wether we need an arbiter by putting in the number of votes present in a replica set into the following "formula"
needArbiter = originalVotes - floor(originalVotes/2) <= originalVotes / 2
If we put in an additional arbiter, the number of votes would be 4: 3 data bearing nodes and 1 arbiter. One node goes down, no problem. Second node goes down, and the replica set will revert to secondary state. Now let's assume one of the two nodes down are is the arbiter – we would be in secondary state while the data bearing nodes only would be able to provide a quorum. We'd have to pay for and maintain an additional arbiter without anything gained from it. So in order to provide a qualified majority again, we would need to add yet another arbiter (making 2 now), without any benefit other than the fact that two arbiters can go down. You basically would need additional arbiters to prevent situations in which the existence of the arbiter you did not need in the first place becomes a problem.
Now let's assume we have 4 data bearing nodes. Since they can not form a qualified majority when 2 of them going down, that would pretty much be the same situation as with a replica set with 3 data bearing nodes, just more expensive. So in order to allow 2 nodes of the replica set being down at the same time, we simply add an arbiter. Do more arbiters make sense? No, even less than with a replica set with two or 3 data bearing nodes, since the probability that 2 data bearing nodes and the arbiter will fail at the same time is very low. And you'd need an uneven number of arbiters.
Conclusion
Imho, with 4 data bearing nodes, the arbiter reaches its limit of usefulness. If you need a high replication factor the percentage of money saved when using an arbiter in comparison to a data bearing node becomes smaller and smaller. Remember, the next step would be 6 data bearing nodes plus an arbiter, so the costs you save is less than 1/6 of your overall costs.
So more generally speaking, the more data bearing nodes you have (the higher your "replication factor" in Mongo terms) the less reasonable it becomes to have additional arbiters. Both from the technical point of view (the probability of a majority of nodes failing the same time becomes lower and lower) and the business point of view (with a high replication factor, the money saved with an arbiter in comparison to the overall costs becomes absurdly small).
Mnemonic:
The lowest uneven number is 1.
I have a scenario where I think having more than 1 Arbiter makes sense.
Problem
I have 3 data bearing nodes in a replicaset. Now I want to distribute my replicaset geographically so that I can mitigate the risk of a datacenter outage.
3 Node Replicaset, does not solve the problem
Primary Datacenter => 2 Data bearing Nodes
Backup Datacenter => 1 Data bearing Node
If that primary datacenter is down and the two out of three nodes in the replicaset would not be available then data bearing node in backup datacenter would not be able to become a primary since majority is not available. So the 3 node configuration does not solve the problem of a datacenter outage.
5 Node replicaset
Primary Datacenter => 2 Data bearing Nodes
Backup Datacenter => 1 Data bearing Node
Third Datacenter => 2 Arbiters
In this configuration I am able to sustain outage of any of the three datacenters and still be able to operate.
Obviously, a more ideal configuration would be to have 4 data bearing nodes and have 1 arbiter. It would give me redundancy in the backup datacenter as well. However since data bearing node is a much more expensive proposition than an arbiter going with 3 data bearing nodes and 2 arbiters makes more sense and I am happy to forgo the redundancy in backup datacenter in favor of the cost saving.
For our special case it makes sense to have 2 arbiters. Let me explain: we have 3 data centers but 1 of these 3 data centers is not suitable to host data bearing members. That's why we host in this data center 2 arbiters for each replica set. The 3 data bearing members of the replSet are hosted in the two other data centers (we want to have 3 instead of 2 data bearing members for resilience reasons). If 1 of the 3 data center goes down or is not reachable due to a network partition, the replSet is still able to elect a primary, thus it's still read and writeable. This wouldn't be possible with only 1 or 0 arbiter. Hence, 2 arbiters may make sense.
Let's see how it may look like. Here are 2 replSets, each with 3 data bearing members and 2 arbiters in 3 data centers, whereas DC3 is the restricted data center:
| |DC1 |DC2 |DC3 |
|----|-----|-----|-----|
|rs1 |m1,m2|m3 |a1,a2|
|rs2 |m1 |m2,m3|a1,a2|
If one data center goes down, which replSet member would become primary?
DC1 goes down:
rs1: m3
rs2: m2 or m3
DC2 goes down:
rs1: m1 or m2
rs2: m1
DC3 goes down:
rs1: m1,m2 or m3
Mongo docs list this three-member configuration: primary, secondary, arbiter, as the minimal architecture of a replica set.
Why would an arbiter be necessary there? If the primary fails, the secondary won't see the heartbeat, so it needs to become primary. In other words, why wouldn't a primary + secondary configuration be sufficient? This related question doesn't seem to address the issue, as it discusses larger numbers of nodes.
Suppose you have only two servers, one primary and one secondary.
If suddenly the secondary can not reach the primary server it could be that the primary is down (in that case the secondary should become primary) but it could be as well a network issue that isolated the secondary (this the secondary is the one that is in deed down).
however, if you have an arbiter and the secondary cannot reach the primary but it CAN reach the arbiter then the issue is with the primary so it must become the new primary. If it CANNOT reach the primary, nor the arbiter, then the secondary knows that the issue is that he is isolated/broken -poor secondary :(- so he must not become the primary
If you bring the Arbiter down to its core it is essentially a none-data holding member used for voting.
One case for an Arbiter is as I state in the linked question: Why do we need an 'arbiter' in MongoDB replication? to break the problems of CAP but that is not its true purpose since you could easily replace that Arbiter with a data holding node and have the same effect.
However, an Arbiter will have a few benefits:
Small footprint
No data
No need to synch
can instantly vote
can be put literally anywhere in your network, app server or even another secondary to boost that part of your network (this comes into partitions).
So an Arbiter is extremely useful, even on one side of a partition (i.e. you have no partitioning in your network).
Now to explain base setup. An Arbiter would NOT be required, you could factor it out for a data holding node, but 3 data holding nodes is not the minimum (that is the minimum you need to keep automatic failover), 2 data holding nodes and 1 Arbiter is actually the minimum.
Now to answer:
In other words, why wouldn't a primary + secondary configuration be sufficient?
Because if one of those goes down there is only 50% of the vote left (2-1 = 1) and 50% is not classed as a sufficient majority for MongoDB to actually vote in a member (judged by the total configured voteable members in your rs.config).
Also in this case MongoDB does not actually know if that last member is the last member. It needs other members to tell it otherwise.
So yes, this is why you need a third guy.
Assume we setup a MongoDB replication without arbiter, If the primary is unavailable, the replica set will elect a secondary to be primary. So I think it's kind of implicit arbiter, since the replica will elect a primary automatically.
So I am wondering why do we need a dedicated arbiter node? Thanks!
I created a spreadsheet to better illustrate the effect of Arbiter nodes in a Replica Set.
It basically comes down to these points:
With an RS of 2 data nodes, losing 1 server brings you below your voting minimum (which is "greater than N/2"). An arbiter solves this.
With an RS of even numbered data nodes, adding an Arbiter increases your fault tolerance by 1 without making it possible to have 2 voting clusters due to a split.
With an RS of odd numbered data nodes, adding an Arbiter would allow a split to create 2 isolated clusters with "greater than N/2" votes and therefore a split brain scenario.
Elections are explained [in poor] detail here. In that document it states that an RS can have 50 members (even number) and 7 voting members. I emphasize "states" because it does not explain how it works. To me it seems that if you have a split happen with 4 members (all voting) on one side and 46 members (3 voting) on the other, you'd rather have the 46 elect a primary and the 4 to be a read-only cluster. But, that's exactly what "limited voting" prevents. In that situation you will actually have a 4 member cluster with a primary and a 46 member cluster that is read only. Explaining how that makes sense is out of the scope of this question and beyond my knowledge.
Its necessary to have a arbiter in a replication for the below reasons:
Replication is more reliable if it has odd number of replica sets. Incase if there is even number of replica sets its better to add a arbiter in the replication.
Arbiters do not hold data in them and they are just to vote in election when there is any node failure.
Arbiter is a light weight process they do not consume much hardware resources.
Arbiters just exchange the user credentials data between the replica set which are encrypted.
Vote during elections,hearbeats and configureation data are not encrypted while communicating in between the replica sets.
It is better to run arbiter on a separate machine rather than along with any one of the replica set to retain high availability.
Hope this helps !!!
This really comes down to the CAP theorem whereby it is stated that if there are equal number of servers on either side of the partition the database cannot maintain CAP (Consistency, Availability, and Partition tolerance). An Arbiter is specifically designed to create an "imbalance" or majority on one side so that a primary can be elected in this case.
If you get an even number of nodes on either side MongoDB will not elect a primary and your set will not accept writes.
Edit
By either side I mean, for example, 2 on one side and 2 on the other. My English wasn't easy to understand there.
So really what I mean is both sides.
Edit
Wikipedia presents quite a good case for explaining CAP: http://en.wikipedia.org/wiki/CAP_theorem
Arbiters are an optional mechanism to allow voting to succeed when you have an even number of mongods deployed in a replicaset. Arbiters are light weight, meant to be deployed on a server that is NOT a dedicated mongo replica, i.e: the server's primary role is some other task, like a redis server. Since they're light they won't interfere (noticeably) with the system's resources.
From the docs :
An arbiter does not have a copy of data set and cannot become a
primary. Replica sets may have arbiters to add a vote in elections of
for primary. Arbiters allow replica sets to have an uneven number of
members, without the overhead of a member that replicates data.
http://docs.mongodb.org/manual/core/replica-set-arbiter/
http://docs.mongodb.org/manual/core/replica-set-elections/#replica-set-elections