In MongoDB tutorial of deploying geographically distributed replica set it is said that
Ensure that a majority of the voting members are within a primary facility, “Site A”. This includes priority 0 members and arbiters.
I am confused by arbiters there since in other place in documentation I found that
There should only be at most one arbiter configured in any replica set.
So how many arbiters at most can be in a replica set? If more that one arbiter allowed, then what is the point to have more than one arbiter in replica set?
Introduction
The fact that "arbiters" is written in plural in the first sentence has style reasons, not technical reasons.
You really should have at most 1 arbiter. Iirc, you technically could have more, but to be honest with you, I never tried it. But let's assume you could for the sake of the explanation below.
You seem to be a bit unsure here, but correctly assume that it does not make any sense to have more than one arbiter.
Recap: What are arbiters there for?
An arbiter exists to provide a quorum in elections.
Take a replica set with two data bearing nodes. That setup will run as expected as long as both instances are up – they form a quorum of 2 votes of 2 original members of a replica set. If one machine goes down, however, we only have 1 vote of 2 originally present, which is not a qualified majority, and the data bearing node still running will subsequently revert to secondary state, making writes impossible.
To prevent that, an arbiter is added to the mix. An arbiter does nothing more than to track which of the available data bearing nodes has the most current data set available and vote for that member in case of an election. So with our replica set with two data bearing nodes, in order to get a qualified majority of votes in case 1 of the nodes forming the replica set goes down, we only need 1 arbiter, since 2/3 votes provides a qualified majority.
Arbiters beyond 2 data bearing nodes
If we had a replica set with 3 data bearing nodes, we would not need an arbiter, since we have 3 voting members, and if 1 member goes down, the others still form a qualified majority needed to hold an election.
A bit more abstract, we can find out wether we need an arbiter by putting in the number of votes present in a replica set into the following "formula"
needArbiter = originalVotes - floor(originalVotes/2) <= originalVotes / 2
If we put in an additional arbiter, the number of votes would be 4: 3 data bearing nodes and 1 arbiter. One node goes down, no problem. Second node goes down, and the replica set will revert to secondary state. Now let's assume one of the two nodes down are is the arbiter – we would be in secondary state while the data bearing nodes only would be able to provide a quorum. We'd have to pay for and maintain an additional arbiter without anything gained from it. So in order to provide a qualified majority again, we would need to add yet another arbiter (making 2 now), without any benefit other than the fact that two arbiters can go down. You basically would need additional arbiters to prevent situations in which the existence of the arbiter you did not need in the first place becomes a problem.
Now let's assume we have 4 data bearing nodes. Since they can not form a qualified majority when 2 of them going down, that would pretty much be the same situation as with a replica set with 3 data bearing nodes, just more expensive. So in order to allow 2 nodes of the replica set being down at the same time, we simply add an arbiter. Do more arbiters make sense? No, even less than with a replica set with two or 3 data bearing nodes, since the probability that 2 data bearing nodes and the arbiter will fail at the same time is very low. And you'd need an uneven number of arbiters.
Conclusion
Imho, with 4 data bearing nodes, the arbiter reaches its limit of usefulness. If you need a high replication factor the percentage of money saved when using an arbiter in comparison to a data bearing node becomes smaller and smaller. Remember, the next step would be 6 data bearing nodes plus an arbiter, so the costs you save is less than 1/6 of your overall costs.
So more generally speaking, the more data bearing nodes you have (the higher your "replication factor" in Mongo terms) the less reasonable it becomes to have additional arbiters. Both from the technical point of view (the probability of a majority of nodes failing the same time becomes lower and lower) and the business point of view (with a high replication factor, the money saved with an arbiter in comparison to the overall costs becomes absurdly small).
Mnemonic:
The lowest uneven number is 1.
I have a scenario where I think having more than 1 Arbiter makes sense.
Problem
I have 3 data bearing nodes in a replicaset. Now I want to distribute my replicaset geographically so that I can mitigate the risk of a datacenter outage.
3 Node Replicaset, does not solve the problem
Primary Datacenter => 2 Data bearing Nodes
Backup Datacenter => 1 Data bearing Node
If that primary datacenter is down and the two out of three nodes in the replicaset would not be available then data bearing node in backup datacenter would not be able to become a primary since majority is not available. So the 3 node configuration does not solve the problem of a datacenter outage.
5 Node replicaset
Primary Datacenter => 2 Data bearing Nodes
Backup Datacenter => 1 Data bearing Node
Third Datacenter => 2 Arbiters
In this configuration I am able to sustain outage of any of the three datacenters and still be able to operate.
Obviously, a more ideal configuration would be to have 4 data bearing nodes and have 1 arbiter. It would give me redundancy in the backup datacenter as well. However since data bearing node is a much more expensive proposition than an arbiter going with 3 data bearing nodes and 2 arbiters makes more sense and I am happy to forgo the redundancy in backup datacenter in favor of the cost saving.
For our special case it makes sense to have 2 arbiters. Let me explain: we have 3 data centers but 1 of these 3 data centers is not suitable to host data bearing members. That's why we host in this data center 2 arbiters for each replica set. The 3 data bearing members of the replSet are hosted in the two other data centers (we want to have 3 instead of 2 data bearing members for resilience reasons). If 1 of the 3 data center goes down or is not reachable due to a network partition, the replSet is still able to elect a primary, thus it's still read and writeable. This wouldn't be possible with only 1 or 0 arbiter. Hence, 2 arbiters may make sense.
Let's see how it may look like. Here are 2 replSets, each with 3 data bearing members and 2 arbiters in 3 data centers, whereas DC3 is the restricted data center:
| |DC1 |DC2 |DC3 |
|----|-----|-----|-----|
|rs1 |m1,m2|m3 |a1,a2|
|rs2 |m1 |m2,m3|a1,a2|
If one data center goes down, which replSet member would become primary?
DC1 goes down:
rs1: m3
rs2: m2 or m3
DC2 goes down:
rs1: m1 or m2
rs2: m1
DC3 goes down:
rs1: m1,m2 or m3
Related
I have read this multiple times, but am trying to understand the message. All documents state to have an odd number of members in a replica set. The reason is because you want to have an odd number of votes to ensure proper election results (no ties). However, having an odd number of member in a replica set, in my opinion, is counter-intuitive as if the primary becomes unavailable, you're left with an even number of members to vote for the new primary. Based off of that, in reality, wouldn't it make more sense to have an even number of members in the replica set? That way if the primary becomes unavailable you'll have an odd number to determine the results with a guarantee of no tie?
For example, if you have 3 nodes and one goes down, you only have 2 left that can vote. As 2 is even, you could have a tie, which is what we're trying to avoid. 2 members to start isn't good as if one becomes unavailable, that leaves you with 1 node, which in that case means no primaries. Based off of my above argument, that would imply that 4 would be good. If one becomes unavailable, you'll have 3 nodes left to vote (an odd number of members). I know this is the opposite of what the documentation states, but to me, this makes sense. The only place where an odd number seems to make sense is if an election occurs with no nodes down.
Thoughts?
-Jim
The purpose of replication is that in the event of a server goes down in a replica-set you can still access the data. This concept is called as availability.
In case the primary node goes down, one of the secondary nodes take its place, as primary; this process is called as failover.
The available nodes decide which node becomes a primary, thru the process of election.
A replica-set can have upto 50 members and upto 7 voting members - maximum.
Why odd number of members?
The failover mechanism means that a majority of the nodes must be available for a primary to be elected.
In a 4 (even) member replica-set the majority is 3: 4/2 + 1
In a 5 (odd) member replica-set the majority is 3: 5/2 + 1
In a 5 member replica-set there is a better chance of having 3 nodes available for the replica-set to be operational. In a 4 member replica-set at least 3 member must be up for the replica-set to be operational. So, with the 5 member replica-set (3 of 5) there is
higher chance of being more operational than the 4 (3 of 4) member replica-set.
It is not just the number of members; there are various factors that affect an election.
Members with higher priority has a chance of getting elected over
others (priorities are numbers, and higher number is higher priority).
Also, a secondary member with the latest replicated data has a
preference over a member which do not have a latest copy. Factors
like write concern affect which member has the latest copy of the
data.
There are arbiter members which can only vote but do not replicate
data or run for an election. Hidden and delayed members cannot vote
or become a primary.
That said, even with even number of available voting nodes election gets completed (and considering the above factors). An election may result in a tie, in such a case the voting process will start again, until a member is elected as primary. Note that an election is just an aspect (or process) of a system - a system that provides high availability and makes your system operational and with data replication.
Also see: Factors and Conditions that Affect Elections.
Could someone help validate our setup
Setting up a 4 node MongoDB cluster
1 primary (write ) , 3 secondaries (read) if primary goes down, 3 secondaries can break tie and elect a secondary to primary
Will this setup work?
is an arbiter required in such a scenario?
Once I set it up this way at the outset, then as load increases all I need to do is keep adding nodes in pairs to the cluster. (Adding nodes in pairs will help us keep up with performance and reduce the frequency of cluster changes, also we are more read heavy than writes, at some point we will have to consider scaling out writes as well )
Help is very much appreciated.
thanks.
Yes, an arbiter is required, otherwise if 2 nodes go down or are otherwise unavailable, you will not have a primary - MongoDB requires a strict majority (>50%) of votes to elect a primary, and in your case that majority number is 3 out of 4 (two out of 4 is not greater than 50%). That number will still be 3 if you add an arbiter, but you will be able to have a primary with 2 data bearing nodes down.
As for why, consider the following possibility:
2 nodes are isolated from the other 2 - they are still up, functional, but cannot talk to each other. There are now 2 votes on either side of this "split" and no way to break the tie - each side is just as valid in terms of voting for a primary, and without the strict majority rule you end up with 2 primaries and no way to resolve writes once the split resolves itself. Add an arbiter to either side of the split and you have no such ambiguity.
This type of scenario has a number of permutations when the number of votes are even which I won't go into here. Suffice it to say that the best practice when running a replica set is to always have an odd number of votes and hence avoid these situations.
I have the following set up in my production environement,
3 nodes, with 1 primary and 2 secondary...
I came to know about the arbiter, Since mongo db itself does the work of election within the replicaset. What is the need of arbiter in the mongo replication?
In which scenario, arbiter will be useful?
Regards,
Harry
The point of an Arbiter is to break the deadlock when an election needs to be held for a Primary. In such that there are a majority of nodes that can make the decision as to which node to elect.
In your current configuration you have an odd number of nodes, so the election process is simple when all nodes are up, and in a failover one of the other nodes will simply be elected.
If you have an even number of nodes in a replica set to begin with and Arbiter may be required in the case where you do not want to commit the same level of hardware to have say a five node* replica set. Here you could use an arbiter on a lower spec machine in order to avoid a deadlock in elections.
An arbiter is also useful if you want to give preference to certain nodes to be elected as the Primary.
Plenty of information in the documentation:
http://docs.mongodb.org/manual/core/replica-set-members/
An arbiter enables MongoDB to garnish majority holdings across partitions without the need for a data holding node.
This all stems from the CAP theorem: http://en.wikipedia.org/wiki/CAP_theorem and essentially it enables for one side to have a majority voting as such stops stale mate elections across paritions within your clusters.
Of course Arbiters can be used only on side too if you want say 4 data holding nodes and 1 that is not.
In addition to breaking ties during election, arbiters do not hold data. An arbiter is lightweight and can run alongside other processes. Since it does not participate in a replication/serving queries, it does not use much CPU/memory. Arbiters have minimal resource requirements and do not require dedicated hardware.
I've read quite a bit about mongodb replica set and how the elections work on a failover. My curiosity is assuming the client will use readPreference set to primary only, is there any advantage to having odd number of members against having even number of members + 1 arbiter?
For example if you have a 3 member replica set you can set all 3 members to be replicas or you can have only 2 replicas and an arbiter (which you can install on a smaller machine). The safety is basically the same, any machine can go down and the replica set is still ok, but if two of them go down then the replica set is in stalemate (it cannot elect a new primary).
The only difference is that in the second case you can use a way smaller machine for the arbiter.
It's actually not true that three data holding nodes provide the same "safety" net as two data holding nodes plus an arbiter.
Consider these cases:
1) one of your nodes loses its disk and you need to fully resync it. If you had three data holding nodes you can resync off of the other secondary, instead of the primary (which would reduce the load on the primary).
2) one of your nodes loses its disk and it takes you a while to located a new one. While that secondary is down you are running with ZERO safety net if you had two nodes and an arbiter since you only have one node with data left, if anything happens to it, you are toast.
Assume we setup a MongoDB replication without arbiter, If the primary is unavailable, the replica set will elect a secondary to be primary. So I think it's kind of implicit arbiter, since the replica will elect a primary automatically.
So I am wondering why do we need a dedicated arbiter node? Thanks!
I created a spreadsheet to better illustrate the effect of Arbiter nodes in a Replica Set.
It basically comes down to these points:
With an RS of 2 data nodes, losing 1 server brings you below your voting minimum (which is "greater than N/2"). An arbiter solves this.
With an RS of even numbered data nodes, adding an Arbiter increases your fault tolerance by 1 without making it possible to have 2 voting clusters due to a split.
With an RS of odd numbered data nodes, adding an Arbiter would allow a split to create 2 isolated clusters with "greater than N/2" votes and therefore a split brain scenario.
Elections are explained [in poor] detail here. In that document it states that an RS can have 50 members (even number) and 7 voting members. I emphasize "states" because it does not explain how it works. To me it seems that if you have a split happen with 4 members (all voting) on one side and 46 members (3 voting) on the other, you'd rather have the 46 elect a primary and the 4 to be a read-only cluster. But, that's exactly what "limited voting" prevents. In that situation you will actually have a 4 member cluster with a primary and a 46 member cluster that is read only. Explaining how that makes sense is out of the scope of this question and beyond my knowledge.
Its necessary to have a arbiter in a replication for the below reasons:
Replication is more reliable if it has odd number of replica sets. Incase if there is even number of replica sets its better to add a arbiter in the replication.
Arbiters do not hold data in them and they are just to vote in election when there is any node failure.
Arbiter is a light weight process they do not consume much hardware resources.
Arbiters just exchange the user credentials data between the replica set which are encrypted.
Vote during elections,hearbeats and configureation data are not encrypted while communicating in between the replica sets.
It is better to run arbiter on a separate machine rather than along with any one of the replica set to retain high availability.
Hope this helps !!!
This really comes down to the CAP theorem whereby it is stated that if there are equal number of servers on either side of the partition the database cannot maintain CAP (Consistency, Availability, and Partition tolerance). An Arbiter is specifically designed to create an "imbalance" or majority on one side so that a primary can be elected in this case.
If you get an even number of nodes on either side MongoDB will not elect a primary and your set will not accept writes.
Edit
By either side I mean, for example, 2 on one side and 2 on the other. My English wasn't easy to understand there.
So really what I mean is both sides.
Edit
Wikipedia presents quite a good case for explaining CAP: http://en.wikipedia.org/wiki/CAP_theorem
Arbiters are an optional mechanism to allow voting to succeed when you have an even number of mongods deployed in a replicaset. Arbiters are light weight, meant to be deployed on a server that is NOT a dedicated mongo replica, i.e: the server's primary role is some other task, like a redis server. Since they're light they won't interfere (noticeably) with the system's resources.
From the docs :
An arbiter does not have a copy of data set and cannot become a
primary. Replica sets may have arbiters to add a vote in elections of
for primary. Arbiters allow replica sets to have an uneven number of
members, without the overhead of a member that replicates data.
http://docs.mongodb.org/manual/core/replica-set-arbiter/
http://docs.mongodb.org/manual/core/replica-set-elections/#replica-set-elections