Could someone help validate our setup
Setting up a 4 node MongoDB cluster
1 primary (write ) , 3 secondaries (read) if primary goes down, 3 secondaries can break tie and elect a secondary to primary
Will this setup work?
is an arbiter required in such a scenario?
Once I set it up this way at the outset, then as load increases all I need to do is keep adding nodes in pairs to the cluster. (Adding nodes in pairs will help us keep up with performance and reduce the frequency of cluster changes, also we are more read heavy than writes, at some point we will have to consider scaling out writes as well )
Help is very much appreciated.
thanks.
Yes, an arbiter is required, otherwise if 2 nodes go down or are otherwise unavailable, you will not have a primary - MongoDB requires a strict majority (>50%) of votes to elect a primary, and in your case that majority number is 3 out of 4 (two out of 4 is not greater than 50%). That number will still be 3 if you add an arbiter, but you will be able to have a primary with 2 data bearing nodes down.
As for why, consider the following possibility:
2 nodes are isolated from the other 2 - they are still up, functional, but cannot talk to each other. There are now 2 votes on either side of this "split" and no way to break the tie - each side is just as valid in terms of voting for a primary, and without the strict majority rule you end up with 2 primaries and no way to resolve writes once the split resolves itself. Add an arbiter to either side of the split and you have no such ambiguity.
This type of scenario has a number of permutations when the number of votes are even which I won't go into here. Suffice it to say that the best practice when running a replica set is to always have an odd number of votes and hence avoid these situations.
Related
I have read this multiple times, but am trying to understand the message. All documents state to have an odd number of members in a replica set. The reason is because you want to have an odd number of votes to ensure proper election results (no ties). However, having an odd number of member in a replica set, in my opinion, is counter-intuitive as if the primary becomes unavailable, you're left with an even number of members to vote for the new primary. Based off of that, in reality, wouldn't it make more sense to have an even number of members in the replica set? That way if the primary becomes unavailable you'll have an odd number to determine the results with a guarantee of no tie?
For example, if you have 3 nodes and one goes down, you only have 2 left that can vote. As 2 is even, you could have a tie, which is what we're trying to avoid. 2 members to start isn't good as if one becomes unavailable, that leaves you with 1 node, which in that case means no primaries. Based off of my above argument, that would imply that 4 would be good. If one becomes unavailable, you'll have 3 nodes left to vote (an odd number of members). I know this is the opposite of what the documentation states, but to me, this makes sense. The only place where an odd number seems to make sense is if an election occurs with no nodes down.
Thoughts?
-Jim
The purpose of replication is that in the event of a server goes down in a replica-set you can still access the data. This concept is called as availability.
In case the primary node goes down, one of the secondary nodes take its place, as primary; this process is called as failover.
The available nodes decide which node becomes a primary, thru the process of election.
A replica-set can have upto 50 members and upto 7 voting members - maximum.
Why odd number of members?
The failover mechanism means that a majority of the nodes must be available for a primary to be elected.
In a 4 (even) member replica-set the majority is 3: 4/2 + 1
In a 5 (odd) member replica-set the majority is 3: 5/2 + 1
In a 5 member replica-set there is a better chance of having 3 nodes available for the replica-set to be operational. In a 4 member replica-set at least 3 member must be up for the replica-set to be operational. So, with the 5 member replica-set (3 of 5) there is
higher chance of being more operational than the 4 (3 of 4) member replica-set.
It is not just the number of members; there are various factors that affect an election.
Members with higher priority has a chance of getting elected over
others (priorities are numbers, and higher number is higher priority).
Also, a secondary member with the latest replicated data has a
preference over a member which do not have a latest copy. Factors
like write concern affect which member has the latest copy of the
data.
There are arbiter members which can only vote but do not replicate
data or run for an election. Hidden and delayed members cannot vote
or become a primary.
That said, even with even number of available voting nodes election gets completed (and considering the above factors). An election may result in a tie, in such a case the voting process will start again, until a member is elected as primary. Note that an election is just an aspect (or process) of a system - a system that provides high availability and makes your system operational and with data replication.
Also see: Factors and Conditions that Affect Elections.
I came across MongoDB official site explaining on having odd number of members replica set up. I also heard of the term Arbiter from the same site, which based on my understanding, it will not be elected as primary and it does participate on election (from https://docs.mongodb.com/manual/core/replica-set-arbiter/).
There is also a post related to Arbiter in Why do we need an 'arbiter' in MongoDB replication? which then relates to CAP theorem, which further gets things more complicated.
First of all, why do we need to make the number of members odd? Also, can someone explain to me what this Arbiter is and what is its role in a given replica set in simple layman English??
Thanks in advance.
In short: it is to stop the two normal nodes of the replica set getting into a split-brain situation if they lose contact with each other.
MongoDB replica sets are designed so that, if one or more members goes down or loses contact, the other members are able to keep going as long as between them they have a majority. The majority clause is important: without that, you might have a situation where the network is split in two, and the nodes on each side of the partition think that they're still carrying on the replica set, and end up with different sets of data.
So to avoid the split brain problem, the nodes of a replica set will not continue if they can't command an absolute majority. An example of this is if you have two nodes, in a replica set like this:
If they lose communication, the outcome is symmetrical:
Each one will reason the same way:
realise it has lost communication with the other
assess whether it is possible to keep the replica set going
realise that 1 node (out of 2) does not constitute a majority
revert to Secondary mode
The difference an Arbiter makes
If there is a third node, then even if the two main nodes lose contact with each other then there will still be one of them in contact with the arbiter. This allows the two main nodes to make different decisions, and keep the replica set going while avoiding the split-brain problem.
Consider the following example of a 3-node replica set:
Whichever way the network partition goes, one node will still be in contact with the arbiter; for example like this:
Node A will:
realise it can contact neither node B nor the arbiter
assess whether it is possible to keep the replica set going
realise that 1 node (out of 3) does not constitute a majority
revert to Secondary mode
Whereas node B is able to react differently:
realise it cannot contact node A, but still has contact with the arbiter
assess whether it is possible to keep the replica set going
realise that 2 nodes (out of 3) do constitute a majority
take over as Primary
This also illustrates how you should deploy an arbiter to get that benefit:
try to put the arbiter on a system independent of both the data-bearing nodes, to maximise the chance of it still being able to communicate with either throughout network problems
it doesn't need to store data, so you don't need high-spec hardware
Just 1 arbiter is enough to break the deadlock; you don't get any benefit from multiple arbiters
Take the example of a 2-member replica set: in the event of a network-partitioning, i.e., the 2 members lost touch of each other, who gets to become the primary? There will be a tie and a need for a tie-breaker. That would not be the case if we have a 3-member replica set: the group that contains two nodes will win and one of them will become primary. That is the basis of the requirement for an odd number of nodes in a replica set. As for an arbiter, it happens to be light weight so that I guess one can save money by having in place a smaller machine, since we do not expect it to hold any data, and that we just need it to be present to vote for primary.
In MongoDB tutorial of deploying geographically distributed replica set it is said that
Ensure that a majority of the voting members are within a primary facility, “Site A”. This includes priority 0 members and arbiters.
I am confused by arbiters there since in other place in documentation I found that
There should only be at most one arbiter configured in any replica set.
So how many arbiters at most can be in a replica set? If more that one arbiter allowed, then what is the point to have more than one arbiter in replica set?
Introduction
The fact that "arbiters" is written in plural in the first sentence has style reasons, not technical reasons.
You really should have at most 1 arbiter. Iirc, you technically could have more, but to be honest with you, I never tried it. But let's assume you could for the sake of the explanation below.
You seem to be a bit unsure here, but correctly assume that it does not make any sense to have more than one arbiter.
Recap: What are arbiters there for?
An arbiter exists to provide a quorum in elections.
Take a replica set with two data bearing nodes. That setup will run as expected as long as both instances are up – they form a quorum of 2 votes of 2 original members of a replica set. If one machine goes down, however, we only have 1 vote of 2 originally present, which is not a qualified majority, and the data bearing node still running will subsequently revert to secondary state, making writes impossible.
To prevent that, an arbiter is added to the mix. An arbiter does nothing more than to track which of the available data bearing nodes has the most current data set available and vote for that member in case of an election. So with our replica set with two data bearing nodes, in order to get a qualified majority of votes in case 1 of the nodes forming the replica set goes down, we only need 1 arbiter, since 2/3 votes provides a qualified majority.
Arbiters beyond 2 data bearing nodes
If we had a replica set with 3 data bearing nodes, we would not need an arbiter, since we have 3 voting members, and if 1 member goes down, the others still form a qualified majority needed to hold an election.
A bit more abstract, we can find out wether we need an arbiter by putting in the number of votes present in a replica set into the following "formula"
needArbiter = originalVotes - floor(originalVotes/2) <= originalVotes / 2
If we put in an additional arbiter, the number of votes would be 4: 3 data bearing nodes and 1 arbiter. One node goes down, no problem. Second node goes down, and the replica set will revert to secondary state. Now let's assume one of the two nodes down are is the arbiter – we would be in secondary state while the data bearing nodes only would be able to provide a quorum. We'd have to pay for and maintain an additional arbiter without anything gained from it. So in order to provide a qualified majority again, we would need to add yet another arbiter (making 2 now), without any benefit other than the fact that two arbiters can go down. You basically would need additional arbiters to prevent situations in which the existence of the arbiter you did not need in the first place becomes a problem.
Now let's assume we have 4 data bearing nodes. Since they can not form a qualified majority when 2 of them going down, that would pretty much be the same situation as with a replica set with 3 data bearing nodes, just more expensive. So in order to allow 2 nodes of the replica set being down at the same time, we simply add an arbiter. Do more arbiters make sense? No, even less than with a replica set with two or 3 data bearing nodes, since the probability that 2 data bearing nodes and the arbiter will fail at the same time is very low. And you'd need an uneven number of arbiters.
Conclusion
Imho, with 4 data bearing nodes, the arbiter reaches its limit of usefulness. If you need a high replication factor the percentage of money saved when using an arbiter in comparison to a data bearing node becomes smaller and smaller. Remember, the next step would be 6 data bearing nodes plus an arbiter, so the costs you save is less than 1/6 of your overall costs.
So more generally speaking, the more data bearing nodes you have (the higher your "replication factor" in Mongo terms) the less reasonable it becomes to have additional arbiters. Both from the technical point of view (the probability of a majority of nodes failing the same time becomes lower and lower) and the business point of view (with a high replication factor, the money saved with an arbiter in comparison to the overall costs becomes absurdly small).
Mnemonic:
The lowest uneven number is 1.
I have a scenario where I think having more than 1 Arbiter makes sense.
Problem
I have 3 data bearing nodes in a replicaset. Now I want to distribute my replicaset geographically so that I can mitigate the risk of a datacenter outage.
3 Node Replicaset, does not solve the problem
Primary Datacenter => 2 Data bearing Nodes
Backup Datacenter => 1 Data bearing Node
If that primary datacenter is down and the two out of three nodes in the replicaset would not be available then data bearing node in backup datacenter would not be able to become a primary since majority is not available. So the 3 node configuration does not solve the problem of a datacenter outage.
5 Node replicaset
Primary Datacenter => 2 Data bearing Nodes
Backup Datacenter => 1 Data bearing Node
Third Datacenter => 2 Arbiters
In this configuration I am able to sustain outage of any of the three datacenters and still be able to operate.
Obviously, a more ideal configuration would be to have 4 data bearing nodes and have 1 arbiter. It would give me redundancy in the backup datacenter as well. However since data bearing node is a much more expensive proposition than an arbiter going with 3 data bearing nodes and 2 arbiters makes more sense and I am happy to forgo the redundancy in backup datacenter in favor of the cost saving.
For our special case it makes sense to have 2 arbiters. Let me explain: we have 3 data centers but 1 of these 3 data centers is not suitable to host data bearing members. That's why we host in this data center 2 arbiters for each replica set. The 3 data bearing members of the replSet are hosted in the two other data centers (we want to have 3 instead of 2 data bearing members for resilience reasons). If 1 of the 3 data center goes down or is not reachable due to a network partition, the replSet is still able to elect a primary, thus it's still read and writeable. This wouldn't be possible with only 1 or 0 arbiter. Hence, 2 arbiters may make sense.
Let's see how it may look like. Here are 2 replSets, each with 3 data bearing members and 2 arbiters in 3 data centers, whereas DC3 is the restricted data center:
| |DC1 |DC2 |DC3 |
|----|-----|-----|-----|
|rs1 |m1,m2|m3 |a1,a2|
|rs2 |m1 |m2,m3|a1,a2|
If one data center goes down, which replSet member would become primary?
DC1 goes down:
rs1: m3
rs2: m2 or m3
DC2 goes down:
rs1: m1 or m2
rs2: m1
DC3 goes down:
rs1: m1,m2 or m3
Doing initial sync to the secondary is a very time consuming process, I haven't found anywhere in MongoDB docs that the primary can accept write operations during the initial Sync, or if not recommended. Is it safe to keep the primary operational (for write) during this process?
Thanks
In order for a primary to accept a write there have to be at least a quorum of voting replica set members available to vote and vote for the same primary. For instance for a 3 member replica set you need at least 2.
A secondary that is in initial sync should be in the Recovering state and according to the documentation can vote http://docs.mongodb.org/manual/reference/replica-states/:
3 RECOVERING Can vote. Members either perform startup self-checks,
or transition from completing a rollback or resync.
Now should you? I think the question depends on how many members were in the set before. If you've been running with 2 data nodes and 1 arbiter, running with 1 data node only for awhile is something only you can answer - yes it's riskier but what's your alternative, being down completely?
If you have 3 data nodes and 1 is down for an initial sync I don't see much issue unless you have very high data redundancy needs.
If you are starting from only having 1 node and you are transitioning into a replica set well you are no worse off then you were before.
Above all else always make certain you have at least 3 members of your replica set, preferably with at least 2 data nodes and generally speaking an odd number of voters.
Assume we setup a MongoDB replication without arbiter, If the primary is unavailable, the replica set will elect a secondary to be primary. So I think it's kind of implicit arbiter, since the replica will elect a primary automatically.
So I am wondering why do we need a dedicated arbiter node? Thanks!
I created a spreadsheet to better illustrate the effect of Arbiter nodes in a Replica Set.
It basically comes down to these points:
With an RS of 2 data nodes, losing 1 server brings you below your voting minimum (which is "greater than N/2"). An arbiter solves this.
With an RS of even numbered data nodes, adding an Arbiter increases your fault tolerance by 1 without making it possible to have 2 voting clusters due to a split.
With an RS of odd numbered data nodes, adding an Arbiter would allow a split to create 2 isolated clusters with "greater than N/2" votes and therefore a split brain scenario.
Elections are explained [in poor] detail here. In that document it states that an RS can have 50 members (even number) and 7 voting members. I emphasize "states" because it does not explain how it works. To me it seems that if you have a split happen with 4 members (all voting) on one side and 46 members (3 voting) on the other, you'd rather have the 46 elect a primary and the 4 to be a read-only cluster. But, that's exactly what "limited voting" prevents. In that situation you will actually have a 4 member cluster with a primary and a 46 member cluster that is read only. Explaining how that makes sense is out of the scope of this question and beyond my knowledge.
Its necessary to have a arbiter in a replication for the below reasons:
Replication is more reliable if it has odd number of replica sets. Incase if there is even number of replica sets its better to add a arbiter in the replication.
Arbiters do not hold data in them and they are just to vote in election when there is any node failure.
Arbiter is a light weight process they do not consume much hardware resources.
Arbiters just exchange the user credentials data between the replica set which are encrypted.
Vote during elections,hearbeats and configureation data are not encrypted while communicating in between the replica sets.
It is better to run arbiter on a separate machine rather than along with any one of the replica set to retain high availability.
Hope this helps !!!
This really comes down to the CAP theorem whereby it is stated that if there are equal number of servers on either side of the partition the database cannot maintain CAP (Consistency, Availability, and Partition tolerance). An Arbiter is specifically designed to create an "imbalance" or majority on one side so that a primary can be elected in this case.
If you get an even number of nodes on either side MongoDB will not elect a primary and your set will not accept writes.
Edit
By either side I mean, for example, 2 on one side and 2 on the other. My English wasn't easy to understand there.
So really what I mean is both sides.
Edit
Wikipedia presents quite a good case for explaining CAP: http://en.wikipedia.org/wiki/CAP_theorem
Arbiters are an optional mechanism to allow voting to succeed when you have an even number of mongods deployed in a replicaset. Arbiters are light weight, meant to be deployed on a server that is NOT a dedicated mongo replica, i.e: the server's primary role is some other task, like a redis server. Since they're light they won't interfere (noticeably) with the system's resources.
From the docs :
An arbiter does not have a copy of data set and cannot become a
primary. Replica sets may have arbiters to add a vote in elections of
for primary. Arbiters allow replica sets to have an uneven number of
members, without the overhead of a member that replicates data.
http://docs.mongodb.org/manual/core/replica-set-arbiter/
http://docs.mongodb.org/manual/core/replica-set-elections/#replica-set-elections