I am learning kubernetes by following the official documentation and in the Creating Highly Available clusters with kubeadm part it's recommended to use 3 masters and 3 workers as a minimum required to set a HA cluster.
This recommendation is given with no explanation about the reasons behind it. In other words, why a 2 masters and 2 workers configuration is not ok with HA ?
You want an uneven number of master eligible nodes, so you can form a proper quorum (two out of three, three out of five). The total number of nodes doesn't actually matter. Smaller installation often make the same node master eligible and data holding at the same time, so in that case you'd prefer an uneven number of nodes. Once you move to a setup with dedicated master eligible nodes, you're freed from that restriction. You could also run 4 nodes with a quorum of 3, but that will make the cluster unavailable if any two nodes die. The worst setup is 2 nodes since you can only safely run with a quorum of 2, so if a node dies you're unavailable.
(This was an answer from here which I think is a good explanation)
This why exactly:
https://etcd.io/docs/v3.3/faq/#why-an-odd-number-of-cluster-members
check then the concept of quorum, you can find plenty of info especially in the pacemaker/corosync documentation
Related
I made a OKD 4.9 cluster.
I successfully installed openshift and made a cluster, but I got a question now.
There is a sentence in OKD document.
"The only supported values is 3, which is the default value"
I think there is an important reason to do not use 5, 7 masters.
Could you guys tell me why?
I think there is an important reason to do not use 5, 7 masters.
Could you guys tell me why?
Technically it is still possible to have 5 or 7 Master Nodes, there is nothing in place to stop you. It is just not supported, so if you have a problem with it you'll have to deal with it yourself.
However, you should understand why this recommendation is there. Many Kubernetes Master components (Controller Manager, Scheduler) are only active on one single Master Node anyway, while the same component on all other Masters are not active. API Server is active-active, so API Server scales nicely.
For etcd, Raft is leader-based; the leader handles all client requests which need cluster consensus. Any request that requires consensus sent to a follower is automatically forwarded to the leader. So for write operations or for consistent reads, there is an overhead if you have more than 3 etcd members. If you have more etcd members, this overhead gets larger, but YMMV.
So it comes down to additional overhead with 5 or 7 Master Nodes.
I've been experimenting with Vert.x high availability features to test horizontal scalability and resiliency. I have a cluster of several nodes based on Hazelcast. I'm creating verticles on any nodes via an HTTP API. Verticles have the HA flag set when they are created.
Testing scalability
If I have n nodes Nn loaded with HA-verticles and if I add one additional node there is no verticle that is migrated from the Nn node on the new one so that the load would be balanced. Is there a way to tell Vert.x to do so, or not ? I believe it's not so simple...
Testing resilience
If I have n nodes Nn loaded with HA-verticles and I kill one of the nodes, all the verticles from that very node are migrated, but are migrated on one single of the remaining nodes that is not always the least loaded one. That destination node may become overloaded and the whole cluster would be at risk of freeze or crash. Same question as before: is there a way to force Vert.x to balance the restarted verticles on all nodes, or at least on the node that is the least loaded ?
Your observations are correct, there is no way:
to distribute verticles from a failed node over the rest of the nodes
to prevent starting verticles in a node that is already loaded
Improving the HA features is not on the Vert.x roadmap.
If, as it seems, you need more than basic failover, I would recommend to use specialized infrastructure tools that can leverage info from monitoring systems and start/stop new nodes as needed.
i want to build a 3 nodes (avoid split brain) symmetric cluster with high availability using replication. In addition I would like to be able to load balanced messages between nodes
how should this be achieved?
option 1: 1 master with 2 slaves
option 2: 3 colocated master/slave
nodes
Option 1 isn't really an option as the slaves will not participate in the voting process which means split-brain will not be mitigated. The only option you have left (of the 2 you listed, of course) is to use 3 colocated master/slaves.
I am trying to configure a Active/Passive cluster with two nodes (using OpenShift). The second passive node should be a hot standby, in other words it is up and running but not doing anything, until the first node dies. Then the passive node becomes active and a new passive node is started.
I have read the High Availability documentation, however it just seems to cover the theory. Furthermore it seems like overkill ( I am thinking there might be an easier way to meet my goal).
Where would I start?
What you are asking for goes against the usual practice for how Kubernetes/OpenShift is used. You wouldn't have hot standby nodes, you would always use all nodes in the cluster. You would then allow for enough additional capacity in your cluster such that loosing a node doesn't cause a problem as other nodes would have enough capacity to then run the applications. In this scenario the Kubernetes scheduler would automatically restart any applications which were on a failed node on the other nodes in the cluster, without you needing to perform any explicit failover steps.
So don't try and do anything special, setup your cluster with the two nodes, with applications being distributed across both. If you need to have the ability to run with only a single node, make sure it has enough capacity to run everything. If over time you add more applications and one node is not enough, add a third node, with all three being used in normal case. You can then handle failure of a single node again.
I wonder about the best strategy with regard to Zookeeper and SolrCloud clusters. Should one Zookeeper cluster be dedicated per SolrCloud cluster or multiple SolrCloud clusters can share one Zookeeper cluster? I guess the former must be a very safe approach but I am wondering if the 2nd option is fine as well.
As far as I know, SolrCloud use Zookeeper to share cluster state (up, down nodes) and to load core shared configurations (solrconfig.xml, schema.xml, etc...) on boot. If you have clients based on SolrJ's CloudSolrServer implementation than they will mostly perform reads of the cluster state.
In this respect, I think it should be fine to share the same ZK ensemble. Many reads and few writes, this is exactly what ZK is designed for.
SolrCloud puts very little load on a ZooKeeper cluster, so if it's purely a performance consideration then there's no problem. It would probably be a waste of resources to have one ZK cluster per SolrCloud if they're all on a local network. Just make sure the ZooKeeper configurations are in separate ZooKeeper paths. For example, using -zkHost :/ for one SolrCloud, and replace "path1" with "path2" for the second one will put the solr files in separate paths within ZooKeeper to ensure they don't conflict.
Note that the ZK cluster should be well-configured and robust, because if it goes down then none of the SolrClouds are going to be able to respond to changes in node availability or state. (If SolrCloud leader is lost, not connectable, or if a node enters recovering state, etc.)