We have 2 server location
A)location_1
B)location_2
total 5 server
location_1 -> have 2 server
location_2 ->have 3 server
we installed K8 in above 5 servers.in that 3 are master node which also act as worker node also 2 more worker node. But we need to attain HA for kubernative cluster
setup is like below
location_1 have 2 master node(it also act as worker node too)
location_2 have 1 master node (it also a worker node) and 2 more worker node -Total 3 nodes
but there might be chance of entire location1 or location 2 get down .So if that case how we can achieve HA as whole ,also individual location wise?
Related
We have 2 datacenters (A and B) on 2 different regions (Eastern and Central). I've setup K8S single cluster where master is on datacenter A and few nodes on both datacenters. Our application contains 3 images (AppServer, DB, ReportingServer). Consider these 2 scenarios when the application is deployed:
1 - All 3 pods are created on the nodes belong to datacenter A. Everything works fine.
2 - DB pod or AppServer pod is created on a node belong to datacenter B and the other 2 pods are created on datacenter A. In this case application is very slow, takes 10-15 mins for it to be in running state (instead of 2-3 mins), login page loads very slowly and logging to the application usually throws error due to timeout.
My question: Is it normal for K8S to behave like this if nodes are in different datacenters? Or my setup is wrong?
i want to build a 3 nodes (avoid split brain) symmetric cluster with high availability using replication. In addition I would like to be able to load balanced messages between nodes
how should this be achieved?
option 1: 1 master with 2 slaves
option 2: 3 colocated master/slave
nodes
Option 1 isn't really an option as the slaves will not participate in the voting process which means split-brain will not be mitigated. The only option you have left (of the 2 you listed, of course) is to use 3 colocated master/slaves.
I'm trying to setup a 2 node Crate cluster, I have set the following configuration values on the 2 nodes:
gateway.recover_after_nodes: 1
gateway.expected_nodes: 2
However the check is failing as per the documentation:
(E / 2) < R <= E where R is the number of recovery nodes, E is the
number of expected nodes.
I see that most available documentation states a 3 node cluster, however at this point I can only start a 2 node cluster as a failover setup.
The behaviour I'm expecting is that if one of the nodes goes down the other node should be able to take up the traffic and once 2nd node comes back up it should sync up with new node.
If anyone has been able to successfully bring up a 2 node Crate cluster, please share the configuration required for the same.
Cheers
It doesn't make sense to run a two node cluster with 1 required node, because this could easily end up in a split brain and set the cluster into a state that it won't be able to recover, that's why you always need more then half of the number of expected nodes.
Let say I want to achieve maximum useable capacity with data resilience on this 3 OSD nodes setup where each node contains 2x 1TB OSDs.
Is it safe run 3 Ceph nodes with 2-way replication?
What are the pros and cons of using 2-way? Will it cause data split-brain?
Last but not least, what domain fault tolerance will it be running on 2-way replication?
Thanks!
Sometimes, even three replica is not enough, e.g. if ssd disks (from cache) fail together or one by one.
http://lists.ceph.com/pipermail/ceph-users-ceph.com/2015-October/005672.html
For two osd you can even set manually 1 replica for minimum and 2 replicas for maximum (I didn't managed to set it automatically in the case of one failed osd of all three osds):
osd pool default size = 2 # Write an object 2 times
osd pool default min size = 1 # Allow writing 1 copy in a degraded state
But this command: ceph osd pool set mypoolname set min_size 1
sets it for a pool, not just the default settings.
For n = 4 nodes each with 1 osd and 1 mon and settings of replica min_size 1 and size 4 three osd can fail, only one mon can fail (the monitor quorum means more than half will survive). 4 + 1 number of monitors is required for two failed monitors (at least one should be external without osd). For 8 monitors (four external monitors) three mon can fail, so even three nodes each with 1 osd and 1 mon can fail. I am not sure that setting of 8 monitors is possible.
Thus, for three nodes each with one monitor and osd the only reasonable settings are replica min_size 2 and size 3 or 2. Only one node can fail.
If you have an external monitors, if you set min_size to 1 (this is very dangerous) and size to 2 or 1 the 2 nodes can be down. But with one replica (no copy, only original data) you can loose your job very soon.
We have the following multi data-center Scenario
Node1 --- Node3
| |
| |
| |
--- ---
Node2 Node4
Node1 and Node3 form a Replica (sort of) Set ( for high availability )
Node 2/Node 4 are Priority 0 members (They should never become Primaries - Solely for read purpose)
Caveat -- what is the best way to design such a situation, since Node 2 and Node4 are not accessible to one another, given the way we configured our VPN/Firewalls;
essentially ruling out any heartbeat between Node2 and Node4.
Thanks Much
Here's what I got in mind:
Don't keep even members in a set. Thus you need another arbiter or set one of node2/4 to non-voting member.
As I'm using C# driver, I'm not sure you are using the same technology to build your application. Anyway, it turns out C# driver obtain a complete available server list from seeds (servers you provided in connection string) and tries to load-balancing requests to all of them. In your situation, I guess you would have application servers running in all 3 data centers. However, you probably don't want, for example, node 1 to accept connections from a different data center. That would significantly slow down the application. So you need some further settings:
Set node 3/4 to hidden nodes.
For applications running in the same data center with node 3/4, don't config the replicaSet parameter in connection string. But config the readPreference=secondary. If you need to write, you'll have to config another connection string to primary node.
If you make the votes of 2 and 4 also 0 then it should act, in failover as though 1 and 2 are only eligible. If you set them to hidden you have to forceably connect to them, MongoDB drivers will intentionally avoid them normally.
Other than that node 2 and 4 have direct access to whatever would be the primary as such I see no other problem.