I'm trying to setup a 2 node Crate cluster, I have set the following configuration values on the 2 nodes:
gateway.recover_after_nodes: 1
gateway.expected_nodes: 2
However the check is failing as per the documentation:
(E / 2) < R <= E where R is the number of recovery nodes, E is the
number of expected nodes.
I see that most available documentation states a 3 node cluster, however at this point I can only start a 2 node cluster as a failover setup.
The behaviour I'm expecting is that if one of the nodes goes down the other node should be able to take up the traffic and once 2nd node comes back up it should sync up with new node.
If anyone has been able to successfully bring up a 2 node Crate cluster, please share the configuration required for the same.
Cheers
It doesn't make sense to run a two node cluster with 1 required node, because this could easily end up in a split brain and set the cluster into a state that it won't be able to recover, that's why you always need more then half of the number of expected nodes.
Related
We have 2 server location
A)location_1
B)location_2
total 5 server
location_1 -> have 2 server
location_2 ->have 3 server
we installed K8 in above 5 servers.in that 3 are master node which also act as worker node also 2 more worker node. But we need to attain HA for kubernative cluster
setup is like below
location_1 have 2 master node(it also act as worker node too)
location_2 have 1 master node (it also a worker node) and 2 more worker node -Total 3 nodes
but there might be chance of entire location1 or location 2 get down .So if that case how we can achieve HA as whole ,also individual location wise?
I have a situation where I have a cluster with a service (we named it A1) and its data which is on a remote storage like cephFS in my case. the number of replica for my service is 1. Assume I have 5 node in my cluster and service A1 reside in node 1. something happens with node 1 network and it lose the connectivity with cephFS cluster and my Kubernetes cluster as well (or docker-swarm). cluster mark it as unreachable and start a new service (we named it A2) on node 2 to keep replica as 1. after for example 15 min node 1 network fixed and node 1 get back to cluster and have service A1 running already (assume it didn't crash while it loses its connectivity with remote storage).
I worked with docker-swarm and recently switched to Kubernetes. I see Kuber has a feature call StatefulSet but when I read about it. it doesn't answer my question. (or I may miss something when I read about it)
Question A: what does cluster do. does it keep A2 and shutdown A1 or let A1 keeps working and shutdown A2 (Logically it should shutdown A1)
Question B (and my primary question as well!): Assume that the cluster wants to shutdown on of these services (for example A1). This service does some save on storage when it wants to shutdown. in this case state A1 save to disk and A2 with newer state saved something before A1 network get fixed.
There must be some locks when we mount the volume to the container in which when it attached to one container other container cant write to that (let A1 failed when want to save its old state data on disk)
The way it works - using docker swarm terminology -
You have a service. A service is a description of some image you'd like to run, how many replicas and so on. Assuming the service specifies at least 1 replica should be running it will create a task that will schedule a container on a swarm node.
So the service is associated with 0 to many tasks, where each task has 0 - if its still starting or 1 container - if the task is running or stopped - which is on a node.
So, when swarm (the orcestrator) detects a node go offline, it principally sees that a number of tasks associated with a service have lost their containers, and so the replication (in terms of running tasks) is no longer correct for the service, and it creates new tasks which in turn will schedule new containers on the available nodes.
On the disconnected node, the swarm worker notices that it has lost connection to the swarm managers so it cleans up all the tasks it is holding onto as it no longer has current information about them. In the process of cleaning the tasks up, the associated containers get stopped.
This is good because when the node finally reconnects there is no race condition where there are two tasks running. Only "A2" is running and "A1" has been shut down.
This is bad if you have a situation where nodes can lose connectivity to the managers frequently, but you need the services to keep running on those nodes regardless, as they will be shut down each time the workers detach.
The process on K8s is pretty much the same just change the terminology.
I recently converted a solr 7.x + zookeeper 3.4.14 to solr 8.3 + zk 3.5.6, and depending on how I start the solr nodes I'm geting a sync exception.
My setup uses 3 zk nodes and 2 solr nodes (let's call it A and B). The collection that has this problem has 1 shard and 2 replicas. I've noticed 2 situations: (1) which works fine and (2) which does not work.
1) This works: I start solr node A, and wait until it's replica is elected leader ("green" in the Solr interface 'Cloud'->'Graph') - which takes about 2 min; and only then start solr node B. Both replicas are active and the one in A is the leader.
2) This does NOT work: I start solr node A, and a few secs after I star solr node B (that is, before the 'A' replica is elected leader - still "Down" in the solr interface). In this case I get the following exception:
ERROR (coreZkRegister-1-thread-2-processing-n:192.168.15.20:8986_solr x:alldata_shard1_replica_n1 c:alldata s:shard1 r:core_node3) [c:alldata s:shard1 r:core_node3 x:alldata_shard1_replica_n1] o.a.s.c.SyncStrategy Sync Failed:java.lang.IndexOutOfBoundsException: Index -1 out of bounds for length 99
It seems that if both solr node are started soon after each other, then ZK cannot elect one as leader.
This error only appears in the solr.log of node A, even if I invert the order of starting nodes.
Has anyone seen this before?
I have several other collections which do not show this problem.
Thanks!
I am learning kubernetes by following the official documentation and in the Creating Highly Available clusters with kubeadm part it's recommended to use 3 masters and 3 workers as a minimum required to set a HA cluster.
This recommendation is given with no explanation about the reasons behind it. In other words, why a 2 masters and 2 workers configuration is not ok with HA ?
You want an uneven number of master eligible nodes, so you can form a proper quorum (two out of three, three out of five). The total number of nodes doesn't actually matter. Smaller installation often make the same node master eligible and data holding at the same time, so in that case you'd prefer an uneven number of nodes. Once you move to a setup with dedicated master eligible nodes, you're freed from that restriction. You could also run 4 nodes with a quorum of 3, but that will make the cluster unavailable if any two nodes die. The worst setup is 2 nodes since you can only safely run with a quorum of 2, so if a node dies you're unavailable.
(This was an answer from here which I think is a good explanation)
This why exactly:
https://etcd.io/docs/v3.3/faq/#why-an-odd-number-of-cluster-members
check then the concept of quorum, you can find plenty of info especially in the pacemaker/corosync documentation
in akka cluster i would like to know a few more details about how the cluster works.
If i have a seed from A to B,C and from C to D and from D to E
then what if nodes D and E are restarted and D does not come up will E know the rest of the cluster? if not isn't that considered a problem?
Assuming you have multiple seed nodes (A to E) and question is not aimed at cluster state management/convergence.
yes, seed node E will try to join the other seed nodes (A,B,C) even though node D is not up. As in the Cluster Usage documentation
Once more than two seed nodes have been started it is no problem to
shut down the first seed node. If the first seed node is restarted it
will first try join the other seed nodes in the existing cluster.