I have kubernetes cluster working fine. I have one master node and 5 worker nodes, and all these are running pods. When all the nodes are on and if the kubernetes master goes down/ powered off, will the worker nodes keep working as normally.?
If master node is down and one of the worker node also goes down and then come back online after sometime. Then will the pod automatically be started on that worker as the master node is still down.?
When all the nodes are on and if the kubernetes master goes down/ powered off, will the worker nodes keep working as normally.?
Yes, they will work in their last state.
If master node is down and one of the worker node also goes down and then come back online after sometime. Then will the pod automatically be started on that worker as the master node is still down.?
No.
As you can read in Kubernetes Components section:
Master components provide the cluster’s control plane. Master components make global decisions about the cluster (for example, scheduling), and detecting and responding to cluster events (starting up a new pod when a replication controller’s ‘replicas’ field is unsatisfied).
Related
I am creating kubernetes cluster which include: 1 master node (M1), 2 worker nodes (W1 and W2)
Using deployment to create pods with replica count 5.
If pod dies kubernetes is re creating the pods. Count remains 5.
Lets suppose if W2 worker node dies due to any reason.
In this case does kubernetes will create a new node or just run all the replicas on the same node W1.
If i want to restore the died node automatically how can i do that?
This mostly depends on how you deployed things. Most cloud-integrated installers and hosted providers (GKE, EKS, AKS, Kops) use a node group of some kind so a fully failed node (machine terminated) would be replaced at that level. If the node is up but jammed, that would generally be solved by cluster-autoscaler starting a new node for you. Some installers that don't make per-cloud assumptions (Kubespray, etc) leave this up to you to handle yourself.
From what I've read about Kubernetes, if the master(s) die, the workers should still be able to function as normal (https://stackoverflow.com/a/39173007/281469), although no new scheduling will occur.
However, I've found this to not be the case when the master can also schedule worker pods. Take a 2-node cluster, where one node is a master and the other a worker, and the master has the taints removed:
If I shut down the master and docker exec into one of the containers on the worker I can see that:
nc -zv ip-of-pod 80
succeeds, but
nc -zv ip-of-service 80
fails half of the time. The Kubernetes version is v1.15.10, using iptables mode for kube-proxy.
I'm guessing that since the kube-proxy on the worker node can't connect to the apiserver, it will not remove the master node from the iptables rules.
Questions:
Is it expected behaviour that kube-proxy won't stop routing to pods on master nodes, or is there something "broken"?
Are any workarounds available for this kind of setup to allow the worker nodes to still function correctly?
I realise the best thing to do is separate the CP nodes but that's not viable for what I'm working on at the moment.
Is it expected behaviour that kube-proxy won't stop routing to pods on
master nodes, or is there something "broken"?
Are any workarounds
available for this kind of setup to allow the worker nodes to still
function correctly?
The cluster master plays the role of decision maker for the various activities in cluster's nodes. This can include scheduling workloads, managing the workloads' lifecycle, scaling etc.. Each node is managed by the master components and contains the services necessary to run pods. The services on a node typically includes the kube-proxy, container runtime and kubelet.
The kube-proxy component enforces network rules on nodes and helps kubernetes in managing the connectivity among Pods and Services. Also, the kube-proxy, acts as an egress-based load-balancing controller which keeps monitoring the the kubernetes API server and continually updates node's iptables subsystem based on it.
In simple terms, the master node only is aware of everything and is in charge of creating the list of routing rules as well based on node addition or deletion etc. kube-proxy plays a kind of enforcer whereby it takes charge of checking with master, syncing the information and enforcing the rules on the list.
If the master node(API server) is down, the cluster will not be able to respond to API commands or deploy nodes. If another master node is not available, there shall be no one else available who can instruct the worker nodes on change in work allocation and hence they shall continue to execute the operations that were earlier scheduled by the master until the time the master node is back and gives different instructions. Inline to it, kube-proxy shall also be unable to get the latest rules by sync up with master, however it shall not stop routing and shall continue to handle the networking and routing functionalities (uses the earlier iptable rules that were determined before the master node went down) that shall allow network communication to your pods provided all pods in worker nodes are still up and running.
Single master node based architecture is not a preferred deployment architecture for production. Considering that resilience and reliability is one of the major business goal of kubernetes, it is recommended as a best practice to have HA cluster based architecture to avoid single point of failure.
Once you remove taints, kubernetes scheduler don't need any tolerations to schedule pods on your master node. So it is as good as your worker node with control plane components running on it and you can also run your workload pods on this node (although its not a recommended practice).
Kube-proxy (https://kubernetes.io/docs/concepts/overview/components/#kube-proxy) is the component deployed on all the nodes of cluster and it handles the networking and routing connection to your pods. So, even if your master node is down kube-proxy still works fine on the worker node and it will route traffic to your pods running on worker node.
If all your pods are running in worker nodes (which are still up and running), then kube-proxy will continue to route traffic to your pods even via service.
There is nothing inherent in Kubernetes that would cause this. The master node role is just for humans, and if you've removed the taints then the nodes are just normal nodes. That said, remember that usual rules about scheduling and resource requests apply so if your pods don't all fit then things wouldn't be scheduled. It's possible your Kubernetes deploy system set up more specialized firewall rules or similar around the control plane nodes, but that would be dependent on that system.
What would be the behavior of a multi node kubernetes cluster if it only has a single master node and if the node goes down?
The control plane would be unavailable. Existing pods would continue to run, however calls to the API wouldn't work, so you wouldn't be able to make any changes to the state of the system. Additionally self-repair systems like pods being restarted on failure would not happen since that functionality lives in the control plane as well.
You wouldn't be able to create or query kubernetes objects(pods, deployments etc) since the required control plane components(api-server and etcd) are not running.
Existing pods on the worker nodes will keep running. If a pod crashes, kubelet on that node would restart it as well.
If worker node goes down while master is down, even the pods created by a controllers like deployment/replicaset won't be re-scheduled to different node since controller-manager(control plane component) is not running.
I am new to the Kubernetes and cluster.
I would like to bring up an High Availability Master Only Kubernetes Cluster(Need Not to!).
I have the 2 Instances/Servers running Kubernetes daemon, and running different kind of pods on both the Nodes.
Now I would like to somehow create the cluster and if the one of the host(2) down, then all the pods from that host(2) should move to the another host(1).
once the host(2) comes up. the pods should float back.
Please let me know if there is any way i can achieve this?
Since your requirement is to have a 2 node master-only cluster and also have HA capabilities then unfortunately there is no straightforward way to achieve it.
Reason being that a 2 node master-only cluster deployed by kubeadm has only 2 etcd pods (one on each node). This gives you no fault tolerance. Meaning if one of the nodes goes down, etcd cluster would lose quorum and the remaining k8s master won't be able to operate.
Now, if you were ok with having an external etcd cluster where you can maintain an odd number of etcd members then yes, you can have a 2 node k8s cluster and still have HA capabilities.
It is possible that master node serves also as a worker node however it is not advisable on production environments, mainly for performance reasons.
By default, kubeadm configures master node so that no workload can be run on it and only regular nodes, added later would be able to handle it. But you can easily override this default behaviour.
In order to enable workload to be scheduled also on master node you need to remove from it the following taint, which is added by default:
kubectl taint nodes --all node-role.kubernetes.io/master-
To install and configure multi-master kubernetes cluster you can follow this tutorial. It describes scenario with 3 master nodes but you can easily customize it to your needs.
I am running a Kubernetes cluster with 3 master and 3 nodes.
I have found this to auto-scale worker nodes based on the pod's status.
https://github.com/kubernetes/autoscaler/tree/master/cluster-autoscaler/cloudprovider/aws
But, I couldn't find any blog or add-on to auto-scale master nodes.
Is there any reason to auto-scale master nodes, if yes how can we do that?
There is no need to autoscale the master nodes. In a practical world, your worker nodes responsibility is to run your work load and your master nodes responsibility is to make sure that your worker nodes are having desired state in the cluster.
Now all the end users will request your application (pods) and as the load increased they need to scale horizontally and more pods should be spawned. If the resources on worker nodes are insufficient to run those nodes, more worker nodes should be spawned.
In large cluster we do not run load on master node, but we need to make sure it is highly available so that there is no single point of failure to orchestrate the worker nodes. For that we can have 3 master multi-master cluster in place.
Worker nodes we worry about the horizontal scalability and In master node we worry about high availability.
But for building large cluster, you need to provide adequate resources to master nodes for handling the orchestration of load on worker nodes.
For more information on building large cluster, please refer official document:
https://kubernetes.io/docs/setup/cluster-large/
In a nutshell, You can even have one master for 1000 worker nodes if you provide enough resources to that node. So, there is no reason to autoscale master comparing to the challenges we face in doing so.