I am exploring the implementation of CI/CD pipeline using kubernetes and jenkins. I am planned to use multi master kubernetes cluster - HA cluster. And using the etcd cluster on the master nodes. I am following the below tutorial for learning,
https://kubernetes.io/docs/setup/independent/high-availability/
Here when I am reading I found that - After Run kubeadm init on master1 and master2, there is a step "Add master1 and master2 to load balancer".
Actually I am confused in this step that , Do I need to use another VM as a load balancer ? Or master0 master1 and master2 setup defaultly giving one load balancer ?
I am not clear about this step about load balancer in HA kubernetes cluster. Can anyone clarify my confusions please?
The short answer is yes, you have to create Load Balancer for masters and add other masters to LB pool manually.
Here is a part from the "Building High-Availability Clusters" document, "Replicating API Servers" section:
Load balancing
At this point, you should have 3 apiservers all working correctly. If you set up a network load balancer, you should
be able to access your cluster via that load balancer, and see traffic
balancing between the apiserver instances. Setting up a load balancer
will depend on the specifics of your platform, for example
instructions for the Google Cloud Platform can be found here.
Note, if you are using authentication, you may need to regenerate your
certificate to include the IP address of the balancer, in addition to
the IP addresses of the individual nodes.
For pods that you deploy into the cluster, the kubernetes service/dns
name should provide a load balanced endpoint for the master
automatically.
For external users of the API (e.g. the kubectl command line
interface, continuous build pipelines, or other clients) you will want
to configure them to talk to the external load balancer’s IP address.
Please also find two parts, related to Cloud load balancer and On Site load balancer in the Creating HA clusters with kubeadm document.
(It's hidden above "Acquire etcd cert") in "Choose one.../Cloud/On-Site" section.
Related
Let's say I have a web application Backend that I want to deploy with the help of Kubernetes, how exactly does scaling work in this case.
I understand scaling in Kubernetes as: We have one a master node that orchestrates multiple worker nodes where each of the worker nodes runs 0-n different containers with the same image. My question is, if this is correct, how does Kubernetes deal with the fact that the same application use the same Port within one worker node? Does the request reach the master node which then handles this problem internally?
Does the request reach the master node which then handles this problem internally?
No, the master nodes does not handle traffic for your apps. Typically traffic meant for your apps arrive to a load balancer or gateway, e.g. Google Cloud Load Balancer or AWS Elastic Load Balancer, then the load balancer forwards the request to a replica of a matching service - this is managed by the Kubernetes Ingress resource in your cluster.
The master nodes - the control plane - is only used for management, e.g. when you deploy a new image or service.
how does Kubernetes deal with the fact that the same application use the same Port within one worker node?
Kubernetes uses a container runtime for your containers. You can try this on your own machine, e.g. when you use docker, you can create multiple containers (instances) of your app, all listening on e.g. port 8080. This is a key feature of containers - the provide network isolation.
On Kubernetes, all containers are tied together with a custom container networking. How this works, depends on what Container Networking Interface-plugin you use in your cluster. Each Pod in your cluster will get its own IP address. All your containers can listen to the same port, if you want - this is an abstraction.
I have a same discussion in k3s github repository, but no one reply. Hope someone can give an answer here.
There are articles talking about the embedded etcd HA solution of k3s like this. One of the key behavior is adding a load balancer solution (EIP like this article or LB from the clound provider) between the agents and masters:
k3s agent --> load balancer --> master
And the architecture of k3s also show that a Fixed Registration Address is necessary.
While, after some research I found that k3s (at least v1.21.5+k3s2) have a internal agent load balancer (config at /var/lib/rancher/k3s/agent/etc/k3s-agent-load-balancer.yaml) which will auto update the master k8s api server list in it. So the out side load balancer is unnecessary?
I got a response from the k3s discussion:
https://github.com/k3s-io/k3s/discussions/4488#discussioncomment-1719009
Our documentation lists a requirement for a "fixed registration endpoint" so that nodes do not rely on a single server being online in order to join the cluster. This endpoint could be a load-balancer or a DNS alias, it's up to you. This is only needed when nodes are registering to the cluster; once they have successfully joined, they use the client load-balancer to communicate directly with the servers without going through the registration endpoint.
I think this is good enough to answer this question.
Yes, an external load balancer is still required to achieve a highly available setup with more than one master node.
Whenever you start a worker node or use the API, you should connect to the external load balancer to ensure you can connect to a running master node if one master is currently down.
The internal load balancer you mentioned above distributes any load within your cluster.
I have created python web-server and created docker image for that. Secondly I created Kubernetes cluster with kubeadm and created a service with type load balancer but I got External IP in Pending state so I came to know that External Load balancer can not be created in kubeadm tool.
So my task is to create pod replica of web-server and access through Pubic IP and the request should be load balanced between pod.
Can someone help me for this ?
I see two options in your scenario an choosing between them is up to you based in your needs.
The fact it's pending forever is expected behavior.
By default, the solution proposed by Kubeadm requires you to define a cloud-provider to be able to use resources like LoadBalancer offered by your Cloud Provider.
Another option is to use a out-of-box solutions as MetalLB.
MetalLB isn't going to allocate an External IP for you, it will allocate a internal IP inside our VPC and you have to create the necessary routing rules to route traffic to this IP.
I am trying to create Kubernetes cluster having 1 master and 2 worker nodes by using the tool kubeadm in my on-premise machines. I am following the Kubernetes official documentation for forming the cluster from the following url:
https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/high-availability/
After installing all the runtime and completing before begin pre-requistics steps, I found in the document as the first step of forming the cluster is Create load balancer for kube-apiserver.
My Doubt
When I created the single master 3 worker nodes cluster using kubespray tool, I did not created any separate load balancer for that. So here when I am following the kubeadm tool, Do I need to create the load balancer actually for forming ?
Why are both tools showing different way, Since I did not created load balancer by using kubespray tool. Now I am trying to create cluster with kubeadm tool.
Speaking of load balancers creation during Kubernetes deployment using Kubeadm it depends on your setup. It is not mandatory to setup load balancer. Your cluster will still work, but without load balancing, it's going to be hard to qualify this cluster as HA.
In a single master setup as it is in your case, the master node manages the etcd database, API server, controller manager and scheduler, along with the worker nodes. However, if that single master node fails, all the worker node fail as well and entire cluster will be lost.
Learn more here: kubernetes-ha-kubeadm.
Kubeadm covers the needs of a life-cycle management for Kubernetes clusters, including self-hosted layouts, dynamic discovery services, etc. Kubespray is more about generic configuration, initial clustering, and bootstrapping.
Kubespray is a good choice when you either are familiar with Ansible or seek a possibility to switch between multiple platforms. If your priority is tight integration with unique features offered by the supported clouds, and you plan to stick with your provider, kops may be a better option.
Deploying a loadbalancer is up to a user and is not covered by ansible roles in Kubespray. By default, it only configures a non-HA endpoint, which points to the access_ip or IP address of the first server node in the kube-master group. It can also configure clients to use endpoints for a given loadbalancer type. More information you can find here: kubespray-lb.
Here you have comparision of Kubernetes deployment tools: Kubernetes Deployment Tools.
I am currently trying to form a high availability Kubernetes cluster having 5 worker nodes and 3 master in on-premise server. I learned about implementation of high availability cluster by checking its documentation. Also I understood the implementation of HA cluster on AWS cloud or Azure cloud using Load Balancer functionality from appropriate cloud provider.
My confusion is that, when I am creating the same high availability Kubernetes cluster in my on-premise server, then how I can use the Load Balancer functionality in implementation ?
You can use keepalived to setup the load balancer for master on your on premise setup. The keepalived daemon can be used to monitor services or systems and to automatically failover to a standby if problems occur. There will be one Active server on the master and other two master will be in backup mode.
I have written a blog on how to setup kubernetes highly available cluster on premise. You can find it at below link:
https://velotio.com/blog/2018/6/15/kubernetes-high-availability-kubeadm
I have used keepalived to setup the load balancer in above blog on my on-premise cluster.
The way we do it is that we put cluster of loadbalancers (simple nginx in reverse proxy in HA) in front of the K8s API and in front of the Ingress.