Can anyone please help me understand the ingress traffic flow to a pod in kubernetes? Any web links or documents are much appreciated.
In my application there is a intermittent connection timed out so i want to understand how the traffic is flowing in to cluster and where do i need to enable tcpdump to understand what is happening when there is timeout.
Your question does not contain enough information to give you a detailed answer. There are different types of ingress controllers, and load balancers as well.
So, suppose:
you are using Azure Kubernetes Service
you are using Azure Load Balancer
you have two types of backend pods, each has its own dedicated service
you are using Nginx as ingress controller which is able to do LAYER 7 (OSI) load balancing
Nginx has also its own pods and a service sits in front of these pods. This service has a Service IP which is available only within the AKS cluster. Due to this, additionally you can use Azure Load Balancer (ALB) to make your backend pods available for the public. ALB is a layer 4 load balancer, which sends the incoming traffic to the worker nodes.
Kube-proxy is running on every worker nodes and able to recognize that the traffic from the ALB was destined to the Nginx service.
See the flow on the image below:
Related
In al the tutorials about Kubernetes cluster I have read I didn't see that they mention to 2 load balancers, but only one for the ingress pods.
However, in a proper production environment, should's we have 2 different load balancers?
to balance between the master nodes for requests to the ApiServer.
to balance between the Ingress podes to control the external traffic.
to balance between the master nodes for requests to the ApiServer.
For all production environments its advised to have load
balancer for API Server. This is the first step as part of K8S HA mode creation. More details are in k8s documentation
to balance between the Ingress podes to control the external traffic.
You are correct for this also it’s definitely required to handle external traffic. All the ingress services are created of LoadBalancer Type in their implementations.
I've different deployments over different namespaces and I would like to expose some of them to the Internet, even if I don't have a static and public IP available.
The different services are deployed on Rancher k3s and every service which should be publicly accessible has an Ingress defined in the same namespace.
I was trying to follow Rancher - How to expose my services publicly?, but I didn't really get what I've to do and, moreover:
Why do we need to define a LoadBalancer? It seems to me that the IngressController used by k3s (Traefik?) already creates one. If this is a must (or a good way to go), how it should the service defined exactly?
I don't have any Rancher UI in my environment. Therefore, is there a way to achieve what described in that link in a declarative way?
Is there a way to use services like No-IP or FreeDNS for the final hostname?
If I get it right, you deployed Kubernetes manually on barebone/vms nodes and now you want to reach you deployments running inside that cluster.
There is two level of loadbalancing in this setup, the one managed by your ingress controller, sounds like it is traefik in your case, and it is recommanded to run a second L4 load balancer in front of your workers to reach the ingress pods that are usually deployed on multiple/all nodes. Traefik, or other lb controllers, will load balancer traffic inside the k8s cluster without issue even if you don't have a L4 load balancer, but it is not recommanded as if you loose this node, no traffic can reach the kubernetes cluster anymore. You "just" need to have your dns resolution pointing at your public ip and routed to one of your worker, or the LB in front of it. However, if you don't have a L4 LB, you'll need to have your ingress pods listening on ports 80 and/or 443.
Most things that you do in Rancher UI is just an easier way to see your k8s objects, all ingress configuration can be achieved via kubectl, k9s (strongly recommand thatone!), lens or other methods. However k8s objects are still k8s objects. In this case, you need to have your services exposed with ClusterIP that are then reachable by the ingress pods.
I've never used such a solution natively from k8s, but when I had too the internet router was able to do this part, once you're there, it is internal routing.
I hope this helps. Ingress can definitely be a tough one to grasp!
Basically I have Following Hdfs Cluster setup using docker-compose:
Node 1 with IP: 192.168.1.1 having service deployed as below:
Namenode1:9000
HMaster1: 8300
ZooKeeper1:1291
Node 2 with IP: 192.168.1.2 having service deployed as below:
Namenode2:9000
ZooKeeper2:1291
How does Traefik / Ngnix - (Ingress Controllers) forwards request to two different services having configured with same port number?
There are several great tutorials on how ingress and load balancing works in kubernetes, e.g. this one by Mark Betz. As a general rule, it helps to think in terms of services and workloads instead of specific nodes where your workloads are running on.
A workload deployed in Kubernetes (a so called Pod) has its own internal IP address, called a ClusterIP. That pod can have one or more ports open, just on that pod-owned ip address.
If you now have several pods to distribute the load, e.g. like 5 web server processes or backend logic, it would be hard for a client (inside the cluster) to keep track of all those pod IPs, because they also change when a pod is updated or just restarted due to a crash. This is why Kubernetes has a so called concept of services. Those provide a stable DNS name and IP which then transparently "forwards" to one of the healthy pods. So your client only needs to know the DNS name and not keep track of the specific pod IPs.
If you now want to expose such a service to the public, there are different ways. Either you set your service to type: LoadBalancer which then sets up some load balancer infrastructure on your cloud provider and routes traffic to the nodes and then to the pods - or - you already have an ingress controller in place and just define the routing based on host names and paths. An ingress controller itself is such a loadbalanced service with an attached cloud load balancer and also has some pods (with e.g. a traefik or nginx container) which then route your packets accordingly.
So coming back to your initial question: If you want to expose a service with several pods of the same kind, then you would first create a Service resource that matches your Pods using the selector and then you create one single ingress resource that provides a hostname/path and references this service. The ingress controller will pick up those ingress resources and configure the traefik or nginx accordingly. The ingress controller doesn't really care about the host IPs and port numbers, because it acts on the internal kubernetes ClusterIPs, so you even don't need (and shouldn't) expose such a service directly when you have an ingress in place.
I hope this answers your question regarding exposing two workloads over an ingress controller. For details, check the Kubernetes docs on Ingresses. Based on the services you named (zookeeper, hdfs) load balancing and ingresses might not be what you need for that case. Zookeeper instances should be internal in most cases and need to be adressed individually, so you might want to check out headless services, for this use case. Also check the Kubernetes docs for a way to run zookeeper.
When using an external load balancer with istio ingress gateways (multiple replicas spread across different nodes), how does it identify which istio ingress gateway it can possibly hit i.e. I can manually access nodeip:nodeport/endpoint for any node manually but how is an external load balancer expected to know all nodes.
Is this manually configured or does the load balancer consume this info from an API
Is there a recommended strategy for bypassing an external load balancer eg. roundrobin across a DNS which is aware of the node ip / port ?
The root of this question is - how do we avoid a single point of failure . Using multiple istio ingress gateway replicas achieves this in istio but then the the external load balancer / load balancer cluster needs to know the replicas . Is this automated or a manual config or is there a single virtual endpoint that the external load balancer hits?
External load balancers are generally configured to do health check on your set of nodes (over /healthz endpoint or some other method), and balance the incoming traffic using an LB algorithm, by sending the packets it receives to one of the healthy nodes over the service's NodePort.
In fact, that's mostly the reason why NodePort type services exist in the first place - they don't have much of an usage by themselves, but they are the intermediate steps between modes LoadBalancer and ClusterIP.
How does the load balancer know about the nodes? It heavily depends on the load balancer. As an example, if you use MetalLB in BGP mode, you need to add your nodes as peers to your external BGP router (either manually or in an automated way). MetalLB takes care of advertising the IPs of the LoadBalancer type services to the router. This means, that router effectively becomes the load balancer of your cluster.
There are also a number of enterprise-grade commercial Kubernetes load balancers out there, such as F5 Big-IP.
Enable ClusterIP for service rather than Node Port. Any LB can be used along with the ingress. But it depends on the platform you are using . It's bare metal or open shift , IBM Cloud, Google cloud. Once the ingress controller ( Metalb, ngnix, Traffic) is able to communicate any LB like F5 GTM or LTM can be set up in front.
https://github.com/kubernetes-sigs/aws-alb-ingress-controller/blob/master/docs/guide/ingress/annotation.md#target-type
In above link it is mentioned that "instance mode" will route traffic to all ec2 instances within cluster on NodePort opened for your service.
so how does kube-proxy make sure that request is served only once in case multiple replicas of pods are running in different instances and how does it makes sure that requests are evenly served from all pods?
As per documentation:
Amazon Elastic Load Balancing Application Load Balancer (ALB) is a popular AWS service that load balances incoming traffic at the application layer (layer 7) across multiple targets, such as Amazon EC2 instances.
The AWS ALB Ingress controller is a controller that triggers the creation of an ALB and the necessary supporting AWS resources whenever a Kubernetes user declares an Ingress resource on the cluster. The Ingress resource uses the ALB to route HTTP[s] traffic to different endpoints within the cluster.
With instance mode, ingress traffic start from ALB and reach Node Port opened for service. Traffic is routed to the container POD within cluster.
Moreover target-type: "instance mode" is default setting in AWS ALB ingress controller and service must be type of "NodePort" or "LoadBalancer" to use this mode.
Managing ALBs is automatic, and you only need to define your ingress resources as you would typically do. ALB ingress controller POD which is running inside the Kubernetes cluster communicates with Kubernetes API and does all the work. However, this POD is only a control plane, it doesn't do any proxying and stuff like that.
Your Application Load Balancer periodically sends requests to its registered targets to test their status. These tests are called health checks. Alb-ingress-controller is performing "health checks" for targets groups. Different "health check's" on target groups can be controlled using annotations.
You can find more information about ALB ingress and NodePort here and here
Hope this help.