How does k8 traffic flow internally? - kubernetes

I have ingress and service with LB. When traffic coming from outside it hits ingress first and then does it goes to pods directly using ingress LB or it goes to service and get the pod ip via selector and then goes to pods? If it's first way, what is the use of services? And which kind, services or ingress uses readinessProbe in the deployment?
All the setup is in GCP
I am new to K8 networks.

A service type LoadBalancer is a external source provided by your cloud and are NOT in Kubernetes cluster. They can work forwarding the request to your pods using node selector, but you can't for example make path rules or redirect, rewrites because this is provided by an Ingress.
Service is an abstraction which defines a logical set of Pods and a policy by which to access them (sometimes this pattern is called a micro-service). The set of Pods targeted by a Service is usually determined by a selector (see below for why you might want a Service without a selector).
Internet
|
[ LoadBalancer ]
--|-----|--
[ Services ]
--| |--
[ Pod1 ] [ Pod2 ]
When you use Ingress, is a component controller by a ingress controller that is basically a pod configured to handle the rules you defined.
To use ingress you need to configure a service for your path, and then this service will reach the pods with configures selectors. You can configure some rules based on path, hostname and them redirect for the service you want. Like this:
Internet
|
[ Ingress ]
--|-----|--
[ Services ]
--| |--
[ Pod1 ] [ Pod2 ]
Ingress exposes HTTP and HTTPS routes from outside the cluster to services within the cluster. Traffic routing is controlled by rules defined on the Ingress resource.
This article has a good explanation between all ways to expose your service.
The readnessProbe is configured in your pod/deployment specs, and kubelet is responsible to evaluate your container healthy.
The kubelet uses readiness probes to know when a Container is ready to start accepting traffic. A Pod is considered ready when all of its Containers are ready. One use of this signal is to control which Pods are used as backends for Services. When a Pod is not ready, it is removed from Service load balancers.
kube-proxy is the responsible to foward the request for the pods.
For example, if you have 2 pods in different nodes, kube-proxy will handle the firewall rules (iptables) and distribute the traffic between your nodes. Each node in your cluster has a kube-proxy running.
kube-proxy can be configured in 3 ways: userspace mode, iptables mode and ipvs mode.
If kube-proxy is running in iptables mode and the first Pod that’s selected does not respond, the connection fails. This is different from userspace mode: in that scenario, kube-proxy would detect that the connection to the first Pod had failed and would automatically retry with a different backend Pod.
References:
https://kubernetes.io/docs/concepts/services-networking/service/
https://kubernetes.io/docs/concepts/services-networking/ingress/

Depends whether your LoadBalancer service exposes the Ingress controller or your application Pods (the first is the correct approach).
The usual way to use Services and Ingresses is like this:
LoadBalancer Service -> Ingress -> ClusterIP Service -> Pods
In that case, the traffic from the Internet first hits the load balancer of your cloud provider (created by the LoadBalancer Service), which forwards it to the Ingress controller (which is one or multiple Pods running NGINX in your cluster), which in turn forward it to your application Pods (by getting the Pods' IP addresses from the ClusterIP Service).
I'm not sure if you currently have this constellation:
Ingress -> LoadBalancer Service -> Pods
In that case, you don't need a LoadBalancer Service there. You need only a ClusterIP Service behind an Ingress, and then you typically expose the Ingress with a LoadBalancer Service.

Related

How does traffic get routed to an ingress controller?

The following questions are about an on-prem K3S setup.
1] How does HTTP/S traffic reach an ingress controller in say K3S?
When I hit any of my nodes on HTTPS port 443 I get the traefik ingress controller. This must be "magic" though because:
There is no process on the host listening on 443 (according to lsof)
The actual nodePort on the traefik service (of type LoadBalancer) is 30492
2] Where is the traefik config located inside the ingress controller pod?
When I shell into my traefik pods I cannot find the config anywhere - /etc/traefik does not even exist. Is everything done via API (from Ingress resource definitions) and not persisted?
3] Is ingress possible without any service of type LoadBalancer? I.e. can I use a nodePort service instead by using an external load balancer (like F5) to balance traffic between nodes and these nodeports?
4] Finally, how do the traefik controller pods "know" when a node is down and stop sending/balancing traffic to pods which no longer exist?
Port-forwarding is responsible for traffic getting mapped to traefik ingress controller by hitting on port 443 and NodePort is generally in between this range 30000-32767 only.
Refer this documentation for more information on port forwarding.
Yes, An Ingress does not expose arbitrary ports or protocols. Exposing services other than HTTP and HTTPS to the internet typically uses a service of type Service.Type=NodePort or Service.Type=LoadBalancer.
Refer this documentation for more information on ingress.
Kubernetes has a health check mechanism to remove unhealthy pods from Kubernetes services (cf readiness probe). As unhealthy pods have no Kubernetes endpoints, Traefik will not forward traffic to them. Therefore, Traefik health check is not available for kubernetesCRD and kubernetesIngress providers.
Refer this documentation for more information on Health check.

Why do we need a load balancer to expose kubernetes services using ingress?

For a sample microservice based architecture deployed on Google kubernetes engine, I need help to validate my understanding :
We know services are supposed to load balance traffic for pod replicaset.
When we create an nginx ingress controller and ingress definitions to route to each service, a loadbalancer is also setup automatically.
had read somewhere that creating nginx ingress controller means an nginx controller (deployment) and a loadbalancer type service getting created behind the scene. I am not sure if this is true.
It seems loadbalancing is being done by services. URL based routing is
being done by ingress controller.
Why do we need a loadbalancer? It is not meant to load balance across multiple instances. It will just
forward all the traffic to nginx reverse proxy created and it will
route requests based on URL.
Please correct if I am wrong in my understanding.
A Service type LoadBalancer and the Ingress is the way to reach your application externally, although they work in a different way.
Service:
In Kubernetes, a Service is an abstraction which defines a logical set of Pods and a policy by which to access them (sometimes this pattern is called a micro-service). The set of Pods targeted by a Service is usually determined by a selector (see below for why you might want a Service without a selector).
There are some types of Services, and of them is the LoadBalancer type that permit you to expose your application externally assigning a externa IP for your service. For each LoadBalancer service a new external IP will be assign to it.
The load balancing will be handled by kube-proxy.
Ingress:
An API object that manages external access to the services in a cluster, typically HTTP.
Ingress may provide load balancing, SSL termination and name-based virtual hosting.
When you setup an ingress (i.e.: nginx-ingress), a Service type LoadBalancer is created for the ingress-controller pods and a Load Balancer in you cloud provider is automatically created and a public IP will be assigned for the nginx-ingress service.
This load balancer/public ip will be used for incoming connection for all your services, and nginx-ingress will be the responsible to handle the incoming connections.
For example:
Supose you have 10 services of LoadBalancer type: This will result in 10 new publics ips created and you need to use the correspondent ip for the service you want to reach.
But if you use a ingress, only 1 IP will be created and the ingress will be the responsible to handle the incoming connection for the correct service based on PATH/URL you defined in the ingress configuration. With ingress you can:
Use regex in path to define the service to redirect;
Use SSL/TLS
Inject custom headers;
Redirect requests for a default service if one of the service failed (default-backend);
Create whitelists based on IPs
Etc...
A important note about Ingress Load balancing in ingress:
GCE/AWS load balancers do not provide weights for their target pools. This was not an issue with the old LB kube-proxy rules which would correctly balance across all endpoints.
With the new functionality, the external traffic is not equally load balanced across pods, but rather equally balanced at the node level (because GCE/AWS and other external LB implementations do not have the ability for specifying the weight per node, they balance equally across all target nodes, disregarding the number of pods on each node).
An ingress controller(nginx for example) pods needs to be exposed outside the kubernetes cluster as an entry point of all north-south traffic coming into the kubernetes cluster. One way to do that is via a LoadBalancer. You could use NodePort as well but it's not recommended for production or you could just deploy the ingress controller directly on the host network on a host with a public ip. Having a load balancer also gives ability to load balance the traffic across multiple replicas of ingress controller pods.
When you use ingress controller the traffic comes from the loadBalancer to the ingress controller and then gets to backend POD IPs based on the rules defined in ingress resource. This bypasses the kubernetes service and load balancing(by kube-proxy at layer 4) offered by kubernetes service.Internally the ingress controller discovers all the POD IPs from the kubernetes service's endpoints and directly route traffic to the pods.
It seems loadbalancing is being done by services. URL based routing is being done by ingress controller.
Services do balance the traffic between pods. But they aren't accessible outside the kubernetes in Google Kubernetes Engine by default (ClusterIP type). You can create services with LoadBalancer type, but each service will get its own IP address (Network Load Balancer) so it can get expensive. Also if you have one application that has different services it's much better to use Ingress objects that provides single entry point. When you create an Ingress object, the Ingress controller (e.g. nginx one) creates a Google Cloud HTTP(S) load balancer. An Ingress object, in turn, can be associated with one or more Service objects.
Then you can get the assigned load balancer IP from ingress object:
kubectl get ingress ingress-name --output yaml
As a result your application in pods become accessible outside the kubernetes cluster:
LoadBalancerIP/url1 -> service1 -> pods
LoadBalancerIP/url2 -> service2 -> pods

Kubernetes LoadBalancer service with hostNetwork binding

I have a query regarding the usage of a LoadBalancer service with hostNetwork
If we set hostNetwork: true, then the pods bind on the host network - to which the external services connect to. If we need only one instance of the pod running - then I believe we do not need a LoadBalancer service for the external services to connect to the pod. I do not see any use-case for a a LoadBalancer service here, or are there any I am missing ?
hostNetwork=true is not the recommended approach for exposing pods outside of the cluster. It has a few limitations:
Only 1 instance of a pod can run on a specific node on the same port
You have to use the nodeIP to access the pod, however, the node IP can change.
If the pod fails, the k8s scheduler may spawn it on a different node.
The recommended way for exposing pods outside of the cluster is via Kubernetes Service Controllers.
All service controllers act as load balancers (they will balance the traffic across all "ready" pods) no matter the Service.spec.type property.
Service.spec.type property can be one of the below:
ClusterIP, NodePort, LoadBalancer, ExternalName
The LoadBalancer type means that k8s will use a cloud provider LoadBalancer to expose the service outside of the cluster (for example AWS Elastic Load balancer if the k8s cluster is running on AWS).
LoadBalancer: Exposes the Service externally using a cloud provider’s
load balancer. NodePort and ClusterIP Services, to which the external
load balancer routes, are automatically created.
More on k8s service types

How Alb Ingress Controller target-type:instance forward requests

https://github.com/kubernetes-sigs/aws-alb-ingress-controller/blob/master/docs/guide/ingress/annotation.md#target-type
In above link it is mentioned that "instance mode" will route traffic to all ec2 instances within cluster on NodePort opened for your service.
so how does kube-proxy make sure that request is served only once in case multiple replicas of pods are running in different instances and how does it makes sure that requests are evenly served from all pods?
As per documentation:
Amazon Elastic Load Balancing Application Load Balancer (ALB) is a popular AWS service that load balances incoming traffic at the application layer (layer 7) across multiple targets, such as Amazon EC2 instances.
The AWS ALB Ingress controller is a controller that triggers the creation of an ALB and the necessary supporting AWS resources whenever a Kubernetes user declares an Ingress resource on the cluster. The Ingress resource uses the ALB to route HTTP[s] traffic to different endpoints within the cluster.
With instance mode, ingress traffic start from ALB and reach Node Port opened for service. Traffic is routed to the container POD within cluster.
Moreover target-type: "instance mode" is default setting in AWS ALB ingress controller and service must be type of "NodePort" or "LoadBalancer" to use this mode.
Managing ALBs is automatic, and you only need to define your ingress resources as you would typically do. ALB ingress controller POD which is running inside the Kubernetes cluster communicates with Kubernetes API and does all the work. However, this POD is only a control plane, it doesn't do any proxying and stuff like that.
Your Application Load Balancer periodically sends requests to its registered targets to test their status. These tests are called health checks. Alb-ingress-controller is performing "health checks" for targets groups. Different "health check's" on target groups can be controlled using annotations.
You can find more information about ALB ingress and NodePort here and here
Hope this help.

Accessing micro service end point from deployed micro service using Kubernetes orchestration

I am trying to deploy my sample micro service Docker image in Kubernetes cluster having 2 node. I explored everything about Pods, Services, Deployment, StatefulSets and Daemon-sets etc.
I am trying to create a sample deployment and Service for that. Here I explored about how deployment provides the scalability and load balancing functionality. And exploring about service discovery by providing Services ClusterIp.
I have two questions:
My scenario is that I am trying to deploy microservice on my on-premise Ubuntu machine. The machine has the IP address of 192.168.1.15. When I am referring Kubernetes, service will also have one clusterIP.
If my microservice end point is /api/v1/loadCustomer, how I can call this end point? Do I need to use clusterIP also ? Can I call simply 192.168.1.15:8080/api/v1/loadCustomers ?
What is the role of clusterIP when I am calling my end point ? Can I directly use port?
I am referring to the following link for exploration:
https://kubernetes.io/docs/concepts/services-networking/connect-applications-service/
tldr:
you can not access the application using the clusterIP from the outside of the cluster. you can access the application using either loadbalancer's IP (type=LoadBalaner) or Node's IP (type=NodePort).
benefit of clusterIP:
As you know that pods can be created and terminated during its life-cycle consequently IP (endpoint IP)address created and terminated.Therefore, clusterIP is static which does not depends of the life-cycle of the pods.
Long Answer
In a Kubernetes cluster
an application or pod has following abstraction.
Endpoint IP and Port:It is provided by the CNI Plugins such as flannel, calico.
Each pod has an IP and tragetPort which is UNIQUE.
you can list and watch the endpoints by the following commands.
kubectl get endpoints --all-namespaces
clusterIP and port : It is provided by the kube-proxy component.
The replicated pods share a clusterIP and Port.
Load-balancing of request to the replicated pods.
internally expose so that other pod can discover it
you can list and watch clusterIP and port with the following command
kubectl get services --all-namespaces
externalIP and port: It can be layer 3-4 load balancer's IP and port or node's IP and Nodeport.
if you want to use loadbalancer's IP and port, you can use type=LoadBalaner in service file.
If you want to use node's IP, you need to use type=NodePort in service file.