I understand that, in Cloud scenarios, a LoadBalancer resource refers to and provisions an external layer 4 load balancer. There is some proprietary integration by which the cluster configures this load balancing. OK.
On-prem we have no such integration so we create our own load balancer outside of the cluster which distributes traffic to all nodes - effectively, in our case to the ingress.
I see no need for the cluster to know anything about this external load balancer.
What is a LoadBalancer* then in an on-prem scenario? Why do I have them? Why do I need one? What does it do? What happens when I create one? What role does the LoadBalancer resource play in ingress traffic. What effect does the LoadBalancer have outside the cluster? Is a LoadBalancer just a way to get a new IP address? How/why/ where does the IP point to and where does the IP come from?
All questions refer to the Service of type “LoadBalancer” inside cluster and not my load balancer outside the cluster of which the cluster has no knowledge.
As pointed out in the comments a kubernetes service of type LoadBalancer can not be used by default with on-prem setups. You can use metallb to setup a service of that type in an on prem environment.
Kubernetes does not offer an implementation of network load balancers (Services of type LoadBalancer) for bare-metal clusters. [...] If you’re not running on a supported IaaS platform (GCP, AWS, Azure…), LoadBalancers will remain in the “pending” state indefinitely when created. [...] MetalLB aims to redress this imbalance by offering a network load balancer implementation that integrates with standard network equipment, so that external services on bare-metal clusters also “just work” as much as possible.
You can for example use the BGP mode to advertise the service's IP to your router, read more on that in the docs.
The project is still in beta but is promoted as production ready and used by several bigger companies.
Edit
Regarding your question in the comments:
Can I just broadcast the MAC address of my node and manually add the IP I am broadcasting to the LoadBalancer service via kubectl edit?
Yes that would work too. That's basically what metallb does, announcing the IP and updating the service.
Why need a software then? Imaging having 500 hosts that come and go with thousends of services of type LoadBalancer that come and go. You need an automation here.
Why does Kubernetes need to know this IP?
It doesn't. If you don't use an external ip, the service is still usable via it's NodePort, see for example the istio docs (with a little more details added by me):
$ kubectl get svc istio-ingressgateway -n istio-system
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S)
istio-ingressgateway LoadBalancer 192.12.129.119 <Pending> [...],80:32123/TCP,443:30994/TCP,[...]
Here the external IP is not set and stays in <Pending>. You can still use the service by pointing your traffic to <Node-IP>:32123 for plain http and to <Node-IP>:30994 for https. As you can see above those ports are mapped to 80 and 443.
If the external ip is set you can direct traffic directly to port 80 and 443 on the external load balancer. Kube-Proxy will create an iptables chain with the destination of you external ip, that basically leads from the external IP over the service ip with a load balancer configuration to a pod ip.
To investigate that set up a service of type LoadBalancer, make sure it has an external ip, connect to the host and run the iptables-save command (e.g. iptables-save | less). Search for the external ip and follow the chain until you end up at the pod.
Related
I'm unable to wrap my head around the concepts of interconnectivity among DNS, Ingress controller, MetalLb and Kubeproxy.
I know what these resources/tools/services are for and get the concepts of them individually but unable to form a picture of them working in tandem.
For example, in a bare metal setup, a client accesses my site - https://mytestsite.com, still having doubts , how effectively it lands to the right pod and where the above mentioned services/resources/tools comes into picture & at what stage ?
Ex. How DNS talks to my MetalLB, if the client accesses my MetalLB hosting my application and how LB inturn speaks to IngressController and finally where does Kube-proxy comes into play here.
I went thru the K8s official documentation and few others as well but still kind of stumped here. Following article is really good but I'm unable to stitch the pieces together.
https://www.disasterproject.com/kubernetes-with-external-dns/
Kindly redirect me to the correct forum, if it is not the right place, thanks.
The ingress-controller creates a service of type LoadBalancer that serves as the entry point into the cluster. In a public cloud environment, a loadbalancer like ELB on AWS would create the counter part and set the externalIP of that service to it's ip. It is like a service of type NodePort but it also has an ExternalIP, which corresponds to the actual ip of the counterpart, a load balancer like ELB on aws.
In a bare metal environment, no external load balancer will be created, so the external ip would stay in <Pending> state forever. Here for example the service of the istio ingress controller:
$ kubectl get svc istio-ingressgateway -n istio-system
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S)
istio-ingressgateway LoadBalancer 192.12.129.119 <Pending> [...],80:32123/TCP,443:30994/TCP,[...]
In that state you would need to call http://<node-ip>:32123 to reach the http port 80 of the ingress controller service, which would be then forwarded to your pod (more on that in a bit).
When you're using metallb, it will update the service with an external ip so you can call http://<ip> instead. Metallb will also announce that ip, e.g. via BGP, so other know where to send traffic to, when someone would call the ip.
I havn't used external DNS and only scanned the article but I guess that you can use that to also have a dns record to be created so someone can call your service by it's domain, not only by it's ip. So you can call http://example.com instead.
This is basically why you run metallb and how it interacts with your ingress controller. The ingress controller creates an entry point into the cluster and metallb configures it and attracts traffic.
Until now the call to http://example.com can reach your cluster, but it needs to also reach the actual application, running in a pod inside the cluster. That's kube-proxy's job.
You read a lot about service of different types and all this kind of stuff, but in the end it all boils down to iptables rules. kube-proxy will create a bunch of those rules, that form a chain.
SSH into any kubernetes worker, run iptables-save | less command and search for the external ip configured on your ingress-controller's service by metallb. You'll find a chain with the destination of you external ip, that basically leads from the external IP over the service ip with a load balancer configuration to a pod ip.
In the end the whole chain would look something like this:
http://example.com
-> http://<some-ip> (domain translated to ip)
-> http://<node-ip>:<node-port> (ingress-controller service)
---
-> http://<cluster-internal-ip>:<some-port> (service of your application)
-> http://<other-cluster-internal-ip>:<some-port> (ip of one of n pods)
where the --- line shows the switch from cluster external to cluster internal traffic. The cluster-internal-ip will be from the configured service-cdir and the other-cluster-internal-ip will be from the configured pod-cidr.
Note that there are different ways to configure cluster internal traffic routing, how to run kube-proxy and some parts might even be a bit simplified, but this should give you a good enough understanding of the overall concept.
Also see this answer on the question 'What is a Kubernetes LoadBalancer On-Prem', that might provide additional input.
Requirement/Problem:
I would like to route traffic from the VPC network to a cluster IP. In AKS I was able to do this by adding an entry in the VNET route table to a node running kube-proxy. I can't seem to be able to do this in EKS. I would like to do this for development environments so I can easily access service cluster IPs without having to forward ports or create load balancers. It's my understanding that kube-proxy uses iptables to forward network traffic.
Question:
Is there something fundamental that won't allow me to route traffic to the cluster network in EKS?
Context:
I'm testing with eks.9 and k8s 1.21
As per my understanding, you should definitely be able to do this by setting proper SecurityGroup settings (which allow traffic to be forwarded to your worker node clusterIP subnet).
And yes, kube-proxy uses iptables to forward traffic but it really depends on the overlay networking driver you have. If you're running flannel for instance, this is true, but perhaps not for calico or cilium, they may use bpf. So, just double check if your overlay network CNI plugin supports forwarding based on iptables.
Another thing you can do (and this will not require creating Load Balancers) is you can change your service type to NodePort or LoadBalancer which will allow you to set a personalized externalIP on your service. This you can provide to the cluster through a subnet configured in your VPC. All the incoming traffic to this subnet will then be forwarded to your services on the desired ports on which they are listening.
I hope this is helpful enough for you to get started.
Loadbalancer is used to handle a request and forward that request to a particular pod, but my question is how does the GKE LoadBalancer work, how is it different from the LoadBalancer we have in minikube, and how should we use LoadBalancer properly.
In GKE when you add a Service of type LoadBalancer, it makes the necessary Google Cloud API calls to create either an external network LB, or an internal TCP/UDP LB.
The cloud.google.com/load-balancer-type: "Internal" annotation denotes an internal LB; otherwise, GKE creates an external network load balancer. This type of LB operates at layer 3/4, and is not an application load balancer, and thus not aware of HTTP requests or headers.
The LB service was designed for deployments in external cloud providers. In Minikube you need to use the tunnel feature to expose it. tunnel runs as a process, creating a network route on the host to the service CIDR of the cluster using the cluster’s IP address as a gateway. The tunnel command exposes the external IP directly to programs running on the host OS.
I have a GKE cluster set up with Cloud NAT, so traffic from any node/container going outward would have the same external IP. (I needed this for whitelisting purposes while working with 3rd-party services).
Now, if I want to deploy a proxy server onto this cluster that does basic traffic forwarding, how do I expose the proxy server "endpoint"? Or more generically, how do I expose a service if I deploy it to this GKE cluster?
Proxy server running behind NAT ?
Bad idea, unless it is only for your kubernetes cluster workload, but you didn't specify anywhere that it should be reachable only by other Pods running in the same cluster.
As you can read here:
Cloud NAT does not implement unsolicited inbound connections from the
internet. DNAT is only performed for packets that arrive as responses
to outbound packets.
So it is not meant to be reachable from outside.
If you want to expose any application within your cluster, making it available for other Pods, use simple ClusterIP Service which is the default type and it will be created as such even if you don't specify its type at all.
Normally, to expose a service endpoint running on a Kubernetes cluster, you have to use one of the Service Types, as Pods have internal IP addresses and are not addressable externally.
The possible service types:
ClusterIP: this also uses an internal IP address, and is therefore not addressable externally.
NodePort: this type opens a port on every node in your Kubernetes cluster, and configures iptables to forward traffic arriving to this port into the Pods providing the actual service.
LoadBalancer: this type opens a port on every node as with NodePort, and also allocates a Google Cloud Load Balancer service, and configures that service to access the port opened on the Kubernetes nodes (actually load balancing the incoming traffic between your operation Kubernetes nodes).
ExternalName: this type configures the Kubernetes internal DNS server to point to the specified IP address (to provide a dynamic DNS entry inside the cluster to connect to external services).
Out of those, NodePort and LoadBalancer are usable for your purposes. With a simple NodePort type Service, you would need publicly accessible node IP addresses, and the port allocated could be used to access your proxy service through any node of your cluster. As any one of your nodes may disappear at any time, this kind of service access is only good if your proxy clients know how to switch to another node IP address. Or you could use the LoadBalancer type Service, in that case you can use the IP address of the configured Google Cloud Load Balancer for your clients to connect to, and expect the load balancer to forward the traffic to any one of the running nodes of your cluster, which would then forward the traffic to one of the Pods providing this service.
For your proxy server to access the Internet as a client, you also need some kind of public IP address. Either you give the Kubernetes nodes public IP addresses (in that case, if you have more than a single node, you'd see multiple source IP addresses, as each node has its own IP address), or if you use private addresses for your Kubernetes nodes, you need a Source NAT functionality, like the one you already use: Cloud NAT
I've setup a kubernetes cluster using kubespray, and now I am trying to follow this guide
root#node1 ~ # kubectl get -n heptio-contour service contour -o wide
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE SELECTOR
contour LoadBalancer 10.233.55.94 <pending> 80:32414/TCP,443:30149/TCP 42m app=contour
It seems EXTERNAL-IP is pending because I am on a bare metal machine (not AWS/GKE etc.)
What do I need to do in order to get an external ip showing there?
Kubernetes offers three ways to expose a service:
1) L4 LoadBalancer: Available only on cloud providers such as GCE and AWS
2) Expose Service via NodePort: The NodePort directive allocates a port on every worker node, which proxy the traffic to the respective Pod.
3) L7 Ingress: The Ingress is a dedicated load balancer (eg. nginx, HAProxy, traefik, vulcand) that redirects incoming HTTP/HTTPS traffic to the respective endpoints
Kubernetes does not offer implementation of network load-balancers (Services of type LoadBalancer) for bare metal clusters.
If you’re not running Kubernetes cluster on a supported IaaS platform (GCP, AWS, Azure…), LoadBalancers will remain in the “pending” state regardless of the time they were created.
The reason is the lack of support of IP routing between the external world and Kubernetes; there is no default implementation of transferring dns zones
used by Ingress to allocate communication to internal of the cluster.
There are external projects to provide bare-metal even in federation clusters mode to be part of standalone or hybrid solution.
It depends on the scale and the maturity of projects you have, so it should begin with choosing a proper load balancer or VIP provider:
https://github.com/google/metallb
https://docs.traefik.io/
https://github.com/kubernetes/contrib/tree/master/keepalived-vip
http://vulcand.github.io/
and deprecated:
http://www.linuxvirtualserver.org/software/ipvs.html
Please notice that in federation clusters (more than one bare metal Kubernetes
cluster) is needed to export IP address of each physical machine to central API
provider and probably it is not covered by the links I provided to you.