I'm managing a small Kubernetes cluster on Azure with Postgres. This cluster is accessible through an Nginx controller with a static IP.
The ingress routes to a ClusterIP to a pod which uses a Postgres instance. This Postgres instance has all IPs blocked, with a few exceptions for my own IP and the static IP of the ingress.
This worked well until I pushed an update this morning, where to my amazement I see in the logs an error that the pods IP address differs from the static ingress IP, and it has a permission error because of it.
My question: how is it possible that my pod, with ClusterIP, has a different outer IP address than the ingress static IP I assigned it?
Note that the pod is easily reached, through the Ingress.
Ingresses and Services handle only incoming pod traffic. Pod outgoing traffic IP depends on Kubernetes networking implementation you use. By default all outgoing connections from pods are source NAT-ed on node level which means pod will have an IP of node which it runs on. So you might want to allow worker node IP addresses in your Postgres.
Related
I have 5 VPS with a public network interface for each, for which I have configured a VPN.
3 nodes are Kubernetes masters where I have set the Kubelet --node-ip flag as their private IP address.
One of the 3 nodes have a HAProxy load balancer for the Kubernetes masters, listening on the private IP, so that all the nodes used the private IP address of the load balancer in order to join the cluster.
2 nodes are Kubernetes workers where I didn't set the Kubelet --node-ip flag so that their node IP is the public address.
The cluster is healthy and I have deploy my application and its dependencies.
Now I'd like to access the app from the Internet, so I've deployed a edge router and created a Kubernetes Service with the type LoadBalancer.
The service is well created but never takes the worker nodes' public IP addresses as EXTERNAL-IP.
Assigning the IP addresses manually works, but obviously want that to be automatic.
I have read about the MetalLb project, but it doesn't seem to fit in my case as it is supposed to have a range of IP addresses to distribute, while here I have one public IP address per node, and not in the same range.
So who can I configure Kubernetes so that my Service of type LoadBalancer gets automatically the public IP addresses as EXTERNAL-IP?
I finally can answer myself in two times.
Without an external Load Balancer
Firstly, in order to solve the problem from my question, the only way I found which worked quite well was to set the externalIPs of my LoadBalancer service with the IP addresses of the Kubernetes worker nodes.
Those nodes were running Traefik and therefor had it listening on ports 80 and 443.
After that, I've created as many A DNS entries as I have Kubernetes worker nodes, pointing each to the Kubernetes respective worker node public IP address. This setup makes the DNS server returning the list of IP addresses, in a random order, and then the web browser will take care of trying the first IP address, then the second one if the first is down and so on.
The downside of this, is when you want to drain a node for maintenance, or when it crashes, the web browser will wast time trying to reach it until it tries the next IP address.
So here come the second option: External Load balancer.
With an external Load Balancer
I took another VPS where I've installed HAproxy and configured a SSL passthrough of the Kubernetes API port so that it load balancer the trafic to the master nodes, without terminating it.
With this solution, I removed the externalIPs field from my Service and I've installed MetalLB with a single IP address configured with this manifest:
apiVersion: v1
kind: ConfigMap
metadata:
namespace: metallb-system
name: config
data:
config: |
address-pools:
- name: staging-public-ips
protocol: layer2
addresses:
- 1.2.3.4/32
When the LoadBalancer Service is created, MetalLB assigns this IP address and calls the Kubernetes APIs accordingly.
This has solved my issue to integrate my Kubernetes cluster with Gitlab.
WARNING: MetalLB will assign only once the IP address so that if you have a second LoadBalancer Service, it will remain in Pending state forever, until you give a new IP address to MetalLB.
The only requirement in Kubernetes networking docs is to open firewall between pods. How does pod to service connectivity works, as service cluster ip range and pod cidrs are different?
Services has an virtual IP assigned. When a Pod communicates with a Service, the Kubeproxy on the local node replaces the virtual IP with an IP to one of the pods that represents the service.
An example:
E.g. Pod-A on Node-A want to send a request to Service-B. Service-B is for example implemented by the pods with label app-b, and in this example Pod-D and Pod-E on Node-C and Node-E. When Pod-A sends the request, the target IP is changed from an virtual IP, to the IP for Pod-D or Pod-E by kubeproxy and the request is routed to one of the pods that represents Service-B.
Layout:
Service-B with selector: app=b
Pod-D with label: app=b
Pod-E with label: app=b
Pod-A should address the Service virtual IP, since pods comes and goes when new versions are deployed. But the virtual IP is translated to a pod with the implementation of the Service.
Can someone help me understand about the IP address I see for cluster IP when I list services.
what is cluster IP (not the service type, but the real IP)?
how it is used?
where does it come from?
can I define the range for cluster IP (like we do for pod network)?
Good question to start learning something new (also for me):
Your concerns are related to kube-proxy by default in K8s cluster it's working in iptables mode.
Every node in a Kubernetes cluster runs a kube-proxy. Kube-proxy is responsible for implementing a form of virtual IP for Services.
In this mode, kube-proxy watches the Kubernetes control plane for the addition and removal of Service and Endpoint objects. For each Service, it installs iptables rules, which capture traffic to the Service’s clusterIP and port, and redirect that traffic to one of the Service’s backend sets. For each Endpoint object, it installs iptables rules which select a backend Pod.
Node components kube-proxy:
kube-proxy is a network proxy that runs on each node in your cluster, implementing part of the Kubernetes Service concept.
kube-proxy maintains network rules on nodes. These network rules allow network communication to your Pods from network sessions inside or outside of your cluster.
kube-proxy uses the operating system packet filtering layer if there is one and it’s available. Otherwise, kube-proxy forwards the traffic itself.
As described here:
Due to these iptables rules, whenever a packet is destined for a service IP, it’s DNATed (DNAT=Destination Network Address Translation), meaning the destination IP is changed from service IP to one of the endpoints pod IP chosen at random by iptables. This makes sure the load is evenly distributed among the backend pods.
When this DNAT happens, this info is stored in conntrack — the Linux connection tracking table (stores 5-tuple translations iptables has done: protocol, srcIP, srcPort, dstIP, dstPort). This is so that when a reply comes back, it can un-DNAT, meaning change the source IP from the Pod IP to the Service IP. This way, the client is unaware of how the packet flow is handled behind the scenes.
There are also different modes, you can find more information here
During cluster initialization you can use --service-cidr string parameter Default: "10.96.0.0/12"
ClusterIP: The IP address assigned to a Service
Kubernetes assigns a stable, reliable IP address to each newly-created Service (the ClusterIP) from the cluster's pool of available Service IP addresses. Kubernetes also assigns a hostname to the ClusterIP, by adding a DNS entry. The ClusterIP and hostname are unique within the cluster and do not change throughout the lifecycle of the Service. Kubernetes only releases the ClusterIP and hostname if the Service is deleted from the cluster's configuration. You can reach a healthy Pod running your application using either the ClusterIP or the hostname of the Service.
Pod IP: The IP address assigned to a given Pod.
Kubernetes assigns an IP address (the Pod IP) to the virtual network interface in the Pod's network namespace from a range of addresses reserved for Pods on the node. This address range is a subset of the IP address range assigned to the cluster for Pods, which you can configure when you create a cluster.
Resources:
Iptables Mode
Network overview
Understanding Kubernetes Kube-Proxy
Hope this helped
The cluster IP is the address where your service can be reached from inside the cluster. You won't be able to ping from the external network the cluster IP unless you do some kind of SSH tunneling. This IP is auto assigned by k8s and it might be possible to define a range (I'm not sure and I don't see why you need to do so).
I've deployed a hello-world application on my Kubernetes cluster. When I access the app via <cluster ip>:<port> in my browser I get the following webpage: hello-kuleuven app webpage.
I understand that from outside the cluster you have to access the app via the cluster IP and the port specified in the deployment file (which in my case is 30001). From inside the cluster you have to contact the master node with its local IP and another port number, in my case 10.111.152.164:8080.
My question is about the last line of the webpage:
Kubernetes listening in 443 available at tcp://10.96.0.1:443
Since, the service is already accessible from inside and outside the cluster by other ports and IP's, I'm not sure what this does.
The IP 10.96.0.1 is a cluster IP of kube-dns service. You can see it using
kubectl get svc -n kube-apiserver
Kubernetes DNS schedules a DNS Pod and Service on the cluster, and configures the kubelets to tell individual containers to use the DNS Service’s IP to resolve DNS names.
So every pod you deploy uses kube-dns service (ClusterIP 10.96.0.1) to resolve the dns names.
Read more about kube dns at kubernetes official document here
I have an app deployment called 'backend-app' running in pods that are on several different nodes. I also have a service that exposes the 'backend-app' to be accessed by other cluster internal pods as my 'frontend-app' pods.
If I use DNS to connect to the 'backend-app' from my different app deployment called 'frontend-app' will the requests be load balanced to each 'backend-app' pod on each node?
It sounds like a NodePort service will only connect to one node and not load balance my requests to others.
For each Service with type: NodePort a port is opened on all nodes (the same port on each). The port is open whether a pod of that service is running on a node or not. The load balancing is done among all pods of all nodes with no preference to a pod that happens to run on the same node to which you connected on the node port (if there is one there at all).
Services automatically load balance to the pods that are assigned to them. See https://kubernetes.io/docs/concepts/services-networking/connect-applications-service/#creating-a-service.
The cluster IP address that is created with a service is the IP address that will automatically select an available pod on any node that is running the pod. You can find the service's cluster IP address by using a DNS lookup.
My confusion came because I didn't realise the cluster IP address was associated with a service, not with a specific Pod.
I'm currently not sure about how NodePort's work with this though.