I'm new to GKE and K8S so please bare with me and my silliness. I currently have a GKE cluster that has two nodes in the default node pool and the cluster is exposed via a LoadBalancer type service.
These nodes are tasked with calling a Compute Engine instance via HTTP. I have a Firewall rule set in GCP to deny ingress traffic to the GCE instance except the one coming from the GKE cluster.
The issue is that the traffic isn't coming from the LoadBalancer's service IP but rather from the nodes themselves, so whitelisting the services' IP has no effect and I have to whitelist the IPs of the nodes instead of the cluster. This is not ideal, since each time a new node is created I have to change the Firewall rule. I understand that once you have a service set up in the cluster, all traffic will be directed towards the IP of the service, so why is this happening? What am I doing wrong? Please let me know if you need more details and thanks in advance.
YAML of the service:
https://i.stack.imgur.com/XBZmE.png
When you create a service on GKE, and you expose it to internet, a load balancer is created. This load balancer manage only the ingress traffic (traffic from internet to your GKE cluster).
When your pod initiate a communication, the traffic is not managed by the load balancer, but by the node that host the pod, if the node have a public IP (Instead of denied the traffic to GCE instance, simply remove the public IP, it's easier and safer!).
If you want to manage the IP for egress traffic originated by your pod, you have to set up a Cloud NAT on your GKE cluster.
Related
I'm trying to understand how traffic can be forwarded to an on-premise Kubernetes cluster.
It's clear to me that in a public Cloud provider, the underlying infrastructure of the Cloud can automatically manage and forward traffic to a Kubernetes distribution, such as EKS, GKE, AKS, by assining a LoadBalancer IP to a Kubernetes Service. Then, after a few seconds, this service will receive an external IP and will be reachable from the outside world.
On the other hand, in an on-premise Kubernetes cluster, by assigning a LoadBalancer IP to a service, it stays on pending forever, unless you assign a node IP, but what if you want to assign a different IP from a private IP range? In order to tackle this, in my homelab, I've deployed metallb inside my K3s cluster. The metallb is configured to use a private IP range of my network, let's say 10.0.0.0/24. Now, services of type LoadBalancer can consume an address of this range, e.g. my Ingress Controller can receive 10.0.2.3 as its external IP.
I can't understand what's metallb doing under the hood. How metallb "listens" to an address of the range and forwards traffic to my cluster. Can this be achieved without a metallb? I've tried setting an ExternalIP directly to a service of type LoadBalancer, but it never managed to claim that specific IP without it.
In addition, I'm aware that this can also be achieved with a "physical" load-balancer solution, such as NGINX and HAProxy, that sits in front of the cluster. To my understanding, technically this does the same thing as metallb. With such a solution configured, an address can be listened and be forwarded to the cluster. But my question here is, can this be achieved without those technologies? Can a Kubernetes Cluster listen to an external address and accept traffic without an intermediate solution? Maybe through Firewall rules and port-forwarding?
Your time is highly appreciated!
This involves some of the core networking concepts like NATing, you can have two networks one local and one external CIDR. For exposing the services you can NAT the local CIDR with external CIDR and configure required firewall rules for making your cluster serve the public.
Requirement/Problem:
I would like to route traffic from the VPC network to a cluster IP. In AKS I was able to do this by adding an entry in the VNET route table to a node running kube-proxy. I can't seem to be able to do this in EKS. I would like to do this for development environments so I can easily access service cluster IPs without having to forward ports or create load balancers. It's my understanding that kube-proxy uses iptables to forward network traffic.
Question:
Is there something fundamental that won't allow me to route traffic to the cluster network in EKS?
Context:
I'm testing with eks.9 and k8s 1.21
As per my understanding, you should definitely be able to do this by setting proper SecurityGroup settings (which allow traffic to be forwarded to your worker node clusterIP subnet).
And yes, kube-proxy uses iptables to forward traffic but it really depends on the overlay networking driver you have. If you're running flannel for instance, this is true, but perhaps not for calico or cilium, they may use bpf. So, just double check if your overlay network CNI plugin supports forwarding based on iptables.
Another thing you can do (and this will not require creating Load Balancers) is you can change your service type to NodePort or LoadBalancer which will allow you to set a personalized externalIP on your service. This you can provide to the cluster through a subnet configured in your VPC. All the incoming traffic to this subnet will then be forwarded to your services on the desired ports on which they are listening.
I hope this is helpful enough for you to get started.
I have installed my kubernetes cluster on Jelastic. Now, I tried to define a service of LoadBalancer type and would like it to be provided with an external IP. The external IP is currently marked as pending. What should I do to make it non-pending? Do I have to provide the worker nodes with an external IPv4?
In my current setup, my worker nodes have no IPv4 because I put an nginx load-balancer in front of the cluster:
The IPv4 is set on the nginx node. Is that a problem? If I want to access my loadbalancer service inside of my kubernetes cluster, what should I do?
For LoadBalancer service type to work, the cloud provider must implemenet the relevant APIs to get it to work.
With regard to Jelastic, as per their docs, they don't support it https://docs.jelastic.com/kubernetes-exposing-services/:
Jelastic PaaS does not support the LocaBalancer service type currently.
In Jelastic Public IP addresses have to be attached to worker nodes.
Every worker node has ingress controller instance running (based oт nginx/haproxy/traefik) with http/https listeners that can forward traffic to the required service.
You have just to bind your domain as CNAME to Environment FQDN and every your worker node can accept requests in RR-DNS mode.
Does this scenario works for you or you have a specific requirement to use external load balancer?
By default, when Public IPs are not attached to worker instances the traffic is going through the Shared Load Balancer.
P.S. If you install Certification Manager Addon to your K8s cluster - you can also issue free Let's Encrypt certificates.
The external IP is perfectly reachable from outside the cluster. It's perfectly reachable from all nodes within the cluster. However, when I try to telnet to the URL from a pod within the cluster that is not on the same node as a pod that is part of the service backend, the connection always times out.
The external IP is reachable by pods that run on the same node as a pod that is part of the service backend.
All pods can perfectly reach the cluster IP of the service.
When I set externalTrafficPolicy to Cluster, the pods are able to reach the external URL regardless of what node they're on.
I am using iptables proxying and kubernetes 1.16
I'm completely at a loss here as to why this is happening. Is someone able to shed some light on this?
From the official doc here,
service.spec.externalTrafficPolicy - denotes if this Service desires to route external traffic to node-local or cluster-wide endpoints. There are two available options: Cluster (default) and Local. Cluster obscures the client source IP and may cause a second hop to another node, but should have good overall load-spreading. Local preserves the client source IP and avoids a second hop for LoadBalancer and NodePort type services, but risks potentially imbalanced traffic spreading.
The service could be either node-local or cluster-wide endpoints. When you define the externalTrafficPolicy as Local, it means node-local. So, other nodes are not able to reach it.
So, you will need to set the externalTrafficPolicy as Cluster instead.
I'm having some troubles with a nginx pod inside a kubernetes cluster located on GCP which should be able to access a service located on app engine.
I have set firewall rules in the app engine to deny all and only allow some ips but the ip which hits my app engine service isn't the IP of the load balancer of my Nginx but instead the IP of one of the node of the cluster.
An image is better than 1000 words, then here's an image of our architecture :
The problem is: The ip which hits app engine's firewall is IP A whereas I thought i'd be IP B. IP A changes everytime I kill/create the cluster. If it were IP B, I could easily open this IP in App engine's firewall rules as I've put her static. Anyone has an idea how to have IP B instead of IP A ?
Thanks
The IP address assigned to your nginx "load balancer" is (likely) not an IP owned or managed by your Kubernetes cluster. Services of type LoadBalancer in GKE use Google Cloud Load Balancers. These are an external abstraction which terminates inbound connections in Google's front-end infrastructure and passes traffic to the individual k8s nodes in the cluster for onward delivery to your k8s-hosted service.
Pods in a Kubernetes cluster will, by default, route egress traffic out of the cluster using the configuration of their host node. In GKE, this route corresponds to the gateway of the VPC in which the cluster (and, by extension, Compute Engine instances) exists. The public IP of cluster nodes will change as they are added and removed from the pool.
A workaround uses a dedicated instance with a static external IP to process egress traffic leaving your VPC (i.e. egress from your cluster). Google has a tutorial for this purpose here: https://cloud.google.com/solutions/using-a-nat-gateway-with-kubernetes-engine
There are k8s-native solutions, but these will be unsuitable in a GKE context at present due to the inability to maintain any node with a non-ephemeral public IP.