In our Kubernetes deployment, we have deployed a WebApp on a deployment controller and created a Load-balancer for external access.
So all the inbound request is getting load-balanced by load-balancer and works fine.
But we are facing issue with our outbound request.In our case external application can only accept traffic from whitelisted IP addresses so we wanted to give load-balancer ip which will then get whitelisted as pods are ephemeral in nature and their IP will not be static.
But as request are originating from pod, it keeps the source ip of pod and then external application drops the request.
Is there a way in which pod can send outbound request using source as service ip, or can source ip be masked by service Ip?
You can potentially use a egress gateway for this. Istio provides Envoy as a egress gateway proxy. From your service inside the cluster all outbound traffic will be routed through this egress proxy. You can configure TLS origination at the proxy before the traffic is send to the external service. You need to then whitelist the IP of the egress gateway in your external service.
Other option will be to have a reverse proxy in front of that external service and terminate traffic from service inside kubernetes and start a new TCP session from the reverse proxy to the external service. In this case the reverse proxy accepts connection from any origin IP but the external service only receives request originated from the proxy. You can configure the proxy to pass the actual client IP in a http header typically X-Forwarded-Host
https://istio.io/docs/tasks/traffic-management/egress/
I am assuming you are using Kubernetes in IPv4 mode. When you are accessing an external IP address from the kubernetes pod, the requests are source NAT'd. This would mean that the packet would have the IP address of the host's (VM?) ethernet interface through which the traffic flows out. Please whitelist this IP and see if that helps
This would be a good reference: https://www.youtube.com/watch?v=0Omvgd7Hg1I
Please note that service IP is useful when other services want to discover and talk to other services and IP table (in kube-proxy ip-table mode) translates it to POD IP. Its not in play for the traffic originating from the given service
Related
I have a Kubernetes cluster installed on a bare metal server, I have installed Metallb for external load balancer and Traefik for reverse proxy engine, this cluster is behind an OpenVPN with subnet 10.1.0.0/24, the ip for the server is 10.1.0.1
For the Metallb I assign ip 10.1.1.0/24 for the pool, so the Traefik LoadBalancer ip is 10.1.1.1
I also have my own Domain Name Server that will be pushed to the server when connected to the VPN
If I create a domain for one of my app inside the Kubernetes Cluster, to what IP should I point my Domain so that I can access my app through the domain from other server that are also connected to the VPN?
I think I misconfigured something but I got stuck
You need to point the domain to the Traefik Loadbalancer IP, which is ‘10.1.1.1’ in your case. The IP address which you have shared is an external IP provided by Metallb.
When the client connects to the application using the domain; the DNS resolution process will resolve the domain to the Traefik LB IP. The traffic will be forwarded to the Traefik LB IP, which will route the traffic to the appropriate service and the pod in your Kubernetes cluster based on the rules defined in your Trafik. Please check this Here is the blog posted by Peter Gillich for your reference.
I have a GKE cluster set up with Cloud NAT, so traffic from any node/container going outward would have the same external IP. (I needed this for whitelisting purposes while working with 3rd-party services).
Now, if I want to deploy a proxy server onto this cluster that does basic traffic forwarding, how do I expose the proxy server "endpoint"? Or more generically, how do I expose a service if I deploy it to this GKE cluster?
Proxy server running behind NAT ?
Bad idea, unless it is only for your kubernetes cluster workload, but you didn't specify anywhere that it should be reachable only by other Pods running in the same cluster.
As you can read here:
Cloud NAT does not implement unsolicited inbound connections from the
internet. DNAT is only performed for packets that arrive as responses
to outbound packets.
So it is not meant to be reachable from outside.
If you want to expose any application within your cluster, making it available for other Pods, use simple ClusterIP Service which is the default type and it will be created as such even if you don't specify its type at all.
Normally, to expose a service endpoint running on a Kubernetes cluster, you have to use one of the Service Types, as Pods have internal IP addresses and are not addressable externally.
The possible service types:
ClusterIP: this also uses an internal IP address, and is therefore not addressable externally.
NodePort: this type opens a port on every node in your Kubernetes cluster, and configures iptables to forward traffic arriving to this port into the Pods providing the actual service.
LoadBalancer: this type opens a port on every node as with NodePort, and also allocates a Google Cloud Load Balancer service, and configures that service to access the port opened on the Kubernetes nodes (actually load balancing the incoming traffic between your operation Kubernetes nodes).
ExternalName: this type configures the Kubernetes internal DNS server to point to the specified IP address (to provide a dynamic DNS entry inside the cluster to connect to external services).
Out of those, NodePort and LoadBalancer are usable for your purposes. With a simple NodePort type Service, you would need publicly accessible node IP addresses, and the port allocated could be used to access your proxy service through any node of your cluster. As any one of your nodes may disappear at any time, this kind of service access is only good if your proxy clients know how to switch to another node IP address. Or you could use the LoadBalancer type Service, in that case you can use the IP address of the configured Google Cloud Load Balancer for your clients to connect to, and expect the load balancer to forward the traffic to any one of the running nodes of your cluster, which would then forward the traffic to one of the Pods providing this service.
For your proxy server to access the Internet as a client, you also need some kind of public IP address. Either you give the Kubernetes nodes public IP addresses (in that case, if you have more than a single node, you'd see multiple source IP addresses, as each node has its own IP address), or if you use private addresses for your Kubernetes nodes, you need a Source NAT functionality, like the one you already use: Cloud NAT
Inbound traffic is pass through the service before going to pods. How about the outbound traffic such as a pod accessing
pods outside the service in the same node
pods outside the service in different node
outside the cluster
does it pass through the service first? If yes, the only reason i can see now if for monitoring and logging or there is anything else?
I think you can distinguish two cases here:
1. Responses from Pod behind a Service
If a client makes a request to a Service, the request is routed to one of the Pods behind the Service. The response from this Pod to the client "passes through the Service" again.
The reason is that the source IP address of the IP datagram has to be changed from the Pod's IP address to the Service's IP address. This is to make the actual Pods that implement a Service transparent to the client. For the client it looks as if the Service IP address and port is a single server and it communicates directly with it.
2. A Pod making an independent request
If a Pod makes an independent request, this request doesn't pass through the Service but directly goes to whatever IP address the Pod is requesting.
For example, if a Pod makes a direct request to another Pod in the cluster, the destination IP address of the request is this Pod's IP address. If a Pod makes a request to a Service in the cluster, the destination IP address of the request is this Service's IP address. If a Pod makes a request to a destination outside the cluster, the destination IP address of the request is the IP address of this external destination.
I have created a k8s service (type=loadbalancer) with a numbers of pods behind. To my understanding, all packets initiazed from the pods will have the source ip as PodIP, wheareas those responding the inbound traffic will have the source ip as LoadBalancer IP. So my questions are:
Is my claim true, or there are times the source IP will be the node IP instead?
Are there any tricks in k8s, which I can change the source IP in the first scenario from PodIP to LB IP??
Any way to specify a designated pod IP??
The Pods are running in the internal network while the load balancer is exposed on the Internet, so the addresses of the packets will look more or less like this:
[pod1] <-----> [load balancer] <-----> [browser]
10.1.0.123 10.1.0.234 201.123.41.53 217.123.41.53
For specifying the pod IP have a look at SessionAffinity.
As user315902 said, Azure ACS k8s exposed service to internet with Azure load balancer.
Architectural diagram of Kubernetes deployed via Azure Container Service:
Is my claim true, or there are times the source IP will be the node IP
instead?
If we expose the service to internet, I think the source IP will be the load balancer public IP address. In ACS, if we expose multiple services to internet, Azure LB will add multiple public IP addresses.
Are there any tricks in k8s, which I can change the source IP in the
first scenario from PodIP to LB IP??
Do you mean you want to use node public IP address to expose the service to internet? if yes, I think we can't use node IP to expose service to internet. In Azure, we had to use LB to expose service to internet.
My nginx access log deployed in a GKE Kubernetes cluster (with type LoadBalancer Kubernetes service) shows internal IPs instead of real visitor IP.
Is there a way to find real IPs anywhere? maybe some log file provided by GKE/Kubernetes?
Right now, the type: LoadBalancer service does a double hop. The external request is balanced among all the cluster's nodes, and then kube-proxy balances amongst the actual service backends.
kube-proxy NATs the request. E.g. a client request from 1.2.3.4 to your external load balancer at 100.99.98.97 gets NATed in the node to 10.128.0.1->10.100.0.123 (node's private IP to pod's cluster IP). So the "src ip" you see in the backend is actually the private IP of the node.
There is a feature planned with a corresponding design proposal for preservation of client IPs of LoadBalancer services.
You could use the real IP module for nginx.
Pass your internal GKE net as a set_real_ip_from directive and you'll see the real client IP in your logs:
set_real_ip_from 192.168.1.0/24;
Typically you would add to the nginx configuration:
The load balancers IP
i.e. the IP that you see in your logs instead of the real client IP currently
The kubernetes network
i.e. the subnet your Pods are in, the "Docker subnet"
Adding of these lines to my nginx.conf HTTP block fixed this issue for me and real visitor IPs started displaying in Stackdriver LogViewer:
http {
...
real_ip_recursive on;
real_ip_header X-Forwarded-For;
set_real_ip_from 127.0.0.1;
set_real_ip_from 192.168.0.0/24;
set_real_ip_from 10.0.0.0/8;
...
}
I'm a happy camper :)