I have a created an nginx pod and nginx clusterIP service and assign an externalIP to that service like below
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
test-nginx ClusterIP 10.110.93.251 192.168.0.10 443/TCP,80/TCP,8000/TCP,5443/TCP 79m
In one of my application pod, I am trying to execute below command and get the fqdn of it.
>>> import socket
>>> socket.getfqdn('192.168.0.10')
'test-nginx.test.svc.cluster.local'
It returns me the nginx service fqdn instead of my host machine fqdn. Is there a way to block dns resolution only for external-ip ? or is there any other workaround for this problem?
You assigned an external ip to a ClusterIP service in Kubernetes, so you can access your application from outside the Cluster, but you are concerned about the Pods having access to that external ip and want to block the dns resolution.
This is not the best approach to your issue, Kubernetes has several ways to expose the services without compromising the security; for what you want, maybe a better option is to implement an Ingress instead.
As you can see in the diagram, the Ingress routes the incoming traffic to the desired service based on configured rules, isolating the outside world from your service and only allowing specific traffic to go in. You can also implement features as TLS termination for your HTTPS traffic, and it performs load balancing by default.
Even further, if your main concern is security within your Cluster, you can take a look at the Istio Service mesh.
Related
I'm unable to wrap my head around the concepts of interconnectivity among DNS, Ingress controller, MetalLb and Kubeproxy.
I know what these resources/tools/services are for and get the concepts of them individually but unable to form a picture of them working in tandem.
For example, in a bare metal setup, a client accesses my site - https://mytestsite.com, still having doubts , how effectively it lands to the right pod and where the above mentioned services/resources/tools comes into picture & at what stage ?
Ex. How DNS talks to my MetalLB, if the client accesses my MetalLB hosting my application and how LB inturn speaks to IngressController and finally where does Kube-proxy comes into play here.
I went thru the K8s official documentation and few others as well but still kind of stumped here. Following article is really good but I'm unable to stitch the pieces together.
https://www.disasterproject.com/kubernetes-with-external-dns/
Kindly redirect me to the correct forum, if it is not the right place, thanks.
The ingress-controller creates a service of type LoadBalancer that serves as the entry point into the cluster. In a public cloud environment, a loadbalancer like ELB on AWS would create the counter part and set the externalIP of that service to it's ip. It is like a service of type NodePort but it also has an ExternalIP, which corresponds to the actual ip of the counterpart, a load balancer like ELB on aws.
In a bare metal environment, no external load balancer will be created, so the external ip would stay in <Pending> state forever. Here for example the service of the istio ingress controller:
$ kubectl get svc istio-ingressgateway -n istio-system
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S)
istio-ingressgateway LoadBalancer 192.12.129.119 <Pending> [...],80:32123/TCP,443:30994/TCP,[...]
In that state you would need to call http://<node-ip>:32123 to reach the http port 80 of the ingress controller service, which would be then forwarded to your pod (more on that in a bit).
When you're using metallb, it will update the service with an external ip so you can call http://<ip> instead. Metallb will also announce that ip, e.g. via BGP, so other know where to send traffic to, when someone would call the ip.
I havn't used external DNS and only scanned the article but I guess that you can use that to also have a dns record to be created so someone can call your service by it's domain, not only by it's ip. So you can call http://example.com instead.
This is basically why you run metallb and how it interacts with your ingress controller. The ingress controller creates an entry point into the cluster and metallb configures it and attracts traffic.
Until now the call to http://example.com can reach your cluster, but it needs to also reach the actual application, running in a pod inside the cluster. That's kube-proxy's job.
You read a lot about service of different types and all this kind of stuff, but in the end it all boils down to iptables rules. kube-proxy will create a bunch of those rules, that form a chain.
SSH into any kubernetes worker, run iptables-save | less command and search for the external ip configured on your ingress-controller's service by metallb. You'll find a chain with the destination of you external ip, that basically leads from the external IP over the service ip with a load balancer configuration to a pod ip.
In the end the whole chain would look something like this:
http://example.com
-> http://<some-ip> (domain translated to ip)
-> http://<node-ip>:<node-port> (ingress-controller service)
---
-> http://<cluster-internal-ip>:<some-port> (service of your application)
-> http://<other-cluster-internal-ip>:<some-port> (ip of one of n pods)
where the --- line shows the switch from cluster external to cluster internal traffic. The cluster-internal-ip will be from the configured service-cdir and the other-cluster-internal-ip will be from the configured pod-cidr.
Note that there are different ways to configure cluster internal traffic routing, how to run kube-proxy and some parts might even be a bit simplified, but this should give you a good enough understanding of the overall concept.
Also see this answer on the question 'What is a Kubernetes LoadBalancer On-Prem', that might provide additional input.
I understand that, in Cloud scenarios, a LoadBalancer resource refers to and provisions an external layer 4 load balancer. There is some proprietary integration by which the cluster configures this load balancing. OK.
On-prem we have no such integration so we create our own load balancer outside of the cluster which distributes traffic to all nodes - effectively, in our case to the ingress.
I see no need for the cluster to know anything about this external load balancer.
What is a LoadBalancer* then in an on-prem scenario? Why do I have them? Why do I need one? What does it do? What happens when I create one? What role does the LoadBalancer resource play in ingress traffic. What effect does the LoadBalancer have outside the cluster? Is a LoadBalancer just a way to get a new IP address? How/why/ where does the IP point to and where does the IP come from?
All questions refer to the Service of type “LoadBalancer” inside cluster and not my load balancer outside the cluster of which the cluster has no knowledge.
As pointed out in the comments a kubernetes service of type LoadBalancer can not be used by default with on-prem setups. You can use metallb to setup a service of that type in an on prem environment.
Kubernetes does not offer an implementation of network load balancers (Services of type LoadBalancer) for bare-metal clusters. [...] If you’re not running on a supported IaaS platform (GCP, AWS, Azure…), LoadBalancers will remain in the “pending” state indefinitely when created. [...] MetalLB aims to redress this imbalance by offering a network load balancer implementation that integrates with standard network equipment, so that external services on bare-metal clusters also “just work” as much as possible.
You can for example use the BGP mode to advertise the service's IP to your router, read more on that in the docs.
The project is still in beta but is promoted as production ready and used by several bigger companies.
Edit
Regarding your question in the comments:
Can I just broadcast the MAC address of my node and manually add the IP I am broadcasting to the LoadBalancer service via kubectl edit?
Yes that would work too. That's basically what metallb does, announcing the IP and updating the service.
Why need a software then? Imaging having 500 hosts that come and go with thousends of services of type LoadBalancer that come and go. You need an automation here.
Why does Kubernetes need to know this IP?
It doesn't. If you don't use an external ip, the service is still usable via it's NodePort, see for example the istio docs (with a little more details added by me):
$ kubectl get svc istio-ingressgateway -n istio-system
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S)
istio-ingressgateway LoadBalancer 192.12.129.119 <Pending> [...],80:32123/TCP,443:30994/TCP,[...]
Here the external IP is not set and stays in <Pending>. You can still use the service by pointing your traffic to <Node-IP>:32123 for plain http and to <Node-IP>:30994 for https. As you can see above those ports are mapped to 80 and 443.
If the external ip is set you can direct traffic directly to port 80 and 443 on the external load balancer. Kube-Proxy will create an iptables chain with the destination of you external ip, that basically leads from the external IP over the service ip with a load balancer configuration to a pod ip.
To investigate that set up a service of type LoadBalancer, make sure it has an external ip, connect to the host and run the iptables-save command (e.g. iptables-save | less). Search for the external ip and follow the chain until you end up at the pod.
I'm running a bare metal Kubernetes cluster with 1 master node and 3 worker Nodes. I have a bunch of services deployed inside with Istio as an Ingress-gateway.
Everything works fine since I can access my services from outside using the ingress-gateway NodePort.
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
istio-ingressgateway LoadBalancer 10.106.9.2 <pending> 15021:32402/TCP,80:31106/TCP,443:31791/TCP 2d23h
istiod ClusterIP 10.107.220.130 <none> 15010/TCP,15012/TCP,443/TCP,15014/TCP 2d23h
In our case the port 31106.
The issues is, I don't want my customer to access my service on port 31106. that's not user friendly. So is there a way to expose the port 80 to the outside ?
In other word, instead of typing http://example.com:31106/ , I want them to be able to type http://example.com/
Any solution could help.
Based on official documentation:
If the EXTERNAL-IP value is set, your environment has an external load balancer that you can use for the ingress gateway. If the EXTERNAL-IP value is <none> (or perpetually <pending>), your environment does not provide an external load balancer for the ingress gateway. In this case, you can access the gateway using the service’s node port.
This is in line with what David Maze wrote in the comment:
A LoadBalancer-type service would create that load balancer, but only if Kubernetes knows how; maybe look up metallb for an implementation of that. The NodePort port number will be stable unless the service gets deleted and recreated, which in this case would mean wholesale uninstalling and reinstalling Istio.
In your situation you need to access the gateway using the NodePort. Then you can configure istio. Everything is described step by step in this doc. You need to choose the instructions corresponding to NodePort and then set the ingress IP depends on the cluster provider. You can also find sample yaml files in the documentation.
I have a api-service with type 'ClusterIp' which is working fine and is accessible on the node with clusterip. I want to access it externally . It's a baremetal installation with kubeadm . I cannot use Loadbalancer or Nodeport.
If I use nginx-ingress that too I will use as 'ClusterIP' so how to get the service externally accessible in either api service or nginx-ingress case .
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
api ClusterIP 10.97.48.17 <none> 80/TCP 41s
ingress-nginx ClusterIP 10.107.76.178 <none> 80/TCP 3h49m
Changes to solve the issue:
nginx configuration on node
in /etc/nginx/sites-available
upstream backend {
server node1:8001;
server node2:8001;
server node3:8001;
}
server_name _;
location / {
proxy_pass http://backend;
# First attempt to serve request as file, then
# as directory, then fall back to displaying a 404.
try_files $uri $uri/ =404;
}
Ran my two services as DaemonSet
ClusterIP services are accesible only within the cluster.
For bare metal clusters, you can use any of the following approaches to make a service available externally. Suggestions are from most recommended to least recommended order:
Use metallb to implement LoadBalancer service type support - https://metallb.universe.tf/. You will need a pool of IP addresses for metallb to handout. It also supports IP sharing mode where you can use same IP for multiple LoadBalancer services.
Use NodePort service. You can access your service from any node IP:node_port address. NodePort service selects random port in node port range by default. You can choose a custom port in node port range using spec.ports.nodePort field in the service specification.
Disadvantage: The node port range by default is 30000-32767. So you cannot bind to any custom port that you want like 8080. Although you can change the node port range with --service-node-port-range flag of kube-api-server, it is not recommended to use it with low port ranges.
Use hostPort to bind a port on the node.
Disadvantage: You don't have fixed IP address because you don't know which node your pod gets scheduled to unless you use nodeAffinity. You can make your pod a daemonset if you want it to be accessible from all nodes on the given port.
If you are dealing with HTTP traffic, another option is installing an IngressController like nginx or Traefik and use Ingress resource. As part of their installation, they use one of the approaches mentioned above to make themselves available externally.
Well, as you can guess by reading the name, ClusterIp is only accessible from inside the cluster.
To make a service accessible from outside the cluster, you havec 3 options :
NodePort Service type
LoadBalancer Service type (you still have to manage your LoadBalancer manually though)
Ingress
There is a fourth option which is hostPort (which is not a service type) but I'd rather keep it from special case when you're absolutely sure that your pod will always be located on the same node (or eventually for debug).
Having this said, then this leaves us only with one solution offered by Kubernetes : Ingress.
When reading through the kubernetes document I noticed that NodePort type service always automatically creates a ClusterIP and ingress traffic targeting NodePort will be routed to ClusterIP. My question is that why is this necessary? Why can't kubeproxy directly does load balancing for this NodePort through forwarding? ClusterIP doesn't seem to be necessary to support NodePort and it seems to introduce additional overhead.
Even Service of type NodePort does not directly contact Pods, they still go through the Service's cluster IP, and its associated Pod selector rules. With newer kubernetes I think you can also influence whether traffic is round-robin or weighted distributed, which wouldn't work if the NodePort directly contacted the Pods
Also, NodePorts are opened on every member of the cluster, but -- in most cases -- you don't have a Pod running on every member of the cluster, so it still has to use the Service IP to route to the actual Node upon which an in-service Pod can service that traffic.
Think of NodePorts an a mechanism to bridge the "outside world" with the "cluster world," rather than a shortcut mechanism to side-step iptables or ipvs