This is more a design question than an issue. We have deployed in our company our own Kubernetes infrastructure and we are trying to use ingresses and NGINX ingress controller to externally expose our services, but since it is not a cloud environment such as GCP or AWS, we can't use service type "LoadBalancer". Should we just expose our ingress controller through a service type "NodePort"? Is that the normal way to go for production environments (non-cloud)?
From what I've read in another post, one suitable recommendation is to use NodePort, and manually point yet another external load balancer to the port on your Kubernetes nodes.
It just seems that exposing the ingress controller through this mechanism is somehow not very practical or robust (e.g. you don’t know what port your service is going to be allocated, and the port might get re-allocated at some point, etc.)
Is there any other mechanism maybe to expose the ingress controller to the external world?
The Loadbalancer service approach is one way to do it but behind it it's nothing more than a nodeport on the cluster.
Even if you use a service that create a LB on cloud provider, the LB needs to have a target port to communicate with the cluster.
When using a nginx-ingress that will mostly handle web requests, it's common usage to put an ingress in front of a nodeport service.
So with this I think using NodePort services is a good idea to do what you want ;)
This is my opinion, I'm interested if anyone else has another way to do it.
You can specify the port via nodePort in the service. Then it would not be random.
Related
Exposing Service from a BareMetal(Kubeadm) Build Kubernetes Cluster to the outside world. I am trying to access my Nginx as a service outside of the cluster to get NGINX output in the web browser.
For that, I have created a deployment and service for NGINX as shown below,
As per my search, found that we have below to expose to outside world
MetalLb
Ingress NGINX
Some HELM resources
I would like to know all these 3 or any more approaches in such way it help me to learn new things.
GOAL
Exposing Service from a BareMetal(Kubeadm) Built Kubernetes Cluster to the outside world.
How Can I make my service has its own public IP to access from the outside cluster?
You need to set up MetalLB to get an external IP address for the LoadBalancer type services. It will give a local network IP address to the service.
Then you can do port mapping (configuration in the router) of incoming traffic of port 80 and port 443 to your external service IP address.
I have done a similar setup you can check it here in detail:
https://developerdiary.me/lets-build-low-budget-aws-at-home/
You need to deploy an ingress controller in your cluster so that it gives you an entrypoint where your applications can be accessed. Traditionally, in a cloud native environment it would automatically provision a LoadBalancer for you that will read the rules you define inside your Ingress object and route your request to the appropriate service.
One of the most commonly used ingress controller is the Nginx Ingress Controller. There are multiple ways you can use to deploy it (mainfests, helm, operators). In case of bare metal clusters, there are multiple considerations which you can read here.
MetalLB is still in beta stage so its your choice if you want to use. If you don't have a hard requirement to expose the ingress controller as a LoadBalancer, you can expose it as a NodePort Service that will accessible across all your nodes in the cluster. You can then map that NodePort Service in your DNS so that the ingress rules are evaluated.
I want to set up a Kubernetes cluster on-Prem directly on VMs. Since we are not talking about using a cloud provider exposing my service as type LoadBalancer may not directly make sense. I understand MetalLB could be an option but I don’t have a pool of IP addresses to assign it.
I want an entry point into my cluster to which I can point my DNS A record to. I have a couple of solutions in mind but not sure if one is better than the other or there are other better solutions.
Exposing my service on NodePort and using an external load balancer. I can make the external LB my entry point.
Running an Ingress Controller on NodePort which routes traffic to my ClusterIP services internally. I could load balance between the Ingress NodePorts using an external LB and make it my entry point.
I only want to expose one service from my cluster to the outside world. In that case I am not sure how using Ingress will add any advantage.
Please help me out my sharing your thoughts and suggestions!
I've just deployed websocket based echo-server on AWS EKS. I see it's running stable and okay but when I was searching for implementation details I was finding only articles that were saying something about nginx ingress controller or AWS application loadbalancer and a lot of troubles with them.
Do I miss anything in my current, vanilla config? Do I need the AWS ALB or nginx ingress controller?
Thank you for all the replies.
All the best.
Do I miss anything in my current, vanilla config?
You probably exposed your echo-server app using service type - ClusterIP or NodePort which is fine if you only need to access your app locally in the cluster (ClusterIP) or using your node IP address (NodePort).
Do I need the AWS ALB or nginx ingress controller?
They both are different things, but they have similar common goal - to make your websocket app available externally and distribute traffic based on defined L7 routing routes. It's good solution if you have multiple deployments. So you need to answer yourself if you need some kind of Ingress Controller. If you are planning to deploy your application into production you should consider using those solutions, but probably it may be fine with service type LoadBalancer.
EDIT:
If you are already using service type LoadBalancer your app is already available externally. Ingress controller provides additional configuration possibilities to configure L7 traffic route to your cluster (Ingress Controllers are often using LoadBalancer under the hood). Check this answer for more details about differences between LoadBalancer and Ingress.
Also check:
Choosing the Right Load Balancer on Amazon: AWS Application Load Balancer vs. NGINX Plus
Configuring Kubernetes Ingress on AWS? Don’t Make These Mistakes
WebSocket - Deploy to Kubernetes
LoadBalancer vs Ingress
I'm not sure how load balancing works with Ingress.
If I understand correctly, what happens is actually something like this:
I fail to see how the load balancing is performed.
What is wrong in the above scheme that I have drawn?
Can you help me rectify it?
Notes:
- The following answer tells me that the Ingress controller itself is of type 'loadbalancer': Ingress service type
- I use kind ClusterIP because I don't want to expose the loadbalancer to the outside world. The following article does not support this claim, where the load balancer would be provided by the service:
https://medium.com/google-cloud/kubernetes-nodeport-vs-loadbalancer-vs-ingress-when-should-i-use-what-922f010849e0
The ClusterIP services themselves perform load balancing. The naming can be confusing as LoadBalancer services are not the only services that involve load balancing - LoadBalancer actually means something more like 'cloud provider please create an external load balancer and point it at this service'. The kubernetes ClusterIP services also load-balance across Pods in different Nodes using the kube-proxy. If you don't want kubernetes to do load balancing then you have to specifically disable it by creating a headless service.
It seems like the first scheme you drew is correct. But I think you get confused in terminology. Particularly in the difference between ingress and ingress-controller.
Ingress is a type of resources in k8s (like Service, Deployment, ReplicaSet etc). We use ingress if we want to expose some services to an external world with binding to some path and host (i.e. myapp.com/api -> my-api-service).
The job of ingress-controller is to handle creation/update/deletion of ingress resources and implement all the functionality needed for ingress. Under the hood ingress-controller is a simple deployment exposed as LoadBalancer or NodePort service depending on where k8s is deployed. And image-controller forwards received request further to one of pods of service which matches host and path in some of the deployed ingress resources.
I got curious and thought: why would I need an Ingress for load balancing on layer 7 if the only thing it does is forward the traffic to a Service that implements the load balancing on layer 4?
Most Ingress controller implementations I looked up, talk to the Kubernetes API server to keep track of all Pods associated with a Service. Instead of forwarding traffic to the Service, they skip the intermediary and directly forward to the Pods. Since an ingress controller operates on layer 7, it enables a more application-oriented load balancing.
all
I knew well about k8s' nodePort and ClusterIP type in services.
But I am very confused about the Ingress way, because how will a request come into a pod in k8s by this Ingress way?
Suppose K8s master IP is 1.2.3.4, after Ingress setup, and can connect to backend service(e.g, myservice) with a port(e.g, 9000)
Now, How can I visit this myservice:9000 outside? i.e, through 1.2.3.4? As there's no entry port on the 1.2.3.4 machine.
And many docs always said visit this via 'foo.com' configed in the ingress YAML file. But that is really funny, because xxx.com definitely needs DNS, it's not a magic to let you new-invent any xxx.com you like be a real website and can map your xxx.com to your machine!
The key part of the picture is the Ingress Controller. It's an instance of a proxy (could be nginx or haproxy or another ingress type) and runs inside the cluster. It acts as an entrypoint and lets you add more sophisticated routing rules. It reads Ingress Resources that are deployed with apps and which define the routing rules. This allows each app to say what the Ingress Controller needs to do for routing to it.
Because the controller runs inside the cluster, it needs to be exposed to the outside world. You can do this by NodePort but if you're using a cloud provider then it's more common to use LoadBalancer. This gives you an external IP and port that reaches the Ingress controller and you can point DNS entries at that. If you do point DNS at it then you have the option to use routing rules base on DNS (such as using different subdomains for different apps).
The article 'Kubernetes NodePort vs LoadBalancer vs Ingress? When should I use what?' has some good explanations and diagrams - here's the diagram for Ingress: