I'm looking for a way to to create an internal HTTPs L7 ingress controller in GCE, but there doesn't seem to be any documentation in regards to how this can be done. I see that there is a way to create an internal a L4 LoadBalancer, but not an L7 ingress. Why is this the case and what are possible solutions or workarounds?
If you want an "internal" ingress-controller setup you simply do not need to expose your Service as type `LoadBalancer. Your ingress-controller will be accessible via the node port.'
If you're wanting an actual load balancer in front of your ingress you'll have to roll your own such as HAproxy.
Related
I don't know if I missed something, but I can't seem to find any posts/doc that is related to my question. Maybe I misunderstand the type ingress in kubernetes, but is it possible to define multiple ingresses that use the same LoadBlancer? Having to start one LoadBalancer for every ingress is costly.
One of the benefit of using ingress it helps to avoid creating an external LoadBalancer for each LoadBalancer type service. On many cloud providers some of the ingress controllers will create the corresponding external Load Balancer resource for each ingress resource. But using Nginx Ingress controller you need one loadBalancer to expose the Nginx Ingress controller itself. Then create multiple ingress resource and have multiple backends. All the backends are served by same external Load Balancer.
From the docs of Nginx Ingress
In this section you can find a common usage scenario where a single
load balancer powered by ingress-nginx will route traffic to 2
different HTTP backend services based on the host name
This is more a design question than an issue. We have deployed in our company our own Kubernetes infrastructure and we are trying to use ingresses and NGINX ingress controller to externally expose our services, but since it is not a cloud environment such as GCP or AWS, we can't use service type "LoadBalancer". Should we just expose our ingress controller through a service type "NodePort"? Is that the normal way to go for production environments (non-cloud)?
From what I've read in another post, one suitable recommendation is to use NodePort, and manually point yet another external load balancer to the port on your Kubernetes nodes.
It just seems that exposing the ingress controller through this mechanism is somehow not very practical or robust (e.g. you don’t know what port your service is going to be allocated, and the port might get re-allocated at some point, etc.)
Is there any other mechanism maybe to expose the ingress controller to the external world?
The Loadbalancer service approach is one way to do it but behind it it's nothing more than a nodeport on the cluster.
Even if you use a service that create a LB on cloud provider, the LB needs to have a target port to communicate with the cluster.
When using a nginx-ingress that will mostly handle web requests, it's common usage to put an ingress in front of a nodeport service.
So with this I think using NodePort services is a good idea to do what you want ;)
This is my opinion, I'm interested if anyone else has another way to do it.
You can specify the port via nodePort in the service. Then it would not be random.
GKE Ingress: https://cloud.google.com/kubernetes-engine/docs/concepts/ingress
Nginx Ingress: https://kubernetes.github.io/ingress-nginx/
Why GKE Ingress
GKE Ingress can be used along with Google's managed SSL certificates. These certificates are deployed in edge servers of load balancer which results in very low TTFB (time to first byte)
What's wrong about GKE Ingress
The HTTP/domain routing is done in the load balancer using 'forward rules' which is very pricy. Costs around $7.2 per rule. Each domain requires one rule.
Why Nginx Ingress
Nginx Ingress also creates (TCP/UP) load balancer where we can specify routing of HTTP/domain using ingress controller. Since the routing is done inside the cluster there are no additional costs on adding domains into the rules
What's wrong about Nginx Ingress
To enable SSL, we can use cert-manager. But as I mentioned above, Google's managed certificate deploy certificates in edge servers which results in very low latency
My Question
Is it possible to use both of them together? So that HTTPS requests first hit GKE ingress which will terminate SSL and route the traffic to Nginx ingress which will route it to corresponding pods
Is not possible to point an Ingress to another Ingress. Furthermore and in your particular case, is also not possible to point a GCE ingress class to Nginx since it relies in an HTTP(S) Load Balancer, which can only have GCE instances/instances groups (basically the node pools in GKE), or GCS buckets as backends.
If you were to deploy an Nginx ingress using GKE, it will spin up a Network Load Balancer which is not a valid backend for the HTTP(S) Load Balancer.
So is neither possible via Ingress nor GCP infrastructure features. However, if you need the GCE ingress class to be hit first, and then, manage further routing with Nginx, you might want to consider having Nginx as a Kubernetes Service/Deployment to manage the incoming traffic once is within the cluster network.
You can create a ClusterIP service for internally accessing your Nginx deployment and from there, using cluster-local hostnames to redirect to other services/applications within the cluster.
I'm not sure how load balancing works with Ingress.
If I understand correctly, what happens is actually something like this:
I fail to see how the load balancing is performed.
What is wrong in the above scheme that I have drawn?
Can you help me rectify it?
Notes:
- The following answer tells me that the Ingress controller itself is of type 'loadbalancer': Ingress service type
- I use kind ClusterIP because I don't want to expose the loadbalancer to the outside world. The following article does not support this claim, where the load balancer would be provided by the service:
https://medium.com/google-cloud/kubernetes-nodeport-vs-loadbalancer-vs-ingress-when-should-i-use-what-922f010849e0
The ClusterIP services themselves perform load balancing. The naming can be confusing as LoadBalancer services are not the only services that involve load balancing - LoadBalancer actually means something more like 'cloud provider please create an external load balancer and point it at this service'. The kubernetes ClusterIP services also load-balance across Pods in different Nodes using the kube-proxy. If you don't want kubernetes to do load balancing then you have to specifically disable it by creating a headless service.
It seems like the first scheme you drew is correct. But I think you get confused in terminology. Particularly in the difference between ingress and ingress-controller.
Ingress is a type of resources in k8s (like Service, Deployment, ReplicaSet etc). We use ingress if we want to expose some services to an external world with binding to some path and host (i.e. myapp.com/api -> my-api-service).
The job of ingress-controller is to handle creation/update/deletion of ingress resources and implement all the functionality needed for ingress. Under the hood ingress-controller is a simple deployment exposed as LoadBalancer or NodePort service depending on where k8s is deployed. And image-controller forwards received request further to one of pods of service which matches host and path in some of the deployed ingress resources.
I got curious and thought: why would I need an Ingress for load balancing on layer 7 if the only thing it does is forward the traffic to a Service that implements the load balancing on layer 4?
Most Ingress controller implementations I looked up, talk to the Kubernetes API server to keep track of all Pods associated with a Service. Instead of forwarding traffic to the Service, they skip the intermediary and directly forward to the Pods. Since an ingress controller operates on layer 7, it enables a more application-oriented load balancing.
I understood Ingress can be used when we want to expose multiple service/routes with a single Load Balancer / public IP.
Now I want to expose my Nginx server to public. I have two choices
Set service type as LoadBalancer voila I got public IP
Use Nginx Ingress Controller
Now I can get my job done with Option 1 when or why would I choose Option 2 whats the advantage of having nginx with Ingress without Ingress ?
There is a difference between ingress rule (ingress) and ingress controller. So, technically, nginx ingress controller and LoadBalancer type service are not comparable. You can compare ingress resource and LoadBalancer type service, which is below.
Generally speaking:
LoadBalancer type service is a L4(TCP) load balancer. You would use it to expose single app or service to outside world. It would balance the load based on destination IP address and port.
Ingress type resource would create a L7(HTTP/S) load balancer. You would use this to expose several services at the same time, as L7 LB is application aware, so it can determine where to send traffic depending on the application state.
ingress and ingress controller relation:
Ingress, or ingress rules are the rules that ingress controller follows to distribute the load. Ingress controller get the packet, checks ingress rules and determines to which service to deliver the packet.
Nginx Ingress Controller
Nginx ingress controller uses LoadBalancer type service actually as entrypoint to the cluster. Then is checks ingress rules and distributes the load. This can be very confusing. You create an ingress resource, it creates the HTTP/S load balancer. It also gives you an external IP address (on GKE, for example), but when you try hitting that IP address, the connection is refused.
Conclusions:
You would use Loadbalancer type service if you would have a single app, say myapp.com that you want to be mapped to an IP address.
You would use ingress resource if you would have several apps, say myapp1.com, myapp1.com/mypath, myapp2.com, .., myappn.com to be mapped to one IP address.
As the ingress is L7 it is able to distinguish between myapp1.com and myapp1.com/mypath, it is able to route the traffic to the right service.
Accepted answer covered a lots of stuff already. All of the reasons are valid, apart from that the reason I am using ingress controller in aws is to minimize cost. I have multiple web applications which are running in kubernetes cluster aws. To access those applications instead of exposing individual application as LoadBalancer and creating individual ELB (each ELB cost money), I expose ingress controller service as LoadBalancer and created ingress rule for each.
Steps involve:
Ingress service, exposed as loadbalancer which created ELB in aws lets say elb1.aws.com
Ingress rule for each web applications, eg example.com, awesome.com, helloworld.com
Route53 mapping all mapped to same ELB, eg:
example.com -> elb1.aws.com
awesome.com -> elb1.aws.com
helloworld.com -> elb1.aws.com