I have a kubernetes objects as below:
a deployment
a service to use with that deployment in step 1
an ingress with backend paths to the service in step 2
I am using Kubernetes Engine in GCP. Once I created an ingress object, it created a load balancer as usual.
So I have my dns, and I added a A record with a domain name test1.<my-domain>.co pointing to the IP of the LoadBalancer created from the ingress
But this is not working. It doesn't load the page.
Then I tried installing ningx ingress controller, and once that is being deployed, it created another load balancer in the gcp. So, I got the IP of that newly created load balancer and switched/changed replace the DNS record IP with newly created Load Balancer's IP. And voila, it started working. So does that mean, an ingress always need an ingress controller to work?
Yes, in order for the Ingress resource to work, the cluster must have an ingress controller running. Only creating an Ingress resource has no effect.
An Ingress controller is responsible for fulfilling the Ingress, usually with a load balancer, which is what you see.
A request from the client lands up on the Ingress managed Load Balancer which is forwarded to the respective Ingress based on the host and path in the original request. Following the routing rules defined in the ingress, request is forwarded to the service from where it lands up on the backend pods.
Ingress resource creating its own Load Balancer seems to be a behaviour followed in GKE. From the GCP docs
When you specify kind:Ingress in the resource manifest, you instruct
GKE to create an Ingress resource. By including annotations and
supporting workloads and Services, you can create a custom Ingress
controller. Otherwise, GKE makes appropriate Google Cloud API calls to
create an external HTTP(S) load balancer.
You can read more about this here.
yes ingress controller is needed to serve user request comes outside the cluster.
when user sends request on load balancer IP of ingress controller ingress controller reads route from ingress resource and forward user request accordingly.
ingress resource is a part of service. means for every service you need to have ingress resource where as a ingress controller can serve multiple ingress resource.
There are mainly two ingress controller used.
nginx
contour.
you can read about them in details.
Related
nginx Ingress is mainly used for path based routing and sub domain based routing to route the request to particular pod and Cloud provider load balancer will provide external ip address to get the requests from external world and which in turn points to ingress.
In Kubernetes, Service components acts as a real load balancer by balancing the load to multiple pods in the cluster. This is my understanding, Am I correct?
Yes you are correct, so the flow goes like
You create the LB on any Cloud provider which gives you endpoint to the internet you can use it with ingress controller. (You can further use the SSL/TLS certificate also with LB to run HTTPS)
If you are using the Nginx ingress controller or any other controller depending on that it will handle or manage the Ingress resource in the cluster.
Once ingress rule decide service to forward traffic, K8s internal service load balance the traffic across running PODs for specific deployment based on rule.
internet > Loadbalancer > ingress > ingress controller checks > service > Loadbalance traffic across avilable POD of that specific service
Default load balancing will be round robbin.
In GKE Ingress documentation
it states that:
When you create an Ingress object, the GKE Ingress controller creates a Google Cloud HTTP(S) Load Balancer and configures it according to the information in the Ingress and its associated Services.
To me it seems that I can not have multiple ingress resources with single GCP ingress controller. Instead, GKE creates a new ingress controller for every ingress resource.
Is this really so, or is it possible to have multiple ingress resources with a single ingress controller in GKE?
I would like to have one GCP LoadBalancer as ingress controller with static IP and DNS configured, and then have multiple applications running in cluster, each application registering its own ingress resource with application specific host and/or path specifications.
Please note that I'm very new to GKE, GCP and Kubernetes in general, so it might be that I have misunderstood something.
I think the question you're actually asking is slightly different than what you have written. You want to know if multiple Ingress resources can be linked to a single GCP Load Balancer, not GKE Ingress controller. Based on the concept of a controller, there is only one GKE Ingress controller in a cluster, which is responsible for fulfilling multiple resources and provisioning multiple load balancers.
So, to answer the question directly (because I've been searching for a straight answer for a long time!):
Combining multiple Ingress resources into a single Google Cloud load
balancer is not supported.
Source: https://cloud.google.com/kubernetes-engine/docs/concepts/ingress
Sad.
However, using the nginx-ingress controller is one way to at least minimize the number of external (GCP) load balancers provisioned (it only provisions a single TCP load balancer), but since the load balancer is for TCP traffic, it cannot terminate SSL, or apply Firewall rules for you (Cloud Armor cannot be used, for instance).
The only way I know of to have a single HTTPS load-balancer in GCP terminate SSL and route traffic to multiple services in GKE is to combine the ingresses into a single resource with all paths and certificates defined in one place.
(If anybody figures out a way to do it with multiple separate ingress resources, I'd love to hear it!)
Yes it is possible to have the single ingress controller for multiple ingress resources.
You can create multiple ingress resources as per path requirement and all will be managed by single ingress controller.
There are multiple ingress controller options also available you can use Nginx also that will create one LB and manage the paths.
Inside Kubernetes if you are creating a service with type LoadBalancer it will create the new LB resource in GCP so make sure your microservice type is ClusterIP and your all traffic goes inside K8s cluster via ingress path.
When you setup the ingress controller it will create one service with type LoadBalancer you can can use that IP in DNS servers to forward the subdomain and path to K8s cluster.
For a sample microservice based architecture deployed on Google kubernetes engine, I need help to validate my understanding :
We know services are supposed to load balance traffic for pod replicaset.
When we create an nginx ingress controller and ingress definitions to route to each service, a loadbalancer is also setup automatically.
had read somewhere that creating nginx ingress controller means an nginx controller (deployment) and a loadbalancer type service getting created behind the scene. I am not sure if this is true.
It seems loadbalancing is being done by services. URL based routing is
being done by ingress controller.
Why do we need a loadbalancer? It is not meant to load balance across multiple instances. It will just
forward all the traffic to nginx reverse proxy created and it will
route requests based on URL.
Please correct if I am wrong in my understanding.
A Service type LoadBalancer and the Ingress is the way to reach your application externally, although they work in a different way.
Service:
In Kubernetes, a Service is an abstraction which defines a logical set of Pods and a policy by which to access them (sometimes this pattern is called a micro-service). The set of Pods targeted by a Service is usually determined by a selector (see below for why you might want a Service without a selector).
There are some types of Services, and of them is the LoadBalancer type that permit you to expose your application externally assigning a externa IP for your service. For each LoadBalancer service a new external IP will be assign to it.
The load balancing will be handled by kube-proxy.
Ingress:
An API object that manages external access to the services in a cluster, typically HTTP.
Ingress may provide load balancing, SSL termination and name-based virtual hosting.
When you setup an ingress (i.e.: nginx-ingress), a Service type LoadBalancer is created for the ingress-controller pods and a Load Balancer in you cloud provider is automatically created and a public IP will be assigned for the nginx-ingress service.
This load balancer/public ip will be used for incoming connection for all your services, and nginx-ingress will be the responsible to handle the incoming connections.
For example:
Supose you have 10 services of LoadBalancer type: This will result in 10 new publics ips created and you need to use the correspondent ip for the service you want to reach.
But if you use a ingress, only 1 IP will be created and the ingress will be the responsible to handle the incoming connection for the correct service based on PATH/URL you defined in the ingress configuration. With ingress you can:
Use regex in path to define the service to redirect;
Use SSL/TLS
Inject custom headers;
Redirect requests for a default service if one of the service failed (default-backend);
Create whitelists based on IPs
Etc...
A important note about Ingress Load balancing in ingress:
GCE/AWS load balancers do not provide weights for their target pools. This was not an issue with the old LB kube-proxy rules which would correctly balance across all endpoints.
With the new functionality, the external traffic is not equally load balanced across pods, but rather equally balanced at the node level (because GCE/AWS and other external LB implementations do not have the ability for specifying the weight per node, they balance equally across all target nodes, disregarding the number of pods on each node).
An ingress controller(nginx for example) pods needs to be exposed outside the kubernetes cluster as an entry point of all north-south traffic coming into the kubernetes cluster. One way to do that is via a LoadBalancer. You could use NodePort as well but it's not recommended for production or you could just deploy the ingress controller directly on the host network on a host with a public ip. Having a load balancer also gives ability to load balance the traffic across multiple replicas of ingress controller pods.
When you use ingress controller the traffic comes from the loadBalancer to the ingress controller and then gets to backend POD IPs based on the rules defined in ingress resource. This bypasses the kubernetes service and load balancing(by kube-proxy at layer 4) offered by kubernetes service.Internally the ingress controller discovers all the POD IPs from the kubernetes service's endpoints and directly route traffic to the pods.
It seems loadbalancing is being done by services. URL based routing is being done by ingress controller.
Services do balance the traffic between pods. But they aren't accessible outside the kubernetes in Google Kubernetes Engine by default (ClusterIP type). You can create services with LoadBalancer type, but each service will get its own IP address (Network Load Balancer) so it can get expensive. Also if you have one application that has different services it's much better to use Ingress objects that provides single entry point. When you create an Ingress object, the Ingress controller (e.g. nginx one) creates a Google Cloud HTTP(S) load balancer. An Ingress object, in turn, can be associated with one or more Service objects.
Then you can get the assigned load balancer IP from ingress object:
kubectl get ingress ingress-name --output yaml
As a result your application in pods become accessible outside the kubernetes cluster:
LoadBalancerIP/url1 -> service1 -> pods
LoadBalancerIP/url2 -> service2 -> pods
GKE Ingress: https://cloud.google.com/kubernetes-engine/docs/concepts/ingress
Nginx Ingress: https://kubernetes.github.io/ingress-nginx/
Why GKE Ingress
GKE Ingress can be used along with Google's managed SSL certificates. These certificates are deployed in edge servers of load balancer which results in very low TTFB (time to first byte)
What's wrong about GKE Ingress
The HTTP/domain routing is done in the load balancer using 'forward rules' which is very pricy. Costs around $7.2 per rule. Each domain requires one rule.
Why Nginx Ingress
Nginx Ingress also creates (TCP/UP) load balancer where we can specify routing of HTTP/domain using ingress controller. Since the routing is done inside the cluster there are no additional costs on adding domains into the rules
What's wrong about Nginx Ingress
To enable SSL, we can use cert-manager. But as I mentioned above, Google's managed certificate deploy certificates in edge servers which results in very low latency
My Question
Is it possible to use both of them together? So that HTTPS requests first hit GKE ingress which will terminate SSL and route the traffic to Nginx ingress which will route it to corresponding pods
Is not possible to point an Ingress to another Ingress. Furthermore and in your particular case, is also not possible to point a GCE ingress class to Nginx since it relies in an HTTP(S) Load Balancer, which can only have GCE instances/instances groups (basically the node pools in GKE), or GCS buckets as backends.
If you were to deploy an Nginx ingress using GKE, it will spin up a Network Load Balancer which is not a valid backend for the HTTP(S) Load Balancer.
So is neither possible via Ingress nor GCP infrastructure features. However, if you need the GCE ingress class to be hit first, and then, manage further routing with Nginx, you might want to consider having Nginx as a Kubernetes Service/Deployment to manage the incoming traffic once is within the cluster network.
You can create a ClusterIP service for internally accessing your Nginx deployment and from there, using cluster-local hostnames to redirect to other services/applications within the cluster.
I understood Ingress can be used when we want to expose multiple service/routes with a single Load Balancer / public IP.
Now I want to expose my Nginx server to public. I have two choices
Set service type as LoadBalancer voila I got public IP
Use Nginx Ingress Controller
Now I can get my job done with Option 1 when or why would I choose Option 2 whats the advantage of having nginx with Ingress without Ingress ?
There is a difference between ingress rule (ingress) and ingress controller. So, technically, nginx ingress controller and LoadBalancer type service are not comparable. You can compare ingress resource and LoadBalancer type service, which is below.
Generally speaking:
LoadBalancer type service is a L4(TCP) load balancer. You would use it to expose single app or service to outside world. It would balance the load based on destination IP address and port.
Ingress type resource would create a L7(HTTP/S) load balancer. You would use this to expose several services at the same time, as L7 LB is application aware, so it can determine where to send traffic depending on the application state.
ingress and ingress controller relation:
Ingress, or ingress rules are the rules that ingress controller follows to distribute the load. Ingress controller get the packet, checks ingress rules and determines to which service to deliver the packet.
Nginx Ingress Controller
Nginx ingress controller uses LoadBalancer type service actually as entrypoint to the cluster. Then is checks ingress rules and distributes the load. This can be very confusing. You create an ingress resource, it creates the HTTP/S load balancer. It also gives you an external IP address (on GKE, for example), but when you try hitting that IP address, the connection is refused.
Conclusions:
You would use Loadbalancer type service if you would have a single app, say myapp.com that you want to be mapped to an IP address.
You would use ingress resource if you would have several apps, say myapp1.com, myapp1.com/mypath, myapp2.com, .., myappn.com to be mapped to one IP address.
As the ingress is L7 it is able to distinguish between myapp1.com and myapp1.com/mypath, it is able to route the traffic to the right service.
Accepted answer covered a lots of stuff already. All of the reasons are valid, apart from that the reason I am using ingress controller in aws is to minimize cost. I have multiple web applications which are running in kubernetes cluster aws. To access those applications instead of exposing individual application as LoadBalancer and creating individual ELB (each ELB cost money), I expose ingress controller service as LoadBalancer and created ingress rule for each.
Steps involve:
Ingress service, exposed as loadbalancer which created ELB in aws lets say elb1.aws.com
Ingress rule for each web applications, eg example.com, awesome.com, helloworld.com
Route53 mapping all mapped to same ELB, eg:
example.com -> elb1.aws.com
awesome.com -> elb1.aws.com
helloworld.com -> elb1.aws.com