We have a multi tenant application and for each tenant we provision separate container image.
Likewise we create a subdomain for each tenant which will be redirected to its own container.
There might be a scenario where 1000s of tenants can exist and its dynamic.
So it has become necessary for us to consider the limitations in ingress controllers for Kubernetes in general before choosing. Especially the nginx-ingress.
Is there any max limitation on number of Ingress resources or rules inside ingress that can be created? Or will there be any performance or scaling issues when too many ingress resources are created ?
Is it better to add a new rule(for each subdomain) in same ingress resource or to create separate ingress resource for each subdomain ?
AFAIK, there are no limits like that, You will either run out of resources or find a choke point first. This article compares resource consumption of several loadbalancers.
As for Nginx-ingress there are few features hidden behind paid nginx plus version as listed here.
If You wish to have dynamic configurations and scalability, You should try envoy based ingress like Ambassador or Istio.
Envoy offers dynamic configuration updates which will not interrupt existing connections. More info here.
Check out this article which compares most of popular kubernetes ingress controllers.
This article shows a great example of pushing HAproxy and Nginx combination to its limits.
Hope it helps.
Related
Using stackoverflow in my work for a long time ago, but actually asking my first question here :)
We currently have a K8s cluster with an Istio Ingress Gateway & cert-manager for our public facing TLS-terminated endpoint, and are willing to setup another public endpoint serving customers' own domain names (from their own dns provider).
Customers could go on their project back-office and ask to connect their project to their own domain name by filling a simple form & pasting the appropriate CNAME/A rule within their DNS provider.
These custom domain names would be served by another Ingress Gateway instance configured with 1 LetsEncrypt certificate per domain, generated from cert-manager using HTTP challenges.
However, I wonder about the relevance & scalability of this stack.
First, and as mentioned by this thread title, is there any limit to the number of Gateway resource that a single ingress gateway instance can handle ? We could group all custom rules into a single Gateway resource, but having a separate Gateway per customer would be far easier for maintenance / update / clean up processes. Theoretically this could go up to a few dozens or even 100+ Gateway resources.
As a side note, have you any specific feedback/recommendation on dynamically generating customers' own certificates & ingress rules using this or other better approach ?
One consequence that I really dislike is that we have to push K8s manifests from one of our microservices (and grants persmissions to do so), using infrastructure code within application codebase and opening potential security flaws ...
Thanks !
I have an existing kind k8s cluster with a bunch of running services and nginx-ingress setup and I would like to add knative to it.
Is there a way of doing this with nginx-ingress, seems like networking for knative is a bit more complex than a normal service installation.
Knative needs more capabilities out of the HTTP routing layer than are exposed via the Kubernetes Ingress resource. (Percentage splits and header rewrite are two of the big ones.)
Unfortunately, so one has written an adapter from Knative's "KIngress" implementation to the nginx ingress. In the future, the gateway API (aka "Ingress V2") may provide these capabilities; in the meantime, you'll need to install one of the other network adapters and ingress implementations. Kourier provides the smallest implementation, while Contour also provides an Ingress implementation if you want to switch from nginx entirely.
I am learning ingress and ingress controller. So I understand that I have to do the following tasks-
ingress controller deployment
create service account
ingress controller service nodeport expose
create ingress resources to attach services.
Now my question is why we need a service account?? And what role should I attach with that service account and how do I use that service account?
What you are asking is very generic and may change a lot, depending on which is your setup (microk8s, minikube, bare-metal and so on) there are a lot of considerations to make.
The Nginx Ingress Controller installation guide for example can help you see how much things change between different environments.
It is also a good idea to simply use the installation resources provided in such guides instead of creating your own resources.. simply because the guide is more complete and ready-to-use basically.
With this said, the reason for the ServiceAccount is that the Ingress Controller Pod needs to be able to access Kubernetes API. Specifically, it needs to watch for resources such as Ingress (obviously), Services, Pods, Endpoints and more.
Imagine that the user (you) creates (or updates, or delete) a Ingress resource, the Ingress Controller needs to notice, parse it, understand what is declared and configure itself to serve the required Services at the configured domains and so on. Similarly, if something changes in the cluster, it may change how the controller needs to serve things.
For example, if you take a look at the Bare-Metal raw YAML definitions of the Nginx Ingress Controller and search for Role you will notice what it needs and also how it is attached to the other resources.
Lastly, serving the Ingress Controller from a NodePort service may not be the most resilient way to do it, it's okay for tests and such, but usually what you want is to have the Ingress Controller Pod to be served at a Load Balanced IP address, so that it is resilient to a single node of your cluster going down.
The Nginx Controller Bare-Metal considerations explains it very well.
I'm trying to figure out which tools from GKE stack I should apply to my use case which is a dynamic deployment of stateful application with dynamic HTTP endpoints.
Stateful in my case means that I don't want any replicas and load-balancing (because the app doesn't scale horizontally at all). I understand though that in k8s/gke nomenclature I'm still going to be using a 'load-balancer' even though it'll act as a reverse proxy and not actually balance any load.
The use case is as follows. I have some web app where I can request for a 'new instance' and in return I get a dynamically generated url (e.g. http://random-uuid-1.acme.io). This domain should point to a newly spawned, single instance of a container (Pod) hosting some web application. Again, if I request another 'new instance', I'll get a http://random-uuid-2.acme.io which will point to another (separate), newly spawned instance of the same app.
So far I figured out following setup. Every time I request a 'new instance' I do the following:
create a new Pod with dynamic name app-${uuid} that exposes HTTP port
create a new Service with NodePort that "exposes" the Pod's HTTP port to the Cluster
create or update (if exists) Ingress by adding a new http rule where I specify that domain X should point at NodePort X
The Ingress mentioned above uses a LoadBalancer as its controller, which is automated process in GKE.
A few issues that I've already encountered which you might be able to help me out with:
While Pod and NodePort are separate resources per each app, Ingress is shared. I am thus not able to just create/delete a resource but I'm also forced to keep track of what has been added to the Ingress to be then able to append/delete from the yaml which is definitely not the way to do that (i.e. editing yamls). Instead I'd probably want to have something like an Ingress to monitor a specific namespace and create rules automatically based on Pod labels. Say I have 3 pods with labels, app-1, app-2 and app-3 and I want Ingress to automatically monitor all Pods in my namespace and create rules based on the labels of these pods (i.e. app-1.acme.io -> reverse proxy to Pod app-1).
Updating Ingress with a new HTTP rule takes around a minute to allow traffic into the Pod, until then I keep getting 404 even though both Ingress and LoadBalancer look as 'ready'. I can't figure out what I should watch/wait for to get a clear message that the Ingress Controller is ready for accepting traffic for newly spawned app.
What would be the good practice of managing such cluster where you can't strictly define Pods/Services manifests because you are creating them dynamically (with different names, endpoints or rules). You surely don't want to create bunch of yaml-s for every application you spawn to maintain. I would imagine something similar to consul templates in case of Consul but for k8s?
I participated in a similar project and our decision was to use Kubernetes Client Library to spawn instances. The instances were managed by a simple web application, which took some customisation parameters, saved them into its database, then created an instance. Because of the database, there was no problem with keeping track of what have been created so far. By querying the database we were able to tell if such deployment was already created or update/delete any associated resources.
Each instance consisted of:
a deployment (single or multi-replica, depending on the instance);
a ClusterIp service (no reason to reserve machine port with NodePort);
an ingress object for shared ingress controller;
and some shared configMaps.
And we also used external DNS and cert manager, one to manage DNS records and another to issue SSL certificates for the ingress. With this setup it took about 10 minutes to deploy a new instance. The pod and ingress controller were ready in seconds but we had to wait for the certificate and it's readiness depended on whether issuer's DNS got our new record. This problem might be avoided by using a wildcard domain but we had to use many different domains so it wasn't an option in our case.
Other than that you might consider writing a Helm chart and make use of helm list command to find existing instances and manage them. Though, this is a rather 'manual' solution. If you want this functionality to be a part of your application - better use a client library for Kubernetes.
I am currently switching from Service Fabric to Kubernetes and was wondering how to do custom and more complex load balancing.
So far I already read about Kubernetes offering "Services" which do load balancing for pods hidden behind them, but this is only available in more plain ways.
What I want to rewrite right now looks like the following in Service Fabric:
I have this interface:
public interface IEndpointSelector
{
int HashableIdentifier { get; }
}
A context keeping track of the account in my ASP.Net application e.g. inherits this. Then, I wrote some code which would as of now do service discovery through the service fabric cluster API and keep track of all services, updating them when any instances die or are being respawned.
Then, based on the deterministic nature of this identifier (due to the context being cached etc.) and given multiple replicas of the target service of a frontend -> backend call, I can reliably route traffic for a certain account to a certain endpoint instance.
Now, how would I go about doing this in Kubernetes?
As I already mentioned, I found "Services", but it seems like their load balancing does not support custom logic and is rather only useful when working with stateless instances.
Is there also a way to have service discovery in Kubernetes which I could use here to replace my existing code at some points?
StatefulSet
StatefulSet is a building block for stateful workload on Kubernetes with certain guarantees.
Stable and unique network identity
StatefulSet Pods have a unique identity that is comprised of an ordinal, a stable network identity, and stable storage.
As an example, if your StatefulSet has the name sharded-svc
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: sharded-svc
And you have e.g. 3 replicas, those will be named by <name>-<ordinal> where ordinal starts from 0 up to replicas-1.
The name of your pods will be:
sharded-svc-0
sharded-svc-1
sharded-svc-2
and those pods can be reached with a dns-name:
sharded-svc-0.sharded-svc.your-namespace.svc.cluster.local
sharded-svc-1.sharded-svc.your-namespace.svc.cluster.local
sharded-svc-2.sharded-svc.your-namespace.svc.cluster.local
given that your Headless Service is named sharded-svc and you deploy it in namespace your-namespace.
Sharding or Partitioning
given multiple replicas of the target service of a frontend -> backend call, I can reliably route traffic for a certain account to a certain endpoint instance.
What you describe here is that your stateful service is what is called sharded or partitioned. This does not come out of the box from Kubernetes, but you have all the needed building blocks for this kind of service. It may happen that it exists an 3rd party service providing this feature that you can deploy, or it can be developed.
Sharding Proxy
You can create a service sharding-proxy consisting of one of more pods (possibly from Deployment since it can be stateless). This app need to watch the pods/service/endpoints in your sharded-svc to know where it can route traffic. This can be developed using client-go or other alternatives.
This service implements the logic you want in your sharding, e.g. account-nr modulus 3 is routed to the corresponding pod ordinal
Update: There are 3rd party proxies with sharding functionallity, e.g. Weaver Proxy
Sharding request based on headers/path/body fields
Recommended reading: Weaver: Sharding with simplicity
Consuming sharded service
To consume your sharded service, the clients send request to your sharding-proxy that then apply your routing or sharding logic (e.g. request with account-nr modulus 3 is routed to the corresponding pod ordinal) and forward the request to the replica of sharded-svc that match your logic.
Alternative Solutions
Directory Service: It is probably easier to implement sharded-proxy as a directory service but it depends on your requirements. The clients can ask your directory service to what statefulSet replica should I send account-nr X and your serice reply with e.g. sharded-svc-2
Routing logic in client: The probably most easy solution is to have your routing logic in the client, and let this logic calculate to what statefulSet replica to send the request.
Services generally run the proxy in kernel space for performance reasons so writing custom code is difficult. Cillium does allow writing eBPF programs for some network features but I don't think service routing is one of them. So that pretty much means working with a userspace proxy instead. If your service is HTTP-based, you could look at some of the existing Ingress controllers to see if any are close enough or allow you to write your own custom session routing logic. Otherwise you would have to write a daemon yourself to handle it.