we're currently trying out traefik and consider using it as ingress controller for our internal kubernetes cluster.
Now I wonder if it is possible to use traefik to loadbalance the kube-apiserver? We have a HA setup with 3 masters.
How would I proceed here?
Basically I just want to loadbalance the API requests from all nodes in the cluster between the 3 masters.
Should I just run traefik outside the cluster?
I'm trying to wrap my head around this... I'm having a hard time to understand how this could work together with traefik as ingress controller.
Thanks for any input, much appreciated!
One way to achieve this is to use the file provider and create a static setup pointing at your API server nodes; something like this (untested)
[file]
[backends]
[backends.backend1]
[backends.backend1.servers]
[backends.backend1.servers.server1]
url = "http://apiserver1:80"
weight = 1
[backends.backend1.servers.server2]
url = "http://apiserver2:80"
weight = 1
[backends.backend1.servers.server3]
url = "http://apiserver3:80"
weight = 1
[frontends]
[frontends.frontend1]
entryPoints = ["http"]
backend = "backend1"
passHostHeader = true
[frontends.frontend1.routes]
[frontends.frontend1.routes.route1]
rule = "Host:apiserver"
(This assumes a simple HTTP-only setup; HTTPS would need some extra setup.)
When Traefik is given this piece of configuration (and whatever else you need to do either via the TOML file or CLI parameters), it will round-robin requests with an apiserver Host header across the three nodes.
Another at least potential option is to create a Service object capturing your API server nodes and another Ingress object referencing that Service and mapping the desirable URL path and host to your API server. This would give you more flexibility as the Service should adjust to changes to your API server automatically, which might be interesting when things like rolling upgrades come into play. One aggravating point though might be that Traefik needs to speak to the API server to process Ingresses and Services (and Endpoints, for that matter), which it cannot if the API server is unavailable. You might need some kind of HA setup or be willing to sustain a certain non-availability. (FWIW, Traefik should recover from temporary downtimes on its own.)
Whether you want run Traefik in-cluster or out-of-cluster is up to you. The former is definitely easier to setup if you want to process API objects as you won't have to pass in API server configuration parameters, though the same restrictions about API server connectivity needs apply if you want to go down the Ingress/Service route. With the file provider approach, you don't need to worry about that -- it is perfectly possible to run Traefik inside Kubernetes without using the Kubernetes provider.
Related
My Kubernetes cluster uses a replicaSet to run N similar pods. Each pod can only handles one websocket connection du to resource limitations. My Kubernetes uses a nginx ingress controller.
Is there any way to make nginx dispatch only one incoming websocket connection per pod and in case of lack of available pods, refuses the incoming connection ?
I'm not super familiar with the Kubernetes Nginx ingress setup, but assuming it exposes some of the Nginx configuration options for setting up groups of servers, in the server function there's a parameter called max_conns that will let you limit the number of connections to a given server. Assuming there's a mapping in the ingress controller, it should be possible to set max_conns=1 for each server that's getting created and added to the Nginx configuration under the hood.
http://nginx.org/en/docs/http/ngx_http_upstream_module.html#server
Edit: a little cursory research and it looks like this is indeed possible. It looks like you can specify this in a ConfigMap as nginx.org/max-conns according to the master list of parameters here: https://github.com/nginxinc/kubernetes-ingress/blob/master/docs/configmap-and-annotations.md
You could use the readinessProbe with a periodSecond relatively low and, obvously, {success,failure}Threshold set to 1 in order to release or not the Pod as fast as possible.
Basically, you could set up a script or a simple HTTP endpoint that returns a failed status code in case a connection has been established: so the Pod endpoint will be removed from the Service endpoints list and will not be selected by the Ingress Controller.
Just keep in mind that this solution could be affected by race-conditions but it's the most simple one: a better solution could be using a Service Mesh but means additional complexity.
In my kubernetes cluster, there is a lot of inter-service communication. I have also enabled horizontal pod autoscaler and we use service resource for all these services.
I need to understand how does the service resource loadbalance the request across the pods.
I did read about SessionAffinity but it only supports ClientIP, if you are not using an Ingress resource.
Want to know if the service can loadbalance based on the load (in terms of cpu, memory, etc.) on a particular pod. In short, does sessionAffinity config support anything other than ClientIP. I dont want to bring in an Ingress resource, as these are not external facing requests, these are inter-service communication.
Thanks in advance.
In short, does sessionAffinity config support anything other than ClientIP
No, it does not. See the model definition here for v1.SessionAffinityConfig: Nor will it; I already feel that sessionAffinity is out of a Service's scope, I'm surprised it exists at all.
You're going to want to use some kind of layer in front of your service if you want to have more control of your connections. There are plenty of Service Meshes that might solve your problem (see isito or Linkerd )
You could also roll your own solution with nginx and send your requests for that service to the nginx pod.
I'm investigating using Traefik as a reverse proxy within a Service Fabric cluster running Dockerized microservices. The official way of running Traefik within Service Fabric is to use the Service Fabric provider. Running Traefik within a Docker container is not recommended according to the Github readme because you can't reach the Service Fabric API through localhost:19080, but instead have to reach it through its public IP. This requires you to install SSL certs to securely talk to the API, which is a bit of a hassle.
I'm curious if there's any advantage using the Traefix Service Fabric provider (which requires complex setup) rather than just running Traefix in a container running with the File provider. Provided your services had ServiceDnsName attributes in the ApplicationManifest, thus allowing Service Fabric DNS to find them, it seems like a much simpler approach. The Traefik configuration would look something like:
[frontends]
[frontends.api]
backend = "api"
passHostHeader = true
[frontends.api.routes.forwarder]
rule = "PathPrefixStrip: /api/"
[frontends.someservice]
backend = "someservice"
passHostHeader = true
[frontends.someservice.routes.forwarder]
rule = "PathPrefixStrip: /SomeService/"
[backends]
[backends.api]
[backends.api.servers.endpoint]
url = "http://Api:11080"
[backends.someservice]
[backends.someservice.servers.endpoint]
url = "http://SomeService:12080"
You'd map port 80 to your Traefix service, which would then dispatch the HTTP call to the appropriate internal service based on the URL prefix.
Advantages:
No need to talk to the Service Fabric API, which is somewhat hacky to do from within a container.
Possibly more secure; Everything is internal, no need to worry about installing certificates
Disadvantages:
Your service routing is now tied to a TOML file within a Docker container rather than integrated into service and application manifest files. No way to modify this without redeploying that container.
I don't believe this would work unless all services were running in a Container (Though I believe if you enabled the reserve proxy, you can resolve the service by name using http://localhost:19081/AppName/ServiceName instead)
To me, it seems like a cleaner approach provided you're not changing and adding services all the time. Usually, the stuff stays fairly static though.
Are there any gotchas that I'm not considering, or any strong argument against doing this?
I will add my two cents to your considerations:
Is a preference between Dynamic and Static configuration.
The benefit is that it will reload with all the new configurations every time a service comes up with Traefik configuration. Today you say "it won't change", but in a few months, weeks or perhaps days, you might face a new requirement and you have to update the file, if the system evolves quickly, like on Microservices solution, you soon will face a hard work of updating this file by hand.
If you are sure it will hardly change, don't be limited by the Service Fabric Configuration or File, Traefik can get the configuration from REST , ETCD, DynamoDB and many other configuration providers to load the rules.
For the File approach, you can create the file in an Azure FileShare and mount as a volume in the container, so you don't need to rebuild the container to take effect.
Then you can just update the file and:
Restart the container to reload the file, or.
An even better approach is using the file.watch setting you don't need to restart the container and it will watch for any changes in the file.
Is there a way to make a kubernetes service available under multiple paths via traefik ingress, but sending requests to the backend only with one path?
e.g. https://example.com/mainpath/api and https://example.com/otherpath/api should both send requests with /mainpath/api to the backend
Sure, use two different ingresses, one relatively normal, the other with the traefik.ingress.kubernetes.io/rewrite-target: /mainpath/api annotation. Your backend will have to check the X-Replaced-Path header if it cares to be able to tell the difference.
When using on premise (running on my own) api gateway like Kong, should it be run in a node as 1 withing the main kubernetes cluster or should it be ran as separate kubernetes cluster?
Unless you have an amazing reason to do otherwise: run Kong within the cluster. Pretty much the last thing you'd want is for all API requests to bomb because of a severed connection between cluster-A and cluster-B, not to mention the horrible latency as requests hop from one layer of abstraction to another.
Taking a page from the nginx Ingress controller, you also have the opportunity to use the Endpoint API to bypass the iptables-based Service machinery, saving even more latency and system resources -- a trick that would be almost impossible with a multi-cluster configuration.
It is my recollection there are even Kong-based Ingress controllers, which could save you even more heartache if their featureset and your needs align