We have a service with multiple replicas which works with storage without transactions and blocking approaches. So we need somehow to synchronize concurrent requests between multiple instances by some "sharding" key. Right now we host this service in Kubernetes environment as a ReplicaSet.
Don't you know any simple out-of-the-box approaches on how to do this to not implement it from scratch?
Here are several of our ideas on how to do this:
Deploy the service as a StatefulSet and implement some proxy API which will route traffic to the specific pod in this StatefulSet by sharding key from the HTTP request. In this scenario, all requests which should be synchronized will be handled by one instance and it wouldn't be a problem to handle this case.
Deploy the service as a StatefulSet and implement some custom logic in the same service to re-route traffic to the specific instance (or process on this exact instance). As I understand it's not possible to have abstract implementation and it would work only in Kubernetes environment.
Somehow expose each pod IP outside the cluster and implement routing logic on the client-side.
Just implement synchronization between instances through some third-party service like Redis.
I would like to try to route traffic to the specific pods. If you know standard approaches how to handle this case I'll be much appreciated.
Thank you a lot in advance!
Another approach would be to put a messaging queue (like Kafka and RabbitMq) in front of your service.
Then your pods will subscribe to the MQ topic/stream. The pod will decide if it should process the message or not.
Also, try looking into service meshes like Istio or Linkerd.
They might have an OOTB solution for your use-case, although I wasn't able to find one.
Remember that Network Policy is not traffic routing !
Pods are intended to be stateless and indistinguishable from one another, pod-networking.
I recommend to Istio. It has special component which is responsible or routing- Envoy. It is a high-performance proxy developed in C++ to mediate all inbound and outbound traffic for all services in the service mesh.
Useful article: istio-envoy-proxy.
Istio documentation: istio-documentation.
Useful Istio explaination https://www.youtube.com/watch?v=e2kowI0fAz0.
But you should be able to create a Deployment per customer group, and a Service per Deployment. The Ingress nginx should be able to be told to map incoming requests by whatever attributes are relevant to specific customer group Services.
Other solution is to use kube-router.
Kube-router can be run as an agent or a Pod (via DaemonSet) on each node and leverages standard Linux technologies iptables, ipvs/lvs, ipset, iproute2.
Kube-router uses IPVS/LVS technology built in Linux to provide L4 load balancing. Each ClusterIP, NodePort, and LoadBalancer Kubernetes Service type is configured as an IPVS virtual service. Each Service Endpoint is configured as real server to the virtual service. The standard ipvsadm tool can be used to verify the configuration and monitor the active connections.
How it works: service-proxy.
Related
I'm learning Kubernetes at the moment and just had a question I'd like to have clarified regarding exposing Services and making Pods accessible to the public internet.
Lets say I have an Java Spring boot application which has an embedded Tomcat server using JSP, MySQL Pod and Memcached (All on separate Pods), and I'd like to expose them as a Service making them publicly available.
I'm confused as to which type of Service each of these Pods would need , and also why. I'm aware that of Ingress and using a single Load Balancer to route requests from Services instead of multiple Load Balancers, but the actual Service type is what I'm finding hard to understand based on the what work the Pod needs to do.
Answering which Service type do you need: it's always ClusterIP.
LoadBalancers and NodePort are reserved for very specific use cases. One requiring to be integrated with a cloud (provisioning loadbalancers), the other requiring your kubernetes nodes being exposed to external clients, allowing connections to non-default ports.
When you don't know or you're not sure: just assume you can't use NodePort or LoadBalancers. As a cluster end-user, developer or Kubernetes beginner: ClusterIP is the only Service type you need.
Exposing your application to clients outside of your SDN, you want to use Ingresses. As again, while LoadBalancers or NodePorts might be suitable technical solutions on paper, they usually aren't in practice -- and when they are, there are security aspect to consider: better dealt with by your cluster administrator.
I've an existing microservice architecture that uses Netflix Eureka and zuul services,
I've deployed a pod that successfully registers on the discover server but when I hit the API it gives a timeout, what I can think is that while registering on the Discovery server the container IP is given because of which it is not accessible.
Is there a way to either map the correct address or redirect the call to the proper URL looking for a easy way, as this needs to be done on multiple services
I think you should be rethinking your design in Kubernetes way! Your Eureka(service discovery), Zuul server (API gateway/ Loadbalancer) are really extra services that you really don't need in the Kubernetes platform.
For Service discovery and load-balancing, you can use Services in Kubernetes.
From Kubernetes documentation:
An abstract way to expose an application running on a set of Pods as a
network service. With Kubernetes, you don't need to modify your
application to use an unfamiliar service discovery mechanism. Kubernetes gives Pods their own IP addresses and a single DNS name for a set of Pods and can load-balance across them.
And for API gateway, you can think about Ingress in Kubernetes.
There are different implementations for Ingress Controllers for Kubernetes. I'm using Ambassador API gateway implementation.
https://kubernetes.io/docs/concepts/services-networking/ingress-controllers/
I've been looking into Kubernetes networking, more specifically, how to serve HTTPS users the most efficient.
I was watching this talk: https://www.youtube.com/watch?v=0Omvgd7Hg1I and from 22:18 he explains what the problem is with a load balancer that is not pod aware. Now, how they solve this in kubernetes is by letting the nodes also act as a 'router' and letting the node pass the request on to another node. (explained at 22:46). This does not seem very efficient, but when looking around SoundCloud (https://developers.soundcloud.com/blog/how-soundcloud-uses-haproxy-with-kubernetes-for-user-facing-traffic) actually seems to do something similar to this but with NodePorts. They say that the overhead costs less than creating a better load balancer.
From what I have read an option might be using an ingress controller. Making sure that there is not more than one ingress controller per node, and routing the traffic to the specific nodes that have an ingress controller. That way there will not be any traffic re-routing needed. However, this does add another layer of routing.
This information is all from 2017, so my question is: is there any pod aware load balancer out there, or is there some other method that does not involve sending the http request and response over the network twice?
Thank you in advance,
Hendrik
EDIT:
A bit more information about my use case:
There is a bare-metal setup with kubernetes. The firewall load balances the incomming data between two HAProxy instances. These HAProxy instances do ssl termination and forward the traffic to a few sites. This includes an exchange setup, a few internal IIS sites and a nginx server for a static web app. The idea is to transform the app servers into kubernetes.
Now my main problem is how to get the requests from HAProxy into kubernetes. I see a few options:
Use the SoundCloud setup. The infrastructure could stay almost the same, the HAProxy server can still operate the way they do now.
I could use an ingress controller on EACH node in the kubernetes cluster and have the firewall load balance between the nodes. I believe it is possible to forward traffic from the ingress controller to server outside the cluster, e.g. exchange.
Some magic load balancer that I do not know about that is pod aware and able to operate outside of the kubernetes cluster.
Option 1 and 2 are relatively simple and quite close in how they work, but they do come with a performance penalty. This is the case when the node that the requests gets forwarded to by the firewall does not have the required pod running, or if another pod is doing less work. The request will get forwarded to another node, thus, using the network twice.
Is this just the price you pay when using Kubernetes, or is there something that I am missing?
How traffic heads to pods depend on whether a managed cluster is used.
Almost all cloud providers can forward traffic in a cloud-native way in their managed K8s clusters. First, you can a managed cluster with some special network settings (e.g. vpc-native cluster of GKE). Then, the only thing you need to do is to create a LoadBalancer typed Service to expose your workload. You can also create Ingresses for your L7 workloads, they are going to be handled by provided IngressControllers (e.g. ALB of AWS).
In an on-premise cluster without any cloud provider(OpenStack or vSphere), the only way to expose workloads is NodePort typed Service. It doesn't mean you can't improve it.
If your cluster is behind reverse proxies (the SoundCloud case), setting externalTrafficPolicy: Local to Services could break traffic forwarding among work nodes. When traffic received through NodePorts, they are forwarded to local Pods or dropped if Pods reside on other nodes. Reserve proxy will mark these NodePort as unhealthy in the backend health check and reject to forward traffic to them. Another choice is to use topology-aware service routing. In this case, local Pods have priorities and traffic is still forwarded between node when no local Pods matched.
For IngressController in on-prem clusters, it is a little different. You may have some work nodes that have EIP or public IP. To expose HTTP(S) services, an IngressController usually deployed on those work nodes through DaemeaSet and HostNetwork such that clients access the IngressController via the well-known ports and EIP of nodes. These work nodes regularly don't accept other workloads (e.g. infra node in OpenShift) and one more forward on the Pod network is needed. You can also deploy the IngressController on all work nodes as well as other workloads, so traffic could be forwarded to a closer Pod if the IngressController supports topology-aware service routing although it can now.
Hope it helps!
I am currently switching from Service Fabric to Kubernetes and was wondering how to do custom and more complex load balancing.
So far I already read about Kubernetes offering "Services" which do load balancing for pods hidden behind them, but this is only available in more plain ways.
What I want to rewrite right now looks like the following in Service Fabric:
I have this interface:
public interface IEndpointSelector
{
int HashableIdentifier { get; }
}
A context keeping track of the account in my ASP.Net application e.g. inherits this. Then, I wrote some code which would as of now do service discovery through the service fabric cluster API and keep track of all services, updating them when any instances die or are being respawned.
Then, based on the deterministic nature of this identifier (due to the context being cached etc.) and given multiple replicas of the target service of a frontend -> backend call, I can reliably route traffic for a certain account to a certain endpoint instance.
Now, how would I go about doing this in Kubernetes?
As I already mentioned, I found "Services", but it seems like their load balancing does not support custom logic and is rather only useful when working with stateless instances.
Is there also a way to have service discovery in Kubernetes which I could use here to replace my existing code at some points?
StatefulSet
StatefulSet is a building block for stateful workload on Kubernetes with certain guarantees.
Stable and unique network identity
StatefulSet Pods have a unique identity that is comprised of an ordinal, a stable network identity, and stable storage.
As an example, if your StatefulSet has the name sharded-svc
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: sharded-svc
And you have e.g. 3 replicas, those will be named by <name>-<ordinal> where ordinal starts from 0 up to replicas-1.
The name of your pods will be:
sharded-svc-0
sharded-svc-1
sharded-svc-2
and those pods can be reached with a dns-name:
sharded-svc-0.sharded-svc.your-namespace.svc.cluster.local
sharded-svc-1.sharded-svc.your-namespace.svc.cluster.local
sharded-svc-2.sharded-svc.your-namespace.svc.cluster.local
given that your Headless Service is named sharded-svc and you deploy it in namespace your-namespace.
Sharding or Partitioning
given multiple replicas of the target service of a frontend -> backend call, I can reliably route traffic for a certain account to a certain endpoint instance.
What you describe here is that your stateful service is what is called sharded or partitioned. This does not come out of the box from Kubernetes, but you have all the needed building blocks for this kind of service. It may happen that it exists an 3rd party service providing this feature that you can deploy, or it can be developed.
Sharding Proxy
You can create a service sharding-proxy consisting of one of more pods (possibly from Deployment since it can be stateless). This app need to watch the pods/service/endpoints in your sharded-svc to know where it can route traffic. This can be developed using client-go or other alternatives.
This service implements the logic you want in your sharding, e.g. account-nr modulus 3 is routed to the corresponding pod ordinal
Update: There are 3rd party proxies with sharding functionallity, e.g. Weaver Proxy
Sharding request based on headers/path/body fields
Recommended reading: Weaver: Sharding with simplicity
Consuming sharded service
To consume your sharded service, the clients send request to your sharding-proxy that then apply your routing or sharding logic (e.g. request with account-nr modulus 3 is routed to the corresponding pod ordinal) and forward the request to the replica of sharded-svc that match your logic.
Alternative Solutions
Directory Service: It is probably easier to implement sharded-proxy as a directory service but it depends on your requirements. The clients can ask your directory service to what statefulSet replica should I send account-nr X and your serice reply with e.g. sharded-svc-2
Routing logic in client: The probably most easy solution is to have your routing logic in the client, and let this logic calculate to what statefulSet replica to send the request.
Services generally run the proxy in kernel space for performance reasons so writing custom code is difficult. Cillium does allow writing eBPF programs for some network features but I don't think service routing is one of them. So that pretty much means working with a userspace proxy instead. If your service is HTTP-based, you could look at some of the existing Ingress controllers to see if any are close enough or allow you to write your own custom session routing logic. Otherwise you would have to write a daemon yourself to handle it.
We have an old service discovery system that requires processes to register its ip:port during startup. On a kubernetes cluster, we exposed a service that enables NodePort. The processes within container can register to the old system with their Pod Ip:port + HostIp. For the clients within the same kubernetes cluster, they should be able to connect to the right process via specific Pod Ip:port. For an external client, it knows the HostIp+NodePort and the specific Pod Ip:port, is there an efficient way to route the client’s request to the specific Pod? Running a proxy on each node to route the traffic (nodeport -> pod) seems inefficient due to additional proxy layer.
I guess you mean you don't want to add a Service of type NodePort as for your case that seems like an additional proxy layer. I can see how it is an additional layer in your case. Typically Kubernetes would be doing the orchestration and the Service would be part of the service-discovery mechanism. It sounds like you could use hostPort. But if you do go this route you should be aware it's not suggested practice as Kubernetes is intended for orchestration.