I have an app with 3 nodes running with a service, which uses zookeeper to elect a leader. And I want the requests to the service to be redirected to a certain pod, depending if it's leader or not. I have a http method in my app that returns if it's leader or not (if this helps in any way).
I'm not aware of a k8s feature for doing this.
What I would do:
write a little application which looks for the endpoints of the existing service, searches the current leader, and creates/updates a 2nd service without selector (https://kubernetes.io/docs/concepts/services-networking/service/#services-without-selectors) with an endpoint pointing to the leader. Then you can use that new service for your dashboard.
To do that, use a StatefulSet, for example, following the example here.
Related
I have my main application which has its own unique state, let's call it Application A.
This application A starts a few processes which does some parsing work and then it collects it and should send it to a database server outside of the Kuberentes cluster.
I would like to run a few copies of this application A in different pods. however, each instance is unique and cannot be replaced as it has its own state. it means that each client has to talk only with the same instance it started the communication with http requests.
How can it be done in Kubernetes?
do I need to define StatefulSet component?
how do I manage that each client (from outside the cluster) will talk every time with the same instance he started communication on the same object id ? for example to get status on that object.
in case the pod die I don't want to recover. is that possible?
1: yes, sort of
2: not necessarily, but might simplify some things
3: if you use ingress, you can use different methods to maintain backend affinity ie. cookie based, source IP based etc. (nginx example: https://kubernetes.github.io/ingress-nginx/examples/affinity/cookie/)
4: you might want to set restartPolicy to Never
With all that said, this really sounds like a bad idea. You should either allow shared state (ie. redis), or statefulset with ability to restart with the same state loaded from local storage. You need to remember that even with the most optimal setup things like this can break (ie. switch to different pod when a backing pod went down, node rescheduling due to cluster scaling etc.)
For the number 4 question. You only need to set up the container-restart-policy. I used this flag to create a pod with this feature: --restart=Never
IMHO, It is not a Kubernetes problem. You could have this scenario in other environments. The idea is to use sticky sessions to have an affinity for all your request. You probably need to search for this setup in your ingress controller documentation. E.g Nginx Ingress
In Kubernetes, is it possible to have 2 services for a single deployment, one which is "standard" and proxies in front of all ready pods, and a second service which sends traffic only the elected leader? If so how? I am using client-go for leader election. These are layer 4 services.
I know that a service can use labels for a selector, but client-go uses an annotation to mark the leader. Using a service without selectors and creating/removing an endpoint in the leader callbacks seems hacky/buggy. Thank you.
In Kubernetes, is it possible to have 2 services for a single deployment, one which is "standard" and proxies in front of all ready pods, and a second service that sends traffic only the elected leader?
Yes, but it seems a bit hacky. The way services work is like this:
Service -> Service labels select endpoints -> Endpoint -> Endpoint labels select pods -> Pods (PodIP)
So you could have your regular "Service" that points to all the pods on your Deployment or StatefulSet which automatically provisions all the Endpoints.
You could also have another set of "Headless Service" + "Endpoint" each manually created individually that you make their labels match with each other and then have that Endpoint manually match with the label of the pod of your choice.
Now with respect to client-go/leaderelection. It seems like it works using an Endpoint or ConfigMap lock for the leader (The example shows a ConfigMap lock). But, looks like you want to use the Endpoint lock. So this package doesn't work with services or labels, it looks like it just works on Endpoint resources. So essentially if you have 3 nodes and want to find the leader you would have to use 3 manually created Endpoint resources. The one that is the leader will always have the annotation.
Now how do you tie it to 2) above? As your client elects or selects the leader then you also have to change the endpoint labels so that they match your manually created headless service. (It could be done in your code too)
You could also just elect to just use Endpoints instead 2) above (No headless service) and have client-go/leaderelection talk directly to the endpoints.
Another option is to take advantage of StatefulSets and its required headless service. So that service will resolve to the IP addresses of all the replicas in your quorum-based cluster. The leader election would be up to the client package (client-go doesn't seem to support this) which is pretty the case for most quorum based applications (K8s, Zookeeper, Kafka, Etcd, etc, etc); the client is the one that finds who the leader is.
✌️
I have 2 Deployment - A(1 replica) and B(4 replica)
I have scheduled job in the POD A and on successful completion it hits endpoint present in one of the Pods from Deployment B through the service for Deployment B.
Is there a way I can hit all the endpoints of 4 PODS from Deployment B on successful completion of job?
Ideally one of the pod is notified!! But is this possible as I don't want to use pub-sub for this.
Is there a way I can hit all the endpoints of 4 PODS from Deployment B on successful completion of job?
But is this possible as I don't want to use pub-sub for this.
As you say, a pub-sub solution is best for this problem. But you don't want to use it.
Use stable network identity for Service B
To solve this without pub-sub, you need a stable network-identity for the pods in Deployment B. To get this, you need to change to a StatefulSet for your service B.
StatefulSets are valuable for applications that require one or more of the following.
Stable, unique network identifiers.
When B is deployed with a StatefulSet, your job or other applications can reach your pods of B, with a stable network identity that is the same for every version of service B that you deploy. Remember that you also need to deploy a Headless Service for your pods.
Scatter pattern: You can have an application aware (e.g. aware of number of pods of Service B) proxy, possibly as a sidecar. Your job sends the request to this proxy. The proxy then sends a request to all your replicas. As described in Designing Distributed Systems: Patterns and Paradigms
Pub-Sub or Request-Reply
If using pub-sub, the job only publish an event. Each pod in B is responsible to subscribe.
In a request-reply solution, the job or a proxy is responsible for watching what pods exists (unless it is a fixed number of pods) in service B, in addition it need to send request to all, if requests fails to any pod (it will happen on deployments sometime) it is responsibly to retry the request to those pods.
So, yes, it is a much more complicated problem in a request-reply way.
Kubernetes service is an abstraction to provide service discovery and load balancing. So if you are using service your request will be sent to one of the backend pods.
To achieve what you want I suggest you create 4 different services each having only one backend pod or use a message queue such as rabbitmq between service A and service B.
You can use headless service. That way Kubernetes won't allocate separate IP address and instead will setup DNS record with IP addresses of all the pods. Then in your application just resolve the record and send notification to all endpoints. But really, this is ideal use case for pub-sub or service discovery system. DNS is too unreliable for this.
PUB-SUB is the best option here. I have similar use case and I am using pub-sub , which is in production for last 6 months.
I have a use case in which front-end application in sending a file to back-end service for processing. And at a time only one request can be processed by backend service pod. And if multiple request came service should autoscale and send that request to new Pod.
So I am finding a way in which I can spawn a new POD against each request and after completion of processing by backend service pod it will return the result to front-end service and destroy itself.
So that each pod only process a single request at a time.
I explore the HPA autoscaling but did not find any suitable way.
Open to use any custom metric server for that, even can use Jobs if they are able to fulfill the above scenario.
So if someone have knowledge or tackle the same use case then help me so that I can also try that solution.
Thanks in advance.
There's not really anything built-in for this that I can think of. You could create a service account for your app that has permissions to create pods, and then build the spawning behavior into your app code directly. If you can get metrics about which pods are available, you could use HPA with Prometheus to ensure there is always at least one unoccupied backend, but that depends on what kind of metrics your stuff exposes.
As already said, there is no built in way for doing this , you need to find custom way to achive this.
One solution can be use of service account and http request to api server to create back end pod as soon as your service is received by front end pod, check status of back end pod and once it is up, forward request to back end.
Second way i can think of using some temp storage ( db or hostpath volume ) and write cronejob in your master to poll that storage and depending on status spawn pod having job container.
I'm in the middle of creating an K8s app that doesn't expose any HTTP endpoints, is just a background app that pulls messages from a message bus and takes some action based on the incoming message. No other apps will interact directly with this background app, only thru posting messages into the message bus.
Scaling is a requirement and most likely will always need to run more than one replica. What is the recommended Service type in Kubernetes to handle this type of workload ?
No service required... just create a Deployment, which will result in a ReplicaSet, which will keep n replicas of your app running.