I have a use case in which front-end application in sending a file to back-end service for processing. And at a time only one request can be processed by backend service pod. And if multiple request came service should autoscale and send that request to new Pod.
So I am finding a way in which I can spawn a new POD against each request and after completion of processing by backend service pod it will return the result to front-end service and destroy itself.
So that each pod only process a single request at a time.
I explore the HPA autoscaling but did not find any suitable way.
Open to use any custom metric server for that, even can use Jobs if they are able to fulfill the above scenario.
So if someone have knowledge or tackle the same use case then help me so that I can also try that solution.
Thanks in advance.
There's not really anything built-in for this that I can think of. You could create a service account for your app that has permissions to create pods, and then build the spawning behavior into your app code directly. If you can get metrics about which pods are available, you could use HPA with Prometheus to ensure there is always at least one unoccupied backend, but that depends on what kind of metrics your stuff exposes.
As already said, there is no built in way for doing this , you need to find custom way to achive this.
One solution can be use of service account and http request to api server to create back end pod as soon as your service is received by front end pod, check status of back end pod and once it is up, forward request to back end.
Second way i can think of using some temp storage ( db or hostpath volume ) and write cronejob in your master to poll that storage and depending on status spawn pod having job container.
Related
I have my main application which has its own unique state, let's call it Application A.
This application A starts a few processes which does some parsing work and then it collects it and should send it to a database server outside of the Kuberentes cluster.
I would like to run a few copies of this application A in different pods. however, each instance is unique and cannot be replaced as it has its own state. it means that each client has to talk only with the same instance it started the communication with http requests.
How can it be done in Kubernetes?
do I need to define StatefulSet component?
how do I manage that each client (from outside the cluster) will talk every time with the same instance he started communication on the same object id ? for example to get status on that object.
in case the pod die I don't want to recover. is that possible?
1: yes, sort of
2: not necessarily, but might simplify some things
3: if you use ingress, you can use different methods to maintain backend affinity ie. cookie based, source IP based etc. (nginx example: https://kubernetes.github.io/ingress-nginx/examples/affinity/cookie/)
4: you might want to set restartPolicy to Never
With all that said, this really sounds like a bad idea. You should either allow shared state (ie. redis), or statefulset with ability to restart with the same state loaded from local storage. You need to remember that even with the most optimal setup things like this can break (ie. switch to different pod when a backing pod went down, node rescheduling due to cluster scaling etc.)
For the number 4 question. You only need to set up the container-restart-policy. I used this flag to create a pod with this feature: --restart=Never
IMHO, It is not a Kubernetes problem. You could have this scenario in other environments. The idea is to use sticky sessions to have an affinity for all your request. You probably need to search for this setup in your ingress controller documentation. E.g Nginx Ingress
If I have a service, pod etc. that I can query using a selector changes its ip address, is there a way to get notified?
For example, say my application needs to have a list of ip addresses of a pod, or the ip address of a service. Since the container can go down and get recreated using kubernetes, is there a way to get notified when the containers go down and get recreated so I can then use the kubernetes API to get the latest values for the ip addresses?
This would be required for things like primary and slave databases etc.
Does kubernetes have a webhook type functionality that can be used to notify my app?
You can use watch API operations.
To watch all Endpoints objects:
GET /api/v1/namespaces/{namespace}/endpoints?watch=true
To watch a specific Endpoints object:
GET /api/v1/watch/namespaces/{namespace}/endpoints/{name}?watch=true
This creates a hanging HTTP GET request and you get notified whenever any of the watched objects changes.
See the Kubernetes API reference.
There is nothing out of the box. You would have to write a controller which can watch and get notified for change of a resource in kubernetes cluster ETCD store. The endpoint controller within kubernetes is an example of that because it updates the Endpoints object whenever IP of a pod behind a service changes.
Another example is ingress controllers which watches for any change in the Endpoints which holds the Pod IPs behind a service.
The watch API in the standard kubernetes client libraries is pretty efficient and widely used.
Is there a reason why you would need to get IPs of Pods rather than deploying a StatefulSet and work with stable SRV records ?
StatefulSet looks a better approach to get stable identities. Master-Slaves topologies are typical use cases for StatefulSets.
https://kubernetes.io/docs/tutorials/stateful-application/basic-stateful-set/
I have 2 Deployment - A(1 replica) and B(4 replica)
I have scheduled job in the POD A and on successful completion it hits endpoint present in one of the Pods from Deployment B through the service for Deployment B.
Is there a way I can hit all the endpoints of 4 PODS from Deployment B on successful completion of job?
Ideally one of the pod is notified!! But is this possible as I don't want to use pub-sub for this.
Is there a way I can hit all the endpoints of 4 PODS from Deployment B on successful completion of job?
But is this possible as I don't want to use pub-sub for this.
As you say, a pub-sub solution is best for this problem. But you don't want to use it.
Use stable network identity for Service B
To solve this without pub-sub, you need a stable network-identity for the pods in Deployment B. To get this, you need to change to a StatefulSet for your service B.
StatefulSets are valuable for applications that require one or more of the following.
Stable, unique network identifiers.
When B is deployed with a StatefulSet, your job or other applications can reach your pods of B, with a stable network identity that is the same for every version of service B that you deploy. Remember that you also need to deploy a Headless Service for your pods.
Scatter pattern: You can have an application aware (e.g. aware of number of pods of Service B) proxy, possibly as a sidecar. Your job sends the request to this proxy. The proxy then sends a request to all your replicas. As described in Designing Distributed Systems: Patterns and Paradigms
Pub-Sub or Request-Reply
If using pub-sub, the job only publish an event. Each pod in B is responsible to subscribe.
In a request-reply solution, the job or a proxy is responsible for watching what pods exists (unless it is a fixed number of pods) in service B, in addition it need to send request to all, if requests fails to any pod (it will happen on deployments sometime) it is responsibly to retry the request to those pods.
So, yes, it is a much more complicated problem in a request-reply way.
Kubernetes service is an abstraction to provide service discovery and load balancing. So if you are using service your request will be sent to one of the backend pods.
To achieve what you want I suggest you create 4 different services each having only one backend pod or use a message queue such as rabbitmq between service A and service B.
You can use headless service. That way Kubernetes won't allocate separate IP address and instead will setup DNS record with IP addresses of all the pods. Then in your application just resolve the record and send notification to all endpoints. But really, this is ideal use case for pub-sub or service discovery system. DNS is too unreliable for this.
PUB-SUB is the best option here. I have similar use case and I am using pub-sub , which is in production for last 6 months.
I'd like to know if it's possible to get information from a pod when it's just created.
I'm spending time in developing a kubernetes controller process that reacts itself when a pod is created in cluster.
When a pod is just created, the service has to be able to get some basic information from pod. For example, ip, annotations...
I'd like to use a java service.
Any ideas?
You can use kubernetes
api-server
to get information regarding
endpoints (service)
. Kubernetes expose its API via REST so, you can use anything to communicate. Also, verify the results using 'kubectl' tool while development. For example, if you want to monitor pods related to service say, myservice.
kubectl get endpoints <myservice_pod> --watch
This will notify you with any activity with pods related to myservice. IMO, in java you have to use polling mechanism to mimic --watch functionality.
well, if you use kubernetes API client you can just watch on changes for all pods and then get their details (assuming you have granted RBAC auth)
Let's say that I have a service with 2 pods (replicas). Each pod contains just one container that is a REST API war that runs on a Tomcat. Moreover, each pod has imagePullPolicy: Always so when there is a new version of the image, it will pull it.
When the container starts, obviously, Tomcat takes some seconds to start. This will happen in both containers.
Is it possible that in a particular time my REST API is not available? I mean, is it possible that both Tomcat aren't started yet and a request fails?
Does K8S use health checking on a pod before attemp to re-run the another one? If so, I could perform an http health checking against my REST API endpoint. Is it the right way?
Thanks in advance. Any advice will be appreciated.
Is it possible that in a particular time my REST API is not available? I mean, is it possible that both Tomcat aren't started yet and a request fails?
Yes. You can prevent this from happening by making sure that at least one of your pods is ready to serve requests before creating the service (and then using rolling updates to avoid downtime as you upgrade you application).
Does K8S use health checking on a pod before attemp to re-run the another one? If so, I could perform an http health checking against my REST API endpoint. Is it the right way?
You should take a look at liveness and readiness probes. They are designed to capture the difference between a container running and the application inside the container being ready to serve requests.