From Kubernetes document, when readiness probe fails, it removes the Pod's IP address from the endpoints of all services that match the pod.
We are thinking about implementing SIGTERM handler to fail the health check and stop the pod from receiving future traffic. That's what we want, no more Inbound traffic. The question is, if the pod contains requests that depend on backend service which are not reside in the same pod, will the pod still be able to complete those outbound requests?
From the docs (emphasis mine):
Sometimes, applications are temporarily unable to serve traffic. For example, an application might need to load large data or configuration files during startup, or depend on external services after startup. In such cases, you don't want to kill the application, but you don't want to send it requests either. Kubernetes provides readiness probes to detect and mitigate these situations. A pod with containers reporting that they are not ready does not receive traffic through Kubernetes Services.
The pod can't be reached through Kubernetes services. You can still make outbound requests, and anyone using the pod name or IP directly will also still be able to reach it.
Related
I use Kubernetes in my project, specially HPA. So, every minute in project we started check-status request for checking if all microservices are available. Availability is defined by simple response from one of replicas (not all) each microservice.
But I have one moment related to HPA. When HPA automatically decides to remove some pods from cluster and my check-status request comes to server at the same time then very often occurs that my API-gateway service push it to deleted pod and doesn't get any response. It means that microservice is unavailable for our server.
My question is what is the best way for setting autoscaler to avoid this cases.
It is not related to HPA in this case but more on how you graceful shut down your pods.
In short, your service/LB is not aware if your pod is ready to accept new requests, so on a SIGTERM signal, your pod should set your readiness probe to false, and give some time for the app to shutdown. If your readiness probe is not healthy, the service won't send new requests to your pod.
Then you can shut it down once all requests have been addressed AND the pod won't receive new requests.
I would advise you of reading these sources:
https://cloud.google.com/blog/products/containers-kubernetes/kubernetes-best-practices-terminating-with-grace
https://pracucci.com/graceful-shutdown-of-kubernetes-pods.html
The problem I'm trying to solve is horizontal scaling for the web application, where some sessions lead to high CPU usage. The idea is to use Readiness probe to inform K8s that pod is loaded with the current task and new traffic has to be sent to another one (HPA will do the work and prepare a new pod).
But I want that session that processing on the initial pod will be active and once work is done the result will be delivered to the user.
The question is does it mean that if readiness probe fail K8s will:
Stop route ALL traffic to the pod, drop current sessions that open through ingress.
Stop route NEW traffic to the pod, but current sessions will be active during the specified timeout.
Thank you in advance.
UPDATE
It seems like I was totally not right in my 1st edit. More correct is to specify that It will Stop route NEW traffic to the pod but TCP connections like ssh will still be alive.
When the Endpoints controller receives the notification that the readiness probe failed, it removes the Pod as an Endpoint in the Service that the Pod is a part of. Then API server sends this information to the kube-proxies running on the worker nodes and kube-proxies update the iptables rules on its node, which is what prevents new connections from being forwarded to this Pod. However it's worth knowing that the TCP protocol is a stateful protocol (unlike HTTP) so existing connections (e.g ssh sessions) will still be active.
One of my micro service is running on Kubernetes.
I would like to specify to K8s load balancer when a pod is busy because the behaviour that I get currently is not ok.
One example:
I have 8 pods running, each pod can process 1 request at a time. Each request take from 70 to 100% of the CPU core allocated to the pod.
But when I send 8 requests to my application, Kubernetes does not dispatch those requests to the 8 pods but try to use only one. And since I'm blocking (via threadpool) each replica of app to use only one thread at a time, of course requests are queued for pod 1.
So my question is: How can I tell Kubernetes that POD 1 is busy and that load-balancer must dispatch request 2 to POD 2 ?
Note: For dev and test purpose I'm using Docker Desktop (Docker for Windows) on Windows 10 and kubectl.
As prometherion suggested you can use the liveness probe and also i would suggest to add the rediness probe together.
you can have a look at the official document : https://kubernetes.io/docs/tasks/configure-pod-container/configure-liveness-readiness-probes/
Sometimes, applications are temporarily unable to serve traffic. For example when, application first need to load large data or configuration files during startup.
In such cases, you don’t want to kill the application, but you don’t want to send traffic either there to pods. K8s provides readiness probes to detect and mitigate these situations. A pod with containers reporting that they are not ready does not receive traffic through Kubernetes Services.
You have to use LivenessProbe
when a Pod will not able to handle a request its IP will be removed from Service endpoints, so no traffic will be forwarded to it.
LivenessProbe can be TCP or HTTP
I have a scenario where it is required to 'prepare' Kubernetes towards taking off/terminating/shutdown a container, but allow it to serve some requests till that happens.
For example, lets assume that there are three methods: StartAction, ProcessAction, EndAction. I want to prevent clients from invoking StartAction when a container is about to be shutdown. However they should be able use ProcessAction and EndAction on that same container (after all Actions have been completed, the container will shutdown).
I was thinking that this is some sort of 'dual' readiness probe, where I basically want to indicate a 'not ready' status but continue to serve requests for already started Actions.
I know that there is a PreStop hook but I am not confident that this serves the need because according to the documentation I suspect that during the PreStop the pod is already taken off the load balancer:
(simultaneous with 3) Pod is removed from endpoints list for service, and are no longer considered part of the set of running Pods for replication controllers. Pods that shutdown slowly cannot continue to serve traffic as load balancers (like the service proxy) remove them from their rotations.
(https://kubernetes.io/docs/concepts/workloads/pods/pod/#termination-of-pods).
Assuming that I must rely on stickiness and must continue serving requests for Actions on containers where those actions were started, is there some recommended practice?
I think you can just implement 2 endpoints in your application:
Custom readiness probe
Shutdown preparation endpointList item
So to make graceful shutdown I think you should firstly call "Shutdown preparation endpoint" which will cause that "Custom readiness probe" will return error so Kubernetes will get out that POD from service load balancer (no new clients will come) but existing TCP connections will be kept (existing clients will operate). After your see in some custom metrics (which your service should provide) that all actions for clients are done you should shutdown containers using standard Kubernetes actions. All those actions should be probably automated somehow using Kubernetes and your application APIs.
Is this a thing?
I have some legacy services which will never run in Kubernetes that I currently make available to my cluster by defining a service and manually uploading an endpoints object.
However, the service is horizontally sharded and we often need to restart one of the endpoints. My google-fu might be weak, but i can't figure out if Kubernetes is clever enough to prevent the Service from repeatedly trying the dead endpoint?
The ideal behavior is that the proxy should detect the outage, mark the endpoint as failed, and at some point when the endpoint comes back re-admit it into the full list of working endpoints.
BTW, I understand that at present, liveness probes are HTTP only. This would need to be a TCP probe because it's a replicated database service that doesn't grok HTTP.
I think the design is for the thing managing the endpoint addresses to add/remove them based on liveness. For services backed by pods, the pod IPs are added to endpoints based on the pod's readiness check. If a pod's liveness check fails, it is deleted and its IP removed from the endpoint.
If you are manually managing endpoint addresses, the burden is currently on you (or your external health checker) to maintain the addresses/notReadyAddresses in the endpoint.