One of my micro service is running on Kubernetes.
I would like to specify to K8s load balancer when a pod is busy because the behaviour that I get currently is not ok.
One example:
I have 8 pods running, each pod can process 1 request at a time. Each request take from 70 to 100% of the CPU core allocated to the pod.
But when I send 8 requests to my application, Kubernetes does not dispatch those requests to the 8 pods but try to use only one. And since I'm blocking (via threadpool) each replica of app to use only one thread at a time, of course requests are queued for pod 1.
So my question is: How can I tell Kubernetes that POD 1 is busy and that load-balancer must dispatch request 2 to POD 2 ?
Note: For dev and test purpose I'm using Docker Desktop (Docker for Windows) on Windows 10 and kubectl.
As prometherion suggested you can use the liveness probe and also i would suggest to add the rediness probe together.
you can have a look at the official document : https://kubernetes.io/docs/tasks/configure-pod-container/configure-liveness-readiness-probes/
Sometimes, applications are temporarily unable to serve traffic. For example when, application first need to load large data or configuration files during startup.
In such cases, you don’t want to kill the application, but you don’t want to send traffic either there to pods. K8s provides readiness probes to detect and mitigate these situations. A pod with containers reporting that they are not ready does not receive traffic through Kubernetes Services.
You have to use LivenessProbe
when a Pod will not able to handle a request its IP will be removed from Service endpoints, so no traffic will be forwarded to it.
LivenessProbe can be TCP or HTTP
Related
From Kubernetes document, when readiness probe fails, it removes the Pod's IP address from the endpoints of all services that match the pod.
We are thinking about implementing SIGTERM handler to fail the health check and stop the pod from receiving future traffic. That's what we want, no more Inbound traffic. The question is, if the pod contains requests that depend on backend service which are not reside in the same pod, will the pod still be able to complete those outbound requests?
From the docs (emphasis mine):
Sometimes, applications are temporarily unable to serve traffic. For example, an application might need to load large data or configuration files during startup, or depend on external services after startup. In such cases, you don't want to kill the application, but you don't want to send it requests either. Kubernetes provides readiness probes to detect and mitigate these situations. A pod with containers reporting that they are not ready does not receive traffic through Kubernetes Services.
The pod can't be reached through Kubernetes services. You can still make outbound requests, and anyone using the pod name or IP directly will also still be able to reach it.
I've recently been reading myself into kubernetes and want to create a StatefulSet for a service of mine.
As far as I understood, a StatefulSet with let's say 5 replicas offers certian dns entries to reach it.
E.g. myservice1.internaldns.net, myservice2.internaldns.net
What would now happen, if one of the pods behind the dns entries goes down, even if it's just for a small amount of time?
I had a hard time finding information on this.
Is the request held until the pod is back? Will it be router to another pod, possibly losing the respective state? Will it just straightup fail?
If you're Pod is not ready, then the traffic is not forwarded to that Pod. So, your service will not load balance traffic to Pods that are not ready.
To decide if the given Pod is ready or not, you should define readinessProbe. I recommend reading the Kubernetes documentation on "Configure Liveness, Readiness and Startup Probes".
I'm following this task Configure Liveness, Readiness and Startup Probes
and it's unclear to me whether a container where the check is made is a container only used to check the availability of a pod? Because it makes sense if pod check container fails therefore api won't let any traffic in to the pod.
So a health check signal must be coming from container where some image or app runs? (sorry, another question)
From the link you provided it seems like they are speaking about Containers and not Pods so the probes are meant to be per containers. When all containers are ready the pod is described as ready too as written in the doc you provided :
The kubelet uses readiness probes to know when a Container is ready to
start accepting traffic. A Pod is considered ready when all of its
Containers are ready. One use of this signal is to control which Pods
are used as backends for Services. When a Pod is not ready, it is
removed from Service load balancers.
So yes, every containers that are running some images or apps are supposed to expose those metrics.
Livenes and readiness probes as described by Ko2r are additional checks inside your containers and verified by kubelet according to the settings fro particular probe:
If the command (defined by health-check) succeeds, it returns 0, and the kubelet considers the Container to be alive and healthy. If the command returns a non-zero value, the kubelet kills the Container and restarts it.
In addition:
The kubelet uses liveness probes to know when to restart a Container. For example, liveness probes could catch a deadlock, where an application is running, but unable to make progress. Restarting a Container in such a state can help to make the application more available despite bugs.
Fro another point of view:
Pod is a top-level resource in the Kubernetes REST API.
As per docs:
Pods are ephemeral. They are not designed to run forever, and when a Pod is terminated it cannot be brought back. In general, Pods do not disappear until they are deleted by a user or by a controller.
Information about controllers can find here:
So the best practise is to use controllers like describe above. You’ll rarely create individual Pods directly in Kubernetes–even singleton Pods. This is because Pods are designed as relatively ephemeral, disposable entities. When a Pod gets created (directly by you, or indirectly by a Controller), it is scheduled to run on a Node in your cluster. The Pod remains on that Node until the process is terminated, the pod object is deleted, the Pod is evicted for lack of resources, or the Node fails.
Note:
Restarting a container in a Pod should not be confused with restarting the Pod. The Pod itself does not run, but is an environment the containers run in and persists until it is deleted
Because Pods represent running processes on nodes in the cluster, it is important to allow those processes to gracefully terminate when they are no longer needed (vs being violently killed with a KILL signal and having no chance to clean up). Users should be able to request deletion and know when processes terminate, but also be able to ensure that deletes eventually complete. When a user requests deletion of a Pod, the system records the intended grace period before the Pod is allowed to be forcefully killed, and a TERM signal is sent to the main process in each container. Once the grace period has expired, the KILL signal is sent to those processes, and the Pod is then deleted from the API server. If the Kubelet or the container manager is restarted while waiting for processes to terminate, the termination will be retried with the full grace period.
The Kubernetes API server validates and configures data for the api objects which include pods, services, replicationcontrollers, and others. The API Server services REST operations and provides the frontend to the cluster’s shared state through which all other components interact.
For example, when you use the Kubernetes API to create a Deployment, you provide a new desired state for the system. The Kubernetes Control Plane records that object creation, and carries out your instructions by starting the required applications and scheduling them to cluster nodes–thus making the cluster’s actual state match the desired state.
Here you can find information about processing pod termination.
There are different probes:
For example for HTTP probe:
even if your app isn’t an HTTP server, you can create a lightweight HTTP server inside your app to respond to the liveness probe.
Command
For command probes, Kubernetes runs a command inside your container. If the command returns with exit code 0 then the container is marked as healthy.
More about probes and best practices.
Hope this help.
For self-healing I want to be able to call a function on a specific pod instance from outside the kube cluster.
So... I have 2 networks, non-kube, and my kube-cluster. My kube-cluster contains one service, myapp and has 8 replicas spread across 4 nodes. If each of these pods was given a VIP 10.1.1.x where x is from 1 to 8, I want to be able to from any app running in non-kube, be able to call any of the API methods exposed in myapp, but forcing a particular replica.
E.g. myapp has one REST GET method,
whatismyvip()
And from a console app running in non-kube I want to be able to ensure I get the following output, by calling the above method on the right pod.
10.1.1.4
10.1.1.6
10.1.1.8
10.1.1.2
10.1.1.5
Why would I want to do something like this? Well, generally the errors that are generated in the kube-cluster service are server errors due to bad data / configuration, rather than the system "going down", i.e. 500 errors, and so I want the client to be smart enough to go
"ok, i got an error from 10.1.1.2, i will try 10.1.1.3"
I'd prefer to do this natively in kubernetes (even if it means custom IngressController) rather than sidecar service mesh, Envoys et al.
Create a loadbalanced service that fronts your replicaset of 8 pods. Define a liveness HTTP probe that pings the whatismyvip() HTTP endpoint.
The host kubelet will restart the pod if you get any unexpected HTTP status codes.
define a liveness HTTP request
Otherwise if what you mean by forcing a particular replica is that you wish to direct your request into a specific pod from outside the cluster, then perhaps a custom solution might be better. Each see if 2 X pods on each node might have a clusterIP service fronting them. Then use a loadbalanced service fronting a custom app that proxies your request for a specific replica or set of pods based on your request.
E.g. a URL query string parameter that targets clusterIP service A, B, C, or D.
https://customrouterapp/whatismyvip?replicaset=B
we are planning to use kubernetes and I am validating, if and how it fits our requirements.
One concern is the following:
I want to build an application/pod, which is connecting to a certain service on the internet (host and port) and keeps the socket alive as long as we need it (usually forever). The number of sockets the application will connect to may vary.
For the inter pod communication we are going to use RabbitMQ.
What is the correct/best practise approach for that purpose?
One pod handling all/multiple sockets?
Replicated pods handling multiple socket?
One Socket per pod?
How do i react, if the number of sockets changes?
At the moment we want to use the gitlab-ci and helm for our CI pipeline.
kubernetes deploys Pods and has two abstractions: Deployments and StatefulSets. The former deploys ephemeral Pods whose hostname and IP changes. The latter retains the state.
If you're deploying kubernetes only for this application, it's an overkill imho. I'd rather use plain Docker or a simpler-than-kubernetes orchestrator such as Docker Swarm Mode or Kontena.
If kubernetes is your only option, you could deploy the app as a StatefulSet. That way its hostname will remain between restarts. Have the app monitor its hostname and connect to the appropriate endpoint. For example, the app-1 Pod connects to endpoint:10001, app-2 Pod connects to endpoint:10002, and so on...
When more Pods are needed to connect to more sockets, either increase the StatefulSet's replicas manually, or write a sidecar application to monitor the no. of sockets and up/down the replicas automatically.