I have two containers，maybe A and B, which A should run before B, but A is a server application, which the final type is Running but not Complete, so I wonder in this way, will B be never executed? So how can I deal with it?
If A and B are part of the same pod, then initContainer is the legacy way to establish ordering.
From the Kubernetes Pod lifecycle, I suppose you mean "Running, but no Terminated"
A pod liveness/readiness probe is in your case a better fit, since the server will not accept request until ready.
Read "Straight to the Point: Kubernetes Probes" from Peter Malina
Both readiness and liveness probe run in parallel throughout the life of a container.
Use the liveness probe to detect an internal failure and restart the container (e.g. HTTP server down).
Use the readiness probe to detect if you can serve traffic (e.g. established DB connection) and wait (not restart) for the container.
A dead container is also not a ready container.
To serve traffic, all containers within a pod must be ready.
You can add a pod readiness gate (stable from 1.14) to specify additional conditions to be evaluated for Pod readiness.
Read also "Kubernetes Liveness and Readiness Probes: How to Avoid Shooting Yourself in the Foot" from Colin Breck
"Should Health Checks call other App Health Checks" compares that approach with the InitContainer approach
Related
I am struggling with defining a sane implementation strategy for Kubernetes probes for my product. Digging into the available guidelines, both official docs and field reports, I am not able to identify a common approach with consensus in the ecosystem; and most probably that's expected.
Here's what I plan to implement as probe definition rules:
Readiness probe:
Enable it for services handling incoming network traffic.
Question: Any other use case where I should consider a readiness probe for a service?
Liveness probe:
This is the most difficult one for me. What I have in mind as a rule is: Don't define it by default and only enable it manually for services where scenarios as deadlocks are detected and only until these bugs are fixed.
I don't see as a healthy approach to just assume that a service will deadlock and leave the liveness probe handle it. First because it is very hard to identify a service deadlock from the probe and second because it would leave bugs unaddressed.
Question: Any other use case where I should consider a liveness probe?
Startup probe:
Enable it only when there is a liveness probe enabled on that service.
Question: Without a liveness probe defined, is there any advantage in defining a startup probe?
Question: Without a liveness probe defined, is there any advantage in
defining a startup probe?
i am not seeing any advantage in defining the just startup probe without liveness. startup probe is there to protect the slow-starting PODs.
Question: Any other use case where I should consider a liveness probe?
If the service depends on DB or any other service and inside the backend service code you have a function or endpoint to check that you can use that as a liveness probe. You are right in the case of leaving the bug unaddressed.
Selling point to keeping the Liveness probe is that it will auto-restart container POD if it's failing while the readiness probe just changes the status from 0/1 to handle the traffic.
if you don't specify the liveness probe it will just decide the container status based on the PID.
If you are using the bash/sh, dumb-init in your docker you might not want PID 1 to trace instead you want child process PID 2 to track in that liveness probe would be required.
TL;DR is the last paragraph, but the rest is here for context if that's not clear enough.
I have a K8s pod running a PHP application. It's split up into an FPM container, and an Nginx container. The liveness and readiness checks are set up to check the container process. So for Nginx, this simply means "is port 443 answering", and for FPM this means "is TCP 9000 answering?".
We already have more intelligent probes ready at /readiness and /liveness endpoints in the PHP application, but where would these fit in?
When the pod was running both Nginx and FPM in a single container it was obvious, because restarting the single container due to a liveness probe failure made sense. For the FPM container, I thought perhaps changing it's probe type from httpGet to command might be the right thing, since you can then run a command that checks the status of the application. Something feels off about that though (mainly that you're no longer checking the main process anymore).
I can probably figure something out where you're checking the service via FPM, but what I want to ask is:
When you have a pod with an FPM container, what is the proper usage of readiness and liveness probes? Should I be asking the application itself whether it feels okay, or should I get all the information to make a decision from FPM?
You need to ask both FPM and the application. Those probes are not for a pod, they are independently configured for each container. (Although a single container's readiness probe failure will take the whole pod out of traffic, the readiness probe anyway is still for each container)
I'm creating multiple pods at the same time in Openshift, and I also want to check the containers inside the pods are working correctly.
Some of these containers can take a while to start-up, and I don't want to wait for one pod to be fully running before starting up the other one.
Are there any Openshift / Kubernetes checks I can do to ensure a container has booted up, while also going ahead with other deployments?
Please configure the Liveness and Readiness Probes
Liveness : Under what circumstances is it appropriate to restart the pod?
Readiness : under what circumstances should we take the pod out of the list of service endpoints so that it no longer responds to
requests?
...Some of these containers can take a while to start-up
Liveness probe is not a good option for containers that requires extended startup time, mainly because you have to set a long time to cater for startup; which is irrelevant after that - result to unable to detect problem on time during execution. Instead, you use startup probe to handle and detect problem during startup and handover to liveness probe upon success; or restart container according to its restartPolicy should the startup probe failed.
According to this documentation, I see that readinessProbe can be used to temporarily halt requests to a pod without having to restart it in order to recover gracefully.
https://kubernetes.io/docs/tasks/configure-pod-container/configure-liveness-readiness-probes/#define-readiness-probes
When I see pod events it looks like the pod is restarted upon Readiness probe failure. Here are the events:
1. Readiness probe failed
2. Created container
3. Started container
4. Killing container with id {}
Tried to modify container restartPolicy to OnFailure hoping this configuration decides pod action upon readinessProbe failure but I see the following error:
The Deployment {} is invalid: spec.template.spec.restartPolicy: Unsupported value: "OnFailure": supported values: "Always"
Which is the right way to stop requests to a pod without having to restart it and letting the application gracefully recover?
There are two type of probes.
Restarts happen due to failing liveness probes.
https://kubernetes.io/docs/tasks/configure-pod-container/configure-liveness-readiness-probes/
liveness probe
The kubelet uses liveness probes to know when to restart a Container. For example, liveness probes could catch a deadlock, where an application is running, but unable to make progress. Restarting a Container in such a state can help to make the application more available despite bugs.
readiness probe
Sometimes, applications are temporarily unable to serve traffic. For example, an application might need to load large data or configuration files during startup, or depend on external services after startup. In such cases, you don’t want to kill the application, but you don’t want to send it requests either. Kubernetes provides readiness probes to detect and mitigate these situations. A pod with containers reporting that they are not ready does not receive traffic through Kubernetes Services.
Today I found a very good essay about probes https://blog.colinbreck.com/kubernetes-liveness-and-readiness-probes-how-to-avoid-shooting-yourself-in-the-foot/
I wanted to know what kubernetes does to check liveness and readiness of a pod and container by default.
I could find the document which mentions how I can add my custom probe and change the probe parameters like initial delay etc. However, could not find the default probe method used by k8s.
By default, Kubernetes starts to send traffic to a pod when all the containers inside the pod start, and restarts containers when they crash. While this can be good enough when you are starting out, but you can make your deployment more robust by creating custom health checks.
By default, Kubernetes just checks container inside the pod is up and starts sending traffic. There is no by default readiness or liveness check provided by kubernetes.
Readiness Probe
Let’s imagine that your app takes a minute to warm up and start. Your service won’t work until it is up and running, even though the process has started. You will also have issues if you want to scale up this deployment to have multiple copies. A new copy shouldn’t receive traffic until it is fully ready, but by default Kubernetes starts sending it traffic as soon as the process inside the container starts. By using a readiness probe, Kubernetes waits until the app is fully started before it allows the service to send traffic to the new copy.
Liveness Probe
Let’s imagine another scenario where your app has a nasty case of deadlock, causing it to hang indefinitely and stop serving requests. Because the process continues to run, by default Kubernetes thinks that everything is fine and continues to send requests to the broken pod. By using a liveness probe, Kubernetes detects that the app is no longer serving requests and restarts the offending pod.
TL/DR: there is no default readiness probe ("should I send this pod traffic?") and the default liveness probe ("should I kill this pod?") is just whether the container is still running.
Kubernetes will not do anything on its own. You will have to decide what liveness and readiness mean to you. There are multiple things you can do, for example, an HTTP get request, issue a command, or connect on a port. It is up to you to decide how you want to make sure your users are happy and everything is running correctly.