GKE readiness/liveness randomly failed - kubernetes

Question

I am facing a very weird problem.
I have a deployment with nginx container(six pods) work on GKE, I use Liveness/Readiness Probe on nginx container, all the pods are working fine.
One day, all pods were dead because of readiness/liveness failed. And I use kubectl describe pod that pop up ...
Liveness probe failed: Get http://10.28.6.19:80/healthz: net/http: request canceled (Client.Timeout exceeded while awaiting headers)
Readiness probe failed: Get http://10.28.6.19:80/healthz: net/http: request canceled (Client.Timeout exceeded while awaiting headers)
I try to delete pods to let six pods reborn, the pod is very normal.
Some posts said: this is pods hit the limit of resource, I swear my pod resource usage is very low. And I don't set limit for my pods.
I have encounter this problem third, until now I still don't know the root cuse and any solution of this kind of problem.
Deployment Spec as below, Nginx will direct repose 200 status code according to /healthz, so nginx config have to add a location
readinessProbe:
httpGet:
path: /healthz
port: 80
initialDelaySeconds: 10
timeoutSeconds: 2
livenessProbe:
httpGet:
path: /healthz
port: 80
initialDelaySeconds: 10
timeoutSeconds: 2

Answer 1

When your app starts hitting the resource limit, the Kubernetes starts throttling your container. Resulting in probe failure. You should always set the resource request and limit for avoiding such events:
containers:
- name: app
image: images.my-company.example/app:v4
resources:
requests:
memory: "64Mi"
cpu: "250m"
limits:
memory: "128Mi"
cpu: "500m"

Answer 2

Liveness probe failed: Get http://10.28.6.19:80/healthz: net/http: request canceled (Client.Timeout exceeded while awaiting headers)
Readiness probe failed: Get http://10.28.6.19:80/healthz: net/http: request canceled (Client.Timeout exceeded while awaiting headers)
The error message says that your HTTP request wasn't successful.
Readiness Probe needs to succeed for the pod to be added as an endpoint for the
service exposing it.
kubectl get po -o wide : Run this command to get the pod's cluster IP
kubectl exec -t [another_pod] -- curl -I [pod's cluster IP] : If this command returns a 200 response, the path is configured properly and the readiness probe should pass. If you get anything other than a 200 response, this is why the readiness probe fails and you need to check your image.

Answer 3

You can try out any of these solutions.
1. Increase the timeoutSeconds to atleast 10 seconds. During peak load time nginx don't provide the response in 2 seconds. Also configure successThreshold and failureThreshold values.
2. Replace httpGet to exec method which will call the same endpoint using curl if needed. This resolves my issue.
3. Isolate the nginx containers to separate node pools to ensure no other deployments are creating any kind of resource constraints to nginx pods.