k8s pod readiness probe fails with connection refused, but pod is serving requests just fine - kubernetes

Question

I'm having a hard time understanding why a pods readiness probe is failing.
Warning Unhealthy 21m (x2 over 21m) kubelet, REDACTED Readiness probe failed: Get http://192.168.209.74:8081/actuator/health: dial tcp 192.168.209.74:8081: connect: connection refused
If I exec into this pod (or in fact into any other I have for that application), I can run a curl against that very URL without issue:
kubectl exec -it REDACTED-l2z5w /bin/bash
$ curl -v http://192.168.209.74:8081/actuator/health
$ curl -v http://192.168.209.74:8081/actuator/health
* Expire in 0 ms for 6 (transfer 0x5611b949ff50)
* Trying 192.168.209.74...
* TCP_NODELAY set
* Expire in 200 ms for 4 (transfer 0x5611b949ff50)
* Connected to 192.168.209.74 (192.168.209.74) port 8081 (#0)
> GET /actuator/health HTTP/1.1
> Host: 192.168.209.74:8081
> User-Agent: curl/7.64.0
> Accept: */*
>
< HTTP/1.1 200
< Set-Cookie: CM_SESSIONID=E62390F0FF8C26D51C767835988AC690; Path=/; HttpOnly
< X-Content-Type-Options: nosniff
< X-XSS-Protection: 1; mode=block
< Cache-Control: no-cache, no-store, max-age=0, must-revalidate
< Pragma: no-cache
< Expires: 0
< X-Frame-Options: DENY
< Content-Type: application/vnd.spring-boot.actuator.v3+json
< Transfer-Encoding: chunked
< Date: Tue, 02 Jun 2020 15:07:21 GMT
<
* Connection #0 to host 192.168.209.74 left intact
{"status":"UP",...REDACTED..}
I'm getting this behavior from both a Docker-for-Desktop k8s cluster on my Mac as well as an OpenShift cluster.
The readiness probe is shown like this in kubectl describe:
Readiness: http-get http://:8081/actuator/health delay=20s timeout=3s period=5s #success=1 #failure=10
The helm chart has this to configure it:
readinessProbe:
failureThreshold: 10
httpGet:
path: /actuator/health
port: 8081
scheme: HTTP
initialDelaySeconds: 20
periodSeconds: 5
successThreshold: 1
timeoutSeconds: 3
I cannot fully rule out that HTTP proxy settings are to blame, but the k8s docs say that HTTP_PROXY is ignored for checks since v1.13, so it shouldn't happen locally.
The OpenShift k8s version is 1.11, my local one is 1.16.

Answer 1

Describing events always show the last event on the resource you are checking. The thing is that the last event logged was an error while checking the readinessProbe.
I tested it in my lab with the following pod manifest:
apiVersion: v1
kind: Pod
metadata:
name: readiness-exec
spec:
containers:
- name: readiness
image: k8s.gcr.io/busybox
args:
- /bin/sh
- -c
- sleep 30; touch /tmp/healthy; sleep 600
readinessProbe:
exec:
command:
- cat
- /tmp/healthy
initialDelaySeconds: 5
periodSeconds: 5
As can be seen, a file /tmp/healthy will be created in the pod after 30 seconds and the readinessProbe will check if the file exists after 5 seconds and repeat the check after every 5 seconds.
Describing this pod will give me that:
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 7m56s default-scheduler Successfully assigned default/readiness-exec to yaki-118-2
Normal Pulling 7m55s kubelet, yaki-118-2 Pulling image "k8s.gcr.io/busybox"
Normal Pulled 7m55s kubelet, yaki-118-2 Successfully pulled image "k8s.gcr.io/busybox"
Normal Created 7m55s kubelet, yaki-118-2 Created container readiness
Normal Started 7m55s kubelet, yaki-118-2 Started container readiness
Warning Unhealthy 7m25s (x6 over 7m50s) kubelet, yaki-118-2 Readiness probe failed: cat: can't open '/tmp/healthy': No such file or directory
The readinessProbe looked for the file 6 times with no success and it's completely right as I configured it to check every 5 seconds and the file was created after 30 seconds.
What you think is a problem is actually the expected behavior. Your Events is telling you that the readinessProbe failed to check 21 minutes ago. It means actually that your pod is healthy since 21 minutes ago.