pod is not changing state from unready to ready - kubernetes

Question

I am using the readiness probe feature in my k8s pod. For this, probe checks if the file /tmp/healthy is present or not. If not, probe mark pod UN-READY. I have observed If I create the file again /tmp/healthy, the pod does not come back to READY state. Does it mean, in pod lifecycle there is only path of READY--> UN-READY but not vice-versa?
/home/ravi/for_others/ric/stub>kubectl describe pod -n myns deployment-eterm
Name: deployment-eterm
Namespace: myns
Priority: 0
Status: Running
IP: 192.168.252.87
IPs:
IP: 192.168.252.87
Controlled By: ReplicaSet/deployment-myns-eterm-micro-6b896556c8
Containers:
Liveness: exec [/bin/sh -c /tmp/liveliness-status-collector.sh] delay=60s timeout=1s period=2s #success=1 #failure=5
Readiness: exec [/bin/sh -c cat /tmp/healthy] delay=60s timeout=1s period=10s #success=1 #failure=1
Environment Variables from:
configmap-myns-eterm-env-micro ConfigMap Optional: false
Environment: <none>
Mounts:
/var/run/secrets/kubernetes.io/serviceaccount from default-token-57jpz (ro)
Conditions:
Type Status
Initialized True
Ready False
ContainersReady False
PodScheduled True
Tolerations: node.kubernetes.io/not-ready:NoExecute for 300s
node.kubernetes.io/unreachable:NoExecute for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 9m11s default-scheduler Successfully assigned myns/deployment-eterm to mnode
Normal Pulled 9m10s kubelet, mnode Container image "porics.microlab.com:5000/eterm:latest" already present on machine
Normal Created 9m10s kubelet, mnode Created container container-myns-eterm
Normal Started 9m10s kubelet, mnode Started container container-myns-eterm
Normal Pulled 9m10s kubelet, mnode Container image "porics.microlab.com:5000/config-proxy:latest" already present on machine
Normal Created 9m10s kubelet, mnode Created container container-myns-configproxy
Normal Started 9m10s kubelet, mnode Started container container-myns-configproxy
Warning Unhealthy 3s (x3 over 23s) kubelet, mnode Readiness probe failed: cat: /tmp/healthy: No such file or directory
/home/ravi/for_others/ric/stub>

Answer 1

The below statement is not true:
Does it mean, in pod lifecycle, there is the only one path of READY--> UN-READY
but not vice-versa?
You can run the below experiment to see the pod transition:
Create a pod with below manifest file:
apiVersion: v1
kind: Pod
metadata:
labels:
test: readiness
name: readiness-test
spec:
nodeName: kube-master
containers:
- name: liveness
image: bash
command: ['bash','-c', 'while true;do echo "$(date): creating /tmp/healthy"; touch /tmp/healthy; sleep 10; echo "$(date): deleting /tmp/healthy";rm /tmp/healthy ;sleep 10;done']
readinessProbe:
exec:
command:
- cat
- /tmp/healthy
initialDelaySeconds: 5
periodSeconds: 2
The above manifest will cause the pod to transition into ready/not-ready every 10 seconds.
kubectl get pod -w
NAME READY STATUS RESTARTS AGE
readiness-test 0/1 Running 0 79s
readiness-test 1/1 Running 0 83s
readiness-test 0/1 Running 0 97s
readiness-test 1/1 Running 0 103s
readiness-test 0/1 Running 0 117s
readiness-test 1/1 Running 0 2m3s
readiness-test 0/1 Running 0 2m17s
readiness-test 1/1 Running 0 2m23s
readiness-test 0/1 Running 0 2m37s
readiness-test 1/1 Running 0 2m43s
readiness-test 0/1 Running 0 2m57s
readiness-test 1/1 Running 0 3m3s
readiness-test 0/1 Running 0 3m17s
readiness-test 1/1 Running 0 3m23s
readiness-test 0/1 Running 0 3m37s
readiness-test 1/1 Running 0 3m43s
readiness-test 0/1 Running 0 3m57s
readiness-test 1/1 Running 0 4m3s
You can see the the readiness status getting changed:
while true; do k get pod readiness-test -o jsonpath='{.status.containerStatuses[*].ready}{"\n"}';sleep 2 ;done
true
true
true
true
true
true
false
false
true
true
true
true
true
true
false
false
true
true
true
true
true
true
false
false
true
true
true
true
true
false
false
false