Kubernetes probe events not visible but failing - kubernetes

Question

I'm using K8S 1.14 and Helm 3.3.1.
I have an app which works when deployed without probes. Then I set two trivial probes:
livenessProbe:
exec:
command:
- ls
- /mnt
initialDelaySeconds: 5
periodSeconds: 5
readinessProbe:
exec:
command:
- ls
- /mnt
initialDelaySeconds: 5
periodSeconds: 5
When I deploy via helm upgrade, the command eventually (~5 mins) fails with:
Error: UPGRADE FAILED: release my-app failed, and has been rolled back due to atomic being set: timed out waiting for the condition
But in the events log there is no trace of any probe:
5m21s Normal ScalingReplicaSet deployment/my-app Scaled up replica set my-app-7 to 1
5m21s Normal Scheduled pod/my-app-7-6 Successfully assigned default/my-app-7-6 to gke-foo-testing-foo-testing-node-po-111-r0cu
5m21s Normal LoadBalancerNegNotReady pod/my-app-7-6 Waiting for pod to become healthy in at least one of the NEG(s): [k8s1-222-default-my-app-80-54]
5m21s Normal SuccessfulCreate replicaset/my-app-7 Created pod: my-app-7-6
5m20s Normal Pulling pod/my-app-7-6 Pulling image "my-registry/my-app:v0.1"
5m20s Normal Pulled pod/my-app-7-6 Successfully pulled image "my-registry/my-app:v0.1"
5m20s Normal Created pod/my-app-7-6 Created container my-app
5m20s Normal Started pod/my-app-7-6 Started container my-app
5m15s Normal Attach service/my-app Attach 1 network endpoint(s) (NEG "k8s1-222-default-my-app-80-54" in zone "europe-west3-a")
19s Normal ScalingReplicaSet deployment/my-app Scaled down replica set my-app-7 to 0
19s Normal SuccessfulDelete replicaset/my-app-7 Deleted pod: my-app-7-6
19s Normal Killing pod/my-app-7-6 Stopping container my-app
Hence the question: what are the probes doing and where?

Answer 1

Try deleting the helm then re-apply it again: helm del --purge <APPNAME>
Also which helm version are you using? Try upgrading to v3.2.1, there's an open issue that tries to fix this incident with previously failed upgrades: https://github.com/helm/helm/issues/5939

Answer 2

I reproduced the same scenario here and everything went fine. The release was deployed and the pod is running. Did you check within the container if the /mnt really exists?
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 3m41s Successfully assigned default/nginx-deployment2-5cdd568667-blsc7 to minikube
Normal Pulling 3m41s kubelet, minikube Pulling image "nginx"
Normal Pulled 3m38s kubelet, minikube Successfully pulled image "nginx" in 2.769840982s
Normal Created 3m38s kubelet, minikube Created container nginx
Normal Started 3m38s kubelet, minikube Started container nginx
NAME READY STATUS RESTARTS AGE
nginx-deployment2-5cdd568667-blsc7 1/1 Running 0 4m59s
apiVersion: apps/v1
kind: Deployment
metadata:
name: nginx-deployment2
spec:
selector:
matchLabels:
app: ameba
replicas: 1
template:
metadata:
labels:
app: ameba
spec:
containers:
- name: nginx
image: nginx
ports:
- containerPort: 80
name: nginx-port
livenessProbe:
exec:
command:
- ls
- /mnt
initialDelaySeconds: 5
periodSeconds: 5
readinessProbe:
exec:
command:
- ls
- /mnt
initialDelaySeconds: 5
periodSeconds: 5

Answer 3

I don't if you image include bash, but if you just want to verify if the directory exists, you can do the samething using others shell commands, try this:
livenessProbe:
exec:
command:
- /bin/bash
- -c
- ls /mnt
initialDelaySeconds: 5
periodSeconds: 5
readinessProbe:
exec:
command:
- /bin/bash
- -c
- ls /mnt
initialDelaySeconds: 5
periodSeconds: 5
In bash you can also try to use the test built-in function:
[[ -d /mnt ]] = The -d verify if the directory /mnt exists.
As an alternative, there is also the command stat:
stat /mnt
If you want to check if the directory has any specific file, use the complete path with filename include.