K8s EKS Error - Readiness probe errored: rpc error - kubernetes

Question

I'm using AWS EKS Fargate to deploy my work. After applying the deployment yaml file, everything goes well in first 10mins, but after that, I failed to access the pod by using kubectl exec <podname> -- bash, when typing kubectl describe pod <podname>, both readinessProbe and livenessProbe return similar messages as below:
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning LoggingDisabled 16m fargate-scheduler Disabled logging because aws-logging configmap was not found. configmap "aws-logging" not found
Normal Scheduled 15m fargate-scheduler Successfully assigned k8s-fargate/k8s-api-5765846f76-d7nws to fargate-ip-10-0-130-250.ap-east-1.compute.internal
Normal Pulling 15m kubelet Pulling image "awsaccid.dkr.ecr.ap-east-1.amazonaws.com/k8s-api-test:1.0.0"
Normal Pulled 14m kubelet Successfully pulled image "awsaccid.dkr.ecr.ap-east-1.amazonaws.com/k8s-api-test:1.0.0" in 1m18.703187993s
Normal Created 14m kubelet Created container k8s-api
Normal Started 14m kubelet Started container k8s-api
Warning Unhealthy 2m18s kubelet Readiness probe errored: rpc error: code = Unknown desc = failed to exec in container: failed to start exec "c2a2e9750a44684104a7e76a92bf7abe814ba29f306b092a48e17b90aab7f2dd": OCI runtime exec failed: exec failed: container_linux.go:380: starting container process caused: resource temporarily unavailable: unknown
Warning Unhealthy 2m13s kubelet Readiness probe errored: rpc error: code = Unknown desc = failed to exec in container: failed to start exec "bcb60f638e2c364adc8694bc12f00660e2b0d7647d3861d3462727976d2df08c": OCI runtime exec failed: exec failed: container_linux.go:380: starting container process caused: resource temporarily unavailable: unknown
Warning Unhealthy 2m8s kubelet Readiness probe errored: rpc error: code = Unknown desc = failed to exec in container: failed to start exec "988d29870b88fdcaae3cedf1071e79d2a786638c801364d71b6c7886f0be79e1": OCI runtime exec failed: exec failed: container_linux.go:380: starting container process caused: resource temporarily unavailable: unknown
Moreover, livenessProbe hasn't restart the pod even it is unhealthy.
I spent a whole day for this but still failed to solve it, anyone knows the problem? Thank you so much
Here's my deploy.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
namespace: k8s-fargate
name: k8s-api
spec:
replicas: 1
selector:
matchLabels:
app: k8s-api
template:
metadata:
labels:
app: k8s-api
spec:
volumes:
- name: k8s-properties
configMap:
name: k8s-properties
containers:
- name: k8s-api
image: awsaccountid.dkr.ecr.ap-east-1.amazonaws.com/k8s-test:1.0.0
imagePullPolicy: IfNotPresent
ports:
- containerPort: 8443
resources:
requests:
memory: "1024Mi"
cpu: "200m"
limits:
memory: "2500Mi"
cpu: "1000m"
volumeMounts:
- name: k8s-properties
mountPath: "/usr/local/folder"
readOnly: false
livenessProbe:
exec:
command:
- cat
- /usr/local/folder/file
initialDelaySeconds: 5
periodSeconds: 30
readinessProbe:
exec:
command:
- cat
- /usr/local/folder/file
initialDelaySeconds: 5
periodSeconds: 5

Answer 1

Problem solved by creating new Docker image. Still have no idea of the error, but problem likely comes from the image container itself.