Kubernetes CronJob startup prob - kubernetes

Question

I'm starting with Kubernetes and I implemented a CronJob that runs a Java jar.
It works fine but what I have observed is that if for some reason (for example, a wrong secret key) the container does not start, the pod will sit there indefinitely with error status : CreateContainerConfigError.
Is there a way to automatically kill the pod when such situation occurs ?
I tried with startup probe as indicated in code below but the probe did not even run.
apiVersion: batch/v1
kind: CronJob
metadata:
name: appName
namespace: appNamespace
labels:
app: appName
release: production
tiers: backend
spec:
jobTemplate:
spec:
backoffLimit: 2
template:
spec:
volumes:
- name: tmp-pod
emptyDir: {}
containers:
- name: appName
image: docker-image
command: ["/bin/bash", "-c"]
args:
- |
touch /tmp/pod/app-started;
java -XX:MaxRAMPercentage=75.0 -Djava.security.egd=file:/dev/urandom -jar /app.jar;
volumeMounts:
- mountPath: /tmp/pod
name: tmp-pod
env:
- name: env_var
value: value
# if app is not started within 5m (30 * 10 = 300s), container will be killed.
startupProbe:
exec:
command:
- cat
- /tmp/pod/app-started
initialDelaySeconds: 5
periodSeconds: 10
failureThreshold: 30
resources:
requests:
memory: "2200Mi"
cpu: "750m"
limits:
memory: "2200Mi"
restartPolicy: OnFailure
schedule: "0 12 * * *"
concurrencyPolicy: Forbid
Does CronJob not support probes ?
Or I'm doing something wrong ?
Would there be another way of killing container that is not able to start after some time ?

Answer 1

I had the same issue, kubernetes v1.23.14
It seems that the kubernetes has a bug with cronjobs probs
Events:
Normal Started 3m38s kubelet Started container k8s-costs
Warning Unhealthy 3m38s kubelet Startup probe failed: cat: /var/run/secrets/vault-envs: No such file or directory
Normal Killing 3m37s kubelet Stopping container k8s-costs
kubelet said "Stopping container" but it still working
So as result I see that pod is Successed and everything fine, no alerts.
But really job execution was failed and already we in a fire

Answer 2

You can use a liveness probe to automatically kill a pod if the container inside it fails to start. A liveness probe continuously checks if the container is running, and if it detects that the container has failed, it restarts the container.
Here is an example of how to add a liveness probe to your pod definition:
apiVersion: batch/v1
kind: CronJob
metadata:
name: appName
namespace: appNamespace
labels:
app: appName
release: production
tiers: backend
spec:
jobTemplate:
spec:
backoffLimit: 2
template:
spec:
volumes:
- name: tmp-pod
emptyDir: {}
containers:
- name: appName
image: docker-image
command: ["/bin/bash", "-c"]
args:
- |
touch /tmp/pod/app-started;
java -XX:MaxRAMPercentage=75.0 -Djava.security.egd=file:/dev/urandom -jar /app.jar;
volumeMounts:
- mountPath: /tmp/pod
name: tmp-pod
env:
- name: env_var
value: value
livenessProbe:
exec:
command:
- cat
- /tmp/pod/app-started
initialDelaySeconds: 5
periodSeconds: 10
failureThreshold: 30
resources:
requests:
memory: "2200Mi"
cpu: "750m"
limits:
memory: "2200Mi"
restartPolicy: OnFailure
schedule: "0 12 * * *"
concurrencyPolicy: Forbid
In the above example, the liveness probe exec command checks for the presence of the file /tmp/pod/app-started. If the file does not exist after the initial delay of 5 seconds, the probe will check every 10 seconds until the failure threshold of 30 is reached. At that point, the container will be killed and restarted.