I have a cronjob that keeps restarting, despite its RestartPolicy set to Never:
apiVersion: batch/v1beta1
kind: CronJob
metadata:
name: cron-zombie-pod-killer
spec:
schedule: "*/9 * * * *"
successfulJobsHistoryLimit: 1
jobTemplate:
spec:
template:
metadata:
name: cron-zombie-pod-killer
spec:
containers:
- name: cron-zombie-pod-killer
image: bitnami/kubectl
command:
- "/bin/sh"
args:
- "-c"
- "kubectl get pods --all-namespaces --field-selector=status.phase=Failed | awk '{print $2 \" --namespace=\" $1}' | xargs kubectl delete pod > /dev/null"
serviceAccountName: pod-read-and-delete
restartPolicy: Never
I would expect it to run every 9th minute, but that's not the case.
What happens is that when there are pods to clean up (so, when there's smth to do for the pod) it would run normally. Once everything is cleared up, it keeps restarting -> failing -> starting, etc. in a loop every second.
Is there something I need to do to tell k8s that the job has been successful, even if there's nothing to do (no pods to clean up)? What makes the job loop in restarts and failures?
That is by design. restartPolicy is not applied to a CronJob, but a Pod it creates.
If restartPolicy is set to Never, it will ust create new pods, if the previous failed. Setting it to OnFailure causes the Pod to be restarted, and prevents the stream of new Pods.
This was discussed in this GitHub issue: Job being constanly recreated despite RestartPolicy: Never #20255
Your kubectl command results in exit code 123 (any invocation exited with a non-zero status) if there are no Pods in Failed state. This causes the Job to fail, and constant restarts.
You can fix that by forcing kubectl command to exit with exit code 0. Add || exit 0 to the end of it:
kubectl get pods --all-namespaces --field-selector=status.phase=Failed | awk '{print $2 \" --namespace=\" $1}' | xargs kubectl delete pod > /dev/null || exit 0
...Once everything is cleared up, it keeps restarting -> failing -> starting, etc. in a loop every second.
When your first command returns no pod, the trailing commands (eg. awk, xargs) fails and returns non-zero exit code. Such exit code is perceived by the controller that the job has failed and therefore start a new pod to re-run the job. You should just exit with zero when there is no pod returned.
Related
I'm running a K8S job, with the following flags:
apiVersion: batch/v1
kind: Job
metadata:
name: my-EP
spec:
template:
metadata:
labels:
app: EP
spec:
restartPolicy: "Never"
containers:
- name: EP
image: myImage
The Job starts, runs my script that runs some application that sends me an email and then terminates. The application returns the exit code to the bash script.
when I run the command:
kubectl get pods, I get the following:
NAME READY STATUS RESTARTS AGE
my-EP-94rh8 0/1 Completed 0 2m2s
Sometimes there are issues, and the network not connected or no license available.
I would like that to be visible to the pod user.
My question is, can I propagate the script exit code to be seen when I run the above get pods command?
I.E instead of the "Completed" status, I would like to see my application exit code - 0, 1, 2, 3....
or maybe there is a way to see it in the Pods Statuses, in the describe command?
currently I see:
Pods Statuses: 0 Running / 1 Succeeded / 0 Failed
Is this possible?
The a non-zero exit code on k8s jobs will fall into the Failed pod status. There really isn't a way for you to have the exit code shown with kubectl get pods but you could output the pod status with -ojson and then pipe it into jq looking for the exit code. Something like the following from this post might work
kubectl get pod pod_name -c container_name-n namespace -ojson | jq .status.containerStatuses[].state.terminated.exitCode
or this, with the items[] in the json
kubectl get pods -ojson | jq .items[].status.containerStatuses[].state.terminated.exitCode
Alternatively, as u/blaimi mentioned, you can do it without jq, like this:
kubectl get pod pod_name -o jsonpath --template='{.status.containerStatuses[*].state.terminated.exitCode}
Goal is to terminate the pod after completion of Job.
This is my yaml file. Currently, my pod status is completed after running the job.
apiVersion: batch/v1
kind: Job
metadata:
# Unique key of the Job instance
name: example-job
spec:
template:
metadata:
name: example-job
spec:
containers:
- name: container-name
image: my-img
command: ["python", "main.py"]
# Do not restart containers after they exit
restartPolicy: Never
# of retries before marking as failed.
backoffLimit: 4
You can configure and remove the jobs once complete
inside the YAML you can configure limit of keeping the PODs
successfulJobsHistoryLimit: 0
failedJobsHistoryLimit: 0
you can set the history limits using above config in YAML.
The .spec.successfulJobsHistoryLimit and .spec.failedJobsHistoryLimit
fields are optional. These fields specify how many completed and
failed jobs should be kept. By default, they are set to 3 and 1
respectively. Setting a limit to 0 corresponds to keeping none of the
corresponding kind of jobs after they finish.
backoffLimit: 4 will retires a number of times and try to run the job before marking it as a failed one.
Read more at : https://kubernetes.io/docs/tasks/job/automated-tasks-with-cron-jobs/#jobs-history-limits
A job of a pod basically terminates itself after the main container of that pod finishes successful. If it returns a failure error code it will retry as many times as you specified in your backoffLimit.
So it seems as if your container does not terminate after it finishes whatever job it is supposed to do. Without knowing anything about your job image I cannot tell you what you need to do exactly.
However, it seems as if you need to adapt your main.py to properly exit after it has done what it is supposed to do.
If you want to delete the pod after completing the task, then just delete the job with kubectl:
$ kubectl delete jobs
You can also use yaml file script to automatically delete the jobs by using the following command:
$kubectl delete -f ./job. yaml
When you delete the job using kubectl, all the created pods get deleted too.
You can check whether these jobs and pods deleted or not with the following commands:
$ kubectl get jobs and $ kubectl get pods
For more details refer to the Jobs.
I have tried the above steps in my own environment and it worked for me.
How to delete the failed jobs in the kubernetes cluster using a cron job in gke?. when i tried to delete the failed jobs using following YAML, it has deleted all the jobs (including running)
apiVersion: batch/v1beta1
kind: CronJob
metadata:
name: XXX
namespace: XXX
spec:
schedule: "*/30 * * * *"
failedJobsHistoryLimit: 1
successfulJobsHistoryLimit: 1
jobTemplate:
spec:
template:
spec:
serviceAccountName: XXX
containers:
- name: kubectl-runner
image: bitnami/kubectl:latest
command: ["sh", "-c", "kubectl delete jobs $(kubectl get jobs | awk '$2 ~ 1/1' | awk '{print $1}')"]
restartPolicy: OnFailure
To delete failed Jobs in GKE you will need to use following command:
$ kubectl delete job $(kubectl get job -o=jsonpath='{.items[?(#.status.failed==1)].metadata.name}')
This command will output the JSON for all jobs and search for jobs that have status.failed field set to 1. It will then pass the failed jobs to $ kubectl delete jobs
This command ran in a CronJob will fail when there are no jobs with status: failed.
As a workaround you can use:
command: ["sh", "-c", "kubectl delete job --ignore-not-found=true $(kubectl get job -o=jsonpath='{.items[?(#.status.failed==1)].metadata.name}'); exit 0"]
exit 0 was added to make sure that the Pod will leave with status code 0
As for part of the comments made under the question:
You will need to modify it it support "Failed" Jobs
I have already tried the following , but it's not deleting the jobs. kubectl delete job $(kubectl get job -o=jsonpath='{.items[?(#.status.Failed==1)].metadata.name}')
#.status.Failed==1 <-- incorrect as JSON is case sensitive
#.status.failed==1 <-- correct
If you were to run incorrect version of this command on following Pods (to show that they failed and aren't still running to completion):
NAME READY STATUS RESTARTS AGE
job-four-9w5h9 0/1 Error 0 5s
job-one-n9trm 0/1 Completed 0 6s
job-three-nhqb6 0/1 Error 0 5s
job-two-hkz8r 0/1 Error 0 6s
You should get the following error :
error: resource(s) were provided, but no name, label selector, or --all flag specified
Above error will also show when there was no jobs passed to $ kubectl delete job.
Running correct version of this command should delete all jobs that failed:
job.batch "job-four" deleted
job.batch "job-three" deleted
job.batch "job-two" deleted
I encourage you to check additional resources:
StackOverflow.com: Questions: Kubectl list delete all completed jobs
Kubernetes.io: Doc
This one visually looks better for me:
kubectl delete job --field-selector=status.phase==Failed
#Dawid Kruk answer is excellent but working on a specific namespace only and not for all namespaces as I needed.
in order to solve it, I've created a simple bash script that gets all failed jobs and delete them -
# Delete failed jobs
failedJobs=$(kubectl get job -A -o=jsonpath='{range .items[?(#.status.failed>=1)]}{.metadata.name}{"\t"}{.metadata.namespace}{"\n"}{end}')
echo "$failedJobs" | while read each
do
array=($each)
jobName=${array[0]}
namespace=${array[1]}
echo "Debug: job name: $jobName is deleted on namespace $namespace"
kubectl delete job $jobName -n $namespace
done
When I try to create Deployment as Type Job, it's not pulling any image.
Below is .yaml:
apiVersion: batch/v1
kind: Job
metadata:
name: copyartifacts
spec:
backoffLimit: 1
template:
metadata:
name: copyartifacts
spec:
restartPolicy: "Never"
volumes:
- name: sharedvolume
persistentVolumeClaim:
claimName: shared-pvc
- name: dockersocket
hostPath:
path: /var/run/docker.sock
containers:
- name: copyartifacts
image: alpine:3.7
imagePullPolicy: Always
command: ["sh", "-c", "ls -l /shared; rm -rf /shared/*; ls -l /shared; while [ ! -d /shared/artifacts ]; do echo Waiting for artifacts to be copied; sleep 2; done; sleep 10; ls -l /shared/artifacts; "]
volumeMounts:
- mountPath: /shared
name: sharedvolume
Can you please guide here?
Regards,
Vikas
There could be two possible reasons for not seeing pod.
The pod hasn't been created yet.
The pod has completed it's task and terminated before you have noticed.
1. Pod hasn't been created:
If pod hasn't been created yet, you have to find out why the job failed to create pod. You can view job's events to see if there are any failure event. Use following command to describe a job.
kubectl describe job <job-name> -n <namespace>
Then, check the Events: field. There might be some events showing pod creation failure with respective reason.
2. Pod has completed and terminated:
Job's are used to perform one-time task rather than serving an application that require to maintain a desired state. When the task is complete, pod goes to completed state then terminate (but not deleted). If your Job is intended for a task that does not take much time, the pod may terminate after completing the task before you have noticed.
As the pod is terminated, kubectl get pods will not show that pod. However, you will able to see the pod using kubectl get pods -a command as it hasn't been deleted.
You can also describe the job and check for completion or success event.
if you use kind created the K8s cluster, all the cluster node run as docker. If you had reboot you computer or VM, the cluster (pod) ip address may change, leeding to the cluster node internet communication failed. In this case, see the cluster manager logs, it has error message. Job created, but pod not.
try to re-create the cluster, or change the node config about ip address.
On my test cluster, starting a pod seems to consistently take ~12 seconds, give or take one. I would like to know if that is reasonable, or if I am doing something wrong, either in configuring the pod, or in measuring the time, or configuring the cluster.
According to https://github.com/kubernetes/kubernetes/issues/3952 and https://medium.com/google-cloud/profiling-gke-startup-time-9052d81e0052, I believe what I am getting is excessively slow.
The way I measure startup is by running the following script and counting how many times it prints "Pending", and that is my startup time in seconds. Since due to the sleep command, I get almost exactly one "Pending" per second.
id=mypod1
tee job.yaml <<EOF
apiVersion: v1
kind: Pod
metadata:
name: clusterrunner-build-${id}
spec:
containers:
- name: clusterrunner-slave
image: jdanekrh/clusterrunner-slave
command: ["bash", "-c", "echo bof; sleep 5; echo lek"]
restartPolicy: Never
EOF
kubectl create -f job.yaml
while kubectl get pod/clusterrunner-build-${id} -o jsonpath='{.status.phase}' | grep Pending; do
sleep 1
done
kubectl logs -f po/clusterrunner-build-${id}
kubectl delete -f job.yaml