Say that I have a job history limit > 1, is there a way to use kubectl to find which jobs that have been spawned by a CronJob?
Use label.
$kubectl get jobs -n namespace -l created-by=cronjob
created-by=cronjob which define at your cronjob.
apiVersion: batch/v1
kind: CronJob
metadata:
name: hello
spec:
schedule: "* * * * *"
jobTemplate:
metadata:
labels:
created-by: cronjob
spec:
template:
spec:
containers:
- name: hello
image: busybox
imagePullPolicy: IfNotPresent
command:
- /bin/sh
- -c
- date; echo Hello from the Kubernetes cluster
restartPolicy: OnFailure
Ideally you would use the --field-selector label to get all jobs belonging to a particular CronJob, but field selector does not support all the fields. The best way I could find is using jsonPath:
Substitute CronJobName to the name of your cronjob to get all jobs that belong to that CronJob
kubectl get jobs $(kubectl get jobs -o=jsonpath='{.items[?(#.metadata.ownerReferences[*].name=="CronJobName")].metadata.name}')
Related
I am trying to create a cronjob , I have written a Springboot application for this and have created a abc-dev.yml file for application configuration
error: unable to recognize "src/java/k8s/abc-dev.yml": no matches for kind "CronJob" in version "apps/v1"
apiVersion: apps/v1
kind: CronJob
metadata:
name: abc-cron-job
spec:
schedule: "* * * * *"
jobTemplate:
spec:
template:
spec:
container:
- name: abc-cron-job
image: busybox
imagePullPolicy: IfNotPresent
command:
- /bin/sh
- -c
- date; echo Hello from the Kubernetes cluster
restartPolicy: OnFailure
If you are running kubernetes 1.20 or lower, the correct apiVersion value is:
apiVersion: batch/v1beta1
If you are running kubernetes 1.21 or higher, its
apiVersion: batch/v1
You can check the api-version of a resource with the
kubectl api-resources
command.
In this case:
kubectl api-resources | grep cronjob | awk -v N=3 '{print $N}'
The output is 'batch/v1'.
Cronjob belongs to batch/v1 k8s api. You should check api version before creating resources in any case they sometimes change.
https://kubernetes.io/docs/concepts/workloads/controllers/cron-jobs/
I have a small instance of influxdb running in my kubernetes cluster.
The data of that instance is stored in a persistent storage.
But I also want to run the backup command from influx at scheduled interval.
influxd backup -portable /backuppath
What I do now is exec into the pod and run it manually.
Is there a way that I can do this automatically?
You can consider running a CronJob with bitnami kubectl which will execute the backup command. This is the same as exec into the pod and run except now you automate it with CronJob.
CronJob is the way to go here. It acts more or less like a crontab, but for Kubernetes.
As an example you could use this
apiVersion: batch/v1
kind: CronJob
metadata:
name: backup
spec:
schedule: 0 8 * * *
jobTemplate:
spec:
template:
spec:
containers:
- name: influxdb-backup
image: influxdb
imagePullPolicy: IfNotPresent
command: ["/bin/sh"]
args:
- "-c"
- "influxd backup -portable /backuppath"
restartPolicy: Never
This will create a Job, everyday at 08:00, executing influxd backup -portable /backuppath. Of course, you have to edit it accordingly, to work on your environment.
This is the solution I have used for this question
apiVersion: v1
kind: ConfigMap
metadata:
name: cm-backupscript
namespace: influx
data:
backupscript.sh: |
#!/bin/bash
echo 'getting pod name'
podName=$(kubectl get pods -n influx --field-selector=status.phase==Running --output=jsonpath={.items..metadata.name})
echo $podName
#echo 'create backup'
kubectl exec -it $podName -n influx -- /mnt/influxBackupScript/influxbackup.sh
echo 'done'
---
apiVersion: batch/v1beta1
kind: CronJob
metadata:
name: backup-cron
namespace: influx
spec:
schedule: "0 2 * * *"
jobTemplate:
spec:
template:
spec:
affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: kubernetes.io/arch
operator: In
values:
- amd64
volumes:
- name: backup-script
configMap:
name: cm-backupscript
defaultMode: 0777
containers:
- name: kubectl
image: bitnami/kubectl:latest
command:
- /bin/sh
- -c
- /mnt/scripts/backupscript.sh
volumeMounts:
- name: backup-script
mountPath: "/mnt/scripts"
restartPolicy: Never
You can either run it as a cronjob and setup the image to be able to connect to the DB, or you can sidecar it alongside your db pod, and set it to run the cron image (i.e. will run as a mostly-idle container in the same pod as your DB)
Is it possible to restart pods automatically based on the time?
For example, I would like to restart the pods of my cluster every morning at 8.00 AM.
Use a cronjob, but not to run your pods, but to schedule a Kubernetes API command that will restart the deployment everyday (kubectl rollout restart). That way if something goes wrong, the old pods will not be down or removed.
Rollouts create new ReplicaSets, and wait for them to be up, before killing off old pods, and rerouting the traffic. Service will continue uninterrupted.
You have to setup RBAC, so that the Kubernetes client running from inside the cluster has permissions to do needed calls to the Kubernetes API.
---
# Service account the client will use to reset the deployment,
# by default the pods running inside the cluster can do no such things.
kind: ServiceAccount
apiVersion: v1
metadata:
name: deployment-restart
namespace: <YOUR NAMESPACE>
---
# allow getting status and patching only the one deployment you want
# to restart
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
name: deployment-restart
namespace: <YOUR NAMESPACE>
rules:
- apiGroups: ["apps", "extensions"]
resources: ["deployments"]
resourceNames: ["<YOUR DEPLOYMENT NAME>"]
verbs: ["get", "patch", "list", "watch"] # "list" and "watch" are only needed
# if you want to use `rollout status`
---
# bind the role to the service account
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: deployment-restart
namespace: <YOUR NAMESPACE>
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: Role
name: deployment-restart
subjects:
- kind: ServiceAccount
name: deployment-restart
namespace: <YOUR NAMESPACE>
And the cronjob specification itself:
apiVersion: batch/v1beta1
kind: CronJob
metadata:
name: deployment-restart
namespace: <YOUR NAMESPACE>
spec:
concurrencyPolicy: Forbid
schedule: '0 8 * * *' # cron spec of time, here, 8 o'clock
jobTemplate:
spec:
backoffLimit: 2 # this has very low chance of failing, as all this does
# is prompt kubernetes to schedule new replica set for
# the deployment
activeDeadlineSeconds: 600 # timeout, makes most sense with
# "waiting for rollout" variant specified below
template:
spec:
serviceAccountName: deployment-restart # name of the service
# account configured above
restartPolicy: Never
containers:
- name: kubectl
image: bitnami/kubectl # probably any kubectl image will do,
# optionaly specify version, but this
# should not be necessary, as long the
# version of kubectl is new enough to
# have `rollout restart`
command:
- 'kubectl'
- 'rollout'
- 'restart'
- 'deployment/<YOUR DEPLOYMENT NAME>'
Optionally, if you want the cronjob to wait for the deployment to roll out, change the cronjob command to:
command:
- bash
- -c
- >-
kubectl rollout restart deployment/<YOUR DEPLOYMENT NAME> &&
kubectl rollout status deployment/<YOUR DEPLOYMENT NAME>
Another quick and dirty option for a pod that has a restart policy of Always (which cron jobs are not supposed to handle - see creating a cron job spec pod template) is a livenessProbe that simply tests the time and restarts the pod on a specified schedule
ex. After startup, wait an hour, then check hour every minute, if hour is 3(AM) fail probe and restart, otherwise pass
livenessProbe:
exec:
command:
- exit $(test $(date +%H) -eq 3 && echo 1 || echo 0)
failureThreshold: 1
initialDelaySeconds: 3600
periodSeconds: 60
Time granularity is up to how you return the date and test ;)
Of course this does not work if you are already utilizing the liveness probe as an actual liveness probe ¯\_(ツ)_/¯
I borrowed idea from #Ryan Lowe but modified it a bit. It will restart pod older than 24 hours
livenessProbe:
exec:
command:
- bin/sh
- -c
- "end=$(date -u +%s);start=$(stat -c %Z /proc/1 | awk '{print int($1)}'); test $(($end-$start)) -lt 86400"
There's a specific resource for that: CronJob
Here an example:
apiVersion: batch/v1beta1
kind: CronJob
metadata:
name: your-cron
spec:
schedule: "*/20 8-19 * * 1-5"
concurrencyPolicy: Forbid
jobTemplate:
spec:
template:
metadata:
labels:
app: your-periodic-batch-job
spec:
containers:
- name: my-image
image: your-image
imagePullPolicy: IfNotPresent
restartPolicy: OnFailure
change spec.concurrencyPolicy to Replace if you want to replace the old pod when starting a new pod. Using Forbid, the new pod creation will be skip if the old pod is still running.
According to cronjob-in-kubernetes-to-restart-delete-the-pod-in-a-deployment
you could create a kind: CronJob with a jobTemplate having containers. So your CronJob will start those containers with a activeDeadlineSeconds of one day (until restart). According to you example, it will be then schedule: 0 8 * * ? for 8:00AM
We were able to do this by modifying the manifest (passing a random param every 3 hours) file of deployment from a CRON job:
We specifically used Spinnaker for triggering deployments:
We created a CRON job in Spinnaker like below:
Configuration step looks like:
The Patch Manifest looks like: (K8S restarts PODS when YAML changes, to counter that check bottom of post)
As there can be a case where all pods can restart at a same time, causing downtime, we have a policy for Rolling Restart where maxUnavailablePods is 0%
spec:
# replicas: 1
strategy:
type: RollingUpdate
rollingUpdate:
maxSurge: 50%
maxUnavailable: 0%
This spawns new pods and then terminates old ones.
I'm trying to set the minimum-container-ttl-duration property on a Kubernetes CronJob. I see a bunch of properties like this that appear to be configurable, but the documentation doesn't appear to show where, in the yml file, they can actually be set.
In this example yml, where would I put this property?
apiVersion: batch/v1beta1
kind: CronJob
metadata:
name: hello
spec:
schedule: "*/1 * * * *"
jobTemplate:
spec:
template:
spec:
containers:
- name: hello
image: busybox
args:
- /bin/sh
- -c
- date; echo Hello from the Kubernetes cluster
restartPolicy: OnFailure
minimum-container-ttl-duration is not a property on CronJob but is a Node-level property set via a command line parameter: kubelet ... --minimum-container-ttl-duration=x.
https://kubernetes.io/docs/concepts/cluster-administration/kubelet-garbage-collection/#user-configuration:
minimum-container-ttl-duration, minimum age for a finished container before it is garbage collected. Default is 0 minute, which means every finished container will be garbage collected.
The usage of this flag is deprecated.
Is there a way to automatically remove completed Jobs besides making a CronJob to clean up completed Jobs?
The K8s Job Documentation states that the intended behavior of completed Jobs is for them to remain in a completed state until manually deleted. Because I am running thousands of Jobs a day via CronJobs and I don't want to keep completed Jobs around.
You can now set history limits, or disable history altogether, so that failed or successful CronJobs are not kept around indefinitely. See my answer here. Documentation is here.
To set the history limits:
The .spec.successfulJobsHistoryLimit and .spec.failedJobsHistoryLimit fields are optional. These fields specify how many completed and failed jobs should be kept. By default, they are set to 3 and 1 respectively. Setting a limit to 0 corresponds to keeping none of the corresponding kind of jobs after they finish.
The config with 0 limits would look like:
apiVersion: batch/v1beta1
kind: CronJob
metadata:
name: hello
spec:
schedule: "*/1 * * * *"
successfulJobsHistoryLimit: 0
failedJobsHistoryLimit: 0
jobTemplate:
spec:
template:
spec:
containers:
- name: hello
image: busybox
args:
- /bin/sh
- -c
- date; echo Hello from the Kubernetes cluster
restartPolicy: OnFailure
This is possible from version 1.12 Alpha with ttlSecondsAfterFinished. An example from Clean Up Finished Jobs Automatically:
apiVersion: batch/v1
kind: Job
metadata:
name: pi-with-ttl
spec:
ttlSecondsAfterFinished: 100
template:
spec:
containers:
- name: pi
image: perl
command: ["perl", "-Mbignum=bpi", "-wle", "print bpi(2000)"]
restartPolicy: Never
Another way using a field-selector:
kubectl delete jobs --field-selector status.successful=1
That could be executed in a cronjob, similar to others answers.
Create a service account, something like my-sa-name
Create a role with list and delete permissions for the resource jobs
Attach the role in the service account (rolebinding)
Create a cronjob that will use the service account that will check for completed jobs and delete them
# 1. Create a service account
apiVersion: v1
kind: ServiceAccount
metadata:
name: my-sa-name
namespace: default
---
# 2. Create a role
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
namespace: default
name: my-completed-jobs-cleaner-role
rules:
- apiGroups: [""]
resources: ["jobs"]
verbs: ["list", "delete"]
---
# 3. Attach the role to the service account
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: my-completed-jobs-cleaner-rolebinding
namespace: default
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: Role
name: my-completed-jobs-cleaner-role
subjects:
- kind: ServiceAccount
name: my-sa-name
namespace: default
---
# 4. Create a cronjob (with a crontab schedule) using the service account to check for completed jobs
apiVersion: batch/v1beta1
kind: CronJob
metadata:
name: jobs-cleanup
spec:
schedule: "*/30 * * * *"
jobTemplate:
spec:
template:
spec:
serviceAccountName: my-sa-name
containers:
- name: kubectl-container
image: bitnami/kubectl:latest
# I'm using bitnami kubectl, because the suggested kubectl image didn't had the `field-selector` option
command: ["sh", "-c", "kubectl delete jobs --field-selector status.successful=1"]
restartPolicy: Never
I've found the below to work
To remove failed jobs:
kubectl delete job $(kubectl get jobs | awk '$3 ~ 0' | awk '{print $1}')
To remove completed jobs:
kubectl delete job $(kubectl get jobs | awk '$3 ~ 1' | awk '{print $1}')
i'm using wernight/kubectl's kubectl image
scheduled a cron deleting anything that is
completed
2 - 9 days old (so I have 2 days to review any failed jobs)
it runs every 30mins so i'm not accounting for jobs that are 10+ days old
apiVersion: batch/v1beta1
kind: CronJob
metadata:
name: cleanup
spec:
schedule: "*/30 * * * *"
jobTemplate:
spec:
template:
spec:
containers:
- name: kubectl-runner
image: wernight/kubectl
command: ["sh", "-c", "kubectl get jobs | awk '$4 ~ /[2-9]d$/ || $3 ~ 1' | awk '{print $1}' | xargs kubectl delete job"]
restartPolicy: Never
I recently built a kubernetes-operator to do this task.
After deploy it will monitor selected namespace and delete completed jobs/pods if they completed without errors/restarts.
https://github.com/lwolf/kube-cleanup-operator
Using jsonpath:
kubectl delete job $(kubectl get job -o=jsonpath='{.items[?(#.status.succeeded==1)].metadata.name}')
As stated in the documentation "It is up to the user to delete old jobs", see http://kubernetes.io/docs/user-guide/jobs/#job-termination-and-cleanup
I would run a pod to do this cleanup based on job name and certain conditions, thus letting kubernetes at least take care of the availability of your process here. You could run a recurring job for this (assuming you run kubernetes 1.5).
A simple way to delete them by running a cron job:
kubectl get jobs --all-namespaces | sed '1d' | awk '{ print $2, "--namespace", $1 }' | while read line; do kubectl delete jobs $line; done