How do I run a Kubernetes Job once in N hours? - kubernetes

The Kubernetes documentation mentions that a CronJob supports the use case of:
Once at a specified point in time
But, I don't see any examples of how this would be possible. Specifically, I'm looking to kick off a job to run once in N hours.

According to documentation, CronJob uses the common Cron format of schedule:
Here are some examples:
schedule: "1 2-14 * * 0-1,5-6" (first minute of every hour from 2am to 2pm UTC on Sun,Mon,Fri,Sat)
schedule: "*/1 * * * *" (every minute)
CronJobs also have some limitations:
A cron job creates a job object about once per execution time of its
schedule. We say “about” because there are certain circumstances where
two jobs might be created, or no job might be created. We attempt to
make these rare, but do not completely prevent them. Therefore, jobs
should be idempotent.
If startingDeadlineSeconds is set to a large value or left unset (the
default) and if concurrencyPolicy is set to Allow, the jobs will
always run at least once.
Jobs may fail to run if the CronJob controller is not running or
broken for a span of time from before the start time of the CronJob to
start time plus startingDeadlineSeconds, or if the span covers
multiple start times and concurrencyPolicy does not allow concurrency.
For example, suppose a cron job is set to start at exactly 08:30:00
and its startingDeadlineSeconds is set to 10, if the CronJob
controller happens to be down from 08:29:00 to 08:42:00, the job will
not start. Set a longer startingDeadlineSeconds if starting later is
better than not starting at all.
The Cronjob is only responsible for creating Jobs that match its
schedule, and the Job in turn is responsible for the management of the
Pods it represents.
The other important thing is that Kubernetes uses UTC exclusively. Make sure you take that into account when you’re creating your schedule.
To run a job just once, you can use kubectl create -f job.yaml started by at command on the admin machine or on the master node.
echo "kubectl create -f job.yaml" | at midnight

It is just as a regular CronJob object but using a format for the cronjob expression to run at a specific point in time:
apiVersion: batch/v1beta1
kind: CronJob
metadata:
name: my-cronjob
namespace: kube-system
spec:
schedule: "7 7 7 7 6"
restartPolicy: OnFailure
jobTemplate:
...
for example this will run “At 07:07, Saturday 7th of July 2018.”
Next occurrence will be in 2029 so you have plenty of time to delete the cronJob object. That is, it will create a new job if you don't delete it as far and as I am aware there are no ways to avoid this.

Related

How to make cron job start after x seconds?

I think I am misunderstanding Kubernetes CronJobs. On the CKAD exam there was a question to have a CronJob run every minute, but it should start after an arbitrary amount of time. I don't see any properties for CronJobs or Jobs to have them start after a specific time. Should that be part of the cron string or am I completely misunderstanding?
You could do something like
#reboot sleep 60 && script.sh
though you don't mention boot time specifically. You can also add sleep to the crontab.
Another way is to create a systemd service (note: on systems with systemd installed)
[Unit]
Description=cronjob with delay
After=(some criteria)
[Service]
Type=oneshot
ExecStart=/pathto/script.sh
[Install]
WantedBy=
You can schedule your CronJob to start at specific date/time and than run every minute or however you would like to set it. There is a powerful online tool that can help you with it. For example:
0 0/10 10/1 ? * * *
will schedule your CronJob to run every 10 mins starting at 10h of the day. Or:
0 0/10 * ? * 6/1 *
will schedule your CronJob to run every 10 mins starting on Friday.
The important thing to keep in mind while using this particular approach is to be aware of the timezone that your cluster is running in:
All CronJob schedule: times are based on the timezone of the
kube-controller-manager.
If your control plane runs the kube-controller-manager in Pods or bare
containers, the timezone set for the kube-controller-manager container
determines the timezone that the cron job controller uses.
More info/examples about scheduling can be found below:
Cron schedule syntax
Running Automated Tasks with a CronJob
maybe you misunderstood the question, it was to terminate the job if it didn't complete in x seconds. you should use the .spec.activeDeadlineSeconds to satisfy the requirment.
you could do
sleep "seconds"; echo " * * * * * command" >> path/to/crontab
I feel like maybe the question is worded incorrectly. It probably meant something like every minute on every Saturday day.
* * * * SAT
Or every minute through range of hours
* 9-17 * * *
https://crontab.guru/examples.html

How to control interval between kubernetes cronjobs (i.e. cooldown period after completion)

If I set up a kubernetes cronjob with for example
spec:
schedule: "*/5 * * * *"
concurrencyPolicy: Forbid
then it will create a job every 5 minutes.
However if the job takes e.g. 4 minutes, then it will create another job 1 minute after the previous job completed.
Is there a way to make it create a job every 5 minutes after the previous job finished?
You might say; just make the schedule */9 * * * * to account for the 4 minutes the job takes, but the job might not be predictable like that.
Unfortunately there is no possibility within Kubernetes CronJob to specify a situation when the timer starts (for example 5 minutes) after a job is completed.
A word about cron:
The software utility cron is a time-based job scheduler in Unix-like computer operating systems. Users that set up and maintain software environments use cron to schedule jobs (commands or shell scripts) to run periodically at fixed times, dates, or intervals.
-- Wikipedia.org: Cron
The behavior of your CronJob within Kubernetes environment can be modified by:
As said Schedule in spec definition
schedule: "*/5 * * * *"
startingDeadline field that is optional and it describe a deadline in seconds for starting a job. If it doesn't start in that time period it will be counted as failed. After a 100 missed schedules it will no longer be scheduled.
Concurrency policy that will specify how concurrent executions of the same Job are going to be handled:
Allow - concurrency will be allowed
Forbid - if previous Job wasn't finished the new one will be skipped
Replace - current Job will be replaced with a new one
Suspend parameter if it is set to true, all subsequent executions are suspended. This setting does not apply to already started executions.
You could refer to official documentation: CronJobs
As it's unknown what type of Job you want to run you could try to:
Write a shell script in type of:
while true
do
HERE_RUN_YOUR_JOB_AND_WAIT_FOR_COMPLETION.sh
sleep 300 # ( 5 * 60 seconds )
done
Create an image that mimics usage of above script and use it as pod in Kubernetes.
Try to get logs from this pod if it's necessary as described here
Another way would be to create a pod that could connect to Kubernetes API.
Take a look on additional resources about Jobs:
Kubernetes.io: Fine parallel processing work queue
Kubernetes.io: Coarse parallel processing work queue
Please let me know if you have any questions to that.

Schedule cron job to never happen?

Here is part of my CronJob spec:
kind: CronJob
spec:
schedule: #{service.schedule}
For a specific environment a cron job is set up, but I never want it to run. Can I write some value into schedule: that will cause it to never run?
I haven't found any documentation for all supported syntax, but I am hoping for something like:
#never or #InABillionYears
#reboot doesn't guarantee that the job will never be run. It will actually be run always when your system is booted/rebooted and it may happen. It will be also run each time when cron daemon is restarted so you need to rely on that "typically it should not happen" on your system...
There are far more certain ways to ensure that a CronJob will never be run:
On Kubernetes level by suspending a job by setting its .spec.suspend field to true
You can easily set it using patch:
kubectl patch cronjobs <job-name> -p '{"spec" : {"suspend" : true }}'
On Cron level. Use a trick based on fact that crontab syntax is not strictly validated and set a date that you can be sure will never happen like 31th of February. Cron will accept that as it doesn't check day of the month in relation to value set in a month field. It just requires that you put valid numbers in both fields (1-31 and 1-12 respectively). You can set it to something like:
* * 31 2 *
which for Cron is perfectly valid value but we know that such a date is impossible and it will never happen.
kind: CronJob
spec:
suspend: true
Why do you need this to be a CronJob in the first place? If you never want it to run, you could specify a simple Job: https://kubernetes.io/docs/concepts/workloads/controllers/job/
I think you can use #reboot,
see: https://en.wikipedia.org/wiki/Cron
#reboot configures a job to run once when the daemon is started. Since cron is typically never restarted, this typically corresponds to the machine being booted.

HOW TO FIX: K8 job stoped and is not schedule any more

I had to stop a job in k8 by killing the pod, and now the job is not schedule anymore.
# Import
- name: cron-xml-import-foreman
schedule: "*/7 * * * *"
args:
- /bin/sh
- -c
/var/www/bash.sh; /usr/bin/php /var/www/import-products.php -->env=prod;
resources:
request_memory: "3Gi"
request_cpu: "2"
limit_memory: "4Gi"
limit_cpu: "4"
Error :
Warning FailedNeedsStart 5m34s (x7883 over 29h) cronjob-controller
Cannot determine if job needs to be started: Too many missed start
time (> 100). Set or decrease .spec.startingDeadlineSeconds or check
clock skew.
According to the official documentation:
If startingDeadlineSeconds is set to a large value or left unset (the
default) and if concurrencyPolicy is set to Allow, the jobs will
always run at least once.
A CronJob is counted as missed if it has failed to be created at its
scheduled time. For example, If concurrencyPolicy is set to Forbid and
a CronJob was attempted to be scheduled when there was a previous
schedule still running, then it would count as missed.
And regarding the concurrencyPolicy
It specifies how to treat concurrent executions of a job that is
created by this cron job.
Check your CronJob configuration and adjust those values accordingly.
Please let me know if that helped.

How do I stop a CronJob from recreating failed Jobs?

When for whatever reasons I delete the pod running the Job that was started by a CronJob, I immediately see a new pod being created. It is only once I delete something like six times the backoffLimit number of pods, that new ones stop being created.
Of course, if I'm actively monitoring the process, I can delete the CronJob, but what if the Pod inside the job fails when I'm not looking? I would like it not to be recreated.
How can I stop the CronJob from persisting in creating new jobs (or pods?), and wait until the next scheduled time if the current job/pod failed? Is there something similar to Jobs' backoffLimit, but for CronJobs?
Set startingDeadlineSeconds to a large value or left unset (the default).
At the same time set .spec.concurrencyPolicy as Forbid and the CronJobs skips the new job run while previous created job is still running.
If startingDeadlineSeconds is set to a large value or left unset (the default) and if concurrencyPolicy is set to Forbid, the job will not be run if failed.
Concurrent policy field you can add to specification to defintion of your CronJob (.spec.concurrencyPolicy), but this is optional.
It specifies how to treat concurrent executions of a job that is created by this CronJob. The spec may specify only one of these three concurrency policies:
Allow (default) - The cron job allows concurrently running jobs
Forbid - The cron job does not allow concurrent runs; if it is time for a new job run and the previous job run hasn’t finished yet, the cron job skips the new job run
Replace - If it is time for a new job run and the previous job run hasn’t finished yet, the cron job replaces the currently running job run with a new job run
It is good to know that currency policy applies just to the jobs created by the same CronJob.
If there are multiple CronJobs, their respective jobs are always allowed to run concurrently.
A CronJob is counted as missed if it has failed to be created at its scheduled time. For example, If concurrencyPolicy is set to Forbid and a CronJob was attempted to be scheduled when there was a previous schedule still running, then it would count as missed.
For every CronJob, the CronJob controller checks how many schedules it missed in the duration from its last scheduled time until now. If there are more than 100 missed schedules, then it does not start the job and logs the error
More information you can find here: CronJobs and AutomatedTask.
I hope it helps.
CronJob creates a job by a "backoffLimit" with a default value (6) in your case, and restart policy by default is (Always)
Better to make backoffLimit > (0) and make restart policy = (Never) and increase startingDeadlineSeconds to be lower than or equal to your interval or you can customize it up on your request to control the run time of each CronJob run
Additionally, you may stop "concurrencyPolicy" >> (Forbid)