Here is part of my CronJob spec:
kind: CronJob
spec:
schedule: #{service.schedule}
For a specific environment a cron job is set up, but I never want it to run. Can I write some value into schedule: that will cause it to never run?
I haven't found any documentation for all supported syntax, but I am hoping for something like:
#never or #InABillionYears
#reboot doesn't guarantee that the job will never be run. It will actually be run always when your system is booted/rebooted and it may happen. It will be also run each time when cron daemon is restarted so you need to rely on that "typically it should not happen" on your system...
There are far more certain ways to ensure that a CronJob will never be run:
On Kubernetes level by suspending a job by setting its .spec.suspend field to true
You can easily set it using patch:
kubectl patch cronjobs <job-name> -p '{"spec" : {"suspend" : true }}'
On Cron level. Use a trick based on fact that crontab syntax is not strictly validated and set a date that you can be sure will never happen like 31th of February. Cron will accept that as it doesn't check day of the month in relation to value set in a month field. It just requires that you put valid numbers in both fields (1-31 and 1-12 respectively). You can set it to something like:
* * 31 2 *
which for Cron is perfectly valid value but we know that such a date is impossible and it will never happen.
kind: CronJob
spec:
suspend: true
Why do you need this to be a CronJob in the first place? If you never want it to run, you could specify a simple Job: https://kubernetes.io/docs/concepts/workloads/controllers/job/
I think you can use #reboot,
see: https://en.wikipedia.org/wiki/Cron
#reboot configures a job to run once when the daemon is started. Since cron is typically never restarted, this typically corresponds to the machine being booted.
Related
If I set up a kubernetes cronjob with for example
spec:
schedule: "*/5 * * * *"
concurrencyPolicy: Forbid
then it will create a job every 5 minutes.
However if the job takes e.g. 4 minutes, then it will create another job 1 minute after the previous job completed.
Is there a way to make it create a job every 5 minutes after the previous job finished?
You might say; just make the schedule */9 * * * * to account for the 4 minutes the job takes, but the job might not be predictable like that.
Unfortunately there is no possibility within Kubernetes CronJob to specify a situation when the timer starts (for example 5 minutes) after a job is completed.
A word about cron:
The software utility cron is a time-based job scheduler in Unix-like computer operating systems. Users that set up and maintain software environments use cron to schedule jobs (commands or shell scripts) to run periodically at fixed times, dates, or intervals.
-- Wikipedia.org: Cron
The behavior of your CronJob within Kubernetes environment can be modified by:
As said Schedule in spec definition
schedule: "*/5 * * * *"
startingDeadline field that is optional and it describe a deadline in seconds for starting a job. If it doesn't start in that time period it will be counted as failed. After a 100 missed schedules it will no longer be scheduled.
Concurrency policy that will specify how concurrent executions of the same Job are going to be handled:
Allow - concurrency will be allowed
Forbid - if previous Job wasn't finished the new one will be skipped
Replace - current Job will be replaced with a new one
Suspend parameter if it is set to true, all subsequent executions are suspended. This setting does not apply to already started executions.
You could refer to official documentation: CronJobs
As it's unknown what type of Job you want to run you could try to:
Write a shell script in type of:
while true
do
HERE_RUN_YOUR_JOB_AND_WAIT_FOR_COMPLETION.sh
sleep 300 # ( 5 * 60 seconds )
done
Create an image that mimics usage of above script and use it as pod in Kubernetes.
Try to get logs from this pod if it's necessary as described here
Another way would be to create a pod that could connect to Kubernetes API.
Take a look on additional resources about Jobs:
Kubernetes.io: Fine parallel processing work queue
Kubernetes.io: Coarse parallel processing work queue
Please let me know if you have any questions to that.
I had to stop a job in k8 by killing the pod, and now the job is not schedule anymore.
# Import
- name: cron-xml-import-foreman
schedule: "*/7 * * * *"
args:
- /bin/sh
- -c
/var/www/bash.sh; /usr/bin/php /var/www/import-products.php -->env=prod;
resources:
request_memory: "3Gi"
request_cpu: "2"
limit_memory: "4Gi"
limit_cpu: "4"
Error :
Warning FailedNeedsStart 5m34s (x7883 over 29h) cronjob-controller
Cannot determine if job needs to be started: Too many missed start
time (> 100). Set or decrease .spec.startingDeadlineSeconds or check
clock skew.
According to the official documentation:
If startingDeadlineSeconds is set to a large value or left unset (the
default) and if concurrencyPolicy is set to Allow, the jobs will
always run at least once.
A CronJob is counted as missed if it has failed to be created at its
scheduled time. For example, If concurrencyPolicy is set to Forbid and
a CronJob was attempted to be scheduled when there was a previous
schedule still running, then it would count as missed.
And regarding the concurrencyPolicy
It specifies how to treat concurrent executions of a job that is
created by this cron job.
Check your CronJob configuration and adjust those values accordingly.
Please let me know if that helped.
When for whatever reasons I delete the pod running the Job that was started by a CronJob, I immediately see a new pod being created. It is only once I delete something like six times the backoffLimit number of pods, that new ones stop being created.
Of course, if I'm actively monitoring the process, I can delete the CronJob, but what if the Pod inside the job fails when I'm not looking? I would like it not to be recreated.
How can I stop the CronJob from persisting in creating new jobs (or pods?), and wait until the next scheduled time if the current job/pod failed? Is there something similar to Jobs' backoffLimit, but for CronJobs?
Set startingDeadlineSeconds to a large value or left unset (the default).
At the same time set .spec.concurrencyPolicy as Forbid and the CronJobs skips the new job run while previous created job is still running.
If startingDeadlineSeconds is set to a large value or left unset (the default) and if concurrencyPolicy is set to Forbid, the job will not be run if failed.
Concurrent policy field you can add to specification to defintion of your CronJob (.spec.concurrencyPolicy), but this is optional.
It specifies how to treat concurrent executions of a job that is created by this CronJob. The spec may specify only one of these three concurrency policies:
Allow (default) - The cron job allows concurrently running jobs
Forbid - The cron job does not allow concurrent runs; if it is time for a new job run and the previous job run hasn’t finished yet, the cron job skips the new job run
Replace - If it is time for a new job run and the previous job run hasn’t finished yet, the cron job replaces the currently running job run with a new job run
It is good to know that currency policy applies just to the jobs created by the same CronJob.
If there are multiple CronJobs, their respective jobs are always allowed to run concurrently.
A CronJob is counted as missed if it has failed to be created at its scheduled time. For example, If concurrencyPolicy is set to Forbid and a CronJob was attempted to be scheduled when there was a previous schedule still running, then it would count as missed.
For every CronJob, the CronJob controller checks how many schedules it missed in the duration from its last scheduled time until now. If there are more than 100 missed schedules, then it does not start the job and logs the error
More information you can find here: CronJobs and AutomatedTask.
I hope it helps.
CronJob creates a job by a "backoffLimit" with a default value (6) in your case, and restart policy by default is (Always)
Better to make backoffLimit > (0) and make restart policy = (Never) and increase startingDeadlineSeconds to be lower than or equal to your interval or you can customize it up on your request to control the run time of each CronJob run
Additionally, you may stop "concurrencyPolicy" >> (Forbid)
Because Kubernetes handles situations where there's a typo in the job spec, and therefore a container image can't be found, by leaving the job in a running state forever, I've got a process that monitors job events to detect cases like this and deletes the job when one occurs.
I'd prefer to just stop the job so there's a record of it. Is there a way to stop a job?
1) According to the K8S documentation here.
Finished Jobs are usually no longer needed in the system. Keeping them around in the system will put pressure on the API server. If the Jobs are managed directly by a higher level controller, such as CronJobs, the Jobs can be cleaned up by CronJobs based on the specified capacity-based cleanup policy.
Here are the details for the failedJobsHistoryLimit property in the CronJobSpec.
This is another way of retaining the details of the failed job for a specific duration. The failedJobsHistoryLimit property can be set based on the approximate number of jobs run per day and the number of days the logs have to be retained. Agree that the Jobs will be still there and put pressure on the API server.
This is interesting. Once the job completes with failure as in the case of a wrong typo for image, the pod is getting deleted and the resources are not blocked or consumed anymore. Not sure exactly what kubectl job stop will achieve in this case. But, when the Job with a proper image is run with success, I can still see the pod in kubectl get pods.
2) Another approach without using the CronJob is to specify the ttlSecondsAfterFinished as mentioned here.
Another way to clean up finished Jobs (either Complete or Failed) automatically is to use a TTL mechanism provided by a TTL controller for finished resources, by specifying the .spec.ttlSecondsAfterFinished field of the Job.
Not really, no such mechanism exists in Kubernetes yet afaik.
You can workaround is to ssh into the machine and run a: (if you're are using Docker)
# Save the logs
$ docker log <container-id-that-is-running-your-job> 2>&1 > save.log
$ docker stop <main-container-id-for-your-job>
It's better to stream log with something like Fluentd, or logspout, or Filebeat and forward the logs to an ELK or EFK stack.
In any case, I've opened this
You can suspend cronjobs by using the suspend attribute. From the Kubernetes documentation:
https://kubernetes.io/docs/tasks/job/automated-tasks-with-cron-jobs/#suspend
Documentation says:
The .spec.suspend field is also optional. If it is set to true, all
subsequent executions are suspended. This setting does not apply to
already started executions. Defaults to false.
So, to pause a cron you could:
run and edit "suspend" from False to True.
kubectl edit cronjob CRON_NAME (if not in default namespace, then add "-n NAMESPACE_NAME" at the end)
you could potentially create a loop using "for" or whatever you like, and have them all changed at once.
you could just save the yaml file locally and then just run:
kubectl create -f cron_YAML
and this would recreate the cron.
The other answers hint around the .spec.suspend solution for the CronJob API, which works, but since the OP asked specifically about Jobs it is worth noting the solution that does not require a CronJob.
As of Kubernetes 1.21, there alpha support for the .spec.suspend field in the Job API as well, (see docs here). The feature is behind the SuspendJob feature gate.
The Kubernetes documentation mentions that a CronJob supports the use case of:
Once at a specified point in time
But, I don't see any examples of how this would be possible. Specifically, I'm looking to kick off a job to run once in N hours.
According to documentation, CronJob uses the common Cron format of schedule:
Here are some examples:
schedule: "1 2-14 * * 0-1,5-6" (first minute of every hour from 2am to 2pm UTC on Sun,Mon,Fri,Sat)
schedule: "*/1 * * * *" (every minute)
CronJobs also have some limitations:
A cron job creates a job object about once per execution time of its
schedule. We say “about” because there are certain circumstances where
two jobs might be created, or no job might be created. We attempt to
make these rare, but do not completely prevent them. Therefore, jobs
should be idempotent.
If startingDeadlineSeconds is set to a large value or left unset (the
default) and if concurrencyPolicy is set to Allow, the jobs will
always run at least once.
Jobs may fail to run if the CronJob controller is not running or
broken for a span of time from before the start time of the CronJob to
start time plus startingDeadlineSeconds, or if the span covers
multiple start times and concurrencyPolicy does not allow concurrency.
For example, suppose a cron job is set to start at exactly 08:30:00
and its startingDeadlineSeconds is set to 10, if the CronJob
controller happens to be down from 08:29:00 to 08:42:00, the job will
not start. Set a longer startingDeadlineSeconds if starting later is
better than not starting at all.
The Cronjob is only responsible for creating Jobs that match its
schedule, and the Job in turn is responsible for the management of the
Pods it represents.
The other important thing is that Kubernetes uses UTC exclusively. Make sure you take that into account when you’re creating your schedule.
To run a job just once, you can use kubectl create -f job.yaml started by at command on the admin machine or on the master node.
echo "kubectl create -f job.yaml" | at midnight
It is just as a regular CronJob object but using a format for the cronjob expression to run at a specific point in time:
apiVersion: batch/v1beta1
kind: CronJob
metadata:
name: my-cronjob
namespace: kube-system
spec:
schedule: "7 7 7 7 6"
restartPolicy: OnFailure
jobTemplate:
...
for example this will run “At 07:07, Saturday 7th of July 2018.”
Next occurrence will be in 2029 so you have plenty of time to delete the cronJob object. That is, it will create a new job if you don't delete it as far and as I am aware there are no ways to avoid this.