Iam trying to create a cron job like this:
apiVersion: batch/v1beta1
kind: CronJob
metadata:
name: postgres-backup
spec:
# Backup the database every day at 2AM
schedule: "*/1 * * * *"
jobTemplate:
spec:
template:
spec:
containers:
- name: postgres-backup
image: postgres:10.4
command: ["/bin/bash"]
args: ["-c", 'PGPASSWORD="$PGPASS" pg_dump -U postgresadmin -h postgres example > /var/backups/backup-$(date +"%m-%d-%Y-%H-%M").sql']
env:
- name: PGPASS
value: admin123
volumeMounts:
- mountPath: /var/backups
name: postgres-pv-volume
restartPolicy: Never
volumes:
- name: postgres-pv-volume
hostPath:
# Ensure the file directory is created.
path: /var/volumes/postgres-backups
type: DirectoryOrCreate
When iam running the crone job the pods getting created with status created. But i was not able to see the backup files in my postgres pod. I think the backup file getting created in the same cronjob pod but how to access the completed pod.
Like mdaniel mentioned in the comment, you are using the volume type hostPath, which is a path on the node, so I guess in order for you to find the file being created, you need to login to that node and look under that path on the node.
And personally I don't think you should try to store it in the pod at all, since Job is run to complete, which means you are not able to retrieve anything in it when the job is completed.
Related
I have a small instance of influxdb running in my kubernetes cluster.
The data of that instance is stored in a persistent storage.
But I also want to run the backup command from influx at scheduled interval.
influxd backup -portable /backuppath
What I do now is exec into the pod and run it manually.
Is there a way that I can do this automatically?
You can consider running a CronJob with bitnami kubectl which will execute the backup command. This is the same as exec into the pod and run except now you automate it with CronJob.
CronJob is the way to go here. It acts more or less like a crontab, but for Kubernetes.
As an example you could use this
apiVersion: batch/v1
kind: CronJob
metadata:
name: backup
spec:
schedule: 0 8 * * *
jobTemplate:
spec:
template:
spec:
containers:
- name: influxdb-backup
image: influxdb
imagePullPolicy: IfNotPresent
command: ["/bin/sh"]
args:
- "-c"
- "influxd backup -portable /backuppath"
restartPolicy: Never
This will create a Job, everyday at 08:00, executing influxd backup -portable /backuppath. Of course, you have to edit it accordingly, to work on your environment.
This is the solution I have used for this question
apiVersion: v1
kind: ConfigMap
metadata:
name: cm-backupscript
namespace: influx
data:
backupscript.sh: |
#!/bin/bash
echo 'getting pod name'
podName=$(kubectl get pods -n influx --field-selector=status.phase==Running --output=jsonpath={.items..metadata.name})
echo $podName
#echo 'create backup'
kubectl exec -it $podName -n influx -- /mnt/influxBackupScript/influxbackup.sh
echo 'done'
---
apiVersion: batch/v1beta1
kind: CronJob
metadata:
name: backup-cron
namespace: influx
spec:
schedule: "0 2 * * *"
jobTemplate:
spec:
template:
spec:
affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: kubernetes.io/arch
operator: In
values:
- amd64
volumes:
- name: backup-script
configMap:
name: cm-backupscript
defaultMode: 0777
containers:
- name: kubectl
image: bitnami/kubectl:latest
command:
- /bin/sh
- -c
- /mnt/scripts/backupscript.sh
volumeMounts:
- name: backup-script
mountPath: "/mnt/scripts"
restartPolicy: Never
You can either run it as a cronjob and setup the image to be able to connect to the DB, or you can sidecar it alongside your db pod, and set it to run the cron image (i.e. will run as a mostly-idle container in the same pod as your DB)
I have a 3 node K8 v1.21 cluster in AWS and looking for SOLID config to run a script using a cronjob. I have seen many documents on here and Google using cronjob and hostPath to Persistent Volumes/Claims to using ConfigMaps, the list goes one.
I keep getting "Back-off restarting failed container/CrashLoopBackOff" errors.
Any help is much appreciated.
cronjob.yaml
The script I am trying to run is basic for testing only
#! /bin/<br/>
kubectl create deployment nginx --image=nginx
Still getting the same error.
kubectl describe pod/xxxx
This hostPath in AWS cluster created using eksctl works.
apiVersion: v1
kind: Pod
metadata:
name: redis-hostpath
spec:
containers:
- image: redis
name: redis-container
volumeMounts:
- mountPath: /test-mnt
name: test-vol
volumes:
- name: test-vol
hostPath:
path: /test-vol
UPDATE
Tried running your config in GCP on a fresh cluster. Only thing I changed was the /home/script.sh to /home/admin/script.sh
Did you test this on your cluster?
Warning FailedPostStartHook 5m27s kubelet Exec lifecycle hook ([/home/mchung/script.sh]) for Container "busybox" in Pod "dumb-job-1635012900-qphqr_default(305c4ed4-08d1-4585-83e0-37a2bc008487)" failed - error: rpc error: code = Unknown desc = failed to exec in container: failed to create exec "0f9f72ccc6279542f18ebe77f497e8c2a8fd52f8dfad118c723a1ba025b05771": cannot exec in a deleted state: unknown, message: ""
Normal Killing 5m27s kubelet FailedPostStartHook
Assuming you're running it in a remote multi-node cluster (since you mentioned AWS in your question), hostPath is NOT an option there for volume mount. Your best choice would be to use a ConfigMap and use it as volume mount.
apiVersion: v1
kind: ConfigMap
metadata:
name: redis-script
data:
script.sh: |
# write down your script here
And then:
apiVersion: batch/v1
kind: CronJob
metadata:
name: redis-job
spec:
schedule: '*/5 * * * *'
jobTemplate:
spec:
template:
spec:
containers:
- name: redis-container
image: redis
args:
- /bin/sh
- -c
- /home/user/script.sh
volumeMounts:
- name: redis-data
mountPath: /home/user/script.sh
subPath: script.sh
volumes:
- name: redis-data
configMap:
name: redis-script
Hope this helps. Let me know if you face any difficulties.
Update:
I think you're doing something wrong. kubectl isn't something you should run from another container / pod. Because it requires the necessary binary to be existed into that container and an appropriate context set. I'm putting a working manifest below for you to understand the whole concept of running a script as a part of cron job:
apiVersion: v1
kind: ConfigMap
metadata:
name: script-config
data:
script.sh: |-
name=StackOverflow
echo "I love $name <3"
---
apiVersion: batch/v1
kind: CronJob
metadata:
name: dumb-job
spec:
schedule: '*/1 * * * *' # every minute
jobTemplate:
spec:
template:
spec:
containers:
- name: busybox
image: busybox:stable
lifecycle:
postStart:
exec:
command:
- /home/script.sh
volumeMounts:
- name: some-volume
mountPath: /home/script.sh
volumes:
- name: some-volume
configMap:
name: script-config
restartPolicy: OnFailure
What it'll do is it'll print some texts in the STDOUT in every minute. Please note that I have put only the commands that container is capable to execute, and kubectl is certainly not one of them which exists in that container out-of-the-box. I hope that is enough to answer your question.
I have a Postgres DB container which is running in a Kubernetes cluster. I need to write a Kubernetes job to connect to the Postgres DB container and run the scripts from SQL file. I need to understand two things here
commands to run SQL script
how to load SQL file in Job.yaml file
Here is my sample yaml file for Kubernetes job
apiVersion: batch/v1
kind: Job
metadata:
name: init-db
spec:
template:
metadata:
name: init-db
labels:
app: init-postgresdb
spec:
containers:
- image: "docker.io/bitnami/postgresql:11.5.0-debian-9-r60"
name: init-db
command:
- psql -U postgres
env:
- name: DB_HOST
value: "knotted-iguana-postgresql"
- name: DB_DATABASE
value: "postgres"
restartPolicy: OnFailure
You have to mount the SQL file as a volumen from a configmap and use the psql cli to execute the commands from mounted file.
To execute commands from file you can change the command parameter on the yaml by this:
psql -a -f sqlCommand.sql
The configmap needs to be created using the file you pretend to mount more info here
kubectl create configmap sqlCommands.sql --from-file=sqlCommands.sql
Then you have to add the configmap and the mount statement on your job yaml and modify the command to use the mounted file.
apiVersion: batch/v1
kind: Job
metadata:
name: init-db
spec:
template:
metadata:
name: init-db
labels:
app: init-postgresdb
spec:
containers:
- image: "docker.io/bitnami/postgresql:11.5.0-debian-9-r60"
name: init-db
command: [ "bin/sh", "-c", "psql -a -f /sqlCommand.sql" ]
volumeMounts:
- name: sqlCommand
mountPath: /sqlCommand.sql
env:
- name: DB_HOST
value: "knotted-iguana-postgresql"
- name: DB_DATABASE
value: "postgres"
volumes:
- name: sqlCommand
configMap:
# Provide the name of the ConfigMap containing the files you want
# to add to the container
name: sqlCommand.sql
restartPolicy: OnFailure
You should make a docker file for the same first, execute it and map the same working docker image to the kubernetes job yaml file.
You can add an entrypoint.sh in docker file, where you can place your scripts to be executed
I am trying to run a shell script at regular interval of 1 minute using a CronJob.
I have created following Cron job in my openshift template:
- kind: CronJob
apiVersion: batch/v2alpha1
metadata:
name: "${APPLICATION_NAME}"
spec:
schedule: "*/1 * * * *"
jobTemplate:
spec:
template:
spec:
containers:
- name: mycron-container
image: alpine:3
imagePullPolicy: IfNotPresent
command: [ "/bin/sh" ]
args: [ "/var/httpd-init/croyscript.sh" ]
volumeMounts:
- name: script
mountPath: "/var/httpd-init/"
volumes:
- name: script
configMap:
name: ${APPLICATION_NAME}-croyscript
restartPolicy: OnFailure
terminationGracePeriodSeconds: 0
concurrencyPolicy: Replace
The following is the configmap inserted as a volume in this job:
- kind: ConfigMap
apiVersion: v1
metadata:
name: ${APPLICATION_NAME}-croyscript
labels:
app: "${APPLICATION_NAME}"
data:
croyscript.sh: |
#!/bin/sh
if [ "${APPLICATION_PATH}" != "" ]; then
mkdir -p /var/httpd-resources/${APPLICATION_PATH}
fi
mkdir temp
cd temp
###### SOME CODE ######
This Cron job is running. as I can see the name of the job getting replaced every 1 min (as scheduled in my job). But it is not executing the shell script croyscript.sh
Am I doing anything wrong here? (Maybe I have inserted the configmap in a wrong way, so Job is not able to access the shell script)
Try below approach
Update permissions on configmap location
volumes:
- name: script
configMap:
name: ${APPLICATION_NAME}-croyscript
defaultMode: 0777
If this one doesnt work, most likely the script in mounted volume might have been with READONLY permissions.
use initContainer to copy the script to different location and set appropriate permissions and use that location in command parameter
I have a cronjob that is completing and outputting several log files.
I want to persist these files and be able access them after the pod has succeeded.
I've found I can access the stdout with oc logs -f <pod>, but I really need to access the log files.
I'm aware Openshift 2 apparently had an environment variable location OPENSHIFT_LOG_DIR that log files were written to, but Openshift 3.5 doesn't appear to have this.
What's my best way of logging and accessing the logs from the CronJob after the pod has succeeded and finished?
After a Job runs to completion, the Pod terminates, but it is not automatically deleted. Since it has completed, you need to use -a to see it. Once you have the Pod name, kubectl logs works as you would expect.
$ kubectl get pods -a
NAME READY STATUS RESTARTS AGE
curator-1499817660-6rzmf 0/1 Completed 0 28d
$ kubectl logs curator-1499817660-6rzmf
2017-07-12 00:01:10,409 INFO ...
A bit late but I hope this answer helps someone facing an almost similar context. For my case I needed to access some files generated by a CronJob and because the pod (and logs) are no longer accessible once the job completes I could not do so I was getting an error:
kubectl logs mongodump-backup-29087654-hj89
Error from server (NotFound): pods "mongodump-backup-27640120-n8p7z" not found
My solution was to deploy a Pod that could access the PVC. The Pod runs a busybox image as below :
apiVersion: v1
kind: Pod
metadata:
name: pvc-inspector
namespace: demos
spec:
containers:
- image: busybox
imagePullPolicy: "IfNotPresent"
name: pvc-inspector
command: ["tail"]
args: ["-f", "/dev/null"]
volumeMounts:
- mountPath: "/tmp"
name: pvc-mount
volumes:
- name: pvc-mount
persistentVolumeClaim:
claimName: mongo-backup-toolset-claim
After deploying this pod side by side to the CronJob I can exec into the pvc-inspector pod and then access files generated by the CronJob :
kubectl exec -it mongodump-backup-29087654-hj89 -- sh
cd tmp
ls
The pvc-inspector has to use the same persistentVolumeClaim as the CronJob and it also has to mount to the same directory as the CronJob.
The CronJob is a simple utility that is doing database backups of Mongo instances :
apiVersion: batch/v1beta1
kind: CronJob
metadata:
name: mongodump-backup
spec:
schedule: "*/5 * * * *" #Cron job every 5 minutes
startingDeadlineSeconds: 60
concurrencyPolicy: Forbid
successfulJobsHistoryLimit: 1
failedJobsHistoryLimit: 2
jobTemplate:
spec:
template:
spec:
containers:
- name: mongodump-backup
image: golide/mongo-backup-toolset:0.0.2
imagePullPolicy: "IfNotPresent"
env:
- name: DATABASE_NAME
value: "claims"
- name: MONGODB_URI
value: mongodb://root:mypasswordhere#mongodb-af1/claims
volumeMounts:
- mountPath: "/tmp"
name: mongodump-volume
command: ['sh', '-c',"./dumpp.sh"]
restartPolicy: OnFailure
volumes:
- name: mongodump-volume
persistentVolumeClaim:
claimName: mongo-backup-toolset-claim