How to run kubectl commands in a cron - kubernetes

Question

I created a schedule configuration inside my Gcloud project to create snapshots of a bunch of virtual disks.
Now I want to add my schedule configuration to my disks, but I dont know how to do it in a automated way, because I have more than 1200 disks.
I tryed to use a POD with a cron inside, but I cannot execute the kubectl command to list all my persistent volumes:
kubectl describe pv | grep "Name" | awk 'NR % 2 == 1' | awk '{print $2}'
I want to use this list with the next command in a Loop to add automatically my programmed schedule to my disks:
gcloud compute disks add-resource-policies [DISK_NAME] --resource-policies [SCHEDULE_NAME] --zone [ZONE]
Thanks in advance for your help.
Edit 1: After some comments I changed my code to add a Kubernetes CronJob, but the result is the same, the code doesn't work (the pod is created, but it gives me an error: ImagePullBackOff):
resource "kubernetes_cron_job" "schedulerdemo" {
metadata {
name = "schedulerdemo"
}
spec {
concurrency_policy = "Replace"
failed_jobs_history_limit = 5
schedule = "*/5 * * * *"
starting_deadline_seconds = 10
successful_jobs_history_limit = 10
job_template {
metadata {}
spec {
backoff_limit = 2
ttl_seconds_after_finished = 10
template {
metadata {}
spec {
container {
name = "scheduler"
image = "imgscheduler"
command = ["/bin/sh", "-c", "date; kubectl describe pv | grep 'Name' | awk 'NR % 2 == 1' | awk '{print $2}'"]
}
}
}
}
}
}
}

Answer 1

Answering the comment:
Ok, shame on me, wrong image name. Now I have an error in the Container Log: /bin/sh: kubectl: not found
It means that the image that you are using doesn't have kubectl installed (or it's not in the PATH). You can use image: google/cloud-sdk:latest. This image already have cloud-sdk installed which includes:
gcloud
kubectl
To run a CronJob that will get the information about PV's and change the configuration of GCP storage you will need following accesses:
Kubernetes/GKE API(kubectl) - ServiceAccount with a Role and RoleBinding.
GCP API (gcloud) - Google Service account with IAM permissions for storage operations.
I found this links helpful when assigning permissions to list PV's:
Kubernetes.io: RBAC
Success.mirantis.com: Article: User unable to list persistent volumes
The recommended way to assign specific permissions for GCP access:
Workload Identity is the recommended way to access Google Cloud services from applications running within GKE due to its improved security properties and manageability.
-- Cloud.google.com: Kubernetes Engine: Workload Identity: How to
I encourage you to read documentation I linked above and check other alternatives.
As for the script used inside of a CronJob. You should look for pdName instead of Name as the pdName is representation of the gce-pd disk in GCP (assuming that we are talking about in-tree plugin).
You will have multiple options to retrieve the disk name from the API to use it in the gcloud command.
One of the options:
kubectl get pv -o yaml | grep "pdName" | cut -d " " -f 8 | xargs -n 1 gcloud compute disks add-resource-policies --zone=ZONE --resource-policies=POLICY
Disclaimer!
Please treat above command only as an example.
Above command will get the PDName attribute from the PV's and iterate with each of them in the command after xargs.
Some of the things to take into consideration when creating a script/program:
Running this command more than once on a single disk will issue an error that you cannot assign multiple policies. You could have a list of already configured disks that do not require assigning a policy.
Consider using .spec.concurrencyPolicy: Forbid instead of Replace. Replaced CronJob will start from the beginning iterating over all of those disks. Command could not complete in the desired time and CronJob will be replaced.
You will need to check for the correct kubectl version as the official support allows +1/-1 version difference between client and a server (cloud-sdk:latest uses v1.19.3).
I highly encourage you to look on other methods to backup your PVC's (like for example VolumeSnapshots).
Take a look on below links for more reference/ideas:
Stackoverflow.com: Answer: Periodic database backup in kubernetes
Stash.run: Guides: Latest: Volumesnapshot: PVC
Velero.io
It's worth to mention that:
CSI drivers are the future of storage extension in Kubernetes. Kubernetes has announced that the in-tree volume plugins are expected to be removed from Kubernetes in version 1.21. For details, see Kubernetes In-Tree to CSI Volume Migration Moves to Beta. After this change happens, existing volumes using in-tree volume plugins will communicate through CSI drivers instead.
-- Cloud.google.com: Kubernetes Engine: Persistent Volumes: GCE PD CSI Driver: Benefits of using
Switching to CSI plugin for your StorageClass will allow you to use Volume Snapshots inside of GKE:
Volume snapshots let you create a copy of your volume at a specific point in time. You can use this copy to bring a volume back to a prior state or to provision a new volume.
-- Cloud.google.com: Kubernetes Engine: Persistent Volumes: Volume snaphosts: How to
Additional resources:
Cloud.google.com: Kubernetes Engine: Persistent Volumes
Cloud.google.com: Kubernetes Engine: Cronjobs: How to
Terraform.io: Kubernetes: CronJob
Cloud.google.com: Compute: Disks: Create snapshot