Kubernetes delete pod job - kubernetes

Question

I wanted to know is it possible to have a job in Kubernetes that will run every hour, and will delete certain pods. I need this as a temporary
stop gap to fix an issue.

Answer 1

Yes, it's possible.
I think the easiest way is just to call the Kubernernes API directly from a job. Considering RBAC is configured, something like this:
apiVersion: batch/v1
kind: Job
metadata:
name: cleanup
spec:
serviceAccountName: service-account-that-has-access-to-api
template:
spec:
containers:
- name: cleanup
image: image-that-has-curl
command:
- curl
- -ik
- -X
- DELETE
- -H
- "Authorization: Bearer $(cat /var/run/secrets/kubernetes.io/serviceaccount/token)"
- https://kubernetes.default.svc.cluster.local/api/v1/namespaces/{namespace}/pods/{name}
restartPolicy: Never
backoffLimit: 4
You can also run a kubectl proxy sidecar to connect to the cluster using localhost. More information here
Or even running plain kubectl in a pod is also an option: Kubernetes - How to run kubectl commands inside a container?

Answer 2

Use a CronJob (1, 2) to run the Job every hour.
K8S API can be accessed from Pod (3) with proper permissions. When a Pod is created a default ServiceAccount is assigned to it (4) by default. The default ServiceAccount has no RoleBinding and hence the default ServiceAccount and also the Pod has no permissions to invoke the API.
If a role (with permissions) is created and mapped to the default ServiceAccount, then all the Pods by default will get those permissions. So, it's better to create a new ServiceAccount instead of modifying the default ServiceAccount.
So, here are steps for RBAC (5)
Create a ServiceAccount
Create a Role with proper permissions (deleting pods)
Map the ServiceAccount with the Role using RoleBinding
Use the above ServiceAccount in the Pod definition
Create a pod/container with the code/commands to delete the pods
I know it's a bit confusing, but that's the way K8S works.

Answer 3

There is another workaround possibly.
You could create a liveness probe (super easy if you have none already) that doesn't run until after one hour and always fail.
livenessProbe:
tcpSocket:
port: 1234
initialDelaySeconds: 3600
This will wait 3600 seconds (1 hour) and then try to connect to port 1234 and if that fails it will kill the container (not the pod!).