I'm having Spring Cloud Dataflow deployed in multiple namespaces of my kubernetes cluster.
Additionally, a task is registered there which is executed from time to time.
Executing a Task in SCDF on kubernetes will create a pod for each execution, and if it's successful, the pod will not be deleted, but set to 'completed' instead.
I am looking for a way to automatically remove those completed pods regularily after a given amount of time (e.g. days). Also, best case scenario would be if that would work across namespaces, but i am not sure if this possible.
Do you know of any way to achieve this?
My first thought was CronJob with busybox, but i am not sure if i can give a CronJob the required permissions to delete ressources in a cluster and it would probably require to be deployed in each namespace that i want to delete ressources from.
Let me know of your thoughts, thanks in advance guys
Related
I have a simple Node.js API service in a Kubernetes cluster and it's connected to a MongoDB. This app will erase all the data from the DB and fill it up with new data every ten minutes.
In my current setup I have a NodePort service for the API and a ClusterIP service for the DB. I think everything works well as long as there is a service. However, I am afraid that if the number of Node.js pods is not one, but say 10, it will delete and upload the database at 10 different times.
Is there any way to ensure that no matter how many Node.js pods there are only once this database is deleted and uploaded every 10 minutes?
I see two ways but both require some code changes:
Define an environment variable to enable the deletion job and split your deployment in two: one deployment of a single replica with the deletion enabled and one deployment with all the other replicas but with the deletion disabled.
Use a statefulset and run the deletion only on the first pod. You can do this by checking the pod name which will always be the same on each pod, for example "myapp-0" for the first pod.
Both case solve your problem but are not that elegant. Something more in line with kubernetes design would be to remove the "deletion every 10 minutes" from your code and place the deletion code in a CLI command. Then create a kubernete CronJob that will run this command every 10 minutes. This way you keep a single, "clean" deployment, and you get all the visibility, features and guarantees from kubernetes cronjobs.
I am trying to achieve the exactly-once delivery on Kafka using Spring-Kafka on Kubernetes.
As far as I understood, the transactional-ID must be set on the producer and it should be the same across restarts, as stated here https://stackoverflow.com/a/52304789/3467733.
The problem arises using this semantic on Kubernetes. How can you get a consistent ID?
To solve this problem I implementend a Spring boot application, let's call it "Replicas counter" that checks, through the Kubernetes API, how many pods there are with the same name as the caller, so I have a counter for every pod replica.
For example, suppose I want to deploy a Pod, let's call it APP-1.
This app does the following:
It perfoms a GET to the Replicas-Counter passing the pod-name as parameter.
The replicas-counter calls the Kubernetes API in order to check how many pods there are with that pod name. So it does a a +1 and returns it to the caller. I also need to count not-ready pods (think about a first deploy, they couldn't get the ID if I wasn't checking for not-ready pods).
The APP-1 gets the id and will use it as the transactional-id
But, as you can see a problem could arise when performing rolling updates, for example:
Suppose we have 3 pods:
At the beginning we have:
app-1: transactional-id-1
app-2: transactional-id-2
app-3: transactional-id-3
So, during a rolling update we would have:
old-app-1: transactional-id-1
old-app-2: transactional-id-2
old-app-3: transactional-id-3
new-app-3: transactional-id-4 (Not ready, waiting to be ready)
New-app-3 goes ready, so Kubernetes brings down the Old-app-3. So time to continue the rolling update.
old-app-1: transactional-id-1
old-app-2: transactional-id-2
new-app-3: transactional-id-4
new-app-2: transactional-id-4 (Not ready, waiting to be ready)
As you can see now I have 2 pods with the same transactional-id.
As far as I understood, these IDs have to be the same across restarts and unique.
How can I implement something that gives me consistent IDs? Is there someone that have dealt with this problem?
The problem with these IDs are only for the Kubernetes Deployments, not for the Stateful-Sets, as they have a stable identifier as name. I don't want to convert all deployment to stateful sets to solve this problem as I think it is not the correct way to handle this scenario.
The only way to guarantee the uniqueness of Pods is to use StatefulSet.
StatefulSets will allow you to keep the number of replicas alive but everytime pod dies it will be replaced with the same host and configuration. That will prevent data loss that is required.
Service in Statefulset must be headless because since each pod is going to be unique, so you are going to need certain traffic to reach certain pods.
Every pod require a PVC (in order to store data and recreate whenever pod is deleted from that data).
Here is a great article describing why StatefulSet should be used in similar case.
I have a requirement to run an ad-hoc job, once in a while. The job needs some state to work. Building the state takes a lot of time. So, it is desired to keep the state persistent and reusable in subsequent runs, for a fast turnaround time. I want this job to be managed as K8s pods.
This is a complete set of requirements:
Pods will go down after work finish. The K8s controller should not try to bring up the pods.
Each pod should have a persistent volume attached to it. There should be 1 volume per pod. I am planning to use EBS.
We should be able to manually bring the pods back up in future.
Future runs may have more or less replicas than the past runs.
I know K8s supports both Jobs and Statefulsets. Is there any Controller which supports both at the same time?
Pods will go down after work finish. The K8s controller should not try
to bring up the pods.
This is what Jobs do - run to completion. You only control whether you wanna retry on exit > 0.
Pods should have a persistent volume attached to
them.
Same volume to all? Will they write or only read? What volume backend do you have, AWS EBS or similar? Depending of answers you might want to split input data between few volumes or use separate volumes to write and then finalization job to assemble in 1 volume (kind of map reduce). Or use volume backend which supports multi-mount RW https://kubernetes.io/docs/concepts/storage/persistent-volumes/#access-modes (see table for ReadWriteMany)
We should be able to manually bring the pods back up in future.
Jobs fit here: You launch it when you need it, and it runs till completion.
Future runs may have more or less replicas than the past runs.
Jobs fit here. Specify different completions or parallelism when you launch a job: https://kubernetes.io/docs/concepts/workloads/controllers/jobs-run-to-completion/#parallel-jobs
StatefulSets are different concept, they mostly used for clustered software which you run continuously and need to persist the role per pod (e.g. shard).
I can get jobs and pods in kubernetes after they have been completed.Then I have 2 choices:
1. delete them, but I can't find the history anymore, maybe I should build a history server by myself?
2. keep them, I wonder if there is any performance problem when lots of pods and jobs accumulate
Which one is kubernetes designed to be? Or a third choice?
All
have kubernetes on Google Computing Engine running, launching pods etc.
Recently learned that in Kubernetes Jobs were added as first class construct.
Could someone enlighten me why they were added? Looks like another level of indirection, but to solve exactly WHAT kind of problems which pods and replication controller were unable to solve?
Jobs are for workloads designed for tasks that will run to completion, then exit.
It is technically still a pod, just another kind. Where a pod while start up and run continuously until it's terminated or it crashes, a job will run a workload, then it will exit with a "Completed" status.
There is also a cronJob type, which will run periodically on set times. If you think of how cronJobs work on a standard Linux system, it should give you an idea of how you might utilize jobs (or cronJobs) on your k8s cluster