I have a simple Node.js API service in a Kubernetes cluster and it's connected to a MongoDB. This app will erase all the data from the DB and fill it up with new data every ten minutes.
In my current setup I have a NodePort service for the API and a ClusterIP service for the DB. I think everything works well as long as there is a service. However, I am afraid that if the number of Node.js pods is not one, but say 10, it will delete and upload the database at 10 different times.
Is there any way to ensure that no matter how many Node.js pods there are only once this database is deleted and uploaded every 10 minutes?
I see two ways but both require some code changes:
Define an environment variable to enable the deletion job and split your deployment in two: one deployment of a single replica with the deletion enabled and one deployment with all the other replicas but with the deletion disabled.
Use a statefulset and run the deletion only on the first pod. You can do this by checking the pod name which will always be the same on each pod, for example "myapp-0" for the first pod.
Both case solve your problem but are not that elegant. Something more in line with kubernetes design would be to remove the "deletion every 10 minutes" from your code and place the deletion code in a CLI command. Then create a kubernete CronJob that will run this command every 10 minutes. This way you keep a single, "clean" deployment, and you get all the visibility, features and guarantees from kubernetes cronjobs.
Related
I'm having Spring Cloud Dataflow deployed in multiple namespaces of my kubernetes cluster.
Additionally, a task is registered there which is executed from time to time.
Executing a Task in SCDF on kubernetes will create a pod for each execution, and if it's successful, the pod will not be deleted, but set to 'completed' instead.
I am looking for a way to automatically remove those completed pods regularily after a given amount of time (e.g. days). Also, best case scenario would be if that would work across namespaces, but i am not sure if this possible.
Do you know of any way to achieve this?
My first thought was CronJob with busybox, but i am not sure if i can give a CronJob the required permissions to delete ressources in a cluster and it would probably require to be deployed in each namespace that i want to delete ressources from.
Let me know of your thoughts, thanks in advance guys
So I have a spring boot app which runs with two replicas. I want to be able to inject whether the app is replica 1 or 2. I want to this as i want my application to run a proccess on startup, however I only want one of the replicas to run the start up proccess
My test.values.yaml.template
spring-boot:
application:
spring:
datasource:
url: url
username: username
password: password
profiles:
active: prod, agent
In general if for any reason you need to make your application replicas distinguishable from each other, then you should use StatefulSet instead of Deployment. Then you can inject the POD name into your container as env variable and use it in your application.
TL;DR:
No, you cannot inject job operations directly on a ReplicaSet.
It's not a helm issue, it's a core Kubernetes concept:
From ReplicaSet’s Documentation:
A ReplicaSet purpose is to maintain a stable set of replica Pods running at any given time. As such, it is often used to guarantee the availability of a specified number of identical Pods.
This actually means that you may never need to manipulate ReplicaSet objects: use a Deployment instead, and define your application in the spec section.
The purpose of a ReplicaSet is to replicate the pods (usually described in a deployment) and ensure the desired number of replicas is always available.
I want to be able to inject whether the app is replica 1 or 2. I want to this as i want my application to run a proccess on startup, however I only want one of the replicas to run the start up proccess
Pods are separate hosts, It's not like two instances of a app running inside the same computer, hence if you need a startup job to make them work, this job needs to be run in each one of them.
A Pod represents a unit of deployment: a single instance of an application in Kubernetes, which might consist of either a single container or a small number of containers that are tightly coupled and that share resources.
For that you can use a InitContainer:
Init containers are exactly like regular containers, except:
Init containers always run to completion.
Each init container must complete successfully before the next one starts
I'll leave you some examples of how to use InitContainers:
Kubernetes.io InitContainer Examples
A Spring-boot Use Case with Kubernetes
Kubernetes.io Configure a Pod Initialization
The InitContainer Pattern
If you have any question let me know in the comments.
If you have any startup process, one of the best option is to make use of init container. Please see more details here
I am trying to achieve the exactly-once delivery on Kafka using Spring-Kafka on Kubernetes.
As far as I understood, the transactional-ID must be set on the producer and it should be the same across restarts, as stated here https://stackoverflow.com/a/52304789/3467733.
The problem arises using this semantic on Kubernetes. How can you get a consistent ID?
To solve this problem I implementend a Spring boot application, let's call it "Replicas counter" that checks, through the Kubernetes API, how many pods there are with the same name as the caller, so I have a counter for every pod replica.
For example, suppose I want to deploy a Pod, let's call it APP-1.
This app does the following:
It perfoms a GET to the Replicas-Counter passing the pod-name as parameter.
The replicas-counter calls the Kubernetes API in order to check how many pods there are with that pod name. So it does a a +1 and returns it to the caller. I also need to count not-ready pods (think about a first deploy, they couldn't get the ID if I wasn't checking for not-ready pods).
The APP-1 gets the id and will use it as the transactional-id
But, as you can see a problem could arise when performing rolling updates, for example:
Suppose we have 3 pods:
At the beginning we have:
app-1: transactional-id-1
app-2: transactional-id-2
app-3: transactional-id-3
So, during a rolling update we would have:
old-app-1: transactional-id-1
old-app-2: transactional-id-2
old-app-3: transactional-id-3
new-app-3: transactional-id-4 (Not ready, waiting to be ready)
New-app-3 goes ready, so Kubernetes brings down the Old-app-3. So time to continue the rolling update.
old-app-1: transactional-id-1
old-app-2: transactional-id-2
new-app-3: transactional-id-4
new-app-2: transactional-id-4 (Not ready, waiting to be ready)
As you can see now I have 2 pods with the same transactional-id.
As far as I understood, these IDs have to be the same across restarts and unique.
How can I implement something that gives me consistent IDs? Is there someone that have dealt with this problem?
The problem with these IDs are only for the Kubernetes Deployments, not for the Stateful-Sets, as they have a stable identifier as name. I don't want to convert all deployment to stateful sets to solve this problem as I think it is not the correct way to handle this scenario.
The only way to guarantee the uniqueness of Pods is to use StatefulSet.
StatefulSets will allow you to keep the number of replicas alive but everytime pod dies it will be replaced with the same host and configuration. That will prevent data loss that is required.
Service in Statefulset must be headless because since each pod is going to be unique, so you are going to need certain traffic to reach certain pods.
Every pod require a PVC (in order to store data and recreate whenever pod is deleted from that data).
Here is a great article describing why StatefulSet should be used in similar case.
I have a requirement to run an ad-hoc job, once in a while. The job needs some state to work. Building the state takes a lot of time. So, it is desired to keep the state persistent and reusable in subsequent runs, for a fast turnaround time. I want this job to be managed as K8s pods.
This is a complete set of requirements:
Pods will go down after work finish. The K8s controller should not try to bring up the pods.
Each pod should have a persistent volume attached to it. There should be 1 volume per pod. I am planning to use EBS.
We should be able to manually bring the pods back up in future.
Future runs may have more or less replicas than the past runs.
I know K8s supports both Jobs and Statefulsets. Is there any Controller which supports both at the same time?
Pods will go down after work finish. The K8s controller should not try
to bring up the pods.
This is what Jobs do - run to completion. You only control whether you wanna retry on exit > 0.
Pods should have a persistent volume attached to
them.
Same volume to all? Will they write or only read? What volume backend do you have, AWS EBS or similar? Depending of answers you might want to split input data between few volumes or use separate volumes to write and then finalization job to assemble in 1 volume (kind of map reduce). Or use volume backend which supports multi-mount RW https://kubernetes.io/docs/concepts/storage/persistent-volumes/#access-modes (see table for ReadWriteMany)
We should be able to manually bring the pods back up in future.
Jobs fit here: You launch it when you need it, and it runs till completion.
Future runs may have more or less replicas than the past runs.
Jobs fit here. Specify different completions or parallelism when you launch a job: https://kubernetes.io/docs/concepts/workloads/controllers/jobs-run-to-completion/#parallel-jobs
StatefulSets are different concept, they mostly used for clustered software which you run continuously and need to persist the role per pod (e.g. shard).
I am trying to create a Kubernetes job that consists of two pods that have to be scheduled on separate nodes in our Hybrid cluster. Our requirement is that one of the pods runs on a Windows Server node and the other pod is running on a Linux node (thus we cannot just run two Docker containers from the same pod, which I know is possible, but would not work in our scenario). The Linux pod (which you can imagine as a client) will communicate over the network with the Windows pod (which you can imagine as a stateful server) exchanging data while the job runs. When the Linux pod terminates, we want to also terminate the Windows pod. However, if one of the pods fail, then we want to fail both pods (as they are designed to be a single job)
Our current design is to write a K8S service that handles the communication between the pods, and then apply the service and the two pods to the cluster to "emulate" a job. However, this is not ideal since the two pods are not tightly coupled as a single job and adds quite a bit of overhead to manually manage this setup (e.g. when failures or the job, we probably need to manually kill the service and deployment of the Windows pod). Plus we would need to deploy a new service for each "job", as we require the Linux pod to always communicate with the same Windows pod for the duration of the job due to underlying state (thus cannot use a single service for all Windows pods).
Any thoughts on how this could be best achieved on Kubernetes would be much appreciated! Hopefully this scenario is supported natively, and I would not need to resort in this kind of pod-service-pod setup that I described above.
Many thanks
I am trying to distinguish your distaste for creating and wiring the Pods from your distaste at having to do so manually. Because, in theory, a Job that creates Pods is very similar to what you are describing, and would be able to have almost infinite customization for those kinds of rules. With a custom controller like that, one need not create a Service for the client(s) to speak to their server, as the Job could create the server Pod first, obtain its Pod-specific-IP, and feed that to the subsequently created client Pods.
I would expect one could create a Job controller using only bash and either curl or kubectl: generate the json or yaml that describes the situation you wish to have, feed it to the kubernetes API (since the Job would have a service account - just like any other in-cluster container), and use normal traps to cleanup after itself. Without more of the specific edge cases loaded in my head it's hard to say if that's a good idea or not, but I believe it's possible.