Is there a way to tell k8s to delete the oldest pods in the deployment first?
For instance I have a deployment which consist of 3 pods. I then decided to scale it up by adding 3 more pods:
kubectl scale deployment some-deployment --replicas=6
After finishing my tests I want it to be 3 pods again, so I scale it down:
kubectl scale deployment some-deployment --replicas=3
Is there a way to configure deployment in a way so that during the scale down process the oldest pods would be removed first? What if I need it the other way around, delete more recent pods first?
This is an open issue. You may find this related item interesting. Currently you are not able to do it. Let's hope it will be fixed soon.
Related
I have a testing scenario to check if the API requests are being handled by another pod if one goes down. I know this is the default behaviour, but I want to stimulate the following scenario.
Pod replicas - 2 (pod A and B)
During my API requests, I want to kill/stop only pod A.
During downtime of A, requests should be handled by B.
I am aware that we can restart the deployment and also scale replicas to 0 and again to 2, but this won't work for me.
Is there any way to kill/stop/crash only pod A?
Any help will be appreciated.
If you want to simulate what happens if one of the pods just gets lost, you can scale down the deployment
kubectl scale deployment the-deployment-name --replicas=1
and Kubernetes will terminate all but one of the pods; you should almost immediately see all of the traffic going to the surviving pod.
But if instead you want to simulate what happens if one of the pods crashes and restarts, you can delete the pod
# kubectl scale deployment the-deployment-name --replicas=2
kubectl get pods
kubectl delete pod the-deployment-name-12345-f7h9j
Once the pod starts getting deleted, the Kubernetes Service should route all of the traffic to the surviving pod(s) (those with Running status). However, the pod is managed by a ReplicaSet that wants there to be 2 replicas, so if one of the pods is deleted, the ReplicaSet will immediately create a new one. This is similar to what would happen if the pod crashes and restarts (in this scenario you'd get the same pod and the same node, if you delete the pod it might come back in a different place).
As you mentioned you can manually kill or restart the pod that is the only solution to test the case or else you can try crashing the one single POD but in the end, it will create the same scenario POD will auto restart.
Or else may you can increase the Graceful shutdown period for deployment so this way POD might take time and stay in terminating state for a good amount of time and you can perform the test.
In kubernetes where pods are controlled by the replicaSet, if you kill a pod it will again be recreated. So the only way to do this is to scale down the number of replicas.
Let's say if your deployment had 4 replicas. You can scale down to 3 by running the command below
kubectl scale deployment <deployment-name> --replicas=3
My example is as show below
kubectl scale deployment hello-world --replicas=3
deployment.apps/hello-world scaled
I'm currently learning Kubernetes and I'm facing a problem with trying to realize a concept using Kubernetes.
I'm looking for something that works like a ReplicationController where I can tell K8s to start 50 replicas. But when I reduce the amount of replicas I need K8s to wait for the pods to terminate by themselves.
I know that there are Jobs but from what I've read it doesn't seem to be the fitting solution, since jobs are kind of a one-time thing. I need to keep the amount of desired pods until I decrease the amount of desired pods.
Basically a behavior like this:
You can use the kind Deployment in background it uses the ReplicationController and ReplicaSets.
ReplicationController is old version while the ReplicaSets is an updated approach to use. In background Kind : Deployment uses.
You can run the number for desired replicas by setting the numbers into the YAML file.
when you scale the deployment it will spin up the number of replicas and at the time of termination, you can again pass the desired replicas.
For example :
kubectl scale deployment test-deployment --replicas=50
Now running replicas are 50 and you want to scale down
kubectl scale deployment test-deployment --replicas=40
You can also check out the HPA
https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale/
I ran kubectl rollout restart deployment.
It created a new pod which is now stuck in Pending state because there are not enough resources to schedule it.
I can't increase the resources.
How do I delete the new pod?
please check if that pod has a Deployment controller (which should be recreating the pod), use:
kubectl get deployments
Then try to delete the Deployment with
Kubectl delete deployment DEPLOYMENT_NAME
Also, I would suggest to check resources allocation on GKE and its usage on your nodes with next command:
kubectl describe nodes | grep -A10 "Allocated resources"
And if you need more resources, try to activate GKE CA (cluster autoscaler) or in case you already have it enabled, then increase the number of nodes on Max value. You can also try to manually add a new node by manually resizing the Nodepool you are using.
I have a Kubernetes deployment which can have multiple replica pods. I wish to horizontally increase and decrease the pods based on some logic in my python application (not custom metrics in hpa).
I have two ways to this:
Using Horizontal Pod Autoscalar and changing minReplicas, maxReplicas though my application by using kubernetes APIs
Directly updating the "/spec/replicas" field in my deployment using the APIs
Both the above things are working for upscale and downscale.
But, when I scale down, I want to remove a particular Pod, and not any other pod.
If I update the minReplicas maxReplicas in HPA, then it randomly deletes a pod.
Same when I update the /spec/replicas field in the deployment.
How can I delete a particular pod while scaling down?
I am not aware of any way to ensure that a particular pod in a ReplicaSet will be deleted during a scale down. You could achieve this behavior with a StatefulSet which will always delete the last pod on scale down.
For example, if we had a StatefulSet foo that was scaled to 3 we would have pods:
foo-0
foo-1
foo-2
And if we scaled the StatefulSet to 2, the controller would delete foo-2. But note that there are other limitations to be aware of with StatefulSet.
I am trying to use Kubernetes Deployment , i would like to know whether this is same as kubectl apply -f deployment.yaml or does this wait for the deployments to be up and running . because when i used kubernetes deployment to create a basic pod which i know will not work, i got this error
Error: Waiting for rollout to finish: 0 of 1 updated replicas are available...
Is this just giving me the error from kubernetes or the entire terraform script fails because of this?
According to the documentation
A Deployment ensures that a specified number of pod “replicas” are running at any one time. In other words, a Deployment makes sure that a pod or homogeneous set of pods are always up and available. If there are too many pods, it will kill some. If there are too few, the Deployment will start more.
So, It will wait to ensure number of expected replicas are up