I have one requirement based on some input value I need to decide the number of active pods I have, let's say in beginning the number would be 1, so there we need to start one pod after some time if number goes to 3, I need to start 2 more pods.
Next day it could happen number goes back to 1, so I need to accordingly remove 2 pods and keep 1 active. How can this be achieved in Kubernetes?
There are few ways to achieve this. The most obvious and manual one would be to use kubectl scale [--resource-version=version] [--current-replicas=count] --replicas=COUNT (-f FILENAME | TYPE NAME) as per this document or this tutorial. You can also consider taking advantage of Kubernetes autoscaling (Horizontal Pod Autoscaler and Cluster Autoscaler) described in this article and this document.
Related
I have kubernetes clusters with prometheus and grafana for monitoring and I am trying to build a dashboard panel that would display the number of pods that have been restarted in the period I am looking at.
Atm I have this query that fills a vector with 1 if the pod's creation time is in the range (meaning it has been restarted during this period) and -1 otherwise.
-sgn((time() - kube_pod_created{cluster="$cluster"}) - $__range_s)
what this looks like
Is there a way to count the number of positive values in this vector and display it? Like in this example just have a box with red 1 inside.
Or maybe there is a better way to accomplish what I am trying.
To display the Pod Restarts we have different Prometheus metrics
kube_pod_container_status_restarts_total. This is the counter metrics and this will record the container restarts.
To calculate the restarts:
If you want to see all pods then,
sum(increase(kube_pod_container_status_restarts_total{namespace="My-Namespace"}[5m])) by(pod)
or If you want Particular Pod then use,
sum(increase(kube_pod_container_status_restarts_total{namespace="My-Namespace", pod="My-Pod"}[5m]))
or to show by container wise use
sum(increase(kube_pod_container_status_restarts_total{namespace="My-Namespace", pod="My-Pod"}[5m])) by(container)
For ReplicaSets I see there is a way to use a Horizontal Pod Autoscaler (HPA) and set a max/min value for the number of replicas allowed. Is there a similar feature for StatefulSets? Since it also allows you to specify the number of replicas to deploy initially? For example, how would I tell Kubernetes to limit the number of pods it can deploy for a given StatefulSets?
I have posted community wiki answer for better visibility.
Jonas well mentioned in the comment:
First sentence in the documentation:
"The Horizontal Pod Autoscaler automatically scales the number of Pods in a replication controller, deployment, replica set or stateful set"
summary, it is possible to set min / max replicas for a statefulset using HPA. In this documentation you will learn how HPA works, how to use it, what is supported etc. HPA will not work only with the objects that can't be scaled, for example, DaemonSets.
See also this related question.
I am wondering if there is a way to set up dynamically environment variables on a scale depending on the high load.
Let's imagine that we have
Kubernetes with service called Generic Consumer which at the beginning have 4 pods. First of all
I would like to set that 75% of pods should have env variable Gold and 25% Platinium. Is that possible? (% can be changed to static number for example 3 nodes Gold, 1 Platinium)
Second question:
If Platinium pod is having a high load is there a way to configure Kubernetes/charts to scale only the Platinium and then decrease it after higher load subsided
So far I came up with creating 2 separate YAML files with diff env variables and replicas numbers.
Obviously, the whole purpose of this is to prioritize some topics
I have used this as a reference https://www.confluent.io/blog/prioritize-messages-in-kafka.
So in the above example, Generic Consumer would be the Kafka consumer which would use env variable to get bucket config
configs.put(ConsumerConfig.PARTITION_ASSIGNMENT_STRATEGY_CONFIG,
BucketPriorityAssignor.class.getName());
configs.put(BucketPriorityConfig.TOPIC_CONFIG, "orders-per-bucket");
configs.put(BucketPriorityConfig.BUCKETS_CONFIG, "Platinum, Gold");
configs.put(BucketPriorityConfig.ALLOCATION_CONFIG, "70%, 30%");
configs.put(BucketPriorityConfig.BUCKET_CONFIG, "Platinum");
consumer = new KafkaConsumer<>(configs);
If you have any alternatives, please let me know!
As was mention in comment section, the most versitale option (and probably the best for your scenario with prioritization) is to keep two separate deployments with gold and platinium labels.
Regarding first question, like #David Maze pointed, pods from Deployment are identical and you cannot have few pods with one label and few others with different. Even if you would create manually (3 pods with gold and 1 with platiunuim) you won't be able to use HPA.
This option allows you to adjustments depends on the situation. For example you would be able to scale one deployment using HPA and another with VPA or both with HPA. Would help you maintain budget, i.e for gold users you might limit to have maximum 5 pods to run simultaneously and for platinium you could set this maximum to 10.
You could consider Istio Traffic Management to routing request, but in my opinion, method with two separated deployments is more suitable.
When creating a kubernetes deployment I set .spec.replicas to my minimum desired number of replicas. Then I create a horizontal pod autoscaler with minimum and maximum replicas.
The easiest way to do the next deployment is to use this same lower bound. When combining it with autoscaling should I set replicas to the minimum desired as used before or should I get the current number of replicas and start from there? This would involve an extra roundtrip to the api, so if it's not needed would be preferable.
There are two interpretations of your question:
1. You have an existing Deployment object and you want to update it - "deploy new version of your app".
In this case you don't need to change either replicas in Deployment object (it's managed by horizontal pod autoscaler) or horizontal pod autoscaler configuration. It will work out of the box. It's enough to change the important bits of deployment spec.
See rolling update documentation for more details.
2. You have an existing Deployment object and you want to create second one with the same application
If you create a separate application it may have a different load characteristics, so it's probably that the desired number of replicas will be different. In any case HPA will adjust it relatively quickly, so IMO setting initial number of replicas to the same number is not needed.
In reference to this doc I think I understand the value of temporarily horizontally scaling a pod during an update. For example, you go from 1 pod to 2 pods - update pod 1 and then delete pod 2.
Is there any value to horizontally scaling Kubernates if you're not doing an update? Won't replicating pods simply decrease the performance of each one?
For example, doubling the number of pods while keeping the amount of RAM fixed just means that each pod will have half as much RAM.
... doubling the number of pods while keeping the amount of RAM fixed ...
I think you are mis-understanding what happens when you horizontally scale pods. Each pod is allocated a certain amount of memory and when you create new pods, each existing pod keeps using that much memory and new pods get that same amount of memory. So the overall memory usage as your horizontally scale pods is increasing linearly with the number of pods that are running.
The only constraint is when you hit the total limit of memory available in your cluster, at which point you won't be able to schedule new pods. At this point, you would need to (manually or automatically) scale the number of nodes in the cluster to add additional resources.
Is there any value to horizontally scaling Kubernates if you're not doing an update?
Yes. If you are serving traffic and the current number of pods can't handle the load, adding new pods can help. For example, during the One million requests per second demo, the author determines how many requests each nginx pod can serve and then scales the number of pods to handle the expected load. If this was a web site, the horizontal pod autoscaler could allow you to dynamically scale the number of nginx pods based on the current number of requests.