I have a project on Kubernetes and only 1 pod is running on each node and it has to be like that. How can I autoscale the nodes I want to build? So how can I create a new node when the load on the node increases. I am using AWS and Azure.
I am not sure why you want to run a pod per node. We have separate groups of applications and each group runs on certain node groups. We use cluster autoscaler to scale down/up nodes based on usage. We then use taints and tolerations on pods/nodes for each group of application. Cluster auto scaler will scale up nodes for the specific application group is pods are in pending status.
added below portion -
One useful article about scaling in kubernetes is here. You can read about Cluster Autoscaler (CA) as well.
I think you can try this with ClusterAutoscaler:
https://github.com/kubernetes/autoscaler/tree/master/cluster-autoscaler
Related
I have a terraform-managed EKS cluster. It used to have 2 nodes on it. I doubled the number of nodes (4).
I have a kubernetes_deployment resource that automatically deploys a fixed number of pods to the cluster. It was set to 20 when I had 2 nodes, and seemed evenly distributed with 10 each. I doubled that number to 40.
All of the new pods for the kubernetes deployment are being scheduled on the first 2 (original) nodes. Now the two original nodes have 20 pods each, while the 2 new nodes have 0 pods. The new nodes are up and ready to go, but I cannot get kubernetes to schedule the new pods on those new nodes.
I am unsure where to even begin searching, as I am fairly new to k8s and ops in general.
A few beginner questions that may be related:
I'm reading about pod affinity, and it seems like I could tell k8s to have a pod ANTI affinity with itself within a deployment. However, I am having trouble setting up the anti-affinity rules. I see that the kubernetes_deployment resource has a scheduling argument, but I can't seem to get the syntax right.
Naively it seems that the issue may be that the deployment somehow isn't aware of the new nodes. If that is the case, how could I reboot the entire deployment (without taking down the already-running pods)?
Is there a cluster level scheduler that I need to set? I was under the impression that the default does round robin, which doesn't seem to be happening at the node level.
EDIT:
The EKS terraform module node_groups submodule has fields for desired/min/max_capacity. To increase my worker nodes, I just increased those numbers. The change is reflected in the aws eks console.
Check a couple of things:
Do your nodes show up correctly in the output of kubectl get nodes -o wide and do they have a state of ready?
Instead of pod affinity look into pod topology spread constraints. Anti affinity will not work with multiple pods.
I have setup GKS in free trail access.
here is screenshot of cluster
I have already setup vm instance in gce. So my kubernets cluster is having less resource for testing i have setup it but i want to know if i delete 1 node out of 3 what will happened
my pods are running in all 3 nodes(disturbed)
So i delete one node will it create a new node with deploy my running pods into another 2 nodes it will become heavy
how do i know its HA using and Scale Up and Scale Down
Please clear my questions
So i delete one node will it create a new node with deploy my running
pods into another 2 nodes it will become heavy
GKE will manage the Nodes using Node pool config.
if inside your GKE you have set 3 nodes and manually remove 1 instance it will auto create new Node in cluster.
You pod might get moved to another node if space is left there or else it will go to pending state and wait for new node to join the GKE cluster.
If you want to redice nodes in GKE you have to redice minimum count in GKE node pool.
If you want to test scale up and down, you can enable auto scaling on Node pool and increase the POD count on cluster, GKE will auto add nodes. Make sure you have set correctly min and max nodes into node pool section for autoscaling.
When you delete a node, its pods are also deleted. Depending on your deployment, i.e. you have Pod scale of 3, one node will hold 2 pods and the other 1. If your app will suffer or not it depends on the actual traffic.
I am using kubernete to manage docker cluster. Right now, I can set up POD autoscale using Horizontal Pod Scaler, that is fine.
And now I think the next step is to autoscale nodes. I think for HPA, the auto-created pod is only started in the already created nodes, but if all the available nodes are utilized and no available resource for any more pods, I think the next step is to automatically create node and have node join the k8s master.
I googled a lot and there are very limited resources to introduce this topic.
Can anyone please point me to any resource how to implement this requirement.
Thanks
One way to do using AWS and setting up your own Kubernetes cluster is by following these steps :
Create an Instance greater than t2.micro (will be master node).
Initialize the Kubernetes cluster using some tools like Kubeadm. After the initialisation would be completed you would get a join command, which needs to e run on all the nodes who want to join the cluster. (Here is the link)
Now create an Autoscaling Group on AWS with start/boot script containing that join command.
Now whenever the utilisation specified by you in autoscaling group is breached the scaling would happen and the node(s) would automatically join the Kubernetes cluster. This would allow the Kubernetes to schedule pods on the newly joined nodes based on the HPA.
(I would suggest to use Flannel as pod network as it automatically removes the node from Kubernetes cluster when it is not available)
kubernetes operations (kops) helps you create, destroy, upgrade and maintain production-grade, highly available, Kubernetes clusters from the command line.
Features:
Automates the provisioning of Kubernetes clusters in AWS and GCE
Deploys Highly Available (HA) Kubernetes Masters
Most of the managed kubernetes service providers provide auto scaling feature of the nodes
Elastic Kubernetes Service EKS- configure cluster auto scalar
Google Kubernetes Engine
GKE Auto Scalar
Auto scaling feature needs to be supported by the underlying cloud provider. Google cloud supports auto scaling during cluster creation or update by passing flags --enable-autoscaling --min-nodes and --max-nodes to the corresponding gcloud commands.
Examples:
gcloud container clusters create mytestcluster --zone=us-central1-b --enable-autoscaling --min-nodes=3 --max-nodes=10 --num-nodes=5
gcloud container clusters update mytestcluster --enable-autoscaling --min-nodes=1 --max-nodes=15
below link would be helpful
https://medium.com/kubecost/understanding-kubernetes-cluster-autoscaling-675099a1db92
I am using Google Container Engine . Now I want auto scaling functionality in my cluster . As per documentation GKE autoscaler is in beta release . I can also enable autoscaling in instance group that is managing cluster nodes .
Cluster autoscaler add/remove nodes so that all scheduled pods have a place to run where instance group add/remove nodes based on different policies like average cpu utilization .
I think by adjusting pods CPU limit and target CPU utilization for pods in Kubernetes autoscaler , Managed Instance Group autoscaling can also be used to resize GKE cluster .
So my question is what should I use ?
Short answer - don't use GCE MIG autoscaling feature. It will just not work properly with your cluster.
See details in this FAQ:
https://github.com/kubernetes/autoscaler/blob/master/cluster-autoscaler/FAQ.md#should-i-use-a-cpu-usage-based-node-autoscaler-with-kubernetes
(read the question linked above and 2 next ones)
As per GCP docs :
"Caution: Do not enable Compute Engine autoscaling for managed instance groups for your cluster nodes. GKE's cluster autoscaler is separate from Compute Engine autoscaling. This can lead to node pools failing to scale up or scale down as the Compute Engine autoscaler will be in conflict with GKE's cluster autoscaler"
More Details :
https://cloud.google.com/kubernetes-engine/docs/concepts/cluster-autoscaler
I have deployed an app using Kubernetes to a Google Cloud Container Engine Cluster.
I got into autoscaling, and I found the following options:
Kubernetes Horizontal Pod Autoscaling (HPA)
As explained here, Kubernetes offers the HPA on deployments. As per the docs:
Horizontal Pod Autoscaling automatically scales the number of pods in a replication controller, deployment or replica set based on observed CPU utilization
Google Cloud Container Cluster
Now I have a Google Cloud Container Cluster using 3 instances, with autoscaling enabled. As per the docs:
Cluster Autoscaler enables users to automatically resize clusters so that all scheduled pods have a place to run.
This means I have two places to define my autoscaling. Hence my questions:
Is a Pod the same as VM instance inside my cluster, or can multiple Pod's run inside a single VM instance?
Are these two parameters doing the same (aka creating/removing VM instances inside my cluster). If not, what is their behaviour compared to one another?
What happens if e.g. I have a number of pods between 3 and 10 and a cluster with number of instances between 1 and 3 and autoscaling kicks in. When and how would both scale?
Many thanks!
Is a Pod the same as VM instance inside my cluster, or can multiple
Pod's run inside a single VM instance?
Multiple Pods can run the same instance (called node in kuberenetes). You can define maximum resources to consume for a POD in the deployment yaml. See the docs. This is an important prerequisite for autoscaling.
Are these two parameters doing the same (aka creating/removing VM
instances inside my cluster). If not, what is their behaviour compared
to one another?
Kubernetes autoscaler will schedule additional PODs in your existing nodes. Google autoscaler will add worker nodes (new instances) to your cluster. Google autoscaler looks at queued up PODs that cannot be scheduled because there is no space in your cluster and when it finds those will add nodes.
What happens if e.g. I have a number of pods between 3 and 10 and a
cluster with number of instances between 1 and 3 and autoscaling kicks
in. When and how would both scale?
By the maximum resource usage you define for your pods google autoscaler will estimate how many new nodes are required to run all queued up, scheduled pods.
Also read this article.