I have a Gke cluster with one node pool attached
I want to make some changes to the node pool though- like adding tags, etc
So I created a new node pool with my new config and attached to the cluster. so now cluster has 2 node pools.
At this point I want to move the pods to the new node pool and destroy the old one
How is this process done? Am I doing this right?
There are multiple ways to move your pods to the new node pool.
One way is to steer your pods to the new node pool using a label selector in your pod spec, as described in the "More fun with node pools" in the Google blog post that announced node pools (with the caveat that you need to forcibly terminate the existing pods for them to be rescheduled). This leaves all nodes in your cluster functional, and you can easily shift the pods back and forth between pools using the labels on the node pools (GKE automatically adds the node pool name as a label to make this easier).
Another way is to follow the tutorial for Migrating workloads to different machine types, which describes how to cordon / drain nodes to shift workloads to the new node pool.
Finally, you can just use GKE to delete your old node pool. GKE will automatically drain nodes prior to deleting them, which will cause your workload to shift to the new pool without you needing to run any extra commands yourself.
You can also create or update an existing node pool to add taint and then update the manifest of the pod you want to move with tolerations. This ensures that only the pod you are interested to move, moves but not others.
Refer this doc for more detail.
You can use :
kubectl drain <node_name>
in order to move all pods from a specific node to other nodes
Related
I am new to Kubernetes. One thing I am not sure of is when creating all of these deployments I am starting to max out my cpu/memory usage on my current node pool.
In this article it states that the SAME configuration of nodes will be created as a "node pool"
In Azure Kubernetes Service (AKS), nodes of the same configuration are grouped together into node pools.
System node pools serve the primary purpose of hosting critical system pods such as CoreDNS and tunnelfront. User node pools serve the primary purpose of hosting your application pods.
Also, you can create "taints (lol what)", tags and labels which only seem to "label" per se the node pool. Not an individual node.
When creating a node pool, you can add taints, labels, or tags to that node pool. When you add a taint, label, or tag, all nodes within that node pool also get that taint, label, or tag.
So with all of that said, it doesn't seem like the control is inside of a node pool's node. So how does it work for nodes in a node pool when deploying workloads and services?
Do I need to worry about managing that or is that automatically managed and pods are created across the plane of nodes in a node pool. I'm not really seeing the documentation for this.
One more thing, "Vertical Pod Autoscaling"
This seems like a good option but doesn't really explain in the documentation what is going on in terms of the nodes in a node pool. Except for this one statement at the end.
This article showed you how to automatically scale resource utilization, such as CPU and memory, of cluster nodes to match application requirements.
The question about the vertical auto scaler (which is really vertical + horizontal (IMHO)) but I understand the reference/verbiage, is what happens if you aren't using this? Do you have to manage each deployment on your own? How do deployments distribute over the individual node pool plane?
I have setup GKS in free trail access.
here is screenshot of cluster
I have already setup vm instance in gce. So my kubernets cluster is having less resource for testing i have setup it but i want to know if i delete 1 node out of 3 what will happened
my pods are running in all 3 nodes(disturbed)
So i delete one node will it create a new node with deploy my running pods into another 2 nodes it will become heavy
how do i know its HA using and Scale Up and Scale Down
Please clear my questions
So i delete one node will it create a new node with deploy my running
pods into another 2 nodes it will become heavy
GKE will manage the Nodes using Node pool config.
if inside your GKE you have set 3 nodes and manually remove 1 instance it will auto create new Node in cluster.
You pod might get moved to another node if space is left there or else it will go to pending state and wait for new node to join the GKE cluster.
If you want to redice nodes in GKE you have to redice minimum count in GKE node pool.
If you want to test scale up and down, you can enable auto scaling on Node pool and increase the POD count on cluster, GKE will auto add nodes. Make sure you have set correctly min and max nodes into node pool section for autoscaling.
When you delete a node, its pods are also deleted. Depending on your deployment, i.e. you have Pod scale of 3, one node will hold 2 pods and the other 1. If your app will suffer or not it depends on the actual traffic.
I am wondering if Kubernetes can automatically move the pods to another node if that node resources are on critical, or if that happens only for pods managed by Replica Sets?
In particular:
What happens when a bare pod (not managed by Replica Sets or similar) is evicted? Is it moved to another node or it is just removed?
In case it is "moved" to a new node, is it really moved or it is recreated? For example, does its age change?
Is it true that only pods managed by Deployments and Replica Sets are moved/recreated on a new node while bare pods are simply removed in case of resource shortage?
What happens when a bare pod (not managed by Replica Sets or similar) is evicted? Is it moved to another node or it is just removed?
Pod is designed as a relatively ephemeral, disposable entity; when it is evicted, it's deleted by a Kubelet agent running on the node. There is no recreating / moving to the other node, it's just removed (for bare pods). The controllers (like Deployment, StatefulSet, DaemonSet) are responsible for placing the replacement pods.
In case it is "moved" to a new node, is it really moved or it is recreated? For example, does its age change?
As I mentioned in the answer from the previous question, in the Kubernetes architecture pods are designed as relatively ephemeral, disposable entities, so there is no "moving". It's recreated as a "fresh" pod on the same or new node. The age parameter is changed, it's starting counting from the beginning as it is a new pod.
Is it true that only pods managed by Deployments and Replica Sets are moved/recreated on a new node while bare pods are simply removed in case of resource shortage?
Not only pods managed by Deployment/ReplicaSets but it's also true for some other controllers (e.g StatefulSet). When a pod is missing (it's got evicted by resource shortage or there is a rolling update), the controllers are making requests to the Kubernetes Scheduler to roll-out new pods. The bare pods, as answered in the first question, are not recreated.
If you want to read more about this process, check my other answer.
A orphan pod will be delete by kubelet/eviction_manager when evicted, no recreation.
No removed, only recreated.
The deploy/replicaset will keep the number of replicas, so a new pod will be created and kube-scheduler will schedule it to a fit node.
From this article(https://cloudplatform.googleblog.com/2018/06/Kubernetes-best-practices-upgrading-your-clusters-with-zero-downtime.html) I learnt that it is possible to create a new node pool, and cordon and drain old nodes one by one, so that workloads get re-scheduled to new nodes in the new pool.
To me, a new node pool seems to indicate a new cluster. The reason: we have two node pools in GKE, and they're listed as two separate clusters.
My question is: after the pods under a service get moved to a new node, if that service is being called from other pods in the old node, will this inter-cluster service call fail?
You don't create a new cluster per se. You upgrade the master(s) and then you create a new node pool with nodes that have a newer version. Make sure the new node pool shares the same network as the original node pool.
If you have a service with one replica (one pod) if that pod is living in one of the nodes you are upgrading you need to allow time for Kubernetes to create a new replica on a different node that is not being upgraded. For that time, your service will be unavailable.
If you have a service with multiple replicas chances are that you won't see any downtime unless for some odd reason all your replicas are scheduled on the same node.
Recommendation: scale your resources which serve your services (Deployments, DaemonSets, StatefulSets, etc) by one or two replicas before doing node upgrades.
StatefulSet tip: You will have some write downtime if you are running something like mysql in a master-slave config when you reschedule your mysql master.
Note that creating a new node Pool does not create a new cluster. You can have multiple node pools within the same cluster. Workloads within the different node pools will still interact with each other since they are in the same cluster.
gcloud container node-pools create (the command to create node pools) requires that you specify the --cluster flag so that the new node pool is created within an existing cluster.
So to answer the question directly, following the steps from that Google link will not cause any service interruption nor will there be any issues with pods from the same cluster communicating with each other during your migration.
For my own deployments I can use node selectors to make them run on a specific node pool. I want to do the same thing for kubernetes master pods.
(Motivation: I upgraded the kubernetes cluster today causing the aforementioned pods to be moved to some random node pool which I want to prevent from happening again in the future.)