I create cluster on Google Kubernetes Engine with Cluster Autoscaler option enabled.
I want to config the scaling behavior such as --scale-down-delay-after-delete according to https://github.com/kubernetes/autoscaler/blob/master/cluster-autoscaler/FAQ.md .
But I found no Pod or Deployment on kube-system which is cluster autoscaler.
Anyone has ideas?
Edit:
I am not saying Horizontal Pod Autoscaler.
And I hope I can configure it as like this :
$ gcloud container clusters update cluster-1 --enable-autoscaling --scan-interval=5 --scale-down-unneeded-time=3m
ERROR: (gcloud.container.clusters.update) unrecognized arguments:
--scan-interval=5
--scale-down-unneeded-time=3m
It is not possible according to https://github.com/kubernetes/autoscaler/issues/966
Probably because there is no way to access the executable (which it seems to be) on GKE.
You can't even view the logs of the autoscaler on GKE: https://github.com/kubernetes/autoscaler/issues/972
One way is to not enable the GKE autoscaler, and then manually install it on a worker node - per the project's docs:
Users can put it into kube-system namespace (Cluster Autoscaler doesn't scale down node with non-mirrored kube-system pods running on them) and set a priorityClassName: system-cluster-critical property on your pod spec (to prevent your pod from being evicted).
https://github.com/kubernetes/autoscaler/tree/master/cluster-autoscaler#deployment
I would also think you could annotate the autoscaler pod(s) with the following:
"cluster-autoscaler.kubernetes.io/safe-to-evict": "false"
If i correclty understand you need this:
Check your deployments name by:
kubectl get deployments
And autoscale it by:
kubectl autoscale deployment your_deployment_name --cpu-percent=100 --min=1 --max=10
Related
I have kubernets cluster in gcp with docker container runtime. I am trying to change docker container runtime into containerd. Following steps shows what I did.
New node pool added ( nodes with containerd )
drained old nodes
Once I perform above steps I am getting " Pod is blocking scale down because it has local storage " warning message.
You need to add the once annotation to POD so that cluster autoscaler can remove that POD from POD safe to evict.
cluster-autoscaler.kubernetes.io/safe-to-evict": "true"
above annotation, you have to add in into POD.
You can read more at : https://cloud.google.com/kubernetes-engine/docs/how-to/cluster-autoscaler-visibility#cluster-not-scalingdown
NoScaleDown example: You found a noScaleDown event that contains a
per-node reason for your node. The message ID is
"no.scale.down.node.pod.has.local.storage" and there is a single
parameter: "test-single-pod". After consulting the list of error
messages, you discover this means that the "Pod is blocking scale down
because it requests local storage". You consult the Kubernetes Cluster
Autoscaler FAQ and find out that the solution is to add a
"cluster-autoscaler.kubernetes.io/safe-to-evict": "true" annotation to
the Pod. After applying the annotation, cluster autoscaler scales down
the cluster correctly.
For further clarification, you can use this command to update the pod's annotation:
kubectl annotate pod <podname> -n <namespace> "cluster-autoscaler.kubernetes.io/safe-to-evict=true"
Had the same error when using Gitlab + Autodevops + GoogleCloud.
The issue is the cm_acme pods's that are spun up to answer the letsencrypt challenges.
e.g. we have pods like this
cm-acme-http-solver-d2tak
hanging around in our cluster so the cluster won't downsize until these pods are destroyed.
A simple
kubectl get pods -A | grep cm-acme
will list all the pods that need to be destroyed with
kubectl delete pod -n {namespace} {pod name}
I have a GKE cluster with an autoscale node pool.
After adding some pods, the cluster starts autoscale and creates a new node but the old running pods start to crash randomly:
I don't think it's directly related to autoscaling unless some of your old nodes are being removed. The autoscaling is triggered by adding more pods but most likely, there is something with your application or connectivity to external services (db for example). I would check the what's going on in the pod logs:
$ kubectl logs <pod-id-that-is-crashing>
You can also check for any other event in the pods or deployment (if you are using a deployment)
$ kubectl describe deployment <deployment-name>
$ kubectl describe pod <pod-id> -c <container-name>
Hope it helps!
I'm looking to update manually with the command kubectl autoscale my maximum number of replicas for auto scaling.
however each time I run the command it creates a new hpa that fails to launch the pod why I don't know at all:(
Do you have an idea how i can update manually with kubectl my HPA ?
https://gist.github.com/zyriuse75/e75a75dc447eeef9e8530f974b19c28a
I think you are mixing two topics here, one is manually scale a pod (you can do it through a deployment applying kubectl scale deploy {mydeploy} --replicas={#repl}). In the other hand you have HPA (Horizontal Pod AutoScaler), in order to do this (HPA) you should have configured any app metrics provider system
e.g:
metrics server
https://github.com/kubernetes/kubernetes/tree/master/cluster/addons/metrics-server
heapster (deprecated) https://github.com/kubernetes-retired/heapster
then you can create a HPA to handle your autoscaling, you can get more info on this link https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale-walkthrough/
Once created you can patch your HPA or deleted it and create it again
kubectl delete hpa hpa-pod -n ns-svc-cas
kubectl autoscale hpa-pod --min={#number} --max={#number} -n ns-svc-cas
easiest way
I have 3 nodes in kubernetes cluster. I create a daemonset and deployed it in all the 3 devices. This daemonset created 3 pods and they were successfully running. But for some reasons, one of the pod failed.
I need to know how can we restart this pod without affecting other pods in the daemon set, also without creating any other daemon set deployment?
Thanks
kubectl delete pod <podname> it will delete this one pod and Deployment/StatefulSet/ReplicaSet/DaemonSet will reschedule a new one in its place
There are other possibilities to acheive what you want:
Just use rollout command
kubectl rollout restart deployment mydeploy
You can set some environment variable which will force your deployment pods to restart:
kubectl set env deployment mydeploy DEPLOY_DATE="$(date)"
You can scale your deployment to zero, and then back to some positive value
kubectl scale deployment mydeploy --replicas=0
kubectl scale deployment mydeploy --replicas=1
Just for others reading this...
A better solution (IMHO) is to implement a liveness probe that will force the pod to restart the container if it fails the probe test.
This is a great feature K8s offers out of the box. This is auto healing.
Also look into the pod lifecycle docs.
kubectl -n <namespace> delete pods --field-selector=status.phase=Failed
I think the above command is quite useful when you want to restart 1 or more failed pods :D
And we don't need to care about name of the failed pod.
Kubernetes Cluster Autoscaler versions are tightly coupled to Kubernetes versions. How can I check what version of Cluster Autoscaler is deployed currently in my Kubernetes cluster?
Running gcloud container clusters describe my-kube-cluster does not return the Cluster Autoscaler version:
nodePools:
- autoscaling:
enabled: true
maxNodeCount: 12
minNodeCount: 3
There's no endpoint in the cluster autoscaler that prints it's version, including /health-check and /metrics. The only place I could find that referenced a version number was this line in the initialisation code, which you might find in the cluster autoscaler logs. Other than that I guess you could use the kubernetes API to query the cluster autoscaler Deployment resource image tag:
kubectl get pods --all-namespaces -o=jsonpath="{..image}" -l app=cluster-autoscaler