Kubernetes Cluster Autoscaler versions are tightly coupled to Kubernetes versions. How can I check what version of Cluster Autoscaler is deployed currently in my Kubernetes cluster?
Running gcloud container clusters describe my-kube-cluster does not return the Cluster Autoscaler version:
nodePools:
- autoscaling:
enabled: true
maxNodeCount: 12
minNodeCount: 3
There's no endpoint in the cluster autoscaler that prints it's version, including /health-check and /metrics. The only place I could find that referenced a version number was this line in the initialisation code, which you might find in the cluster autoscaler logs. Other than that I guess you could use the kubernetes API to query the cluster autoscaler Deployment resource image tag:
kubectl get pods --all-namespaces -o=jsonpath="{..image}" -l app=cluster-autoscaler
Related
I have kubernets cluster in gcp with docker container runtime. I am trying to change docker container runtime into containerd. Following steps shows what I did.
New node pool added ( nodes with containerd )
drained old nodes
Once I perform above steps I am getting " Pod is blocking scale down because it has local storage " warning message.
You need to add the once annotation to POD so that cluster autoscaler can remove that POD from POD safe to evict.
cluster-autoscaler.kubernetes.io/safe-to-evict": "true"
above annotation, you have to add in into POD.
You can read more at : https://cloud.google.com/kubernetes-engine/docs/how-to/cluster-autoscaler-visibility#cluster-not-scalingdown
NoScaleDown example: You found a noScaleDown event that contains a
per-node reason for your node. The message ID is
"no.scale.down.node.pod.has.local.storage" and there is a single
parameter: "test-single-pod". After consulting the list of error
messages, you discover this means that the "Pod is blocking scale down
because it requests local storage". You consult the Kubernetes Cluster
Autoscaler FAQ and find out that the solution is to add a
"cluster-autoscaler.kubernetes.io/safe-to-evict": "true" annotation to
the Pod. After applying the annotation, cluster autoscaler scales down
the cluster correctly.
For further clarification, you can use this command to update the pod's annotation:
kubectl annotate pod <podname> -n <namespace> "cluster-autoscaler.kubernetes.io/safe-to-evict=true"
Had the same error when using Gitlab + Autodevops + GoogleCloud.
The issue is the cm_acme pods's that are spun up to answer the letsencrypt challenges.
e.g. we have pods like this
cm-acme-http-solver-d2tak
hanging around in our cluster so the cluster won't downsize until these pods are destroyed.
A simple
kubectl get pods -A | grep cm-acme
will list all the pods that need to be destroyed with
kubectl delete pod -n {namespace} {pod name}
I have deployed application on kubernetes cluster and for monitoring using prometheus and grafana. For kubernetes pods monitoring using Grafana dashboard: Kubernetes cluster monitoring (via Prometheus) https://grafana.com/grafana/dashboards/315
I had imported the dashboard using id 315 and its reflecting without pod name and containers name instead getting pod_name . Can anyone pls help how can i get pod name and container name in dashboard.
Provided tutorial was updated 2 years ago.
Current version of Kubernetes is 1.17. As per tags, tutorial was tested on Prometheus v. 1.3.0, Kubernetes v.1.4.0 and Grafana v.3.1.1 which are quite old at the moment.
In requirements you have statement:
Prometheus will use metrics provided by cAdvisor via kubelet service (runs on each node of Kubernetes cluster by default) and via kube-apiserver service only.
In Kubernetes 1.16 metrics labels like pod_name and container_name was removed. Instead of that you need to use pod and container. You can verify it here.
Any Prometheus queries that match pod_name and container_name labels (e.g. cadvisor or kubelet probe metrics) must be updated to use pod and container instead.
Please check this Github Thread about dashboard bug for more information.
Solution
Please change pod_name to pod in your query.
Kubernetes version v1.16.0 has Removed cadvisor metric labels pod_name and container_name to match instrumentation guidelines. Any Prometheus queries that match pod_name and container_name labels (e.g. cadvisor or kubelet probe metrics) must be updated to use pod and container instead.
You can check:
https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG/CHANGELOG-1.16.md#metrics-changes
I need to create K8s autoscale setup for spark application which will be running - on premise and AWS both as docker images.By scale, I mean (scale up and down of nodes) from on-premise to AWS cloud using cluster autoscaler or by other means
I browsed so many articles like how to set up K8 cluster on AWS/ HPA & CA scaling but could not get concrete directions to follow
I am looking for any direction which can help me understand from where i should start/steps to follow to setup such K8s cluster.
Regarding Cluster Autoscaler:
Cluster Autoscaler is a tool that automatically adjusts the size of the Kubernetes cluster when one of the following conditions is true:
- there are pods that failed to run in the cluster due to insufficient resources,
- there are nodes in the cluster that have been underutilized for an extended period of time and their pods can be placed on other existing nodes.
The cluster autoscaler on Azure dynamically scales Kubernetes worker nodes. It runs as a deployment in your cluster.
This README will help you get cluster autoscaler running on your Azure Kubernetes cluster.
Regarding HPA:
The Horizontal Pod Autoscaler automatically scales the number of pods in a replication controller, deployment or replica set based on observed CPU utilization or other custom metrics. HPA normally fetches metrics from a series of aggregated APIs:
- metrics.k8s.io
- custom.metrics.k8s.io
- external.metrics.k8s.io
Metrics-server needs to be launched separately if you wish to base on something more than just CPU utilization. More info can be found here and here.
How to make it work?
HPA is being supported by kubectl by default:
kubectl create - creates a new autoscaler
kubectl get hpa - lists your autoscalers
kubectl describe hpa - gets a detailed description of autoscalers
kubectl delete - deletes an autoscaler
Example:
kubectl autoscale rs foo --min=2 --max=5 --cpu-percent=80 creates an autoscaler for replication set foo, with target CPU utilization set to 80% and the number of replicas between 2 and 5.
Here is a detailed documentation of how to use kubectl autoscale command.
I create cluster on Google Kubernetes Engine with Cluster Autoscaler option enabled.
I want to config the scaling behavior such as --scale-down-delay-after-delete according to https://github.com/kubernetes/autoscaler/blob/master/cluster-autoscaler/FAQ.md .
But I found no Pod or Deployment on kube-system which is cluster autoscaler.
Anyone has ideas?
Edit:
I am not saying Horizontal Pod Autoscaler.
And I hope I can configure it as like this :
$ gcloud container clusters update cluster-1 --enable-autoscaling --scan-interval=5 --scale-down-unneeded-time=3m
ERROR: (gcloud.container.clusters.update) unrecognized arguments:
--scan-interval=5
--scale-down-unneeded-time=3m
It is not possible according to https://github.com/kubernetes/autoscaler/issues/966
Probably because there is no way to access the executable (which it seems to be) on GKE.
You can't even view the logs of the autoscaler on GKE: https://github.com/kubernetes/autoscaler/issues/972
One way is to not enable the GKE autoscaler, and then manually install it on a worker node - per the project's docs:
Users can put it into kube-system namespace (Cluster Autoscaler doesn't scale down node with non-mirrored kube-system pods running on them) and set a priorityClassName: system-cluster-critical property on your pod spec (to prevent your pod from being evicted).
https://github.com/kubernetes/autoscaler/tree/master/cluster-autoscaler#deployment
I would also think you could annotate the autoscaler pod(s) with the following:
"cluster-autoscaler.kubernetes.io/safe-to-evict": "false"
If i correclty understand you need this:
Check your deployments name by:
kubectl get deployments
And autoscale it by:
kubectl autoscale deployment your_deployment_name --cpu-percent=100 --min=1 --max=10
I am running Kubernetes cluster 1.5.3 on IBM Bluemix, I would like to get the pod's resources utilization (memory and cpu) as raw data points. Is Kubernetes expose such API?
➜ bluemix git:(master) ✗ k cluster-info
Kubernetes master is running at https://x:x
Heapster is running at
https://x:x/api/v1/proxy/namespaces/kube-system/services/heapster
KubeDNS is running at
https://x:x/api/v1/proxy/namespaces/kube-system/services/kube-dns
kubernetes-dashboard is running at
https://x:x/api/v1/proxy/namespaces/kube-system/services/kubernetes-dashboard
You can use heapster or kube-state-metrics to achieve this. In many kubernetes deployments heapsteris already installed. Both can be easily deployed in-cluster.