My deployment never auto-scale on DigitalOcean. I have on my devspace.yaml
deployments:
- name: app
namespace: "mynamespace"
helm:
componentChart: true
values:
replicas: 1
autoScaling:
horizontal:
maxReplicas: 3
averageCPU: 5m
# averageRelativeCPU: 1
containers:
- name: app
image: pablorsk/app
Always has 1 replica. I try with small values on averageCPU like 5m o averageRelativeCPU like 1, but never upgrade replicas on cluster.
$ kubectl get hpa
NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE
app Deployment/app <unknown>/5m 1 3 1 13d
This is my node configuration on DigitalOcean:
HPA installation is required for auto-scale deployments.
kubectl apply -f https://github.com/kubernetes-sigs/metrics-server/releases/latest/download/components.yaml
Then, you can see values for TARGETS
$ kubectl get hpa
NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE
app Deployment/app 216/5m 1 3 1 1d
More information on metrics-server repository.
Related
So I wish to limit resources used by pod running for each of my namespace, and therefor want to use resource quota.
I am following this tutorial.
It works well, but I wish something a little different.
When trying to schedule a pod which will go over the limit of my quota, I am getting a 403 error.
What I wish is the request to be scheduled, but waiting in a pending state until one of the other pod end and free some resources.
Any advice?
Instead of using straight pod definitions (kind: Pod) use deployment.
Why?
Pods in Kubernetes are designed as relatively ephemeral, disposable entities:
You'll rarely create individual Pods directly in Kubernetes—even singleton Pods. This is because Pods are designed as relatively ephemeral, disposable entities. When a Pod gets created (directly by you, or indirectly by a controller), the new Pod is scheduled to run on a Node in your cluster. The Pod remains on that node until the Pod finishes execution, the Pod object is deleted, the Pod is evicted for lack of resources, or the node fails.
Kubernetes assumes that for managing pods you should a workload resources instead of creating pods directly:
Pods are generally not created directly and are created using workload resources. See Working with Pods for more information on how Pods are used with workload resources.
Here are some examples of workload resources that manage one or more Pods:
Deployment
StatefulSet
DaemonSet
By using deployment you will get very similar behaviour to the one you want.
Example below:
Let's suppose that I created pod quota for a custom namespace, set to "2" as in this example and I have two pods running in this namespace:
kubectl get pods -n quota-demo
NAME READY STATUS RESTARTS AGE
quota-demo-1 1/1 Running 0 75s
quota-demo-2 1/1 Running 0 6s
Third pod definition:
apiVersion: v1
kind: Pod
metadata:
name: quota-demo-3
spec:
containers:
- name: quota-demo-3
image: nginx
ports:
- containerPort: 80
Now I will try to apply this third pod in this namespace:
kubectl apply -f pod.yaml -n quota-demo
Error from server (Forbidden): error when creating "pod.yaml": pods "quota-demo-3" is forbidden: exceeded quota: pod-demo, requested: pods=1, used: pods=2, limited: pods=2
Not working as expected.
Now I will change pod definition into deployment definition:
apiVersion: apps/v1
kind: Deployment
metadata:
name: quota-demo-3-deployment
labels:
app: quota-demo-3
spec:
selector:
matchLabels:
app: quota-demo-3
template:
metadata:
labels:
app: quota-demo-3
spec:
containers:
- name: quota-demo-3
image: nginx
ports:
- containerPort: 80
I will apply this deployment:
kubectl apply -f deployment-v3.yaml -n quota-demo
deployment.apps/quota-demo-3-deployment created
Deployment is created successfully, but there is no new pod, Let's check this deployment:
kubectl get deploy -n quota-demo
NAME READY UP-TO-DATE AVAILABLE AGE
quota-demo-3-deployment 0/1 0 0 12s
We can see that a pod quota is working, deployment is monitoring resources and waiting for the possibility to create a new pod.
Let's now delete one of the pod and check deployment again:
kubectl delete pod quota-demo-2 -n quota-demo
pod "quota-demo-2" deleted
kubectl get deploy -n quota-demo
NAME READY UP-TO-DATE AVAILABLE AGE
quota-demo-3-deployment 1/1 1 1 2m50s
The pod from the deployment is created automatically after deletion of the pod:
kubectl get pods -n quota-demo
NAME READY STATUS RESTARTS AGE
quota-demo-1 1/1 Running 0 5m51s
quota-demo-3-deployment-7fd6ddcb69-nfmdj 1/1 Running 0 29s
It works the same way for memory and CPU quotas for namespace - when the resources are free, deployment will automatically create new pods.
I have managed to install Prometheus and it's adapter and I want to use one of the pod metrics for autoscaling
kubectl get --raw "/apis/custom.metrics.k8s.io/v1beta1" | jq . |grep "pods/http_request".
"name": "pods/http_request_duration_milliseconds_sum",
"name": "pods/http_request",
"name": "pods/http_request_duration_milliseconds",
"name": "pods/http_request_duration_milliseconds_count",
"name": "pods/http_request_in_flight",
Checking api I want to use pods/http_request and added it to my HPA configuration
---
apiVersion: autoscaling/v2beta2
kind: HorizontalPodAutoscaler
metadata:
name: app
namespace: app
spec:
scaleTargetRef:
apiVersion: apps/v1
kind: Deployment
name: app
minReplicas: 4
maxReplicas: 8
metrics:
- type: Pods
pods:
metric:
name: http_request
target:
type: AverageValue
averageValue: 200
After applying the yaml and check the hpa status it shows up as <unkown>
$ k apply -f app-hpa.yaml
$ k get hpa
NAME REFERENCE TARGETS
app Deployment/app 306214400/2000Mi, <unknown>/200 + 1 more...
But when using other pod metrics such as pods/memory_usage_bytes the value is properly detected
Is there a way to check the proper values for this metric? and how do I properly add it for my hpa configuration
Reference https://www.ibm.com/support/knowledgecenter/SSBS6K_3.2.0/manage_cluster/hpa.html
1st deploy metrics server, it should be up and running.
$ kubectl apply -f https://github.com/kubernetes-sigs/metrics-server/releases/latest/download/components.yaml
Then in a few sec. metrics server deployed. check HPA it should resolved.
$ kubectl get deployment -A
NAMESPACE NAME READY UP-TO-DATE AVAILABLE AGE
.
.
kube-system metrics-server 1/1 1 1 34s
$ kubectl get hpa
NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE
ha-xxxx-deployment Deployment/xxxx-deployment 1%/5% 1 10 1 6h46m
Following the example here https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale-walkthrough/#autoscaling-on-metrics-not-related-to-kubernetes-objects, I have created installed metrics-server and modified it as follows:
spec:
containers:
- command:
- metrics-server
- --secure-port=8443
- --kubelet-insecure-tls=true
- --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname
image: docker.io/bitnami/metrics-server:0.3.7-debian-10-r89
imagePullPolicy: IfNotPresent
name: metrics-server
ports:
- containerPort: 8443
name: https
protocol: TCP
resources: {}
My nodes are listed when queried:
kubectl get --raw "/apis/metrics.k8s.io/v1beta1/nodes"
{"kind":"NodeMetricsList","apiVersion":"metrics.k8s.io/v1beta1","metadata":{"selfLink":"/apis/metrics.k8s.io/v1beta1/nodes"},"items":[{"metadata":{"name":"eo-test-metrics-35lks","selfLink":"/apis/metrics.k8s.io/v1beta1/nodes/eo-test-metrics-35lks","creationTimestamp":"2020-11-04T04:05:58Z"},"timestamp":"2020-11-04T04:05:28Z","window":"30s","usage":{"cpu":"770120208n","memory":"934476Ki"}},{"metadata":{"name":"eo-test-metrics-35lkp","selfLink":"/apis/metrics.k8s.io/v1beta1/nodes/eo-test-metrics-35lkp","creationTimestamp":"2020-11-04T04:05:58Z"},"timestamp":"2020-11-04T04:05:25Z","window":"30s","usage":{"cpu":"483763591n","memory":"850756Ki"}}]}
But, the HPA targets remain 'unknown':
kubectl get hpa
NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE
php-apache Deployment/php-apache <unknown>/50% 1 10 1 31m
Running top nodes works but top pods does not
kubectl top nodes
NAME CPU(cores) CPU% MEMORY(bytes) MEMORY%
eo-test-metrics-35lkp 93m 4% 934Mi 30%
eo-test-metrics-35lks 166m 8% 1053Mi 33%
Top pods return error error: Metrics not available:
kubectl top pods
W1104 11:19:27.854485 62798 top_pod.go:266] Metrics not available for pod default/php-apache-d4cf67d68-blt2c, age: 13h1m51.854463s
error: Metrics not available for pod default/php-apache-d4cf67d68-blt2c, age: 13h1m51.854463s
This is on Kubernetes server version v1.19.3 and metrics server version 0.3.6
The logs from metrics-server
E1104 21:21:56.496129 1 reststorage.go:160] unable to fetch pod metrics for pod default/php-apache-d4cf67d68-blt2c: no metrics known for pod
E1104 21:22:10.945091 1 reststorage.go:160] unable to fetch pod metrics for pod default/php-apache-d4cf67d68-blt2c: no metrics known for pod
E1104 21:22:26.496814 1 reststorage.go:160] unable to fetch pod metrics for pod default/php-apache-d4cf67d68-blt2c: no metrics known for pod
The issue is resolved when Docker 19.03 is used on Kubernetes version 1.19 in relation to this upstream issue: https://github.com/kubernetes/kubernetes/issues/94281
Use this version of metrics server
git clone https://github.com/kodekloudhub/kubernetes-metrics-server
kubectl apply -f kubernetes-metrics-server
Then follow the same instructions as mentioned in this link
It will increase the number of pods as the load goes up
BUT I noticed the autoscaler does not scale down the deployment upon the load stopping. It might take some time until it scales down again
Note: This version of metrics server could only be used in a dev or learning environment.
I used Katakoda environment for testing the answer.
I would like to know how it's possible to set a priorityClass by default for all pods in a specific namespace without using a
globalvalue: true
may be with admission controller but i don't know.
Do you have an concret example for that ?
PriorityClass : A PriorityClass is a non-namespaced object
PriorityClass also has two optional fields: globalDefault and description.
The globalDefault field indicates that the value of this PriorityClass should be used for Pods without a priorityClassName.
Only one PriorityClass with globalDefault set to true can exist in the system. If there is no PriorityClass with globalDefault set, the priority of Pods with no priorityClassName is zero.
Create Priority Class as using below yaml (no globalDefault flag is set)
apiVersion: scheduling.k8s.io/v1
kind: PriorityClass
metadata:
name: high-priority
value: 1000000
description: "This priority class should be used for pods."
$ kubectl get priorityclasses.scheduling.k8s.io
NAME VALUE GLOBAL-DEFAULT AGE
high-priority 1000000 false 10s
Now add priority class to pod manifest and schedule them in your namespace
$ kubectl create namespace priority-test
namespace/priority-test created
$ kubectl get namespaces
NAME STATUS AGE
default Active 43m
kube-node-lease Active 43m
kube-public Active 43m
kube-system Active 43m
priority-test Active 5s
Example : pod.yaml
apiVersion: v1
kind: Pod
metadata:
name: nginx
spec:
containers:
- name: nginx
image: nginx
priorityClassName: high-priority
$ kubectl apply -f pod.yaml -n priority-test
pod/nginx created
ubuntu#k8s-master-1:~$ kubectl get all -n priority-test
NAME READY STATUS RESTARTS AGE
pod/nginx 1/1 Running 0 25s
$ kubectl describe pod -n priority-test nginx | grep -i priority
Namespace: priority-test
Priority: 1000000
Priority Class Name: high-priority
Normal Scheduled <unknown> default-scheduler Successfully assigned priority-test/nginx to worker-1
Currently per namespace priorities are not possible.
But you can achieve similar result if instead you set default priorityClass with globalDefault: true and e.g. value: 1000. Next create another lower priority class and with e.g. value: 100 and add it to all dev/staging pods.
Btw. not directly related to the question but it would be much easier to accomplish what you need if you use nodeSelectors and schedule dev pods to separate nodes. This way production pods don't have to compete for resources with non-essential pods.
I went through both daemonset doesn't create any pods and DaemonSet doesn't create any pods: v1.1.2 before asking this question. Here is my problem.
Kubernetes cluster is running on CoreOS
NAME=CoreOS
ID=coreos
VERSION=1185.3.0
VERSION_ID=1185.3.0
BUILD_ID=2016-11-01-0605
PRETTY_NAME="CoreOS 1185.3.0 (MoreOS)"
ANSI_COLOR="1;32"
HOME_URL="https://coreos.com/"
BUG_REPORT_URL="https://github.com/coreos/bugs/issues"
I refer to https://coreos.com/kubernetes/docs/latest/getting-started.html guide and created 3 etcd, 2 masters and 42 nodes. All applications running in the cluster without issue.
I got a requirement of setting up logging with fluentd-elasticsearch and downloaded yaml files in https://github.com/kubernetes/kubernetes/tree/master/cluster/addons/fluentd-elasticsearch deployed fluentd deamonset.
kubectl create -f fluentd-es-ds.yaml
I could see it got created but none of pod created.
kubectl --namespace=kube-system get ds -o wide
NAME DESIRED CURRENT NODE-SELECTOR AGE CONTAINER(S) IMAGE(S) SELECTOR
fluentd-es-v1.22 0 0 alpha.kubernetes.io/fluentd-ds-ready=true 4h fluentd-es gcr.io/google_containers/fluentd-elasticsearch:1.22 k8s-app=fluentd-es,kubernetes.io/cluster-service=true,version=v1.22
kubectl --namespace=kube-system describe ds fluentd-es-v1.22
Name: fluentd-es-v1.22
Image(s): gcr.io/google_containers/fluentd-elasticsearch:1.22
Selector: k8s-app=fluentd-es,kubernetes.io/cluster-service=true,version=v1.22
Node-Selector: alpha.kubernetes.io/fluentd-ds-ready=true
Labels: k8s-app=fluentd-es
kubernetes.io/cluster-service=true
version=v1.22
Desired Number of Nodes Scheduled: 0
Current Number of Nodes Scheduled: 0
Number of Nodes Misscheduled: 0
Pods Status: 0 Running / 0 Waiting / 0 Succeeded / 0 Failed
No events.
I verified below details according to the comments in above SO questions.
kubectl api-versions
apps/v1alpha1
authentication.k8s.io/v1beta1
authorization.k8s.io/v1beta1
autoscaling/v1
batch/v1
batch/v2alpha1
certificates.k8s.io/v1alpha1
extensions/v1beta1
policy/v1alpha1
rbac.authorization.k8s.io/v1alpha1
storage.k8s.io/v1beta1
v1
I could see below logs in one kube-controller-manager after restart.
I0116 20:48:25.367335 1 controllermanager.go:326] Starting extensions/v1beta1 apis
I0116 20:48:25.367368 1 controllermanager.go:328] Starting horizontal pod controller.
I0116 20:48:25.367795 1 controllermanager.go:343] Starting daemon set controller
I0116 20:48:25.367969 1 horizontal.go:127] Starting HPA Controller
I0116 20:48:25.369795 1 controllermanager.go:350] Starting job controller
I0116 20:48:25.370106 1 daemoncontroller.go:236] Starting Daemon Sets controller manager
I0116 20:48:25.371637 1 controllermanager.go:357] Starting deployment controller
I0116 20:48:25.374243 1 controllermanager.go:364] Starting ReplicaSet controller
The other one has below log.
I0116 23:16:23.033707 1 leaderelection.go:295] lock is held by {master.host.name} and has not yet expired
Am I missing something? Appreciate your help on figure out the issue.
I found the solution after studying https://github.com/kubernetes/kubernetes/blob/master/cluster/addons/fluentd-elasticsearch/fluentd-es-ds.yaml
There is nodeSelector: set as alpha.kubernetes.io/fluentd-ds-ready: "true"
But nodes doesn't have a label like that. What I did is add the label as below to one node to check whether it's working.
kubectl label nodes {node_name} alpha.kubernetes.io/fluentd-ds-ready="true"
After that, I could see fluentd pod started to run
kubectl --namespace=kube-system get pods
NAME READY STATUS RESTARTS AGE
fluentd-es-v1.22-x1rid 1/1 Running 0 6m
Thanks.