k8s Resource Quotas definition - kubernetes

I want to ignore the resource request that is defined in the service manifest:
resources:
requests:
memory: "512Mi"
cpu: "500m"
limits:
memory: "1024Mi"
cpu: "1000m"
because sometimes developers specifies requests more than they need, and it causes other services to not have enough resources.
I read about Resource Quotas, which I need to define at the namespace level.
My question is: If I define Resource Quotas in the namespace, will the resource request at the service be considered? ignored? or anything else?
Thanks.

What you can use here is a ResourceQuota something like:
apiVersion: v1
kind: ResourceQuota
metadata:
name: mem-cpu
spec:
hard:
cpu: "100m"
memory: "100Mi"
requests.cpu: "500m"
requests.memory: "512Mi"
limits.cpu: "1"
limits.memory: 1Gi
So if you have defined the resource quota (the value which cannot be exceeded by sum of cpu & memory requests respectively) in a namespace to 500m cpu & 512Mi of memory and you worry a single pod might take it up all, you can also define cpu & memory resources in the ResourceQuota (as you see in the example above).
A pod with resource requests:
resources:
limits:
memory: "512Mi"
cpu: "500m"
requests:
memory: "200Mi"
cpu: "200m"
will result in an error:
Error from server (Forbidden): error when creating "so2.yaml": pods "quota-mem-cpu-demo" is forbidden: exceeded quota: mem-cpu, requested: cpu=200m,memory=200Mi, used: cpu=0,memory=0, limited: cpu=100m,memory=100Mi
While this would work:
resources:
limits:
memory: "512Mi"
cpu: "500m"
requests:
memory: "100Mi"
cpu: "100m"
So you can control what resources a user requests.

If I define Resource Quotas in the namespace, will the resource request at the service be considered? ignored? or anything else?
As said in the name, this is a Quota, meaning that the team using the namespace can use up to that quota, but no more. As explained in the documentation, this is enforced at creation time. When a Resource Quota is set, users are required to define resources in the manifests:
If quota is enabled in a namespace for compute resources like cpu and memory, users must specify requests or limits for those values; otherwise, the quota system may reject pod creation.
Hint: Use the LimitRanger admission controller to force defaults for pods that make no compute resource requirements.

Related

Dapr sidecar injector logs show regular expression error when applying sidecar limits

I want to deploy a Dapr application with limitations on the sidecar
annotations:
dapr.io/enabled: "true"
dapr.io/app-id: "simple-js"
dapr.io/app-port: "5001"
dapr.io/sidecar-cpu-limit: “200m”
dapr.io/sidecar-cpu-request: “100m”
dapr.io/sidecar-memory-limit: “200Mi”
dapr.io/sidecar-memory-request: “100Mi”
but get these error messages:
time="2022-12-18T15:57:52.123897777Z" level=warning msg="couldn't set container resource requirements: error parsing sidecar cpu limit: quantities must match the regular expression '^([+-]?[0-9.]+)([eEinumkKMGTP][-+]?[0-9])$'. using defaults" instance=dapr-sidecar-injector-858f7757f8-6dzdq scope=dapr.injector.container type=log ver=1.9.5
time="2022-12-18T16:07:57.422789042Z" level=warning msg="couldn't set container resource requirements: error parsing sidecar memory limit: quantities must match the regular expression '^([+-]?[0-9.]+)([eEinumkKMGTP][-+]?[0-9])$'. using defaults" instance=dapr-sidecar-injector-858f7757f8-6dzdq scope=dapr.injector.container type=log ver=1.9.5
Rookie error. It turned out when assembling the annotations from https://docs.dapr.io/operations/hosting/kubernetes/kubernetes-production/#sidecar-resource-settings and / or as suggested https://kubernetes.io/docs/tasks/administer-cluster/manage-resources/quota-memory-cpu-namespace/#create-a-pod I introduced special characters / double quotes which were not distinguishable in my editor (VS Code with my fancy 3270 Nerdfont):
annotations:
dapr.io/enabled: "true"
dapr.io/app-id: "simple-js"
dapr.io/app-port: "5001"
dapr.io/sidecar-cpu-limit: “200m”
dapr.io/sidecar-cpu-request: “100m”
dapr.io/sidecar-memory-limit: “200Mi”
dapr.io/sidecar-memory-request: “100Mi”
Replacing the corrupt double quotes with correct ones fixed the problem
annotations:
dapr.io/enabled: "true"
dapr.io/app-id: "simple-js"
dapr.io/app-port: "5001"
dapr.io/sidecar-cpu-limit: "200m"
dapr.io/sidecar-cpu-request: "100m"
dapr.io/sidecar-memory-limit: "200Mi"
dapr.io/sidecar-memory-request: "100Mi"
Now the limits also show when describing the pod with kubectl
Limits:
cpu: 200m
memory: 200Mi
Requests:
cpu: 100m
memory: 100Mi

Kubernetes Resource Requests and Limits

I'm new to kubernetes. I'm just wondering is there any downside if i'm set the value for kubernetes container resource requests and limits as max as possible like this?
resources:
limits:
cpu: '3'
memory: 1Gi
requests:
cpu: '2'
memory: 256Mi
You should set requests to the minimum values your pod needs and limits to the max you allow it to use. It helps Kubernetes to schedule pods properly.
If the requests value is too high, then Kubernetes may not have any node that fulfills these requirements and your pod may not run at all.
Check this link for more details: https://sysdig.com/blog/kubernetes-limits-requests/

GKE autopilot has scaled up my container resources contary to resource requests

I have a container running in a GKE autopilot K8s cluster. I have the following in my deployment manifest (only relevant parts included):
apiVersion: apps/v1
kind: Deployment
spec:
template:
spec:
containers:
resources:
requests:
memory: "250Mi"
cpu: "512m"
So I've requested the minimum resources that GKE autopilot allows for normal pods. Note that I have not specified a limits.
However, having applied the manifest and looking at the yaml I see that it does not match what's in the manifest I applied:
resources:
limits:
cpu: 750m
ephemeral-storage: 1Gi
memory: 768Mi
requests:
cpu: 750m
ephemeral-storage: 1Gi
memory: 768Mi
Any idea what's going on here? Why has GKE scaled up the resources. This is costing me more money unnecessarily?
Interestingly it was working as intended until recently. This behaviour only seemed to start in the past few days.
If the resources that you've requested are following:
memory: "250Mi"
cpu: "512m"
Then they are not compliant with the minimal amount of resources that GKE Autopilot will assign. Please take a look on the documentation:
NAME
Normal Pods
CPU
250 mCPU
Memory
512 MiB
Ephemeral storage
10 MiB (per container)
-- Cloud.google.com: Kubernetes Engine: Docs: Concepts: Autopilot overview: Allowable resource ranges
As you can see the amount of memory you've requested was too small and that's why you saw the following message (and the manifest was modified to increate the requests/limits):
Warning: Autopilot increased resource requests for Deployment default/XYZ to meet requirements. See http://g.co/gke/autopilot-resources.
To fix that you will need to assign resources that are within the limits of the documentation, I've included in the link above.

Error creating: pods is forbidden:<pod> failed quota: <namespace>: must specify limits.memory

I created two deployments (deployment happening with a kubenetes operator and there are other activities, like service creation, secret creation etc., also there though i feel they are not related to this error) and expected for the pods to come up but pods dint come up.
when I checked the events I found there is below error for both the pods(i am listing one)
60m Warning FailedCreate replicaset/sast-edge-5577bdcf6c Error creating: pods "sas-edge-5577bdcf6c-94ps9" is forbidden: failed quota: app-xxxx-stable-biw9247u7z: must specify limits.memory
When I describe pod I see that the limits have been specified
image: registry.xxx.xxxx.com/xxkl/xxx-xxxx:1.4
imagePullPolicy: IfNotPresent
name: nsc-core
ports:
- containerPort: 3000
protocol: TCP
resources:
limits:
memory: 500Mi
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
I also checked the quota for the NS
Name: app-xxxx-stable-biw9247u7z
Namespace: app-xxxx-stable-biw9247u7z
Resource Used Hard
-------- ---- ----
limits.memory 4072Mi 8Gi
I am not sure why kubernetes is not the specified resource limit. Need help.
Forbidden Failed quota error comes when any of the containers in the pod does not have limits and requests in the spec and that includes init containers too. Adding limits and requests to all containers should solve the error.
Add something like this under spec section
containers:
- name: nsc-core
image: registry.xxx.xxxx.com/xxkl/xxx-xxxx:1.4
resources:
limits:
cpu: "2"
memory: 2000Mi
requests:
cpu: "0.2"
memory: 1500Mi

Why does a pod with one container of equal requests and limits classify as a Burstable pod?

In the kubernetes documentation here the conditions for a pod that is classified as Burstable in regards to resource QOS is defined as
If requests and optionally limits are set (not equal to 0) for one or
more resources across one or more containers, and they are not equal,
then the pod is classified as Burstable. When limits are not
specified, they default to the node capacity.
so basically stated differently:
requests set for one or more resources (cpu/memory) across one or more containers in the pod.
limits are optional: if set, they should be not be equal to the requests of the same
resource.
But then later on the documentation gives the following as an example of Burstable pod:
containers:
name: foo
resources:
limits:
cpu: 10m
memory: 1Gi
requests:
cpu: 10m
memory: 1Gi
name: bar
Note: Container bar has no resources specified.
This example fulfils condition 1. However, it doesn't satisfy condition 2, since the limits and requests are set for one container but they are equal.
So why is this pod classified as a Burstable pod?
K8s documentation containing QOS explanation and examples: https://github.com/kubernetes/community/blob/master/contributors/design-proposals/node/resource-qos.md#qos-classes
The evaluation of the Quality of Service (QoS) is done by scheduler on the whole pod, i.e. container by container and then taking the lowest evaluation.
Take a look at this example:
apiVersion: v1
kind: Pod
metadata:
name: class
spec:
containers:
- name: container1
image: busybox
command: ["sh"]
args: ["-c","sleep 3600"]
resources:
requests:
memory: 100Mi
cpu: 200m
limits:
memory: 100Mi
cpu: 200m
- name: container2
image: busybox
command: ["sh"]
args: ["-c","sleep 3600"]
resources:
requests:
memory: 100Mi
cpu: 200m
container1 has Guaranteed QoS, because it has both requests and limits defined, and they are equals.
container2 has Burstable QoS, because it hasn't limits defined, but only requests.
class pod is evaluated, based on both containers and taking the lowest evaluation:
min(Guaranteed, Burstable) = Burstable
Reference: https://kubernetes.io/docs/tasks/configure-pod-container/quality-service-pod/