I'm new to kubernetes. I'm just wondering is there any downside if i'm set the value for kubernetes container resource requests and limits as max as possible like this?
resources:
limits:
cpu: '3'
memory: 1Gi
requests:
cpu: '2'
memory: 256Mi
You should set requests to the minimum values your pod needs and limits to the max you allow it to use. It helps Kubernetes to schedule pods properly.
If the requests value is too high, then Kubernetes may not have any node that fulfills these requirements and your pod may not run at all.
Check this link for more details: https://sysdig.com/blog/kubernetes-limits-requests/
Related
I have created a deployment with the following resources:
resources:
requests:
memory: "128Mi"
cpu: "0.45"
limits:
memory: "128Mi"
cpu: "0.8"
Using the minikube metrics server I can see that my pod CPU usage is below the requested of 450m and is only using around 150m. Shouldn't it always use 450m as a minimum value since I requested it in my .yaml file? The CPU usage goes up only if I dramatically increase the workload of the deployment. Can I have my deployment use 450m as baseline and not go below that value?
The requested value is a hint for the scheduler to help good placement of the workload. If your application does not make use of the requested resources, this is fine.
The limit will ensure no more resources are used: For CPU it will be throttled, if more RAM is used, the workload is killed (out of memory).
I have a resource block for my pod like -
resources:
limits:
cpu: 3000m
memory: 512Mi
requests:
memory: 512Mi
does it by default take request allocation for CPU (i.e 3000m) which is mentioned in resource limits (3000m). Because in my case it taking 3000m as default cpu in request even though I have not mentioned it.
What you observed is correct, K8s will assign the requests.cpu that matches the limits.cpu when you only define the limits.cpu and not the requests.cpu. Official document here.
sourced from kubernetes documentation
If you specify a CPU limit for a Container but do not specify a CPU request, Kubernetes automatically assigns a CPU request that matches the limit. Similarly, if a Container specifies its own memory limit, but does not specify a memory request, Kubernetes automatically assigns a memory request that matches the limit
I have a container running in a GKE autopilot K8s cluster. I have the following in my deployment manifest (only relevant parts included):
apiVersion: apps/v1
kind: Deployment
spec:
template:
spec:
containers:
resources:
requests:
memory: "250Mi"
cpu: "512m"
So I've requested the minimum resources that GKE autopilot allows for normal pods. Note that I have not specified a limits.
However, having applied the manifest and looking at the yaml I see that it does not match what's in the manifest I applied:
resources:
limits:
cpu: 750m
ephemeral-storage: 1Gi
memory: 768Mi
requests:
cpu: 750m
ephemeral-storage: 1Gi
memory: 768Mi
Any idea what's going on here? Why has GKE scaled up the resources. This is costing me more money unnecessarily?
Interestingly it was working as intended until recently. This behaviour only seemed to start in the past few days.
If the resources that you've requested are following:
memory: "250Mi"
cpu: "512m"
Then they are not compliant with the minimal amount of resources that GKE Autopilot will assign. Please take a look on the documentation:
NAME
Normal Pods
CPU
250 mCPU
Memory
512 MiB
Ephemeral storage
10 MiB (per container)
-- Cloud.google.com: Kubernetes Engine: Docs: Concepts: Autopilot overview: Allowable resource ranges
As you can see the amount of memory you've requested was too small and that's why you saw the following message (and the manifest was modified to increate the requests/limits):
Warning: Autopilot increased resource requests for Deployment default/XYZ to meet requirements. See http://g.co/gke/autopilot-resources.
To fix that you will need to assign resources that are within the limits of the documentation, I've included in the link above.
I want to ignore the resource request that is defined in the service manifest:
resources:
requests:
memory: "512Mi"
cpu: "500m"
limits:
memory: "1024Mi"
cpu: "1000m"
because sometimes developers specifies requests more than they need, and it causes other services to not have enough resources.
I read about Resource Quotas, which I need to define at the namespace level.
My question is: If I define Resource Quotas in the namespace, will the resource request at the service be considered? ignored? or anything else?
Thanks.
What you can use here is a ResourceQuota something like:
apiVersion: v1
kind: ResourceQuota
metadata:
name: mem-cpu
spec:
hard:
cpu: "100m"
memory: "100Mi"
requests.cpu: "500m"
requests.memory: "512Mi"
limits.cpu: "1"
limits.memory: 1Gi
So if you have defined the resource quota (the value which cannot be exceeded by sum of cpu & memory requests respectively) in a namespace to 500m cpu & 512Mi of memory and you worry a single pod might take it up all, you can also define cpu & memory resources in the ResourceQuota (as you see in the example above).
A pod with resource requests:
resources:
limits:
memory: "512Mi"
cpu: "500m"
requests:
memory: "200Mi"
cpu: "200m"
will result in an error:
Error from server (Forbidden): error when creating "so2.yaml": pods "quota-mem-cpu-demo" is forbidden: exceeded quota: mem-cpu, requested: cpu=200m,memory=200Mi, used: cpu=0,memory=0, limited: cpu=100m,memory=100Mi
While this would work:
resources:
limits:
memory: "512Mi"
cpu: "500m"
requests:
memory: "100Mi"
cpu: "100m"
So you can control what resources a user requests.
If I define Resource Quotas in the namespace, will the resource request at the service be considered? ignored? or anything else?
As said in the name, this is a Quota, meaning that the team using the namespace can use up to that quota, but no more. As explained in the documentation, this is enforced at creation time. When a Resource Quota is set, users are required to define resources in the manifests:
If quota is enabled in a namespace for compute resources like cpu and memory, users must specify requests or limits for those values; otherwise, the quota system may reject pod creation.
Hint: Use the LimitRanger admission controller to force defaults for pods that make no compute resource requirements.
I'm running a small node in gcloud with 2 pods running. Google cloud console shows all resources utilization
<40% cpu utilization
about 8k n\w bytes
about 64 disk bytes.
When adding the next pod, it fails with below error.
FailedScheduling:Failed for reason PodExceedsFreeCPU and possibly others
Based on the numbers I see in google console, ~60% CPU is available. is there anyway to get more logs? Am I missing something obvious here?
Thanks in advance !
As kubernetes reserve some space if more cpu or memory is needed you should check the capacity allocated by the cluster instead of the utilization.
kubectl describe nodes
You can find a deeper description about the capacity of the nodes in: http://kubernetes.io/docs/user-guide/compute-resources/
In your helm chart or Kubernetes yaml, check the resources section. Even if you have free capacity, if your request would put the cluster over, even if your pod etc wouldn't actually use that much, it will fail to schedule. The request is asking for a reservation of capacity. IE:
spec:
serviceAccountName: xxx
containers:
- name: xxx
image: xxx
command:
- cat
tty: true
resources:
requests:
memory: "256Mi"
cpu: "250m"
limits:
memory: "256Mi"
cpu: "250m"
If the value for cpu there could make the cluster oversubscribed, it won't schedule the pod. So make sure your request reflect actual typical usage. If your requests do reflect actual typical usage, then you need more capacity.