I have a pod with the following volume specified
- name: file-storage
emptyDir:
medium: Memory
also have the ephemeral storage specified in the requests and limits:
limits:
cpu: 2
ephemeral-storage: 2Gi
memory: 2Gi
requests:
cpu: 1
ephemeral-storage: 2Gi
memory: 2Gi
Now I am wondering, will the emptyDir.medium take precedence so that I get to use the RAM or the I get the ephemeral-storage and let kubernetes decide the best for me?
The emptyDir spec with medium: Memory get stored on RAM and count towards your request and limits of memory that you specify. If you remove the medium: Memory then they will be stored on the ephemeral-storage.
This is because medium: Memory spec is created on tmpfs that is allocated from RAM.
If you would like to read more about this please look at the following links:
https://kubernetes.io/docs/concepts/storage/volumes/#emptydir
https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/#resource-emphemeralstorage-consumption
Hope this answers your question, let me know if you need more clarity on this.
If you do not specify where to create an emptyDir , it gets created on the disk space of the Kubernetes node.
If you need a small scratch volume, you can define it to be created in RAM. As soon as you start using emptyDir.medium it starts using RAM.
You can check the same by creating a deployment of busybox, exec into the pod and run df -h and then you can check you would get a tmpfs (RAM) FileSystem type.
The default size of a RAM-based emptyDir is half the RAM of the node it runs on. With limits and requests please try and check what the size of disk comes out to be. Is it still from the node or does it pick up from limits values.
Please check this excercise for better understanding
https://www.alibabacloud.com/blog/kubernetes-volume-basics-emptydir-and-persistentvolume_594834
Related
I don't understand how replication works in Kubernetes.
I understand that two replicas on different nodes will provide fault tolerance for the application, but I don’t understand this:
Suppose the application is given the following resources:
---
apiVersion: v1
kind: Pod
metadata:
name: frontend
spec:
containers:
- name: app
image: images.my-company.example/app:v4
resources:
requests:
memory: "1G"
cpu: "1"
limits:
memory: "1G"
cpu: "1"
The application has two replicas. Thus, in total, 2 CPUs and 2G RAM are available for applications.
But what happens if the application receives a request with a size of 1.75G? After all, only 1G RAM is available in one replica. Will the request be distributed among all replicas?
Answer for Harsh Manvar
Maybe you misunderstood me?
What you explained is not entirely true.
Here is a real, working deployment of four replicas:
$ kubectl get deployment dev-phd-graphql-server-01-master-deployment
NAME READY UP-TO-DATE AVAILABLE AGE
dev-phd-graphql-server-01-master-deployment 4/4 4 4 6d15h
$ kubectl describe deployment dev-phd-graphql-server-01-master-deployment
...
Limits:
cpu: 2
memory: 4G
Requests:
cpu: 2
memory: 4G
...
No, it won't get distributed one replica will start simply and the other will stay in pending state.
If you will describe that pending POD(replica) it show this error :
0/1 nodes available: insufficient cpu, insufficient memory
kubectl describe pod POD-name
K8s will check for the requested resource
requests:
memory: "1G"
cpu: "1"
if mentioned minimum requested resources available it will deploy the replica and other will goes in pending state.
Update
But what happens if the application receives a request with a size of
1.75G? After all, only 1G RAM is available in one replica.
requests:
memory: "1G"
cpu: "1"
limits:
memory: "1G"
cpu: "1"
If you have a set request of 1 GB and application start using the 1.75 GB it will kill or restart the POD due to hitting the limit.
But yes in some cases container might can exceeds the limit if Node has memory available.
A Container can exceed its memory request if the Node has memory
available. But a Container is not allowed to use more than its memory
limit. If a Container allocates more memory than its limit, the
Container becomes a candidate for termination. If the Container
continues to consume memory beyond its limit, the Container is
terminated. If a terminated Container can be restarted, the kubelet
restarts it, as with any other type of runtime failure.
Read more at : https://kubernetes.io/docs/tasks/configure-pod-container/assign-memory-resource/#exceed-a-container-s-memory-limit
You might would like to read this also : https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/#how-pods-with-resource-limits-are-run
See for these kinds of circumstances you need to have an idea of how large of a request it can get and accordingly setup your resource request and limits.
If you feel there can be a request as big as 1.75GB you have tackle it in your source code.
For example you might have a conversion job which takes a lot of resources. You can make it a celery task and host the celery worker in another node group which is made for large tasks (A AWS t3.xlarge for example)
Anyways such large tasks will not generate a result immediately so I don't see a problem in running them asynchronously and giving back the result later maybe even in a websocket message. This will keep you main server from getting clumped up and also will help you to efficiently scale your large tasks
I have created a deployment with the following resources:
resources:
requests:
memory: "128Mi"
cpu: "0.45"
limits:
memory: "128Mi"
cpu: "0.8"
Using the minikube metrics server I can see that my pod CPU usage is below the requested of 450m and is only using around 150m. Shouldn't it always use 450m as a minimum value since I requested it in my .yaml file? The CPU usage goes up only if I dramatically increase the workload of the deployment. Can I have my deployment use 450m as baseline and not go below that value?
The requested value is a hint for the scheduler to help good placement of the workload. If your application does not make use of the requested resources, this is fine.
The limit will ensure no more resources are used: For CPU it will be throttled, if more RAM is used, the workload is killed (out of memory).
I tried to build angular js application in my openshidt free account :
openshift.com
The issue is when I build angular project dependencies are so big that I need 1gi for the build.
And I don't know why openshift limit the memory to 25%.
I tried to add resources in my config :
But still limit to 25%
resources:
requests:
cpu: "100m"
memory: "1Gi"
Hope you have any idea for this.
Thanks
François
Setting the memory request does not have an effect on OpenShift Online Starter (the "free account on openshift.com"). The limit would default to 512 MiB and other values (requests, CPU limit) will be set by the ClusterResourceOverride admission controller that is in place. In order to have your build use up to 1 GiB memory, you should specify only the memory limit within the build configuration:
resources:
limits:
memory: 1Gi
Look at LimitRange object in your namespace. https://kubernetes.io/docs/tasks/administer-cluster/manage-resources/memory-default-namespace/ That might be limiting your max memory.
Look at kubectl get -o=yaml POD_NAME for your pod, is your configuration actually taking effect?
If the answer to #2 is no, then maybe delete and re-apply your pod to the system.
In kubernetes I can currently limit the CPU and Memory of the containers, but what about the hard disk size of the containers.
For example, how could I avoid that someone runs a container in my k8s worker node that stores internally .jpg files making this container grow and grow with the time.
Not talking about Persistent Volume and Persistent Volume Claim. I'm talking that if someone makes an error in the container and write inside the container filesystem I want to control it.
Is there a way to limit the amount of disk used by the containers?
Thank you.
There is some support for this; the tracking issues are #361, #362 and #363. You can define requests and/or limits on the resource called ephemeral-storage, like so (for a Pod/PodTemplate):
spec:
containers:
- name: foo
resources:
requests:
ephemeral-storage: 50Mi
limits:
ephemeral-storage: 50Mi
The page Reserve Compute Resources for System Daemons has some additional information on this feature.
I'm running a small node in gcloud with 2 pods running. Google cloud console shows all resources utilization
<40% cpu utilization
about 8k n\w bytes
about 64 disk bytes.
When adding the next pod, it fails with below error.
FailedScheduling:Failed for reason PodExceedsFreeCPU and possibly others
Based on the numbers I see in google console, ~60% CPU is available. is there anyway to get more logs? Am I missing something obvious here?
Thanks in advance !
As kubernetes reserve some space if more cpu or memory is needed you should check the capacity allocated by the cluster instead of the utilization.
kubectl describe nodes
You can find a deeper description about the capacity of the nodes in: http://kubernetes.io/docs/user-guide/compute-resources/
In your helm chart or Kubernetes yaml, check the resources section. Even if you have free capacity, if your request would put the cluster over, even if your pod etc wouldn't actually use that much, it will fail to schedule. The request is asking for a reservation of capacity. IE:
spec:
serviceAccountName: xxx
containers:
- name: xxx
image: xxx
command:
- cat
tty: true
resources:
requests:
memory: "256Mi"
cpu: "250m"
limits:
memory: "256Mi"
cpu: "250m"
If the value for cpu there could make the cluster oversubscribed, it won't schedule the pod. So make sure your request reflect actual typical usage. If your requests do reflect actual typical usage, then you need more capacity.