This page provides the following example from a container resource manifest of a pod configuration file:
resources:
limits:
cpu: "1"
requests:
cpu: "0.5"
args:
- -cpus
- "2"
And there are some notes describing what this means:
The args section of the configuration file provides arguments for the
container when it starts. The -cpus "2" argument tells the Container
to attempt to use 2 CPUs.
Recall that by setting -cpu "2", you configured the Container to
attempt to use 2 CPUs, but the Container is only being allowed to use
about 1 CPU. The container's CPU use is being throttled, because the
container is attempting to use more CPU resources than its limit.
What does it mean to say that the container is being "told to attempt to use 2 CPUs"?
From the example and note, it seems to imply that the container is being told to use more resources than what the limit constrains, but why would anyone do that?
I was assuming that the args above indicate across how many CPUs the container can span its resource utilization, but that the aggregate utilization across all of those CPUs would have to remain within the specified limits. But given the notes on the page, it seems to not work the way I expected, and that the args: -cpus is used for some other purpose/benefit, but I can't seem to glean what that is.
This example is misleading. There is nothing special about the -cpu 2 in the args section. The args section is just a way to provide paramaters to the container. The same way you would if running the container manually with docker run. In this example the command will end up being: docker run someimage:latest -cpus 2. The -cpus 2 portion gets passed along to the entrypoint of the image and (if supported) will get used by the application running inside that container.
I would imagine the aim was to show that you can have an application (container) that takes in a cpu parameter to control how much CPU that application consumes. But still limit the container usage from a k8's level using limits in the pod manifest.
Related
We have an application running under kubernetes that is NET6.0. This application is a controller and starts up 10 worker processes. The issue we are experiencing is that frequently these worker processes are being killed by kubernetes and have the exit code of 137. From my research that indicates that they were kill because they are consuming too much memory.
To make this issue further difficult to troubleshoot, it only happens in our production environment after a period of time. Our production environment is also very locked down, the docker images all run with a readonly root filesystem, with a non-root user and very low priviledges. So to monitor the application we created a dashboard that reports various things, the two I will focus on are these pieces of data:
DotnetTotalMemory = GC.GetTotalMemory(false) / (1024 * 1024),
WorkingSetSize = process.WorkingSet64 / (1024 * 1024),
The interesting thing is that the "DotnetTotalMemory" ranges anywhere from 200mb to 400mb, but the "WorkingSetSize" starts out between 400mb to 600mb, but at times it jumps up to 1300mb, even when the "DotnetTotalMemory" is hovering at 200mb.
Our quota is as follows:
resources:
limits:
cpu: '5'
memory: 10Gi
requests:
cpu: 1250m
memory: 5Gi
From what I have read, the limit amount is recognized as the "available system memory" for dotnet and is passed to it through some mechanism similar to docker run --memory=XX, correct?
I switched to Workstation GC and that seems to make them slightly more stable. Another thing I tried was setting the 'DOTNET_GCConserveMemory' environment variable to '9', again it seems to help some. But I can't get past the fact that the process seems to have 1100mb+ of memory that is not managed by the GC. Is there a way for me to reduce the working set used by these processes?
I need to restrict an app/deployment to run on specific cpus only (say 0-3 or just 1 or 2 etc.) I found out about CPU Manager and tried implement it with static policy but not able to achieve what I intend to.
I tried the following so far:
Enabled cpu manager static policy on kubelet and verified that it is enabled
Reserved the cpu with --reserved-cpus=0-3 option in the kubelet
Ran a sample nginx deployment with limits equal to requests and cpu of integer value i.e. QoS of guaranteed is ensured and able to validate the cpu affinity with taskset -c -p $(pidof nginx)
So, this makes my nginx app to be restricted to run on all cpus other than reserved cpus (0-3), i.e. if my machine has 32 cpus, the app can run on any of the 4-31 cpus. And so can any other apps/deployments that will run. As I understand, the reserved cpus 0-3 will be reserved for system daemons, OS daemons etc.
My questions-
Using the Kubernetes CPU Manager features, is it possible to pin certain cpu to an app/pod (in this case, my nginx app) to run on a specific cpu only (say 2 or 3 or 4-5)? If yes, how?
If point number 1 is possible, can we perform the pinning at container level too i.e. say Pod A has two containers Container B and Container D. Is it possible to pin cpu 0-3 to Container B and cpu 4 to Container B?
If none of this is possible using Kubernetes CPU Manager, what are the alternatives that are available at this point of time, if any?
As I understand your question, you want to set up your dedicated number of CPU for each app/pod. As I've searched.
I am only able to find some documentation that might help. The other one is a Github topic I think this is a workaround to your problem.
This is a disclaimer, based from what I've read, searched and understand there is no direct solution for this issue, only workarounds. I am still searching further for this.
I am searching for an specific information regarding kubernetes requests and limits and still didn't find an answer or just didn't understand quite well. Say I've defined two containers A and B for a single pod, both with its resources limits and requests:
A:
RAM request: 1Gi
RAM limit: 2Gi
B:
RAM request: 1Gi
RAM limit: 2Gi
So, we have a PoD limit of 4Gi (total). Suppose the A container exceeded the limit (say +1Gi), but B is consuming 64Mi only. So, my questions are:
What happens to the pod? Is it evicted?
Is the container A restarted?
Is the container A allowed to use the B's available RAM?
Thanks!
What happens to the pod? Is it evicted?
If the memory limit of a container is exceeded, the kernel's OOM killer is invoked and terminates the container's process. The Pod then starts a new container on the same Node.
(CPU limits use a different mechanism (CFS Bandwidth Control) that throttles the processes' CPU cycles instead of terminating the process.)
Is the container A restarted?
Yes.
Is the container A allowed to use the B's available RAM?
The memory is tracked separately for each container. They are not pooled together into the same limit.
Just to add some details
Memory request: Is the memory reserved for container, whether it is used completely or not.
Memory Limit: Is a restriction limit of max memory this container is supposed to use. So when containers memory requests exceeds, then whether to allocate or not depends on the free memory available in the machine running that container at that point of time
To answer your queries, from my understanding:
If Container A reaches its Memory limit of 2GI, it OOMed and this will restart the containers.
If Container A exceeds its Memory request of 1GI, it tries to get the required memory from whats available of the machine(max to what limit is set)
Hope this answers you queries
I am trying to set a deployment of containers in Kubernetes. I want the resource utilization to be controlled. I am referring this.
An example config from the docs -
resources:
limits:
memory: "200Mi"
requests:
memory: "100Mi"
command: ["stress"]
args: ["--vm", "1", "--vm-bytes", "150M", "--vm-hang", "1"]
But I am not able to clearly understand the differences between requests and args fields. limits is somewhat clear that the container should not be using more than the limit amount of resource.
What purpose does args serve exactly. Here, it is stated that this is the resource the container would start with. Then how is it different from requests ?
resources:
limits:
memory: "200Mi"
requests:
memory: "100Mi"
Resource has request and limit field.
It means minimum 100Mi memory should be allocated to the container and this values is sufficient to run the container. In case of spike in traffic, it can burst memory consumption upto 200Mi. It is kind of upper bound. If it exceeds more than 200Mi the container will get killed/restarted.
Args are being passed to command(stress container) as command line arguments.
Stress Tool Docs
DockerImageForStress
looks like stress is consuming --vm-bytes=150M memory passed as an arg
I think with the help of stress tool, the docs are trying to indicate the container can consume memory between request and limit values.
Regards the args, since it's the only thing wasn't answered in the duplicated answer:
Args are not related to the resources definition, it just describes which arguments you pass to your docker container on start.
In the case of the example, the image is probably running java code, which and the user decided to pass some arguments related to memory to it.
If the image was using a different image, for example, node, the args could be some arguments to the node code running inside the container.
Hope that it answers your question.
I have a 4 cores CPU, I create a Kubernetes Pod with CPU resource limit 100m, which mean it will occupy 1/10 of a core power.
I wondering in this case, 100m is not even a full core, if my app is a multithread app, will my app's threads run in parallel? Or all the threads will run in the part of core (100 milli core) only?
Can anyone further explain the mechanism behind?
The closest answer I found so far is this one:
For a single-threaded program, a cpu usage of 0.1 means that if you
could freeze the machine at a random moment in time, and look at what
each core is doing, there is a 1 in 10 chance that your single thread
is running at that instant. The number of cores on the machine does
not affect the meaning of 0.1. For a container with multiple threads,
the container's usage is the sum of its thread's usage (per previous
definition.) There is no guarantee about which core you run on, and
you might run on a different core at different points in your
container's lifetime. A cpu limit of 0.1 means that your usage is not
allowed to exceed 0.1 for a significant period of time. A cpu request
of 0.1 means that the system will try to ensure that you are able to
have a cpu usage of at least 0.1, if your thread is not blocking
often.
I think above sound quite logical. Based on my question, 100m core of CPUs power will spread across all the CPU cores, which mean multithreading should work in Kubernetes.
Update:
In addition, this answer explain quite well that, although it might be running a thread in single core (or less than one core power as per question), due to operating system's scheduling capability, it will still try to run the instruction unit in parallel, but not exceed the clocking power (100m as per question) as specified.
Take a look to this documentation related to resources in Kubernetes:
You can use resources as described in the article:
To specify a CPU request for a Container, include the
resources:requests field in the Container resource manifest. To
specify a CPU limit, include resources:limits.
In this exercise, you create a Pod that has one Container. The
Container has a request of 0.5 CPU and a limit of 1 CPU. Here is the
configuration file for the Pod:
apiVersion: v1
kind: Pod
metadata:
name: cpu-demo
namespace: cpu-example
spec:
containers:
- name: cpu-demo-ctr
image: vish/stress
resources:
limits:
cpu: "1"
requests:
cpu: "0.5"
args:
- -cpus
- "2"
Additional to your question:
Yes it is not gonna run in parallels (multi core threads). But you can show few core for your application in pod and then use multi threads to execute it.
The args section of the configuration file provides arguments for
the Container when it starts. The -cpus "2" argument tells the
Container to attempt to use 2 CPUs.
I had a close look at the GitHub Issue Thread in question. There is a bit of back and forth in the thread, but I think I made sense of it and would like to share a couple of things that seem to be missing from the answers so far:
100m is not the same as a 1/10 of core power. It is an absolute quantity of CPU time and will remain the same regardless of the number of cores in the node.
While CPU time might well given on multiple cores of the node, true parallelism still depends on having a CPU limit that is well over the CPU required by a single thread. Otherwise, your multi-threaded application will run concurrently (i.e. threads take turns on the same CPU core) rather than in parallel (i.e. threads running on separate cores).