I want to create kubernetes cluster, and what I need to do is, I have two servers. (Nodes) with 10x Intel Xeon and 32gb ram each. Can I run one application on both of them, so the application can use 20 cores, and 64gb of ram?
Of course you can't. If you run application in kubernetes, then kubernetes will create a structure called pod to help run it. Each of your server will be a node in kubernetes. There can be many pods running this application ,but for each pod it can only use the resource of one node where the pods assigned.( a pod is like a container, but I don't know how it works)
If your application really needs 20 cores and 64gb, then you need a node that actually has 20 cores and 64gb.
But if you just want to make the full use of the resource, you can check the resource limit of your application (or calculate it by your own) and calculate how many copies you need to fully use your resource. For example, if your application needs at least 1 core and 1gb to run, then you can use a replicas and make 20 copy(may be less).
No, it is not possible to span a single pod across 2 different physical/virtual nodes. One single application running inside your pod(minimal object in K8s) can only utilize the maximum capacity(virtually) of the underlying node on which the pod is scheduled.
As #youth Wu said, try to check the resource limit of your application and start with the minimal one and if your application really needs resources more then the available resources on a node then, you have to scale your nodes vertically without any choice.
Related
Am using GKE(google managed kubernetes) and I have requirement where I want to leave around 10% of memory on each Node as Idle so that during burst workload scenarios, the pod's already deployed on that Node can make use of those idle resources (within limit range)
Basically What I want to achieve is, I want to avoid a scenario where Pod's get scheduled onto a Node till 100% resources are consumed and assuming all the Pod's/Services are utilizing their allocated resources (set via requests) and one of the POD has a burst workload scenario or the pod got restarted and it needs more memory during boot up, then it should be able to make use of those idle resources
After going through the documentation I have come across this, but since GKE is a managed service, these properties aren't exposed anywhere, are there any other ways to achieve the same ?
GKE is a managed service and therefore you will not be able to costumize the worker node kublet parameters like --eviction-hard or --system-reserved.
As a workaround, you need to calculate your pod's memory requests and memory limits in order to configure a maximum number of pods per node, in this way you can controle the number of pods that run on your node and the spare CPU and memory to be used by your pods in case of a burst.
I am new to K8s. I am researching benefits of using k8s on GKE/AKS over Azure VM Scalesets used in our system.
For k8s, let's say we deploy worker application to 5 nodes cluster, each node can runs 10 pods. Each node is 16 core, 64G memory VM.
My question is, since we are paying cloud service provider by node, not by pod, why we want to scale down pod on the node at all?
Does it make more sense to horizontally scale up and down nodes only, while having maximum pods on each node?
Does 10 pods, 20 pods, 30 pods, 40 pods, 50 pods makes more sense, meanwhile 11 pods, 21 pods, 31 pods, 41 pods etc sounds wasting resources we paid?
I must missed key points of doing pods scaling.
Please point them out.
Thanks.
If all your pods are all identical, then yes, it likely makes sense to scale them only in node-sized equivalents (but maybe in that case you want to actually make sure the nodes can scale in increments that make sense to your workload size -- e.g. do you really need another 32 or 64 or 96 cores worth of your application every time? This is one of the things that the new Google batch for Kubernetes product tries to address -- including rightsizing of machines).
But think if you have a heterogeneous set of workloads (far more likely with k8s!) -- then one of the advantages of k8s is that you can binpack different workloads onto the same node.
Imagine if one workload needs a lot of RAM, but not much CPU, and another workload needs a lot of CPU and not a lot of RAM -- you wouldn't want them to both scale up in one-node increments, you'd want the pods to scale with the demands on each application, and the nodes to scale when you can no longer binpack onto the existing machines.
Because it is easier to think in terms of scaling policies for a single deployment at a time.
Because you can set the system to collapse low utilization nodes together and then delete unused nodes.
Because it gives you better stats to track your utilization rate.
I want to know the recommendation set for pod size. I.e. when to put application within pod or at what size it will be better to use machine itself in place of pod.
Ex. when to think of coming out of k8s and used as external service for some application, when pod required 8GB or 16GB or 32GB? Same for CPU intensive.
Because if pod required 16GB or 16 CPU and we have a machine/node of the same size then I think there is no sense of running pod on that machine. If we run in that scenario then it will be like we will having 10 pods and which required 8 Nodes.
Hopes you understand my concern.
So if some one have some recommendation for that then please share your thoughts on that. Some references will be more better.
Recommendation for ideal range:
size of pods in terms of RAM and CPU
Pods is to nodes ratio, i.e. number of pods per nodes
Whether good for stateless or stateful or both type of application or not
etc.
Running 16cpu/16gb pod on 16cpu/16gb machine is normal. Why not? You think of pods to be tiny but there is no such requirement. Pods can be gigantic, there is no issue with that. Remember container is just a process on a node, why you refuse to run a fat process on a fat node? Kubernetes adds very nice orchestration level to containers, why not make use of it?
There is no such thing as a universal or recommended pod size. Asking for recommended pod size is the same as asking for a recommended size for VM or bare metal server. It is totally up to your application. If your application requires 16 or 64 GB of RAM - this is the recommended size for you, you see?
Regarding pods to nodes ratio - current upper limit of Kubernetes is 110 pods per node. Everything below that watermark is fine. The only thing is that recommended master node size increases with total number of pods. If you have like 1000 pods - you go with small to medium size master nodes. If you have over 10 000 pods - you should increase your master nodes size.
Regarding statefulness - stateless application generally survive better. But often state also should be stored somewhere and stored reliably. So if you plan your application as a set of microservices - create as much stateless apps you can and as few stateful as you can. Ideally, only the relational databases should be truly stateful.
I have played around a little bit with docker and kubernetes. Need some advice here on - Is it a good idea to have one POD on a VM with all these deployed in multiple (hybrid) containers?
This is our POC plan:
Customers to access (nginx reverse proxy) with a public API endpoint. eg., abc.xyz.com or def.xyz.com
List of containers that we need
Identity server Connected to SQL server
Our API server with Hangfire. Connected to SQL server
The API server that connects to Redis Server
The Redis in turn has 3 agents with Hangfire load-balanced (future scalable)
Setup 1 or 2 VMs?
Combination of Windows and Linux Containers, is that advisable?
How many Pods per VM? How many containers per Pod?
Should we attach volumes for DB?
Thank you for your help
Cluster size can be different depending on the Kubernetes platform you want to use. For managed solutions like GKE/EKS/AKS you don't need to create a master node but you have less control over our cluster and you can't use latest Kubernetes version.
It is safer to have at least 2 worker nodes. (More is better). In case of node failure, pods will be rescheduled on another healthy node.
I'd say linux containers are more lightweight and have less overhead, but it's up to you to decide what to use.
Number of pods per VM is defined during scheduling process by the kube-scheduler and depends on the pods' requested resources and amount of resources available on cluster nodes.
All data inside running containers in a Pod are lost after pod restart/deletion. You can import/restore DB content during pod startup using Init Containers(or DB replication) or configure volumes to save data between pod restarts.
You can easily decide which container you need to put in the same Pod if you look at your application set from the perspective of scaling, updating and availability.
If you can benefit from scaling, updating application parts independently and having several replicas of some crucial parts of your application, it's better to put them in the separate Deployments. If it's required for the application parts to run always on the same node and if it's fine to restart them all at once, you can put them in one Pod.
I'm trying to build a web app where each user gets their own instance of the app, running in its own container. I'm new to kubernetes so I'm probably not understanding something correctly.
I will have a few physical servers to use, which in kubernetes as I understand are called nodes. For each node, there is a limitation of 100 pods. So if I am building the app so that each user gets their own pod, will I be limited to 100 users per physical server? (If I have 10 servers, I can only have 500 users?) I suppose I could run multiple VMs that act as nodes on each physical server but doesn't that defeat the purpose of containerization?
The main issue in having too many pods in a node is because it will degrade the node performance and makes is slower(and sometimes unreliable) to manage the containers, each pod is managed individually, increasing the amount will take more time and more resources.
When you create a POD, the runtime need to keep a constant track, doing probes (readiness and Liveness), monitoring, Routing rules many other small bits that adds up to the load in the node.
Containers also requires processor time to run properly, even though you can allocate fractions of a CPU, adding too many containers\pod will increase the context switch and degrade the performance when the PODs are consuming their quota.
Each platform provider also set their own limits to provide a good quality of service and SLAs, overloading the nodes is also a risk, because a node is a single point of failure, and any fault in high density nodes might have a huge impact in the cluster and applications.
You should either consider:
Smaller nodes and add more nodes to the cluster or
Use Actors instead, where each client will be one Actor. And many actor will be running in a single container. To make it more balanced around the cluster, you partition the actors into multiple containers instances.
Regarding the limits, this thread has a good discussion about the concerns
Because of the hard limit if you have 10 servers you're limited to 1000 pods.
You might want to count also control plane pods in your 1000 available pods. Usually located in the namespace kube-system it can include (but is not limited to) :
node log exporters (1 per node)
metrics exporters
kube proxy (usually 1 per node)
kubernetes dashboard
DNS (scaling according to the number of nodes)
controllers like certmanager
A pretty good rule of thumb could be 80-90 application pods per node, so 10 nodes will be able to handle 800-900 clients considering you don't have any other big deployment on those nodes.
If you're using containers in order to gain perfs, creating node VMs will be against your goal. But if you're using containers as a way to deploy coherent environments and scale stateless applications then using VMs as node can make sense.
There are no magic rules and your context will dictate what to do.
As managing a virtualization cluster and a kubernetes cluster may skyrocket your infrastructure complexity, maybe kubernetes is not the most efficient tool to manage your workload.
You may also want to take a look at Nomad wich does not seem to have those kind of limitations and may provide features that are closer to your needs.