Is there any way we can correctly guess how much resource limits we need to keep for running deployments on kubernetes clusters.
Yes, you can guess that single threaded application most likely won't need more that 1 CPU.
For any other programs: no, there is not easy way to guess it. Every application is different, and reacts differently under different workloads.
The easiest way to figure out how many resources it needs is to run it and measure it.
Run some benchmarks/profilers and see how application behaves. Then make decisions based on that.
Related
We have a deployment stack with about 20 microservices/pods. Each deployment goes to its own namespace. To make sure that the cpu and memory are guaranteed for each pod and not shared, we set the request amounts the same as limit amount. Now we sometimes need to deploy more stack into the same performance cluster, e.g. testing different releases of the same stack. The question is whether having more than one deployment in one cluster can invalidate the test result due to shared network or some other reasons?
Initially we were thinking to create one cluster for each performance testing to make sure it is isolated and test results are correct but creating a new cluster and maintaining it a very costly. We also thought about making sure each deployment goes to one node to avoid load testing on one stack impact the others but I'm not sure if that really helps. Please share your knowledge on this as Kubernetes is almost new to us.
If the containers are running on the same underlying hosts then bleedthrough is always possible. If you set all pods into Guaranteed QoS mode (aka requests == limits) then it at least reduces the bleedthrough to a minimum. Running things on one cluster is always fine but if you want to truly reduce the crosstalk to zero then you would need dedicated workload nodes for each.
We’re in the process of migrating our aging monolith to a more robust solution and landed on Kubernetes as the most appropriate platform to achieve what we’re looking for. At the same time, we’re looking to split out and isolate our client data for security and improved privacy.
What we’re considering is ultimately having one database per customer, and embedding those connection details into a deployment for each of them. We’d then build a routing service of some kind that would link a client’s request to their respective deployment/service.
Because our individual clients vary wildly in size (we have some that generate thousands of requests per minute, and others that are hundreds per day), we like the option of having the ability to scale them independently through ReplicaSets on the deployments.
However, I have some concerns regarding upper limits of how many deployments can exist/be successfully managed within a cluster, as we’d be looking at potentially hundreds of different clients, which will continue to grow. I also have concerns of costs, and how having dedicated resources (essentially an entire VM) for our smaller clients might impact our budgets.
So my questions are:
is this a good idea at all? Why or why not, and if not, are there alternative architectures we could look at to achieve the same thing?
is this solution more expensive than it needs to be?
I’d appreciate any insights you could offer, thank you!
I can think of a couple options for this situations:
Deploying separate clusters for each customer. This also allows you to size your clusters properly for each customers and configure autoscaling accordingly for each of them. The drawback is that each cluster has a management fee of 0.10$ per hour, but you get full guarantee that everything is isolated, and you can use the cluster autoscaler to make sure that only the VMs that are actually needed for each customer are running. For smaller customers, you may wanna use this with small (and cheap) machine types.
Another option would be to, as mentioned in the comments, use namespaces. However you would have to configure the cluster properly as there exist ways of accessing services in different namespaces.
Implement customer isolation in your own software running on a cluster. This would imply forcing your software to access only the database for a given customer, but I would not recommend to go this route.
we have a small problem with the kubernetes cluster.
Because one of our applications is so demanding that sometimes consume all of our resources and finally some of pods are killed. The real problem starts when system pods like flannel or cache became removed.
Is there a recommended way to control what is being removed? How "save" system pods? Maybe someone has experience in this topic?
One of the ideas is to change QoS for all pods/apps from the kube-system to "Guaranteed". But I'm afraid that this will not work well if we limit resources, even with a large margin.
Btw. where can I read about what (default) requirements system services have? How set it on cluster creation phase?
The second idea is setting the Eviction Policy and/or Taints and Tolerations, but there is a anxiety that our key application will be (re)moved as one of the first. Unfortunately it currently works only in one copy and the initialization can take up to several minutes, so switching between nodes is currently unacceptable and impossible.
The final idea is to use Priority and Preemption, but from what I see in the 1.8.1 documentation is still in the "alpha" phase, and I have serious concerns about the stability of this solution.
Maybe there is something else I did not think about? I will be happy to listen other proposals.
I need an Akka cluster to run multiple CPU intensive jobs. I cannot predict how much CPU power I need. Sometimes load is high, while at other times, there isn't much load. I guess autoscaling is a good option, which means, example: I should be able to specify that I need minimum 2 and maximum 10 Actors. The cluster should scale up or down along with a cool off period as load goes up or down. Is there a way to do that?
I am guessing, maybe one can make an Docker image of the codebase, and autoscale it using Kubernetes. Is it possible? Is there a native Akka solution?
Thanks
If you consider a project like hseeberger/constructr and its issue 179, a native Akka solution should be based on akka/akka-management:
This repository contains interfaces to inspect, interact and manage various Parts of Akka, primarily Akka Cluster. Future additions may extend these concepts to other parts of Akka.
There is a demo for kubernetes.
I've rather a teoretical question which I can't answer with the reousrces found online. The question is: what's the rule to decide how to compose containers in POD? . Let me explain with an example.
I've these microservices:
Authentication
Authorization
Serving content
(plus) OpenResty to forward the calls form one to the other and orhcestarate the flow. (is there a possibility to do so natively in K8?, it seems to have services base on nginx+lua, but not sure how it works)
For the sake of the example I avoid Databases and co, I assume they are external and not managed by kubernetes
Now, what's the correct way here LEFT or RIGHT of the image?
LEFT : this seems easier to make it working, everything works on "localhost" , the downside is that it looses a bit the benefit of the microservices. For example, if the auth become slows and it would need more instances, I've to duplicate the whole pod and not just that service.
RIGHT seems a bit more complex, need services to expose each POD to the other PODs. Yet, here, I could duplicate auth as I need without duplicating the other containers. On the other hand I'll have a lot of pods since each pod is basically a container.
It is generally recommended to keep different services in different pods or better deployments that will scale independently. The reasons are what is generally discussed as benefits of a microservices architecture.
A more loose coupling allowing the different services to be developed independently in their own languages/technologies,
be deployed and updated independently and
also to scale independently.
The exception are what is considered a "helper application" to assist a "primary application". Examples given in the k8s docs are data pullers, data pushers and proxies. In those cases a share file system or exchange via loopback network interface can help with critical performance use cases. A data puller can be a side-car container for an nginx container pulling a website to serve from a GIT repository for example.
right image, each in own pod. multi containers in a pod should really only be used when they are highly coupled or needed for support of the main container such as a data loader.
With separate pods, it allows for each service to be updated and deployed independently. It also allows for more efficient scaling. in the future, you may need 2 or 3 content pods but still only one authorization. if they are all together you scale them all since you don't have a choice with them all together in the same pod.
Right image is better option. Easier management, upgrades, scaling.
Should choose the right side of the structure, on the grounds that the deployment of the left side of the architecture model is tight coupling is not conducive to a module according to the actual needs of the business expansion capacity.