I'm new to Kubernetes and want to know the best approach to this problem.
I have a varying set of large models (~5GB) that need to be loaded into memory for my application to run. The app handles a request that specifies which model it needs, but the actual task is the same. I don't want to load all of the models with a single pod for cost reasons, and so models can be added/removed more easily. Can I have a single service with pods that each have a different subset of the resources loaded (1 or 2 each), with a request being directed to the pod that has the model it needs? Or do I need to make a service for each model, and then a gateway service in front of all of them?
I'm thinking it might be possible with process namespaces, but I'm not sure how customizable a service master can be in terms of parsing a request parameter and sending it the right namespace.
Related
I have an application that currently is deployed multiple times to an Kubernetes cluster because it needs to call different sources based on the URLs that it is called with. You could call them lanes. These deployments are well automated by a CI/CD pipeline.
The problem is, most of these lanes are not frequently used, yet they still need to be available. And we have a lot of applications following this pattern. I'd love to be able to deploy less pods that can handle the traffic of all lanes and still call the appropriate dependencies.
I know we could 'fix' this problem by incorporating switching logic within the application and passing the lane in a header or something, but that seems like a can of worms and can be problematic in production if that logic isn't necessary there.
I know you can have a single deployment with multiple ingresses.
Is it possible to use the ingress API to accomplish something like this in my kube.yml where I could choose or rewrite outgoing urls based on which ingress was called?
https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.19/
Is there something in the nginx API that I could use in my kube.yml to accomplish this?
https://docs.nginx.com/nginx-ingress-controller/
Thanks for the help!
Lets say we have following apps ,
API app : Responsible for serving the user requests.
Backend app: Responsible for handling the user requests which are long running tasks. It updates the progress to database (postgres) and distributed cache (Redis).
Both apps are scalable service. Single Backend app handles multiple tenants e.g. Customer here but one customer is assigned to single backend app only.
I have a usecase where I need API layer to connect to specific replica which is handling that customer. Do we have a common Pattern for this ?
Few strategies in mind
Pub/Sub: Problem is we want sync guranteed response , probably using Redis
gRPC : Using POD IP to connect to specific pod is not a standard way
Creating a Service at runtime by adding labels to the replicas and use those. -- Looks promising
Do let me know if there is common pattern or example architecture of this or standard way of doing this?
Note :[Above is a simulation of production usecase, names and actual use case is changed]
You should aim to keep your services stateless, in a Kubernetes environment there is no telling when one pod might be replaced by another due to worker node maintenance.
If you have long running task that cannot be completed during the configured grace period for pods to shutdown during a worked node drain/evacuation you need to implement some kind of persistent work queue as your are think about in option 1. I suggest you look into the saga pattern.
Another pattern we usually employ is to let the worker service write the current state of the job into the database and let the client pull the status every few seconds. This does however require some way of handling half finished jobs that might be abandoned by pods that are forced to shutdown.
I was reading the tutorial on deploying a Cassandra ring and zookeeper with statefulsets. What I don't understand is if I decide to add another replica into the statefulset, how do I notify the other PODS that there is another one. What are best practices for it? I want to be able for one POD to redirect request to another POD in my custom application in case the request doesn't belong to it (ie. it doesn't have the data)
Well, seems like you want to run a clustered application inside kubernetes. It is not something that kubernetes is directly responsible for. The cluster coordination for given solution should be handled within it, and a response to a "how to" question can not be generic.
Most of the softwares out there will have some kind of coordination, discovery and registration mechanism. Be it preconfigured members, external dioscovery catalog/db or some networ broadcasting.
StatefulSet helps a lot in it by retaining network identity under service/pod, or helping to keep storage, so you can ie. always point your new replicas to register with first replica (or preferably one of the first two, cause what if your no.1 is the one that restarted), but as a wrote above, this is pretty much depending on capabilities available on the solution you want to deploy.
I've rather a teoretical question which I can't answer with the reousrces found online. The question is: what's the rule to decide how to compose containers in POD? . Let me explain with an example.
I've these microservices:
Authentication
Authorization
Serving content
(plus) OpenResty to forward the calls form one to the other and orhcestarate the flow. (is there a possibility to do so natively in K8?, it seems to have services base on nginx+lua, but not sure how it works)
For the sake of the example I avoid Databases and co, I assume they are external and not managed by kubernetes
Now, what's the correct way here LEFT or RIGHT of the image?
LEFT : this seems easier to make it working, everything works on "localhost" , the downside is that it looses a bit the benefit of the microservices. For example, if the auth become slows and it would need more instances, I've to duplicate the whole pod and not just that service.
RIGHT seems a bit more complex, need services to expose each POD to the other PODs. Yet, here, I could duplicate auth as I need without duplicating the other containers. On the other hand I'll have a lot of pods since each pod is basically a container.
It is generally recommended to keep different services in different pods or better deployments that will scale independently. The reasons are what is generally discussed as benefits of a microservices architecture.
A more loose coupling allowing the different services to be developed independently in their own languages/technologies,
be deployed and updated independently and
also to scale independently.
The exception are what is considered a "helper application" to assist a "primary application". Examples given in the k8s docs are data pullers, data pushers and proxies. In those cases a share file system or exchange via loopback network interface can help with critical performance use cases. A data puller can be a side-car container for an nginx container pulling a website to serve from a GIT repository for example.
right image, each in own pod. multi containers in a pod should really only be used when they are highly coupled or needed for support of the main container such as a data loader.
With separate pods, it allows for each service to be updated and deployed independently. It also allows for more efficient scaling. in the future, you may need 2 or 3 content pods but still only one authorization. if they are all together you scale them all since you don't have a choice with them all together in the same pod.
Right image is better option. Easier management, upgrades, scaling.
Should choose the right side of the structure, on the grounds that the deployment of the left side of the architecture model is tight coupling is not conducive to a module according to the actual needs of the business expansion capacity.
Looking at the design docs for Resource Quotas, it appears they can be set on a namespace level. That makes sense, to limit total resource usage of a Namespace.
In my organization, there are many teams, each working on their own microservice. The microservices need to communicate with one another, so I want them all to be in the same namespace, for ease of service discovery. However, I'd like to put some protections in place so that one of the microservices doesn't accidentally take over all the resources in the namespace.
I was planning on giving each microservice its own Service Account. Ideally, I would like to be able to set resource quotas on those service accounts. But I can't find any way in the documentation to do that.
Is that possible? If not, is something like this planned for the future?
Thanks.
Quota is per namespace. I don't think it is likely to be updated to subdivide a namespace by user (or service account). Right now, resources don't have the concept of an "owner", so it isn't possible to attribute a pod in a namespace to a particular creator. It gets even muddier when replication controllers create pods automatically. If one person creates a replication controller and another person scales it up, which one of them "owns" those pods? Those are the sorts of questions that would need clear answers before ownership could factor into quota.