I have a couple of Stateful Sets, one is dependent on another one, I need the pods in the first Stateful Set to be in Ready state before the 2nd Stateful Set is started to be scaled.
I'm looking to see if there is a way to link the two Stateful Sets to achieve this in an automatic way, instead of me manually doing it. I have the same concern about Deployments as well.
Currently I don't use Helm or any other tools, just kubectl.
I would advise against doing this as it would increase coupling between different services.
A better way of handling your problem is for containers in the dependent service to check if the other service is available. If it isn't, it's ok to crash the container since Kubernetes provides self-healing by automatically restarting containers for you. This way, the dependent service will try to connect to the service and if the latter isn't available, then the dependent service will crash and try again later using exponential back-off.
Related
I have my main application which has its own unique state, let's call it Application A.
This application A starts a few processes which does some parsing work and then it collects it and should send it to a database server outside of the Kuberentes cluster.
I would like to run a few copies of this application A in different pods. however, each instance is unique and cannot be replaced as it has its own state. it means that each client has to talk only with the same instance it started the communication with http requests.
How can it be done in Kubernetes?
do I need to define StatefulSet component?
how do I manage that each client (from outside the cluster) will talk every time with the same instance he started communication on the same object id ? for example to get status on that object.
in case the pod die I don't want to recover. is that possible?
1: yes, sort of
2: not necessarily, but might simplify some things
3: if you use ingress, you can use different methods to maintain backend affinity ie. cookie based, source IP based etc. (nginx example: https://kubernetes.github.io/ingress-nginx/examples/affinity/cookie/)
4: you might want to set restartPolicy to Never
With all that said, this really sounds like a bad idea. You should either allow shared state (ie. redis), or statefulset with ability to restart with the same state loaded from local storage. You need to remember that even with the most optimal setup things like this can break (ie. switch to different pod when a backing pod went down, node rescheduling due to cluster scaling etc.)
For the number 4 question. You only need to set up the container-restart-policy. I used this flag to create a pod with this feature: --restart=Never
IMHO, It is not a Kubernetes problem. You could have this scenario in other environments. The idea is to use sticky sessions to have an affinity for all your request. You probably need to search for this setup in your ingress controller documentation. E.g Nginx Ingress
I am trying to deploy multiple pods in k8s like say MySQL, Mango, Redis etc
Can i create a single deployment resource for this and have multiple containers defined in template section? Is this allowed? If so, how will replication behave in this case?
Thanks
Pavan
I am trying to deploy multiple pods in k8s like say MySQL, Mango,
Redis etc
From microservices architecture perspective it is actually quite a bad idea to place all those containers in a single Pod. Keep in mind that a Pod is a smallest deployable unit that can be created and managed by Kubernetes. There are quite many good reasons you don't want to have all above mentioned services in a single Pod. Difficulties in scaling such solution is just one of them.
Can i create a single deployment resource for this and have multiple
containers defined in template section? Is this allowed? If so, how
will replication behave in this case?
No, it is not allowed in Kubernetes. As to Deployments and StatefulSets, (which you need for statefull applications such as databases) both manage Pods that are based on identical container spec so it is not possible to have a Deployment or StatefulSet consisting of different types of Pods, based on different specs.
To sum up:
Many Deployments and StatefulSets objects, serving for different purposes are the right solution.
A deployment can have multiple containers inside of it.
Generaly it's used to have one master container for the app and some sidecar container that are needed for the app. I don't have an example right now.
Still it's a best practice to split deployments for scalling purpose, your front may need to scale more than the back depending on cache and you may not want to have pods too big. For cahing purpose like redis it's better to have a cluster on the side as each time a pod start or stop, you will loose data.
It's common having multiple containers per Pod in order to share namespaces and volumes between them: take as example the Ambassador pattern that is used to present the application to outside adding a layer for the authentication, making it totally transparent to the main app.
Other examples using the sidecar pattern consist of log parsers or configurators that hot reload credentials without the main app to worry about it.
That's the theory, according to your needs you have to use one deployment per component, so a Deployment for your app, a StatefulSet for the DB and so on. Keep in mind to use a container per process and a Kubernetes resource per backing service.
I am running a pod that has three containers. need to update the image of one of the container without doing a rolling upgrade.
How do I get the container image updated without touching/restarting the other two containers?
If you are asking yourself this question, maybe you should reconsider some things.
As stated in the others comment/answers, a pod once created is one unit whatever is inside of it.
If you ever needs to scale some part of the pod and not the rest or do updates of just a part and don't want to restart the rest (a caching system for example), you should look to take out the container from you deployment and create another independent one.
You must delete and recreate the entire Pod. If you look at the API definition of the Kubernetes PodSpec, both the containers and initContainers fields contain the phrase Cannot be updated. A Pod is immutable once created, and the various higher-level objects work by creating and destroying Pods.
In the standard practice I'm used to:
Always use higher-level objects to manage your containers, most often Deployments; don't deal directly in Pods. You can change the Pod spec embedded in a Deployment spec, and the deployment controller will create new pods matching the new spec and then tear down the old pods (that is, a zero-downtime rolling upgrade).
Put only one container in a Pod if at all possible. You mention an initContainer so this may not be 100% possible for you, but the only way you can achieve "update A without restarting B" is to have them be in different Pods (and therefore different Deployments).
Don't stress about restarting things. If a Node fails, or you need to update a container image like you show, or in some other cases, a Pod can get stopped and deleted and a new Pod started in its place. This is totally normal, and if you have a Deployment with multiple replicas, fairly transparent. Your application may need to do a little work to be happy in this environment (not keeping state only in memory or local disk).
I don't think you can, because a pod an atomic unit, contains all the containers.
As I have been using kubernetes more I keep on seeing the reference that a pod can contain 1 container or more and I have even looked at examples.
My question is whether there is a case where this would be best practice and more efficient to create multi container pods since you can scale and replicate your pods coupling it with a service.
Thanks in advance
A Pod can contain multiple containers, but for the most portion of the situations, it makes perfect sense for the Pod to be simply an abstraction over a single running container.
In what situations does it make sense to have a multi-container deployed Pod?
What comes to my mind are the scenarios where you have a primary Pod running, but you need to tightly couple helper processes, such as a log watcher. In those situations, it makes perfect sense to actually have multiple containers running inside a single pod.
Another big example that comes to my mind is from the Istio project, which is a platform made to connect, manage and secure microservices and is generally referred as a Service Mesh.
A huge part of what it does and is able to accomplish to provide a greater control and customization over the deployed microservices network, is due to the fact that it deploys a sidecar proxy, denominated Envoy, throughout the environment intercepting all network communication between microservices.
Here, you can check an example of load balancing in a Istio service mesh. As you can see the Proxy is deployed inside the Pod, intercepting all communication that goes through it.
I am trying to create a Kubernetes job that consists of two pods that have to be scheduled on separate nodes in our Hybrid cluster. Our requirement is that one of the pods runs on a Windows Server node and the other pod is running on a Linux node (thus we cannot just run two Docker containers from the same pod, which I know is possible, but would not work in our scenario). The Linux pod (which you can imagine as a client) will communicate over the network with the Windows pod (which you can imagine as a stateful server) exchanging data while the job runs. When the Linux pod terminates, we want to also terminate the Windows pod. However, if one of the pods fail, then we want to fail both pods (as they are designed to be a single job)
Our current design is to write a K8S service that handles the communication between the pods, and then apply the service and the two pods to the cluster to "emulate" a job. However, this is not ideal since the two pods are not tightly coupled as a single job and adds quite a bit of overhead to manually manage this setup (e.g. when failures or the job, we probably need to manually kill the service and deployment of the Windows pod). Plus we would need to deploy a new service for each "job", as we require the Linux pod to always communicate with the same Windows pod for the duration of the job due to underlying state (thus cannot use a single service for all Windows pods).
Any thoughts on how this could be best achieved on Kubernetes would be much appreciated! Hopefully this scenario is supported natively, and I would not need to resort in this kind of pod-service-pod setup that I described above.
Many thanks
I am trying to distinguish your distaste for creating and wiring the Pods from your distaste at having to do so manually. Because, in theory, a Job that creates Pods is very similar to what you are describing, and would be able to have almost infinite customization for those kinds of rules. With a custom controller like that, one need not create a Service for the client(s) to speak to their server, as the Job could create the server Pod first, obtain its Pod-specific-IP, and feed that to the subsequently created client Pods.
I would expect one could create a Job controller using only bash and either curl or kubectl: generate the json or yaml that describes the situation you wish to have, feed it to the kubernetes API (since the Job would have a service account - just like any other in-cluster container), and use normal traps to cleanup after itself. Without more of the specific edge cases loaded in my head it's hard to say if that's a good idea or not, but I believe it's possible.