I have an application that has 14 different services. Some of the services are dependent on other services. I am trying to find a good way to deploy these in the right sequences without using thread sleeps.
Is there a way to tell kuberentes a service deployment tree like don't deploy service B or service C until Service A is in a container and the status is running?\
Is there s good way to use kubectl to poll service A so I can do a while loop until I know it's up and running then run the scripts to deploy service B and C?
This is not how Kubernetes works. You can kind of shim it with an initContainer that blocks until dependencies are available (usually via kubectl in a while loop, but you get fancy you can try using --wait).
But the expectation is that you set up your applications to be "eventually consistent" when it comes to inter-service dependencies. In practical terms, this usually means just crashing if a dependent service isn't available, and it will just be restarted until things succeed.
You can use the readiness probe to hit health check APIs of your application which is being deployed and in those health check APIs you can test the other service pods availability by hitting their APIs or service
Related
I have two external AcriveMQ url, say mqA and mqB which are used by pod my-pod.
Both mqA and mqB are not in kubernetes cluster, and are not managed by me.
mqA is to be normally used, thus when its unavailable (0.1% probability) then mqB can be used.
Currently if mqA is broken then my-pod crashes and tries to restart.
For me, it's ok to restart the pod or some reasonable fowntime for switching.
So basically I'd like to have a kind of faileover for external service.
Is there a common approach for this in Kubernetes? Like an externalname would be fine. So when mqA is not pingable then it will switch externalName to mqB.
Unfortunately Kubernetes does not have native concept for running active-passive workloads. At this stage you will be better off having fail over at the client side or have some custom operator logic and proxy that will do the fail over for you.
Here's very good information that you may find interesting about active/standby proposal to kubernetes devs with some possible temporary solutions:
https://github.com/amelbakry/kubernetes-active-passive
https://github.com/andreykaipov/active-standby-controller
https://github.com/kubernetes/kubernetes/issues/45300#issuecomment-737658596
I'm kinda new to Kubernets and I think I understand the basics of the whole system but most of the stuff I have read was about how to use kubectl to start a service and deployment and stuff.
But in my use case I have this web API running (built in ASP.net core) that takes a request, does some processing and depending on the input data has to start a secondary process.
A Kubernetes job with restart policy OnFailure seemed to be the way to implement those secondary processes but I can't find any resources on how the web server can be used to start this job.
You can use Kubernetes API to create a Job(or any kubernetes resource) from your application running inside the cluster. You can either install kubectl inside your applications's container and call it from your application code or use a kubernetes client library(https://github.com/kubernetes-client/csharp) to talk to kubernetes API server.
See the following answer for more details:
Kubernetes - Finding out how many replicas there are in a service?
We are now on our journey to break our monolith (on-prem pkg (rpm/ova)) into services (dockers).
In the process we are evaluation envoy/istio as our communication and security layer, it looks great when running as sidecar in k8s, or each service on a separate machie.
As we are going to deliver several services within one machine, and can't deliver it within k8s, I'm not sure if we can use envoy, I didn't find any reference on using envoy in additional ways, are there additional deployment methods I can use to enjoy it?
You can run part of your services on Kubernetes and part on VMs.
I have Consul running in my cluster and each node runs a consul-agent as a DaemonSet. I also have other DaemonSets that interact with Consul and therefore require a consul-agent to be running in order to communicate with the Consul servers.
My problem is, if my DaemonSet is started before the consul-agent, the application will error as it cannot connect to Consul and subsequently get restarted.
I also notice the same problem with other DaemonSets, e.g Weave, as it requires kube-proxy and kube-dns. If Weave is started first, it will constantly restart until the kube services are ready.
I know I could add retry logic to my application, but I was wondering if it was possible to specify the order in which DaemonSets are scheduled?
Kubernetes itself does not provide a way to specific dependencies between pods / deployments / services (e.g. "start pod A only if service B is available" or "start pod A after pod B").
The currect approach (based on what I found while researching this) seems to be retry logic or an init container. To quote the docs:
They run to completion before any app Containers start, whereas app Containers run in parallel, so Init Containers provide an easy way to block or delay the startup of app Containers until some set of preconditions are met.
This means you can either add retry logic to your application (which I would recommend as it might help you in different situations such as a short service outage) our you can use an init container that polls a health endpoint via the Kubernetes service name until it gets a satisfying response.
retry logic is preferred over startup dependency ordering, since it handles both the initial bringup case and recovery from post-start outages
I am trying to create a Kubernetes job that consists of two pods that have to be scheduled on separate nodes in our Hybrid cluster. Our requirement is that one of the pods runs on a Windows Server node and the other pod is running on a Linux node (thus we cannot just run two Docker containers from the same pod, which I know is possible, but would not work in our scenario). The Linux pod (which you can imagine as a client) will communicate over the network with the Windows pod (which you can imagine as a stateful server) exchanging data while the job runs. When the Linux pod terminates, we want to also terminate the Windows pod. However, if one of the pods fail, then we want to fail both pods (as they are designed to be a single job)
Our current design is to write a K8S service that handles the communication between the pods, and then apply the service and the two pods to the cluster to "emulate" a job. However, this is not ideal since the two pods are not tightly coupled as a single job and adds quite a bit of overhead to manually manage this setup (e.g. when failures or the job, we probably need to manually kill the service and deployment of the Windows pod). Plus we would need to deploy a new service for each "job", as we require the Linux pod to always communicate with the same Windows pod for the duration of the job due to underlying state (thus cannot use a single service for all Windows pods).
Any thoughts on how this could be best achieved on Kubernetes would be much appreciated! Hopefully this scenario is supported natively, and I would not need to resort in this kind of pod-service-pod setup that I described above.
Many thanks
I am trying to distinguish your distaste for creating and wiring the Pods from your distaste at having to do so manually. Because, in theory, a Job that creates Pods is very similar to what you are describing, and would be able to have almost infinite customization for those kinds of rules. With a custom controller like that, one need not create a Service for the client(s) to speak to their server, as the Job could create the server Pod first, obtain its Pod-specific-IP, and feed that to the subsequently created client Pods.
I would expect one could create a Job controller using only bash and either curl or kubectl: generate the json or yaml that describes the situation you wish to have, feed it to the kubernetes API (since the Job would have a service account - just like any other in-cluster container), and use normal traps to cleanup after itself. Without more of the specific edge cases loaded in my head it's hard to say if that's a good idea or not, but I believe it's possible.