we are planning to use kubernetes and I am validating, if and how it fits our requirements.
One concern is the following:
I want to build an application/pod, which is connecting to a certain service on the internet (host and port) and keeps the socket alive as long as we need it (usually forever). The number of sockets the application will connect to may vary.
For the inter pod communication we are going to use RabbitMQ.
What is the correct/best practise approach for that purpose?
One pod handling all/multiple sockets?
Replicated pods handling multiple socket?
One Socket per pod?
How do i react, if the number of sockets changes?
At the moment we want to use the gitlab-ci and helm for our CI pipeline.
kubernetes deploys Pods and has two abstractions: Deployments and StatefulSets. The former deploys ephemeral Pods whose hostname and IP changes. The latter retains the state.
If you're deploying kubernetes only for this application, it's an overkill imho. I'd rather use plain Docker or a simpler-than-kubernetes orchestrator such as Docker Swarm Mode or Kontena.
If kubernetes is your only option, you could deploy the app as a StatefulSet. That way its hostname will remain between restarts. Have the app monitor its hostname and connect to the appropriate endpoint. For example, the app-1 Pod connects to endpoint:10001, app-2 Pod connects to endpoint:10002, and so on...
When more Pods are needed to connect to more sockets, either increase the StatefulSet's replicas manually, or write a sidecar application to monitor the no. of sockets and up/down the replicas automatically.
Related
I have an issue with k8s volumes
The structure
We talk about 2 services (pods) that service one generates files and the other service expose those files over the web.
Service number one needs a UDP connection open to the world and also a TCP connection to communicate between other pods.
(two services, one for UDP and one for TCP)
Service Number two needs a connection to the world (web).
(one service that connects to ingress)
The Issue
I need volume in memory between those two pods, to speed up the R/W process.
The solution I checked, to work with the multi-container structure using EmptyDir volume
(there is an option to run this volume in memory)
The problem with this solution is that I can't connect the k8s service object for those containers, service connects only to pods, and only the pod gets IP but containers not.
there is any solution or idea for this situation?
p.s. I'm running on AKS if it's matters.
If you really want to use a multi-container pod, why don’t you create two Kubernetes service resources? One mapping to ContainerPort A and one mapping to containerPort B? This way you would Expose every Container independently.
Keep in mind, that EmptyDir is not „in memory“ it is just a shared volume on the same host, accessible only for the containers that are sharing the emptyDir.
Edit: as #Matt pointed out I was wrongly informed: I was not aware of the emptyDir.medium="Memory" setting.
So another solution could be to go with two independent pods and a dedicated volume (either on host level or a Kubernetes persistent volume). With taints and tolerations you can then ensure, that both pods are scheduled on the same node where the actual volume is attached.
Most of the examples I come across to use Kafka in Kubernetes is to deploy it as a headless service but I am not able to get the answer yet on why it should be headless and not Cluster IP? In my opinion cluster, IP provides the load balancing in which we ensure out of the box that not only one of the broker gets loaded always with its resources as I see with headless the Kafka clients be it sarma or java client tries to pick always the first IP from the DNS lookup and connects to it, will this not be a bottleneck if there are around 100+ clients trying to do the same and open connection to the first IP? or Kafka handles this inbuilt already which I am still trying to understand how it really happens.
When there is no differentiation between various instances of a services(replicas of a pod serving a stateless application), you can expose them under a ClusterIP service as connecting to any of the replica to serve the current request is okay. This is not the case with stateful services(like Kafka, databases etc). Each instance is responsible for it's own data. Each instance might be owning a different partition/topic etc. The instances of the service are not exact "replicas". Solutions for running such stateful services on Kubernetes usually use headless services and/or statefulsets so that each instance of the service has a unique identity. Such stateful applications usually have their own clustering technology that rely on each instance in the cluster having a unique identity.
Now that you know why stable identities are required for stateful applications and how statefulsets with headless services provide stable identities, you can check how your Kafka distributions might using them to run Kafka on kubernetes.
This blog post explains how strimzi does it:
For StatefulSets – which Strimzi is using to run the Kafka brokers –
you can use the Kubernetes headless service to give each of the pods a
stable DNS name. Strimzi is using these DNS names as the advertised
addresses for the Kafka brokers. So with Strimzi:
The initial connection is done using a regular Kubernetes service to
get the metadata.
The subsequent connections are opened using the DNS
names given to the pods by another headless Kubernetes service.
It's used in cases where communication to specific Pods is needed.
For example, A monitoring service must be able to reach all pods behind a service, to check their status, so it needs the addresses of all Pods and not just any one of them. This would be a use case of headless service.
Or when there is a cluster of Pods being set up, it's important to coordinate with the Pods to keep the cluster working for consumers. In Kafka, this work is done by Zookeeper. thus a headless service is needed by Zookeeper
Stateful:
Kafka streaming platform maintain replicas of partition across kafka brokers based on RELICATION_FACTOR. It maintains it data across persistent storage. When it comes to K8s ; stateful type is suggested; Pods in StatefulSets are not interchangeable: each Pod has a unique identifier that is maintained no matter where it is scheduled.
Headless:
To maintain internal communication between PODS. Lets not forget Zookeeper orchestrates kafka brokers.
Thanks
Within POD they should know eachother who is running and who stopped
One of my micro service is running on Kubernetes.
I would like to specify to K8s load balancer when a pod is busy because the behaviour that I get currently is not ok.
One example:
I have 8 pods running, each pod can process 1 request at a time. Each request take from 70 to 100% of the CPU core allocated to the pod.
But when I send 8 requests to my application, Kubernetes does not dispatch those requests to the 8 pods but try to use only one. And since I'm blocking (via threadpool) each replica of app to use only one thread at a time, of course requests are queued for pod 1.
So my question is: How can I tell Kubernetes that POD 1 is busy and that load-balancer must dispatch request 2 to POD 2 ?
Note: For dev and test purpose I'm using Docker Desktop (Docker for Windows) on Windows 10 and kubectl.
As prometherion suggested you can use the liveness probe and also i would suggest to add the rediness probe together.
you can have a look at the official document : https://kubernetes.io/docs/tasks/configure-pod-container/configure-liveness-readiness-probes/
Sometimes, applications are temporarily unable to serve traffic. For example when, application first need to load large data or configuration files during startup.
In such cases, you don’t want to kill the application, but you don’t want to send traffic either there to pods. K8s provides readiness probes to detect and mitigate these situations. A pod with containers reporting that they are not ready does not receive traffic through Kubernetes Services.
You have to use LivenessProbe
when a Pod will not able to handle a request its IP will be removed from Service endpoints, so no traffic will be forwarded to it.
LivenessProbe can be TCP or HTTP
I have played around a little bit with docker and kubernetes. Need some advice here on - Is it a good idea to have one POD on a VM with all these deployed in multiple (hybrid) containers?
This is our POC plan:
Customers to access (nginx reverse proxy) with a public API endpoint. eg., abc.xyz.com or def.xyz.com
List of containers that we need
Identity server Connected to SQL server
Our API server with Hangfire. Connected to SQL server
The API server that connects to Redis Server
The Redis in turn has 3 agents with Hangfire load-balanced (future scalable)
Setup 1 or 2 VMs?
Combination of Windows and Linux Containers, is that advisable?
How many Pods per VM? How many containers per Pod?
Should we attach volumes for DB?
Thank you for your help
Cluster size can be different depending on the Kubernetes platform you want to use. For managed solutions like GKE/EKS/AKS you don't need to create a master node but you have less control over our cluster and you can't use latest Kubernetes version.
It is safer to have at least 2 worker nodes. (More is better). In case of node failure, pods will be rescheduled on another healthy node.
I'd say linux containers are more lightweight and have less overhead, but it's up to you to decide what to use.
Number of pods per VM is defined during scheduling process by the kube-scheduler and depends on the pods' requested resources and amount of resources available on cluster nodes.
All data inside running containers in a Pod are lost after pod restart/deletion. You can import/restore DB content during pod startup using Init Containers(or DB replication) or configure volumes to save data between pod restarts.
You can easily decide which container you need to put in the same Pod if you look at your application set from the perspective of scaling, updating and availability.
If you can benefit from scaling, updating application parts independently and having several replicas of some crucial parts of your application, it's better to put them in the separate Deployments. If it's required for the application parts to run always on the same node and if it's fine to restart them all at once, you can put them in one Pod.
I am trying to build a service that needs to be connected to a socket over the internet without downtime. The service will be reading and publishing info to a message queue, messages should be published only once and in the order received.
For this reason I thought of deploying it into Kubernetes where I can automatically have multiple replicas in case one process fails, i.e. just one process (pod) should be running all time, not multiple pods publishing the same messages to the queue.
These requests need to be routed through a proxy with a static IP, otherwise I cannot connect to the socket. I understand this may not be a standard use case as a reverse proxy as it is normally use with load balancers such as Nginx.
How is it possible to build this kind of forward proxy in Kubernetes?
I will be deploying this on Google Container Engine.
Assuming you're happy to use Terraform, you can use this:
https://github.com/GoogleCloudPlatform/terraform-google-nat-gateway
However, there's one caveat and that is it may inbound traffic to other clusters in that same region/zone.
Is the LoadBalancer that you need?
kubernetes create external loadbalancer,you can see this doc.