I would like to expose one pod that has a lot of ports (including big port range, thousands of them) to the cluster members (namely as a ClusterIP service). So manually listing them in the service definition is not really possible (Kubernetes does not support exposing port ranges yet).
The container in the pod will run Samba AD DC (here I am just showing that there are really a lot of ports): https://wiki.samba.org/index.php/Samba_AD_DC_Port_Usage
I have been trying to find out how to expose the whole pod (like DMZ on a service); if it is possible at all. I am not sure if this is the best approach to get the goal I want (exposing the whole pod to the internal cluster network).
To summarize the question, is there any way to expose the whole ports (or the whole pod let's say) to the internal cluster network (or to any network of choice) using Service?
I am not sure if I am missing something that can be done better in this regard.
Related
Until now, our solution ran on three containers on the same IP.
One of the containers had Redis. When the other two containers (communicating via message passing on Redis) needed to reach the Redis, they simply utilized port 6379. Since the Redis had the same machine/IP this was simple and straight forward.
Now there is a need to run these three dockers on Kuberenetes and each docker container is now getting its own unique IP address. How does one manage that correctly?
What if additional Redis containers are needed. How does each container knows which one to couple with?
When switching to Kubernetes, you have to rely on the K8s architecture itself and think in a cloud-native manner.
Consider that a pod is ephemeral: its IP will, potentially and eventually, change, anytime. You can not rely on a pod's IP.
What you can do is create a service (a clusterIP works great for this use case) that will serve as the entry point for every replicas of your Redis pod. The replicas should rely on the service, and K8s will take care of updating the list of : backing the service.
You can find a great tutorial of doing this here.
In a multiple node cluster we want to expose a service handling UDP traffic. There are two requirements:
We want the service to be backed up by multiple pods (possibly running on different nodes) in order to scale horizontally.
The service needs the UDP source IP address of the client (i.e., should use DNAT instead of SNAT)
Is that possible?
We currently use a NodePort service with externalTrafficPolicy: local. This forces DNAT but only the pod running on the requested node is receiving the traffic.
There doesn't seem to be a way to spread the load over multiple pods on multiple mnodes.
I already looked at this Kubernetes tutorial and also this article here.
The Problem
I feel like there is a need for some explanation before facing the actual issue(s) in order to understand why things do not work as expected:
Usually what happens when using NodePort is that you expose a port on every node in your cluster. When making a call to node1:port the traffic will then (same as with a ClusterIP type) be forwarded to one Pod that matches the selector, regardless of that Pod being on node1 or another node.
Now comes the tricky part.
When using externalTrafficPolicy: Local, packages that arrive on a node that does not have a Pod on it will be dropped.
Perhaps the following illustration explains the behavior in a more understandable way.
NodePort with default externalTrafficPolicy: Cluster:
package --> node1 --> forwards to random pod on any node (node1 OR node2 OR ... nodeX)
NodePort with externalTrafficPolicy: Local:
package --> node1 --> forwards to pod on node1 (if pod exists on node1)
package --> node1 --> drops package (if there is no pod on node1)
So in essence to be able to properly distribute the load when using externalTrafficPolicy: Local two main issues need to be addressed:
There has to be a Pod running on every node in order for packages not to be dropped
The client has to send packages to multiple nodes in order for the load to be distributed
The solution
The first issue can be resolved rather easily by using a DaemonSet. It will ensure that one instance of the Pod runs on every node in the cluster.
Alternatively one could also use a simple Deployment, manage the replicas manually and ensure proper distribution across the nodes by using podAntiAffinity. This approach would take more effort to maintain since replicas must be adjusted manually but can be useful if you want to have more than just 1 Pod on each node.
Now for the second issue.
The easiest solution would be to let the client implement logic on his part and send requests to all the nodes in a round robin principle, however, that is not a very practical and/or realistic way of doing it.
Usually when using NodePort there is still a load balancer of some kind in front of it to distribute the load (not taking about the Kubernetes service type LoadBalancer here). This may seem redundant since by default NodePort will distribute the traffic across all the Pods anyways, however, the node that gets requested still gets the traffic and then another hop happens. Furthermore if only the same node is addressed at all time, once that node goes down (for whatever reason) traffic will never reach any of the Pods anyways. So for those (and many other reasons) a load balancer should always be used in combination with NodePort. To solve the issue simply configure the load balancer to preserve the source IP of the original client.
Furthermore, depending on what cloud you are running on, there is a chance of you being able to configure a service type LoadBalancer instead of NodePort (which basically is a NodePort service + a load balancer in front of it as described above) , configure it with externalTrafficPolicy: Local and address the first issue as described earlier and you achieved what you wanted to do.
For instance, I have a bare-metal cluster with 3 nodes ich with some instance exposing the port 105. In order to expose it on external Ip address I can define a service of type NodePort with "externalIPs" and it seems to work well. In the documentation it says to use a load balancer but I didn't get well why I have to use it and I worried to do some mistake.
Can somebody explain whay I have to use external (MetalLB, HAProxy etc) Load Balancer with Bare-metal kubernetes cluster?
You don't have to use it, it's up to you to choose if you would like to use NodePort or LoadBalancer.
Let's start with the difference between NodePort and LoadBalancer.
NodePort is the most primitive way to get external traffic directly to your service. As the name implies, it opens a specific port on all the Nodes (the VMs), and any traffic that is sent to this port is forwarded to the service.
LoadBalancer service is the standard way to expose a service to the internet. It gives you a single IP address that will forward all traffic to your service.
You can find more about that in kubernetes documentation.
As for the question you've asked in the comment, But NodePort with "externalIPs" option is doing exactly the same. I see only one tiny difference is that the IP should be owned by one of the cluster machin. So where is the profit of using a loadBalancer? let me answer that more precisely.
There are the advantages & disadvantages of ExternalIP:
The advantages of using ExternalIP is:
You have full control of the IP that you use. You can use IP that belongs to your ASN >instead of a cloud provider’s ASN.
The disadvantages of using ExternalIP is:
The simple setup that we will go thru right now is NOT highly available. That means if the node dies, the service is no longer reachable and you’ll need to manually remediate the issue.
There is some manual work that needs to be done to manage the IPs. The IPs are not dynamically provisioned for you thus it requires manual human intervention
Summarizing the pros and cons of both, we can conclude that ExternalIP is not made for a production environment, it's not highly available, if node dies the service will be no longer reachable and you will have to manually fix that.
With a LoadBalancer if node dies the service will be recreated automatically on another node. So it's dynamically provisioned and there is no need to configure it manually like with the ExternalIP.
I'm a beginner in Kubernetes and I have a situation as following: I have two differents Pods: PodA and PodB. Firstly, I want to expose PodA to the outside world, so I create a Service (type NodePort or LoadBalancer) for PodA, which is not difficult to understand for me.
Then I want PodA communicate to PodB, and after several hours googling, I found the answer is that I also need to create a Service (type ClusterIP if I want to keep PodB only visible inside the cluster) for PodB, and if I do so, I can let PodA and PodB comminucate to each other. But the problem is I also found this article. According to this webpage, they say that the communication between pods on the same node can be done via cbr0, a Network Bridge, or the communication between pods on different nodes can be done via a route table of the cluster, and they don't mention anything to the Service object (which means we don't need Service object ???).
In fact, I also read the documents of K8s and I found in the Cluster Networking
Cluster Networking
...
2. Pod-to-Pod communications: this is the primary focus of this document.
...
where they also focus on to the Pod-to-Pod communications, but there is no stuff relevant to the Service object.
So, I'm really confusing right now and my question is: Could you please explain to me the connection between these stuff in the article and the Service object? The Service object is a high-level abstract of the cbr0 and route table? And in the end, how can the Pods can communicate to each other?
If I misunderstand something, please, point it out for me, I really appreciate that.
Thank you guys !!!
Motivation behind using a service in a Kubernetes cluster.
Kubernetes Pods are mortal. They are born and when they die, they are not resurrected. If you use a Deployment to run your app, it can create and destroy Pods dynamically.
Each Pod gets its own IP address, however in a Deployment, the set of Pods running in one moment in time could be different from the set of Pods running that application a moment later.
This leads to a problem: if some set of Pods (call them “backends”) provides functionality to other Pods (call them “frontends”) inside your cluster, how do the frontends find out and keep track of which IP address to connect to, so that the frontend can use the backend part of the workload?
That being said, a service is handy when your deployments (podA and podB) are dynamically managed.
Your PodA can always communicate with PodB if it knows the address or the DNS name of PodB. In a cluster environment, there may be multiple replicas of PodB, or an instance of PodB may die and be replaced by another instance with a different address and different name. A Service is an abstraction to deal with this situation. If you use a Service to expose your PodB, then all pods in the cluster can talk to an instance of PodB using that service, which has a fixed name and fixed address no matter how many instances of PodB exists and what their addresses are.
First, I read it as you are dealing with two applications, e.g. ApplicationA and ApplicationB. Don't use the Pod abstraction when you reason about your architecture. On Kubernetes, you are dealing with a distributed system, and it is designed so that you should have multiple instances of your Application, e.g. for High Availability. Each instance of your application is a Pod.
Deploy your applications ApplicationA and ApplicationB as a Deployment resource. Then it is easy do do rolling upgrades without downtime, and Kubernetes will restart any instance of your application if it crash.
For every Deployment or for you, application, create one Service resource, (e.g. ServiceA and ServiceB). When you communicate from ApplicationA to another application, use the Service, e.g. ServiceB. The service will load balance your requests to the instances of the other application, and you can upgrade your Deployment without downtime.
1.Cluster networking : As the name suggests, all the pods deployed in the cluster will be connected by implementing any kubernetes network model like DANM, flannel
Check this link to see how to create a cluster network.
Creating cluster network
With the CNI installed (by implementing cluster network), every pod will get an IP.
2.Service objects created with type ClusterIP, points to the this IPs (via endpoint) created internally to communicate.
Answering your question, Yes, The Service object is a high-level abstract of the cbr0 and route table.
You can use service object to communicate between pods.
You can also implement service mesh like envoy / Istio if the network is complex.
I have setup an on-premises Kubernetes cluster, and I want to be ensure that my services that are not in Kubernetes, but exist on a separate class B are able to consume those services that have migrated to Kubernetes. There's a number of ways of doing this by all accounts and I'm looking for the simplest one.
Ingress + controller seems to be the one favoured - and it's interesting because of the virtual hosts and HAProxy implementation. But where I'm getting confused is how to set up the Kubernetes service:
We've not a great deal of choice - ClusterIP won't be sufficient to expose it to the outside, or NodePort. LoadBalancer seems to be a simpler, cut down way of switching between network zones - and although there are OnPrem solutions (metalLB), seems to be far geared towards cloud solutions.
But if I stick with NodePort, then my entry into the network is going to be on a non-standard port number, and I would prefer it to be over standard port; particuarly if running a percentage of traffic for that service over non-kube, and the rest over kubernetes (for testing purposes, I'd like to monitor the traffic over a period of time before I bite the bullet and move 100% of traffic for the given microservice to kubernetes). In that case it would be better those services would be available across the same port (almost always 80 because they're standard REST micro-services). More than that, if I have to re-create the service for whatever reason, I'm pretty sure the port will change, and then all traffic will not be able to enter the Kubernetes cluster and that's a frightening proposition.
What are the suggested ways of handling communication between existing on-prem and Kubernetes cluster (also on prem, different IP/subnet)?
Is there anyway to get traffic coming in without changing the network parameters (class B's the respective networks are on), and not being forced to use NodePort?
NodePort service type may be good at stage or dev environments. But i recommend you to go with LoadBalancer type service (Nginx ingress controller is one). The advantage for this over other service types are
You can use standard port (Rather random Nodeport generated by your kubernetes).
Your service is load balanced. (Load balancing will be taken care by ingress controller).
Fixed port (it will not change unless you modify something in ingress object).