I've been looking at setting up an Ingress controller for a bare-metal Kubernetes cluster. I started looking at Ingress controllers, but these seem to only work well for HTTP services that are reachable via port 80 or 443. If you need to expose a TCP or UDP service on an arbitrary port, it seems possible to do with the Nginx or HAProxy Ingress controllers, but your cluster ends up sharing a single port range. Please let me know if I've misunderstood this.
If you need to expose and load balance TCP or UDP services on arbitrary ports, how would you do it? I was thinking of using ClientIP so that services get their own VIP and can use any ports they want, but the question then becomes, how do you route traffic to those VIPs and give them friendly DNS names? Is there a solution for this already, or do you have to build one yourself? Using NodePort or any solution that means namespaces have to share a single port range isn't really scalable or desirable. Especially if Bob in namespace 1 absolutely needs his service to be reachable on port 8000, but Linda in namespace 2 is already using that port.
Any clarification, potential solutions, or help in general will be much appreciated.
The github issue is an interesting read, and there are some clever workarounds like starting with HTTPS then using ALPN to switch to a custom protocol: https://github.com/kubernetes/kubernetes/issues/23291 but of course then your clients need to know how to do that.
But if the protocols for these TCP and UDP services using the same port are different and don't have a way to interoperate, then the ingress controller needs to be able to allocate the equivalent of a distinct routable IP address- either with the cloud provider, or with the proprietary infrastructure, however that is handled, per exposed service.
I have not looked closely though my sense is that the packaged ingress controllers from nginx and haproxy are not going to have that automation. It would have to be built in coordination with the available infrastructure automation.
Related
We want to run a Wowza streaming engine among other containers in our Kubernetes cluster on Azure Kubernetes Service (AKS). Wowza uses various ports, some with TCP, some with UDP protocol.
We need to expose these ports to the outside world. We can't seem to find a way to set up a load balancer that can forward both TCP and UDP ports.
A LoadBalancer service does not support mixed protocols until an upcoming version of K8s, and it will be even longer until this version is available in AKS: link
We have tried using nginx-ingress, but it has the same limitation due to the underlying K8s limitation: see comment from author here
It would seem like citrix-ingress allows this according to its documentation, but we have a lot of problems making it work at all...
Is there any way to do this that we may have missed? Want to make sure we are not missing something obvious.
This is a community wiki answer posted for better visibility. Feel free to edit it when the final solution would be available (k8s v1.20 on AKS).
There is an open enhancement: Support of mixed protocols in Services with type=LoadBalancer #1435 which when implemented will enable the creation of a LoadBalancer Service that has different port definitions with different protocols. The stable release is planned for k8s v1.20.
Unfortunately, it is not possible to use both TCP and UDP ports with LoadBalancer Service currently. It would be however when the above solution is in place.
You can run your pods with hostNetwork mode and then manually set up a load balancer on top of that or similar. This is not recommended both for security and automation reasons.
I have setup an on-premises Kubernetes cluster, and I want to be ensure that my services that are not in Kubernetes, but exist on a separate class B are able to consume those services that have migrated to Kubernetes. There's a number of ways of doing this by all accounts and I'm looking for the simplest one.
Ingress + controller seems to be the one favoured - and it's interesting because of the virtual hosts and HAProxy implementation. But where I'm getting confused is how to set up the Kubernetes service:
We've not a great deal of choice - ClusterIP won't be sufficient to expose it to the outside, or NodePort. LoadBalancer seems to be a simpler, cut down way of switching between network zones - and although there are OnPrem solutions (metalLB), seems to be far geared towards cloud solutions.
But if I stick with NodePort, then my entry into the network is going to be on a non-standard port number, and I would prefer it to be over standard port; particuarly if running a percentage of traffic for that service over non-kube, and the rest over kubernetes (for testing purposes, I'd like to monitor the traffic over a period of time before I bite the bullet and move 100% of traffic for the given microservice to kubernetes). In that case it would be better those services would be available across the same port (almost always 80 because they're standard REST micro-services). More than that, if I have to re-create the service for whatever reason, I'm pretty sure the port will change, and then all traffic will not be able to enter the Kubernetes cluster and that's a frightening proposition.
What are the suggested ways of handling communication between existing on-prem and Kubernetes cluster (also on prem, different IP/subnet)?
Is there anyway to get traffic coming in without changing the network parameters (class B's the respective networks are on), and not being forced to use NodePort?
NodePort service type may be good at stage or dev environments. But i recommend you to go with LoadBalancer type service (Nginx ingress controller is one). The advantage for this over other service types are
You can use standard port (Rather random Nodeport generated by your kubernetes).
Your service is load balanced. (Load balancing will be taken care by ingress controller).
Fixed port (it will not change unless you modify something in ingress object).
If i use node port in yml file it give a port more than 30000
but when my user want to use it they do not want to remember that port and want to use 80. my kubernetes cluster is on baremetal.
How can i solve that?
Kubernetes doesn't allow you to expose low ports via the Node Port service type by design. The idea is that there is a significant chance of a port conflict if users are allowed to set low port numbers for their Node Port services.
If you really want to use port 80, you're going to have to either use a Load Balancer service type, or route your traffic through an Ingress. If you were on a cloud service, then either option would be fairly straight forward. However, since you're on bare metal, both options are going to be very involved. You're going to have to configure the load balancer or ingress functionality yourself in order to use either option, and it's going to be rough, sorry.
If you want to go forward with this, you'll have to read through a bunch of documentation to figure out what you want to implement and how to implement it.
https://www.weave.works/blog/kubernetes-faq-how-can-i-route-traffic-for-kubernetes-on-bare-metal
According to api-server docs you can use --service-node-port-range parameter for api-server or specify it to kubeadm configuration when bootstrapping your cluster see github issue
From a quick read of the Kubernetes docs, I noticed that the kube-proxy behaves as a Level-4 proxy, and perhaps works well for TCP/IP traffic (s.a. typically HTTP traffic).
However, there are other protocols like SIP (that could be over TCP or UDP), RTP (that is over UDP), and core telecom network signaling protocols like DIAMETER (over TCP or SCTP) or likewise M3UA (over SCTP). Is there a way to handle such traffic in application running in a Kubernetes minion ?
In my reading, I have come across the notion of Ingress API of Kuberntes, but I understood that it is a way to extend the capabilities of the proxy. Is that correct ?
Also, it is true that currently there is no known implementation (open-source or closed-source) of Ingress API, that can allow a Kubernetes cluster to handle the above listed type of traffic ?
Finally, other than usage of the Ingress API, is there no way to deal with the above listed traffic, even if it has performance limitations ?
Also, it is true that currently there is no known implementation (open-source or closed-source) of Ingress API, that can allow a Kubernetes cluster to handle the above listed type of traffic ?
Probably, and this IBM study on IBM Voice Gateway "Setting up high availability"
(here with SIPs (Session Initiation Protocol), like OpenSIPS)
Kubernetes deployments
In Kubernetes terminology, a single voice gateway instance equates to a single pod, which contains both a SIP Orchestrator container and a Media Relay container.
The voice gateway pods are installed into a Kubernetes cluster that is fronted by an external SIP load balancer.
Through Kubernetes, a voice gateway pod can be scheduled to run on a cluster of VMs. The framework also monitors pods and can be configured to automatically restart a voice gateway pod if a failure is detected.
Note: Because auto-scaling and auto-discovery of new pods by a SIP load balancer in Kubernetes are not currently supported, an external SIP.
And, to illustrate Kubernetes limitations:
Running IBM Voice Gateway in a Kubernetes environment requires special considerations beyond the deployment of a typical HTTP-based application because of the protocols that the voice gateway uses.
The voice gateway relies on the SIP protocol for call signaling and the RTP protocol for media, which both require affinity to a specific voice gateway instance. To avoid breaking session affinity, the Kubernetes ingress router must be bypassed for these protocols.
To work around the limitations of the ingress router, the voice gateway containers must be configured in host network mode.
In host network mode, when a port is opened in either of the voice gateway containers, those identical ports are also opened and mapped on the base virtual machine or node.
This configuration also eliminates the need to define media port ranges in the kubectl configuration file, which is not currently supported by Kubernetes. Deploying only one pod per node in host network mode ensures that the SIP and media ports are opened on the host VM and are visible to the SIP load balancer.
That network configuration put in place for Kubernetes is best illustrated in this answer, which describes the elements involved in pod/node-communication:
It is possible to handle TCP and UDP traffic from clients to your service, but it slightly depends where you run Kubernetes.
Solutions
A solution which working everywhere
It is possible to use Ingress for both TCP and UDP protocols, not only with HTTP. Some of the Ingress implementations has a support of proxying that types of traffic.
Here is an example of that kind of configuration for Nginx Ingress controller for TCP:
apiVersion: v1
kind: ConfigMap
metadata:
name: tcp-configmap-example
data:
9000: "default/example-go:8080" here is a "$namespace/$service_name:$port"
And UDP:
apiVersion: v1
kind: ConfigMap
metadata:
name: udp-configmap-example
data:
53: "kube-system/kube-dns:53" # here is a "$namespace/$service_name:$port"
So, actually, you can run your application which needs plain UDP and TCP connections with some limitations (you need somehow manage a load balancing if you have more than one pod etc).
But if you now have an application which can do it now, without Kubernetes - I don't think that you will have any problems with that after migration to Kubernetes.
A Small example of a traffic flow
For SIP UDP traffic, for an example, you can prepare configuration like this:
Client -> Nginx Ingress (UDP) -> OpenSIPS Load balancer (UDP) -> Sip Servers (UDP).
So, the client will send packets to Ingress, it will forward it to OpenSIPS, which will manage a state of your SIP cluster and send clients packets to a proper SIP server.
A solution only for Clouds
Also, if you will run in on Cloud, you can use ServiceType LoadBalancer for your Service and get TCP and UDP traffic to your application directly thru External Load Balancer provided by a cloud platform.
About SCTP
What about SCTP, unfortunately, no, that is not supported yet, but you can track a progress here.
With regard to SCTP support in k8s: it has been merged recently into k8s as alpha feature. SCTP is supported as a new protocol type in Service, NetworkPolicy and Pod definitions. See the PR here: https://github.com/kubernetes/kubernetes/pull/64973
Some restrictions exist:
the handling of multihomed SCTP associations was not in the scope of the PR. The support of multihomed SCTP associations for the cases when NAT is used is a much broader topic which affects also the current SCTP kernel modules that handle NAT for the protocol. See an example here: https://datatracker.ietf.org/doc/html/draft-ietf-tsvwg-natsupp-12
From k8s perspective one would also need a CNI plugin that supports the assignment of multiple IP addresses (on multiple interfaces preferably) to pods, so the pod can establish multihomed SCTP association. Also one would need an enhanced Service/Endpoint/DNS controller to handle those multiple IP addresses on the right way.
the support of SCTP as protocol for type=LoadBalancer Services is up to the load balancer implementation, which is not a k8s issue
in order to use SCTP in NetworkPolicy one needs a CNI plugin that supports SCTP in NetworkPolicies
Is it possible to expose pods application of different ports on single IP on different port for example that
microservices-cart LoadBalancer 10.15.251.89 35.195.135.146 80:30721/TCP
microservices-comments LoadBalancer 10.15.249.230 35.187.190.124 80:32082/TCP
microservices-profile LoadBalancer 10.15.244.188 35.195.255.183 80:31032/TCP
would look like
microservices-cart LoadBalancer 10.15.251.89 35.195.135.146 80:30721/TCP
microservices-comments LoadBalancer 10.15.249.230 35.195.135.146 81:32082/TCP
microservices-profile LoadBalancer 10.15.244.188 35.195.135.146 82:31032/TCP
Reusing the same external IP is usually accomplished by using ingress resources.
See https://kubernetes.io/docs/concepts/services-networking/ingress/
But you'll have to route with paths instead of ports.
One possible solution is to combine NodePort and a reverse proxy. NodePort expose pods on different ports on all nodes. The reverse proxy serves as the entrance and redirects traffic to nodes.
One way or another you'll have to consolidate onto the same pod.
You can create a deployment that proxies each of the ports to the appropriate service. There are plenty of ways to create a TCP proxy - via nginx, node via package, there's a Go package maintained by Google; whatever you're most comfortable with.
First of all, if you're building a microservices app, you need an api gateway. It can have an external IP address and communicate with other pods using internal services. One possible way is using nginx. You can watch a guide about api gateways here.