I would like to build a web application that processes video from users' webcams. It looks like WebRTC is ideal for this project. But, I'm having a hard time creating a peer connection between the user's machine and a pod in my Kubernetes cluster. How would you connect these two peers?
This question on Server Fault discusses the issue I'm running into: WEBRTC MCU/SFU inside kubernetes - Port Ranges. WebRTC wants a bunch of ports open so users can create peer connections with the server but Kubernetes has ports closed by default. Here's a rephrasing of my question: How to create
RTCPeerConnections connecting multiple users to an application hosted in a Kubernetes cluster? How should network ports be setup?
The closest I've come to finding a solution is Orchestrating GPU-accelerated streaming apps using WebRTC, their code is available on GitHub. I don't fully understand their approach, I believe it depends on Istio.
The document you link to is helpful, Orchestrating GPU-accelerated streaming apps using WebRTC
What they do to allow for RTCPeerConnection is:
Use two separate Node pools (group of Nodes):
Default Node pool - for most components, using Ingress and load balancer
TURN Node pool - for STUN/TURN service
STUN/TURN service
The STUN/TURN service is network bound and deployed to dedicated nodes. It is deployed with one instance on each node in the node pool. This can be done on Kubernetes using a DaemonSet. In addition this service should use host networking, e.g. all nodes has its ports accessible from Internet. Activate host networking for the PodTemplate in your DaemonSet:
hostNetwork: true
They use coturn as STUN/TURN server.
The STUN/TURN service is run as a DaemonSet on each node of the TURN node pool. The coTURN process needs to allocate a fixed block of ports bound to the host IP address in order to properly serve relay traffic. A single coTURN instance can serve thousands of concurrent STUN and TURN requests based on the machine configuration.
Network
This part of their network diagram shows that some services are served over https with an ingress gateway, whereas the STUN/TURN service is through a different connection using dtls/rtp to the nodes exposed via host network.
Related
Let's say I have a web application Backend that I want to deploy with the help of Kubernetes, how exactly does scaling work in this case.
I understand scaling in Kubernetes as: We have one a master node that orchestrates multiple worker nodes where each of the worker nodes runs 0-n different containers with the same image. My question is, if this is correct, how does Kubernetes deal with the fact that the same application use the same Port within one worker node? Does the request reach the master node which then handles this problem internally?
Does the request reach the master node which then handles this problem internally?
No, the master nodes does not handle traffic for your apps. Typically traffic meant for your apps arrive to a load balancer or gateway, e.g. Google Cloud Load Balancer or AWS Elastic Load Balancer, then the load balancer forwards the request to a replica of a matching service - this is managed by the Kubernetes Ingress resource in your cluster.
The master nodes - the control plane - is only used for management, e.g. when you deploy a new image or service.
how does Kubernetes deal with the fact that the same application use the same Port within one worker node?
Kubernetes uses a container runtime for your containers. You can try this on your own machine, e.g. when you use docker, you can create multiple containers (instances) of your app, all listening on e.g. port 8080. This is a key feature of containers - the provide network isolation.
On Kubernetes, all containers are tied together with a custom container networking. How this works, depends on what Container Networking Interface-plugin you use in your cluster. Each Pod in your cluster will get its own IP address. All your containers can listen to the same port, if you want - this is an abstraction.
I'm having an issue with load balancing persistent tcp connections to my kubernetes replicas.
I have Unity3D clients outside of the kubernetes cluster.
My cluster is a baremetal cluster with metallb installed composed out of 3 nodes: 1 master and 2 workers.
As I have read there are two approaches:
1) client connects to all replicas and each time it needs to send a request it will do so on a random connection out of those that it has previously established. Periodically, it refreshes connections (in case autoscale happened or some of the persistent connections died).
The problem here is, I'm not sure how to access all replicas externally, headless services cannot be exposed externally.
2) service mesh ? I have vaguely read/understood that they might establish persistent tcp on your behalf. So something like this :
unity3d client <----persistent connection ---> controller <---persistent connection----> replicas
However, I'm not sure how to accomplish this and I'm not sure what will happen if the controller itself fails, will all the clients get their connections dropped ? As I see it, it will come down to the same issue as the one from 1), which is allowing a client to connect to multiple different replicas at the same time with a persistent TCP connection.
Part of question comes as a complement to this : https://learnk8s.io/kubernetes-long-lived-connections
In order to enable external traffic to your cluster you need an Ingress Gateway. Your ingress gateway could be the standard nginx Ingress, a gateway provided by a mesh like the Istio Gateway or a more specialized edge gateway like ambassador, traefik, kong, gloo, etc.
There are at least two ways you can perform load balancing in K8s:
Using a Service resource which is just a set of iptables rules managed by the kube-proxy process. This is L4 load balancing only. No L7 application protocols like HTTP2 or gRPC are supported. Depending on your case, this type of LB might not be ideal for long lived connections as connections will rarely be closed.
Using the L7 load balancing offered by any of the ingress controllers which will skip the iptables routing (using a headless Service) and allow for more advanced load balancing algorithms.
In order to benefit from the latter case you still need to ensure that connections are eventually terminated which is often done from the client to the proxy (while reusing connections from the proxy to the upstream). I'm not familiar with Unity3D connections but if terminating them is not an option you won't be able to do much load balancing after all.
When the controller fails, connections will be dropped and your client could either graciously re-attempt the connection or panic. It depends on how you code it.
I have a Kubernetes cluster (Kubernetes 1.13, Weave Net CNI) that has no direct access to an internal company network. There is an authentication-free SOCKS5 proxy that can (only) be reached from the cluster, and which resolves and connects to resources in the internal network:
Consider some 3rd party Docker Images used on Pods that don't have any explicit proxy support, and just want a resolvable DNS name and target port to connect to a TCP-based service (which might be HTTP(S), but doesn't have to be).
What kind of setup would you propose to bind the Pods and Company Network Services together?
The only two things comes to my mind are:
1) Run the Socks5 docker image as a sidecar: https://hub.docker.com/r/serjs/go-socks5-proxy/
2) Use Transparent Proxy Redirector on the nodes - https://github.com/darkk/redsocks
I have deployed two POD-s with hostnetwork set to true. When the POD-s are deployed on same OpenShfit node then everything works fine since they can discover each other using node IP.
When the POD-s are deployed on different OpenShift nodes then they cant discover each other, I get no route to host if I want to point one POD to another using node IP. How to fix this?
The uswitch/kiam (https://github.com/uswitch/kiam) service is a good example of a use case.
it has an agent process that runs on the hostnetwork of all worker nodes because it modifies a firewall rule to intercept API requests (from containers running on the host) to the AWS api.
it also has a server process that runs on the hostnetwork to access the AWS api since the AWS api is on a subnet that is only available to the host network.
finally... the agent talks to the server using GRPC which connects directly to one of the IP addresses that are returned when looking up the kiam-server.
so you have pods of the agent deployment running on the hostnetwork of node A trying to connect to kiam server running on the hostnetwork of node B.... which just does not work.
furthermore, this is a private service... it should not be available from outside the network.
If you want the two containers to be share the same physical machine and take advantage of loopback for quick communications, then you would be better off defining them together as a single Pod with two containers.
If the two containers are meant to float over a larger cluster and be more loosely coupled, then I'd recommend taking advantage of the Service construct within Kubernetes (under OpenShift) and using that for the appropriate discovery.
Services are documented at https://kubernetes.io/docs/concepts/services-networking/service/, and along with an internal DNS service (if implemented - common in Kubernetes 1.4 and later) they provide a means to let Kubernetes manage where things are, updating an internal DNS entry in the form of <servicename>.<namespace>.svc.cluster.local. So for example, if you set up a Pod with a service named "backend" in the default namespace, the other Pod could reference it as backend.default.svc.cluster.local. The Kubernetes documentation on the DNS portion of this is available at https://kubernetes.io/docs/concepts/services-networking/dns-pod-service/
This also avoids the "hostnetwork=true" complication, and lets OpenShift (or specifically Kubernetes) manage the networking.
If you have to absolutely use hostnetwork, you should be creating router and then use those routers to have the communication between pods. You can create ha proxy based router in opeshift, reference here --https://docs.openshift.com/enterprise/3.0/install_config/install/deploy_router.html
In preparation for HIPAA compliance, we are transitioning our Kubernetes cluster to use secure endpoints across the fleet (between all pods). Since the cluster is composed of about 8-10 services currently using HTTP connections, it would be super useful to have this taken care of by Kubernetes.
The specific attack vector we'd like to address with this is packet sniffing between nodes (physical servers).
This question breaks down into two parts:
Does Kubernetes encrypts the traffic between pods & nodes by default?
If not, is there a way to configure it such?
Many thanks!
Actually the correct answer is "it depends". I would split the cluster into 2 separate networks.
Control Plane Network
This network is that of the physical network or the underlay network in other words.
k8s control-plane elements - kube-apiserver, kube-controller-manager, kube-scheduler, kube-proxy, kubelet - talk to each other in various ways. Except for a few endpoints (eg. metrics), it is possible to configure encryption on all endpoints.
If you're also pentesting, then kubelet authn/authz should be switched on too. Otherwise, the encryption doesn't prevent unauthorized access to the kubelet. This endpoint (at port 10250) can be hijacked with ease.
Cluster Network
The cluster network is the one used by the Pods, which is also referred to as the overlay network. Encryption is left to the 3rd-party overlay plugin to implement, failing which, the app has to implement.
The Weave overlay supports encryption. The service mesh linkerd that #lukas-eichler suggested can also achieve this, but on a different networking layer.
The replies here seem to be outdated. As of 2021-04-28 at least the following components seem to be able to provide an encrypted networking layer to Kubernetes:
Istio
Weave
linkerd
cilium
Calico (via Wireguard)
(the list above was gained via consultation of the respective projects home pages)
Does Kubernetes encrypts the traffic between pods & nodes by default?
Kubernetes does not encrypt any traffic.
There are servicemeshes like linkerd that allow you to easily introduce https communication between your http service.
You would run a instance of the service mesh on each node and all services would talk to the service mesh. The communication inside the service mesh would be encrypted.
Example:
your service -http-> localhost to servicemesh node - https-> remoteNode -http-> localhost to remote service.
When you run the service mesh node in the same pod as your service the localhost communication would run on a private virtual network device that no other pod can access.
No, kubernetes does not encrypt traffic by default
I haven't personally tried it, but the description on the Calico software defined network seems oriented toward what you are describing, with the additional benefit of already being kubernetes friendly
I thought that Calico did native encryption, but based on this GitHub issue it seems they recommend using a solution like IPSEC to encrypt just like you would a traditional host