I'm trying to wrap my head around how kubernetes (k8s) utilises ports. Having read the API documentation as well as the available docs, I'm not sure how the port mapping and port flow works.
Let's say I have three containers with an externally hosted database, my k8s cluster is three on-prem CoreOS nodes, and there is a software-defined load balancer in front of all three nodes to forward traffic to all three nodes on ports 3306 and 10082.
Container A utilises incoming port 8080, needs to talk to Container B and C, but does not need external access. It is defined with Replication Controller A that has 1 replica.
Container B utilises incoming port 8081 to talk to Container A and C, but needs to access the external database on port 3306. It is defined with Replication Controller B that has 2 replicas.
Container C utilises incoming port 8082, needs to talk to Container A and B, but also needs external access on port 10082 for end users. It is defined with Replication Controller C that has 3 replicas.
I have three services to abstract the replication controllers.
Service A selects Replication Controller A and needs to forward incoming traffic on port 9080 to port 8080.
Service B selects Replication Controller B and needs to forward incoming traffic on ports 9081 and 3306 to ports 8081 and 3306.
Service C selects Replication Controller C and needs to forward incoming traffic on port 9082 to port 8082.
I have one endpoint for the external database, configured to on port 3306 with an IPv4 address.
Goals:
Services need to abstract Replication Controller ports.
Service B needs to be able to be reached from an external system on port 3306
on all nodes.
Service C needs to be able to be reached from an external system on port 10082
on all nodes.
With that:
When would I use each of the types of ports; i.e. port, targetPort, nodePort, etc.?
Thanks for the very detailed setup, but I still have some questions.
1) When you say "Container" {A,B,C} do you mean Pod? Or are A, B, C containers in the same Pod?
2) "Container B utilises incoming port 8081 to talk to Container A and C" - What do you mean that it uses an INcoming port to talk to other containers? Who opens the connection, to whom, and on what destination port?
3) "needs to access the external database on port 3306" but later "needs to be able to be reached from an external system on port 3306" - Does B access an external database or is it serving a database on 3306?
I'm confused on where traffic is coming in and where it is going out in this explanation.
In general, you should avoid thinking in terms of nodes and you should avoid thinking about pods talking to pods (or containers to containers). You have some number of Services, each of which is backed by some number of Pods. Client pods (usually) talk to Services. Services receive traffic on a port and send that traffic to the corresponding targetPort on Pods. Pods receive traffic on a containerPort.
None of that requires hostPorts or nodePorts. The last question is which of these Services need to be accessed from outside the cluster, and what is your environment capable of wrt load-balancing.
If you answer this far, then I can come back for round 2 :)
Related
Problem statement:
I cannot access services running in pods within third party containers that don't listen on a few specific ports when using istio-sidecar
Facts:
I am running on a network with firewalled connections, so only a handful of ports can be used to communicate across the nodes, I cannot change that.
I have some third party containers running within pods that listen on ports that do not belong to the handful that is allowed
Without istio, I can do an iptables REDIRECT on an initContainer and just use any port I want
With istio-sidecar, the envoy catch-all iptables rules forward the ORIGINAL_DST to envoy with the original port, so it always tries to connect to a port that nobody is listening at. I see envoy receiving it and trying to connect to the pod at the port that I faked, the one that is allowed in the network, not the one the service is listening at.
I am trying to avoid using a socat-like solution that runs another process copying from one port to another.
I can use any kind of iptables rules and/or istio resources, EnvoyFilters etc....
My istio setup is the standard sidecar setup with nothing particular to it.
We deployed 2 cassandra datacenters in 2 different openshift clusters (one for each openshift cluster).
Each cassandra datacenter has one seed pod (pod-0)
We used bitnami helm-chart ( https://github.com/bitnami/charts/tree/master/bitnami/cassandra)
Now we would like to connect both cassandra datacentres in order to synchronize the data.
How can we do it?
I suppose that we need to expose Cassandra traffic using an openshift route. But which port and service to expose/use?
I can see that we have:
a service called myrelease-cassandra, type ClusterIP, targeting 9042/TCP (cql)
a service called myrelease-cassandra-headless, Headless(no ip), targeting 7000/TCP (intra), 7001/TCP (tls),7199/TCP (jmx),9042/TCP (cql)
I tried a couple of solution but so far I didn't succeed:
for example I can see from the logs that the Cassandra rings are trying to connect over the port 9042, but the Openshift routes are accessible through the port 443:
if I set the external seed (basically the other openshift cluser route) with the port number (e.g. my-os-route:443) I see errors saying: host could not be resolved.
if I do not set the port number I see a timeout because the port is 9042 instead of 443
we configured cassandra to use ssl, and the routes in openshift are accessible thorugh an F5 loadbalancer which targets all the infra nodes of opsnhift
UPDATE 1
Basically we would like to replicate this architecture (but with 2 k8s clusters and without the operator)
https://itnext.io/managing-a-multi-site-cassandra-cluster-on-multiple-kubernetes-with-casskop-multicasskop-cf407c297701
or
https://docs.k8ssandra.io/components/k8ssandra-operator/
UPDATE 2
Basically if there would be a way to configure the intra node port as 7000 but saying to cassandra to use another port to connect to other cassandra hosts it would work.
Something like Elasticsearch where you have http.port and http.publish_port Elasticsearch configuration
As you already know, these are the three basic attributes required for nodes to form a cluster:
the same cluster_name
inter-node network connectivity for gossip
common seeds
For two DCs in different K8s clusters to be able to form a cluster, you'll need to setup the first DC with bi-directional TCP network connectivity enabled on gossip port 7000 between the two K8s clusters.
You will also need to make sure that the pods are configured to use a service for the seeds list in cassandra.yaml. This seeds service must be exposed so that it is accessible to the other K8s cluster.
Once the first DC is operational, you will need to replicate the configuration on to the second DC in the other K8s cluster such that:
the second DC has the same cluster_name
uses the same seeds service as DC1 in the seeds list of the pods
have network connectivity to gossip on port 7000
Ideally, the second DC also has a seeds service configured so they can be used by both DCs. Cheers!
Can you use hostPort instead of nodePort. I've been able to get two independent OpenShift K8s kubes connected into a single cluster, using cass-operator, but it required a bit of hijinks. Can you see this doc https://docs.google.com/document/d/1YuS0FaCKIu_Sa9XMDRSI17MqMOHqzScR6SO7XVBImz4/edit
I have an EKS Kubernetes cluster. High level the setup is:
a) There is an EC2 instance, lets call it "VM" or "Host"
b) In the VM, there is a POD running 2 containers: Side Car HAProxy Container + MyApp Container
What happens is that when external requests come, inside of HAProxy container, I can see that the source IP is the "Host" IP. As the Host has a single IP, there can be a maximum of 64K connections to HAProxy.
I'm curious to know how to workaround this problem as I want to be able to make like 256K connections per Host.
I'm not sure is you understand reason for 64k limit so try to explain it
At first that is a good answer about 64k limitations
Let's say that HAProxy (192.168.100.100) listening at port 8080 and free ports at Host (192.168.1.1) are 1,353~65,353, so you have combination of:
source 192.168.1.1:1353~65353 → destination 192.168.100.100:8080
That is 64k simultaneous connections. I don't know how often NAT table is updating, but after update unused ports will be reused. So simultaneous is important
If your only problem is limit of connections per IP, here is couple solutions:
Run multiple HAProxyes. Three containers increase limit to 64,000 X 3 = 192,000
Listen multiple ports on HAProxy (check about SO_REUSEPORT). Three ports (8080, 8081, 8082) increase max number of connections to 192,000
Host interface IP is acting like a gateway for Docker internal network so I not sure if it is possible to set couple IPs for Host or HAProxy. At least I didn't find information about it.
It turns that in Kubernetes one can configure how we want clients to access the service and the choice that we had was nodePort. When we changed it to hostPort, the source IP was seen in the haproxy container and hence the limitation that I was having was removed.
If this option would have failed, my next option was to try the recommendation in the other response which was to have haproxy listening in multiple ports. Thankfully that was not needed.
Thanks!
I've got a lift 'n shift deployment type (i.e. by no means cloud-native) and I'd like to setup sticky sessions so that the requests keep being handled by the same pod if it's available (from the client's perspective).
Client --> LB --> Ingress --> Service --> Deployment
Due to the fact that LB does SNAT, I think service.spec.sessionAffinityConfig.clientIP will work, but because all the requests would be coming with the same source IP of the loadbalancer, the workload won't be truly balanced across all the pods in the deployment.
Can you think of any way to consider source IP & port pair in the sticky session behavior?
Edit 1: The deployment runs in Oracle Cloud. We're using the Oracle Cloud Loadbalancer service in plain TCP mode (i.e. OSI Layer4).
What the question describes is actually a default traffic management behavior in K8s. The packets within each TCP session target the same pod. The TCP session is initiated from the certain source IP (in our case the LB) and source port (which is different for each session), and this session remains "sticky" for its whole duration.
I am experimenting with a service discovery scheme on Kubernetes. I have 20+ GRPC services that can be grouped and deployed as applications on Kubernetes. Each application serves several of these services with a common GRPC server. There is a service to publish this GRPC port, and I have labels on those services that identify which GRPC servers are running there.
For instance, I have APP1 application serving GRPC services a,b,c. There is a service in front of APP1 connected to the port 8000, with labels a,b,c. So when a component in the cluster needs to connect to service, say, "b", it looks up services that have the label "b", and connects to port 8000 of one of those. This way, I can group the GRPC services in different ways, deploy them, and they all find each other.
I started thinking about an alternative approach. Instead of having one service with labels for each app, I want to have multiple services (one for each GRPC service) for the same app:port with different names. So in this new scheme APP1 would have three services, a, b, and c, all connected to the same app:port. The clients would simply look up the name "b" to find the GRPC server "b".
The question is: do you see any potential problems with having multiple services with different names that are connected to the same port of the same app, exposing the same port? That is, addresses a:8000, b:8000, c:8000 all pointing to APP1:8000.
To be honest I don't see any problem as long as your application identifies internally that the client is trying to talk to either a:8000, b:8000, or c:8000. Essentially, you will find to just a single port 8000 in this case in the container. This would be analogous to different HTTP endpoints per service some like https://myendpoint:8000/a, https://myendpoint:8000/b, and https://myendpoint/c.
Note that 8000 would be the port in the container but Kubernetes will use a random port on the node to forward the traffic to 8000 in the container.