request has no response when visit nodePort service - kubernetes

I deployment a kubernetes cluster with 5 nodes: matser worker1 worker2 worker3 worker4.
And I create a deployment with 1 replica, it was arranged on worker4, expose port 7777
create a service:
apiVersion: v1
kind: Service
metadata:
name: service-test
spec:
type: NodePort
selector:
app: app
ports:
- name: http
protocol: TCP
port: 80
targetPort: 7777
nodePort: 31000
After create service, I send a request to worker4:31000/test ,it responses immediately.
But when I request other nodes on 31000, such as master:31100/test , worker1:31100/test. It has no response, and sometime it will response, but it cost such a long time.
when I use lsof to show port usage, it different
[root#worker4 ~]# lsof -i:31000
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
wrapper 5251 root 5u IPv4 33957 0t0 TCP localhost:32000->localhost:31000 (ESTABLISHED)
java 5355 root 13u IPv6 35851 0t0 TCP localhost:31000->localhost:32000 (ESTABLISHED)
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
kube-prox 9679 root 13u IPv6 3746350 0t0 TCP *:31000 (LISTEN)
so how can I visit nodePort service on other nodes.

Nodeport goes through extra network hop and uses IP table load balancing at L4 layer provided by kube proxy.So it's expected to be slow particularly if you access a pod from a a node where it's not scheduled. Also kube proxy need to be running in nodes from where you want to access a pod via nodeport service.
I would suggest to use a reverse proxy such as nginx as ingress or L7 load balancer for faster performance.

Related

HAProxy Not Working with Kubernetes NodePort for Backend (Bare Metal)

I have a host running HAProxy already. It's been up and running since before I did anything with Kubernetes. It works flawlessly as a reverse proxy and SSL terminator for various web things in various Docker containers on various other host machines.
Now I have a Kubernetes cluster up and running across some of those other machines. I've created the NodePort Service that exposes port 30080 on each worker node, as follows:
apiVersion: v1
kind: Service
metadata:
name: snginx
labels:
app: nginx
spec:
type: NodePort
externalTrafficPolicy: Local #Cluster or Local
selector:
app: nginx
ports:
- protocol: TCP
port: 8080
targetPort: 80
nodePort: 30080
From the machine running HAProxy (which is not part of the cluster), I can curl the NodePort successfully ( curl 10.0.0.20:30080 ), and I get "Welcome to nginx!..." However, if I set that NodePort as a backend in HAProxy, I get a 503 "No server is available", and HAProxy traffic log says:
localhost haproxy[18489]: [redactedIP]:49399 [30/Aug/2021:19:24:00.764] http-in~ load/load 66/0/-1/-1/67 503 212 - - SC-- 1/1/0/0/3 0/0 "GET / HTTP/1.1"
The haproxy admin log says:
Aug 30 20:07:13 localhost haproxy[18839]: Server load/load is DOWN, reason: Layer4 connection problem, info: "General socket error (Permission denied)"
However, I've disabled the firewall with
sudo systemctl disable --now firewalld
and verified the status is not running. Also, SELinux was disabled when I installed the cluster. Also, I can ping 10.0.0.20 just fine.
"load" is the hostname I'm using for testing load balancing (i.e. load.mydomain.com).
Also, if I use PAT on my physical router to route directly to that NodePort, from outside the building, it works as expected.
What gives? What's the difference between the proxied request and curl?
Thank you.
SELinux is the difference. That is, SELinux on the HAProxy host (not a cluster node):
"SELinux only allows the web server to make outbound connections to a limited set of ports"
That is, you can't make an outbound http request to any port in the NodePort range (30000-32768) without opening that port on the "client", which is the HAProxy server in this case.
sudo semanage port --add --type http_port_t --proto tcp 30080

How are the various Istio Ports used?

Question
I am trying to learn Istio and I am setting up my Istio Ingress-Gateway. When I set that up, there are the following port options (as indicated here):
Port
NodePort
TargetPort
NodePort makes sense to me. That is the port that the Ingress-Gateway will listen to on each worker node in the Kubernetes cluster. Requests that hit there are going to route into the Kubernetes cluster using the Ingress Gateway CRDs.
In the examples, Port is usually set to the common port for its matching traffic (80 for http, and 443 for https, etc). I don't understand what Istio needs this port for, as I don't see any traffic using anything but the NodePort.
TargetPort is a mystery to me. I have seen some documentation on it for normal Istio Gateways (that says it is only applicable when using ServiceEntries), but nothing that makes sense for an Ingress-Gateway.
My question is this, in relation to an Ingress-Gateway (not a normal Gateway) what is a TargetPort?
More Details
In the end, I am trying to debug why my ingress traffic is getting a "connection refused" response.
I setup my Istio Operator following this tutorial with this configuration:
apiVersion: install.istio.io/v1alpha1
kind: IstioOperator
metadata:
name: istio-controlplane
namespace: istio-system
spec:
components:
ingressGateways:
- enabled: true
k8s:
service:
ports:
- name: http2
port: 80
nodePort: 30980
hpaSpec:
minReplicas: 2
name: istio-ingressgateway
pilot:
enabled: true
k8s:
hpaSpec:
minReplicas: 2
profile: default
I omitted the TargetPort from my config because I found this release notes that said that Istio will pick safe defaults.
With that I tried to follow the steps found in this tutorial.
I tried the curl command indicated in that tutorial:
curl -s -I -H Host:httpbin.example.com "http://10.20.30.40:30980/status/200"
I got the response of Failed to connect to 10.20.30.40 port 30980: Connection refused
But I can ping 10.20.30.40 fine, and the command to get the NodePort returns 30980.
So I got to thinking that maybe this is an issue with the TargetPort setting that I don't understand.
A check of the istiod logs hinted that I may be on the right track. I ran:
kubectl logs -n istio-system -l app=istiod
and among the logs I found:
warn buildGatewayListeners: skipping privileged gateway port 80 for node istio-ingressgateway-dc748bc9-q44j7.istio-system as it is an unprivileged pod
warn gateway has zero listeners for node istio-ingressgateway-dc748bc9-q44j7.istio-system
So, if you got this far, then WOW! I thank you for reading it all. If you have any suggestions on what I need to set TargetPort to, or if I am missing something else, I would love to hear it.
Port, Nodeport and TargetPort are not Istio concepts, but Kubernetes ones, more specifically of Kubernetes Services, which is why there is no detailed description of that in the Istio Operator API.
The Istio Operator API exposes the options to configure the (Kubernetes) Service of the Ingress Gateway.
For a description of those concepts, see the documentation for Kubernetes Service.
See also
Difference between targetPort and port in Kubernetes Service definition
So the target port is where the containers of the Pod of the Ingress Gateway receive their traffic.
Therefore I think, that the configuration of ports and target ports is application specific and the mapping 80->8080 is more or less arbitrary, i.e. a "decision" of the application.
Additional details:
The Istio Operator describes the Ingress Gateway, which itself consists of a Kubernetes Service and a Kubernetes Deployment. Usually it is deployed in istio-system. You can inspect the Kubernetes Service of istio-ingressgateway and it will match the specification of that YAML.
Therefore the Istio Ingress Gateway is actually talking to its containers.
However, this is mostly an implementation detail of the Istio Ingress Gateway and is not related to a Service and a VirtualService which you define for your apps.
The Ingressgateway is itself a Service and receives traffic on the port you define (i.e. 80) and forwards it to 8080 on its containers. Then it processes the traffic according to the rules which are configured by Gateways and VirtualServices and sends it to the Service of the application.
I still don't really understand what TargetPort is doing, but I got the tutorial working.
I went back an uninstalled Istio (by deleting the operator configuration and then the istio namespaces). I then re-installed it but I took the part of my configuration out that specified the node port.
I then ran a kubectl get namespace istio-ingressgateway -o yaml -n istio-system. That showed me what the istio ingress gateway was using as its defaults for the port. I then went and updated my yaml for the operator to match (except for my desired custom NodePort). That worked.
In the end, the yaml looked like this:
apiVersion: install.istio.io/v1alpha1
kind: IstioOperator
metadata:
name: istio-controlplane
namespace: istio-system
spec:
components:
ingressGateways:
- enabled: true
k8s:
service:
ports:
- name: status-port
nodePort: 32562
port: 15021
protocol: TCP
targetPort: 15021
- name: http2
nodePort: 30980
port: 80
protocol: TCP
targetPort: 8080
- name: https
nodePort: 32013
port: 443
protocol: TCP
targetPort: 8443
hpaSpec:
minReplicas: 2
name: istio-ingressgateway
pilot:
enabled: true
k8s:
hpaSpec:
minReplicas: 2
profile: default
I would still like to understand what the TargetPort is doing. So if anyone can answer with that (again, in context of the Istio Ingress Gateway service (not an istio gateway)), then I will accept that answer.
Configuring the istio-gateway with a service will create a kubernetes service with the given port configuration, which (as in a different answer already mentioned) isn't an istio concept, but a kubernetes one. So we need to take a look at the underlying kubernetes mechanisms.
The service that will be created is of type LoadBalancer by default. Also your cloud provider will create an external LoadBalancer that forwards traffic coming to it on a certain port to the cluster.
$ kubectl get svc -n istio-system
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S)
istio-ingressgateway LoadBalancer 10.107.158.5 1.2.3.4 15021:32562/TCP,80:30980/TCP,443:32013/TCP
You can see the internal ip of the service as well as the external ip of the external loadbalancer and in the PORT(S) column that for example your port 80 is mapped to port 30980. Behind the scenes kube-proxy takes your config and configures a bunch of iptables chains to set up routing of traffic to the ingress-gateway pods.
If you have access to a kubernetes host you can investigate those using the iptables command. First start with the KUBE-SERVICES chain.
$ iptables -t nat -nL KUBE-SERVICES | grep ingressgateway
target prot opt source destination
KUBE-SVC-TFRZ6Y6WOLX5SOWZ tcp -- 0.0.0.0/0 10.107.158.5 /* istio-system/istio-ingressgateway:status-port cluster IP */ tcp dpt:15021
KUBE-FW-TFRZ6Y6WOLX5SOWZ tcp -- 0.0.0.0/0 1.2.3.4 /* istio-system/istio-ingressgateway:status-port loadbalancer IP */ tcp dpt:15021
KUBE-SVC-G6D3V5KS3PXPUEDS tcp -- 0.0.0.0/0 10.107.158.5 /* istio-system/istio-ingressgateway:http2 cluster IP */ tcp dpt:80
KUBE-FW-G6D3V5KS3PXPUEDS tcp -- 0.0.0.0/0 1.2.3.4 /* istio-system/istio-ingressgateway:http2 loadbalancer IP */ tcp dpt:80
KUBE-SVC-7N6LHPYFOVFT454K tcp -- 0.0.0.0/0 10.107.158.5 /* istio-system/istio-ingressgateway:https cluster IP */ tcp dpt:443
KUBE-FW-7N6LHPYFOVFT454K tcp -- 0.0.0.0/0 1.2.3.4 /* istio-system/istio-ingressgateway:https loadbalancer IP */ tcp dpt:443
You'll see that there are basically six chains, two for each port you defined: 80, 433 and 15021 (on the far right).
The KUBE-SVC-* are for cluster internal traffic, the KUBE-FW-* for cluster external traffic. If you take a closer look you can see that the destination is the (external|internal) ip and one of the ports. So the traffic arriving on the node's network interface is for example for the destination 1.2.3.4:80. You can now follow down that chain, in my case KUBE-FW-G6D3V5KS3PXPUEDS:
iptables -t nat -nL KUBE-FW-G6D3V5KS3PXPUEDS | grep KUBE-SVC
target prot opt source destination
KUBE-SVC-LBUWNFSUU3FNPZ7L all -- 0.0.0.0/0 0.0.0.0/0 /* istio-system/istio-ingressgateway:http2 loadbalancer IP */
follow that one as well
$ iptables -t nat -nL KUBE-SVC-LBUWNFSUU3FNPZ7L | grep KUBE-SEP
target prot opt source destination
KUBE-SEP-RZL3ZLWSG2M7ZJYD all -- 0.0.0.0/0 0.0.0.0/0 /* istio-system/istio-ingressgateway:http2 */ statistic mode random probability 0.50000000000
KUBE-SEP-F7W3YTTYPP5NEPJ7 all -- 0.0.0.0/0 0.0.0.0/0 /* istio-system/istio-ingressgateway:http2 */
where you see the service endpoints, which are round robin loadbalanced by 50:50, and finally (choosing one of them):
$ iptables -t nat -nL KUBE-SEP-RZL3ZLWSG2M7ZJYD | grep DNAT
target prot opt source destination
DNAT tcp -- 0.0.0.0/0 0.0.0.0/0 /* istio-system/istio-ingressgateway:http2 */ tcp to:172.17.0.4:8080
where they end up being DNATed to 172.17.0.4:8080, which is the ip of one of the istio-ingressgateway pods ip and port 8080.
If you don't have access to a host/don't run in a public cloud environment, you wont have an external loadbalancer, so you wont find any KUBE-FW-* chains (also the EXTERNAL-IP in the service will stay in <pending>). In that case you would be using <nodeip>:<nodeport> to access the cluster from externally, for which iptables chains are also created. Run iptables -t nat -nL KUBE-NODEPORTS | grep istio-ingressgateway which will also show you three KUBE-SVC-* chains, which you can follow down to the DNAT as shown above.
So the targetport (like 8080) is used to configure networking in kubernetes and also istio uses it to defines on which ports the ingressgateway pods bind to. You can kubectl describe pod <istio-ingressgateway-pod> where you'll find the defined ports (8080 and 8443) as container ports. Change them to whatever (above 1000) and they will change accordingly. Next you would apply a Gateway with spec.servers where you define those ports 8080,8443 to configure envoy (= istio-ingressgateway, the one you define with spec.selector) to listen on those ports and a VirtualService to define who to handle the received requests. See also my other answer on that topic.
Why didn't you initial config work? If you omit the targetport, istio will bind to the port you define (80). That requires istio to run as root, otherwise the ingressgateway is unable to bind to ports below 1000. You can change it by setting values.gateways.istio-ingressgateway.runAsRoot=true in the operator, also see the release note you mentioned. In that case the whole traffic flow from above would look exactly the same, except the ingressgateway pod would bind to 80,443 instead of 8080,8443 and the DNAT would <pod-ip>:(80|443) instead of <pod-ip:(8080|8443)>.
So you basically just misunderstood the release note: If you don't run the istio-ingressgateway pod as root you have to define the targetPort or alternatively omit the whole k8s.service overlay (in that case istio will choose safe ports itself).
Note that I greped for KUBE-SVC, KUBE-SEP and DNAT. There will always be a bunch of KUBE-MARK-MASQ and KUBE-MARK-DROP that not really matter for now. If you want to learn more about the topic, there are some great articles about this out there, like this one.

What is the need of target port for pods running in Kubernetes cluster [duplicate]

This question already has answers here:
What does it mean for a Service to be of type NodePort, and have both port and targetPort specified?
(2 answers)
Closed 4 years ago.
When a container runs on a machine, one has to specify the port on which it should run, so other services in the machine can access to this container via port.
But in kubernetes, each pod has their own IP address and user can mention incoming port, so other pods can communicate via IP:Port address.
So what is the need for having target port. I feel one can set any port for "target port" field and the other pods will be seamlessly able to communicate with it.
---
apiVersion: v1
kind: Service
metadata:
name: my-service
spec:
ports:
- port: 8080
targetPort: 8070
nodePort: 31222
protocol: TCP
selector:
component: my-service-app
port: is the port used by the k8s service
target port: is the port on which the pod is serving the app
nodePort: is the port on which the service is exposed outside the cluster

Istio (1.0) intra ReplicaSet routing - support traffic between pods in a Kubernetes Deployment

How does Istio support IP based routing between pods in the same Service (or ReplicaSet to be more specific)?
We would like to deploy a Tomcat application with replica > 1 within an Istio mesh. The app runs Infinispan, which is using JGroups to sort out communication and clustering. JGroups need to identify its cluster members and for that purpose there is the KUBE_PING (Kubernetes discovery protocol for JGroups). It will consult K8S API with a lookup comparable to kubectl get pods. The cluster members can be both pods in other services and pods within the same Service/Deployment.
Despite our issue being driven by rather specific needs the topic is generic. How do we enable pods to communicate with each other within a replica set?
Example: as a showcase we deploy the demo application https://github.com/jgroups-extras/jgroups-kubernetes. The relevant stuff is:
apiVersion: v1
items:
- apiVersion: extensions/v1beta1
kind: Deployment
metadata:
name: ispn-perf-test
namespace: my-non-istio-namespace
spec:
replicas: 3
< -- edited for brevity -- >
Running without Istio, the three pods will find each other and form the cluster. Deploying the same with Istio in my-istio-namespace and adding a basic Service definition:
kind: Service
apiVersion: v1
metadata:
name: ispn-perf-test-service
namespace: my-istio-namespace
spec:
selector:
run : ispn-perf-test
ports:
- protocol: TCP
port: 7800
targetPort: 7800
name: "one"
- protocol: TCP
port: 7900
targetPort: 7900
name: "two"
- protocol: TCP
port: 9000
targetPort: 9000
name: "three"
Note that output below is wide - you might need to scroll right to get the IPs
kubectl get pods -n my-istio-namespace -o wide
NAME READY STATUS RESTARTS AGE IP NODE
ispn-perf-test-558666c5c6-g9jb5 2/2 Running 0 1d 10.44.4.63 gke-main-pool-4cpu-15gb-98b104f4-v9bl
ispn-perf-test-558666c5c6-lbvqf 2/2 Running 0 1d 10.44.4.64 gke-main-pool-4cpu-15gb-98b104f4-v9bl
ispn-perf-test-558666c5c6-lhrpb 2/2 Running 0 1d 10.44.3.22 gke-main-pool-4cpu-15gb-98b104f4-x8ln
kubectl get service ispn-perf-test-service -n my-istio-namespace
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
ispn-perf-test-service ClusterIP 10.41.13.74 <none> 7800/TCP,7900/TCP,9000/TCP 1d
Guided by https://istio.io/help/ops/traffic-management/proxy-cmd/#deep-dive-into-envoy-configuration, let's peek into the resulting Envoy conf of one of the pods:
istioctl proxy-config listeners ispn-perf-test-558666c5c6-g9jb5 -n my-istio-namespace
ADDRESS PORT TYPE
10.44.4.63 7900 TCP
10.44.4.63 7800 TCP
10.44.4.63 9000 TCP
10.41.13.74 7900 TCP
10.41.13.74 9000 TCP
10.41.13.74 7800 TCP
< -- edited for brevity -- >
The Istio doc describes the listeners above as
Receives outbound non-HTTP traffic for relevant IP:PORT pair from
listener 0.0.0.0_15001
and this all makes sense. The pod ispn-perf-test-558666c5c6-g9jb5 can reach itself on 10.44.4.63 and the service via 10.41.13.74. But... what if the pod sends packets to 10.44.4.64 or 10.44.3.22? Those IPs are not present among the listeners so afaik the two "sibling" pods are non-reachable for ispn-perf-test-558666c5c6-g9jb5.
Can Istio support this today - then how?
You are right that HTTP routing only supports local access or remote access by service name or service VIP.
That said, for your particular example, above, where the service ports are named "one", "two", "three", the routing will be plain TCP as described here. Therefore, your example should work. The pod ispn-perf-test-558666c5c6-g9jb5 can reach itself on 10.44.4.63 and the other pods at 10.44.4.64 and 10.44.3.22.
If you rename the ports to "http-one", "http-two", and "http-three" then HTTP routing will kick in and the RDS config will restrict the remote calls to ones using recognized service domains.
To see the difference in the RDF config look at the output from the following command when the port is named "one", and when it is changed to "http-one".
istioctl proxy-config routes ispn-perf-test-558666c5c6-g9jb5 -n my-istio-namespace --name 7800 -o json
With the port named "one" it will return no routes, so TCP routing will apply, but in the "http-one" case, the routes will be restricted.
I don't know if there is a way to add additional remote pod IP addresses to the RDS domains in the HTTP case. I would suggest opening an Istio issue, to see if it's possible.

Prevent public IP address binding on Kubernetes single master/node set-up

I'm following the instructions here to spin up a single node master kubernetes install. And then planning to make a website hosted within it available via an nginx ingress controller hosted directly on the internet (on a physical server, not GCE, AWS or other cloud).
Set-up works as expected and I can hit the load balancer and flow through the ingress to the target echoheaders instance, get my output and everything looks great. Good stuff.
The trouble comes when I portscan the server's public internet IP and see all these open ports besides the ingress port (80).
Open TCP Port: 80 http
Open TCP Port: 4194
Open TCP Port: 6443
Open TCP Port: 8081
Open TCP Port: 10250
Open TCP Port: 10251
Open TCP Port: 10252
Open TCP Port: 10255
Open TCP Port: 38654
Open TCP Port: 38700
Open TCP Port: 39055
Open TCP Port: 39056
Open TCP Port: 44667
All of the extra ports correspond to cadvisor, skydns and the various echo headers and nginx instances, which for security reasons should not be bound to the public IP address of the server. All of these are being injected into the host's KUBE-PORTALS-HOST iptable with bindings to the server's public IP by kube-proxy.
How can I get hypercube to tell kube-proxy to only bind to docker IP (172.x) or private cluster IP (10.x) addresses?
You should be able to set the bind address on kube-proxy (http://kubernetes.io/docs/admin/kube-proxy/):
--bind-address=0.0.0.0: The IP address for the proxy server to serve on (set to 0.0.0.0 for all interfaces)