problem to implement route forwarding method - minikube

I have developed one application using minikube and one pod. I am using IPVS mode here. The application works when I have used Masq forwarding methods. But it does not work for route forwarding methods. Should I need to configure anything when I am using route forwarding method?
Here I have attached IPVS list for masq and routing forwarding method.
# ipvsadm -Ln
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
-> RemoteAddress:Port Forward Weight ActiveConn InActConn
TCP 10.96.0.1:443 wlc
-> 147.214.68.51:8443 Masq 1 2 0
TCP 10.96.0.10:53 wlc
-> 172.17.0.2:53 Masq 1 0 0
-> 172.17.0.3:53 Masq 1 0 0
TCP 10.108.116.175:5000 wlc
-> 172.17.0.4:5000 Masq 1 0 0
TCP 127.0.0.1:32673 wlc
-> 172.17.0.4:5000 Masq 1 0 0
TCP 147.214.68.51:32673 wlc
-> 172.17.0.4:5000 Masq 1 0 0
TCP 172.17.0.1:32673 wlc
-> 172.17.0.4:5000 Masq 1 0 0
TCP 10.96.0.10:9153 wlc
-> 172.17.0.2:9153 Masq 1 0 0
-> 172.17.0.3:9153 Masq 1 0 0
UDP 10.96.0.10:53 wlc
-> 172.17.0.2:53 Masq 1 0 0
-> 172.17.0.3:53 Masq 1 0 0
# ipvsadm -Ln
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
-> RemoteAddress:Port Forward Weight ActiveConn InActConn
TCP 10.96.0.1:443 wlc
-> 147.214.68.51:8443 Route 1 0 0
TCP 10.96.0.10:53 wlc
-> 172.17.0.2:53 Route 1 0 0
-> 172.17.0.3:53 Route 1 0 0
TCP 10.108.116.175:5000 wlc
-> 172.17.0.4:5000 Route 1 0 0
TCP 127.0.0.1:32673 wlc
-> 172.17.0.4:5000 Route 1 0 0
TCP 147.214.68.51:32673 wlc
-> 172.17.0.4:5000 Route 1 3 0
TCP 172.17.0.1:32673 wlc
-> 172.17.0.4:5000 Route 1 0 0
TCP 10.96.0.10:9153 wlc
-> 172.17.0.2:9153 Route 1 0 0
-> 172.17.0.3:9153 Route 1 0 0
UDP 10.96.0.10:53 wlc
-> 172.17.0.2:53 Route 1 0 0
-> 172.17.0.3:53 Route 1 0 0

I got three suggestions for you:
Try to debug and see if you meet all the prerequisites and things from the check list.
Also, here you can find a guide describing LVS-DR Cluster using the route forwarding method (Chapter 13).
Consider using kube-router.
Providing high performance is at the core of all the design choices
made in Kube-router. Be it the use of IPVS/LVS for service proxy or
the use of direct routing across the nodes for pod networking etc.
A documentation with guide can be found here.
If that don't help you'll need to provide more details regarding your configuration for us to work on.
Please let me know if that helped.

Related

kubernetes: can't ping pods

I created a k8s cluster which network configuration to pod is podSubnet: 172.168.0.0/12
Then, I find that can't ping those pod's IP.
for example, the deployment of metrics
# on k8s-master01 node:
$ kubectl get po -n kube-system -o wide
metrics-server-545b8b99c6-r2ql5 1/1 Running 0 5d1h 172.171.14.193 k8s-node02 <none> <none>
# ping 172.171.14.193 -c 2
PING 172.171.14.193 (172.171.14.193) 56(84) bytes of data.
^C
--- 172.171.14.193 ping statistics ---
2 packets transmitted, 0 received, 100% packet loss, time 1016ms
# this is route table
# route -n
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
0.0.0.0 10.180.104.1 0.0.0.0 UG 0 0 0 eth0
10.180.104.0 0.0.0.0 255.255.255.0 U 0 0 0 eth0
172.17.0.0 0.0.0.0 255.255.0.0 U 0 0 0 docker0
172.161.125.0 10.180.104.110 255.255.255.192 UG 0 0 0 tunl0
172.162.195.0 10.180.104.109 255.255.255.192 UG 0 0 0 tunl0
172.169.92.64 10.180.104.108 255.255.255.192 UG 0 0 0 tunl0
172.169.244.192 0.0.0.0 255.255.255.255 UH 0 0 0 cali06e1673851f
172.169.244.192 0.0.0.0 255.255.255.192 U 0 0 0 *
172.171.14.192 10.180.104.111 255.255.255.192 UG 0 0 0 tunl0
that shows metric pod host on k8s-node02. This is k8s-node02's route table
# route -n
Kernel IP routing table of k8s-master01
Destination Gateway Genmask Flags Metric Ref Use Iface
0.0.0.0 10.180.104.1 0.0.0.0 UG 0 0 0 eth0
10.180.104.0 0.0.0.0 255.255.255.0 U 0 0 0 eth0
169.254.169.254 10.180.104.11 255.255.255.255 UGH 0 0 0 eth0
172.17.0.0 0.0.0.0 255.255.0.0 U 0 0 0 docker0
172.161.125.0 10.180.104.110 255.255.255.192 UG 0 0 0 tunl0
172.162.195.0 10.180.104.109 255.255.255.192 UG 0 0 0 tunl0
172.169.92.64 10.180.104.108 255.255.255.192 UG 0 0 0 tunl0
172.169.244.192 10.180.104.107 255.255.255.192 UG 0 0 0 tunl0
172.171.14.192 0.0.0.0 255.255.255.192 U 0 0 0 *
172.171.14.193 0.0.0.0 255.255.255.255 UH 0 0 0 cali872eed170f4
172.171.14.194 0.0.0.0 255.255.255.255 UH 0 0 0 cali7d7625dd37e
172.171.14.203 0.0.0.0 255.255.255.255 UH 0 0 0 calid4e258f95f6
172.171.14.204 0.0.0.0 255.255.255.255 UH 0 0 0 cali5cf96eb1028
in fact, all pods can't access. I created a service based on example deployment.
# kubectl describe svc my-service
Name: my-service
Namespace: default
Labels: <none>
Annotations: <none>
Selector: app=demo-nginx
Type: ClusterIP
IP Families: <none>
IP: 10.100.75.139
IPs: 10.100.75.139
Port: http 80/TCP
TargetPort: 80/TCP
Endpoints: 172.161.125.14:80,172.161.125.15:80,172.171.14.203:80
Session Affinity: None
Events: <none>
# ping 10.100.75.139 -c 1
PING 10.100.75.139 (10.100.75.139) 56(84) bytes of data.
64 bytes from 10.100.75.139: icmp_seq=1 ttl=64 time=0.077 ms
# nc -vz 10.100.75.139 80
Ncat: Version 7.50 ( https://nmap.org/ncat )
Ncat: Connection timed out.
I suppose that the root cause is route table but not sure. Would you please help to fix this issue??
please feel free to let me know if you need more information.
Thanks a lot in advance.
BR//
In connect mode, Ncat initiates a connection (or sends UDP data) to a service that is listening somewhere. For those familiar with socket programming, connect mode is like using the connect function. In listen mode, Ncat waits for an incoming connection (or data receipt), like using the bind and listen to functions.
A connection timed out response indicates that your connection is not working, which could mean your firewall is blocking the port. Test the connection status by adding a rule that accepts connections on the required port.
There might be issues with traffic rules, try adding outbound traffic rules in security groups if you face the error again. Even after adding the traffic rules, if you face the same error then there must be some restricting rules on the node which is blocking you from accessing the port.
Refer to Testing Network services for more information.

kubernetes v1.18.8 installation issue [closed]

Closed. This question does not meet Stack Overflow guidelines. It is not currently accepting answers.
This question does not appear to be about a specific programming problem, a software algorithm, or software tools primarily used by programmers. If you believe the question would be on-topic on another Stack Exchange site, you can leave a comment to explain where the question may be able to be answered.
Closed 2 years ago.
Improve this question
I have deployed Kubernetes cluster v1.18.8 with kubeadm on production environment.Cluster setup is 3 Master and 3 Worker nodes with external Kube-api loadbalancer, etcd residing in Master nodes.Didn't see any issue during installation and all pods in kube-system are running. However when i get error when i run below command i get error:
kubectl get cs
NAME STATUS MESSAGE ERROR
controller-manager Unhealthy Get http://127.0.0.1:10252/healthz: dial tcp 127.0.0.1:10252: connect: connection refused
scheduler Unhealthy Get http://127.0.0.1:10251/healthz: dial tcp 127.0.0.1:10251: connect: connection refused
etcd-0 Healthy {"health":"true"}
While troubleshooting i found that the ports are not being listened.
sudo netstat -tlpn |grep kube
tcp 0 0 127.0.0.1:10248 0.0.0.0:* LISTEN 132584/kubelet
tcp 0 0 127.0.0.1:10249 0.0.0.0:* LISTEN 133300/kube-proxy
tcp 0 0 127.0.0.1:10257 0.0.0.0:* LISTEN 197705/kube-control
tcp 0 0 127.0.0.1:10259 0.0.0.0:* LISTEN 213741/kube-schedul
tcp6 0 0 :::10250 :::* LISTEN 132584/kubelet
tcp6 0 0 :::6443 :::* LISTEN 132941/kube-apiserv
tcp6 0 0 :::10256 :::* LISTEN 133300/kube-proxy
If i check the same thing on development enviroment kubernetes cluster(v1.17) i see no issue.
kubectl get cs
NAME STATUS MESSAGE ERROR
controller-manager Healthy ok
scheduler Healthy ok
etcd-0 Healthy {"health":"true"}
sudo netstat -tlpn |grep 102
tcp 0 0 127.0.0.1:10257 0.0.0.0:* LISTEN 2141/kube-controlle
tcp 0 0 127.0.0.1:10259 0.0.0.0:* LISTEN 2209/kube-scheduler
tcp 0 0 127.0.0.1:10248 0.0.0.0:* LISTEN 1230/kubelet
tcp 0 0 127.0.0.1:10249 0.0.0.0:* LISTEN 2668/kube-proxy
tcp6 0 0 :::10256 :::* LISTEN 2668/kube-proxy
tcp6 0 0 :::10250 :::* LISTEN 1230/kubelet
tcp6 0 0 :::10251 :::* LISTEN 2209/kube-scheduler
tcp6 0 0 :::10252 :::* LISTEN 2141/kube-controlle
On newly created prodction cluster i have deployed nginx and another application just to test how the kubernetes components behave, didn't see any error.
Is it the expected behaviour in version v1.18? Will really apprecite any help on this.
NOTE: No port is being blocked in internal communication
The command Kubectl get componentstatus is depreciated in newer version(1.19) and it already has many issues.
The main point to note here is that Kubernetes has disabled insecure serving of
these components for older versions(atleast from v1.18). Hence i couldn't see kube-controller and kube-scheduler being listned on 1051 and 1052 ports. To restore this functionality you can remove the --port=0 from their manifests files(Not recommended as this can expose their metrics to the whole internet) that you can see inside:
/etc/kubernetes/manifests/
I commented out --port=0 field from the manifest file just to check this, kubectl get componentstatus command worked.

Minikube / Docker proxy is using port 80

I am using minikube in no-driver mode ( sudo minikube start --vm-driver none ) and I can't free port 80.
With
sudo netstat -nlplute
I get:
tcp 0 0 192.168.0.14:2380 0.0.0.0:* LISTEN 0 58500 7200/etcd
tcp6 0 0 :::80 :::* LISTEN 0 62030 8681/docker-proxy
tcp6 0 0 :::8080 :::* LISTEN 0 57318 8656/docker-proxy
I tried to stop minikube, but it doesn't seem to be working when using driver=none
How should I free port 80 ?
EDIT: Full netstat ouput
➜ ~ sudo netstat -nlpute
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address Foreign Address State User Inode PID/Program name
tcp 0 0 127.0.0.53:53 0.0.0.0:* LISTEN 102 35399 1019/systemd-resolv
tcp 0 0 127.0.0.1:631 0.0.0.0:* LISTEN 0 6629864 11358/cupsd
tcp 0 0 127.0.0.1:5432 0.0.0.0:* LISTEN 128 45843 1317/postgres
tcp 0 0 127.0.0.1:6942 0.0.0.0:* LISTEN 1000 14547489 16086/java
tcp 0 0 127.0.0.1:10248 0.0.0.0:* LISTEN 0 58474 1053/kubelet
tcp 0 0 127.0.0.1:10249 0.0.0.0:* LISTEN 0 71361 10409/kube-proxy
tcp 0 0 127.0.0.1:45801 0.0.0.0:* LISTEN 0 57445 1053/kubelet
tcp 0 0 192.168.0.14:2379 0.0.0.0:* LISTEN 0 56922 7920/etcd
tcp 0 0 127.0.0.1:2379 0.0.0.0:* LISTEN 0 56921 7920/etcd
tcp 0 0 192.168.0.14:2380 0.0.0.0:* LISTEN 0 56917 7920/etcd
tcp 0 0 127.0.0.1:2381 0.0.0.0:* LISTEN 0 56084 7920/etcd
tcp 0 0 127.0.0.1:63342 0.0.0.0:* LISTEN 1000 14549242 16086/java
tcp 0 0 0.0.0.0:111 0.0.0.0:* LISTEN 0 15699 1/init
tcp 0 0 127.0.0.1:10257 0.0.0.0:* LISTEN 0 60857 7889/kube-controlle
tcp 0 0 127.0.0.1:10259 0.0.0.0:* LISTEN 0 56932 7879/kube-scheduler
tcp 0 0 127.0.0.1:5939 0.0.0.0:* LISTEN 0 48507 2205/teamviewerd
tcp6 0 0 ::1:631 :::* LISTEN 0 6629863 11358/cupsd
tcp6 0 0 :::8443 :::* LISTEN 0 55158 7853/kube-apiserver
tcp6 0 0 :::44444 :::* LISTEN 1000 16217187 7252/___go_build_gi
tcp6 0 0 :::32028 :::* LISTEN 0 74556 10409/kube-proxy
tcp6 0 0 :::10250 :::* LISTEN 0 58479 1053/kubelet
tcp6 0 0 :::30795 :::* LISTEN 0 74558 10409/kube-proxy
tcp6 0 0 :::10251 :::* LISTEN 0 56926 7879/kube-scheduler
tcp6 0 0 :::10252 :::* LISTEN 0 60851 7889/kube-controlle
tcp6 0 0 :::30285 :::* LISTEN 0 74559 10409/kube-proxy
tcp6 0 0 :::31406 :::* LISTEN 0 74557 10409/kube-proxy
tcp6 0 0 :::111 :::* LISTEN 0 15702 1/init
tcp6 0 0 :::80 :::* LISTEN 0 16269016 16536/docker-proxy
tcp6 0 0 :::8080 :::* LISTEN 0 16263128 16524/docker-proxy
tcp6 0 0 :::10256 :::* LISTEN 0 75123 10409/kube-proxy
udp 0 0 0.0.0.0:45455 0.0.0.0:* 115 40296 1082/avahi-daemon:
udp 0 0 224.0.0.251:5353 0.0.0.0:* 1000 16274723 23811/chrome --type
udp 0 0 224.0.0.251:5353 0.0.0.0:* 1000 16270144 23728/chrome
udp 0 0 224.0.0.251:5353 0.0.0.0:* 1000 16270142 23728/chrome
udp 0 0 0.0.0.0:5353 0.0.0.0:* 115 40294 1082/avahi-daemon:
udp 0 0 127.0.0.53:53 0.0.0.0:* 102 35398 1019/systemd-resolv
udp 0 0 192.168.0.14:68 0.0.0.0:* 0 12307745 1072/NetworkManager
udp 0 0 0.0.0.0:111 0.0.0.0:* 0 18653 1/init
udp 0 0 0.0.0.0:631 0.0.0.0:* 0 6628156 11360/cups-browsed
udp6 0 0 :::5353 :::* 115 40295 1082/avahi-daemon:
udp6 0 0 :::111 :::* 0 15705 1/init
udp6 0 0 :::50342 :::* 115 40297 1082/avahi-daemon:
Ive reproduced your environment (--vm-driver=none). At first I thought it might be connected with minikube built-in configuration, however clean Minikube don't use port 80 on default configuration.
$ kubectl version
Client Version: version.Info{Major:"1", Minor:"18", GitVersion:"v1.18.0", GitCommit:"9e991415386e4cf155a24b1da15becaa390438d8", GitTreeState:"clean", BuildDate:"2020-03-25T14:58:59Z", GoVersion:"go1.13.8", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"17", GitVersion:"v1.17.0", GitCommit:"70132b0f130acc0bed193d9ba59dd186f0e634cf", GitTreeState:"clean", BuildDate:"2019-12-07T21:12:17Z", GoVersion:"go1.13.4", Compiler:"gc", Platform:"linux/amd64"}
$ minikube version
minikube version: v1.6.2
$ sudo netstat -nlplute
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address Foreign Address State User Inode PID/Program name
tcp 0 0 127.0.0.1:10257 0.0.0.0:* LISTEN 0 49556 9345/kube-controlle
tcp 0 0 127.0.0.1:10259 0.0.0.0:* LISTEN 0 50223 9550/kube-scheduler
tcp 0 0 127.0.0.53:53 0.0.0.0:* LISTEN 101 15218 752/systemd-resolve
tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 0 21550 1541/sshd
tcp 0 0 127.0.0.1:44197 0.0.0.0:* LISTEN 0 51016 10029/kubelet
tcp 0 0 127.0.0.1:10248 0.0.0.0:* LISTEN 0 51043 10029/kubelet
tcp 0 0 127.0.0.1:10249 0.0.0.0:* LISTEN 0 52581 10524/kube-proxy
tcp 0 0 127.0.0.1:2379 0.0.0.0:* LISTEN 0 49728 9626/etcd
tcp 0 0 10.156.0.11:2379 0.0.0.0:* LISTEN 0 49727 9626/etcd
tcp 0 0 10.156.0.11:2380 0.0.0.0:* LISTEN 0 49723 9626/etcd
tcp 0 0 127.0.0.1:2381 0.0.0.0:* LISTEN 0 49739 9626/etcd
tcp6 0 0 :::10256 :::* LISTEN 0 52577 10524/kube-proxy
tcp6 0 0 :::22 :::* LISTEN 0 21552 1541/sshd
tcp6 0 0 :::8443 :::* LISTEN 0 49120 9419/kube-apiserver
tcp6 0 0 :::10250 :::* LISTEN 0 51050 10029/kubelet
tcp6 0 0 :::10251 :::* LISTEN 0 50217 9550/kube-scheduler
tcp6 0 0 :::10252 :::* LISTEN 0 49550 9345/kube-controlle
udp 0 0 127.0.0.53:53 0.0.0.0:* 101 15217 752/systemd-resolve
udp 0 0 10.156.0.11:68 0.0.0.0:* 100 15574 713/systemd-network
udp 0 0 127.0.0.1:323 0.0.0.0:* 0 23984 2059/chronyd
udp6 0 0 ::1:323 :::* 0 23985 2059/chronyd
Good description for what docker-proxy is used you can check this article.
When a container starts with its port forwarded to the Docker host on which it runs, in addition to the new process that runs inside the container, you may have noticed an additional process on the Docker host called docker-proxy
This docker-proxy might be something similar like docker zombie process where container was removed, however allocated port wasn't unlocked. Unfortunately it seems that this is recurrent docker issue occuring across versions and OS since 2016. As I mentioned I think currently there is no fix for this, however you can find workaround.
cd /usr/libexec/docker/
ln -s docker-proxy-current docker-proxy
service docker restart
===
$ sudo service docker stop
$ sudo service docker start
===
$ sudo service docker stop
# remove all internal docker network: rm /var/lib/docker/network/files/
$ sudo service docker start
===
$ sudo systemctl stop docker
$ sudo systemctl start docker
There are a few github threads mentioning about this issue. For more information please check this and this thread.
After checking that my port 8080 was also used by docker proxy, I did
$ docker ps
and notices that both port 80 and port 8080 are used by traefik controller:
$ kubectl get services
traefik-ingress-service ClusterIP 10.96.199.177 <none> 80/TCP,8080/TCP 25d
When I checked for traefik service, I found:
kind: Service
apiVersion: v1
metadata:
name: traefik-ingress-service
spec:
selector:
k8s-app: traefik-ingress-lb
ports:
- protocol: TCP
port: 80
name: web
- protocol: TCP
port: 8080
name: admin
So, I think this is why I get a docker-proxy. If I need it to use another port, I can change it here. My bad :(

Too many TCP ports in CLOSE WAIT condition in kafka broker

Too many TCP Connections are in CLOSE_WAIT status in a kafka broker causing DisconnectionException in kafka clients.
tcp6 27 0 172.31.10.143:9092 172.31.0.47:45138 ESTABLISHED -
tcp6 25 0 172.31.10.143:9092 172.31.46.69:41612 CLOSE_WAIT -
tcp6 25 0 172.31.10.143:9092 172.31.0.47:45010 CLOSE_WAIT -
tcp6 25 0 172.31.10.143:9092 172.31.46.69:43000 CLOSE_WAIT -
tcp6 194 0 172.31.10.143:8080 172.31.20.219:45952 CLOSE_WAIT -
tcp6 25 0 172.31.10.143:9092 172.31.20.219:48006 CLOSE_WAIT -
tcp6 1 0 172.31.10.143:9092 172.31.0.47:44582 CLOSE_WAIT -
tcp6 25 0 172.31.10.143:9092 172.31.46.69:42828 CLOSE_WAIT -
tcp6 25 0 172.31.10.143:9092 172.31.46.69:41934 CLOSE_WAIT -
tcp6 25 0 172.31.10.143:9092 172.31.46.69:41758 CLOSE_WAIT -
tcp6 25 0 172.31.10.143:9092 172.31.46.69:41584 CLOSE_WAIT -
tcp6 25 0 172.31.10.143:9092 172.31.46.69:41852 CLOSE_WAIT -
tcp6 1 0 172.31.10.143:9092 172.31.0.47:44342 CLOSE_WAIT -
Error in debezium
connect-prod | 2019-02-14 06:28:54,885 INFO || [Consumer clientId=consumer-3, groupId=4] Error sending fetch request (sessionId=1727876188, epoch=INITIAL) to node 2: org.apache.kafka.common.errors.DisconnectException. [org.apache.kafka.clients.FetchSessionHandler] connect-prod | 2019-02-14 06:28:55,448 INFO || [Consumer clientId=consumer-1, groupId=4] Error sending fetch request (sessionId=1379896198, epoch=INITIAL) to node 2: org.apache.kafka.common.errors.DisconnectException. [org.apache.kafka.clients.FetchSessionHandler]
What can be the reason behind this?
It appears that this is a known issue in Kafka 2.1.0.
https://issues.apache.org/jira/browse/KAFKA-7697
I think the connections stuck in Close_wait is a side effect of the real problem.
This issue has been fixed in Kafka version 2.1.1 which should be released in a few days. Looking forward to it.

Unable to connect to PostgreSQL on Google Cloud Instance

I have postgreSQL runiing on my google cloud instance and i added firewall rule "tcp 5432" on Google cloud firewall but still i am unable to connect, even telnet is not working.
officetaskpy#instance-1:/etc/postgresql/9.5/main$ netstat -ntpl
(Not all processes could be identified, non-owned process info
will not be shown, you would have to be root to see it all.)
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
tcp 0 0 127.0.0.1:5910 0.0.0.0:* LISTEN 9020/Xvnc
tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN -
tcp 0 0 0.0.0.0:5432 0.0.0.0:* LISTEN -
tcp 0 0 0.0.0.0:44801 0.0.0.0:* LISTEN 16023/phantomjs
tcp 0 0 0.0.0.0:53619 0.0.0.0:* LISTEN 812/phantomjs
tcp6 0 0 :::22 :::* LISTEN -
tcp6 0 0 :::5432 :::* LISTEN -
Result of netstat command
Above is my firewall rule. Is there anything which i am missing here.