I want to manually delete iptables rules for debugging.
I have several rules created by kube-proxy based on service nettools:
# kubectl get endpoints nettools
NAME ENDPOINTS AGE
nettools 172.16.27.138:7493 1h
And its iptables rules：
# iptables-save|grep nettools
-A KUBE-SEP-6DFMUWHMXOYMFWKG -s 172.16.27.138/32 -m comment --comment "default/nettools:web" -j KUBE-MARK-MASQ
-A KUBE-SEP-6DFMUWHMXOYMFWKG -p tcp -m comment --comment "default/nettools:web" -m tcp -j DNAT --to-destination 172.16.27.138:7493
-A KUBE-SERVICES -d 10.0.1.2/32 -p tcp -m comment --comment "default/nettools:web cluster IP" -m tcp --dport 7493 -j KUBE-SVC-INDS3KD6I5PFKUWF
-A KUBE-SVC-INDS3KD6I5PFKUWF -m comment --comment "default/nettools:web" -j KUBE-SEP-6DFMUWHMXOYMFWKG
However，I cannot delete those rules:
# iptables -D KUBE-SVC-INDS3KD6I5PFKUWF -m comment --comment "default/nettools:web" -j KUBE-SEP-6DFMUWHMXOYMFWKG
iptables v1.4.21: Couldn't load target `KUBE-SEP-6DFMUWHMXOYMFWKG':No such file or directory
# iptables -D KUBE-SERVICES -d 10.0.1.2/32 -p tcp -m comment --comment "default/nettools:web cluster IP" -m tcp --dport 7493 -j KUBE-SVC-INDS3KD6I5PFKUWF
iptables v1.4.21: Couldn't load target `KUBE-SVC-INDS3KD6I5PFKUWF':No such file or directory
There are multiple tables in play when dealing with iptables. filter table is the default if nothing is specified. The rules that you are trying to delete are part of the nat table.
Just add -t nat to your rules to delete those rules.
Example:
# iptables -t nat -D KUBE-SVC-INDS3KD6I5PFKUWF -m comment --comment "default/nettools:web" -j KUBE-SEP-6DFMUWHMXOYMFWKG
Related
Closed. This question is not about programming or software development. It is not currently accepting answers.
This question does not appear to be about a specific programming problem, a software algorithm, or software tools primarily used by programmers. If you believe the question would be on-topic on another Stack Exchange site, you can leave a comment to explain where the question may be able to be answered.
Closed last month.
Improve this question
I currently have a cluster with 2 nodes:
node1 - 192.168.1.20 (master, worker)
node2 - 192.168.1.30 (worker)
When running a pod on either of the nodes there is no outbound internet connection.
The problem seems to not be related to DNS since even pinging a public IP address of google.com does not work.
pod1-node1$ traceroute to google.com (216.58.215.46), 30 hops max, 46 byte packets
1 192-168-1-20.kubernetes.default.svc.home-cloud.local (192.168.1.20) 0.008 ms 0.009 ms 0.009 ms
2 *
Nodes have internet access and everything works there
node1$ ping 216.58.215.46 -> OK
node2$ ping 216.58.215.46 -> OK
Ip tables on master node:
-P PREROUTING ACCEPT
-P INPUT ACCEPT
-P OUTPUT ACCEPT
-P POSTROUTING ACCEPT
-N DOCKER
-N DOCKER-INGRESS
-N KUBE-FIREWALL
-N KUBE-KUBELET-CANARY
-N KUBE-LOAD-BALANCER
-N KUBE-MARK-DROP
-N KUBE-MARK-MASQ
-N KUBE-NODE-PORT
-N KUBE-POSTROUTING
-N KUBE-SERVICES
-A PREROUTING -m comment --comment "kubernetes service portals" -j KUBE-SERVICES
-A PREROUTING -m addrtype --dst-type LOCAL -j DOCKER-INGRESS
-A PREROUTING -m addrtype --dst-type LOCAL -j DOCKER
-A OUTPUT -m comment --comment "kubernetes service portals" -j KUBE-SERVICES
-A OUTPUT -m addrtype --dst-type LOCAL -j DOCKER-INGRESS
-A OUTPUT ! -d 127.0.0.0/8 -m addrtype --dst-type LOCAL -j DOCKER
-A POSTROUTING -m comment --comment "kubernetes postrouting rules" -j KUBE-POSTROUTING
-A POSTROUTING -o docker_gwbridge -m addrtype --src-type LOCAL -j MASQUERADE
-A POSTROUTING -s 172.17.0.0/16 ! -o docker0 -j MASQUERADE
-A POSTROUTING -s 172.18.0.0/16 ! -o docker_gwbridge -j MASQUERADE
-A POSTROUTING -s 172.17.0.2/32 -d 172.17.0.2/32 -p tcp -m tcp --dport 9000 -j MASQUERADE
-A POSTROUTING -s 172.17.0.3/32 -d 172.17.0.3/32 -p udp -m udp --dport 34197 -j MASQUERADE
-A POSTROUTING -s 172.17.0.3/32 -d 172.17.0.3/32 -p tcp -m tcp --dport 27015 -j MASQUERADE
-A DOCKER -i docker0 -j RETURN
-A DOCKER -i docker_gwbridge -j RETURN
-A DOCKER ! -i docker0 -p tcp -m tcp --dport 9000 -j DNAT --to-destination 172.17.0.2:9000
-A DOCKER ! -i docker0 -p udp -m udp --dport 34197 -j DNAT --to-destination 172.17.0.3:34197
-A DOCKER ! -i docker0 -p tcp -m tcp --dport 27015 -j DNAT --to-destination 172.17.0.3:27015
-A DOCKER-INGRESS -p tcp -m tcp --dport 3306 -j DNAT --to-destination 172.18.0.2:3306
-A DOCKER-INGRESS -p tcp -m tcp --dport 9980 -j DNAT --to-destination 172.18.0.2:9980
-A DOCKER-INGRESS -j RETURN
-A KUBE-FIREWALL -j KUBE-MARK-DROP
-A KUBE-LOAD-BALANCER -j KUBE-MARK-MASQ
-A KUBE-MARK-DROP -j MARK --set-xmark 0x8000/0x8000
-A KUBE-MARK-MASQ -j MARK --set-xmark 0x4000/0x4000
-A KUBE-NODE-PORT -p tcp -m comment --comment "Kubernetes nodeport TCP port for masquerade purpose" -m set --match-set KUBE-NODE-PORT-TCP dst -j KUBE-MARK-MASQ
-A KUBE-POSTROUTING -m comment --comment "Kubernetes endpoints dst ip:port, source ip for solving hairpin purpose" -m set --match-set KUBE-LOOP-BACK dst,dst,src -j MASQUERADE
-A KUBE-POSTROUTING -m mark ! --mark 0x4000/0x4000 -j RETURN
-A KUBE-POSTROUTING -j MARK --set-xmark 0x4000/0x0
-A KUBE-POSTROUTING -m comment --comment "kubernetes service traffic requiring SNAT" -j MASQUERADE --random-fully
-A KUBE-SERVICES -m comment --comment "Kubernetes service lb portal" -m set --match-set KUBE-LOAD-BALANCER dst,dst -j KUBE-LOAD-BALANCER
-A KUBE-SERVICES ! -s 10.233.64.0/18 -m comment --comment "Kubernetes service cluster ip + port for masquerade purpose" -m set --match-set KUBE-CLUSTER-IP dst,dst -j KUBE-MARK-MASQ
-A KUBE-SERVICES -m addrtype --dst-type LOCAL -j KUBE-NODE-PORT
-A KUBE-SERVICES -m set --match-set KUBE-CLUSTER-IP dst,dst -j ACCEPT
-A KUBE-SERVICES -m set --match-set KUBE-LOAD-BALANCER dst,dst -j ACCEPT
From within a pod I get a strange looking ip route result. My docker overlay network is '10.233.64.0/18' configured with calico.
pod1-node1$ ip route
default via 169.254.1.1 dev eth0
169.254.1.1 dev eth0 scope link
I also have metallb installed and it installs a kube-proxy but not sure how exactly it works or if it can be related to the problem.
Any advice is greatly appreciated.
Cheers Alex
EDIT: Cluster is bare metal, installed with kubespray, I haven't installed any CNI manually
It turned out to be a bug in the CNI - Calico.
I manually set the calico version in my kubespray to 3.22.1 and the problem was solved.
I then encountered other connectivity issues between my nodes and opted to use flannel.
Since then everything works fine.
My advise to others after spending a week of research myself :
Container Networking is HARD
Take it slow and do not cut corners ! If you are missing some
knowledge about a topic - stop and learn at least the basics of the
said topic before continuing.
Diagnose your problem - most commonly it is DNS or Packet Routing:
Get the IP of google.com on another machine and ping the IP from within a pod - if it works you have a DNS problem if it doesn't then probably CNI related (packet routing).
Learn how resolved works and how it is configured
This will allow you to check what DNS is actually used on your node and within your pod.
Learn how kubernetes DNS works and why it is needed:
It allows pod to service communication to work among other things. It also needs upstream DNS addresses so that it can resolve things outside your cluster.
Learn how IP tables work and how masquerading works
It allows you to understand how network packets are routed between your node and your pods, using the bridge interface.
Learn what a bridge interface is
The docker0 bridge interface allows all your containers (and pods by extension) to talk to outside world.
Read this very informational article on how calico interacts with your IP tables to create nat masquerading rules on the fly:
https://medium.com/#bikramgupta/pod-network-troubleshooting-while-using-calico-on-kubernetes-ee78b731d4d8
Context
Maybe there is unnecessary redundancy in the iptables rules generated by kubeadm init for kube-proxy:
iptables -t filter -S
output:
-P INPUT ACCEPT
-P FORWARD DROP
-P OUTPUT ACCEPT
-N KUBE-EXTERNAL-SERVICES
-N KUBE-FIREWALL
-N KUBE-FORWARD
-N KUBE-SERVICES
-A INPUT -m conntrack --ctstate NEW -m comment --comment "kubernetes externally-visible service portals" -j KUBE-EXTERNAL-SERVICES
-A INPUT -j KUBE-FIREWALL
-A FORWARD -m comment --comment "kubernetes forwarding rules" -j KUBE-FORWARD
-A FORWARD -s 10.244.0.0/16 -j ACCEPT
-A FORWARD -d 10.244.0.0/16 -j ACCEPT
-A OUTPUT -m conntrack --ctstate NEW -m comment --comment "kubernetes service portals" -j KUBE-SERVICES
-A OUTPUT -j KUBE-FIREWALL
-A KUBE-FIREWALL -m comment --comment "kubernetes firewall for dropping marked packets" -m mark --mark 0x8000/0x8000 -j DROP
-A KUBE-FORWARD -m comment --comment "kubernetes forwarding rules" -m mark --mark 0x4000/0x4000 -j ACCEPT
-A KUBE-FORWARD -s 10.244.0.0/16 -m comment --comment "kubernetes forwarding conntrack pod source rule" -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A KUBE-FORWARD -d 10.244.0.0/16 -m comment --comment "kubernetes forwarding conntrack pod destination rule" -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
The 10.244.0.0/16 range corresponds to the pod overlay network.
Lets focus on the FORWARD chain.
-P FORWARD DROP
-N KUBE-FORWARD
-A FORWARD -m comment --comment "kubernetes forwarding rules" -j KUBE-FORWARD
-A FORWARD -s 10.244.0.0/16 -j ACCEPT
-A FORWARD -d 10.244.0.0/16 -j ACCEPT
-A KUBE-FORWARD -m comment --comment "kubernetes forwarding rules" -m mark --mark 0x4000/0x4000 -j ACCEPT
-A KUBE-FORWARD -s 10.244.0.0/16 -m comment --comment "kubernetes forwarding conntrack pod source rule" -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A KUBE-FORWARD -d 10.244.0.0/16 -m comment --comment "kubernetes forwarding conntrack pod destination rule" -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
Question:
Why the KUBE-FORWARD accepts packages within the overlay network, when their connection state is RELATED or ESTABLISHED if FORWARD chain will accept all packet traffic within the overlay network regardless their connection state?
Note:
The kubernetes cluster works fine.
This duplication is there, because the default FORWARD policy can be disabled for some reason, and Kubernetes still wants to forward packets that either:
Are marked with "masqueradeMark" (those can start new connections)
Are part of already established connection
You can try reading the comments in the k8s source: https://github.com/kubernetes/kubernetes/blob/master/pkg/proxy/iptables/proxier.go#L1325
In general, one should expect some duplication in iptables rules when those rules are automatically managed. It makes it easier to code the automation.
I'm using kubernetes 1.6.7 cluster, 3 nodes as master (node1, node2 and node3).
The apiserver cluster ip as bellow:
root#node1:/etc/kubernetes/manifests# kubectl get svc
NAME CLUSTER-IP EXTERNAL-IP PORT(S) AGE
kubernetes 10.233.0.1 <none> 443/TCP 14d
root#node1:/etc/kubernetes/manifests# kubectl get endpoints
kubernetes 172.16.10.11:6443,172.16.10.12:6443,172.16.10.13:6443 14d
The problem is when node2 was down, tiller pod can't access 10.233.0.1:443, logs as bellow:
[storage] 2018/08/28 06:46:34 listing all releases with filter
[storage/driver] 2018/08/28 06:46:34 list: failed to list: Get https://10.233.0.1:443/api/v1/namespaces/kube-system/configmaps?labelSelector=OWNER%3DTILLER: dial tcp 10.233.0.1:443: connect: no route to host
and the service kubernetes still has 3 endpoints, but i think the 172.16.10.12 should be delete from the endpoint list. iptables as bellow:
-A KUBE-SVC-LRLUS54FOYJDJ5GT -m comment --comment "default/wangxj35-nginx:nginx" -j KUBE-SEP-GGIOR777CPKVSIB7
-A KUBE-SVC-NPX46M4PTMTKRN6Y -m comment --comment "default/kubernetes:https" -m recent --rcheck --seconds 10800 --reap --name KUBE-SEP-CS4Z4M5BFAAWBFXQ --mask 255.255.255.255 --rsource -j KUBE-SEP-CS4Z4M5BFAAWBFXQ
-A KUBE-SVC-NPX46M4PTMTKRN6Y -m comment --comment "default/kubernetes:https" -m recent --rcheck --seconds 10800 --reap --name KUBE-SEP-MQZA2ZXHX6GRRY22 --mask 255.255.255.255 --rsource -j KUBE-SEP-MQZA2ZXHX6GRRY22
-A KUBE-SVC-NPX46M4PTMTKRN6Y -m comment --comment "default/kubernetes:https" -m recent --rcheck --seconds 10800 --reap --name KUBE-SEP-ZX77NMWHNJWLXYWF --mask 255.255.255.255 --rsource -j KUBE-SEP-ZX77NMWHNJWLXYWF
-A KUBE-SVC-NPX46M4PTMTKRN6Y -m comment --comment "default/kubernetes:https" -m statistic --mode random --probability 0.33332999982 -j KUBE-SEP-CS4Z4M5BFAAWBFXQ
-A KUBE-SVC-NPX46M4PTMTKRN6Y -m comment --comment "default/kubernetes:https" -m statistic --mode random --probability 0.50000000000 -j KUBE-SEP-MQZA2ZXHX6GRRY22
-A KUBE-SVC-NPX46M4PTMTKRN6Y -m comment --comment "default/kubernetes:https" -j KUBE-SEP-ZX77NMWHNJWLXYWF
-A KUBE-SEP-CS4Z4M5BFAAWBFXQ -p tcp -m comment --comment "default/kubernetes:https" -m recent --set --name KUBE-SEP-CS4Z4M5BFAAWBFXQ --mask 255.255.255.255 --rsource -m tcp -j DNAT --to-destination 172.16.10.11:6443
-A KUBE-SEP-MQZA2ZXHX6GRRY22 -p tcp -m comment --comment "default/kubernetes:https" -m recent --set --name KUBE-SEP-MQZA2ZXHX6GRRY22 --mask 255.255.255.255 --rsource -m tcp -j DNAT --to-destination 172.16.10.12:6443
-A KUBE-SEP-ZX77NMWHNJWLXYWF -p tcp -m comment --comment "default/kubernetes:https" -m recent --set --name KUBE-SEP-ZX77NMWHNJWLXYWF --mask 255.255.255.255 --rsource -m tcp -j DNAT --to-destination 172.16.10.13:6443
Could anyone help me where the static pods service created by kubernetes source code?
Thanks.
Controller.go creates the kubernetes service:
// UpdateKubernetesService attempts to update the default Kube service.
func (c *Controller) UpdateKubernetesService(reconcile bool)> error {
// Update service & endpoint records.
Also, to work around this problem, kubernetes1.9.0 has added --endpoint-reconciler-type=lease to allow apiserver to track the alive apiserver, see issue51698, while for release below 1.9.0, you can unset the --apiserver-count flag to let alive apiserver to preempt the only one endpoint position.
So, I was experimenting and added a Service with an ExternalIP set to the end hosts ipv6 address.
This is a configuration that's accepted, but not supported, in Kubernetes, and causes the kube-proxy to do fail.
From the kube-proxy logs:
-A KUBE-SERVICES -m comment --comment "default/frontend: external IP" -m tcp -p tcp -d 207.154.225.168/32 --dport 80 -j KUBE-MARK-MASQ
-A KUBE-SERVICES -m comment --comment "default/frontend: external IP" -m tcp -p tcp -d 207.154.225.168/32 --dport 80 -m physdev ! --physdev-is-in -m addrtype ! --src-type LOCAL -j KUBE-SVC-GYQQTB6TY565JPRW
-A KUBE-SERVICES -m comment --comment "default/frontend: external IP" -m tcp -p tcp -d 207.154.225.168/32 --dport 80 -m addrtype --dst-type LOCAL -j KUBE-SVC-GYQQTB6TY565JPRW
-A KUBE-SERVICES -m comment --comment "default/frontend: external IP" -m tcp -p tcp -d 2a03:b0c0:3:d0::43bb:4001/32 --dport 80 -j KUBE-MARK-MASQ
-A KUBE-SERVICES -m comment --comment "default/frontend: external IP" -m tcp -p tcp -d 2a03:b0c0:3:d0::43bb:4001/32 --dport 80 -m physdev ! --physdev-is-in -m addrtype ! --src-type LOCAL -j KUBE-SVC-GYQQTB6TY565JPRW
E0502 07:38:39.913815 1 proxier.go:1312] Failed to execute iptables-restore: exit status 2 (iptables-restore v1.4.21: host/network 2a03:b0c0:3:d0::43bb:4001' not found
Error occurred at line: 53
Tryiptables-restore -h' or 'iptables-restore --help' for more information.
This is the problem:
There is no "default/frontend" service defined. I have explicitly deleted it,
kubectl get services --namespace=default
NAME CLUSTER-IP EXTERNAL-IP PORT(S) AGE
kubernetes 10.3.0.1 443/TCP 3d
However, kube-proxy on all my nodes still attempts to add these, which causes all services created after this misstep to break.
So, the question is, how do I purge this ghost of a service from my cluster?
One solution seems to be to reboot the API server/Kubernetes master. Not a pleasant one, but services work again at least.
I tried to improve my server security by setting up some iptables firewall rules. The result is that Facebook login with Omniauth stopped working. In my logs I see that Facebook is sending some packages to my server ports 37035 and 41198 at least. Why? There is nothing running in those ports.
Can someone say which ports I should open so that Facebook login with Omniauth could start working again on my site.
The rules I applied are:
# Delete all existing rules
iptables -X
# Set default rules
iptables -P INPUT DROP
iptables -P FORWARD ACCEPT
iptables -P OUTPUT ACCEPT
# Allow ssh in
iptables -A INPUT -i eth0 -p tcp --dport 22 -m state --state NEW,ESTABLISHED -j ACCEPT
iptables -A OUTPUT -o eth0 -p tcp --sport 22 -m state --state ESTABLISHED -j ACCEPT
# Allow incoming HTTP
iptables -A INPUT -i eth0 -p tcp --dport 80 -m state --state NEW,ESTABLISHED -j ACCEPT
iptables -A OUTPUT -o eth0 -p tcp --sport 80 -m state --state ESTABLISHED -j ACCEPT
# Allow outgoing SSH
iptables -A OUTPUT -o eth0 -p tcp --dport 22 -m state --state NEW,ESTABLISHED -j ACCEPT
iptables -A INPUT -i eth0 -p tcp --sport 22 -m state --state ESTABLISHED -j ACCEPT
# Allow ping from outside
iptables -A INPUT -p icmp --icmp-type echo-request -j ACCEPT
iptables -A OUTPUT -p icmp --icmp-type echo-reply -j ACCEPT
# Allow pingging other servers
iptables -A OUTPUT -p icmp --icmp-type echo-request -j ACCEPT
iptables -A INPUT -p icmp --icmp-type echo-reply -j ACCEPT
# Allow loopback access
iptables -A INPUT -i lo -j ACCEPT
iptables -A OUTPUT -o lo -j ACCEPT
# Allow sendmail and postfix
iptables -A INPUT -i eth0 -p tcp --dport 25 -m state --state NEW,ESTABLISHED -j ACCEPT
iptables -A OUTPUT -o eth0 -p tcp --sport 25 -m state --state ESTABLISHED -j ACCEPT
# Allow dns lookups
iptables -A OUTPUT -p udp -o eth0 --dport 53 -j ACCEPT
iptables -A INPUT -p udp -i eth0 --sport 53 -j ACCEPT
# Prevent dos attacks - upgrade to hashlimit if needed
iptables -A INPUT -p tcp --dport 80 -m limit --limit 25/minute --limit-burst 100 -j ACCEPT
# Log dropped packages
iptables -N LOGGING
iptables -A INPUT -j LOGGING
iptables -A LOGGING -m limit --limit 2/min -j LOG --log-prefix "IPTables Packet Dropped: " --log-level 7
iptables -A LOGGING -j DROP
Here is an example log entry from my syslog (My IP is filtered)
IPTables Packet Dropped: IN=eth0 OUT= MAC=40:40:ea:31:ac:8d:64:00:f1:cd:1f:7f:08:00 SRC=69.171.224.54 DST=my_ip LEN=56 TOS=0x00 PREC=0x00 TTL=86 ID=0 DF PROTO=TCP SPT=443 DPT=44605 WINDOW=14480 RES=0x00 ACK SYN URGP=0
Finally got an answer for this question. https://superuser.com/questions/479503/why-are-ports-30000-to-60000-needed-when-browsing-the-net
Ports from 32768 to 61000 are needed for http connections.