I’m wondering if anyone has been able to get Kubernetes running properly over the Wireguard VPN.
I created a 2 node cluster on 2 VM’s linked by wireguard. The master node with the full control plane works fine and can accept worker nodes over the wireguard interface. I set the nodeip for kubelet to the wireguard ip and also set the iface argument for flannel to use the wireguard interface instead of the default. This seems to work well so far.
The problem arises when I try to join the worker node into the cluster via the join command.
Note that I also edited the node ip of kubelet to be the wireguard ip on the worker node.
On join all traffic to the node is dropped by the “Kubernetes Firewall”. By the kubernetes firewall I mean if you check iptables after issuing the join command on the worker node you will see KUBE-FIREWALL which drops all marked packets. The firewall is standard as its the same on the master but I presume that the piece I’m missing is what to do to get traffic flowing on the worker node after joining to the master node.
I’m unable to even ping google.com or communicate with the master over the Wireguard tunnel. Pods can’t be scheduled either. I have manually deleted the KUBE-FIREWALL rule as a test which then allows pods to be scheduled and regular traffic to flow on the worker node but Kubelet will quickly recreate the rule after around a minute.
I’m thinking a route needs to be created before the join or something along those lines.
Has anyone tried this before would really appreciate any suggestions for this.
After getting some help I figured out that the issue was Wiregaurd related. Specifically when running wg-quick as a service which apparently creates an ip rule that routes ALL outgoing traffic via wg0 interface, except WG background secured channel. This causes issues when trying to connect a worker to a cluster and so simply manually creating and starting the wg0 interface with something like the below will work
ip link add dev wg0 type wireguard
ip addr add 10.0.0.4/24 dev wg0
wg addconf wg0 /etc/wireguard/wg0.conf
ip link set wg0 up
Related
I have tried to change the NIC ip of the worker node directly. It seems that the master node automatically updates the ip information of the worker node. And it does not have any negative impact on the kubernetes cluster. Is it the simple and correct way to change the worker node ip? Or are there some other important steps that I have missed?
I created a mini cluster using kubeadm with two ubuntu18.04 VMs in one public network.
Indeed changing IP address of the worker node doesn't affect the cluster at all unless new IP address doesn't interfere with --pod-network-cidr.
Kubelet is responsible for it and it uses several options:
The kubelet is the primary "node agent" that runs on each node. It can
register the node with the apiserver using one of: the hostname; a
flag to override the hostname; or specific logic for a cloud provider.
For instance if you decide to change a hostname of worker node, it will become unreachable.
There are two ways to change IP address properly:
Re-join the worker node with new IP (already changed) to the cluster
Configure kubelet to advertise specific IP address.
Last option can be done by following:
modifying /etc/systemd/system/kubelet.service.d/10-kubeadm.conf with adding KUBELET_EXTRA_ARGS=--node-ip %NEW_IP_ADDRESS%.
sudo systemctl daemon-reload since config file was changed
sudo systemctl restart kubelet.service
Useful links:
Specify internal ip for worker nodes - (it's a bit old in terms of how it's done (it should be done as I described above), however the idea is the same).
CLI tools - Kubelet
I brought up openshift container platform on top of Google Cloud Platform.I used Multus for having Multiple networks to pods.
I used Macvlan CNI and whereabouts instead of static in ipam to avoid IP conflicts. I deployed three pods , two in same node and one in another node . The pods were assigned net1 interface without IP conflicts and unique mac address.
The problem is if the pods are in same node , pod communication happened via ping . But across nodes , pod communication is not happening , showing destination host unreachable. one more thing if I check route using the command "ip route get net1ipaddress" , the route is existing .
For example,
ip route get 192.168.1.1
192.168.1.1 via 10.0.32.1 dev ens4 src 10.0.32.3 uid 0
cache
But communication is not happening . How could I check further?
Many Thanks.
In order to access external network, your GCP network must allow all traffics for multiple mac addresses using macvlan from one computer instance. So you should check if promiscuous mode(no filter for mac addresses) is available on the GCP first. AFAIK, public cloud network does not allow that... So you can use ipvlan instead of macvlan as the alternative.
I am implementing a cluster-api controller using Kubernetes as the infrastructure provider - that is, I am trying to run Kubernetes Nodes as Kubernetes Pods and form a cluster within a cluster.
I have this working apart from network connectivity between Pods of the inner cluster (running on Pods of the infrastructure cluster), but I am stuck as to what the issue is.
I am running on GKE, using their default CNI implementation. I am then attempting to use Calico for an overlay implementation of the inner cluster, using IP-in-IP encapsulation so the Nodes of the infrastructure cluster do not need to know how to route inner cluster Pod IPs.
I am creating the infrastructure cluster as follows (the UBUNTU image is needed for the ipip kernel module required by Calico's IP-in-IP encapsulation.
gcloud container clusters create management-cluster --image-type=UBUNTU
I then deploy a number of nginx Pods to the inner cluster. If they land on the same inner cluster Node, they can connect to eachother. If they land on separate inner cluster Nodes they cannot, so I assume this means the IP-in-IP tunnel isn't working properly, but I am not sure why. This fails even if the inner cluster Nodes (Pods) land on the same infrastructure (outer cluster) Node. Pod and Service CIDR ranges of the two clusters do not overlap.
I realise this is not a supported use case for Calico, but I cannot see a reason why it is not possible and would like to get it working. Do the outer cluster Nodes need to support forwarding IP-in-IP packets? They are configured to forward IPv4 packets, but I am not sure if that is enough.
I guess more information is required to give a concrete reason for why this isn't working, but I am not too sure what that would be at this point and would be grateful for any direction.
It was necessary to allow ipencap protocol on the GKE nodes:
iptables -C FORWARD -p ipencap -j ACCEPT || iptables -A FORWARD -p ipencap -j ACCEPT
External firewall logs show blocked connection from < node IP >:< big port >.
The current cluster uses calico networking.
How do I detect which pod trying to connect?
This would usually be pretty hard to work out, you would have to check the NAT table on the node where the packets exited to the public internet.
I have a cluster with container range 10.101.64.0/19 on a net A and subnet SA with ranges 10.101.0.0/18. On the same subnet, there is VM in GCE with IP 10.101.0.4 and it can be pinged just fine from within the cluster, e.g. from a node with 10.101.0.3. However, if I go to a pod on this node which got address 10.101.67.191 (which is expected - this node assigns addresses 10.101.67.0/24 or something), I don't get meaningful answer from that VM I want to access from this pod. Using tcpdump on icmp, I can see that when I ping that VM machine from the pod, the ping gets there but I don't receive ACK in the pod. Seems like VM is just throwing it away.
Any idea how to resolve it? Some routes or firewalls? I am using the same topology in the default subnet created by kubernetes where this work but I cannot find anything relevant which could explain this (there are some routes and firewall rules which could influence it but I wasn't successful when trying to mimic them in my subnet)
I think it is a firewall issue.
Here I've already provided the solution on Stakoverflow.
It may help to solve your case.