minikube mount broken on VPN - kubernetes

So I'm having issues with minikube mount command while on Big-IP VPN. Basically, the command looks like it's able to ssh into the minikube VM, but for whatever reason, minikube can't mount the host folder.
$ minikube mount --v=10 app_shared_sec:/app/shared/sec
Mounting app_shared_sec into /app/shared/sec on the minikube VM
This daemon process needs to stay alive for the mount to still be accessible...
ufs starting
Using SSH client type: native
&{{{<nil> 0 [] [] []} docker [0x140f940] 0x140f910 [] 0s} 127.0.0.1 57930 <nil> <nil>}
About to run SSH command:
sudo umount /app/shared/sec;
SSH cmd err, output: Process exited with status 32: umount: /app/shared/sec: not mounted.
Using SSH client type: native
&{{{<nil> 0 [] [] []} docker [0x140f940] 0x140f910 [] 0s} 127.0.0.1 57930 <nil> <nil>}
About to run SSH command:
sudo mkdir -p /app/shared/sec || true;
sudo mount -t 9p -o trans=tcp,port=51501,dfltuid=1001,dfltgid=1001,version=9p2000.u,msize=262144 192.168.99.1 /app/shared/sec;
sudo chmod 775 /app/shared/sec || true;
SSH cmd err, output: <nil>: mount: /app/shared/sec: mount(2) system call failed: Connection timed out.
Running netstat within the minikube VM seems to point to it being able to read the host.
$ netstat -nr
Kernel IP routing table
Destination Gateway Genmask Flags MSS Window irtt Iface
0.0.0.0 10.0.2.2 0.0.0.0 UG 0 0 0 eth0
10.0.2.0 0.0.0.0 255.255.255.0 U 0 0 0 eth0
10.0.2.2 0.0.0.0 255.255.255.255 UH 0 0 0 eth0
172.17.0.0 0.0.0.0 255.255.0.0 U 0 0 0 docker0
192.168.99.0 0.0.0.0 255.255.255.0 U 0 0 0 eth1
The issue appears to be because VPN blows away the vboxnet interface on the host, minikube VM has no way of communicating with it, causing the mount to fail. When trying to re-create the route, Big-IP seems to watch for changes and removes it. Not sure what else to do at this point.

Related

Trying phppgadmin docker container to view postgres database on host: it says login failed

I want to view the host postgresql with phppgadmin docker container
My host is archlinux and with postgresql server running on it.
I have /var/lib/postgres/data/postgresql.conf
listen_addresses = "*"
and
/var/lib/postgres/data/pg_hba.conf
host all all 172.17.0.0/16 password
I want to view the postgresql tables. So i using phppgadmin docker with the following command
docker run --name='phppgadmin' --rm \
--publish=8888:80 \
-e PHP_PG_ADMIN_SERVER_HOST="127.0.0.1" \
dockage/phppgadmin:latest
Now i can open the phppgadmin from 127.0.0.1:8888/phppgadmin
But when i try to login it says login failed
I have a django project on my host using the using hosts postgresql. That works well with the settings
DATABASES = {
'default': {
'ENGINE': 'django.db.backends.postgresql_psycopg2',
'NAME': ‘<db_name>’,
'USER': '<db_username>',
'PASSWORD': '<password>',
'HOST': '127.0.0.1',
'PORT': '5432',
}
}
Also my netstat output on host
$ netstat -nrv
Kernel IP routing table
Destination Gateway Genmask Flags MSS Window irtt Iface
0.0.0.0 192.168.1.1 0.0.0.0 UG 0 0 0 wlp3s0
172.17.0.0 0.0.0.0 255.255.0.0 U 0 0 0 docker0
172.18.0.0 0.0.0.0 255.255.0.0 U 0 0 0 br-1c7e732767f4
172.20.0.0 0.0.0.0 255.255.0.0 U 0 0 0 br-17604ffc4858
192.168.1.0 0.0.0.0 255.255.255.0 U 0 0 0 wlp3s0
on my docker container
$ netstat -nrv
Kernel IP routing table
Destination Gateway Genmask Flags MSS Window irtt Iface
0.0.0.0 172.17.0.1 0.0.0.0 UG 0 0 0 eth0
172.17.0.0 0.0.0.0 255.255.0.0 U 0 0 0 eth0
From netstat -nrv on my phppgadmin docker container
$ netstat -nrv
Kernel IP routing table
Destination Gateway Genmask Flags MSS Window irtt Iface
0.0.0.0 172.17.0.1 0.0.0.0 UG 0 0 0 eth0
172.17.0.0 0.0.0.0 255.255.0.0 U 0 0 0 eth0
So the ip address of the host is 172.17.0.1
Change PHP_PG_ADMIN_SERVER_HOST="127.0.0.1" to PHP_PG_ADMIN_SERVER_HOST="172.17.0.1"
docker run --name='phppgadmin' --rm \
--publish=8888:80 \
-e PHP_PG_ADMIN_SERVER_HOST="172.17.0.1" \
dockage/phppgadmin:latest
with:
I have /var/lib/postgres/data/postgresql.conf
listen_addresses = "localhost,127.0.0.1,172.17.0.1"
and
/var/lib/postgres/data/pg_hba.conf
host all all 172.17.0.0/16 md5
Open 127.0.0.1:8888/phppgadmin and login with non Superuser
For Superuser its still not working.

No route to host from some Kubernetes containers to other containers in same cluster

This is a Kubespray deployment using calico. All the defaults are were left as-is except for the fact that there is a proxy. Kubespray ran to the end without issues.
Access to Kubernetes services started failing and after investigation, there was no route to host to the coredns service. Accessing a K8S service by IP worked. Everything else seems to be correct, so I am left with a cluster that works, but without DNS.
Here is some background information:
Starting up a busybox container:
# nslookup kubernetes.default
Server: 169.254.25.10
Address: 169.254.25.10:53
** server can't find kubernetes.default: NXDOMAIN
*** Can't find kubernetes.default: No answer
Now the output while explicitly defining the IP of one of the CoreDNS pods:
# nslookup kubernetes.default 10.233.0.3
;; connection timed out; no servers could be reached
Notice that telnet to the Kubernetes API works:
# telnet 10.233.0.1 443
Connected to 10.233.0.1
kube-proxy logs:
10.233.0.3 is the service IP for coredns. The last line looks concerning, even though it is INFO.
$ kubectl logs kube-proxy-45v8n -nkube-system
I1114 14:19:29.657685 1 node.go:135] Successfully retrieved node IP: X.59.172.20
I1114 14:19:29.657769 1 server_others.go:176] Using ipvs Proxier.
I1114 14:19:29.664959 1 server.go:529] Version: v1.16.0
I1114 14:19:29.665427 1 conntrack.go:52] Setting nf_conntrack_max to 262144
I1114 14:19:29.669508 1 config.go:313] Starting service config controller
I1114 14:19:29.669566 1 shared_informer.go:197] Waiting for caches to sync for service config
I1114 14:19:29.669602 1 config.go:131] Starting endpoints config controller
I1114 14:19:29.669612 1 shared_informer.go:197] Waiting for caches to sync for endpoints config
I1114 14:19:29.769705 1 shared_informer.go:204] Caches are synced for service config
I1114 14:19:29.769756 1 shared_informer.go:204] Caches are synced for endpoints config
I1114 14:21:29.666256 1 graceful_termination.go:93] lw: remote out of the list: 10.233.0.3:53/TCP/10.233.124.23:53
I1114 14:21:29.666380 1 graceful_termination.go:93] lw: remote out of the list: 10.233.0.3:53/TCP/10.233.122.11:53
All pods are running without crashing/restarts etc. and otherwise services behave correctly.
IPVS looks correct. CoreDNS service is defined there:
# ipvsadm -ln
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
-> RemoteAddress:Port Forward Weight ActiveConn InActConn
TCP 10.233.0.1:443 rr
-> x.59.172.19:6443 Masq 1 0 0
-> x.59.172.20:6443 Masq 1 1 0
TCP 10.233.0.3:53 rr
-> 10.233.122.12:53 Masq 1 0 0
-> 10.233.124.24:53 Masq 1 0 0
TCP 10.233.0.3:9153 rr
-> 10.233.122.12:9153 Masq 1 0 0
-> 10.233.124.24:9153 Masq 1 0 0
TCP 10.233.51.168:3306 rr
-> x.59.172.23:6446 Masq 1 0 0
TCP 10.233.53.155:44134 rr
-> 10.233.89.20:44134 Masq 1 0 0
UDP 10.233.0.3:53 rr
-> 10.233.122.12:53 Masq 1 0 314
-> 10.233.124.24:53 Masq 1 0 312
Host routing also looks correct.
# ip r
default via x.59.172.17 dev ens3 proto dhcp src x.59.172.22 metric 100
10.233.87.0/24 via x.59.172.21 dev tunl0 proto bird onlink
blackhole 10.233.89.0/24 proto bird
10.233.89.20 dev calib88cf6925c2 scope link
10.233.89.21 dev califdffa38ed52 scope link
10.233.122.0/24 via x.59.172.19 dev tunl0 proto bird onlink
10.233.124.0/24 via x.59.172.20 dev tunl0 proto bird onlink
x.59.172.16/28 dev ens3 proto kernel scope link src x.59.172.22
x.59.172.17 dev ens3 proto dhcp scope link src x.59.172.22 metric 100
172.17.0.0/16 dev docker0 proto kernel scope link src 172.17.0.1 linkdown
I have redeployed this same cluster in separate environments with flannel and calico with iptables instead of ipvs. I have also disabled the docker http proxy after deploy temporarily. None of which makes any difference.
Also:
kube_service_addresses: 10.233.0.0/18
kube_pods_subnet: 10.233.64.0/18
(They do not overlap)
What is the next step in debugging this issue?
I highly recommend you to avoid using latest busybox image to troubleshoot DNS. There are few issues reported regarding dnslookup on versions newer than 1.28.
v 1.28.4
user#node1:~$ kubectl exec -ti busybox busybox | head -1
BusyBox v1.28.4 (2018-05-22 17:00:17 UTC) multi-call binary.
user#node1:~$ kubectl exec -ti busybox -- nslookup kubernetes.default
Server: 169.254.25.10
Address 1: 169.254.25.10
Name: kubernetes.default
Address 1: 10.233.0.1 kubernetes.default.svc.cluster.local
v 1.31.1
user#node1:~$ kubectl exec -ti busyboxlatest busybox | head -1
BusyBox v1.31.1 (2019-10-28 18:40:01 UTC) multi-call binary.
user#node1:~$ kubectl exec -ti busyboxlatest -- nslookup kubernetes.default
Server: 169.254.25.10
Address: 169.254.25.10:53
** server can't find kubernetes.default: NXDOMAIN
*** Can't find kubernetes.default: No answer
command terminated with exit code 1
Going deeper and exploring more possibilities, I've reproduced your problem on GCP and after some digging I was able to figure out what is causing this communication problem.
GCE (Google Compute Engine) blocks traffic between hosts by default; we have to allow Calico traffic to flow between containers on different hosts.
According to calico documentation, you can do it by creating a firewall allowing this communication rule:
gcloud compute firewall-rules create calico-ipip --allow 4 --network "default" --source-ranges "10.128.0.0/9"
You can verify the rule with this command:
gcloud compute firewall-rules list
This is not present on the most recent calico documentation but it's still true and necessary.
Before creating firewall rule:
user#node1:~$ kubectl exec -ti busybox2 -- nslookup kubernetes.default
Server: 10.233.0.3
Address 1: 10.233.0.3 coredns.kube-system.svc.cluster.local
nslookup: can't resolve 'kubernetes.default'
command terminated with exit code 1
After creating firewall rule:
user#node1:~$ kubectl exec -ti busybox2 -- nslookup kubernetes.default
Server: 10.233.0.3
Address 1: 10.233.0.3 coredns.kube-system.svc.cluster.local
Name: kubernetes.default
Address 1: 10.233.0.1 kubernetes.default.svc.cluster.local
It doesn't matter if you bootstrap your cluster using kubespray or kubeadm, this problem will happen because calico needs to communicate between nodes and GCE is blocking it as default.
This is what works for me, I tried to install my k8s cluster using kubespray configured with calico as CNI and containerd as container runtime
iptables -P INPUT ACCEPT
iptables -P FORWARD ACCEPT
iptables -P OUTPUT ACCEPT
iptables -F
[delete coredns pod]

kubernetes : unable to join a node

I have created a master and am trying to join a node to create a cluster. When I try the join command I get the below error. Both the nodes are on the same network. The error message indicates that no routing exist to the host. I'm not sure how to establish a route to the host. Any help is appreciated.
sudo kubeadm join --token d23afe.14fde99cd03def7e 192.168.178.24:6443 --discovery-token-ca-cert-hash sha256:6a5e2674825e683bbdfe9bab512b03c556bcf89d8648317a64372bb44746bb39
[preflight] Running pre-flight checks.
[WARNING SystemVerification]: docker version is greater than the most recently validated version. Docker version: 18.02.0-ce. Max validated version: 17.03
[WARNING FileExisting-crictl]: crictl not found in system path
[discovery] Trying to connect to API Server "192.168.178.24:6443"
[discovery] Created cluster-info discovery client, requesting info from "https://192.168.178.24:6443"
[discovery] Failed to request cluster info, will try again: [Get https://192.168.178.24:6443/api/v1/namespaces/kube-public/configmaps/cluster-info: dial tcp 192.168.178.24:6443: getsockopt: no route to host]
Here's the output of sudo route. Unfortunately, I have little knowledge to troubleshoot from this output
Here's the output of
`sudo route
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
default 192.168.178.1 0.0.0.0 UG 202 0 0 eth0
10.32.0.0 0.0.0.0 255.240.0.0 U 0 0 0 weave
link-local 0.0.0.0 255.255.0.0 U 205 0 0 datapath
link-local 0.0.0.0 255.255.0.0 U 210 0 0 vethwe-datapath
link-local 0.0.0.0 255.255.0.0 U 211 0 0 vethwe-bridge
link-local 0.0.0.0 255.255.0.0 U 212 0 0 vxlan-6784
172.17.0.0 0.0.0.0 255.255.0.0 U 0 0 0 docker0
192.168.178.0 0.0.0.0 255.255.255.0 U 202 0 0 eth0
`
I managed to identify the issue. The issue was with the weave net plugin. I did a tear down and reinstalled the plugin. I was then able to join the node. Thanks all for your suggestions.

Cannot curl kubelet read-only port

I have a heapster pod running on one of the nodes in my Kubernetes cluster. It is able to get http://<node-with-heapster-pod>:10255/stats/summary just fine, but whenever it runs the same get request on another node, it cannot. When I run curl from within any given node I can access that port, but when I curl any node from another machine I get the following error:
Failed to connect to 128.180.120.229 port 10255: No route to host
The following is the netstat output for all ports on which the kubelet is listening:
netstat -ap | grep -i "listen" | grep "kubelet"
tcp 0 0 localhost:10248 0.0.0.0:* LISTEN 7562/kubelet
tcp6 0 0 [::]:4194 [::]:* LISTEN 7562/kubelet
tcp6 0 0 [::]:10250 [::]:* LISTEN 7562/kubelet
tcp6 0 0 [::]:10255 [::]:* LISTEN 7562/kubelet
unix 2 [ ACC ] STREAM LISTENING 621349 7562/kubelet /var/run/dockershim.sock
I apologize for the messy last column. Any ideas why this may be? My iptables rules are set up to accept all incoming connections, and any node can ping port 10250 fine, just not 10255.
you may not have ip_forward enabled on your system. can you check this settings?
sysctl -n net.ipv4.ip_forward
If anybody still cares, port 10255 is the kubelet's read only port and may or may not be configured. You can confirm this by accessing the worker node in question then looking at the kubelet's startup command.
systemctl status kubelet-worker.service
Some on-prem kubernetes solutions set this to 0 as mentioned below
https://kubernetes.io/docs/reference/command-line-tools-reference/kubelet/
--read-only-port int32 The read-only port for the Kubelet to serve on with no authentication/authorization (set to 0 to disable) (default 10255) (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.)

Docker PostgreSQL service: could not bind IPv6 socket: Cannot assign requested address

I am running a PostgreSQL service with Docker. For some reason, PostgreSQL wants to bind to IPV6 - although I haven't specified that anywhere (at least as far as I'm aware).
As a result of this, I am unable to connect to PG. Relevant details follow below:
Dockerfile
FROM postgres:9.6
RUN apt-get update \
&& apt-get -y install apt-utils \
&& apt-get -y install python3 \
&& apt-get -y install postgresql-plpython3-9.6
COPY sql /docker-entrypoint-initdb.d/
EXPOSE 5432
# Add VOLUMEs to allow backup of config, logs and databases
VOLUME ["/etc/postgresql", "/var/log/postgresql", "/var/lib/postgresql"]
PostgreSQL log file contents
LOG: received fast shutdown request
LOG: aborting any active transactions
waiting for server to shut down....LOG: autovacuum launcher shutting down
LOG: shutting down
LOG: database system is shut down
done
server stopped
PostgreSQL init process complete; ready for start up.
LOG: could not bind IPv6 socket: Cannot assign requested address
HINT: Is another postmaster already running on port 5432? If not, wait a few seconds and retry.
LOG: database system was shut down at 2017-10-09 21:22:22 UTC
LOG: MultiXact member wraparound protections are now enabled
LOG: database system is ready to accept connections
LOG: autovacuum launcher started
I run the container using the following command: docker run --name my_db_service_cntnr image_tag
When I run the following command: docker container port my_db_service_cntnr, I get nothing returned:
me#yourbox:~/path/to/pgdb$ docker container port my_db_service_cntnr
me#yourbox:~/path/to/pgdb$
I know PostgreSQL is running in the container:
me#yourbox:~/path/to/pgdb$ docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
824ffe17c5b9 df:pg "docker-entrypoint..." 16 hours ago Up 5 minutes 5432/tcp my_db_service_cntnr
me#yourbox:/path/to/pgdb$ docker container inspect my_db_service_cntnr | grep Address
"LinkLocalIPv6Address": "",
"SecondaryIPAddresses": null,
"SecondaryIPv6Addresses": null,
"GlobalIPv6Address": "",
"IPAddress": "172.17.0.2",
"MacAddress": "02:42:ac:11:00:02",
"IPAddress": "172.17.0.2",
"GlobalIPv6Address": "",
"MacAddress": "02:42:ac:11:00:02"
Yet when I attempt to connect to PostgreSQL (using default port of 5432), it fails to connect to the database:
Attempting to connect via psql
me#yourbox:~/path/to/pgdb$ psql -h 172.17.0.2 -U postgres -p 5432
psql: could not connect to server: Connection refused
Is the server running on host "172.17.0.2" and accepting
TCP/IP connections on port 5432?
me#yourbox:~/path/to/pgdb$
It seems port 5432 is not being listened to on my machine, despite me specifying that the PG image EXPOSE port 5432:
me#yourbox:~/path/to/pgdb$ sudo lsof -i -P | grep -i "listen"
lighttpd 1477 www-data 4u IPv4 22342 0t0 TCP *:80 (LISTEN)
dnsmasq 1645 nobody 5u IPv4 26954 0t0 TCP CEBERUS:53 (LISTEN)
master 2182 root 12u IPv4 28720 0t0 TCP localhost:25 (LISTEN)
master 2182 root 13u IPv6 28721 0t0 TCP ip6-localhost:25 (LISTEN)
rhythmbox 3149 me 17u IPv4 33925 0t0 TCP *:3689 (LISTEN)
rhythmbox 3149 me 18u IPv6 33926 0t0 TCP *:3689 (LISTEN)
cupsd 8432 root 10u IPv6 87004 0t0 TCP ip6-localhost:631 (LISTEN)
cupsd 8432 root 11u IPv4 87005 0t0 TCP localhost:631 (LISTEN)
What is causing this error, and how do I resolve it?
This isn't a complete answer, but should get you closer to an answer. It covers the dockerism's that are required for your debug steps in the question.
Run a postgres container
$ CID=$(docker run -d postgres)
$ echo $CID
48024dc71aa446...
Get the PID of the container
$ PID=$(docker inspect -f {{.State.Pid}} $CID)
$ echo $PID
7994
Get the logs from the container, check for errors.
$ docker logs $CID
Process list from the container
$ docker exec -ti $CID ps -ef
UID PID PPID C STIME TTY TIME CMD
postgres 1 0 0 23:19 ? 00:00:00 postgres
postgres 49 1 0 23:19 ? 00:00:00 postgres: checkpointer process
postgres 50 1 0 23:19 ? 00:00:00 postgres: writer process
postgres 51 1 0 23:19 ? 00:00:00 postgres: wal writer process
postgres 52 1 0 23:19 ? 00:00:00 postgres: autovacuum launcher pr
postgres 53 1 0 23:19 ? 00:00:00 postgres: stats collector proces
postgres 54 1 0 23:19 ? 00:00:00 postgres: bgworker: logical repl
root 66 0 0 23:26 ? 00:00:00 ps -ef
Run ss in the container looking for listening tcp processes (like lsof)
$ docker exec -ti $CID ss -lntp
State Recv-Q Send-Q Local Address:Port Peer Address:Port
LISTEN 0 128 *:5432 *:*
LISTEN 0 128 :::5432 :::*
Outside the container won't report on ports in container namespaces
$ ss -lntp
State Recv-Q Send-Q Local Address:Port Peer Address:Port
LISTEN 0 128 *:22 *:* users:(("sshd",pid=592,fd=3))
LISTEN 0 128 :::22 :::* users:(("sshd",pid=592,fd=4))
From the host, you can use nsenter to enter the containers namespace and run commands.
$ nsenter -t $PID -n ss -lntp
State Recv-Q Send-Q Local Address:Port Peer Address:Port
LISTEN 0 128 *:5432 *:* users:(("postgres",pid=7994,fd=3))
LISTEN 0 128 :::5432 :::* users:(("postgres",pid=7994,fd=4))
$ nsenter -t $PID -n ip address show
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
52: eth0#if53: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default
link/ether 02:42:ac:11:00:03 brd ff:ff:ff:ff:ff:ff link-netnsid 0
inet 172.17.0.3/16 scope global eth0
valid_lft forever preferred_lft forever
Get the IP of the container from Docker
$ IP=$(docker inspect -f '{{.NetworkSettings.Networks.bridge.IPAddress}}' $CID)
$ echo $IP
172.17.0.3
Test the connection
$ psql -h $IP -U postgres -p 5432
Mapped ports
With a mapped port, the ports on the host changes slightly
$ CID=$(docker run -d -p 5432:5432 postgres)
$ echo $CID
020f72394fcd...
Now the container has a port configured
$ docker container port $CID
5432/tcp -> 0.0.0.0:5432
$ docker inspect -f {{.NetworkSettings.Ports}} $CID
map[5432/tcp:[{0.0.0.0 5432}]]
The port will also be listening on the host
$ ss -lntp
State Recv-Q Send-Q Local Address:Port Peer Address:Port
LISTEN 0 128 *:22 *:* users:(("sshd",pid=592,fd=3))
LISTEN 0 128 :::22 :::* users:(("sshd",pid=592,fd=4))
LISTEN 0 128 :::5432 :::* users:(("docker-proxy",pid=8571,fd=4))
And you can connect to localhost or your hosts IP now
$ psql -h 127.0.0.1 -U postgres -p 5432