I am trying to ping and access a k8s service from two different pods. However, I am able to access from one pod while from other pod it is inaccessible. What could be the reason for that ? And how can I fix?
--- service accessible from the pod
/home/ravi>sudo docker exec -it bc1104f73042 /bin/bash
root#kpimon-go:/opt# ps -eaf
UID PID PPID C STIME TTY TIME CMD
root 1 0 0 15:01 ? 00:00:00 ./kpimon
root 21 0 0 15:27 pts/0 00:00:00 /bin/bash
root 36 21 0 15:27 pts/0 00:00:00 ps -eaf
root#kpimon-go:/opt# cat /etc/resolv.conf
nameserver 10.96.0.10
search ricxapp.svc.cluster.local svc.cluster.local cluster.local
options ndots:5
root#kpimon-go:/opt# cat /etc/hosts
# Kubernetes-managed hosts file.
127.0.0.1 localhost
::1 localhost ip6-localhost ip6-loopback
fe00::0 ip6-localnet
fe00::0 ip6-mcastprefix
fe00::1 ip6-allnodes
fe00::2 ip6-allrouters
10.244.0.253 kpimon-go
root#kpimon-go:/opt# ping service-ricplt-e2mgr-http
ping: service-ricplt-e2mgr-http: Name or service not known ----------> unable to ping on k8s service
root#kpimon-go:/opt#
--- service inaccessible from the pod
/home/ravi>sudo docker exec -it 1df20eaae850 /bin/bash
root#rtmgr:/# cat /etc/hosts
# Kubernetes-managed hosts file.
127.0.0.1 localhost
::1 localhost ip6-localhost ip6-loopback
fe00::0 ip6-localnet
fe00::0 ip6-mcastprefix
fe00::1 ip6-allnodes
fe00::2 ip6-allrouters
10.244.0.250 rtmgr
root#rtmgr:/#
root#rtmgr:/# cat /etc/resolv.conf
nameserver 10.96.0.10
search ricplt.svc.cluster.local svc.cluster.local cluster.local
options ndots:5
root#rtmgr:/#
root#rtmgr:/#
root#rtmgr:/# ping service-ricplt-e2mgr-http ----------> unable to ping on k8s service
PING service-ricplt-e2mgr-http.ricplt.svc.cluster.local (10.101.210.88) 56(84) bytes of data.
64 bytes from service-ricplt-e2mgr-http.ricplt.svc.cluster.local (10.101.210.88): icmp_seq=1 ttl=64 time=0.059 ms
64 bytes from service-ricplt-e2mgr-http.ricplt.svc.cluster.local (10.101.210.88): icmp_seq=2 ttl=64 time=0.107 ms
^C
--- service-ricplt-e2mgr-http.ricplt.svc.cluster.local ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1025ms
rtt min/avg/max/mdev = 0.059/0.083/0.107/0.024 ms
root#rtmgr:/#
root#rtmgr:/#
-- info about k8s pods and services
/home/ravi>kubeclt get pods -A
NAMESPACE NAME READY STATUS RESTARTS AGE
kube-system coredns-5644d7b6d9-ngpd6 1/1 Running 13 9d
kube-system coredns-5644d7b6d9-pvzvz 1/1 Running 13 9d
kube-system etcd-ubuntu-20 1/1 Running 13 9d
kube-system kube-apiserver-ubuntu-20 1/1 Running 13 9d
kube-system kube-controller-manager-ubuntu-20 1/1 Running 13 9d
kube-system kube-flannel-ds-hrrml 1/1 Running 13 9d
kube-system kube-proxy-7hbwm 1/1 Running 13 9d
kube-system kube-scheduler-ubuntu-20 1/1 Running 13 9d
ricinfra deployment-tiller-ricxapp-68f777c4d4-lcmff 1/1 Running 1 4h22m
ricinfra tiller-secret-generator-vgkr9 0/1 Completed 0 4h22m
ricplt deployment-ricplt-a1mediator-669cc74647-mw7bb 1/1 Running 1 4h21m
ricplt deployment-ricplt-alarmmanager-57785458dd-ks5t2 1/1 Running 1 4h20m
ricplt deployment-ricplt-appmgr-77986c9cbb-h2mq8 1/1 Running 1 4h21m
ricplt deployment-ricplt-e2mgr-5dd878f58b-bhvss 1/1 Running 5 4h21m
ricplt deployment-ricplt-e2term-alpha-5d58997d8f-8kcrz 1/1 Running 1 4h21m
ricplt deployment-ricplt-o1mediator-5ddd66b4d6-h8nt8 1/1 Running 1 4h20m
ricplt deployment-ricplt-rtmgr-788975975b-kgxfj 1/1 Running 3 4h21m
ricplt deployment-ricplt-submgr-68fc656488-hcccs 1/1 Running 1 4h21m
ricplt deployment-ricplt-vespamgr-84f7d87dfb-mp76j 1/1 Running 1 4h20m
ricplt r4-influxdb-influxdb2-0 1/1 Running 1 4h12m
ricplt r4-infrastructure-kong-7995f4679b-brhdq 2/2 Running 6 4h22m
ricplt r4-infrastructure-prometheus-alertmanager-5798b78f48-hbn8k 2/2 Running 2 4h22m
ricplt r4-infrastructure-prometheus-server-c8ddcfdf5-vr8dk 1/1 Running 1 4h22m
ricplt statefulset-ricplt-dbaas-server-0 1/1 Running 1 4h21m
ricxapp ricxapp-kpimon-go-7b8bb97ffb-5x76c 1/1 Running 1 3h51m
/home/ravi>
/home/ravi>
/home/ravi>kubeclt get svc -A
NAMESPACE NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
default kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 9d
kube-system kube-dns ClusterIP 10.96.0.10 <none> 53/UDP,53/TCP,9153/TCP 9d
ricinfra service-tiller-ricxapp ClusterIP 10.98.94.194 <none> 44134/TCP 4h22m
ricplt aux-entry ClusterIP 10.105.149.143 <none> 80/TCP,443/TCP 4h22m
ricplt r4-influxdb-influxdb2 ClusterIP 10.110.14.243 <none> 80/TCP 4h13m
ricplt r4-infrastructure-kong-proxy NodePort 10.107.12.178 <none> 32080:32080/TCP,32443:32443/TCP 4h22m
ricplt r4-infrastructure-prometheus-alertmanager ClusterIP 10.104.86.76 <none> 80/TCP 4h22m
ricplt r4-infrastructure-prometheus-server ClusterIP 10.102.224.176 <none> 80/TCP 4h22m
ricplt service-ricplt-a1mediator-http ClusterIP 10.105.45.1 <none> 10000/TCP 4h21m
ricplt service-ricplt-a1mediator-rmr ClusterIP 10.108.188.147 <none> 4561/TCP,4562/TCP 4h21m
ricplt service-ricplt-alarmmanager-http ClusterIP 10.111.239.130 <none> 8080/TCP 4h20m
ricplt service-ricplt-alarmmanager-rmr ClusterIP 10.106.30.195 <none> 4560/TCP,4561/TCP 4h20m
ricplt service-ricplt-appmgr-http ClusterIP 10.110.110.91 <none> 8080/TCP 4h21m
ricplt service-ricplt-appmgr-rmr ClusterIP 10.110.96.28 <none> 4561/TCP,4560/TCP 4h21m
ricplt service-ricplt-dbaas-tcp ClusterIP None <none> 6379/TCP 4h22m
ricplt service-ricplt-e2mgr-http ClusterIP 10.101.210.88 <none> 3800/TCP 4h21m
ricplt service-ricplt-e2mgr-rmr ClusterIP 10.101.245.34 <none> 4561/TCP,3801/TCP 4h21m
ricplt service-ricplt-e2term-prometheus-alpha ClusterIP 10.97.95.213 <none> 8088/TCP 4h21m
ricplt service-ricplt-e2term-rmr-alpha ClusterIP 10.100.36.142 <none> 4561/TCP,38000/TCP 4h21m
ricplt service-ricplt-e2term-sctp-alpha NodePort 10.108.215.136 <none> 36422:32222/SCTP 4h21m
ricplt service-ricplt-o1mediator-http ClusterIP 10.96.196.67 <none> 9001/TCP,8080/TCP,3000/TCP 4h20m
ricplt service-ricplt-o1mediator-tcp-netconf NodePort 10.104.237.252 <none> 830:30830/TCP 4h20m
ricplt service-ricplt-rtmgr-http ClusterIP 10.105.27.42 <none> 3800/TCP 4h21m
ricplt service-ricplt-rtmgr-rmr ClusterIP 10.110.0.158 <none> 4561/TCP,4560/TCP 4h21m
ricplt service-ricplt-submgr-http ClusterIP None <none> 3800/TCP 4h21m
ricplt service-ricplt-submgr-rmr ClusterIP None <none> 4560/TCP,4561/TCP 4h21m
ricplt service-ricplt-vespamgr-http ClusterIP 10.98.139.191 <none> 8080/TCP,9095/TCP 4h21m
ricxapp aux-entry ClusterIP 10.99.152.66 <none> 80/TCP,443/TCP 4h22m
ricxapp service-ricxapp-kpimon-go-http ClusterIP 10.109.170.194 <none> 8080/TCP 3h51m
ricxapp service-ricxapp-kpimon-go-rmr ClusterIP 10.111.137.11 <none> 4560/TCP,4561/TCP 3h51m
/home/ravi>
Does the inaccessible pod's namespace same with service? If not, try to
ping service-ricplt-e2mgr-http.<namespace>
You are not able to ping the service in K8s.
Check whether you are using the same namespace to ping the pod. You can also use curl or wget
#curl <servicename>:Port number
#wget -O- IP address:Port number
For additional info refer to the link Ways to connect
Related
k3s - can't access from one pod to another if pods on different nodes
Update:
I've narrowed the issue down - it's pods that are on other master nodes that can't communicate with those on the original master
pods on rpi4-server1 - the original cluster - can communicate with pods on rpi-worker01 and rpi3-worker02
pods on rpi4-server2 are unable to communicate with the others
I'm trying to run a HighAvailability cluster with embedded DB and using flannel / vxlan
I'm trying to setup a project with 5 services in k3s
When all of the pods are contained on a single node, they work together fine.
As soon as I add other nodes into the system and pods are deployed to them, the links seem to break.
In troubleshooting I've exec'd into one of the pods and tried to curl another. When they are on the same node this works, if the second service is on another node it doesn't.
I'm sure this is something simple that I'm missing, but I can't work it out! Help appreciated.
Key details:
Using k3s and native traefik
Two rpi4s as servers (High Availability) and two rpi3s as worker nodes
metallb as loadbalancer
Two services - blah-interface and blah-svc are configured as LoadBalancer to allow external access. The others blah-server, n34 and test-apisas NodePort to support debugging, but only really need internal access
Info on nodes, pods and services....
pi#rpi4-server1:~/Projects/test_demo_2020/test_kube_config/testchart/templates $ sudo kubectl get nodes --all-namespaces -o wide
NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME
rpi4-server1 Ready master 11h v1.17.0+k3s.1 192.168.0.140 <none> Raspbian GNU/Linux 10 (buster) 4.19.75-v7l+ docker://19.3.5
rpi-worker01 Ready,SchedulingDisabled <none> 10h v1.17.0+k3s.1 192.168.0.41 <none> Raspbian GNU/Linux 10 (buster) 4.19.66-v7+ containerd://1.3.0-k3s.5
rpi3-worker02 Ready,SchedulingDisabled <none> 10h v1.17.0+k3s.1 192.168.0.142 <none> Raspbian GNU/Linux 10 (buster) 4.19.75-v7+ containerd://1.3.0-k3s.5
rpi4-server2 Ready master 10h v1.17.0+k3s.1 192.168.0.143 <none> Raspbian GNU/Linux 10 (buster) 4.19.75-v7l+ docker://19.3.5
pi#rpi4-server1:~/Projects/test_demo_2020/test_kube_config/testchart/templates $ sudo kubectl get pods --all-namespaces -o wide
NAMESPACE NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
kube-system helm-install-traefik-l2z6l 0/1 Completed 2 11h 10.42.0.2 rpi4-server1 <none> <none>
test-demo n34-5c7b9475cb-zjlgl 1/1 Running 1 4h30m 10.42.0.32 rpi4-server1 <none> <none>
kube-system metrics-server-6d684c7b5-5wgf9 1/1 Running 3 11h 10.42.0.26 rpi4-server1 <none> <none>
metallb-system speaker-62rkm 0/1 Pending 0 99m <none> rpi-worker01 <none> <none>
metallb-system speaker-2shzq 0/1 Pending 0 99m <none> rpi3-worker02 <none> <none>
metallb-system speaker-2mcnt 1/1 Running 0 99m 192.168.0.143 rpi4-server2 <none> <none>
metallb-system speaker-v8j9g 1/1 Running 0 99m 192.168.0.140 rpi4-server1 <none> <none>
metallb-system controller-65895b47d4-pgcs6 1/1 Running 0 90m 10.42.0.49 rpi4-server1 <none> <none>
test-demo blah-server-858ccd7788-mnf67 1/1 Running 0 64m 10.42.0.50 rpi4-server1 <none> <none>
default nginx2-6f4f6f76fc-n2kbq 1/1 Running 0 22m 10.42.0.52 rpi4-server1 <none> <none>
test-demo blah-interface-587fc66bf9-qftv6 1/1 Running 0 22m 10.42.0.53 rpi4-server1 <none> <none>
test-demo blah-svc-6f8f68f46-gqcbw 1/1 Running 0 21m 10.42.0.54 rpi4-server1 <none> <none>
kube-system coredns-d798c9dd-hdwn5 1/1 Running 1 11h 10.42.0.27 rpi4-server1 <none> <none>
kube-system local-path-provisioner-58fb86bdfd-tjh7r 1/1 Running 31 11h 10.42.0.28 rpi4-server1 <none> <none>
kube-system traefik-6787cddb4b-tgq6j 1/1 Running 0 4h50m 10.42.1.23 rpi4-server2 <none> <none>
default testdemo2020-testchart-6f8d44b496-2hcfc 1/1 Running 1 6h31m 10.42.0.29 rpi4-server1 <none> <none>
test-demo test-apis-75bb68dcd7-d8rrp 1/1 Running 0 7m13s 10.42.1.29 rpi4-server2 <none> <none>
pi#rpi4-server1:~/Projects/test_demo_2020/test_kube_config/testchart/templates $ sudo kubectl get svc --all-namespaces -o wide
NAMESPACE NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE SELECTOR
default kubernetes ClusterIP 10.43.0.1 <none> 443/TCP 11h <none>
kube-system kube-dns ClusterIP 10.43.0.10 <none> 53/UDP,53/TCP,9153/TCP 11h k8s-app=kube-dns
kube-system metrics-server ClusterIP 10.43.74.118 <none> 443/TCP 11h k8s-app=metrics-server
kube-system traefik-prometheus ClusterIP 10.43.78.135 <none> 9100/TCP 11h app=traefik,release=traefik
test-demo blah-server NodePort 10.43.224.128 <none> 5055:31211/TCP 10h io.kompose.service=blah-server
default testdemo2020-testchart ClusterIP 10.43.91.7 <none> 80/TCP 10h app.kubernetes.io/instance=testdemo2020,app.kubernetes.io/name=testchart
test-demo traf-dashboard NodePort 10.43.60.155 <none> 8080:30808/TCP 10h io.kompose.service=traf-dashboard
test-demo test-apis NodePort 10.43.248.59 <none> 8075:31423/TCP 7h11m io.kompose.service=test-apis
kube-system traefik LoadBalancer 10.43.168.18 192.168.0.240 80:30688/TCP,443:31263/TCP 11h app=traefik,release=traefik
default nginx2 LoadBalancer 10.43.249.123 192.168.0.241 80:30497/TCP 92m app=nginx2
test-demo n34 NodePort 10.43.171.206 <none> 7474:30474/TCP,7687:32051/TCP 72m io.kompose.service=n34
test-demo blah-interface LoadBalancer 10.43.149.158 192.168.0.242 80:30634/TCP 66m io.kompose.service=blah-interface
test-demo blah-svc LoadBalancer 10.43.19.242 192.168.0.243 5005:30005/TCP,5006:31904/TCP,5002:30685/TCP 51m io.kompose.service=blah-svc
Hi you issue could be related to the following issue.
After configuring the network under /etc/systemd/network/eth0.network (filename may differ in your case, since i am using arch linux on all pis)
[Match]
Name=eth0
[Network]
Address=x.x.x.x/24 # ip of node
Gateway=x.x.x.x # ip of gateway router
Domains=default.svc.cluster.local svc.cluster.local cluster.local
DNS=10.x.x.x # k3s dns ip x.x.x.x # ip of gateway router
After that I removed the 10.x.x.x routes with ip route del 10.x.x.x dev [flannel|cni0] on every node and restarted them.
I have previously setup kubernetes clusters in dev environments, using private servers without any issues.
Now i created a new cluster in a datacenter (hetzner)
I been trying to get everything working for several days now, reinstalling the servers many times, facing the same issues every time.
Most of my services seem to have network issues, for example the dashboard, dockerreg ui, ... cannot access the resources loaded by the web interfaces. Even pushing a container to the private dockerreg start but stops and timeout after few seconds.
If i configure any of the services with issues to the node port they work find.
So this is probably an issue with the kube-proxy.
All of my servers (3x master node and 2x worker node) have a public and private ip address. when i get a list of pods, all thoses that are running on the host ip, use the external ip instead of the internal ip.
How can i bind these to use the internal ip only?
kubectl get pods -o wide -n kube-system
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
calico-kube-controllers-65b8787765-zj728 1/1 Running 2 12h 192.168.57.14 k8s-master-001 <none> <none>
calico-node-cxn2p 1/1 Running 1 12h <external ip> k8s-master-003 <none> <none>
calico-node-k9g7n 1/1 Running 1 12h <external ip> k8s-master-002 <none> <none>
calico-node-mt8r7 1/1 Running 2 12h <external ip> k8s-master-001 <none> <none>
calico-node-pww9q 1/1 Running 1 12h <external ip> k8s-worker-002 <none> <none>
calico-node-wlg8g 1/1 Running 2 12h <external ip> k8s-worker-001 <none> <none>
coredns-5c98db65d4-lrzj8 1/1 Running 0 12h 192.168.20.1 k8s-worker-002 <none> <none>
coredns-5c98db65d4-s6tzv 1/1 Running 1 12h 192.168.102.17 k8s-worker-001 <none> <none>
etcd-k8s-master-001 1/1 Running 2 12h <external ip> k8s-master-001 <none> <none>
etcd-k8s-master-002 1/1 Running 1 12h <external ip> k8s-master-002 <none> <none>
etcd-k8s-master-003 1/1 Running 1 12h <external ip> k8s-master-003 <none> <none>
kube-apiserver-k8s-master-001 1/1 Running 2 12h <external ip> k8s-master-001 <none> <none>
kube-apiserver-k8s-master-002 1/1 Running 2 12h <external ip> k8s-master-002 <none> <none>
kube-apiserver-k8s-master-003 1/1 Running 1 12h <external ip> k8s-master-003 <none> <none>
kube-controller-manager-k8s-master-001 1/1 Running 3 12h <external ip> k8s-master-001 <none> <none>
kube-controller-manager-k8s-master-002 1/1 Running 1 12h <external ip> k8s-master-002 <none> <none>
kube-controller-manager-k8s-master-003 1/1 Running 1 12h <external ip> k8s-master-003 <none> <none>
kube-proxy-mlsnp 1/1 Running 1 12h <external ip> k8s-master-003 <none> <none>
kube-proxy-mzck9 1/1 Running 2 12h <external ip> k8s-worker-001 <none> <none>
kube-proxy-p7vfz 1/1 Running 1 12h <external ip> k8s-master-002 <none> <none>
kube-proxy-s55fr 1/1 Running 2 12h <external ip> k8s-master-001 <none> <none>
kube-proxy-tz6zn 1/1 Running 1 12h <external ip> k8s-worker-002 <none> <none>
kube-scheduler-k8s-master-001 1/1 Running 3 12h <external ip> k8s-master-001 <none> <none>
kube-scheduler-k8s-master-002 1/1 Running 1 12h <external ip> k8s-master-002 <none> <none>
kube-scheduler-k8s-master-003 1/1 Running 1 12h <external ip> k8s-master-003 <none> <none>
traefik-ingress-controller-gxthm 1/1 Running 1 35m 192.168.57.15 k8s-master-001 <none> <none>
traefik-ingress-controller-rdv8j 1/1 Running 0 35m 192.168.160.133 k8s-master-003 <none> <none>
traefik-ingress-controller-w4t4t 1/1 Running 0 35m 192.168.1.133 k8s-master-002 <none> <none>
im running kubernetes 1.15.3, using CRIO and calico.
all servers are on the 10.0.0.0/24 subnet
I expect the pods running on the node ip, to use the interanal ip instead of the external ip
--- Edit 16/09/2019
The cluster is initialized using the following command
sudo kubeadm init --config=kubeadm-config.yaml --upload-certs
My kubeadm-config.yaml
apiVersion: kubeadm.k8s.io/v1beta2
kind: InitConfiguration
localAPIEndpoint:
advertiseAddress: "10.0.0.2"
bindPort: 6443
---
apiVersion: kubeadm.k8s.io/v1beta2
kind: ClusterConfiguration
kubernetesVersion: stable
controlPlaneEndpoint: "10.0.0.200:6443"
apiServer:
certSANs:
- "k8s.deb-ict.com"
networking:
serviceSubnet: "10.96.0.0/12"
podSubnet: "192.168.0.0/16"
I am deploying statefulset in my local PC (for doing research) follow this link
In this step:
kubectl run -i --tty --image busybox:1.28 dns-test --restart=Never --rm
nslookup web-0.nginx
I meet this error:
nslookup web-0.nginx
Server: 10.96.0.10
Address 1: 10.96.0.10
nslookup: can't resolve 'web-0.nginx'
My pod and node are still working correctly and my coredns is running correctly
kube-system coredns-fb8b8dccf-hbrhw 1/1 Running 0 26m
kube-system coredns-fb8b8dccf-rmrwp 1/1 Running 0 26m
nguyen#kmaster:~/Documents$ kubectl get --all-namespaces=true -o wide pods
NAMESPACE NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
default busybox 1/1 Running 1 65m 10.244.1.218 knode <none> <none>
default web-0 1/1 Running 0 75m 10.244.1.215 knode <none> <none>
default web-1 1/1 Running 0 75m 10.244.1.216 knode <none> <none>
kube-system coredns-fb8b8dccf-hbrhw 1/1 Running 0 51m 10.244.1.219 knode <none> <none>
kube-system coredns-fb8b8dccf-rmrwp 1/1 Running 0 51m 10.244.0.37 kmaster <none> <none>
kube-system etcd-kmaster 1/1 Running 20 20d 192.168.146.132 kmaster <none> <none>
kube-system kube-apiserver-kmaster 1/1 Running 514 20d 192.168.146.132 kmaster <none> <none>
kube-system kube-controller-manager-kmaster 1/1 Running 144 20d 192.168.146.132 kmaster <none> <none>
kube-system kube-flannel-ds-amd64-ndpjq 1/1 Running 0 76m 192.168.146.129 knode <none> <none>
kube-system kube-flannel-ds-amd64-s2vhp 1/1 Running 0 76m 192.168.146.132 kmaster <none> <none>
kube-system kube-proxy-dk5jd 1/1 Running 6 20d 192.168.146.132 kmaster <none> <none>
kube-system kube-proxy-ts79l 1/1 Running 2 20d 192.168.146.129 knode <none> <none>
kube-system kube-scheduler-kmaster 1/1 Running 172 20d 192.168.146.132 kmaster <none> <none>
nguyen#kmaster:~$ kubectl get svc
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 21d
nginx ClusterIP None <none> 80/TCP 6h8m
Did I miss something? Someone can help me.
Thank you!
nginx statefulset is deployed in default namespace as shown below
default web-0 1/1 Running 0 75m 10.244.1.215 knode <none> <none>
default web-1 1/1 Running 0 75m 10.244.1.216 knode <none> <none>
This is how you should test
master $ kubectl get po
NAME READY STATUS RESTARTS AGE
web-0 1/1 Running 0 1m
web-1 1/1 Running 0 1m
master $ kubectl get svc
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 35m
nginx ClusterIP None <none> 80/TCP 2m
master $ kubectl run -i --tty --image busybox:1.28 dns-test --restart=Never --rm
If you don't see a command prompt, try pressing enter.
/ # nslookup nginx
Server: 10.96.0.10
Address 1: 10.96.0.10 kube-dns.kube-system.svc.cluster.local
Name: nginx
Address 1: 10.40.0.1 web-0.nginx.default.svc.cluster.local
Address 2: 10.40.0.2 web-1.nginx.default.svc.cluster.local
/ #
/ # nslookup web-0.nginx
Server: 10.96.0.10
Address 1: 10.96.0.10 kube-dns.kube-system.svc.cluster.local
Name: web-0.nginx
Address 1: 10.40.0.1 web-0.nginx.default.svc.cluster.local
/ # nslookup web-0.nginx.default.svc.cluster.local
Server: 10.96.0.10
Address 1: 10.96.0.10 kube-dns.kube-system.svc.cluster.local
Name: web-0.nginx.default.svc.cluster.local
Address 1: 10.40.0.1 web-0.nginx.default.svc.cluster.local
I have a K8s cluster (1 master, 2 workers) running on 3 vagrant VMs on my computer.
I've installed kubernetes dashboard, like explained here.
All my pods are running correctly:
kubectl get pods -o wide --namespace=kube-system
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
coredns-fb8b8dccf-n5cpm 1/1 Running 1 61m 10.244.0.4 kmaster.example.com <none> <none>
coredns-fb8b8dccf-qwcr4 1/1 Running 1 61m 10.244.0.5 kmaster.example.com <none> <none>
etcd-kmaster.example.com 1/1 Running 1 60m 172.42.42.100 kmaster.example.com <none> <none>
kube-apiserver-kmaster.example.com 1/1 Running 1 60m 172.42.42.100 kmaster.example.com <none> <none>
kube-controller-manager-kmaster.example.com 1/1 Running 1 60m 172.42.42.100 kmaster.example.com <none> <none>
kube-flannel-ds-amd64-hcjsm 1/1 Running 1 61m 172.42.42.100 kmaster.example.com <none> <none>
kube-flannel-ds-amd64-klv4f 1/1 Running 3 56m 172.42.42.102 kworker2.example.com <none> <none>
kube-flannel-ds-amd64-lmpnd 1/1 Running 2 59m 172.42.42.101 kworker1.example.com <none> <none>
kube-proxy-86qsw 1/1 Running 1 59m 10.0.2.15 kworker1.example.com <none> <none>
kube-proxy-dp29s 1/1 Running 1 61m 172.42.42.100 kmaster.example.com <none> <none>
kube-proxy-gqqq9 1/1 Running 1 56m 10.0.2.15 kworker2.example.com <none> <none>
kube-scheduler-kmaster.example.com 1/1 Running 1 60m 172.42.42.100 kmaster.example.com <none> <none>
kubernetes-dashboard-5f7b999d65-zqbbz 1/1 Running 1 28m 10.244.1.3 kworker1.example.com <none> <none>
As you can see the dashboard is in "Running" status.
I also ran kubectl proxy and it's serving on 127.0.0.1:8001.
But when I try to open http://127.0.0.1:8001/api/v1/namespaces/kube-system/services/https:kubernetes-dashboard:/proxy/ I have the error:
This site can’t be reached
127.0.0.1 refused to connect.
ERR_CONNECTION_REFUSED
I'm trying to open the dashboard directly on my computer, not inside the vagram VM. Could that be the problem? If yes, how to solve it ? I'm able to ping my VM from my computer without any issue.
Thanks for helping me.
EDIT
Here is the ouput of kubectl get svc -n kube-system
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
kube-dns ClusterIP 10.96.0.10 <none> 53/UDP,53/TCP,9153/TCP 96m
kubernetes-dashboard NodePort 10.109.230.83 <none> 443:30089/TCP 63m
Kubernetes dashboard runs only in the cluster as default. You can control it with get svc command:
kubectl get svc -n kube-system
Default type of that service is ClusterIp, to reach from outside of the cluster yo have to change it to NodePort.
To change it follow this doc.
Created a local cluster using Vagrant + Ansible + VirtualBox. Manually deploying works fine, but when using Helm:
:~$helm install stable/nginx-ingress --name nginx-ingress-controller --set rbac.create=true
Error: forwarding ports: error upgrading connection: error dialing backend: dial tcp 10.0.52.15:10250: i/o timeout
Kubernetes cluster info:
:~$kubectl get nodes,po,deploy,svc,ingress --all-namespaces -o wide
NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME
node/ubuntu18-kube-master Ready master 32m v1.13.3 10.0.51.15 <none> Ubuntu 18.04.1 LTS 4.15.0-43-generic docker://18.6.1
node/ubuntu18-kube-node-1 Ready <none> 31m v1.13.3 10.0.52.15 <none> Ubuntu 18.04.1 LTS 4.15.0-43-generic docker://18.6.1
NAMESPACE NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
default pod/nginx-server 1/1 Running 0 40s 10.244.1.5 ubuntu18-kube-node-1 <none> <none>
default pod/nginx-server-b8d78876d-cgbjt 1/1 Running 0 4m25s 10.244.1.4 ubuntu18-kube-node-1 <none> <none>
kube-system pod/coredns-86c58d9df4-5rsw2 1/1 Running 0 31m 10.244.0.2 ubuntu18-kube-master <none> <none>
kube-system pod/coredns-86c58d9df4-lfbvd 1/1 Running 0 31m 10.244.0.3 ubuntu18-kube-master <none> <none>
kube-system pod/etcd-ubuntu18-kube-master 1/1 Running 0 31m 10.0.51.15 ubuntu18-kube-master <none> <none>
kube-system pod/kube-apiserver-ubuntu18-kube-master 1/1 Running 0 30m 10.0.51.15 ubuntu18-kube-master <none> <none>
kube-system pod/kube-controller-manager-ubuntu18-kube-master 1/1 Running 0 30m 10.0.51.15 ubuntu18-kube-master <none> <none>
kube-system pod/kube-flannel-ds-amd64-jffqn 1/1 Running 0 31m 10.0.51.15 ubuntu18-kube-master <none> <none>
kube-system pod/kube-flannel-ds-amd64-vc6p2 1/1 Running 0 31m 10.0.52.15 ubuntu18-kube-node-1 <none> <none>
kube-system pod/kube-proxy-fbgmf 1/1 Running 0 31m 10.0.52.15 ubuntu18-kube-node-1 <none> <none>
kube-system pod/kube-proxy-jhs6b 1/1 Running 0 31m 10.0.51.15 ubuntu18-kube-master <none> <none>
kube-system pod/kube-scheduler-ubuntu18-kube-master 1/1 Running 0 31m 10.0.51.15 ubuntu18-kube-master <none> <none>
kube-system pod/tiller-deploy-69ffbf64bc-x8lkc 1/1 Running 0 24m 10.244.1.2 ubuntu18-kube-node-1 <none> <none>
NAMESPACE NAME READY UP-TO-DATE AVAILABLE AGE CONTAINERS IMAGES SELECTOR
default deployment.extensions/nginx-server 1/1 1 1 4m25s nginx-server nginx run=nginx-server
kube-system deployment.extensions/coredns 2/2 2 2 32m coredns k8s.gcr.io/coredns:1.2.6 k8s-app=kube-dns
kube-system deployment.extensions/tiller-deploy 1/1 1 1 24m tiller gcr.io/kubernetes-helm/tiller:v2.12.3 app=helm,name=tiller
NAMESPACE NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE SELECTOR
default service/kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 32m <none>
default service/nginx-server NodePort 10.99.84.201 <none> 80:31811/TCP 12s run=nginx-server
kube-system service/kube-dns ClusterIP 10.96.0.10 <none> 53/UDP,53/TCP 32m k8s-app=kube-dns
kube-system service/tiller-deploy ClusterIP 10.99.4.74 <none> 44134/TCP 24m app=helm,name=tiller
Vagrantfile:
...
Vagrant.configure(VAGRANTFILE_API_VERSION) do |config|
$hosts.each_with_index do |(hostname, parameters), index|
ip_address = "#{$subnet}.#{$ip_offset + index}"
config.vm.define vm_name = hostname do |vm_config|
vm_config.vm.hostname = hostname
vm_config.vm.box = box
vm_config.vm.network "private_network", ip: ip_address
vm_config.vm.provider :virtualbox do |vb|
vb.gui = false
vb.name = hostname
vb.memory = parameters[:memory]
vb.cpus = parameters[:cpus]
vb.customize ['modifyvm', :id, '--macaddress1', "08002700005#{index}"]
vb.customize ['modifyvm', :id, '--natnet1', "10.0.5#{index}.0/24"]
end
end
end
end
Workaround for VirtualBox issue: set diffenrent macaddress and internal_ip.
It is interesting to find a solution that can be placed in one of the configuration files: vagrant, ansible roles. Any ideas on the problem?
Error: forwarding ports: error upgrading connection: error dialing backend: dial tcp 10.0.52.15:10250: i/o timeout
You're getting bitten by a very common kubernetes-on-Vagrant bug: the kubelet believes its IP address is eth0, which is the NAT interface in Vagrant, versus using (what I hope you have) the :private_address network in your Vagrantfile. Thus, since all kubelet interactions happen directly to it (and not through the API server), things like kubectl exec and kubectl logs will fail in exactly the way you see.
The solution is to force kubelet to bind to the private network interface, or I guess you could switch your Vagrantfile to use the bridge network, if that's an option for you -- just so long as the interface isn't the NAT one.
The question is about how you manage TLS Certificates in the cluster, ensure that port 10250 is reachable.
Here is an example of how i fix it when i try to run exec a pod running in node (instance aws in my case),
resource "aws_security_group" "My_VPC_Security_Group" {
...
ingress {
description = "TLS from VPC"
from_port = 10250
to_port = 10250
protocol = "tcp"
cidr_blocks = ["0.0.0.0/0"]
}
}
For more details you can visit [1]: http://carnal0wnage.attackresearch.com/2019/01/kubernetes-unauth-kublet-api-10250.html