When I using dig command to test the CoreDNS server,it shows: connection timed out; no servers could be reached:
[root#ops001 ~]# /opt/k8s/bin/kubectl exec -ti soa-user-service-5c8b744d6d-7p9hr -n dabai-fat /bin/sh
/ # dig -t A kubernetes.default.svc.cluster.local. #10.254.0.2
; <<>> DiG 9.12.4-P2 <<>> -t A kubernetes.default.svc.cluster.local. #10.254.0.2
;; global options: +cmd
;; connection timed out; no servers could be reached
when I ping server,it success.
[root#ops001 ~]# /opt/k8s/bin/kubectl exec -ti soa-user-service-5c8b744d6d-7p9hr -n dabai-fat /bin/sh
/ # ping 10.254.0.2
PING 10.254.0.2 (10.254.0.2): 56 data bytes
64 bytes from 10.254.0.2: seq=0 ttl=64 time=0.100 ms
64 bytes from 10.254.0.2: seq=1 ttl=64 time=0.071 ms
64 bytes from 10.254.0.2: seq=2 ttl=64 time=0.094 ms
64 bytes from 10.254.0.2: seq=3 ttl=64 time=0.087 ms
why the dig could not connect to DNS server althrough the network is ok?This is my CoreDNS service:
when azshara-k8s03‘s node connection to CoreDNS server:
/ # telnet 10.254.0.2 53
Connection closed by foreign host
when azshara-k8s02‘s and azshara-k8s01‘s node connection to CoreDNS server:
/ # telnet 10.254.0.2 53
telnet: can't connect to remote host (10.254.0.2): Connection refused
I just confusing why port 53 is not open,when I scan from host using same command,the port 53 is open:
I finally find some server's kube-proxy not start,and the route foward rule not refresh,using this command to start kube-proxy fix this problem:
systemctl start kube-proxy
Related
Running a clean install of microk8s 1.19 on Fedora 32, I am able to ping an external IP address, but when I try to wget, I get "no route to host" (this is the output of commands run from a busybox pod):
/ # wget x.x.x.x
Connecting to x.x.x.x (x.x.x.x:80)
wget: can't connect to remote host (x.x.x.x): No route to host
/ # ping x.x.x.x
PING x.x.x.x (x.x.x.x): 56 data bytes
64 bytes from x.x.x.x: seq=0 ttl=127 time=1.209 ms
64 bytes from x.x.x.x: seq=1 ttl=127 time=0.765 ms
Finally found https://github.com/ubuntu/microk8s/issues/408
Had to enable masquerade in the firewall zone associated with the bridge interface, or in my case, my ethernet connection.
when I using apt-get command to update the kubernetes(v1.15.2) pods, it failed:
root#nginx-deployment-5754944d6c-7gbds:/# apt-get update
Err http://security.debian.org wheezy/updates Release.gpg
Temporary failure resolving 'security.debian.org'
Err http://http.debian.net wheezy Release.gpg
Temporary failure resolving 'http.debian.net'
Err http://http.debian.net wheezy-updates Release.gpg
Temporary failure resolving 'http.debian.net'
Err http://nginx.org wheezy Release.gpg
Temporary failure resolving 'nginx.org'
Reading package lists... Done
W: Failed to fetch http://http.debian.net/debian/dists/wheezy/Release.gpg Temporary failure resolving 'http.debian.net'
W: Failed to fetch http://http.debian.net/debian/dists/wheezy-updates/Release.gpg Temporary failure resolving 'http.debian.net'
W: Failed to fetch http://security.debian.org/dists/wheezy/updates/Release.gpg Temporary failure resolving 'security.debian.org'
W: Failed to fetch http://nginx.org/packages/mainline/debian/dists/wheezy/Release.gpg Temporary failure resolving 'nginx.org'
W: Some index files failed to download. They have been ignored, or old ones used instead.
now I could ping success to my kube-dns(ip:10.96.0.10,coredns version 1.6.7):
root#nginx-deployment-5754944d6c-7gbds:/# cat /etc/resolv.conf
nameserver 10.96.0.10
search default.svc.cluster.local svc.cluster.local cluster.local
options ndots:5
root#nginx-deployment-5754944d6c-7gbds:/# ping 10.96.21.92
PING 10.96.21.92 (10.96.21.92): 48 data bytes
^C--- 10.96.21.92 ping statistics ---
11 packets transmitted, 0 packets received, 100% packet loss
root#nginx-deployment-5754944d6c-7gbds:/# ping 10.96.0.10
PING 10.96.0.10 (10.96.0.10): 48 data bytes
56 bytes from 10.96.0.10: icmp_seq=0 ttl=64 time=0.103 ms
56 bytes from 10.96.0.10: icmp_seq=1 ttl=64 time=0.094 ms
56 bytes from 10.96.0.10: icmp_seq=2 ttl=64 time=0.068 ms
56 bytes from 10.96.0.10: icmp_seq=3 ttl=64 time=0.066 ms
56 bytes from 10.96.0.10: icmp_seq=4 ttl=64 time=0.060 ms
56 bytes from 10.96.0.10: icmp_seq=5 ttl=64 time=0.064 ms
why the pods could not access to the network? Now I could not install any tool to check the pods network problem in this pod. What should I do to find out where is going wrong?
I tried to create a busybox and test the kube-dns like this:
[miao#MeowK8SMaster1 ~]$ kubectl exec -it busybox -- nslookup kubernetes
Server: 10.96.0.10
Address 1: 10.96.0.10
nslookup: can't resolve 'kubernetes'
command terminated with exit code 1
I understand why you need to exec into coredns pod.
It, however, allows only coredns binary to be executed (not any shell).
E.g:
k exec -it <<coredns podname>> -n kube-system -- ./coredns -version
This will returns the version of the coredns binary that is running.
I am trying to run kubectl commands from inside pod to communicate with cluster and to delete pod with --graceperiod=0 through a monitoring script after holding the pod in delete state through extended grace period to hold the pod deletion with prestop hook. But was not able to connect to cluster IP and neither able to ping the pod itself.
[root#pod01 /]# kubectl exec dnsutils cat /etc/resolv.conf
Unable to connect to the server: dial tcp 196.19.0.1:443: connect: network is unreachable
[root#pod01 /]# cat /etc/resolv.conf
nameserver 196.19.0.2
search namespace.svc.cluster.local svc.cluster.local cluster.local
options ndots:5
[root#pod01 /]# kubectl exec -ti dnsutils -- nslookup kubernetes.default
Unable to connect to the server: dial tcp 196.19.0.1:443: connect: network is unreachable
[root#pod01 /]# ping namespace.svc.cluster.local
ping: namespace.svc.cluster.local: Name or service not known
[root#pod01 /]# ^C
[root#pod01 /]# nslookup Cluster_IP
bash: nslookup: command not found
[root#pod01 /]# ping Cluster_IP port
connect: Network is unreachable
[root#pod01 /]# cat /etc/hosts
# Kubernetes-managed hosts file.
127.0.0.1 localhost
::1 localhost ip6-localhost ip6-loopback
fe00::0 ip6-localnet
fe00::0 ip6-mcastprefix
fe00::1 ip6-allnodes
fe00::2 ip6-allrouters
196.18.9.8 pod01
[root#pod01 /]# ping 196.18.9.8
connect: Network is unreachable
[root#pod01 /]# nslookup 196.18.9.8
bash: nslookup: command not found
[root#pod01 /]# ping localhost
PING localhost (127.0.0.1) 56(84) bytes of data.
64 bytes from localhost (127.0.0.1): icmp_seq=1 ttl=64 time=0.014 ms
64 bytes from localhost (127.0.0.1): icmp_seq=2 ttl=64 time=0.013 ms
64 bytes from localhost (127.0.0.1): icmp_seq=3 ttl=64 time=0.015 ms
64 bytes from localhost (127.0.0.1): icmp_seq=4 ttl=64 time=0.015 ms
^C
--- localhost ping statistics ---
4 packets transmitted, 4 received, 0% packet loss, time 3065ms
any help.? I am not able to run any of these to get the pod interact outside and I am on CentOS 7
The network components are down in a pod with terminated state and won't be able to communicate with cluster.
I am not able to access kubernetes service on localhost:nodePort but can access the same on 127.0.0.1:nodePort. Also i can access service from browser like masterip:nodePort
below is output
[root#k8s-master ~]# kubectl get services
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 4h26m
nginx NodePort 10.109.106.21 <none> 80:30893/TCP 7m24s
[root#k8s-master ~]# curl -I 127.0.0.1:30893
HTTP/1.1 200 OK
Server: nginx/1.17.5
Date: Wed, 13 Nov 2019 15:48:44 GMT
Content-Type: text/html
Content-Length: 612
Last-Modified: Tue, 22 Oct 2019 14:30:00 GMT
Connection: keep-alive
ETag: "5daf1268-264"
Accept-Ranges: bytes
[root#k8s-master ~]# curl -I localhost:30893
^C
[root#k8s-master ~]# ping localhost
PING localhost (127.0.0.1) 56(84) bytes of data.
64 bytes from localhost (127.0.0.1): icmp_seq=1 ttl=64 time=0.094 ms
64 bytes from localhost (127.0.0.1): icmp_seq=2 ttl=64 time=0.077 ms
64 bytes from localhost (127.0.0.1): icmp_seq=3 ttl=64 time=0.069 ms
^C
--- localhost ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2000ms
rtt min/avg/max/mdev = 0.069/0.080/0.094/0.010 ms
Now i am worried that if localhost do not work.Is it that my network is not properly configured for k8 cluster
How to fix this issue.
There is an old reported issue here that seems to be related to your problem.
According to that, it may be something to do with IPv6 and a solution is to add the --ipv4 option to each execution of the curl command, or apply a permanent solution for the host by disabling IPv6 altogether.
I am trying to access a Flask server running in one Openshift pod from other.
For that I created a service as below.
$ oc get svc
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
my-web-app ClusterIP 172.30.216.112 <none> 8080/TCP 8m
$ oc describe svc my-web-app
Name: my-web-app
Namespace: neo4j-sys
Labels: app=my-web-app
Annotations: openshift.io/generated-by=OpenShiftNewApp
Selector: app=my-web-app,deploymentconfig=my-web-app
Type: ClusterIP
IP: 172.30.216.112
Port: 8080-tcp 8080/TCP
TargetPort: 8080/TCP
Endpoints: 172.20.203.104:5000,172.20.49.150:5000
Session Affinity: None
Events: <none>
1)
First, I ping ed from one pod to other pod and got response.
(app-root) sh-4.2$ ping 172.20.203.104
PING 172.20.203.104 (172.20.203.104) 56(84) bytes of data.
64 bytes from 172.20.203.104: icmp_seq=1 ttl=64 time=5.53 ms
64 bytes from 172.20.203.104: icmp_seq=2 ttl=64 time=0.527 ms
64 bytes from 172.20.203.104: icmp_seq=3 ttl=64 time=3.10 ms
64 bytes from 172.20.203.104: icmp_seq=4 ttl=64 time=2.12 ms
64 bytes from 172.20.203.104: icmp_seq=5 ttl=64 time=0.784 ms
64 bytes from 172.20.203.104: icmp_seq=6 ttl=64 time=6.81 ms
64 bytes from 172.20.203.104: icmp_seq=7 ttl=64 time=18.2 ms
^C
--- 172.20.203.104 ping statistics ---
7 packets transmitted, 7 received, 0% packet loss, time 6012ms
rtt min/avg/max/mdev = 0.527/5.303/18.235/5.704 ms
But, when I tried curl, it is not responding.
(app-root) sh-4.2$ curl 172.20.203.104
curl: (7) Failed connect to 172.20.203.104:80; Connection refused
(app-root) sh-4.2$ curl 172.20.203.104:8080
curl: (7) Failed connect to 172.20.203.104:8080; Connection refused
2)
After that I tried to reach cluster IP from one pod. In this case, both ping, curl not reachable.
(app-root) sh-4.2$ ping 172.30.216.112
PING 172.30.216.112 (172.30.216.112) 56(84) bytes of data.
From 172.20.49.1 icmp_seq=1 Destination Host Unreachable
From 172.20.49.1 icmp_seq=4 Destination Host Unreachable
From 172.20.49.1 icmp_seq=2 Destination Host Unreachable
From 172.20.49.1 icmp_seq=3 Destination Host Unreachable
^C
--- 172.30.216.112 ping statistics ---
7 packets transmitted, 0 received, +4 errors, 100% packet loss, time 6002ms
pipe 4
(app-root) sh-4.2$ curl 172.30.216.112
curl: (7) Failed connect to 172.30.216.112:80; No route to host
Please let me know where I am going wrong here. Why the above cases #1, #2 failing. How to access the clusterIP services.
I am completely new to the services and accessing them and hence I might be missing some basics.
I gone through other answers How can I access the Kubernetes service through ClusterIP 1. But, it is for Nodeport which is not helping me.
Updates based on below comment from Graham Dumpleton, below are my observations.
This is the Flask server log which I am running in the pods for information.
* Serving Flask app "wsgi" (lazy loading)
* Environment: production
WARNING: This is a development server. Do not use it in a production deployment.
Use a production WSGI server instead.
* Debug mode: off
* Running on http://127.0.0.1:5000/ (Press CTRL+C to quit)
127.0.0.1 - - [14/Nov/2019 04:54:53] "GET / HTTP/1.1" 200 -
127.0.0.1 - - [14/Nov/2019 04:55:05] "GET / HTTP/1.1" 200 -
Is your pod listening on external interfaces on pod 8080?
If I understand question correctly, my intention here is just to communicate between pods via clusterIP service. I am not looking for accessing pods from external interfaces(other projects or through web urls as load balancer service)
If you get into the pod, can you do curl $HOSTNAME:8080?
Yes, if I am running as localhost or 127.0.0.1, I am getting response from the same pod where I run this as expected.
(app-root) sh-4.2$ curl http://127.0.0.1:5000/
Hello World!
(app-root) sh-4.2$ curl http://localhost:5000/
Hello World!
But, if I tried with my-web-app or service IP(clusterIP). I am not getting response.
(app-root) sh-4.2$ curl http://172.30.216.112:5000/
curl: (7) Failed connect to 172.30.216.112:5000; No route to host
(app-root) sh-4.2$ curl my-web-app:8080
curl: (7) Failed connect to my-web-app:8080; Connection refused
(app-root) sh-4.2$ curl http://my-web-app:8080/
curl: (7) Failed connect to my-web-app:8080; Connection refused
With pod IP also I am not getting response.
(app-root) sh-4.2$ curl http://172.20.49.150:5000/
curl: (7) Failed connect to 172.20.49.150:5000; Connection refused
(app-root) sh-4.2$ curl 172.20.49.150
curl: (7) Failed connect to 172.20.49.150:80; Connection refused
I am answering my own question. Here is how my issue got resolved based on inputs from Graham Dumpleton.
Initially, I used to connect Flask server as below.
from flask import Flask
application = Flask(__name__)
if __name__ == "__main__":
application.run()
This bind the server to http://127.0.0.1:5000/ by default.
As part of resolution I changed the bind to 0.0.0.0 as below
from flask import Flask
application = Flask(__name__)
if __name__ == "__main__":
application.run(host='0.0.0.0')
And log as below after that.
* Serving Flask app "wsgi" (lazy loading)
* Environment: production
WARNING: This is a development server. Do not use it in a production deployment.
Use a production WSGI server instead.
* Debug mode: off
* Running on http://0.0.0.0:5000/ (Press CTRL+C to quit)
After that pods successfully communicated via clusterIP. Below are the service details(increased one more pod)
$ oc describe svc my-web-app
Name: my-web-app
Namespace: neo4j-sys
Labels: app=my-web-app
Annotations: openshift.io/generated-by=OpenShiftNewApp
Selector: app=my-web-app,deploymentconfig=my-web-app
Type: ClusterIP
IP: 172.30.4.250
Port: 8080-tcp 8080/TCP
TargetPort: 5000/TCP
Endpoints: 172.20.106.184:5000,172.20.182.118:5000,172.20.83.40:5000
Session Affinity: None
Events: <none>
Below is the successful response.
(app-root) sh-4.2$ curl http://172.30.4.250:8080 //with clusterIP which is my expectation
Hello World!
(app-root) sh-4.2$ curl http://172.20.106.184:5000 // with pod IP
Hello World!
(app-root) sh-4.2$ curl $HOSTNAME:5000 // with $HOSTNAME
Hello World!