I want to invoke service in my kuberetes cluster, so I try to invoke servcie(the backend pod deployment by deplyment) like this:
curl http://soa-illidan-hub-service.dabai-fat.svc.cluster.local:11024
it tell me curl: (7) Failed to connect to soa-illidan-hub-service.dabai-fat.svc.cluster.local port 11024: Host is unreachable, and I am ping:
/ # ping soa-illidan-hub-service.dabai-fat.svc.cluster.local
PING soa-illidan-hub-service.dabai-fat.svc.cluster.local (10.254.42.62): 56 data bytes
64 bytes from 10.254.42.62: seq=0 ttl=64 time=0.051 ms
64 bytes from 10.254.42.62: seq=1 ttl=64 time=0.063 ms
64 bytes from 10.254.42.62: seq=2 ttl=64 time=0.057 ms
why the curl tell me host is unreachable? and. I could using endpoint ip to invoke servcie,should I using ip? using servcie ip is a good practice?
Try
wget -qO- http://soa-illidan-hub-service.dabai-fat.svc.cluster.local
or try directly to pod ip
wget -qO- POD_IP:PORT
I finnally find the reason, the pod's running node kube-proxy stopped. Try to start kube-proxy will fix this, in my centos 7.6 start like this:
systemctl start kube-proxy
Related
I have recently followed the tutorial on how to use Kubernetes with Windows pods ( https://learn.microsoft.com/en-us/azure/container-service/kubernetes/container-service-kubernetes-windows-walkthrough ). I decided to extend the example to two services, one front calling the one in the back. Simplified:
https://gist.github.com/sebug/f478f1cfd0a793e8d556c6001bbbe142
But now when I connect to one of the front nodes:
kubectl exec -it samplefront-2836659004-4m824 -- powershell
I can't ping the other service:
PS C:\> ping sample-back
Ping request could not find host sample-back. Please check the name and try again.
I heard that it may be because of the two network interfaces and the wrong DNS server being chosen, but I have not found a way to specify anything in the deployment.
Windows IP Configuration
Ethernet adapter vEthernet (Container NIC 7baf5cc0):
Connection-specific DNS Suffix . :
Link-local IPv6 Address . . . . . : fe80::f182:e2e7:7bce:ed60%33
IPv4 Address. . . . . . . . . . . : 10.244.0.211
Subnet Mask . . . . . . . . . . . : 255.255.255.0
Default Gateway . . . . . . . . . : 10.244.0.1
Ethernet adapter vEthernet (Container NIC ae765bad):
Connection-specific DNS Suffix . : 10jheu23yh0ujpey5vzw0q45qg.ax.internal.cloudapp.net
Link-local IPv6 Address . . . . . : fe80::c4dc:b785:9cd:2a7b%37
IPv4 Address. . . . . . . . . . . : 172.31.245.122
Subnet Mask . . . . . . . . . . . : 255.255.240.0
Default Gateway . . . . . . . . . : 172.31.240.1
Can't resolve another service's hostname inside my Kubernetes.
This is a by design behavior. Because the cluster IP does not exist.
In Kubernetes, all the services in a cluster are handled by kube-proxy. kube-proxy runs on every node in the cluster, and what it does it write iptables rules for each service (Linux node, same as windows). These iptables rules manage the traffic towards the service IPs. They don’t actually have any rules for ICMP, because it’s not needed.
But we can ping pod IP or pod's DNS.
For example, we can use this command to list pods IP addresses:
root#k8s-master-9F42C511-0:~# kubectl get pods -o wide
NAME READY STATUS RESTARTS AGE IP NODE
azure-vote-back-3048739398-8zx8b 1/1 Running 0 18m 10.244.1.2 k8s-agent-9f42c511-0
azure-vote-front-837696400-tglpn 1/1 Running 0 18m 10.244.1.3 k8s-agent-9f42c511-0
Then we use one pod to ping those IP addresses:
root#k8s-master-9F42C511-0:~# kubectl exec -it azure-vote-front-837696400-tglpn -- /bin/bash
root#azure-vote-front-837696400-tglpn:/app# ping 10.244.1.3
PING 10.244.1.3 (10.244.1.3): 56 data bytes
64 bytes from 10.244.1.3: icmp_seq=0 ttl=64 time=0.063 ms
64 bytes from 10.244.1.3: icmp_seq=1 ttl=64 time=0.052 ms
^C--- 10.244.1.3 ping statistics ---
2 packets transmitted, 2 packets received, 0% packet loss
round-trip min/avg/max/stddev = 0.052/0.057/0.063/0.000 ms
root#azure-vote-front-837696400-tglpn:/app# ping 10.244.1.4
PING 10.244.1.4 (10.244.1.4): 56 data bytes
64 bytes from 10.244.1.4: icmp_seq=0 ttl=64 time=0.102 ms
64 bytes from 10.244.1.4: icmp_seq=1 ttl=64 time=0.098 ms
^C--- 10.244.1.4 ping statistics ---
2 packets transmitted, 2 packets received, 0% packet loss
round-trip min/avg/max/stddev = 0.098/0.100/0.102/0.000 ms
Also, we can ping pod's A record. In kubernetes, pod's A record in the form of pod-ip-address.my-namespace.pod.cluster.local.
For example, a pod with IP 1.2.3.4 in the namespace default with a DNS name of cluster.local would have an entry: 1-2-3-4.default.pod.cluster.local
In my lab, my pod's A record like this:
root#k8s-master-9F42C511-0:~# kubectl exec -it azure-vote-front-837696400-tglpn -- /bin/bash
root#azure-vote-front-837696400-tglpn:/app# ping 10-244-1-2.default.pod.cluster.local
PING 10-244-1-2.default.pod.cluster.local (10.244.1.2): 56 data bytes
64 bytes from 10.244.1.2: icmp_seq=0 ttl=64 time=0.103 ms
64 bytes from 10.244.1.2: icmp_seq=1 ttl=64 time=0.087 ms
64 bytes from 10.244.1.2: icmp_seq=2 ttl=64 time=0.096 ms
^C--- 10-244-1-2.default.pod.cluster.local ping statistics ---
3 packets transmitted, 3 packets received, 0% packet loss
round-trip min/avg/max/stddev = 0.087/0.095/0.103/0.000 ms
So, we can't ping cluster IP address, but we can use URL to test it. We can ping pod's IP address, and A record.
Update:
Sorry for my mistake, the k8s A record rules work for Linux Agent, but does not work for windows agent.
More information about windows server containers, please refer to this article.
So after trying out different scenarios I figured I'd delete the setup and try again, specifying a specific version of microsoft/iis - and it worked:
https://gist.github.com/sebug/0f7776668fff4e0e6b3f3d313846afa6
kripke:Documents/Projets/ScaledSample% kubectl exec -it samplefront-1226573881-21bbh -- ping sample-back
Pinging sample-back [10.0.216.120] with 32 bytes of data:
Reply from 10.0.216.120: bytes=32 time<1ms TTL=128
Reply from 10.0.216.120: bytes=32 time<1ms TTL=128
Reply from 10.0.216.120: bytes=32 time<1ms TTL=128
Reply from 10.0.216.120: bytes=32 time<1ms TTL=128
Ping statistics for 10.0.216.120:
Packets: Sent = 4, Received = 4, Lost = 0 (0% loss),
Approximate round trip times in milli-seconds:
Minimum = 0ms, Maximum = 0ms, Average = 0ms
My hypothesis is that either I fell on a more well-behaved instance after having recreated the resource group and cluster or that this specifying which exact version of the container image did the trick.
On my local I run a mysql container and then ping it from another container on the same network:
$ docker run -d tutum/mysql
$ docker run -it plumsempy/plum bash
# ping MYSQL_CONTAINER_ID
PING 67e35427d638 (198.105.244.24): 56 data bytes
64 bytes from 198.105.244.24: icmp_seq=0 ttl=37 time=0.243 ms
...
That is good. Then, using Kubernetes(minikube) locally, I deploy tutum/mysql using the following YAML:
...
- name: mysql
image: tutum/mysql
...
There is nothing else for the mysql container. Then I deploy it, ssh into the minikube pod, spin up a random container and try pinging the mysql container inside the pod this time:
$ kubectl create -f k8s-deployment.yml
$ minikube ssh
$ docker ps
$ docker run -it plumsempy/plum bash
# ping MYSQL_CONTAINER_ID_INSIDE_MINIKUBE
PING mysql (198.105.244.24): 56 data bytes
^C--- mysql ping statistics ---
10 packets transmitted, 0 packets received, 100% packet loss
# traceroute MYSQL_CONTAINER_ID_INSIDE_MINIKUBE
traceroute to aa7f7ed7af01 (198.105.244.24), 30 hops max, 60 byte packets
1 172.17.0.1 (172.17.0.1) 0.031 ms 0.009 ms 0.007 ms
2 10.0.2.2 (10.0.2.2) 0.156 ms 0.086 ms 0.050 ms
3 * * *
4 * * *
5 dtr02gldlca-tge-0-2-0-1.gldl.ca.charter.com (96.34.102.201) 16.153 ms 16.107 ms 16.077 ms
6 crr01lnbhca-bue-200.lnbh.ca.charter.com (96.34.98.188) 18.753 ms 18.011 ms 30.642 ms
7 crr01mtpkca-bue-201.mtpk.ca.charter.com (96.34.96.63) 30.779 ms 30.523 ms 30.428 ms
8 bbr01mtpkca-bue-2.mtpk.ca.charter.com (96.34.2.24) 24.089 ms 23.900 ms 23.814 ms
9 bbr01ashbva-tge-0-1-0-1.ashb.va.charter.com (96.34.3.139) 26.061 ms 25.949 ms 36.002 ms
10 10ge9-10.core1.lax1.he.net (65.19.189.177) 34.027 ms 34.436 ms 33.857 ms
11 100ge12-1.core1.ash1.he.net (184.105.80.201) 107.873 ms 107.750 ms 104.078 ms
12 100ge3-1.core1.nyc4.he.net (184.105.223.166) 100.554 ms 100.478 ms 100.393 ms
13 xerocole-inc.10gigabitethernet12-4.core1.nyc4.he.net (216.66.41.242) 109.184 ms 111.122 ms 111.018 ms
14 * * *
15 * * *
...(til it ends)
the plumsempy/plum can be any container since they are both on the same network and same pod, the pinging should go through. The question is Why can I not reach mysql on minikube and how could I fix that?
From k8s multi-container pod docs:
Pods share fate, and share some resources, such as storage volumes and IP addresses.
Hence the mysql container is reachable from the plum container at the IP address 127.0.0.1.
Also, since mysql runs on port 3306 by default, you probably want telnet 127.0.0.1 3306 to check if it's reachable (ping uses ICMP which doesn't have the concept of ports).
I guess the container ID just don't work with Kubernetes. You can also see, that the container ID resolved to the public IP 198.105.244.24, which looks wrong.
You have multiple ways to contact this pod:
get the pod IP via kubectl describe -f k8s-deployment.yml
create a service for that pod and do one of these (assuming the service name is mysql):
use environment variables like ping ${MYSQL_SERVICE_HOST}
use DNS like ping mysql.default.svc.cluster.local
ubuntu#ip-172-31-45-35:~/cse303.p2$ node qloader.js
events.js:72
throw er; // Unhandled 'error' event
^
Error: failed to connect to [52.33.215.205:27017]
at null.<anonymous> (/home/ubuntu/cse303.p2/node_modules/mongoose/node_modules/mongodb/lib/mong odb/connection/server.js:556:74)
at EventEmitter.emit (events.js:106:17)
at null.<anonymous> (/home/ubuntu/cse303.p2/node_modules/mongoose/node_modules/mongodb/lib/mongodb/connection/connection_pool.js:156:15)
at EventEmitter.emit (events.js:98:17)
at Socket.<anonymous> (/home/ubuntu/cse303.p2/node_modules/mongoose/node_modules/mongodb/lib/mongodb/connection/connection.js:534:10)
at Socket.EventEmitter.emit (events.js:95:17)
at net.js:441:14
at process._tickCallback (node.js:415:13)
above is the error message i got when trying to do 'node' to run my database.
I have set all my security group to 'all traffic', I suppose it's not a problem with my security group.
When I ping that ip address it can connect, this is what is shows.
ubuntu#ip-172-31-45-35:~/cse303.p2$ ping 52.33.215.205
PING 52.33.215.205 (52.33.215.205) 56(84) bytes of data.
64 bytes from 52.33.215.205: icmp_seq=1 ttl=63 time=1.37 ms
64 bytes from 52.33.215.205: icmp_seq=2 ttl=63 time=1.27 ms
64 bytes from 52.33.215.205: icmp_seq=3 ttl=63 time=1.21 ms
64 bytes from 52.33.215.205: icmp_seq=4 ttl=63 time=1.25 ms
^C
--- 52.33.215.205 ping statistics ---
4 packets transmitted, 4 received, 0% packet loss, time 3004ms
rtt min/avg/max/mdev = 1.218/1.281/1.376/0.072 ms
So it can connect to that address, but when i add that port 27017, it does't work.
ubuntu#ip-172-31-45-35:~/cse303.p2$ ping 52.33.215.205:27017
ping: unknown host 52.33.215.205:27017
I am not sure whether my mongod is running properly, this is what i got.
ubuntu#ip-172-31-22-135:~$ sudo service mongodb start
mongodb start/running, process 1382
You can't ping a port. Ping is ICMP and not TCP or UDP. The line mongodb start/running, process 1382 indicates that MongoDB is running.
It is likely that MongoDB's bind IP is set to 127.0.0.1. In the MongoDB configuration, check the value of bind_ip. It is likely set to localhost (127.0.0.1) and thus will only accept connections from the local VM.
There are many guides out there that can help, such as this or this.
I'm running into an issue with the MongoReplicaClient while trying to connect to a remote replica set. For some reason MongoReplicaSetClient is just returning an empty set for any replica set that is not on my local host. The strange part is that MongoClient works fine. I'm not sure how to debug this any further since it is not erroring out in anyway. Networking appears to be ruled out since connecting via MongoClient works fine. Does anyone have an idea why this could be happening?
[Update]
After further investigation it appears that this issue affects Pymongo 2.6.3 when the local host is unable to resolve the remote hostname without using it's FQDN. The problem is that if the FQDN is provided to MongoReplicaSetClient in this situation it will just return an empty set. Here's how I reproduced the issue:
Environment
python2.7 (2.7.3-0ubuntu3.4)
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=12.04
DISTRIB_CODENAME=precise
DISTRIB_DESCRIPTION="Ubuntu 12.04.2 LTS"
Procedure
1) Confirm that remote hostname cannot resolve
05:03:48 [js-dev][503]$ ping -c3 hq-sre-mongodb-01
ping: unknown host hq-sre-mongodb-01
05:03:59 [js-dev][504]$ ping -c3 hq-sre-mongodb-01.eng.nutanix.com
PING hq-sre-mongodb-01.eng.nutanix.com (10.1.56.11) 56(84) bytes of data.
64 bytes from hq-sre-mongodb-01.eng.nutanix.com (10.1.56.11): icmp_req=1 ttl=63 time=0.222 ms
64 bytes from hq-sre-mongodb-01.eng.nutanix.com (10.1.56.11): icmp_req=2 ttl=63 time=0.217 ms
64 bytes from hq-sre-mongodb-01.eng.nutanix.com (10.1.56.11): icmp_req=3 ttl=63 time=0.247 ms
Test on pymongo 2.5
1) Connect to the replica set using hostname (Breaks as expected)
>>> MongoReplicaSetClient('hq-sre-mongodb-01', replicaSet='rs0')
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/usr/local/lib/python2.7/dist-packages/pymongo/mongo_replica_set_client.py", line 516, in __init__
raise ConnectionFailure(str(e))
pymongo.errors.ConnectionFailure: hq-sre-mongodb-01:27017: [Errno -2] Name or service not known
2) Connect to the replica set using FQDN (Works)
>>> pymongo.version
'2.5'
>>> MongoReplicaSetClient('hq-sre-mongodb-01.eng.nutanix.com', replicaSet='rs0')
MongoReplicaSetClient([u'sv2-sre-mongodb-03:27017', u'hq-sre-mongodb-01:27017', u'sv2-sre-mongodb-01:27017', u'sv2-sre-mongodb-02:27017', u'hq-sre-mongodb-02:27017'])
Test on pymongo 2.6.3
1) Connect to the replica set using hostname (Breaks as expected)
>>> pymongo.version
'2.6.3'
>>> MongoReplicaSetClient('hq-sre-mongodb-01', replicaSet='rs0')
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/usr/local/lib/python2.7/dist-packages/pymongo/mongo_replica_set_client.py", line 745, in __init__
raise ConnectionFailure(str(e))
pymongo.errors.ConnectionFailure: hq-sre-mongodb-01:27017: [Errno -2] Name or service not known
2) Connect to the replica set using FQDN (Returns an empty set)
>>> pymongo.version
'2.6.3'
>>> MongoReplicaSetClient('hq-sre-mongodb-01.eng.nutanix.com', replicaSet='rs0')
MongoReplicaSetClient([])
Confirmation
You can see that once the localhost can resolve by hostname the issue does not present itself any longer in 2.6.3. It's almost as if MongoReplicaSetClient doesn't know how to handle a FQDN.
1) Confirm remote hostname resolves
05:33:32 [js-dev][501]$ ping -c3 hq-sre-mongodb-01
PING hq-sre-mongodb-01.eng.nutanix.com (10.1.56.11) 56(84) bytes of data.
64 bytes from hq-sre-mongodb-01.eng.nutanix.com (10.1.56.11): icmp_req=1 ttl=63 time=0.263 ms
64 bytes from hq-sre-mongodb-01.eng.nutanix.com (10.1.56.11): icmp_req=2 ttl=63 time=0.259 ms
64 bytes from hq-sre-mongodb-01.eng.nutanix.com (10.1.56.11): icmp_req=3 ttl=63 time=0.281 ms
05:33:36 [js-dev][502]$ ping -c3 hq-sre-mongodb-01.eng.nutanix.com
PING hq-sre-mongodb-01.eng.nutanix.com (10.1.56.11) 56(84) bytes of data.
64 bytes from hq-sre-mongodb-01.eng.nutanix.com (10.1.56.11): icmp_req=1 ttl=63 time=0.154 ms
64 bytes from hq-sre-mongodb-01.eng.nutanix.com (10.1.56.11): icmp_req=2 ttl=63 time=0.256 ms
64 bytes from hq-sre-mongodb-01.eng.nutanix.com (10.1.56.11): icmp_req=3 ttl=63 time=0.275 ms
2) Confirm MongoReplicaSetClient connection using both hostname and FQDN
>>> pymongo.version
'2.6.3'
>>> MongoReplicaSetClient('hq-sre-mongodb-01', replicaSet='rs0')
MongoReplicaSetClient([u'sv2-sre-mongodb-03:27017', 'hq-sre-mongodb-01:27017', u'sv2-sre-mongodb-01:27017', u'sv2-sre-mongodb-02:27017', u'hq-sre-mongodb-02:27017'])
>>> MongoReplicaSetClient('hq-sre-mongodb-01.eng.nutanix.com', replicaSet='rs0')
MongoReplicaSetClient([u'sv2-sre-mongodb-03:27017', u'hq-sre-mongodb-01:27017', u'sv2-sre-mongodb-01:27017', u'sv2-sre-mongodb-02:27017', u'hq-sre-mongodb-02:27017'])
This is a PyMongo bug. I've reported it in PYTHON-608 and I'll fix it for the next release of PyMongo, version 2.7, due out in about a month.
The behavior you saw in PyMongo 2.5 was buggy: The MongoReplicaSetClient connected to the FQDN of the member you provided, but the member wasn't actually reachable using its hostname from the replica set config. Auto-reconnect, failover, and read preferences wouldn't work, so the client should've raised an error instead of allowing you to proceed.
The PyMongo 2.6.3 behavior is buggy too, obviously. In 2.6.3, the client knew it couldn't reach any members using the hostnames it discovered, but it didn't raise an error. Instead, it allowed you to create an unusable client. The correct behavior is to raise an error if none of the hostnames in the replica set config are reachable from the client.
This appears to be related to networking issues between the mongod nodes. If your host is having DNS issues and the stars align you may see this issue. In my testing I've seen MongoReplicaSetClient return an empty list if all nodes are not reachable by both hostname and FQDN. However, after a running a few test scenarios I cannot reliably reproduce the issue so I'll close this out. If you run into this, ensure that DNS connectivity to your mongod nodes is fully functional and you should be all set.
#!/usr/bin/perl
use Net::Ping;
$p = Net::Ping->new();
my $main_ip="$ARGV[0]";
if ($p->ping($main_ip,1)){
$result=true;
print "$main_ip is alive \n";
}else{
print "$main_ip is down \n";
}
I am using above perl script to ping check the server. It worked fine all the cases except for IP 192.168.0.168.
$ perl test.pl 192.168.0.168
192.168.0.168 is down
]$ ping 192.168.0.168
PING 192.168.0.168 (192.168.0.168) 56(84) bytes of data.
64 bytes from 192.168.0.168: icmp_seq=1 ttl=64 time=0.304 ms
64 bytes from 192.168.0.168: icmp_seq=2 ttl=64 time=0.355 ms
64 bytes from 192.168.0.168: icmp_seq=3 ttl=64 time=2.94 ms
64 bytes from 192.168.0.168: icmp_seq=4 ttl=64 time=0.388 ms
--- 192.168.0.168 ping statistics ---
4 packets transmitted, 4 received, 0% packet loss, time 3292ms
rtt min/avg/max/mdev = 0.304/0.997/2.944/1.124 ms
]$ ping 192.168.0.18
PING 192.168.0.18 (192.168.0.18) 56(84) bytes of data.
From 192.168.0.181 icmp_seq=2 Destination Host Unreachable
From 192.168.0.181 icmp_seq=3 Destination Host Unreachable
From 192.168.0.181 icmp_seq=4 Destination Host Unreachable
--- 192.168.0.18 ping statistics ---
4 packets transmitted, 0 received, +3 errors, 100% packet loss, time 3292ms
pipe 3
]$ perl test.pl 192.168.0.18
192.168.0.18 is down
I had no clue even I have increase ping timeout but the results same
The only problem i can think of is, that the ping command uses the ICMP protocol as default, while Net::Ping uses TCP. You can switch Net::Ping to ICMP by creating your object like this:
my $p = Net::Ping->new( 'icmp' );
Take note, that making ICMP pings requires root privilege on Unix.
In order to send icmp packets you must have rights to create raw sockets, i.e. have root rights.
I suppose you running ping.pl as ordinary user, but you need to be root
ls -al `which ping`
-rws--x--x 1 root root 39640 Dec 17 2011 /bin/ping
^
|
suid bit
ping program has a suid bit, which allows to run ping program with root rights.
By default Net::Ping tries to connect to echo port (7/tcp), if it gets ECONNREFUSED - this means that host is up but refuses connection (nothing is listening on that port). if connect breaks on timeout, this means that host is down.
But! I can block all connection to 7/tcp by firewall:
iptables -I INPUT -p tcp --dport 7 -j DROP
and... voila, i get down instead of alive
So, you should check firewall on your failure pinged host