Since a couple of months we are facing a lot of timeout on the Uber estimates API. We are making requests using curl, here is what the verbose output print on my test server when it timeouts
curl -v \
-H 'Authorization: Token [my token]' \
-H 'Accept-Language: fr_FR' \
-H 'Content-Type: application/json' \
'https://api.uber.com/v1.2/estimates/price? start_latitude=48.8676689&start_longitude=2.3677804&end_latitude=48.8791163&end_longitude=2.3560725'
* Trying 104.36.195.168...
When curl halts on connection I try to nc at the same time on the 443 port
nc -vz 104.36.195.157 443
No response either
After a few seconds the same nc command respond success but the curl call still halts.
After a few minutes curl will timeout and retry and it will finally work here is the curl output
* Trying 104.36.194.191...
* TCP_NODELAY set
* connect to 104.36.194.191 port 443 failed: Connection timed out
* Trying 104.36.195.168...
* TCP_NODELAY set
* After 85223ms connect time, move on!
* connect to 104.36.195.168 port 443 failed: Connection timed out
* Trying 104.36.195.165...
* TCP_NODELAY set
* Connected to api.uber.com (104.36.195.165) port 443 (#0)
After the curl manage to connect successfully, I get the response from the API.
Sometime the curl call will work directly and respond the right API result.
Also, when I ping api.uber.com it sometimes does not respond
$ dig api.uber.com
; <<>> DiG 9.11.3-1ubuntu1.3-Ubuntu <<>> api.uber.com
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 12152
;; flags: qr rd ra; QUERY: 1, ANSWER: 7, AUTHORITY: 0, ADDITIONAL:
;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 65494
;; QUESTION SECTION:
;api.uber.com. IN A
;; ANSWER SECTION:
api.uber.com. 60 IN CNAME frontends.uber.com.
frontends.uber.com. 59 IN CNAME frontends-dca1.uber.com.
frontends-dca1.uber.com. 8 IN A 104.36.195.158
frontends-dca1.uber.com. 8 IN A 104.36.194.159
frontends-dca1.uber.com. 8 IN A 104.36.194.134
frontends-dca1.uber.com. 8 IN A 104.36.195.165
frontends-dca1.uber.com. 8 IN A 104.36.195.162
;; Query time: 44 msec
;; SERVER: 127.0.0.53#53(127.0.0.53)
;; WHEN: Thu Jan 17 12:33:30 CET 2019
;; MSG SIZE rcvd: 174
$ ping 104.36.195.158
PING 104.36.195.158 (104.36.195.158) 56(84) bytes of data.
^C
--- 104.36.195.158 ping statistics ---
3 packets transmitted, 0 received, 100% packet loss, time 2047ms
$ ping 104.36.194.159
PING 104.36.194.159 (104.36.194.159) 56(84) bytes of data.
64 bytes from 104.36.194.159: icmp_seq=1 ttl=48 time=86.0 ms
64 bytes from 104.36.194.159: icmp_seq=2 ttl=48 time=85.6 ms
64 bytes from 104.36.194.159: icmp_seq=3 ttl=48 time=85.6 ms
^C
--- 104.36.194.159 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2002ms
rtt min/avg/max/mdev = 85.639/85.772/86.031/0.183 ms
My test server is an Ubuntu 18 on AWS EC2.
I tried to reproduce the issue on my local environment (Mint 18.3 Sylvia Ubuntu 16.04) but it always work!
When the test server was on Ubuntu 16.04 we didn't seem to have the problem either. Is there a problem on my current test environment ?
Is anyone experiencing the same behavior ?
Thanks for your time!
A few days ago the problem solved itself without any action on my side.
Related
I recently started using flutter to create an app that has video and audio calling, so I thought of using WebRTC to accomplish this. However, when I tried to create a local WebRTC server from the WebRTC Github it doesn't let me connect to it. When I run the server from the command prompt it tells me that the server is listening at the given local IP, but when I try to connect to it, it says the site can't be reached. Also when I ping the url it says the ping request could not find the host. Any kind of help would really be appreciated.
WebRTC server: https://github.com/flutter-webrtc/flutter-webrtc-server
Starting server in command prompt
netstat when server running
pinging server
The IP 0.0.0.0 is not the IP address the server is binding/listening too.
It is a common way that servers allow configurion 0.0.0.0 to express binding to all IPv4 the host has (see https://en.wikipedia.org/wiki/0.0.0.0 ). To acually reach your server use whatever IP you have on the host (if you are starting the server on your workstation 127.0.0.1 ought to work)
Ping on Windows is only ICMP if I am not mistaken, this means a TCP port number is not part of it's expected input. Eg. "0.0.0.0:8086" is not a valid target for that command (see https://learn.microsoft.com/en-us/windows-server/administration/windows-commands/ping )
>ping 0.0.0.0:8086
Ping request could not find host 0.0.0.0:8086. Please check the name and try again.
>ping 0.0.0.0
Pinging 0.0.0.0 with 32 bytes of data:
PING: transmit failed. General failure.
PING: transmit failed. General failure.
Ping statistics for 0.0.0.0:
Packets: Sent = 2, Received = 0, Lost = 2 (100% loss),
Control-C
^C
>ping 127.0.0.1
Pinging 127.0.0.1 with 32 bytes of data:
Reply from 127.0.0.1: bytes=32 time<1ms TTL=128
Reply from 127.0.0.1: bytes=32 time<1ms TTL=128
Ping statistics for 127.0.0.1:
Packets: Sent = 2, Received = 2, Lost = 0 (0% loss),
Approximate round trip times in milli-seconds:
Minimum = 0ms, Maximum = 0ms, Average = 0ms
Control-C
^C
Since what you are trying to connect to is a webserver you could use something like curl instead to try to verify your connection. Should be available without any further install if you are on Windows 10 1803 or newer.
>curl https://0.0.0.0:8086
curl: (7) Failed to connect to 0.0.0.0 port 8086: Address not available
>curl https://127.0.0.1:8086
<html><head><title>Unauthorized</title></head><body><h1>401 Unauthorized</h1></body></html>
Some browsers might support https://0.0.0.0:8086 in a similar way server do but I wouldn't rely on it. Go with https://127.0.0.1:8086 or a non localhost IP address of the host.
I have recently followed the tutorial on how to use Kubernetes with Windows pods ( https://learn.microsoft.com/en-us/azure/container-service/kubernetes/container-service-kubernetes-windows-walkthrough ). I decided to extend the example to two services, one front calling the one in the back. Simplified:
https://gist.github.com/sebug/f478f1cfd0a793e8d556c6001bbbe142
But now when I connect to one of the front nodes:
kubectl exec -it samplefront-2836659004-4m824 -- powershell
I can't ping the other service:
PS C:\> ping sample-back
Ping request could not find host sample-back. Please check the name and try again.
I heard that it may be because of the two network interfaces and the wrong DNS server being chosen, but I have not found a way to specify anything in the deployment.
Windows IP Configuration
Ethernet adapter vEthernet (Container NIC 7baf5cc0):
Connection-specific DNS Suffix . :
Link-local IPv6 Address . . . . . : fe80::f182:e2e7:7bce:ed60%33
IPv4 Address. . . . . . . . . . . : 10.244.0.211
Subnet Mask . . . . . . . . . . . : 255.255.255.0
Default Gateway . . . . . . . . . : 10.244.0.1
Ethernet adapter vEthernet (Container NIC ae765bad):
Connection-specific DNS Suffix . : 10jheu23yh0ujpey5vzw0q45qg.ax.internal.cloudapp.net
Link-local IPv6 Address . . . . . : fe80::c4dc:b785:9cd:2a7b%37
IPv4 Address. . . . . . . . . . . : 172.31.245.122
Subnet Mask . . . . . . . . . . . : 255.255.240.0
Default Gateway . . . . . . . . . : 172.31.240.1
Can't resolve another service's hostname inside my Kubernetes.
This is a by design behavior. Because the cluster IP does not exist.
In Kubernetes, all the services in a cluster are handled by kube-proxy. kube-proxy runs on every node in the cluster, and what it does it write iptables rules for each service (Linux node, same as windows). These iptables rules manage the traffic towards the service IPs. They don’t actually have any rules for ICMP, because it’s not needed.
But we can ping pod IP or pod's DNS.
For example, we can use this command to list pods IP addresses:
root#k8s-master-9F42C511-0:~# kubectl get pods -o wide
NAME READY STATUS RESTARTS AGE IP NODE
azure-vote-back-3048739398-8zx8b 1/1 Running 0 18m 10.244.1.2 k8s-agent-9f42c511-0
azure-vote-front-837696400-tglpn 1/1 Running 0 18m 10.244.1.3 k8s-agent-9f42c511-0
Then we use one pod to ping those IP addresses:
root#k8s-master-9F42C511-0:~# kubectl exec -it azure-vote-front-837696400-tglpn -- /bin/bash
root#azure-vote-front-837696400-tglpn:/app# ping 10.244.1.3
PING 10.244.1.3 (10.244.1.3): 56 data bytes
64 bytes from 10.244.1.3: icmp_seq=0 ttl=64 time=0.063 ms
64 bytes from 10.244.1.3: icmp_seq=1 ttl=64 time=0.052 ms
^C--- 10.244.1.3 ping statistics ---
2 packets transmitted, 2 packets received, 0% packet loss
round-trip min/avg/max/stddev = 0.052/0.057/0.063/0.000 ms
root#azure-vote-front-837696400-tglpn:/app# ping 10.244.1.4
PING 10.244.1.4 (10.244.1.4): 56 data bytes
64 bytes from 10.244.1.4: icmp_seq=0 ttl=64 time=0.102 ms
64 bytes from 10.244.1.4: icmp_seq=1 ttl=64 time=0.098 ms
^C--- 10.244.1.4 ping statistics ---
2 packets transmitted, 2 packets received, 0% packet loss
round-trip min/avg/max/stddev = 0.098/0.100/0.102/0.000 ms
Also, we can ping pod's A record. In kubernetes, pod's A record in the form of pod-ip-address.my-namespace.pod.cluster.local.
For example, a pod with IP 1.2.3.4 in the namespace default with a DNS name of cluster.local would have an entry: 1-2-3-4.default.pod.cluster.local
In my lab, my pod's A record like this:
root#k8s-master-9F42C511-0:~# kubectl exec -it azure-vote-front-837696400-tglpn -- /bin/bash
root#azure-vote-front-837696400-tglpn:/app# ping 10-244-1-2.default.pod.cluster.local
PING 10-244-1-2.default.pod.cluster.local (10.244.1.2): 56 data bytes
64 bytes from 10.244.1.2: icmp_seq=0 ttl=64 time=0.103 ms
64 bytes from 10.244.1.2: icmp_seq=1 ttl=64 time=0.087 ms
64 bytes from 10.244.1.2: icmp_seq=2 ttl=64 time=0.096 ms
^C--- 10-244-1-2.default.pod.cluster.local ping statistics ---
3 packets transmitted, 3 packets received, 0% packet loss
round-trip min/avg/max/stddev = 0.087/0.095/0.103/0.000 ms
So, we can't ping cluster IP address, but we can use URL to test it. We can ping pod's IP address, and A record.
Update:
Sorry for my mistake, the k8s A record rules work for Linux Agent, but does not work for windows agent.
More information about windows server containers, please refer to this article.
So after trying out different scenarios I figured I'd delete the setup and try again, specifying a specific version of microsoft/iis - and it worked:
https://gist.github.com/sebug/0f7776668fff4e0e6b3f3d313846afa6
kripke:Documents/Projets/ScaledSample% kubectl exec -it samplefront-1226573881-21bbh -- ping sample-back
Pinging sample-back [10.0.216.120] with 32 bytes of data:
Reply from 10.0.216.120: bytes=32 time<1ms TTL=128
Reply from 10.0.216.120: bytes=32 time<1ms TTL=128
Reply from 10.0.216.120: bytes=32 time<1ms TTL=128
Reply from 10.0.216.120: bytes=32 time<1ms TTL=128
Ping statistics for 10.0.216.120:
Packets: Sent = 4, Received = 4, Lost = 0 (0% loss),
Approximate round trip times in milli-seconds:
Minimum = 0ms, Maximum = 0ms, Average = 0ms
My hypothesis is that either I fell on a more well-behaved instance after having recreated the resource group and cluster or that this specifying which exact version of the container image did the trick.
On my local I run a mysql container and then ping it from another container on the same network:
$ docker run -d tutum/mysql
$ docker run -it plumsempy/plum bash
# ping MYSQL_CONTAINER_ID
PING 67e35427d638 (198.105.244.24): 56 data bytes
64 bytes from 198.105.244.24: icmp_seq=0 ttl=37 time=0.243 ms
...
That is good. Then, using Kubernetes(minikube) locally, I deploy tutum/mysql using the following YAML:
...
- name: mysql
image: tutum/mysql
...
There is nothing else for the mysql container. Then I deploy it, ssh into the minikube pod, spin up a random container and try pinging the mysql container inside the pod this time:
$ kubectl create -f k8s-deployment.yml
$ minikube ssh
$ docker ps
$ docker run -it plumsempy/plum bash
# ping MYSQL_CONTAINER_ID_INSIDE_MINIKUBE
PING mysql (198.105.244.24): 56 data bytes
^C--- mysql ping statistics ---
10 packets transmitted, 0 packets received, 100% packet loss
# traceroute MYSQL_CONTAINER_ID_INSIDE_MINIKUBE
traceroute to aa7f7ed7af01 (198.105.244.24), 30 hops max, 60 byte packets
1 172.17.0.1 (172.17.0.1) 0.031 ms 0.009 ms 0.007 ms
2 10.0.2.2 (10.0.2.2) 0.156 ms 0.086 ms 0.050 ms
3 * * *
4 * * *
5 dtr02gldlca-tge-0-2-0-1.gldl.ca.charter.com (96.34.102.201) 16.153 ms 16.107 ms 16.077 ms
6 crr01lnbhca-bue-200.lnbh.ca.charter.com (96.34.98.188) 18.753 ms 18.011 ms 30.642 ms
7 crr01mtpkca-bue-201.mtpk.ca.charter.com (96.34.96.63) 30.779 ms 30.523 ms 30.428 ms
8 bbr01mtpkca-bue-2.mtpk.ca.charter.com (96.34.2.24) 24.089 ms 23.900 ms 23.814 ms
9 bbr01ashbva-tge-0-1-0-1.ashb.va.charter.com (96.34.3.139) 26.061 ms 25.949 ms 36.002 ms
10 10ge9-10.core1.lax1.he.net (65.19.189.177) 34.027 ms 34.436 ms 33.857 ms
11 100ge12-1.core1.ash1.he.net (184.105.80.201) 107.873 ms 107.750 ms 104.078 ms
12 100ge3-1.core1.nyc4.he.net (184.105.223.166) 100.554 ms 100.478 ms 100.393 ms
13 xerocole-inc.10gigabitethernet12-4.core1.nyc4.he.net (216.66.41.242) 109.184 ms 111.122 ms 111.018 ms
14 * * *
15 * * *
...(til it ends)
the plumsempy/plum can be any container since they are both on the same network and same pod, the pinging should go through. The question is Why can I not reach mysql on minikube and how could I fix that?
From k8s multi-container pod docs:
Pods share fate, and share some resources, such as storage volumes and IP addresses.
Hence the mysql container is reachable from the plum container at the IP address 127.0.0.1.
Also, since mysql runs on port 3306 by default, you probably want telnet 127.0.0.1 3306 to check if it's reachable (ping uses ICMP which doesn't have the concept of ports).
I guess the container ID just don't work with Kubernetes. You can also see, that the container ID resolved to the public IP 198.105.244.24, which looks wrong.
You have multiple ways to contact this pod:
get the pod IP via kubectl describe -f k8s-deployment.yml
create a service for that pod and do one of these (assuming the service name is mysql):
use environment variables like ping ${MYSQL_SERVICE_HOST}
use DNS like ping mysql.default.svc.cluster.local
Running this command in the mongodb installation file from mongodb.org
./mongo ds045907.mongolab.com:45907/database -u user -p password
I changed Database, user, and password for anonymity.
results in this
Error: couldn't connect to server ds045907.mongolab.com:45907 src/mongo/shell/mongo.js:93
exception: connect failed
Maybe i'm being blocked by a server firewall? I have no problem using git or brew or pip...
Here are a few things you can try, but you can always feel free to contact us at support#mongolab.com. I'm sure we can get to the bottom of this.
Anonymous mongo shell connection
Mongo will let you connect without authenticating. You can do very little with an unauthenticated connection, but you can use it as a test to separate a connectivity problem from a credentials problem.
% mongo ds045907.mongolab.com:45907
MongoDB shell version: 2.0.7
connecting to: ds045907.mongolab.com:45907/test
> db.version()
2.2.2
> db.runCommand({ping:1})
{ "ok" : 1 }
> exit
bye
If you can connect without authenticating and run the commands as shown above, but trying to connect with authentication fails, then you have a problem with the credentials. If, however, connecting doesn't work even without supplying credentials then you have a connectivity problem.
ping
That server does allow ICMP traffic, so make sure it's reachable from wherever you are.
% ping ds045907.mongolab.com
PING ec2-107-20-85-188.compute-1.amazonaws.com (107.20.85.188): 56 data bytes
64 bytes from 107.20.85.188: icmp_seq=0 ttl=41 time=99.744 ms
64 bytes from 107.20.85.188: icmp_seq=1 ttl=41 time=99.475 ms
64 bytes from 107.20.85.188: icmp_seq=2 ttl=41 time=99.930 ms
^C
--- ec2-107-20-85-188.compute-1.amazonaws.com ping statistics ---
3 packets transmitted, 3 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 99.475/99.716/99.930/0.187 ms
traceroute
If ping fails, use traceroute (or tracert on Windows) to try to figure out where the problem is. Once the trace reaches AWS, however, it will trail off. That's normal. AWS prevents traces from seeing too far into their networks. Make sure that the last IP on your list is owned by Amazon using some kind of IP reverse lookup tool (many on the Web).
% traceroute ds045907.mongolab.com
traceroute to ec2-107-20-85-188.compute-1.amazonaws.com (107.20.85.188), 64 hops max, 52 byte packets
1 192.168.1.1 (192.168.1.1) 1.092 ms 0.865 ms 1.047 ms
2 192.168.27.1 (192.168.27.1) 1.414 ms 1.330 ms 1.224 ms
... snipped to protect the innocent ...
14 72.21.220.83 (72.21.220.83) 87.777 ms
72.21.220.75 (72.21.220.75) 87.406 ms
205.251.229.55 (205.251.229.55) 99.363 ms
15 72.21.222.145 (72.21.222.145) 87.703 ms
178.236.3.24 (178.236.3.24) 98.662 ms
72.21.220.75 (72.21.220.75) 87.708 ms
16 216.182.224.55 (216.182.224.55) 87.312 ms 86.791 ms 89.005 ms
17 * 216.182.224.55 (216.182.224.55) 91.373 ms *
18 216.182.224.55 (216.182.224.55) 121.754 ms * *
19 * * *
20 * * *
21 * * *
22 * * *
23 * * *
24 * * *
It's a connection problem at your side. I tried it but got a login failure message:
MongoDB shell version: 1.6.5
connecting to: ds045907.mongolab.com:45907/database
Mon Dec 24 01:12:31 uncaught exception: login failed
exception: login failed
#!/usr/bin/perl
use Net::Ping;
$p = Net::Ping->new();
my $main_ip="$ARGV[0]";
if ($p->ping($main_ip,1)){
$result=true;
print "$main_ip is alive \n";
}else{
print "$main_ip is down \n";
}
I am using above perl script to ping check the server. It worked fine all the cases except for IP 192.168.0.168.
$ perl test.pl 192.168.0.168
192.168.0.168 is down
]$ ping 192.168.0.168
PING 192.168.0.168 (192.168.0.168) 56(84) bytes of data.
64 bytes from 192.168.0.168: icmp_seq=1 ttl=64 time=0.304 ms
64 bytes from 192.168.0.168: icmp_seq=2 ttl=64 time=0.355 ms
64 bytes from 192.168.0.168: icmp_seq=3 ttl=64 time=2.94 ms
64 bytes from 192.168.0.168: icmp_seq=4 ttl=64 time=0.388 ms
--- 192.168.0.168 ping statistics ---
4 packets transmitted, 4 received, 0% packet loss, time 3292ms
rtt min/avg/max/mdev = 0.304/0.997/2.944/1.124 ms
]$ ping 192.168.0.18
PING 192.168.0.18 (192.168.0.18) 56(84) bytes of data.
From 192.168.0.181 icmp_seq=2 Destination Host Unreachable
From 192.168.0.181 icmp_seq=3 Destination Host Unreachable
From 192.168.0.181 icmp_seq=4 Destination Host Unreachable
--- 192.168.0.18 ping statistics ---
4 packets transmitted, 0 received, +3 errors, 100% packet loss, time 3292ms
pipe 3
]$ perl test.pl 192.168.0.18
192.168.0.18 is down
I had no clue even I have increase ping timeout but the results same
The only problem i can think of is, that the ping command uses the ICMP protocol as default, while Net::Ping uses TCP. You can switch Net::Ping to ICMP by creating your object like this:
my $p = Net::Ping->new( 'icmp' );
Take note, that making ICMP pings requires root privilege on Unix.
In order to send icmp packets you must have rights to create raw sockets, i.e. have root rights.
I suppose you running ping.pl as ordinary user, but you need to be root
ls -al `which ping`
-rws--x--x 1 root root 39640 Dec 17 2011 /bin/ping
^
|
suid bit
ping program has a suid bit, which allows to run ping program with root rights.
By default Net::Ping tries to connect to echo port (7/tcp), if it gets ECONNREFUSED - this means that host is up but refuses connection (nothing is listening on that port). if connect breaks on timeout, this means that host is down.
But! I can block all connection to 7/tcp by firewall:
iptables -I INPUT -p tcp --dport 7 -j DROP
and... voila, i get down instead of alive
So, you should check firewall on your failure pinged host