Can't resolve another service's hostname inside my Kubernetes Windows setup - kubernetes

I have recently followed the tutorial on how to use Kubernetes with Windows pods ( https://learn.microsoft.com/en-us/azure/container-service/kubernetes/container-service-kubernetes-windows-walkthrough ). I decided to extend the example to two services, one front calling the one in the back. Simplified:
https://gist.github.com/sebug/f478f1cfd0a793e8d556c6001bbbe142
But now when I connect to one of the front nodes:
kubectl exec -it samplefront-2836659004-4m824 -- powershell
I can't ping the other service:
PS C:\> ping sample-back
Ping request could not find host sample-back. Please check the name and try again.
I heard that it may be because of the two network interfaces and the wrong DNS server being chosen, but I have not found a way to specify anything in the deployment.
Windows IP Configuration
Ethernet adapter vEthernet (Container NIC 7baf5cc0):
Connection-specific DNS Suffix . :
Link-local IPv6 Address . . . . . : fe80::f182:e2e7:7bce:ed60%33
IPv4 Address. . . . . . . . . . . : 10.244.0.211
Subnet Mask . . . . . . . . . . . : 255.255.255.0
Default Gateway . . . . . . . . . : 10.244.0.1
Ethernet adapter vEthernet (Container NIC ae765bad):
Connection-specific DNS Suffix . : 10jheu23yh0ujpey5vzw0q45qg.ax.internal.cloudapp.net
Link-local IPv6 Address . . . . . : fe80::c4dc:b785:9cd:2a7b%37
IPv4 Address. . . . . . . . . . . : 172.31.245.122
Subnet Mask . . . . . . . . . . . : 255.255.240.0
Default Gateway . . . . . . . . . : 172.31.240.1

Can't resolve another service's hostname inside my Kubernetes.
This is a by design behavior. Because the cluster IP does not exist.
In Kubernetes, all the services in a cluster are handled by kube-proxy. kube-proxy runs on every node in the cluster, and what it does it write iptables rules for each service (Linux node, same as windows). These iptables rules manage the traffic towards the service IPs. They don’t actually have any rules for ICMP, because it’s not needed.
But we can ping pod IP or pod's DNS.
For example, we can use this command to list pods IP addresses:
root#k8s-master-9F42C511-0:~# kubectl get pods -o wide
NAME READY STATUS RESTARTS AGE IP NODE
azure-vote-back-3048739398-8zx8b 1/1 Running 0 18m 10.244.1.2 k8s-agent-9f42c511-0
azure-vote-front-837696400-tglpn 1/1 Running 0 18m 10.244.1.3 k8s-agent-9f42c511-0
Then we use one pod to ping those IP addresses:
root#k8s-master-9F42C511-0:~# kubectl exec -it azure-vote-front-837696400-tglpn -- /bin/bash
root#azure-vote-front-837696400-tglpn:/app# ping 10.244.1.3
PING 10.244.1.3 (10.244.1.3): 56 data bytes
64 bytes from 10.244.1.3: icmp_seq=0 ttl=64 time=0.063 ms
64 bytes from 10.244.1.3: icmp_seq=1 ttl=64 time=0.052 ms
^C--- 10.244.1.3 ping statistics ---
2 packets transmitted, 2 packets received, 0% packet loss
round-trip min/avg/max/stddev = 0.052/0.057/0.063/0.000 ms
root#azure-vote-front-837696400-tglpn:/app# ping 10.244.1.4
PING 10.244.1.4 (10.244.1.4): 56 data bytes
64 bytes from 10.244.1.4: icmp_seq=0 ttl=64 time=0.102 ms
64 bytes from 10.244.1.4: icmp_seq=1 ttl=64 time=0.098 ms
^C--- 10.244.1.4 ping statistics ---
2 packets transmitted, 2 packets received, 0% packet loss
round-trip min/avg/max/stddev = 0.098/0.100/0.102/0.000 ms
Also, we can ping pod's A record. In kubernetes, pod's A record in the form of pod-ip-address.my-namespace.pod.cluster.local.
For example, a pod with IP 1.2.3.4 in the namespace default with a DNS name of cluster.local would have an entry: 1-2-3-4.default.pod.cluster.local
In my lab, my pod's A record like this:
root#k8s-master-9F42C511-0:~# kubectl exec -it azure-vote-front-837696400-tglpn -- /bin/bash
root#azure-vote-front-837696400-tglpn:/app# ping 10-244-1-2.default.pod.cluster.local
PING 10-244-1-2.default.pod.cluster.local (10.244.1.2): 56 data bytes
64 bytes from 10.244.1.2: icmp_seq=0 ttl=64 time=0.103 ms
64 bytes from 10.244.1.2: icmp_seq=1 ttl=64 time=0.087 ms
64 bytes from 10.244.1.2: icmp_seq=2 ttl=64 time=0.096 ms
^C--- 10-244-1-2.default.pod.cluster.local ping statistics ---
3 packets transmitted, 3 packets received, 0% packet loss
round-trip min/avg/max/stddev = 0.087/0.095/0.103/0.000 ms
So, we can't ping cluster IP address, but we can use URL to test it. We can ping pod's IP address, and A record.
Update:
Sorry for my mistake, the k8s A record rules work for Linux Agent, but does not work for windows agent.
More information about windows server containers, please refer to this article.

So after trying out different scenarios I figured I'd delete the setup and try again, specifying a specific version of microsoft/iis - and it worked:
https://gist.github.com/sebug/0f7776668fff4e0e6b3f3d313846afa6
kripke:Documents/Projets/ScaledSample% kubectl exec -it samplefront-1226573881-21bbh -- ping sample-back
Pinging sample-back [10.0.216.120] with 32 bytes of data:
Reply from 10.0.216.120: bytes=32 time<1ms TTL=128
Reply from 10.0.216.120: bytes=32 time<1ms TTL=128
Reply from 10.0.216.120: bytes=32 time<1ms TTL=128
Reply from 10.0.216.120: bytes=32 time<1ms TTL=128
Ping statistics for 10.0.216.120:
Packets: Sent = 4, Received = 4, Lost = 0 (0% loss),
Approximate round trip times in milli-seconds:
Minimum = 0ms, Maximum = 0ms, Average = 0ms
My hypothesis is that either I fell on a more well-behaved instance after having recreated the resource group and cluster or that this specifying which exact version of the container image did the trick.

Related

Why can the third host receive dataflow between host no.1 and host no.2?(Three hosts are all virtual machines installed in VMware with NAT mode)

I installed three operating systems(let's say 3 hosts) in VMware, all with NAT mode. 3 hosts are named centos, centos 1,centos 2.(As the pic shows below)
3 hosts in VMware
The first host's IP address is 192.168.248.132, the second is 192.168.248.136, and we don't need to know third host's IP because it's not related to this issue.
I typed the command "ping 192.168.248.136", and the output on the screen is:
PING 192.168.248.136 (192.168.248.136) 56(84) bytes of data.
64 bytes from 192.168.248.136: icmp_seq=1 ttl=64 time=0.435 ms
64 bytes from 192.168.248.136: icmp_seq=2 ttl=64 time=0.313 ms
64 bytes from 192.168.248.136: icmp_seq=3 ttl=64 time=0.385 ms
This means ping command has succeeded and host no.2(whose IP addr is 192.168.248.136) has received ICMP and replied.
Meanwhile, I typed the command "tcpdump -i ens33" in host no.3. If everything had worked correctly, host no.3 would not have received any data between host no.1 and host no.2, because ICMP is neither broadcast nor multicast, so only host no.1 and 2 can send and receive. Also, host no.3's network interface is not promiscuous mode, so it can only receive it's own frame. The output from host no.3 below can show it is not promiscuous mode.
[root#localhost usr]# ifconfig
ens33: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 192.168.248.137 netmask 255.255.255.0 broadcast 192.168.248.255
inet6 fe80::b488:bc2c:3770:a95f prefixlen 64 scopeid 0x20<link>
ether 00:0c:29:0d:dc:86 txqueuelen 1000 (Ethernet)
RX packets 351081 bytes 512917768 (489.1 MiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 34947 bytes 2166260 (2.0 MiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
flag is 4163<UP,BROADCAST,RUNNING,MULTICAST>, "PROMISC" is not mentioned, so it is not promiscuous mode.
However, after I typed "tcpdump -i ens33" in host no.3, something appeared on the screen:
06:28:11.511233 IP 192.168.248.132 > 192.168.248.136: ICMP echo request, id 3137, seq 5, length 64
06:28:11.511503 IP 192.168.248.136 > 192.168.248.132: ICMP echo reply, id 3137, seq 5, length 64
Host no.3 received the dataflow between no.1 and 2, and this was supposed to be sent to no.2, but no.3 received it.
So here comes the question, why can host no.3 receive packet which was not supposed to be sent to it?
tcpdump by default activates "promiscuous mode" making it able to see anything on the network it is connected to (even if not explicitly sent to it).
the three hosts seem to be connected to a virtual switch that do not isolate the hosts from each other.

could not invoke kubernetes service by internal endpont

I want to invoke service in my kuberetes cluster, so I try to invoke servcie(the backend pod deployment by deplyment) like this:
curl http://soa-illidan-hub-service.dabai-fat.svc.cluster.local:11024
it tell me curl: (7) Failed to connect to soa-illidan-hub-service.dabai-fat.svc.cluster.local port 11024: Host is unreachable, and I am ping:
/ # ping soa-illidan-hub-service.dabai-fat.svc.cluster.local
PING soa-illidan-hub-service.dabai-fat.svc.cluster.local (10.254.42.62): 56 data bytes
64 bytes from 10.254.42.62: seq=0 ttl=64 time=0.051 ms
64 bytes from 10.254.42.62: seq=1 ttl=64 time=0.063 ms
64 bytes from 10.254.42.62: seq=2 ttl=64 time=0.057 ms
why the curl tell me host is unreachable? and. I could using endpoint ip to invoke servcie,should I using ip? using servcie ip is a good practice?
Try
wget -qO- http://soa-illidan-hub-service.dabai-fat.svc.cluster.local
or try directly to pod ip
wget -qO- POD_IP:PORT
I finnally find the reason, the pod's running node kube-proxy stopped. Try to start kube-proxy will fix this, in my centos 7.6 start like this:
systemctl start kube-proxy

Flutter WebRTC server is not letting me connect

I recently started using flutter to create an app that has video and audio calling, so I thought of using WebRTC to accomplish this. However, when I tried to create a local WebRTC server from the WebRTC Github it doesn't let me connect to it. When I run the server from the command prompt it tells me that the server is listening at the given local IP, but when I try to connect to it, it says the site can't be reached. Also when I ping the url it says the ping request could not find the host. Any kind of help would really be appreciated.
WebRTC server: https://github.com/flutter-webrtc/flutter-webrtc-server
Starting server in command prompt
netstat when server running
pinging server
The IP 0.0.0.0 is not the IP address the server is binding/listening too.
It is a common way that servers allow configurion 0.0.0.0 to express binding to all IPv4 the host has (see https://en.wikipedia.org/wiki/0.0.0.0 ). To acually reach your server use whatever IP you have on the host (if you are starting the server on your workstation 127.0.0.1 ought to work)
Ping on Windows is only ICMP if I am not mistaken, this means a TCP port number is not part of it's expected input. Eg. "0.0.0.0:8086" is not a valid target for that command (see https://learn.microsoft.com/en-us/windows-server/administration/windows-commands/ping )
>ping 0.0.0.0:8086
Ping request could not find host 0.0.0.0:8086. Please check the name and try again.
>ping 0.0.0.0
Pinging 0.0.0.0 with 32 bytes of data:
PING: transmit failed. General failure.
PING: transmit failed. General failure.
Ping statistics for 0.0.0.0:
Packets: Sent = 2, Received = 0, Lost = 2 (100% loss),
Control-C
^C
>ping 127.0.0.1
Pinging 127.0.0.1 with 32 bytes of data:
Reply from 127.0.0.1: bytes=32 time<1ms TTL=128
Reply from 127.0.0.1: bytes=32 time<1ms TTL=128
Ping statistics for 127.0.0.1:
Packets: Sent = 2, Received = 2, Lost = 0 (0% loss),
Approximate round trip times in milli-seconds:
Minimum = 0ms, Maximum = 0ms, Average = 0ms
Control-C
^C
Since what you are trying to connect to is a webserver you could use something like curl instead to try to verify your connection. Should be available without any further install if you are on Windows 10 1803 or newer.
>curl https://0.0.0.0:8086
curl: (7) Failed to connect to 0.0.0.0 port 8086: Address not available
>curl https://127.0.0.1:8086
<html><head><title>Unauthorized</title></head><body><h1>401 Unauthorized</h1></body></html>
Some browsers might support https://0.0.0.0:8086 in a similar way server do but I wouldn't rely on it. Go with https://127.0.0.1:8086 or a non localhost IP address of the host.

Cannot ping containers in the same pod in Kubernetes(minikube)

On my local I run a mysql container and then ping it from another container on the same network:
$ docker run -d tutum/mysql
$ docker run -it plumsempy/plum bash
# ping MYSQL_CONTAINER_ID
PING 67e35427d638 (198.105.244.24): 56 data bytes
64 bytes from 198.105.244.24: icmp_seq=0 ttl=37 time=0.243 ms
...
That is good. Then, using Kubernetes(minikube) locally, I deploy tutum/mysql using the following YAML:
...
- name: mysql
image: tutum/mysql
...
There is nothing else for the mysql container. Then I deploy it, ssh into the minikube pod, spin up a random container and try pinging the mysql container inside the pod this time:
$ kubectl create -f k8s-deployment.yml
$ minikube ssh
$ docker ps
$ docker run -it plumsempy/plum bash
# ping MYSQL_CONTAINER_ID_INSIDE_MINIKUBE
PING mysql (198.105.244.24): 56 data bytes
^C--- mysql ping statistics ---
10 packets transmitted, 0 packets received, 100% packet loss
# traceroute MYSQL_CONTAINER_ID_INSIDE_MINIKUBE
traceroute to aa7f7ed7af01 (198.105.244.24), 30 hops max, 60 byte packets
1 172.17.0.1 (172.17.0.1) 0.031 ms 0.009 ms 0.007 ms
2 10.0.2.2 (10.0.2.2) 0.156 ms 0.086 ms 0.050 ms
3 * * *
4 * * *
5 dtr02gldlca-tge-0-2-0-1.gldl.ca.charter.com (96.34.102.201) 16.153 ms 16.107 ms 16.077 ms
6 crr01lnbhca-bue-200.lnbh.ca.charter.com (96.34.98.188) 18.753 ms 18.011 ms 30.642 ms
7 crr01mtpkca-bue-201.mtpk.ca.charter.com (96.34.96.63) 30.779 ms 30.523 ms 30.428 ms
8 bbr01mtpkca-bue-2.mtpk.ca.charter.com (96.34.2.24) 24.089 ms 23.900 ms 23.814 ms
9 bbr01ashbva-tge-0-1-0-1.ashb.va.charter.com (96.34.3.139) 26.061 ms 25.949 ms 36.002 ms
10 10ge9-10.core1.lax1.he.net (65.19.189.177) 34.027 ms 34.436 ms 33.857 ms
11 100ge12-1.core1.ash1.he.net (184.105.80.201) 107.873 ms 107.750 ms 104.078 ms
12 100ge3-1.core1.nyc4.he.net (184.105.223.166) 100.554 ms 100.478 ms 100.393 ms
13 xerocole-inc.10gigabitethernet12-4.core1.nyc4.he.net (216.66.41.242) 109.184 ms 111.122 ms 111.018 ms
14 * * *
15 * * *
...(til it ends)
the plumsempy/plum can be any container since they are both on the same network and same pod, the pinging should go through. The question is Why can I not reach mysql on minikube and how could I fix that?
From k8s multi-container pod docs:
Pods share fate, and share some resources, such as storage volumes and IP addresses.
Hence the mysql container is reachable from the plum container at the IP address 127.0.0.1.
Also, since mysql runs on port 3306 by default, you probably want telnet 127.0.0.1 3306 to check if it's reachable (ping uses ICMP which doesn't have the concept of ports).
I guess the container ID just don't work with Kubernetes. You can also see, that the container ID resolved to the public IP 198.105.244.24, which looks wrong.
You have multiple ways to contact this pod:
get the pod IP via kubectl describe -f k8s-deployment.yml
create a service for that pod and do one of these (assuming the service name is mysql):
use environment variables like ping ${MYSQL_SERVICE_HOST}
use DNS like ping mysql.default.svc.cluster.local

perl ping failure

#!/usr/bin/perl
use Net::Ping;
$p = Net::Ping->new();
my $main_ip="$ARGV[0]";
if ($p->ping($main_ip,1)){
$result=true;
print "$main_ip is alive \n";
}else{
print "$main_ip is down \n";
}
I am using above perl script to ping check the server. It worked fine all the cases except for IP 192.168.0.168.
$ perl test.pl 192.168.0.168
192.168.0.168 is down
]$ ping 192.168.0.168
PING 192.168.0.168 (192.168.0.168) 56(84) bytes of data.
64 bytes from 192.168.0.168: icmp_seq=1 ttl=64 time=0.304 ms
64 bytes from 192.168.0.168: icmp_seq=2 ttl=64 time=0.355 ms
64 bytes from 192.168.0.168: icmp_seq=3 ttl=64 time=2.94 ms
64 bytes from 192.168.0.168: icmp_seq=4 ttl=64 time=0.388 ms
--- 192.168.0.168 ping statistics ---
4 packets transmitted, 4 received, 0% packet loss, time 3292ms
rtt min/avg/max/mdev = 0.304/0.997/2.944/1.124 ms
]$ ping 192.168.0.18
PING 192.168.0.18 (192.168.0.18) 56(84) bytes of data.
From 192.168.0.181 icmp_seq=2 Destination Host Unreachable
From 192.168.0.181 icmp_seq=3 Destination Host Unreachable
From 192.168.0.181 icmp_seq=4 Destination Host Unreachable
--- 192.168.0.18 ping statistics ---
4 packets transmitted, 0 received, +3 errors, 100% packet loss, time 3292ms
pipe 3
]$ perl test.pl 192.168.0.18
192.168.0.18 is down
I had no clue even I have increase ping timeout but the results same
The only problem i can think of is, that the ping command uses the ICMP protocol as default, while Net::Ping uses TCP. You can switch Net::Ping to ICMP by creating your object like this:
my $p = Net::Ping->new( 'icmp' );
Take note, that making ICMP pings requires root privilege on Unix.
In order to send icmp packets you must have rights to create raw sockets, i.e. have root rights.
I suppose you running ping.pl as ordinary user, but you need to be root
ls -al `which ping`
-rws--x--x 1 root root 39640 Dec 17 2011 /bin/ping
^
|
suid bit
ping program has a suid bit, which allows to run ping program with root rights.
By default Net::Ping tries to connect to echo port (7/tcp), if it gets ECONNREFUSED - this means that host is up but refuses connection (nothing is listening on that port). if connect breaks on timeout, this means that host is down.
But! I can block all connection to 7/tcp by firewall:
iptables -I INPUT -p tcp --dport 7 -j DROP
and... voila, i get down instead of alive
So, you should check firewall on your failure pinged host