Environment
Oracle Solaris 11 for SPARC
Running in a Non-primary (Guest) Logical Domain (LDOM).
Logged in with root access.
Problem
My application uses libpcap to capture network traffic. When my application (myTestApp) calls libpcap findalldevs, my application only sees one network interface ("lo0"), yet ifconfig -a shows many more interfaces.
My application is statically linked to libpcap (version 1.3). The build machine is SunOS RS-T5120-01 5.10 Generic_141444-09 sun4v sparc SUNW,SPARC-Enterprise-T5120.
Any ideas why my application can't see all the network interfaces ?
Linux command Line Sample Output
# tcpdump --version
tcpdump version 4.1.1
libpcap version 1.1.1
# uname -a
SunOS g99dnpi802-LD 5.11 11.1 sun4v sparc sun4v
# ./myTestApp -adapters
[Available Adapters]
name: "lo0", description: "", address: 127.0.0.1, mask: 255.0.0.0
# ifconfig -a
lo0: flags=2001000849<UP,LOOPBACK,RUNNING,MULTICAST,IPv4,VIRTUAL> mtu 8232 index 1
inet 127.0.0.1 netmask ff000000
net0: flags=100001000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4,PHYSRUNNING> mtu 1500 index 2
inet 10.99.220.15 netmask ffffff00 broadcast 10.99.220.255
ether 0:14:4f:fa:e0:8d
net1: flags=100001000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4,PHYSRUNNING> mtu 1500 index 3
inet 10.99.193.210 netmask ffffff80 broadcast 10.99.193.255
ether 0:14:4f:f9:d0:9c
lo0: flags=2002000849<UP,LOOPBACK,RUNNING,MULTICAST,IPv6,VIRTUAL> mtu 8252 index 1
inet6 ::1/128
net0: flags=120002000840<RUNNING,MULTICAST,IPv6,PHYSRUNNING> mtu 1500 index 2
inet6 ::/0
ether 0:14:4f:fa:e0:8d
net1: flags=120002000840<RUNNING,MULTICAST,IPv6,PHYSRUNNING> mtu 1500 index 3
inet6 ::/0
ether 0:14:4f:f9:d0:9c
# tcpdump -i net1
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on net1, link-type EN10MB (Ethernet), capture size 65535 bytes
09:32:29.520815 IP g99dnpi802-LD.ssh > 10.99.8.102.65436: Flags [P.], seq 3397909586:3397909718, ack 1479093081, win 64240, length 132
09:32:29.520860 IP g99dnpi802-LD.ssh > 10.99.8.102.65436: Flags [P.], seq 132:232, ack 1, win 64240, length 100
09:32:29.521644 IP 10.99.8.102.65436 > g99dnpi802-LD.ssh: Flags [.], ack 132, win 16379, length 0
09:32:29.680844 00:14:4f:f9:8d:84 (oui Unknown) > Broadcast, ethertype Unknown (0xcafe), length 90:
0x0000: 0500 ad85 0939 ffff 0001 ffff 809c 7401 .....9........t.
0x0010: 0000 004c 0000 0000 8070 00ab 0000 0000 ...L.....p......
0x0020: 0000 0000 0000 0000 0043 ffff 2074 6167 .........C...tag
0x0030: 6d61 7374 0672 0014 4ff9 8d84 5f31 3362 mast.r..O..._13b
0x0040: 650a 0000 0000 0000 84f9 0aab e...........
[update]
Here is the (edited) output of running the following truss command on the build machine and the customer machine.
truss –f –a –vall –l –d –o truss.txt ./myTestApp -adapters
truss on build machine
14365/1: 0.0751 so_socket(PF_INET, SOCK_DGRAM, IPPROTO_IP, "", SOV_DEFAULT) = 3
14365/1: 0.0753 so_socket(PF_INET6, SOCK_DGRAM, IPPROTO_IP, "", SOV_DEFAULT) = 4
14365/1: 0.0755 ioctl(3, SIOCGLIFNUM, 0xFFBE9F50) = 0
14365/1: 0.0757 ioctl(3, SIOCGLIFCONF, 0xFFBE9F40) = 0
14365/1: 0.0804 ioctl(3, SIOCGLIFFLAGS, 0xFFBE9DC8) = 0
14365/1: 0.0806 ioctl(3, SIOCGLIFNETMASK, 0xFFBE9C50) = 0
14365/1: 0.0809 open64("/dev/lo", O_RDWR) Err#2 ENOENT
14365/1: 0.0811 open64("/dev/lo0", O_RDWR) Err#2 ENOENT
14365/1: 0.0813 ioctl(3, SIOCGLIFFLAGS, 0xFFBE9DC8) = 0
14365/1: 0.0815 ioctl(3, SIOCGLIFNETMASK, 0xFFBE9C50) = 0
14365/1: 0.0817 ioctl(3, SIOCGLIFBRDADDR, 0xFFBE9AD8) = 0
14365/1: 0.0819 open64("/dev/e1000g", O_RDWR) = 5
truss on customer machine
6346/1: 0.0315 so_socket(PF_INET, SOCK_DGRAM, IPPROTO_IP, 0, SOV_DEFAULT) = 3
6346/1: 0.0319 so_socket(PF_INET6, SOCK_DGRAM, IPPROTO_IP, 0, SOV_DEFAULT) = 5
6346/1: 0.0320 ioctl(3, SIOCGLIFNUM, 0xFFBEA830) = 0
6346/1: 0.0321 ioctl(3, SIOCGLIFCONF, 0xFFBEA820) = 0
6346/1: 0.0322 ioctl(3, SIOCGLIFFLAGS, 0xFFBEA6A8) = 0
6346/1: 0.0323 ioctl(3, SIOCGLIFNETMASK, 0xFFBEA530) = 0
6346/1: 0.0327 open64("/dev/lo", O_RDWR) Err#2 ENOENT
6346/1: 0.0328 open64("/dev/lo0", O_RDWR) = 6
6346/1: 0.0345 ioctl(3, SIOCGLIFFLAGS, 0xFFBEA6A8) = 0
6346/1: 0.0346 ioctl(3, SIOCGLIFNETMASK, 0xFFBEA530) = 0
6346/1: 0.0347 ioctl(3, SIOCGLIFBRDADDR, 0xFFBEA3B8) = 0
6346/1: 0.0347 open64("/dev/net", O_RDWR) Err#21 EISDIR
6346/1: 0.0349 ioctl(3, SIOCGLIFFLAGS, 0xFFBEA6A8) = 0
6346/1: 0.0349 ioctl(3, SIOCGLIFNETMASK, 0xFFBEA530) = 0
6346/1: 0.0350 ioctl(3, SIOCGLIFBRDADDR, 0xFFBEA3B8) = 0
6346/1: 0.0351 open64("/dev/net", O_RDWR) Err#21 EISDIR
Related
I wanted to see just the DATA packets in the underlying transfers in gRPC. I ran the greeter server-client example application. Both are running on localhost. When I captured traffic on port 50051, I get the following trace (Omitting a few lines for conciseness):
$ sudo tcpdump -i lo port 50051
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on lo, link-type EN10MB (Ethernet), capture size 262144 bytes
14:55:43.756358 IP6 ip6-localhost.58242 > ip6-localhost.50051: Flags [S], seq 3711036939, win 65476, options [mss 65476,sackOK,TS val 3380654929 ecr 0,nop,wscale 7], length 0
14:55:43.756379 IP6 ip6-localhost.50051 > ip6-localhost.58242: Flags [S.], seq 60801323, ack 3711036940, win 65464, options [mss 65476,sackOK,TS val 3380654929 ecr 3380654929,nop,wscale 7], length 0
...
...
14:55:43.760075 IP6 ip6-localhost.50051 > ip6-localhost.58242: Flags [P.], seq 224:241, ack 396, win 512, options [nop,nop,TS val 3380654933 ecr 3380654933], length 17
14:55:43.760091 IP6 ip6-localhost.58242 > ip6-localhost.50051: Flags [.], ack 241, win 512, options [nop,nop,TS val 3380654933 ecr 3380654933], length 0
14:55:43.760440 IP6 ip6-localhost.58242 > ip6-localhost.50051: Flags [F.], seq 396, ack 241, win 512, options [nop,nop,TS val 3380654933 ecr 3380654933], length 0
14:55:43.760588 IP6 ip6-localhost.58242 > ip6-localhost.50051: Flags [R.], seq 397, ack 241, win 512, options [nop,nop,TS val 3380654933 ecr 3380654933], length 0
Note that the above trace has some SYN, ACK, and FIN packets too.
However, when I wanted to extract just the SYN and ACK packets, it gives me no output?!
$ sudo tcpdump -i lo 'tcp port 50051 and tcp[tcpflags] & (tcp-syn|tcp-ack) != 0'
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on lo, link-type EN10MB (Ethernet), capture size 262144 bytes
^C
0 packets captured
0 packets received by filter
0 packets dropped by kernel
I do get the packets back when I reverse the condition:
sudo tcpdump -i lo 'tcp port 50051 and not tcp[tcpflags] & (tcp-syn|tcp-ack) != 0'
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on lo, link-type EN10MB (Ethernet), capture size 262144 bytes
15:05:09.644989 IP6 ip6-localhost.58658 > ip6-localhost.50051: Flags [S], seq 1905191489, win 65476, options [mss 65476,sackOK,TS val 3381220818 ecr 0,nop,wscale 7], length 0
15:05:09.645011 IP6 ip6-localhost.50051 > ip6-localhost.58658: Flags [S.], seq 3628833616, ack 1905191490, win 65464, options [mss 65476,sackOK,TS val 3381220818 ecr 3381220818,nop,wscale 7], length 0
...
...
15:05:09.649368 IP6 ip6-localhost.50051 > ip6-localhost.58658: Flags [P.], seq 224:241, ack 396, win 512, options [nop,nop,TS val 3381220822 ecr 3381220822], length 17
15:05:09.649382 IP6 ip6-localhost.58658 > ip6-localhost.50051: Flags [.], ack 241, win 512, options [nop,nop,TS val 3381220822 ecr 3381220822], length 0
15:05:09.649768 IP6 ip6-localhost.58658 > ip6-localhost.50051: Flags [F.], seq 396, ack 241, win 512, options [nop,nop,TS val 3381220822 ecr 3381220822], length 0
15:05:09.649929 IP6 ip6-localhost.58658 > ip6-localhost.50051: Flags [R.], seq 397, ack 241, win 512, options [nop,nop,TS val 3381220823 ecr 3381220822], length 0
How do I remove the SYN/ACK/FIN packets from the tcpdump trace?
My test setup looks as following:
Ubuntu 22.4
Kernel 5.15.1025 Realtime
I210 enp1s0 (10.1.180.98)
I225 enp2s0 (10.1.180.97)
Netgear GS108 Switch
enp1s0 and enp2s0 are connected to the switch
sending UDP Packets over enp1s0 to multicast address 224.0.0.22
listening on enp2s0 (-> external loop back)
open62541 UDP pubsub
General:
ifconig
enp1s0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 10.1.180.98 netmask 255.255.255.0 broadcast 10.1.180.255
inet6 fe80::36fc:cf83:b6f7:e7eb prefixlen 64 scopeid 0x20<link>
ether 00:07:32:a5:c3:88 txqueuelen 1000 (Ethernet)
RX packets 10823 bytes 3936173 (3.9 MB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 287226 bytes 29921782 (29.9 MB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
device memory 0x7fe00000-7fe1ffff
enp2s0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 10.1.180.97 netmask 255.255.255.0 broadcast 10.1.180.255
inet6 fe80::a22:bab1:5e74:d3ad prefixlen 64 scopeid 0x20<link>
ether 00:07:32:a5:c3:89 txqueuelen 1000 (Ethernet)
RX packets 287442 bytes 29411683 (29.4 MB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 3506 bytes 174754 (174.7 KB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
device memory 0x7fc00000-7fcfffff
lo: flags=73<UP,LOOPBACK,RUNNING> mtu 65536
inet 127.0.0.1 netmask 255.0.0.0
inet6 ::1 prefixlen 128 scopeid 0x10<host>
loop txqueuelen 1000 (Local Loopback)
RX packets 10698 bytes 924534 (924.5 KB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 10698 bytes 924534 (924.5 KB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
# netstat -rn
Kernel IP routing table
Destination Gateway Genmask Flags MSS Window irtt Iface
0.0.0.0 10.1.180.10 0.0.0.0 UG 0 0 0 enp1s0
0.0.0.0 10.1.180.10 0.0.0.0 UG 0 0 0 enp2s0
10.1.180.0 0.0.0.0 255.255.255.0 U 0 0 0 enp2s0
10.1.180.0 0.0.0.0 255.255.255.0 U 0 0 0 enp1s0
169.254.0.0 0.0.0.0 255.255.0.0 U 0 0 0 enp2s0
# netstat -g
IPv6/IPv4 Group Memberships
Interface RefCnt Group
--------------- ------ ---------------------
lo 1 224.0.0.251
lo 1 224.0.0.1
enp1s0 1 224.0.0.251
enp1s0 1 224.0.0.1
enp2s0 1 224.0.0.22
enp2s0 1 224.0.0.251
enp2s0 1 224.0.0.1
lo 1 ff02::fb
lo 1 ip6-allnodes
lo 1 ff01::1
enp1s0 1 ff02::fb
enp1s0 1 ff02::1:fff7:e7eb
enp1s0 1 ip6-allnodes
enp1s0 1 ff01::1
enp2s0 1 ff02::fb
enp2s0 1 ff02::1:ff74:d3ad
enp2s0 1 ip6-allnodes
enp2s0 1 ff01::1
# iptables -L
Chain INPUT (policy ACCEPT)
target prot opt source destination
Chain FORWARD (policy ACCEPT)
target prot opt source destination
Chain OUTPUT (policy ACCEPT)
target prot opt source destination
befor sending:
# netstat -i
Kernel Interface table
Iface MTU RX-OK RX-ERR RX-DRP RX-OVR TX-OK TX-ERR TX-DRP TX-OVR Flg
enp1s0 1500 12397 0 0 0 572786 0 0 0 BMRU
enp2s0 1500 562000 0 0 0 4015 0 0 0 BMRU
lo 65536 12782 0 0 0 12782 0 0 0 LRU
# netstat -s -u
IcmpMsg:
InType3: 6576
OutType3: 6576
Udp:
5710 packets received
902 packets to unknown port received
0 packet receive errors
576693 packets sent
0 receive buffer errors
0 send buffer errors
IgnoredMulti: 259
UdpLite:
IpExt:
InMcastPkts: 110
OutMcastPkts: 567399
InBcastPkts: 259
InOctets: 54256683
OutOctets: 52916072
InMcastOctets: 10142
OutMcastOctets: 50498445
InBcastOctets: 19383
InNoECTPkts: 574627
MPTcpExt:
# ethtool -S enp2s0 | grep rx
rx_packets: 561920
rx_bytes: 59893407
rx_broadcast: 5508
rx_multicast: 556412
rx_crc_errors: 0
rx_no_buffer_count: 0
rx_missed_errors: 0
rx_long_length_errors: 0
rx_short_length_errors: 0
rx_align_errors: 0
rx_flow_control_xon: 0
rx_flow_control_xoff: 0
rx_long_byte_count: 59893407
rx_smbus: 0
os2bmc_rx_by_bmc: 0
os2bmc_rx_by_host: 0
rx_hwtstamp_cleared: 0
rx_lpi_counter: 0
rx_errors: 0
rx_length_errors: 0
rx_over_errors: 0
rx_frame_errors: 0
rx_fifo_errors: 0
rx_queue_0_packets: 561750
rx_queue_0_bytes: 57629925
rx_queue_0_drops: 0
rx_queue_0_csum_err: 0
rx_queue_0_alloc_failed: 0
rx_queue_1_packets: 0
rx_queue_1_bytes: 0
rx_queue_1_drops: 0
rx_queue_1_csum_err: 0
rx_queue_1_alloc_failed: 0
rx_queue_2_packets: 148
rx_queue_2_bytes: 13290
rx_queue_2_drops: 0
rx_queue_2_csum_err: 0
rx_queue_2_alloc_failed: 0
rx_queue_3_packets: 22
rx_queue_3_bytes: 2512
rx_queue_3_drops: 0
rx_queue_3_csum_err: 0
rx_queue_3_alloc_failed: 0
after sending:
# netstat -i
Kernel Interface table
Iface MTU RX-OK RX-ERR RX-DRP RX-OVR TX-OK TX-ERR TX-DRP TX-OVR Flg
enp1s0 1500 12465 0 0 0 618087 0 0 0 BMRU
enp2s0 1500 607349 0 0 0 4031 0 0 0 BMRU
lo 65536 12800 0 0 0 12800 0 0 0 LRU
# netstat -s -u
IcmpMsg:
InType3: 6588
OutType3: 6588
Udp:
5715 packets received
902 packets to unknown port received
0 packet receive errors
621972 packets sent
0 receive buffer errors
0 send buffer errors
IgnoredMulti: 263
UdpLite:
IpExt:
InMcastPkts: 112
OutMcastPkts: 612677
InBcastPkts: 263
InOctets: 58289081
OutOctets: 56953872
InMcastOctets: 10222
OutMcastOctets: 54527991
InBcastOctets: 19816
InNoECTPkts: 619936
MPTcpExt:
# ethtool -S enp2s0 | grep rx
rx_packets: 607351
rx_bytes: 64748001
rx_broadcast: 5666
rx_multicast: 601685
rx_crc_errors: 0
rx_no_buffer_count: 0
rx_missed_errors: 0
rx_long_length_errors: 0
rx_short_length_errors: 0
rx_align_errors: 0
rx_flow_control_xon: 0
rx_flow_control_xoff: 0
rx_long_byte_count: 64748001
rx_smbus: 0
os2bmc_rx_by_bmc: 0
os2bmc_rx_by_host: 0
rx_hwtstamp_cleared: 0
rx_lpi_counter: 0
rx_errors: 0
rx_length_errors: 0
rx_over_errors: 0
rx_frame_errors: 0
rx_fifo_errors: 0
rx_queue_0_packets: 607176
rx_queue_0_bytes: 62302224
rx_queue_0_drops: 0
rx_queue_0_csum_err: 0
rx_queue_0_alloc_failed: 0
rx_queue_1_packets: 0
rx_queue_1_bytes: 0
rx_queue_1_drops: 0
rx_queue_1_csum_err: 0
rx_queue_1_alloc_failed: 0
rx_queue_2_packets: 153
rx_queue_2_bytes: 13861
rx_queue_2_drops: 0
rx_queue_2_csum_err: 0
rx_queue_2_alloc_failed: 0
rx_queue_3_packets: 22
rx_queue_3_bytes: 2512
rx_queue_3_drops: 0
rx_queue_3_csum_err: 0
rx_queue_3_alloc_failed: 0
Dropwatch output is as followed:
# sudo dropwatch -l ksa
2 drops at igmp_rcv+10c (0xffffffff9dd7202c) [software]
1 drops at unix_stream_connect+36a (0xffffffff9ddbb10a) [software]
2 drops at ip_rcv_finish_core.constprop.0+19c (0xffffffff9dd1930c) [software]
2048 drops at ip_rcv_finish_core.constprop.0+19c (0xffffffff9dd1930c) [software]
2036 drops at ip_rcv_finish_core.constprop.0+19c (0xffffffff9dd1930c) [software]
1 drops at __udp4lib_lib_mcast_deliver+31f (0xffffffff9dd5d67f) [software]
1 drops at __udp4lib_lib_mcast_deliver+31f (0xffffffff9dd5d67f) [software]
If I run this setup (exatly same UDP packets with tcpdump) with a real second windows device, receiving works. But this "external loopback" dosn't receive anything (I want to create so a TSN setup, so the windows machine is no option).
If I don't specify the interface for receiving, I get the packets (but don't know if they come from the loopback)
Following steps I tried without success:
Disabling RP_FILTER (in any combination for all available interfaces)
promisc mode on (but the ethtool output says that there is no problem on the NIC side)
What did I missed?
Best regards,
Patrick
My goal is to send UDP multicast packets on the first interface and receive them on the second interface (for performance analysis and for simulating a current missing Master hardware).
I have an influxdb service running in my kubernetes cluster and it exposes the following two ports
---
apiVersion: v1
kind: Service
metadata:
labels:
task: influxdb
name: influxdb
namespace: my-namespace
spec:
type: NodePort
ports:
- port: 8086
name: influxdb-port
targetPort: 8086
nodePort: 30101
- port: 2003
name: graphite-port
targetPort: 2003
nodePort: 30103
selector:
k8s-app: influxdb
I also have the graphite enabled in the influxdb.conf as follows
[[graphite]]
enabled = true
bind-address = ":2003"
protocol = "tcp"
When I deploy this service in my cluster, I am able to write to the graphite database port from outside the k8s cluster as well as any of the k8s node.
echo "local.random.diceroll 4 `date +%s`" | nc -v 10.233.26.252 2003
Ncat: Version 7.50 ( https://nmap.org/ncat )
Ncat: Connected to 10.233.26.252:2003.
Ncat: 35 bytes sent, 0 bytes received in 0.01 seconds.
However if I run the same command from a client pod running in the same namespace the command hangs. From the same client pod I am able to writing the influxdb port without any issue.
Following is the tcpdump trace of the nc command being sent to the graphite port 2003 from the client pod
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on any, link-type LINUX_SLL (Linux cooked), capture size 65535 bytes
03:02:17.133367 In b2:7c:b9:93:b8:9c ethertype IPv4 (0x0800), length 76: 10.233.102.131.60272 > 10.233.26.252.cfinger: Flags [S], seq 3653742255, win 29200, options [mss 1460,sackOK,TS val 629141248 ecr 0,nop,wscale 7], length 0
03:02:17.133414 Out ethertype IPv4 (0x0800), length 76: 10.233.102.131.60272 > 10.233.71.26.cfinger: Flags [S], seq 3653742255, win 29200, options [mss 1460,sackOK,TS val 629141248 ecr 0,nop,wscale 7], length 0
03:02:17.133783 In ethertype IPv4 (0x0800), length 76: 10.233.71.26.cfinger > 10.233.102.131.60272: Flags [S.], seq 4245034624, ack 3653742256, win 28960, options [mss 1460,sackOK,TS val 629140002 ecr 629141248,nop,wscale 7], length 0
03:02:17.133791 Out ee:ee:ee:ee:ee:ee ethertype IPv4 (0x0800), length 76: 10.233.26.252.cfinger > 10.233.102.131.60272: Flags [S.], seq 4245034624, ack 3653742256, win 28960, options [mss 1460,sackOK,TS val 629140002 ecr 629141248,nop,wscale 7], length 0
03:02:17.133805 In b2:7c:b9:93:b8:9c ethertype IPv4 (0x0800), length 68: 10.233.102.131.60272 > 10.233.26.252.cfinger: Flags [.], ack 1, win 229, options [nop,nop,TS val 629141248 ecr 629140002], length 0
03:02:17.133809 Out ethertype IPv4 (0x0800), length 68: 10.233.102.131.60272 > 10.233.71.26.cfinger: Flags [.], ack 1, win 229, options [nop,nop,TS val 629141248 ecr 629140002], length 0
03:02:17.134036 In b2:7c:b9:93:b8:9c ethertype IPv4 (0x0800), length 103: 10.233.102.131.60272 > 10.233.26.252.cfinger: Flags [P.], seq 1:36, ack 1, win 229, options [nop,nop,TS val 629141249 ecr 629140002], length 35
03:02:17.134049 Out ethertype IPv4 (0x0800), length 103: 10.233.102.131.60272 > 10.233.71.26.cfinger: Flags [P.], seq 1:36, ack 1, win 229, options [nop,nop,TS val 629141249 ecr 629140002], length 35
03:02:17.134235 In ethertype IPv4 (0x0800), length 68: 10.233.71.26.cfinger > 10.233.102.131.60272: Flags [.], ack 36, win 227, options [nop,nop,TS val 629140002 ecr 629141249], length 0
03:02:17.134258 Out ee:ee:ee:ee:ee:ee ethertype IPv4 (0x0800), length 68: 10.233.26.252.cfinger > 10.233.102.131.60272: Flags [.], ack 36, win 227, options [nop,nop,TS val 629140002 ecr 629141249], length 0
^C
10 packets captured
10 packets received by filter
I had to use the -q0 option instead of -v. The -q0 option makes sure that the netcat connection is closed after sending the data.
My Vagrantfile
hosts = {
"host01" => "192.168.11.101",
"host02" => "192.168.11.102",
}
Vagrant.configure("2") do |config|
config.ssh.username = "root"
config.ssh.password = "vagrant"
config.ssh.insert_key = "true"
hosts.each_with_index do |(name,ip),index|
config.vm.define name do |machine|
machine.vm.box = "centos7"
machine.vm.box_check_update = false
machine.vm.hostname = name
machine.vm.synced_folder "/data", "/data"
machine.vm.network :private_network, ip: ip
machine.vm.provider "virtualbox" do |v|
v.name = name
v.customize ["modifyvm", :id, "--memory", 2048]
end
end
end
end
ansible tamplate for generate /etc/host
127.0.0.1 localhost
{% for host in groups['all'] %}
{{ hostvars[host]['ansible_' + iface].ipv4.address }} {{ host }}
{% endfor %}
ansible task
- name: Create the hosts file for all machines
template: src=hosts.j2 dest=/etc/hosts
But I get the result
[root#host01 ~]# cat /etc/hosts
127.0.0.1 localhost
10.0.2.15 host01
10.0.2.15 host02
ifconfig
docker0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 172.17.0.1 netmask 255.255.0.0 broadcast 0.0.0.0
inet6 fe80::42:49ff:fed1:eebb prefixlen 64 scopeid 0x20<link>
ether 02:42:49:d1:ee:bb txqueuelen 0 (Ethernet)
RX packets 77 bytes 6065 (5.9 KiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 99 bytes 8572 (8.3 KiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
enp0s3: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 10.0.2.15 netmask 255.255.255.0 broadcast 10.0.2.255
inet6 fe80::a00:27ff:fede:e0e prefixlen 64 scopeid 0x20<link>
ether 08:00:27:de:0e:0e txqueuelen 1000 (Ethernet)
RX packets 785483 bytes 57738892 (55.0 MiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 777457 bytes 1957320412 (1.8 GiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
enp0s8: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 192.168.11.101 netmask 255.255.255.0 broadcast 192.168.11.255
ether 08:00:27:15:2c:64 txqueuelen 1000 (Ethernet)
RX packets 41445 bytes 39878552 (38.0 MiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 18055 bytes 2113498 (2.0 MiB)
I found in ifconfig, only enp0s8 inet 192.168.11.102 is different on host01 and host02
host01 and host02 got same ip ??
host01 has a docker registry.
host01, curl http://host01:5006/v2/_catalog works
host02, curl http://host01:5006/v2/_catalog not work
host01 and host02 got same ip ??
Yes. That's how Vagrant works and is able to connect for orchestration purposes to machines created from a variety of boxes coming from different publishers.
There's nothing strange about that, they are connected to different virtual switches in VirtualBox.
I just want host01 and host02 can access each other by hostname .
Use the other interface as the value of iface in your Jinja2 template (you did not show it in the question anyway).
This question is unlikely to help any future visitors; it is only relevant to a small geographic area, a specific moment in time, or an extraordinarily narrow situation that is not generally applicable to the worldwide audience of the internet. For help making this question more broadly applicable, visit the help center.
Closed 10 years ago.
FIXED!
It turned out it was caused by our firewall which would remove idle connections from its session list after 1 hour. We increased the timeout to 24 hours and setup a eventmachine periodic timer to reconnect the connection every 23 hours. That's a workaround until eventmachine 1.0.0 hits stable which will allow us to set SO_KEEPALIVE which should resolve our issues
The Problem was
We are using an eventmachine based implementation to push messages to the apple apns. It basically works nicely until it doesn't... :) We were trying to debug this quite hevily and now narrowed it down to something strange happening on the socket to apple after a while.
so, usually, until you send a notification to the apns server, the socket is completely quiet.
If you send a notification this is what tcpdump spits out (sudo tcpdump -vv -i bond0 tcp port 2195):
18:05:23.672477 IP (tos 0x0, ttl 64, id 47828, offset 0, flags [DF], proto TCP (6), length 489) my-worker-hostname.50669 > st11p01st-interface013-bz.push.apple.com.2195: P 3894:4331(437) ack 2724 win 91 <nop,nop,timestamp 893114182 2332608880>
18:05:23.776055 IP (tos 0x0, ttl 48, id 33720, offset 0, flags [DF], proto TCP (6), length 52) st11p01st-interface013-bz.push.apple.com.2195 > my-worker-hostname.50669: ., cksum 0x7844 (correct), 2724:2724(0) ack 4331 win 159 <nop,nop,timestamp 2332623235 893114182>
Nothing suspicious so far imho.
However, after a while (a while being a random amount of time) our worker process starts sending packets to the apple server every 1-2 minutes even though no push notification has been triggered by us:
17:55:06.009741 IP (tos 0x0, ttl 64, id 51807, offset 0, flags [DF], proto TCP (6), length 489) my-worker-hostname.54853 > st11p01st-interface002-bz.push.apple.com.2195: P 0:437(437) ack 1 win 91 <nop,nop,timestamp 892959766 2935299208>
17:56:25.881823 IP (tos 0x0, ttl 64, id 51808, offset 0, flags [DF], proto TCP (6), length 489) my-worker-hostname.54853 > st11p01st-interface002-bz.push.apple.com.2195: P 0:437(437) ack 1 win 91 <nop,nop,timestamp 892979734 2935299208>
17:58:25.877756 IP (tos 0x0, ttl 64, id 51809, offset 0, flags [DF], proto TCP (6), length 489) my-worker-hostname.54853 > st11p01st-interface002-bz.push.apple.com.2195: P 0:437(437) ack 1 win 91 <nop,nop,timestamp 893009734 2935299208>
17:59:12.030887 IP (tos 0x0, ttl 64, id 20781, offset 0, flags [DF], proto TCP (6), length 489) my-worker-hostname.59335 > st11p01st-interface013-bz.push.apple.com.2195: P 3749093679:3749094116(437) ack 4206642630 win 91 <nop,nop,timestamp 893021272 2330366860>
17:59:12.345740 IP (tos 0x0, ttl 64, id 20782, offset 0, flags [DF], proto TCP (6), length 489) my-worker-hostname.59335 > st11p01st-interface013-bz.push.apple.com.2195: P 0:437(437) ack 1 win 91 <nop,nop,timestamp 893021351 2330366860>
17:59:12.977805 IP (tos 0x0, ttl 64, id 20783, offset 0, flags [DF], proto TCP (6), length 489) my-worker-hostname.59335 > st11p01st-interface013-bz.push.apple.com.2195: P 0:437(437) ack 1 win 91 <nop,nop,timestamp 893021509 2330366860>
As soon as this starts, the push notifications don't work anymore until we restart the worker.
I'm running out of ideas here...
UPDATE:
after waiting another while without sending notifications and nothing happening on the socket at all I've just initiated another push, this caused the described behavior once again:
19:10:44.951026 IP (tos 0x0, ttl 64, id 47829, offset 0, flags [DF], proto TCP (6), length 489) my-worker-hostname.50669 > st11p01st-interface013-bz.push.apple.com.2195: P 4331:4768(437) ack 2724 win 91 <nop,nop,timestamp 894094502 2332623235>
19:10:45.361786 IP (tos 0x0, ttl 64, id 47830, offset 0, flags [DF], proto TCP (6), length 489) my-worker-hostname.50669 > st11p01st-interface013-bz.push.apple.com.2195: P 4331:4768(437) ack 2724 win 91 <nop,nop,timestamp 894094605 2332623235>
19:10:46.185822 IP (tos 0x0, ttl 64, id 47831, offset 0, flags [DF], proto TCP (6), length 489) my-worker-hostname.50669 > st11p01st-interface013-bz.push.apple.com.2195: P 4331:4768(437) ack 2724 win 91 <nop,nop,timestamp 894094811 2332623235>
19:10:47.837788 IP (tos 0x0, ttl 64, id 47832, offset 0, flags [DF], proto TCP (6), length 489) my-worker-hostname.50669 > st11p01st-interface013-bz.push.apple.com.2195: P 4331:4768(437) ack 2724 win 91 <nop,nop,timestamp 894095223 2332623235>
19:10:51.133744 IP (tos 0x0, ttl 64, id 47833, offset 0, flags [DF], proto TCP (6), length 489) my-worker-hostname.50669 > st11p01st-interface013-bz.push.apple.com.2195: P 4331:4768(437) ack 2724 win 91 <nop,nop,timestamp 894096048 2332623235>
19:10:57.725824 IP (tos 0x0, ttl 64, id 47834, offset 0, flags [DF], proto TCP (6), length 489) my-worker-hostname.50669 > st11p01st-interface013-bz.push.apple.com.2195: P 4331:4768(437) ack 2724 win 91 <nop,nop,timestamp 894097696 2332623235>
19:11:10.913826 IP (tos 0x0, ttl 64, id 47835, offset 0, flags [DF], proto TCP (6), length 489) my-worker-hostname.50669 > st11p01st-interface013-bz.push.apple.com.2195: P 4331:4768(437) ack 2724 win 91 <nop,nop,timestamp 894100992 2332623235>