Setup: VMWare ESXi 6.0 Update 2, with a Centos 7 guest VM using vmic1
Problem: Centos 7 cannot ping default gateway, but can ping other hosts on the same network. As seen from the output below (both Centos and Windows10) these hosts all report the same MAC for the FW (default gateway) in their ARP tables. The Centos host pings the Windows box with no problem and vice-versa. Win10 has no issues pinging the default gateway either. Furthermore, the mac address for the Centos box is also listed in the FW's ARP table at the bottom. Can someone help identify the problem here?
!===============Centos
[root@cloud ~]# ifconfig
eno16777984: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 192.168.255.10 netmask 255.255.255.0 broadcast 192.168.255.255
inet6 fe80::20c:29ff:fe0f:5e8b prefixlen 64 scopeid 0x20<link>
ether 00:0c:29:0f:5e:8b txqueuelen 1000 (Ethernet)
RX packets 12 bytes 1410 (1.3 KiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 62 bytes 6726 (6.5 KiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
[root@cloud ~]# arp -n
Address HWtype HWaddress Flags Mask Iface
192.168.255.1 ether 84:b5:9c:2c:40:50 C eno16777984
[root@cloud ~]# route -n
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
0.0.0.0 192.168.255.1 0.0.0.0 UG 0 0 0 eno16777984
169.254.0.0 0.0.0.0 255.255.0.0 U 1002 0 0 eno16777984
192.168.122.0 0.0.0.0 255.255.255.0 U 0 0 0 virbr0
192.168.255.0 0.0.0.0 255.255.255.0 U 0 0 0 eno16777984
[root@cloud ~]# ping 192.168.255.101
PING 192.168.255.101 (192.168.255.101) 56(84) bytes of data.
64 bytes from 192.168.255.101: icmp_seq=1 ttl=128 time=0.544 ms
64 bytes from 192.168.255.101: icmp_seq=2 ttl=128 time=0.474 ms
64 bytes from 192.168.255.101: icmp_seq=3 ttl=128 time=0.436 ms
[root@cloud ~]# firewall-cmd --state
not running
[root@cloud ~]#
[root@cloud ~]# cat /etc/sysconfig/network-scripts/ifcfg-eno16777984
TYPE=Ethernet
BOOTPROTO=static
DEFROUTE=yes
PEERDNS=yes
PEERROUTES=yes
IPV4_FAILURE_FATAL=no
IPV6INIT=yes
IPV6_AUTOCONF=yes
IPV6_DEFROUTE=yes
IPV6_PEERDNS=yes
IPV6_PEERROUTES=yes
IPV6_FAILURE_FATAL=no
NAME=eno16777984
UUID=c6a00e9f-ab1f-4a5b-92fa-13b15aff8cb8
DEVICE=eno16777984
ONBOOT=yes
HWADDR=00:0c:29:0f:5e:8b
IPADDR=192.168.255.10
NETMASK=255.255.255.0
GATEWAY=192.168.255.1
NETWORK=192.168.255.0
DNS1=8.8.8.8
[root@cloud ~]#
!============= Windows 10
Windows IP Configuration
Ethernet adapter INET:
Connection-specific DNS Suffix . :
Link-local IPv6 Address . . . . . : fe80::3ca2:6e9d:7a95:c9fc%28
IPv4 Address. . . . . . . . . . . : 192.168.255.101
Subnet Mask . . . . . . . . . . . : 255.255.255.0
Default Gateway . . . . . . . . . : 192.168.255.1
C:\Users\admin>arp -a
Interface: 192.168.255.101 --- 0x1c
Internet Address Physical Address Type
192.168.255.1 84-b5-9c-2c-40-50 dynamic
192.168.255.10 00-0c-29-0f-5e-8b dynamic
192.168.255.255 ff-ff-ff-ff-ff-ff static
224.0.0.22 01-00-5e-00-00-16 static
224.0.0.251 01-00-5e-00-00-fb static
224.0.0.252 01-00-5e-00-00-fc static
<ping from Win10 to Centos>
C:\Users\admin>ping 192.168.255.10
Pinging 192.168.255.10 with 32 bytes of data:
Reply from 192.168.255.10: bytes=32 time<1ms TTL=64
Reply from 192.168.255.10: bytes=32 time<1ms TTL=64
<ping from Win10 to gateway>
C:\Users\admin>ping 192.168.255.10
Pinging 192.168.255.10 with 32 bytes of data:
Reply from 192.168.255.10: bytes=32 time<1ms TTL=64
Reply from 192.168.255.10: bytes=32 time<1ms TTL=64
!================Firewall (default gateway)
root@MyDoorMat> show arp
MAC Address Address Name Interface Flags
00:0c:29:0f:5e:8b 192.168.255.10 192.168.255.10 vlan.255 none
bc:5f:f4:ea:28:64 192.168.255.101 192.168.255.101 vlan.255 none
since the IP was reallocated unknowingly, there was remnant proxy-arp and src-nat pools utilizing this 192.168.255.10 IP. After ripping it out, works fine. I setup a pcap fw filter to catch the packets, and threw it in wireshark and sure enough the packet made it to the firewall, but it gave no response. I should have done this earlier, but relied on the security flow session monitor instead, which apparently doesn't show flows in which replies are not sent.
<ping from Win10 to gateway>
C:\Users\admin>ping 192.168.255.10
Pinging 192.168.255.10 with 32 bytes of data:
Reply from 192.168.255.10: bytes=32 time<1ms TTL=64
Reply from 192.168.255.10: bytes=32 time<1ms TTL=64
above, you pinged to CentOS from Win10 not gateway.
can you ping to gateway 192.168.255.1 from win10 and centOS and share output?
-
Haridas
!======Centos
[root@cloud ~]# ping 192.168.255.1
PING 192.168.255.1 (192.168.255.1) 56(84) bytes of data.
^C
--- 192.168.255.1 ping statistics ---
5 packets transmitted, 0 received, 100% packet loss, time 3999ms
!======Windows
C:\Users\admin>ping 192.168.255.1
Pinging 192.168.255.1 with 32 bytes of data:
Reply from 192.168.255.1: bytes=32 time<1ms TTL=64
Reply from 192.168.255.1: bytes=32 time<1ms TTL=64
Reply from 192.168.255.1: bytes=32 time<1ms TTL=64
Reply from 192.168.255.1: bytes=32 time<1ms TTL=64
Ping statistics for 192.168.255.1:
Packets: Sent = 4, Received = 4, Lost = 0 (0% loss),
Approximate round trip times in milli-seconds:
Minimum = 0ms, Maximum = 0ms, Average = 0ms
!=========Firewall
root@MyDoorMat> ping 192.168.255.101
PING 192.168.255.101 (192.168.255.101): 56 data bytes
64 bytes from 192.168.255.101: icmp_seq=0 ttl=128 time=1.636 ms
64 bytes from 192.168.255.101: icmp_seq=1 ttl=128 time=1.724 ms
64 bytes from 192.168.255.101: icmp_seq=2 ttl=128 time=1.635 ms
^C
--- 192.168.255.101 ping statistics ---
3 packets transmitted, 3 packets received, 0% packet loss
round-trip min/avg/max/stddev = 1.635/1.665/1.724/0.042 ms
root@MyDoorMat> ping 192.168.255.10
PING 192.168.255.10 (192.168.255.10): 56 data bytes
^C
--- 192.168.255.10 ping statistics ---
5 packets transmitted, 0 packets received, 100% packet loss
root@MyDoorMat>
are you running all three VMs in same ESXi host?
You have connected all VMs to same PortGroup?
are you using VLANs?
-
Haridas Vhadade
Here is a shot of the vSphere, vSwitch3 Properties that show the detection of the Windows10 host in the picture above. Note* only the Windows10 host is detected, even though the ARP table in Centos shows both the FW and the Windows10 box.
TCPDUMP output
[root@cloud ~]# tcpdump -i eno16777984
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eno16777984, link-type EN10MB (Ethernet), capture size 65535 bytes
19:26:48.886945 IP 192.168.255.101.db-lsp-disc > 192.168.255.255.db-lsp-disc: UDP, length 156
19:26:48.887622 IP 192.168.255.10.49434 > 8.8.8.8.domain: 59355+ PTR? 255.255.168.192.in-addr.arpa. (46)
19:26:53.892742 IP 192.168.255.10.49434 > 8.8.8.8.domain: 59355+ PTR? 255.255.168.192.in-addr.arpa. (46)
19:26:53.895343 ARP, Request who-has 192.168.255.1 tell 192.168.255.10, length 28
19:26:53.896401 ARP, Reply 192.168.255.1 is-at 84:b5:9c:2c:40:50 (oui Unknown), length 46
19:26:58.897562 IP 192.168.255.10.40117 > 8.8.8.8.domain: 61171+ PTR? 101.255.168.192.in-addr.arpa. (46)
19:27:03.899392 IP 192.168.255.10.40117 > 8.8.8.8.domain: 61171+ PTR? 101.255.168.192.in-addr.arpa. (46)
19:27:08.728137 IP 192.168.255.101 > 192.168.255.10: ICMP echo request, id 5, seq 63016, length 40
19:27:08.728211 IP 192.168.255.10 > 192.168.255.101: ICMP echo reply, id 5, seq 63016, length 40
19:27:08.904814 IP 192.168.255.10.39270 > 8.8.8.8.domain: 7460+ PTR? 8.8.8.8.in-addr.arpa. (38)
19:27:09.730404 IP 192.168.255.101 > 192.168.255.10: ICMP echo request, id 5, seq 63018, length 40
19:27:09.730451 IP 192.168.255.10 > 192.168.255.101: ICMP echo reply, id 5, seq 63018, length 40
19:27:10.736274 IP 192.168.255.101 > 192.168.255.10: ICMP echo request, id 5, seq 63019, length 40
19:27:10.736319 IP 192.168.255.10 > 192.168.255.101: ICMP echo reply, id 5, seq 63019, length 40
19:27:11.742474 IP 192.168.255.101 > 192.168.255.10: ICMP echo request, id 5, seq 63021, length 40
19:27:11.742522 IP 192.168.255.10 > 192.168.255.101: ICMP echo reply, id 5, seq 63021, length 40
19:27:13.462554 ARP, Request who-has 192.168.255.10 (00:0c:29:0f:5e:8b (oui Unknown)) tell 192.168.255.101, length 46
19:27:13.462579 ARP, Reply 192.168.255.10 is-at 00:0c:29:0f:5e:8b (oui Unknown), length 28
19:27:13.908396 IP 192.168.255.10.39270 > 8.8.8.8.domain: 7460+ PTR? 8.8.8.8.in-addr.arpa. (38)
19:27:18.913739 IP 192.168.255.10.33051 > 8.8.8.8.domain: 30597+ PTR? 10.255.168.192.in-addr.arpa. (45)
19:27:18.919347 ARP, Request who-has 192.168.255.1 tell 192.168.255.10, length 28
19:27:18.920431 ARP, Reply 192.168.255.1 is-at 84:b5:9c:2c:40:50 (oui Unknown), length 46
19:27:18.953719 IP 192.168.255.101.db-lsp-disc > 192.168.255.255.db-lsp-disc: UDP, length 156
19:27:23.918329 IP 192.168.255.10.33051 > 8.8.8.8.domain: 30597+ PTR? 10.255.168.192.in-addr.arpa. (45)
19:27:28.923428 IP 192.168.255.10.38986 > 8.8.8.8.domain: 21924+ PTR? 1.255.168.192.in-addr.arpa. (44)
19:27:33.925394 IP 192.168.255.10.38986 > 8.8.8.8.domain: 21924+ PTR? 1.255.168.192.in-addr.arpa. (44)
19:27:49.020404 IP 192.168.255.101.db-lsp-disc > 192.168.255.255.db-lsp-disc: UDP, length 156
^C
27 packets captured
27 packets received by filter
0 packets dropped by kernel
[root@cloud ~]#
since the IP was reallocated unknowingly, there was remnant proxy-arp and src-nat pools utilizing this 192.168.255.10 IP. After ripping it out, works fine. I setup a pcap fw filter to catch the packets, and threw it in wireshark and sure enough the packet made it to the firewall, but it gave no response. I should have done this earlier, but relied on the security flow session monitor instead, which apparently doesn't show flows in which replies are not sent.