ESXi 6.0 CENTOS 7 VM network issues

[esxi77]

Setup: VMWare ESXi 6.0 Update 2, with a Centos 7 guest VM using vmic1

Problem: Centos 7 cannot ping default gateway, but can ping other hosts on the same network. As seen from the output below (both Centos and Windows10) these hosts all report the same MAC for the FW (default gateway) in their ARP tables. The Centos host pings the Windows box with no problem and vice-versa. Win10 has no issues pinging the default gateway either. Furthermore, the mac address for the Centos box is also listed in the FW's ARP table at the bottom. Can someone help identify the problem here?

!===============Centos

[root@cloud ~]# ifconfig

eno16777984: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500

        inet 192.168.255.10  netmask 255.255.255.0  broadcast 192.168.255.255

        inet6 fe80::20c:29ff:fe0f:5e8b  prefixlen 64  scopeid 0x20<link>

        ether 00:0c:29:0f:5e:8b  txqueuelen 1000  (Ethernet)

        RX packets 12  bytes 1410 (1.3 KiB)

        RX errors 0  dropped 0  overruns 0  frame 0

        TX packets 62  bytes 6726 (6.5 KiB)

        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

[root@cloud ~]# arp -n

Address                  HWtype  HWaddress           Flags Mask            Iface

192.168.255.1            ether   84:b5:9c:2c:40:50  C                     eno16777984

[root@cloud ~]# route -n

Kernel IP routing table

Destination     Gateway         Genmask         Flags Metric Ref    Use Iface

0.0.0.0         192.168.255.1   0.0.0.0         UG    0      0        0 eno16777984

169.254.0.0     0.0.0.0         255.255.0.0     U     1002   0        0 eno16777984

192.168.122.0   0.0.0.0         255.255.255.0   U     0      0        0 virbr0

192.168.255.0   0.0.0.0         255.255.255.0   U     0      0        0 eno16777984

[root@cloud ~]# ping 192.168.255.101

PING 192.168.255.101 (192.168.255.101) 56(84) bytes of data.

64 bytes from 192.168.255.101: icmp_seq=1 ttl=128 time=0.544 ms

64 bytes from 192.168.255.101: icmp_seq=2 ttl=128 time=0.474 ms

64 bytes from 192.168.255.101: icmp_seq=3 ttl=128 time=0.436 ms

[root@cloud ~]# firewall-cmd --state

not running

[root@cloud ~]#

[root@cloud ~]# cat /etc/sysconfig/network-scripts/ifcfg-eno16777984

TYPE=Ethernet

BOOTPROTO=static

DEFROUTE=yes

PEERDNS=yes

PEERROUTES=yes

IPV4_FAILURE_FATAL=no

IPV6INIT=yes

IPV6_AUTOCONF=yes

IPV6_DEFROUTE=yes

IPV6_PEERDNS=yes

IPV6_PEERROUTES=yes

IPV6_FAILURE_FATAL=no

NAME=eno16777984

UUID=c6a00e9f-ab1f-4a5b-92fa-13b15aff8cb8

DEVICE=eno16777984

ONBOOT=yes

HWADDR=00:0c:29:0f:5e:8b

IPADDR=192.168.255.10

NETMASK=255.255.255.0

GATEWAY=192.168.255.1

NETWORK=192.168.255.0

DNS1=8.8.8.8

[root@cloud ~]#

!============= Windows 10

Windows IP Configuration

Ethernet adapter INET:

   Connection-specific DNS Suffix  . :
   Link-local IPv6 Address . . . . . : fe80::3ca2:6e9d:7a95:c9fc%28
   IPv4 Address. . . . . . . . . . . : 192.168.255.101
   Subnet Mask . . . . . . . . . . . : 255.255.255.0
   Default Gateway . . . . . . . . . : 192.168.255.1

C:\Users\admin>arp -a

Interface: 192.168.255.101 --- 0x1c
  Internet Address      Physical Address      Type
  192.168.255.1         84-b5-9c-2c-40-50     dynamic
  192.168.255.10        00-0c-29-0f-5e-8b     dynamic
  192.168.255.255       ff-ff-ff-ff-ff-ff     static
  224.0.0.22            01-00-5e-00-00-16     static
  224.0.0.251           01-00-5e-00-00-fb     static
  224.0.0.252           01-00-5e-00-00-fc     static

<ping from Win10 to Centos>

C:\Users\admin>ping 192.168.255.10

Pinging 192.168.255.10 with 32 bytes of data:
Reply from 192.168.255.10: bytes=32 time<1ms TTL=64
Reply from 192.168.255.10: bytes=32 time<1ms TTL=64

<ping from Win10 to gateway>

C:\Users\admin>ping 192.168.255.10

Pinging 192.168.255.10 with 32 bytes of data:
Reply from 192.168.255.10: bytes=32 time<1ms TTL=64
Reply from 192.168.255.10: bytes=32 time<1ms TTL=64

!================Firewall (default gateway)

root@MyDoorMat> show arp

MAC Address       Address         Name                      Interface           Flags

00:0c:29:0f:5e:8b 192.168.255.10  192.168.255.10            vlan.255            none

bc:5f:f4:ea:28:64 192.168.255.101 192.168.255.101           vlan.255            none

[vHaridas]

<ping from Win10 to gateway>

C:\Users\admin>ping 192.168.255.10

Pinging 192.168.255.10 with 32 bytes of data:
Reply from 192.168.255.10: bytes=32 time<1ms TTL=64
Reply from 192.168.255.10: bytes=32 time<1ms TTL=64

above, you pinged to CentOS from Win10 not gateway.

can you ping to gateway 192.168.255.1 from win10 and centOS and share output?

-

Haridas

[esxi77]

!======Centos

[root@cloud ~]# ping 192.168.255.1
PING 192.168.255.1 (192.168.255.1) 56(84) bytes of data.
^C
--- 192.168.255.1 ping statistics ---
5 packets transmitted, 0 received, 100% packet loss, time 3999ms

!======Windows

C:\Users\admin>ping 192.168.255.1

Pinging 192.168.255.1 with 32 bytes of data:
Reply from 192.168.255.1: bytes=32 time<1ms TTL=64
Reply from 192.168.255.1: bytes=32 time<1ms TTL=64
Reply from 192.168.255.1: bytes=32 time<1ms TTL=64
Reply from 192.168.255.1: bytes=32 time<1ms TTL=64

Ping statistics for 192.168.255.1:
    Packets: Sent = 4, Received = 4, Lost = 0 (0% loss),
Approximate round trip times in milli-seconds:
    Minimum = 0ms, Maximum = 0ms, Average = 0ms

!=========Firewall

root@MyDoorMat> ping 192.168.255.101
PING 192.168.255.101 (192.168.255.101): 56 data bytes
64 bytes from 192.168.255.101: icmp_seq=0 ttl=128 time=1.636 ms
64 bytes from 192.168.255.101: icmp_seq=1 ttl=128 time=1.724 ms
64 bytes from 192.168.255.101: icmp_seq=2 ttl=128 time=1.635 ms
^C
--- 192.168.255.101 ping statistics ---
3 packets transmitted, 3 packets received, 0% packet loss
round-trip min/avg/max/stddev = 1.635/1.665/1.724/0.042 ms

root@MyDoorMat> ping 192.168.255.10
PING 192.168.255.10 (192.168.255.10): 56 data bytes
^C
--- 192.168.255.10 ping statistics ---
5 packets transmitted, 0 packets received, 100% packet loss

root@MyDoorMat>

[vHaridas]

are you running all three VMs in same ESXi host?

You have connected all VMs to same PortGroup?

are you using VLANs?

-

Haridas Vhadade

[esxi77]

Here is a shot of the vSphere, vSwitch3 Properties that show the detection of the Windows10 host in the picture above. Note* only the Windows10 host is detected, even though the ARP table in Centos shows both the FW and the Windows10 box.

TCPDUMP output

[root@cloud ~]# tcpdump -i eno16777984

tcpdump: verbose output suppressed, use -v or -vv for full protocol decode

listening on eno16777984, link-type EN10MB (Ethernet), capture size 65535 bytes

19:26:48.886945 IP 192.168.255.101.db-lsp-disc > 192.168.255.255.db-lsp-disc: UDP, length 156

19:26:48.887622 IP 192.168.255.10.49434 > 8.8.8.8.domain: 59355+ PTR? 255.255.168.192.in-addr.arpa. (46)

19:26:53.892742 IP 192.168.255.10.49434 > 8.8.8.8.domain: 59355+ PTR? 255.255.168.192.in-addr.arpa. (46)

19:26:53.895343 ARP, Request who-has 192.168.255.1 tell 192.168.255.10, length 28

19:26:53.896401 ARP, Reply 192.168.255.1 is-at 84:b5:9c:2c:40:50 (oui Unknown), length 46

19:26:58.897562 IP 192.168.255.10.40117 > 8.8.8.8.domain: 61171+ PTR? 101.255.168.192.in-addr.arpa. (46)

19:27:03.899392 IP 192.168.255.10.40117 > 8.8.8.8.domain: 61171+ PTR? 101.255.168.192.in-addr.arpa. (46)

19:27:08.728137 IP 192.168.255.101 > 192.168.255.10: ICMP echo request, id 5, seq 63016, length 40

19:27:08.728211 IP 192.168.255.10 > 192.168.255.101: ICMP echo reply, id 5, seq 63016, length 40

19:27:08.904814 IP 192.168.255.10.39270 > 8.8.8.8.domain: 7460+ PTR? 8.8.8.8.in-addr.arpa. (38)

19:27:09.730404 IP 192.168.255.101 > 192.168.255.10: ICMP echo request, id 5, seq 63018, length 40

19:27:09.730451 IP 192.168.255.10 > 192.168.255.101: ICMP echo reply, id 5, seq 63018, length 40

19:27:10.736274 IP 192.168.255.101 > 192.168.255.10: ICMP echo request, id 5, seq 63019, length 40

19:27:10.736319 IP 192.168.255.10 > 192.168.255.101: ICMP echo reply, id 5, seq 63019, length 40

19:27:11.742474 IP 192.168.255.101 > 192.168.255.10: ICMP echo request, id 5, seq 63021, length 40

19:27:11.742522 IP 192.168.255.10 > 192.168.255.101: ICMP echo reply, id 5, seq 63021, length 40

19:27:13.462554 ARP, Request who-has 192.168.255.10 (00:0c:29:0f:5e:8b (oui Unknown)) tell 192.168.255.101, length 46

19:27:13.462579 ARP, Reply 192.168.255.10 is-at 00:0c:29:0f:5e:8b (oui Unknown), length 28

19:27:13.908396 IP 192.168.255.10.39270 > 8.8.8.8.domain: 7460+ PTR? 8.8.8.8.in-addr.arpa. (38)

19:27:18.913739 IP 192.168.255.10.33051 > 8.8.8.8.domain: 30597+ PTR? 10.255.168.192.in-addr.arpa. (45)

19:27:18.919347 ARP, Request who-has 192.168.255.1 tell 192.168.255.10, length 28

19:27:18.920431 ARP, Reply 192.168.255.1 is-at 84:b5:9c:2c:40:50 (oui Unknown), length 46

19:27:18.953719 IP 192.168.255.101.db-lsp-disc > 192.168.255.255.db-lsp-disc: UDP, length 156

19:27:23.918329 IP 192.168.255.10.33051 > 8.8.8.8.domain: 30597+ PTR? 10.255.168.192.in-addr.arpa. (45)

19:27:28.923428 IP 192.168.255.10.38986 > 8.8.8.8.domain: 21924+ PTR? 1.255.168.192.in-addr.arpa. (44)

19:27:33.925394 IP 192.168.255.10.38986 > 8.8.8.8.domain: 21924+ PTR? 1.255.168.192.in-addr.arpa. (44)

19:27:49.020404 IP 192.168.255.101.db-lsp-disc > 192.168.255.255.db-lsp-disc: UDP, length 156

^C

27 packets captured

27 packets received by filter

0 packets dropped by kernel

[root@cloud ~]#

[esxi77 · Accepted solution]

since the IP was reallocated unknowingly, there was remnant proxy-arp and src-nat pools utilizing this 192.168.255.10 IP. After ripping it out, works fine. I setup a pcap fw filter to catch the packets, and threw it in wireshark and sure enough the packet made it to the firewall, but it gave no response. I should have done this earlier, but relied on the security flow session monitor instead, which apparently doesn't show flows in which replies are not sent.