VMWARE ESXi 7.0 hosts cannot communicate on same vlan but can to vlans

[bwg1234]

So I have 2 VMware esxi hosts that are not connected to vcenter. They are completely independent. I have VMware 1 VM host configured with VLAN 363 with an IP of 10.236.3.1 and I have Vmware 2 VM host with ip 10.236.3.2. I cannot ping from .1 to .2 or .2 to .1 but can ping everything in the VLAN that is not on the ESXi hosts. VMWARE 1 hosts that are on 363 can ping other VLAN 363 hosts sitting on VMWARE 1. Same for VMWARE 2. Both servers are identical. Both are using the same Broadcom dual port 10gig fiber ports and both have virtual switches with the same virtual nic applied. The physical nics both plug into the same physical cisco switch. Subnet is /24.  I have been on the phone with Vmware support for 6 hours and have no resolution. Any assistance would be greatly appreciated. 

[scott28tt]

You’ve not said anything about subnet masks, NICs, physical switch ports, how your virtual switches are configured, and probably more things that could help others to help you…

[bwg1234]

Let me clarify some other details. Both servers are identical. Both are using the same Broadcom dual port 10gig fiber ports and both have virtual switches with the same virtual nic applied. The physical nics both plug into the same physical cisco switch. Subnet is /24. 

[netgolf324]

Are you port channeling on the switch? Have you checked your arp tables on the switch? 

[a_p_]

When you say "VMWARE 1 hosts that are on 363 can ping other VLAN 363 hosts sitting on VMWARE 1." I assume that you are talking about VMs rather than hosts, right?
So what I understand is that the VMs on the same host (ans same port group) can communicate between each other, but not with anything outside the host.

Please check the physical switch settings, to ensure that it doesn't have port security enabled, which limits the number of allowed MAC addresses on a single port. In doubt provide the show run output for the ESXi host's interfaces on the Cisco switch.

André

[srsns]

To add some additional clarification to bwg1234's comments:

1. Guest VMs on both hosts can communicate to everything else on VLAN 363, including guest VMs on the third, unrelated ESXi host.
2. ARP entries for guest VMs on the other host are NOT populated in the ARP table for the local guest VM.
3. If you configure static ARP entries on the guest VM on each host pointing to the guest VM on the other host, then the VMs start communicating with each other.

This implies several things:

1. The trunking configuration, VLAN configuration, and IP addressing are all configured properly.
2. The problem appears to be directly related to broadcast traffic, but not all broadcast traffic.
3. ONLY broadcast traffic between guest VM's on these two hosts seems to be impacted.

Is there any type of identifier or configuration that could be causing a conflict for broadcast traffic between VMs on these two independent hosts?

Note: I'm the network guy.  Any VMware specific information will come from bwg1234

Thanks,
Blake

[srsns]

@a_p_ 

Thanks for the reply.  To clarify, guest VM's can communicate with other guest VMs on the same host.  They can also communicate with any other IP on the 363 VLAN except guest VM's on the second ESXi host in question.

[netgolf324]

I think switch configurations would be helpful to see. Also, the vswitch settings. Like a_p said, I would would look at the port configuration. Is promiscuous mode enabled? 

[bwg1234]

Promiscuous is currently set to reject on both v-switches. We have tried both settings though and we still get the same results. 

[srsns]

Switch configurations are extremely simple.  The interface is trunked with VLAN 363 tagged on the interface and included in spanning-tree on both ports.

Again, the guest VMs can communicated with every other IP on the same VLAN except guests that reside on the other host.  This is true for both hosts.

Regarding promiscuous mode, this would allow the NIC to receive and process packets destined to IP addresses that don't reside on the local host.  How does that apply here?  When the ARP entry is manually configured on each guest VM, then communication between the VMs functions as expected.

[AlexAckerman]

What is the gateway address on that VLAN?  I usually see a .1 address reserved for the gateway.  It’s a Cisco physical switch so what is the virtual address for the switch itself on that VLaN?

[srsns]

He gave .1 and .2 as examples.  Those are not the actual IP address.  The gateway is .1.  The hosts are various other IPs in the same /24 subnet.

[alex-mar]

..are there any non-default segment profiles ?

[DimitarStoyanov]

If you have more than one uplink test the communications when one of them is disconnected.

[DurhamNeil]

Post ‘show run int’ on your switchports, post screenshots of your vswitches & settings. Also, do you have ACLs on that VLAN? You guys are being ambiguous with your terminology, no wonder you’ve on to support for 6 hours.

[frennzyb]

Do you have PVLANs setup anywhere?

[grimsrue]

I know this is a late reply and I hope you figure out your issue. If you are still fighting with your connection.......

It sounds like you either have a gateway issue or dup IP address issue. Check to make sure you do not have the same IP address configured on one of your other Cisco interfaces or on another VM or Host VMK. Try a different set of IP address on your VMs.

Can the VMs or VMKs ping the Gateway?
Are you ESXi hosts VMKs sitting in VLAN 363 or are they running in a different VLAN?
If the ESXi hosts are sitting in the same VLAN as the VMs then move the Hosts off to a different VLAN/Subnet.
If you are not already doing this, I would put your ESXi Host VMK0 on it own dedicated portgroup and then put the VMs in their own dedicated portgroup.
Are you tagging the VLAN at the portgroup or are your trunking the VLAN down to the OS in the VMs?

Check to see if spanning tree is detecting a loop on the cisco switch

Are the Cisco interface setup in a vPC? If so do you have a LAG setup on your vDS or did you set you portgroup teaming to use "IP Hash"

Set MAC Learning and Forged Transmits to "Accept" in the portgroups security

It would also help us by using terminology correctly. Saying 'VMWare ESXi hosts' then saying 'VMWare 1 or 2 VM host' is confusing. Be specific about what you are describing. A ESXi server is normally referred to as the "Host" and the VMs running on the ESXi host are just referred to as "VMs". Based on what we are reading in the original post its hard to determine if you are referring the ESXi host IP configuration or the network configured of two VMs sitting on the ESXi hosts.

Also as a few people have mentioned it would be helpful to see a "show int ethx/x" or "show int ethx/x switchport" on your Cisco switch.

[GirgisHady]

Can you please let us if you found a solution or what did you do with this issue ?