vSwitch bug? ARP not passed on VLANs

[sgadsby]

Hi,

I want to run some VMs in different VLANs. No problem you say, just allocate a different port group on the required VLAN & configure switching appropriately.

BUT there seems to be an issue with Layer 2 broadcasts not getting through ESX 3.5 vSwitches.

A direct consequence of this is that VMs that are migrated to another host will not be visible on the VLAN.

This is because ARP broadcasts by the pSwitches do not get through to the VM in its new location!

To make it work, I had to remove the VLAN reference from the port group, and send that vlan down the trunk untagged. When I did this, the switches were able to locate the VM in its new location (via normal port-flooding) and route correctly. The limitation of course is that only one VLAN can be send down the trunk untagged. Therefore all my VMs need to be in a single VLAN if I want VMotion to work properly...

Has anyone else seen this? Is there a known issue with the vSwitches or am I missing something?

It is possible I suppose that it is my pSwitches not broadcasting inside the VLAN, but that seems almost absurd and I more readily blame the ESX vSwitches.

Please advise,

Simon.

--

I need to read "Xen and the art of VMware sales"

[kimono]

I would be very suprised if it was a Vswitch bug. This sort of thing is almost always switch config.

What sort of physical switch are you using?

/kimono/

[sgadsby]

Nortel L2/3 Copper Switches, running Alteon OS with latest firmware (1.4.2).

It's inside an IBM Bladecenter H.

--

I need to read "Xen and the art of VMware sales"

[admin]

There's a vSwitch setting called "Notify Switches". It's set to yes by default. Is it set to yes on your vSwitches? I'm including a screenshot of mine.

Chris

[Erik_Zandboer]

Simon,

If "notify switches" is enabled, your switch does not even need to send ARP broadcasts. When a VM is vmotioned to antoher host, the host takes care of this by telling the switch (via ARP) that the VM is now available through (one of) his own pNICs... I am very sure that this is no bug in the vSwitch, almost every single environment I install uses trunked (dot1q) ports to the pSwitches. Never seen this fail before...

Are you perhaps having issues using both tagged and untagged frames through the same ports? I know some switches do not cope with this very well (or not at all). I always make sure that either all packets are tagged, or all packets are untagged over a pNIC.

[sgadsby]

Thanks chuss/Erik - I REPENT! I REPENT!

As surmised the problem seems to be with my pSwitch and not the vSwitch afterall.

"Notify Switches" was definitely enabled, and I am running all vlans tagged to ESX now.

The problem seems to lay with the switch having both a L2 Forwarding Database and a L3 ARP cache. The former is successfully updated by the RARPs coming from the VM on the new ESX host, however the ARP cache is not updated. So basically everything works on the Server VLAN, but routing via the L3 pSwitch fails until a correct ARP is established for the server.

I imagine I wouldn't have this problem if I used an upstream router. How frustrating is it that I can see a device on Layer 2 but not on Layer 3 on the same device!! Unbelievable - I will raise this with the switching folks.

Thx for yr input.

--

I need to read "Xen and the art of VMware sales"

[sgadsby]

Layer 3 pSwitch issue

[Erik_Zandboer]

SMILE

It is always usefull to doubt functionality. In this case, it must have pointed you to the pSwitch quite fast, because vSwitches simply WORK... Good to see you solved your problem (that is, if the networking guys can straighten it out for you). Just out of curiosity, what pSwitch are you using?