Hi John,
I turned off "forward all" in the meantime.
Setup: there is 1 "classic" vSwitch and 1 DvS.
- The management interface is bound to vmnic0 and vmnic1 and they go to two physically separated Gigabit switches belonging to the management network.
- There are 10 DvS Portgroups. DvS is connected to the 10Gig network.
- One DvS Portgroup is called VSAN, has the corresponding vmkernel port and is exclusively bound to vmnic2 on every host (with vmnic3 on standby). These are all physically linked to the same switch.
- Another one is called vMotion, is bound to vmnic3 (with vmnic2 on standby) and does what the name implies.
- The other 8 are VM-networking portgroups for the VM's. They are all bound to vmnic3 (with vmnic2 on standby).
- There is a 2x 10G LACP Inter-Switch-Link (ISL) between the two Netgear switches. In case of a switch-outage, the dvPortgroups can fail over to the other switch.
- The VSAN and vMotion VLAN's are not routable.
- All VLAN's are available as tagged on all ports of these ESXi servers and on the ISL's LACP LAG.
- VSAN multicast traffic goes to both ports of all 4 nodes as this VLAN is available as tagged on both switches (but only on the ESXi servers) so it travels the ISL.
- VSAN unicast traffic stays inside the same chassis as the destination mac-addresses exist only on the ports of that same switch (where all the "vmnic2" are connected to) unless the vmkernel-port fails over to the other vmnic and thus the other physical switch.
Summary: one switch is "active" for VSAN traffic only and the other switch is "active" for all other traffic incl. vMotion. Management traffic is isolated on it's own physical network.
In both Netgear switches, under "switching -> Multicast -> Bridge Multicast Forwarding", all VLAN's are on the default setting of "forward unregistered" (used to be "forward all" for the VSAN VLAN but not anymore).
I have not activated IGMP Snooping either at the moment. A networking guy told me it's not needed because the multicast traffic coming from the VSAN vmkernel interfaces stay within the unroutable VLAN (as it's a single L2 network).
