I was reading the design guide for NSX and in there it was mentioned that the VXLAN VLAN ID on a vDS has to be the same across all racks in a given DC. VLAN ID for Mgmt, Storage and vMotion can differ; however the VXLAN VLAN ID has to be consistent. Can someone explain why that is?
The vSphere Distributed Switch can only support one VXLAN transport/VTEP VMkernel port. You therefore need to use the same VLAN ID globally across the environment (although you can back each occurrence with a different IP subnet) or use a separate VDS. In the latter scenario you will get dvPortgroups created across ALL the VDSs that back the hosts in your Transport Zone when you create a Logical Switch but each will be tagged with the associated VLAN ID. If you use one VDS you're restricted to a single dvPortgroup and therefore a single VLAN tag.