vSAN Traffic on VXLAN?

[MC1903]

Quick Q. Does GSS support running vSAN data-path traffic over VXLAN?

I just stumbled across this Dell EMC VxRail Multirack Deployment Guide (https://infohub.delltechnologies.com/static/media/60cee101-e75e-4b72-a654-da28a1438865.pdf) where they are running ESXi Host Management, vMotion and vSAN traffic in VXLAN overlays; to allow distribution of vSAN nodes to different data center racks (fault domains), when using a layer 3 spine/leaf network architecture.

In this configuration the Dell network switches are providing the VXLAN overlay/underlay; there is no NSX involvement with the traffic.

I am wondering if this is a specific Dell VxRail supported configuration or if there is GSS support for non-VxRail vSAN configurations using 'network switch hardware' provisioned VXLAN overlays for the ESXi host management & storage traffic?

Thanks

M

[nachogonzalez]

Hello, hope you are doing fine:

Just read the document and I think there is a misconception here

Extract from the pdf:
shows the five minimum required VxRail VLANs, which are shown as solid colored lines that are attached to virtual networks, and that are associated with VTEP/VXLANs at the leaf layer and shown as corresponding dashed lines. Each leaf pair is configured with an identical VTEP and uses the same IP address on a loopback address. Each VTEP is associated statically on the leaf switches. The dashed lines represent the VXLAN tunnel for a given VxRail VLAN.​

So, to the ESXi cluster you will have a vSAN VLAN, on the rack interconnection ath the Leaf and spine level, the communication will be done via VXLAN.
(VXLAN as a protocol is not VMware-Propietary, some other vendors use it)

Hope that works

[NikolayKulikov · Accepted solution]

Hi,

First of all, you could use L3 leaf/spine without any overlays - it's completely supported to has L3 between nodes (All hosts in your vSAN cluster must be connected to a vSAN Layer 2 or Layer 3 network.) You should set statiс routes on ESXi (pre 7.0U1) or set though GUI (7.0U1+ example).

There is no special requirement vSAN for physical network except regular performance, low latency and reliability so you could use switch-based overlays for vSAN in case it's fast enough, but personally, I'll prefer to build vSAN network in the easiest and simplest way - leaf/spine with L2 inside the rack and L3 between the racks (or sites if it's stretched cluster).

Also you should check these two docs - https://docs.vmware.com/en/VMware-vSphere/7.0/vsan-network-design-guide/GUID-40DE7266-BBE8-49DB-907E... + https://core.vmware.com/resource/vsan-stretched-cluster-guide#sec4-sub3 

[MC1903]

Thank you @NikolayKulikov 

I thought the layer 3 reference only applied to vSAN witness traffic in a stretched cluster configuration, as I have not seen a VMware use case that shows a single site cluster can also use layer 3 for the vSAN data traffic; which is needed in a L3 leaf/spine multi-rack vSAN node setup.

I agree the simple way always make sense; no point in complicating life.

Cheers

M

[TheBobkin]

@MC1903 While I am unaware of the VXLAN-support side, we do support vSAN data-traffic via L3, examples and more details can be found on docs.vmware e.g.:

https://docs.vmware.com/en/VMware-vSphere/7.0/vsan-network-design-guide/GUID-EB8A266B-AE4B-45E4-B148...

[MC1903]

Hello @TheBobkin 

Thank you for also clarifying the layer 3 vSAN data-traffic support position. For the time being I will assume the vSAN data-traffic within a VXLAN overlay is a Dell VxRail / Dell Networking specific use case that is supported by Dell.

I found that page after the reply from @NikolayKulikov .Looking at the Layer 3, Single Site, Multiple Racks diagram, either I am being slow (very possible at this stage!) or it is not showing the vSAN data-traffic routed across different VLANs/subnets (as VLAN10 is show as layer 2 in more than one rack) as they would be in a true multi rack layer 3 leaf/spine network configuration.

Surely it should actually look something like this? The vSAN data-traffic for nodes within the same rack are on the same VLAN, but inter-rack vSAN data-traffic is routed between nodes in different racks, via the spine layer (the 'R1' router in this example)?

Thanks

M