VSAN on two distributed switched

[vmwarelicense12]

We have 8 ESXi hosts in one vSAN cluster using one dvs.

Is it possible to configure 4 of the hosts to use one dvs for vsan and the 4 other hosts to use another dvs (same vsan vlan)?

[AlexanderLiucka · Accepted solution]

vmwarelicense123,

Yes, you can.

[vmwarelicense12]

This is our setup.

3 esxi in one cluster, where each have:

2 uplinks on dvs1 used for management, vmotion, vsan, vxlan (all)

2 uplinks on dvs2 witch I whant to use for vsan.

I then try to migrate vmk2 (vsan) from dvs1 to dvs2. The kernel port is migrated, but reports that the host cannot se the other vsan nodes.

What am I doing wrong?

[AlexanderLiucka]

Check the health status of your dvs2. I'm almost sure that you have not configure correctly the vsan vlan on the hardware switch to which go the 2 uplinks of dvs2.

[vmwarelicense12]

ohh. dear.. ohh... dear.. I forgot to set MTU to 1600 on the new dvs!!

MTU is 1600 om the old dvs i migrate from, so now it all work at expected.

[AlexanderLiucka]

This is strange.

When you have wrong MTU, the vsan health will reportit as incorrect MTU size which is very different from "the host cannot see the other vsan nodes".

Also it is very strange that, you have set the MTU at 1600 not at 9000? the Default MTU is 1500 which is very close to your 1600.

[TheBobkin]

Hello Alexander,

They are likely using NSX which requires 1600 MTU.

Bob

[vmwarelicense12]

Hi TheBobkin,

Exactly... we are using NSX;)

[AlexanderLiucka]

Hello TheBobkin,

You are correct that they use vxlan and for vxlan the minimal MTU is 1600 but they can't put vSan, vMotion, iSCSI, NFS, FCoE, Management, etc over vxlan.

This mean that the vsan and the vxlan are on the same level. To work vxlan the dvs has to have minimum MTU of 1600 or higher, but for the vsan the default MTU of 1500 is enough.

Again I don't know how vmwarelicense123 will receive the error that the vsan host can not see the other vsan nodes when you can not put vSan, vMotion, iSCSI, NFS, FCoE, Management, etc over vxlan which was not working for the wrong MTU?

P.S.

https://storagehub.vmware.com/t/vmware-vsan/vmware-r-vsan-tm-network-design/nsx/

VMware Virtual SAN and VMware NSX Compatibility - Virtual Blocks