Because vSAN and Management both use the same TCP stack, if vmk0 (Management) and vmk1 (WitnessPg) are on the same subnet, you will observe a multi-homing situation (KB 2010877).
You could tag only vmk0 for "vSAN Traffic" if you desire, as this is a supported configuration. Keep in mind you will want to isolate/protect that network from non-administrative access.
I go into some more detail here: Understanding the vSAN Witness Host - Traffic Tagging - Virtual Blocks
Thanks Jase for the info. Since it's a new build, we will request our vendor to split the vmkernels instead of using both Management and Witness traffic on just one vmkernel.
No worries. Completely supported.
Just remember that you'll need to use Static Routes.
Some more questions if you don't mind :
If we use the option to "Override default gateway" on the new vmkernel to point to the Witness gateway, do we still need to add the static routes?
Eg. Management Network Gateway : 192.168.15.1
Witness Traffic Gateway : 192.168.118.1 (Newly created)
Remote Witness Appliance : 192.168.28.100
In the VMkernel creation screen, the IP set for example for Host A is 192.168.118.10, Override default gateway is set to 118.1
1.) How do we confirm that it's NOT using the Management network stack, as we are trunking the 2 Vlans (15 and 118) through one physical link.
2.) If the Management network is able to reach the remote witness appliance (eg. 192.168.28.100), does it matter if it goes through the Management stack as it's separated by vlan/IP? (meaning normally 192.168.15.x is able to ping 192.168.28.x anyway)