2-Node Shared Witness Network Config

[Tibmeister]

So, question for those much smarter than me; why do we need to have the vSAN service enablement on a vmk on the Shared Witness that is located at the main datacenter and the 2-Node clusters connect over a WAN link?  I constantly get plagued with the "vSAN: MTU check (ping with large packet size)" error in SkyLine between the 2-Node clusters and the Shared Witness.  

I have the pSwitch port set for 1500 MTU, the vDS itself set for 1500 MTU, and the vmk0 set for 1500 MTU on the Shared Witness, and the vSAN service is enabled on vmk0 since I only have one vmk.  I have always been boggled why we need the service enabled and why the check isn't smart enough to know that the Shared Witness only has the witness components and is over a WAN and as such a jumbo ping will never work.

[TheBobkin]

@Tibmeister 

"why do we need to have the vSAN service enablement on a vmk on the Shared Witness"

Because without this it would become isolated from all vSAN clusters as nodes will only ever send vSAN traffic over vsan/witness-tagged interfaces. This is like asking why one would need vMotion traffic tagged in order to transmit vMotion data and thus do vMotions.

With regard to the MTU alert - if it is only getting like 7-800 max MTU (e.g. due to tunnel) then this being triggered can be expected and if not using 9000 for vsan traffic at all in that cluster then consider silencing it if it is too bothersome.

[Tibmeister]

I forgot one portion of my question, which is why do we need the vSAN service enabled on the witness appliance instead of the vSAN Witness Traffic flag like the hosts in the cluster have?

Not sure about the "7-800 max MTU (e.g. due to tunnel)" comment.  My understanding of the alert is that it checks for 9000 MTU on all vmks that have the vSAN service enablement, which leads back to the original question of why vSAN Service instead of just Witness Traffic on the witness appliance?

[TheBobkin]

@Tibmeister witness-type traffic is only to distinguish outbound traffic toward a Witness in case the vsan-network used on the data-nodes is not reachable over L3. A Witness should never have witness-type traffic enabled on any vmk, why would it, as it never needs to talk to another Witness.

Actually the jumbo frame health check runs whether 1500 or 9000 MTU is in use, the difference is how it slices these packets e.g. if it is unable to cleanly slice these into 1500 byte segments (as this IS what is set on the vmks on each side) it can fail.

[Tibmeister]

Interesting, because most WAN links can't do 1500 MTU, most times it's more like 1400 for most IPSec type connections, including the VeloCloud SD-WAN solution I currently am using, so hopefully it's not strict coded at 1500 otherwise it will continue to always fail and not be a useful alarm.

[TheBobkin]

IMO it is not a useful alert where only triggered by Witness connection anyway as Witness has an exception in place that it won't be rejected membership based on not being able to send/receive membership join datagrams at full set MTU (with df bit set) whereas a data-node will be (e.g. if you change the MTU beyond what can be sent with df bit and leave-rejoin cluster it will be isolated).

There is also some Witness related feature-gap for that health test in vSAN/vC 7.x that I have seen this trigger in some environments but not others (likely related to back-end network aspects that we wouldn't have insight into) but I am not aware of when/if that gap is intended to be filled.

[Tibmeister]

So, it's one of those that can be ignored based on context.  Would be nice to just not have that check occur on links that have the witness traffic tag to avoid false alerts and only trigger on actual vSAN data paths...

[depping]

that is a good feature request, will see if I can file it for you.

[depping]

Filed it.