A couple of additional comments:
- PFC is meant for layer 2 storage technologies like FCoE in order to create a congestion control mechanism similar to buffer credits in a fiber channel network. Being that Virtual SAN leverages TCP between nodes, TCP congestion avoidance (AIMD & slow start) on top of PFC would potentially create issues and add complexity you don't need. Instead you would provide bandwidth guarantees for traffic classes so you don't starve storage or give the storage traffic a priority queue with a bandwidth cap so it doesn't starve other traffic.
- As noted above routed traffic is not supported. This is generally true of all vmkernel traffic other than management. In addition you'd create more complexity by having to enable L3 multicast if you aren't already using it
- I'm curious what your use case is here for >10GbE. SAS would red line at 6gbps so maximum throughput would ultimately be back-end limited with 40gbps, and 10GbE configured with LACP (if you can), assuming you have spaced your IP addressing properly, would result in a stream of writes utilizing 2 x 10GbE ports which is more than you could handle on the back-end assuming you are not packing multiple controllers and disk groups. If you are planning multiple controllers and disk groups I could see where you might attempt to push more than 10gbps, but being that Virtual SAN is designed to consume no more than 10% CPU on a host, I am not sure how far you could ultimately push it without hitting a CPU wall (not something I've personally tested)
What kind of workloads are you planning to run on this? Generally speaking VM workloads are small, random IO which tends to consume little bandwidth and starve storage IOPS rather than bandwidth. I'm really curious about your use case here.
