Hi,
RDMA support for vSAN 7.0U2 are great news. However as far as I know and also mentioned in https://blogs.vmware.com/virtualblocks/2021/03/24/vsan-7-update-2-rdma/ article, only RoCEv2 is currently supported. But what if, like we do, a customer has an InfiniBand fabric already in place and even connected to VSAN enabled ESXi hosts, could that be utilised also for the VSAN traffic. From a kernel's perspective the application programming interface (IB verbs) are just the same, RoCEv2 is just a different transport layer built on top of the TCP/IP semantics.
The reason why we have Infiniband is to provide SR-IOV VFs to guests, which are then able to natively connect to a Lustre distributed storage and a HPC. The IB does also offer a by-design lossless transport compared to Ethernet fabrics, because Ethernet has to rely on PFC frames to stop transmitting RoCE-frames in case of congestion. IB instead has a credit based flow control. So in a theoretical sense, VSAN kernel adapter would have its own PF on Infiniband adapter and maybe a dedicated subnet manager to avoid mixing the packets with flows from virtual machines?
