Perttu
Enthusiast
Enthusiast

vSAN 7.0U2 RDMA - no support yet for InfiniBand adapters?

Hi, 

RDMA support for vSAN 7.0U2 are great news. However as far as I know and also mentioned in https://blogs.vmware.com/virtualblocks/2021/03/24/vsan-7-update-2-rdma/  article, only RoCEv2 is currently supported. But what if, like we do, a customer has an InfiniBand fabric already in place and even connected to VSAN enabled ESXi hosts, could that be utilised also for the VSAN traffic. From a kernel's perspective the application programming interface (IB verbs) are just the same, RoCEv2 is just a different transport layer built on top of the TCP/IP semantics.

The reason why we have Infiniband is to provide SR-IOV VFs to guests, which are then able to natively connect to a Lustre distributed storage and a HPC. The IB does also offer a by-design lossless transport compared to Ethernet fabrics, because Ethernet has to rely on PFC frames to stop transmitting RoCE-frames in case of congestion. IB instead has a credit based flow control. So in a theoretical sense, VSAN kernel adapter would have its own PF on Infiniband adapter and maybe a dedicated subnet manager to avoid mixing the packets with flows from virtual machines? 

0 Kudos
4 Replies
depping
Leadership
Leadership

I am not sure I understand the question, right now, only RoCEv2 is supported, anything else is not supported. 

 

0 Kudos
Perttu
Enthusiast
Enthusiast

That is exactly what I also thought. This was merely a rhetorical question. Any possibility that VMware is thinking to add native InfiniBand support soon in future releases, because it is rather straightforward to support Infiniband when RoCEv2 is already supported. 

I mean, the application (VSAN) already uses IB (verbs API) semantics for RDMA. RoCEv2 is just a one implementation of this. IB is another (and was the original one). So a step to support IB should not be much to do if the vmkernel structure can be fitted into IB. 

 

0 Kudos
depping
Leadership
Leadership

I can't comment on roadmap publicly. I would recommend to reach out to a local vSAN/VMware representative to get details on roadmap and priority.

0 Kudos
Perttu
Enthusiast
Enthusiast

Perfectly understandable. But let the engineers know, that least a one party is looking forward for this.