VMware Cloud Community
warriorcookie
Contributor
Contributor

Poor SR-IOV 10Gb performance

I've been using VMXNET3 for my network device and recently implemented SR-IOV. I've noticed a huge drop in performance.

Host is SuperMicro X9DRH-7F, 2x E5-2667v2, nic is x520-DA1 connected to my switch via DAC.

I don't know what my max output is across the physical network as I don't have any other 10Gb clients.  iperf and ping show good 1Gb performance.

Using iperf between guests on the same host with VMXNET3 I get a little over 6Gbps.

Switching to SR-IOV, iperf from FreeBSD server dropped as low as 3Gbps to my windows VM and 4.8Gbps to my Ubuntu VM.

Doing a loopback test with iperf I get 40+Gbps on the FreeBSD (FreeNAS) VM, 9.5Gbps on the windows VM and 25Gbps on the Ubuntu VM.

I've tried TSO and LRO, a number of different combinations of sysctl tuneables and the changes are very little if anything at all. Only switching back to VMXNET3 gets me back to 6Gbps.

I would appreciate some help narrowing this issue down. I'm hoping to achieve 9+Gbps, or even more between guests on the same host.

SuperMicro X9DRH-7F Dual E5-2667 v2 88gb DDR3 ECC ESXI 6.7 u3
Reply
0 Kudos
3 Replies
scott28tt
VMware Employee
VMware Employee

The speed of your physical NIC and underlying physical network are irrelevant if the VMs are connected to the same port group in your vSwitch. The packets between those VMs never touch the driver for the physical NIC, never mind the hardware.


-------------------------------------------------------------------------------------------------------------------------------------------------------------

Although I am a VMware employee I contribute to VMware Communities voluntarily (ie. not in any official capacity)
VMware Training & Certification blog
Reply
0 Kudos
warriorcookie
Contributor
Contributor

I understood that was the case for vmxnet3, but i thought sr-iov exposed the nic hardware directly to the vm bypassing the host kernel?

Either way, any suggestions for tracking down the bottleneck?

SuperMicro X9DRH-7F Dual E5-2667 v2 88gb DDR3 ECC ESXI 6.7 u3
Reply
0 Kudos
JerryFusion
Contributor
Contributor

OP, did you ever discover more regarding your issue? I suspect processor P-states/C-states in BIOS might be causing your poor performance.

Reply
0 Kudos