2 Replies Latest reply on Jun 30, 2020 4:05 PM by warriorcookie

    Poor SR-IOV 10Gb performance

    warriorcookie Lurker

      I've been using VMXNET3 for my network device and recently implemented SR-IOV. I've noticed a huge drop in performance.


      Host is SuperMicro X9DRH-7F, 2x E5-2667v2, nic is x520-DA1 connected to my switch via DAC.

      I don't know what my max output is across the physical network as I don't have any other 10Gb clients.  iperf and ping show good 1Gb performance.


      Using iperf between guests on the same host with VMXNET3 I get a little over 6Gbps.


      Switching to SR-IOV, iperf from FreeBSD server dropped as low as 3Gbps to my windows VM and 4.8Gbps to my Ubuntu VM.


      Doing a loopback test with iperf I get 40+Gbps on the FreeBSD (FreeNAS) VM, 9.5Gbps on the windows VM and 25Gbps on the Ubuntu VM.


      I've tried TSO and LRO, a number of different combinations of sysctl tuneables and the changes are very little if anything at all. Only switching back to VMXNET3 gets me back to 6Gbps.


      I would appreciate some help narrowing this issue down. I'm hoping to achieve 9+Gbps, or even more between guests on the same host.