I have SR-IOV working with ESX 5.1, an Intel X520 DA2 (82599EB), on a Dell R420 Server. The guest is Ubuntu 12.04.
The documentation is here http://pubs.vmware.com/vsphere-51/index.jsp?topic=%2Fcom.vmware.vsphere.networking.doc%2FGUID-C5043E...
However, there are some gotchas and undocumented steps that threw me off.
First, just install the NIC and make sure it shows up normally under Host->Configuration->Network Adapters. Now, follow the instructions in the above link.
The 82599 uses the ixbge driver. So, when the documentation says "Expand Kernel Module Configuration > Kernel Module and select your kernel module.", it is referring to "ixgbe". The X520 DA2 is a dual port NIC, so I entered 16,16 for the max_vfs setting.
Be sure to let the task for changing the host profile finish before attaching and applying the profile to the host. Then reboot the host.
Be sure the NIC is cabled to a switch on boot up or the virtual functions may not show up in VMware. Also, I used a bad cable for one of the ports, and half my virtual functions didn't show up. (Luckily my other cable was good so I figured it out.)
One of your goals is to get the ixgbe driver to create the Virtual Functions. You can tell if this is working by going to the ESX VGA console and select view logs and then select the Vmkernel logs. Search for ixgbe with the slash / key. If you see an error of some sort in the ixgbe logs, then you need to resolve that. It could be your computer BIOS out of date, appropriate settings not enabled, or using a wrong PCI slot or whatever.
By the way, I did not need to update the ESX ixgbe driver. I have the 3.7.13 driver which comes with 5.1. (See Host->Hardware Status->Software Components->net-ixgbe)
For me, ultimately, I needed to move the X520 to a different PCI slot before the ixgbe driver would create the virtual functions. I had an i350 that I purchased with the R420 and I needed to move that to the second slot and move the X520 to the primary PCI slot. Before I moved it, the ixgbe driver complained that I may not have SR-IOV enabled in my BIOS (but I did).
Once you resolve the driver errors, you should NOT see the network adapters under Host->Configuration->Network Adapters. If you see them there, SR-IOV is not working. You should only see the Virtual Functions and those will be under Host->Configuration->(Hardware) Advanced Settings in the DirectPass through list.
Yes, that's ultimately how you know you've got SR-IOV working: the Virtual Functions show up as DirectPath I/O capable devices under Advanced Settings.
Now, you just need to assign the virtual functions to virtual machines. To do this, shutdown the virtual machine. Edit settings -> Add.... -> PCI Device -> Select the virtual function from the list. You DO NOT add a VMware network adapter. VMware is just passing through the device at the PCI level - it's up to the guest driver to actually talk to the virtual function over PCI and expose it as an OS network adapter.
Now, when you boot the guest VM, a NIC Virtual Function will be exposed to the guest on the guest PCI bus. The guest must have the driver for the VF mode of the NIC. An up to date Ubuntu 12.04 has a built-in ixgbevf driver that supports virtual functions - no need to install a new driver. So, when I boot Ubuntu, I get a new eth interface that I can immediately use (and it works).
One issue in this process is that when you boot with a new NIC and then enable SR-IOV you'll at first have vmnics showing up that VMware may want to use for management, software FCoE, etc. and then disappearing after you enable SR-IOV and reboot. You'll probably need to reconfigure the management network via the VGA console at some point. As I mentioned earlier, I had two NICs: an i350 and an x520. So, I always had some NIC I could use for management. If you only have one NIC, you could lock yourself out. In that case, you might need to install another NIC to recover. If you have a dual port NIC, you could try 16,0 or 0,16 for the max_vfs so only one port is enabled for SR-IOV. I didn't try that so you're on your own there.
One thing that bothered me is that I had a host profile that was fixated on the MAC address of some NIC it wanted to use for software FCoE and complained about it being missing. I couldn't get it to stop complaining about that and ultimately had to disable that check (right click on profile -> Enable/Disable Profile Configuration -> Storage Configuration -> Uncheck "Software FCoE Configuration", click OK.).
Hopefully, this information will help you avoid the struggle I went though to get this working. If the VMware folks are listening, I'd recommend enhancing the SR-IOV documentation to include: The names of drivers (e.g. ixgbe) for supported NICs, the fact that the physical NIC will not show up in Configuration->Network Adapters after SR-IOV is enabled, the location where the Virtual Functions will appear (in the DirectPath I/O list), removing the misleading phrase "after you create the virtual functions" (because you don't create them - they just show up in DirectPath I/O), the steps to add the PCI device (!), some tips on browsing the logs for driver errors, and the fact that Ubuntu 12.04 supports the ixgbe virtual functions.
Rick Masters
