ESXI 6.0 pass-through multiple PCI devices

[TempleKa]

First of all I'm a bit new to ESXI so I'm no expert. I'm trying to pass-through 2xPCI network cards to 2 VMs (1 for each). Problem is ESXI doesn't power on one of the VMs and gives me an error about device already in use. Can't I pass-through different PCI devices to different VMs or what am I doing wrong?

The network cards both appear on this PCI bridge. I removed one of the cards since I couldn't even do individual pass-through (when I enabled one, both of them enabled).

[bluefirestorm]

From this VMware KB VMware vSphere VMDirectPath I/O: Requirements for Platforms and Devices (2142307) | VMware KB

PCI Functions behind legacy PCI Bridges

VMware strongly recommends that PCI Functions assigned for VMDirectPath I/O be placed behind PCI Express root ports or switch downstream ports.

VMware discourages VMDirectPath I/O assignment of PCI Functions behind conventional PCI bridges or PCIe-to-PCI/PCI-X bridges. PCI Functions behind PCIe to PCI/PCI-X bridges or PCI conventional bridges must be collectively assigned for VMDirectPath I/O to the same virtual machine.

These bridges take ownership of PCI transactions sent by PCI Functions behind them by placing the bridge’s PCI requester ID on the transactions. This forces the ESXi host to program IOMMU translations using the PCI Bridge’s requester ID, implying that all PCI Functions behind the bridges must be placed in the same IOMMU domain and therefore be collectively assigned to the same virtual machine.

[TempleKa]

Ok I'm sorry as I said I'm new at this and I haven't understood much of what you said. What I think I understood is that I'm doing the pass-through wrong. Could you simplify the answer a bit as to how I'm supposed to pass-through those PCI cards?

[bluefirestorm · Accepted solution]

Okay, let's focus on the key statements. You could also replace the words "VMDirectPath I/O" with "pass-through" and "PCI Functions" with "PCI cards".

VMware discourages VMDirectPath I/O assignment of PCI Functions behind conventional PCI bridges or PCIe-to-PCI/PCI-X bridges.

What you are trying to do, passthrough of PCI card through a PCI bridge is discouraged. The Realtek 8169 PCI cards you have are behind the Intel 82801 PCI bridge.

PCI Functions behind PCIe to PCI/PCI-X bridges or PCI conventional bridges must be collectively assigned for VMDirectPath I/O to the same virtual machine.

You are trying to assign two different Realtek 8169 PCI cards behind the same PCI bridge to two different VMs; which contradicts this statement.

To apply this statement to your situation. The Realtek 8169 PCI cards behind the Intel 82801 PCI bridge must be assigned for passthrough to the same virtual machine.

[TempleKa]

Okay I think I understood. You're trying to say that I can't pass-through 2 or more devices to individual VMs if they are on the same PCI / PCI-E bridge?

If the above is correct here's why I was trying to do it. I noticed that if i keep the cards virtualized their performance is pretty bad for my Freenas VM (as in I'm not getting the expected throughput). In this case what should I do to get better performance? Should I change the network cards with something else or are there any other solutions? I'm asking because the cards work as intended once I pass-through to the VM but not when virtualized.

[bluefirestorm]

It is not me who made the rules. That is what the KB says. I just translated it/applied it to the situation that you have.

What happened to the other VM? You now mention only FreeNAS VM, what about the other VM that you were trying to assign the other Realtek 8169 card for?

It is hard to give definitive advice as I don't know what PCIe slot(s) you have that are free/open to put in another Gigabit Ethernet card. But I suppose you got one Realtek 8169 passthrough working; you could just add another Gigabit Ethernet card that sits in its own PCIe slot (not behind the same bridge) for passthrough to the other VM. You may also look at multiport Ethernet cards, or cards that has SR-IOV functionality (be it single port or multi-port). SR-IOV will get more complicated.

There are lots of other forum topics about NAS (or FreeNAS in particular) performance somehow being "capped" in ESXi at a certain read/write rates even though a 10Gbps Ethernet card is used. You could also search for such topics either in this forum or in the FreeNAS forum. So to get ample throughput may not be simply just putting in a 10Gbps Ethernet for your FreeNAS VM.

Have a look at this one for starters

https://forums.freenas.org/index.php?threads/sync-writes-or-why-is-my-esxi-nfs-so-slow-and-why-is-is...

[TempleKa]

I just realized I'm a bit "new" at this and figured out that you already gave me a solution to the problem when you said that PCI pass-through on same PCI bridge does work if they're assigned to the same VM.

I just had to reassign the network cards between VMs and of course everything works now. My other VM is pfSense and that requires 2 network cards so I assigned it the 2 PCI cards and the 3rd card that is a PCI-E card I assigned to Freenas. Problem solved. Thank you very much for your assistance.