VMware Cloud Community
yogaraj1
Contributor
Contributor
‎11-19-2021 03:45 PM

Mapping host to host pci to guest pci address

Is there a way to map pci address from host and corresponding pci device address in the virtual machine? I am looking for a solution that can work for pci-pt, sriov and virtual nics (vmxnet, e1000, etc).

I checked this article https://kb.vmware.com/s/article/2047927 . But the calculated mapping does not look correct.

0 Kudos
6 Replies
bluefirestorm
Champion
Champion
‎11-19-2021 05:45 PM

It is very irresponsible to make an accusation/allegation that the KB calculation is wrong without presenting any of your sample calculation using the said steps in the KB.

Even if your calculation was not matching, it would have been better to ask for confirmation/advice presenting a sample mapping as to what you are doing wrong that is leading to mismatched values; instead of plainly saying that mapping is not correct.

It is highly unlikely the KB is wrong. The chances are much higher that you are wrong in your calculation/interpretation.

Anyway, if you notice, there is also a Python program attachment in the KB.

0 Kudos
yogaraj1
Contributor
Contributor
‎11-20-2021 04:21 PM

@bluefirestorm Thanks for your response. Sorry for being naive. I ran the python program before creating this post. let me give the details. The pci passthrough configuration is in pcipassthru-grep-from-vmx.txt

The program output is given below,

ybaskaravel@lnx2:~/Downloads$ python KB_2047927_pciSlots.py -f TPC-L3R6-39-001.vmx
[0000:00]-+-17.0--[x]--00.0: ethernet0 : [224 ]
[0000:00]-+-10.0--: scsi0 : [16 ]
[0000:00]-+-15.0--[x]--00.0: pcipassthru0 : [160 ]
[0000:00]-+-16.0--[x]--00.0: pcipassthru1 : [192 ]
[0000:00]-+-18.0--[x]--00.0: pcipassthru2 : [256 ]
[0000:00]-+-15.1--[x]--00.0: pcipassthru3 : [1184 ]
[0000:00]-+-18.0--: pcibridge7 : [24 ]
[0000:00]-+-17.0--: pcibridge6 : [23 ]
[0000:00]-+-16.0--: pcibridge5 : [22 ]
[0000:00]-+-15.0--: pcibridge4 : [21 ]
[0000:00]-+-11.0--[x]--00.0: vmci0 : [32 ]
[0000:00]-+-11.0--: pcibridge0 : [17 ]

Does program only give parent slot number? How can I calculate [x] ? Even parent slot numbers marked in red above does not match lspci -t output from the virtual machine (refer to lspci-from-vm.txt)

+-15.0-[03]--+-00.0 Intel Corporation Ethernet 10G 2P X520 Adapter [8086:154d]
|            \-00.1 Intel Corporation Ethernet 10G 2P X520 Adapter [8086:154d]
+-15.1-[04]--
+-16.0-[0b]--
+-18.0-[1b]--+-00.0 Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection [8086:10fb] 
           | \-00.1 Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection [8086:10fb]

The expected mapping is given below,

pcipassthru0 - 0000:04:00.0 (host) - 03:00.0 (guest)
pciPassthru1 - 0000:04:00.1 (host) - 03:00.1 (guest)
pciPassthru2 - 0000:81:00.0 (host) - 1b:00.0 (guest)
pciPassthru3 - 0000:81:00.1 (host) - 1b:00.1 (guest)

Let me know if I miss something in the calculation or any further information is needed to understand the problem.

Preview file
1 KB
Preview file
2 KB
0 Kudos
bluefirestorm
Champion
Champion
‎11-20-2021 07:41 PM

I think the algorithm in the KB would work only for virtual devices and not for PCIe passthrough devices.

If you look at the resulting calculation, it would have resulted in the first X520 device (pcipassthru0) and second 82599ES device (pcipassthru3) to be in the same virtual slot within the VM. Both the physical X520 and 82599 NICs that you have are two devices in one PCIe slot.

You already know the host ID slot (be it from lspci from ESXi console or the vmx). If I am not mistaken, you should be able to see where the PCIe passthrough device is assigned from the vmware.log when the PCIe passthrough device is registered, plus whatever memory map address assigned and size allocated. You should see something like "Registered a PCI device for ....."

 

0 Kudos
yogaraj1
Contributor
Contributor
‎11-22-2021 03:01 PM

Thanks again!

Even for the virtual device (vmxnet3 in this case), the calculation gives only the parent pci bridge address.

Calculation:

[0000:00]-+-17.0--[x]--00.0: ethernet0 : [224 ]

On VM:

-[0000:00]-+-00.0 Intel Corporation 440BX/ZX/DX - 82443BX/ZX/DX Host bridge [8086:7190]
           +-17.0-[13]----00.0 VMware VMXNET3 Ethernet Controller [15ad:07b0]

I am looking for a way to calculate exact pci address of the virtual device.

0 Kudos
bluefirestorm
Champion
Champion
‎11-22-2021 05:33 PM

I have no idea how the [x] is assigned. The python2 program also just simply hardcodes the [x] in the output string.

From the looks of things, it looks like it is the bus number and it is plainly in sequence. You can see this in both lspci (in Linux VMs) or HWinfo64 (on Windows VMs).

PCI devices 0x01 (1) and 0x11 (2) has 1 each, while PCIe devices 0x15, 0x16, 0x17, 0x18 has 8 each and thus 0x15 starts at 0x03 - 0x0a (03 - 10), 0x16 starts at 0b - 0x12 (11-18) and so on.

 

yogaraj1
Contributor
Contributor
‎11-23-2021 02:54 PM

@bluefirestorm Yeah, the bus numbers are in sequence in with pci bridges. If this mapping remains fixed, I think we have a solution (not very straightforward though). In case of pci-pt, we are thinking to do the mapping only for first device in each nic and increment the func number for the following nics. Also, we will test if it works for sr-iov.

-[0000:00]-+-00.0 Intel Corporation 440BX/ZX/DX - 82443BX/ZX/DX Host bridge [8086:7190] 
           +-01.0-[01]-- 
           +-11.0-[02]-- 
           +-15.0-[03]--
           +-15.1-[04]-- 
           +-15.2-[05]-- 
           +-15.3-[06]-- 
           +-15.4-[07]-- 
           +-15.5-[08]-- 
           +-15.6-[09]-- 
           +-15.7-[0a]-- 
           +-16.0-[0b]-- 
           +-16.1-[0c]-- 
           +-16.2-[0d]-- 
           +-16.3-[0e]-- 
           +-16.4-[0f]-- 
           +-16.5-[10]-- 
           +-16.6-[11]-- 
           +-16.7-[12]-- 
           +-17.0-[13]--
           +-17.1-[14]-- 
           +-17.2-[15]-- 
           +-17.3-[16]-- 
           +-17.4-[17]-- 
           +-17.5-[18]-- 
           +-17.6-[19]-- 
           +-17.7-[1a]-- 
           +-18.0-[1b]--
           +-18.1-[1c]-- 
           +-18.2-[1d]-- 
           +-18.3-[1e]-- 
           +-18.4-[1f]-- 
           +-18.5-[20]-- 
           +-18.6-[21]-- 
           \-18.7-[22]--

Btw, vmware.log files for both host and VM contains only the host pci addresses. It does not seem to log the guest pci address.

Thanks a lot for you help!

0 Kudos