VMware Cloud Community
PetrNo
Contributor
Contributor

PCI Passthrough not working with gen 3.0 x4 card

Hello, we have a PCIe card which is Gen 3.0 x4 and we are trying to pass it to a VM Running Ubuntu 22.04 64 bit. the ESXI version we tested was 6.7 and 7.0. The vendor of the card has also tried the same with Windows 10 as a client VM, with the same result.

we have been using PCI passthrough with other cards and it was always working properly, but with this card it does not seem to work.

The card vendor has explained after reviewing the lspci -vvv than the pci properties are not correct and that is why their card and driver do not work.

The actual slot we have tried was Gen 3.0 x8 and Gen 3.0 x16, it behaves the same in both slots.

The lscpi -vvv from within the VM (card does NOT work):

 

 

03:00.0 Non-VGA unclassified device: DekTec Digital Video B.V. Device b87e
      DeviceName: pciPassthru0
      Physical Slot: 160
      Control: I/O- Mem+ BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
      Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
      Interrupt: pin A routed to IRQ 18
      Region 0: Memory at ffa00000 (64-bit, prefetchable) [size=16K]
      Capabilities: [50] MSI: Enable- Count=1/1 Maskable- 64bit+
           Address: 0000000000000000  Data: 0000
      Capabilities: [68] MSI-X: Enable- Count=1 Masked-
           Vector table: BAR=0 offset=00000160
           PBA: BAR=0 offset=00000170
      Capabilities: [78] Power Management version 3
           Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
           Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME-
      Capabilities: [80] Express (v2) Endpoint, MSI 00
           DevCap:    MaxPayload 128 bytes, PhantFunc 0, Latency L0s <64ns, L1 <1us
                 ExtTag- AttnBtn- AttnInd- PwrInd- RBE- FLReset- SlotPowerLimit 0.000W
           DevCtl:    CorrErr- NonFatalErr- FatalErr- UnsupReq-
                 RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
                 MaxPayload 128 bytes, MaxReadReq 128 bytes
           DevSta:    CorrErr- NonFatalErr- FatalErr- UnsupReq- AuxPwr- TransPend-
           LnkCap:    Port #0, Speed 5GT/s, Width x32, ASPM L0s, Exit Latency L0s <64ns
                 ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp-
           LnkCtl:    ASPM Disabled; RCB 64 bytes, Disabled- CommClk-
                 ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
           LnkSta:    Speed 5GT/s (ok), Width x32 (ok)
                 TrErr- Train- SlotClk- DLActive- BWMgmt- ABWMgmt-
           DevCap2: Completion Timeout: Not Supported, TimeoutDis- NROPrPrP- LTR-
                 10BitTagComp- 10BitTagReq- OBFF Not Supported, ExtFmt- EETLPPrefix-
                 EmergencyPowerReduction Not Supported, EmergencyPowerReductionInit-
                 FRS- TPHComp- ExtTPHComp-
                 AtomicOpsCap: 32bit- 64bit- 128bitCAS-
           DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis- LTR- OBFF Disabled,
                 AtomicOpsCtl: ReqEn-
           LnkCtl2: Target Link Speed: 2.5GT/s, EnterCompliance- SpeedDis-
                 Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
                 Compliance De-emphasis: -6dB
           LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete- EqualizationPhase1-
                 EqualizationPhase2- EqualizationPhase3- LinkEqualizationRequest-
                 Retimer- 2Retimers- CrosslinkRes: unsupported
      Capabilities: [100 v1] Virtual Channel
           Caps: LPEVC=0 RefClk=100ns PATEntryBits=1
           Arb:  Fixed- WRR32- WRR64- WRR128-
           Ctrl: ArbSelect=Fixed
           Status:    InProgress-
           VC0:  Caps: PATOffset=00 MaxTimeSlots=1 RejSnoopTrans-
                 Arb:  Fixed- WRR32- WRR64- WRR128- TWRR128- WRR256-
                 Ctrl: Enable+ ID=0 ArbSelect=Fixed TC/VC=ff
                 Status:      NegoPending- InProgress-
      Capabilities: [200 v1] Vendor Specific Information: ID=1172 Rev=0 Len=044 <?>
      Capabilities: [300 v1] Secondary PCI Express
           LnkCtl3: LnkEquIntrruptEn- PerformEqu-
           LaneErrStat: 0
      Kernel modules: DtPcie

 

 

 

The lspci -vvv from a physical machine (card DOES work):

 

 

04:00.0 Non-VGA unclassified device: DekTec Digital Video B.V. Device b87e
        Physical Slot: 4
        Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR+ FastB2B- DisINTx+
        Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
        Latency: 0, Cache Line Size: 32 bytes
        Interrupt: pin A routed to IRQ 29
        NUMA node: 0
        Region 0: Memory at 33fffe00000 (64-bit, prefetchable) [size=16K]
        Capabilities: [50] MSI: Enable- Count=1/1 Maskable- 64bit+
                Address: 0000000000000000  Data: 0000
        Capabilities: [68] MSI-X: Enable+ Count=1 Masked-
                Vector table: BAR=0 offset=00000160
                PBA: BAR=0 offset=00000170
        Capabilities: [78] Power Management version 3
                Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
                Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME-
        Capabilities: [80] Express (v2) Endpoint, MSI 00
                DevCap: MaxPayload 256 bytes, PhantFunc 0, Latency L0s <64ns, L1 <1us
                        ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset- SlotPowerLimit 26.000W
                DevCtl: CorrErr- NonFatalErr- FatalErr- UnsupReq-
                        RlxdOrd- ExtTag+ PhantFunc- AuxPwr- NoSnoop+
                        MaxPayload 256 bytes, MaxReadReq 512 bytes
                DevSta: CorrErr- NonFatalErr- FatalErr- UnsupReq- AuxPwr- TransPend-
                LnkCap: Port #1, Speed 8GT/s, Width x4, ASPM not supported
                        ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp+
                LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk+
                        ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
                LnkSta: Speed 8GT/s (ok), Width x4 (ok)
                        TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
                DevCap2: Completion Timeout: Range A, TimeoutDis+ NROPrPrP- LTR-
                         10BitTagComp- 10BitTagReq- OBFF Not Supported, ExtFmt+ EETLPPrefix-
                         EmergencyPowerReduction Not Supported, EmergencyPowerReductionInit-
                         FRS- TPHComp- ExtTPHComp-
                         AtomicOpsCap: 32bit- 64bit- 128bitCAS-
                DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis- LTR- OBFF Disabled,
                         AtomicOpsCtl: ReqEn-
                LnkCap2: Supported Link Speeds: 2.5-8GT/s, Crosslink- Retimer- 2Retimers- DRS-
                LnkCtl2: Target Link Speed: 8GT/s, EnterCompliance- SpeedDis-
                         Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
                         Compliance De-emphasis: -6dB
                LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete+ EqualizationPhase1+
                         EqualizationPhase2+ EqualizationPhase3+ LinkEqualizationRequest-
                         Retimer- 2Retimers- CrosslinkRes: unsupported
        Capabilities: [100 v1] Virtual Channel
                Caps:   LPEVC=0 RefClk=100ns PATEntryBits=1
                Arb:    Fixed- WRR32- WRR64- WRR128-
                Ctrl:   ArbSelect=Fixed
                Status: InProgress-
                VC0:    Caps:   PATOffset=00 MaxTimeSlots=1 RejSnoopTrans-
                        Arb:    Fixed- WRR32- WRR64- WRR128- TWRR128- WRR256-
                        Ctrl:   Enable+ ID=0 ArbSelect=Fixed TC/VC=01
                        Status: NegoPending- InProgress-
        Capabilities: [200 v1] Vendor Specific Information: ID=1172 Rev=0 Len=044 <?>
        Capabilities: [300 v1] Secondary PCI Express
                LnkCtl3: LnkEquIntrruptEn- PerformEqu-
                LaneErrStat: 0
        Kernel driver in use: DtPcie
        Kernel modules: DtPcie

 

 

The vendor says:

I am noticing a few weird numbers in the lspci output, namely:

DevCap:    MaxPayload 128 bytes (should say 256), PhantFunc 0, Latency L0s <64ns, L1 <1us

                 ExtTag- AttnBtn- AttnInd- PwrInd- RBE- FLReset- SlotPowerLimit 0.000W

LnkCap:    Port #0, Speed 5GT/s (should be 8GT/s), Width x32 (should be x4), ASPM L0s, Exit Latency L0s <64ns

                 ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp-

LnkSta:    Speed 5GT/s (ok), Width x32 (ok)

                 TrErr- Train- SlotClk- DLActive- BWMgmt- ABWMgmt-

 

The DTA-2174B is a PCIe Gen3 x4 card, so link capability should indicate the maximum supported link speed is 8GT/s and link width x4, but somehow in this VM the max link speed is reported as 5GT/s and max link width as x32. Also the link status indicates the card is configured for 5GT/s (which is possible when PCIe is Gen 2 slot) and the width is configured as 32-bit which if that is true this might explain why the driver fails; the system will try to use 32-links where there are only 4 links available, which will undoubtedly result in issues with reading/writing the memory space of the card. Having negotiated a link width of x32 is also strange in itself; as I doubt the slot the card is inserted in is x32 slot (probable is a x16).

 

Does anybody have any hint or experience in fine tuning the way the PCIe passthrough will work?

 

Thanks.

Reply
0 Kudos
0 Replies