vSphere 6.0: VM cannot boot after assigning PCI passthrough QAT Co-processor by PowerCLI

[niexf]

The PCI device was an Intel QAT (Quick Assistant Technology) device.

This (add PCI passthrough then poweron) can be done by vSphere Client, or vSphere Web Client

but failed to do it via powershell CLI: when the device was added via powershell CLI, the VM cannot boot, no matter by powercli or vSphere Client.

The error message from powershell:

DEBUG: 7/13/2017 12:36:44 PM Start-VM  The operation for the entity

"colin-61-4" failed with the following message: "The systemId does not match

the current system or the deviceId, and the vendorId does not match the device

currently at 97:0.0.". The systemId does not match the current system or the

deviceId, and the vendorId does not match the device currently at 97:0.0.

Failed to start the virtual machine. An error was received from the ESX host

while powering on VM colin-61-4.

I tried to get /dump the PCI device with powercli passed through the VM via both vSphere Client and powercli, they seemed identical. details attached in the end of this post

The only suspect was the VendorID was dumped to a negative integer, could it be that the Vendor ID was saved incorrectly when adding the device via powercli for that the error message said venderid mismatch?

And there's a thread reporting similar issue like this, which seemed a bug in vSphere web interface: ESXi 6.0 - VM can't boot after assigning PCIE GPU passthrough : vmware

1.  script used to dump the device:

    $pass_set1 = $vm | Get-PassthroughDevice -Type 'Pci'

    ForEach ($d in $pass_set1) {

        Write-Debug $d.Uid

        Write-Debug($d | Format-List | Out-String)

    }   

2. when the device was added via vSphere Client, in this case, the VM can boot smoothly.

the device was dumped as followed:

DEBUG: 7/13/2017 12:10:08 PM Get-PassthroughDevice

DEBUG:

/VIServer=root@10.32.228.84:443/PassthroughDevice=10.32.228.154-VirtualMachine-

vm-1945-<class> Co-processor-0000:61:00.0/

DEBUG:

Bus           : 97

ClassId       : 2880

DeviceId      : 14280

Function      : 0

Slot          : 0

State         : Active

VendorId      : -32634

Uid           :

/VIServer=root@10.32.228.84:443/PassthroughDevice=10.32.228.154

                -VirtualMachine-vm-1945-<class> Co-processor-0000:61:00.0/

VendorName    : Intel Corporation

Name          : <class> Co-processor

VMId          : VirtualMachine-vm-1945

VM            : colin-61-4

VMHostId      : HostSystem-host-1415

VMHost        : 10.32.228.154

Key           : 13000

ExtensionData : VMware.Vim.VirtualMachinePciPassthroughInfo

Client        : VMware.VimAutomation.ViCore.Impl.V1.VimClient

3. when the device was added via powercli, VM cannot boot.

the device was dumped as followed:

DEBUG: 7/13/2017 1:23:42 PM Get-PassthroughDevice

DEBUG:

/VIServer=root@10.32.228.84:443/PassthroughDevice=10.32.228.154-VirtualMachine-

vm-1945-<class> Co-processor-0000:61:00.0/

DEBUG:

Bus           : 97

ClassId       : 2880

DeviceId      : 14280

Function      : 0

Slot          : 0

State         : Active

VendorId      : -32634

Uid           :

/VIServer=root@10.32.228.84:443/PassthroughDevice=10.32.228.154

                -VirtualMachine-vm-1945-<class> Co-processor-0000:61:00.0/

VendorName    : Intel Corporation

Name          : <class> Co-processor

VMId          : VirtualMachine-vm-1945

VM            : colin-61-4

VMHostId      : HostSystem-host-1415

VMHost        : 10.32.228.154

Key           : 13000

ExtensionData : VMware.Vim.VirtualMachinePciPassthroughInfo

Client        : VMware.VimAutomation.ViCore.Impl.V1.VimClient

[LucD]

Would you mind showing the Get-PassThroughDevice -VMHost xxx and Add-PassThroughDevice lines as well.

How do you select the correct device when there are multiple ones on that ESXi node?

$vmName = 'MyVM'

$esx = Get-VMHost -VM $vmName

Get-PassthroughDevice -VMHost $esx | select *

[niexf]

Thanks, LucD,

here's the add script and debug log:

1. script:

    Write-Debug "get existing passthrough device in vm host"

    $host_pass = $vm | Get-VMHost | Get-PassthroughDevice -Type $passtype

    ForEach ($d in $host_pass) {

        Write-Debug $d.Uid

    }   

    $new_pass = $host_pass | ?{$_.Name.Contains("$passname") -And $_.Uid.Contains(":${passid}:")}

    if($new_pass.count -eq 0) {

        Write-Debug "passthrough device $passid didn't exist"

        exit 0

    }   

2. debug log:

DEBUG: Test_Add_Passthr colin-61-4 Pci 61 Co-processor

DEBUG: vmname: colin-61-4

DEBUG: passthr type: Pci

DEBUG: passthr id: 61

DEBUG: passthr name: Co-processor

DEBUG: 7/13/2017 12:14:51 PM Get-VM

DEBUG: get existing passthrough device in vm

DEBUG: 7/13/2017 12:14:51 PM Get-PassthroughDevice

DEBUG: get existing passthrough device in vm host

DEBUG: 7/13/2017 12:15:03 PM Get-VMHost

DEBUG: 7/13/2017 12:15:03 PM Get-PassthroughDevice

DEBUG: /VIServer=root@10.32.228.84:443/PassthroughDevice=10.32.228.154--<class>

Co-processor-0000:3d:00.0/

DEBUG: /VIServer=root@10.32.228.84:443/PassthroughDevice=10.32.228.154--<class>

Co-processor-0000:3e:00.0/

DEBUG: /VIServer=root@10.32.228.84:443/PassthroughDevice=10.32.228.154--<class>

Co-processor-0000:3f:00.0/

DEBUG: /VIServer=root@10.32.228.84:443/PassthroughDevice=10.32.228.154--<class>

Co-processor-0000:60:00.0/

DEBUG: /VIServer=root@10.32.228.84:443/PassthroughDevice=10.32.228.154--<class>

Co-processor-0000:61:00.0/

DEBUG: /VIServer=root@10.32.228.84:443/PassthroughDevice=10.32.228.154--<class>

Co-processor-0000:88:00.0/

DEBUG: /VIServer=root@10.32.228.84:443/PassthroughDevice=10.32.228.154--<class>

Co-processor-0000:89:00.0/

DEBUG: /VIServer=root@10.32.228.84:443/PassthroughDevice=10.32.228.154--<class>

Audio device-0000:00:1f.3/

DEBUG:

/VIServer=root@10.32.228.84:443/PassthroughDevice=10.32.228.154--LSI2008-0000:0

1:00.0/

DEBUG: /VIServer=root@10.32.228.84:443/PassthroughDevice=10.32.228.154--MGA

G200e [Pilot] ServerEngines (SEP1)-0000:03:00.0/

DEBUG: /VIServer=root@10.32.228.84:443/PassthroughDevice=10.32.228.154--<class>

Ethernet controller-0000:40:00.0/

DEBUG: /VIServer=root@10.32.228.84:443/PassthroughDevice=10.32.228.154--<class>

Ethernet controller-0000:40:00.1/

DEBUG: /VIServer=root@10.32.228.84:443/PassthroughDevice=10.32.228.154--<class>

Ethernet controller-0000:40:00.2/

DEBUG: /VIServer=root@10.32.228.84:443/PassthroughDevice=10.32.228.154--<class>

Ethernet controller-0000:40:00.3/

DEBUG: /VIServer=root@10.32.228.84:443/PassthroughDevice=10.32.228.154--<class>

Ethernet controller-0000:62:00.0/

DEBUG: /VIServer=root@10.32.228.84:443/PassthroughDevice=10.32.228.154--<class>

Ethernet controller-0000:62:00.1/

DEBUG: /VIServer=root@10.32.228.84:443/PassthroughDevice=10.32.228.154--<class>

Ethernet controller-0000:62:00.2/

DEBUG: /VIServer=root@10.32.228.84:443/PassthroughDevice=10.32.228.154--<class>

Ethernet controller-0000:62:00.3/

DEBUG: /VIServer=root@10.32.228.84:443/PassthroughDevice=10.32.228.154--<class>

Memory controller-0000:63:00.0/

DEBUG: /VIServer=root@10.32.228.84:443/PassthroughDevice=10.32.228.154--<class>

Signal processing controller-0000:64:00.0/

DEBUG: /VIServer=root@10.32.228.84:443/PassthroughDevice=10.32.228.154--82571EB

Gigabit Ethernet Controller-0000:81:00.0/

DEBUG: /VIServer=root@10.32.228.84:443/PassthroughDevice=10.32.228.154--82571EB

Gigabit Ethernet Controller-0000:81:00.1/

DEBUG: /VIServer=root@10.32.228.84:443/PassthroughDevice=10.32.228.154--<class>

Ethernet controller-0000:8a:00.0/

DEBUG: /VIServer=root@10.32.228.84:443/PassthroughDevice=10.32.228.154--<class>

Ethernet controller-0000:8a:00.1/

DEBUG: /VIServer=root@10.32.228.84:443/PassthroughDevice=10.32.228.154--<class>

Ethernet controller-0000:8a:00.2/

DEBUG: /VIServer=root@10.32.228.84:443/PassthroughDevice=10.32.228.154--<class>

Ethernet controller-0000:8a:00.3/

DEBUG: /VIServer=root@10.32.228.84:443/PassthroughDevice=10.32.228.154--<class>

Memory controller-0000:8b:00.0/

DEBUG: /VIServer=root@10.32.228.84:443/PassthroughDevice=10.32.228.154--<class>

Signal processing controller-0000:8c:00.0/

DEBUG: /VIServer=root@10.32.228.84:443/PassthroughDevice=10.32.228.154--GT218

[GeForce 210]-0000:af:00.0/

DEBUG: /VIServer=root@10.32.228.84:443/PassthroughDevice=10.32.228.154--High

Definition Audio Controller-0000:af:00.1/

DEBUG: add a new passthrough device: 61

Bus           : 97

ClassId       : 2880

DeviceId      : 14280

Function      : 0

Slot          : 0

State         : Active

VendorId      : -32634

Uid           : /VIServer=root@10.32.228.84:443/PassthroughDevice=10.32.228.154

                -VirtualMachine-vm-1945-<class> Co-processor-0000:61:00.0/

VendorName    : Intel Corporation

Name          : <class> Co-processor

VMId          : VirtualMachine-vm-1945

VM            : colin-61-4

VMHostId      : HostSystem-host-1415

VMHost        : 10.32.228.154

Key           : 13000

ExtensionData : VMware.Vim.VirtualMachinePciPassthroughInfo

Client        : VMware.VimAutomation.ViCore.Impl.V1.VimClient

[LucD]

That looks quite ok.

Are you sure you are powering on the VM on the ESXi node that has the passthrough device you attached to the VM?

[niexf]

Sure, I did the power on via both vSphere Client and powercli (Start-Vm), neither worked. both show the same error message

(as the vm name was passed to the script only once for both operations of add PCI and power on, it cannot be a wrong VM be powered on)

if I add the PCI device via vSphere then the VM can be powered on smoothly via both vSphere Client and powercli

[LucD]

That looks like a "feature".

I suggest you open a SR