Is PowerCLI assigning the incorrect deviceId for PCI Passthrough?

[NetAppJamesGrif]

Version stuffs:

PS C:\project> $PSVersionTable

Name                           Value

----                           -----

PSVersion                      5.1.17763.771

PSEdition                      Desktop

PSCompatibleVersions           {1.0, 2.0, 3.0, 4.0...}

BuildVersion                   10.0.17763.771

CLRVersion                     4.0.30319.42000

WSManStackVersion              3.0

PSRemotingProtocolVersion      2.3

SerializationVersion           1.1.0.1

PS C:\project> Get-Module -ListAvailable | Where-Object { $_.CompanyName -like "VMware*" } | Select-Object ModuleType,Version,Name | Format-Table -AutoSize

ModuleType Version         Name

---------- -------         ----

    Script 6.7.0.11233116  VMware.DeployAutomation

    Script 6.7.0.11233116  VMware.ImageBuilder

  Manifest 11.5.0.14912921 VMware.PowerCLI

    Script 6.7.0.14898114  VMware.Vim

    Script 11.5.0.14898113 VMware.VimAutomation.Cis.Core

    Script 11.0.0.10379994 VMware.VimAutomation.Cloud

    Script 11.5.0.14898112 VMware.VimAutomation.Common

    Script 11.5.0.14899560 VMware.VimAutomation.Core

    Script 6.5.4.7567193   VMware.VimAutomation.HA

    Script 11.5.0.14900247 VMware.VimAutomation.Hcx

    Script 7.10.0.14653756 VMware.VimAutomation.HorizonView

    Script 11.3.0.13990093 VMware.VimAutomation.License

    Script 11.5.0.14900141 VMware.VimAutomation.Nsxt

    Script 10.0.0.7893924  VMware.VimAutomation.PCloud

    Script 11.5.0.14898111 VMware.VimAutomation.Sdk

    Script 11.0.0.10380515 VMware.VimAutomation.Security

    Script 11.5.0.14899557 VMware.VimAutomation.Srm

    Script 11.5.0.14901686 VMware.VimAutomation.Storage

    Script 1.3.0.0         VMware.VimAutomation.StorageUtility

    Script 11.2.0.12483615 VMware.VimAutomation.Vds

    Script 11.5.0.14912923 VMware.VimAutomation.Vmc

    Script 10.0.0.7893921  VMware.VimAutomation.vROps

    Script 2.0.0.0         VMware.vSphereDSC

    Script 6.5.1.7862888   VMware.VumAutomation

PS C:\project>

#####

#####

First attempt using the most simple syntax:

$VMs = Get-VApp $vApp | Get-VM | Sort-Object | ? { $_.Name -notmatch '.*CONSOLE.*|.*DC1|.*GW' }

$kitBfc = $VMhost | Get-PassthroughDevice | Where-Object -Property Name -EQ 'Emulex LightPulse LPe16000 PCIe Fibre Channel Adapter' | Sort-Object -Property Uid | Select-Object -Skip 0 -First 8

$kitAfc = $VMhost | Get-PassthroughDevice | Where-Object -Property Name -EQ 'Emulex LightPulse LPe16000 PCIe Fibre Channel Adapter' | Sort-Object -Property Uid | Select-Object -Skip 8 -First 8

$skip = 0

foreach ($VM in $VMs) {

    $VM.ExtensionData.Reload()

    $VM | Get-PassthroughDevice | Remove-PassthroughDevice

    $VM | Add-PassthroughDevice -PassthroughDevice $($kitBfc | Select-Object -Skip $skip -First 2)

}

<#

RESULTING VMX

-------

pciPassthru0.id = "00000:031:00.0"

pciPassthru0.deviceId = "0xffff"

pciPassthru0.vendorId = "0x10df"

pciPassthru0.systemId = "5dc21a50-2fb7-fe86-c9f1-4c5262353bbc"

pciPassthru0.present = "TRUE"

pciPassthru1.id = "00000:031:00.1"

pciPassthru1.deviceId = "0xffff"

pciPassthru1.vendorId = "0x10df"

pciPassthru1.systemId = "5dc21a50-2fb7-fe86-c9f1-4c5262353bbc"

pciPassthru1.present = "TRUE"

WHEN ATTEMPTING TO START:

The systemId does not match the current system or the deviceId, and the vendorId does not match the device currently at 31:0.0.

#>

#####

#####

Then attempted using the syntax from here: https://communities.vmware.com/thread/573657

$VMs = Get-VApp $vApp | Get-VM | Sort-Object | ? { $_.Name -notmatch '.*CONSOLE.*|.*DC1|.*GW' }

$kitBfc = $VMhost | Get-PassthroughDevice | Where-Object -Property Name -EQ 'Emulex LightPulse LPe16000 PCIe Fibre Channel Adapter' | Sort-Object -Property Uid | Select-Object -Skip 0 -First 8

$kitAfc = $VMhost | Get-PassthroughDevice | Where-Object -Property Name -EQ 'Emulex LightPulse LPe16000 PCIe Fibre Channel Adapter' | Sort-Object -Property Uid | Select-Object -Skip 8 -First 8

$skip = 0

foreach ($VM in $VMs) {

    $VM.ExtensionData.Reload()

    $VM | Get-PassthroughDevice | Remove-PassthroughDevice

    # https://communities.vmware.com/thread/573657

    $currentFC = $kitBfc | Select-Object -Skip $skip -First 2

    $spec = New-Object VMware.Vim.VirtualMachineConfigSpec

    $spec.deviceChange = New-Object VMware.Vim.VirtualDeviceConfigSpec[] (2)

    $spec.deviceChange[0] = New-Object VMware.Vim.VirtualDeviceConfigSpec

    $spec.deviceChange[0].operation = 'add'

    $spec.deviceChange[0].device = New-Object VMware.Vim.VirtualPCIPassthrough

    $spec.deviceChange[0].device.key = "-100"

    $spec.deviceChange[0].device.backing = New-Object VMware.Vim.VirtualPCIPassthroughDeviceBackingInfo

    $spec.deviceChange[0].device.backing.deviceName = $currentFC[0].Name

    $spec.deviceChange[0].device.backing.id = $currentFC[0].ExtensionData.PciDevice.Id

    $spec.deviceChange[0].device.backing.deviceId = $currentFC[0].ExtensionData.PciDevice.DeviceId

    $spec.deviceChange[0].device.backing.systemId = $currentFC[0].ExtensionData.SystemId

    $spec.deviceChange[0].device.backing.vendorId = $currentFC[0].VendorId

    $spec.deviceChange[1] = New-Object VMware.Vim.VirtualDeviceConfigSpec

    $spec.deviceChange[1].operation = 'add'

    $spec.deviceChange[1].device = New-Object VMware.Vim.VirtualPCIPassthrough

    $spec.deviceChange[1].device.key = "-100"

    $spec.deviceChange[1].device.backing = New-Object VMware.Vim.VirtualPCIPassthroughDeviceBackingInfo

    $spec.deviceChange[1].device.backing.deviceName = $currentFC[1].Name

    $spec.deviceChange[1].device.backing.id = $currentFC[1].ExtensionData.PciDevice.Id

    $spec.deviceChange[1].device.backing.deviceId = $currentFC[1].ExtensionData.PciDevice.DeviceId

    $spec.deviceChange[1].device.backing.systemId = $currentFC[1].ExtensionData.SystemId

    $spec.deviceChange[1].device.backing.vendorId = $currentFC[1].VendorId

    $VM.ExtensionData.ReconfigVM_Task($spec)

    $skip = $skip + 2

    $VM.ExtensionData.Reload()

}

<#

RESULTING VMX

-------

pciPassthru0.id = "00000:031:00.0"

pciPassthru0.deviceId = "0x8980"

pciPassthru0.vendorId = "0x10df"

pciPassthru0.systemId = "5dc21a50-2fb7-fe86-c9f1-4c5262353bbc"

pciPassthru0.present = "TRUE"

pciPassthru1.id = "00000:031:00.1"

pciPassthru1.deviceId = "0x8980"

pciPassthru1.vendorId = "0x10df"

pciPassthru1.systemId = "5dc21a50-2fb7-fe86-c9f1-4c5262353bbc"

pciPassthru1.present = "TRUE"

WHEN ATTEMPTING TO START:

The systemId does not match the current system or the deviceId, and the vendorId does not match the device currently at 31:0.0.

#>

#####

#####

Finally assigned using HTML5 UI

<#

EXPECTED VMX

    pciPassthru0.id = "00000:031:00.0"

    pciPassthru0.deviceId = "0xe200"

    pciPassthru0.vendorId = "0x10df"

    pciPassthru0.systemId = "5dc21a50-2fb7-fe86-c9f1-4c5262353bbc"

    pciPassthru0.present = "TRUE"

    pciPassthru1.id = "00000:031:00.1"

    pciPassthru1.deviceId = "0xe200"

    pciPassthru1.vendorId = "0x10df"

    pciPassthru1.systemId = "5dc21a50-2fb7-fe86-c9f1-4c5262353bbc"

    pciPassthru1.present = "TRUE"

NO ISSUE BOOTING

#>

What's going on here?

Also, how can one add a code box to this editor?

I tried <pre></pre> and <code></code>

[LucD]

Method 1 & 2 indeed look like a "feature".
Not sure if it will show anything, but did you try using Code Capture while doing the action via the Web Client?

I would, in any case, suggest opening an SR.

You can copy HTML code directly in your reply.
I normally get my syntax-coloured code from VisualStudio Code.

[NetAppJamesGrif]

Looks like I don't have that:

[NetAppJamesGrif]

Looks like that's a 6.7 feature? We're running 6.5u3.

[LucD]

You can use the HTML5 Fling in vSphere 6.5 afaik.

Kyle did a post on that, see https://www.kmruddy.com/2018/powercli-code-capture/

[NetAppJamesGrif · Accepted solution]

Alright, I'm back after the Ops team updated VCSA to 6.7.

I hit a snag though in that mapping what the WebUI used for DeviceId did *not* align with *anything*

#---------------ReconfigVM_Task---------------

$spec = New-Object VMware.Vim.VirtualMachineConfigSpec

$spec.DeviceChange = New-Object VMware.Vim.VirtualDeviceConfigSpec[] (2)

$spec.DeviceChange[0] = New-Object VMware.Vim.VirtualDeviceConfigSpec

$spec.DeviceChange[0].Device = New-Object VMware.Vim.VirtualPCIPassthrough

$spec.DeviceChange[0].Device.Backing = New-Object VMware.Vim.VirtualPCIPassthroughDeviceBackingInfo

$spec.DeviceChange[0].Device.Backing.SystemId = '5d67cacd-3ea2-f27c-5400-001b21949eb0'                      # $ThisVMDevices[0].ExtensionData.SystemId

$spec.DeviceChange[0].Device.Backing.VendorId = 4319                                                        # $ThisVMDevices[0].ExtensionData.PciDevice.VendorId

$spec.DeviceChange[0].Device.Backing.Id = '0000:12:00.0'                                                    # $ThisVMDevices[0].ExtensionData.PciDevice.Id

$spec.DeviceChange[0].Device.Backing.DeviceId = '-1e00'                                                     # $ThisVMDevices[0].ExtensionData.PciDevice.DeviceId ? '-7680'

$spec.DeviceChange[0].Device.Backing.DeviceName = 'Emulex LightPulse LPe16000 PCIe Fibre Channel Adapter'   # $ThisVMDevices[0].ExtensionData.PciDevice.DeviceName

$spec.DeviceChange[0].Device.DeviceInfo = New-Object VMware.Vim.Description

$spec.DeviceChange[0].Device.DeviceInfo.Summary = 'New PCI device'

$spec.DeviceChange[0].Device.DeviceInfo.Label = 'New PCI device'

$spec.DeviceChange[0].Device.Key = -101

$spec.DeviceChange[0].Operation = 'add'

$spec.DeviceChange[1] = New-Object VMware.Vim.VirtualDeviceConfigSpec

$spec.DeviceChange[1].Device = New-Object VMware.Vim.VirtualPCIPassthrough

$spec.DeviceChange[1].Device.Backing = New-Object VMware.Vim.VirtualPCIPassthroughDeviceBackingInfo

$spec.DeviceChange[1].Device.Backing.SystemId = '5d67cacd-3ea2-f27c-5400-001b21949eb0'

$spec.DeviceChange[1].Device.Backing.VendorId = 4319

$spec.DeviceChange[1].Device.Backing.Id = '0000:12:00.1'

$spec.DeviceChange[1].Device.Backing.DeviceId = '-1e00'

$spec.DeviceChange[1].Device.Backing.DeviceName = 'Emulex LightPulse LPe16000 PCIe Fibre Channel Adapter'

$spec.DeviceChange[1].Device.DeviceInfo = New-Object VMware.Vim.Description

$spec.DeviceChange[1].Device.DeviceInfo.Summary = 'New PCI device'

$spec.DeviceChange[1].Device.DeviceInfo.Label = 'New PCI device'

$spec.DeviceChange[1].Device.Key = -102

$spec.DeviceChange[1].Operation = 'add'

$spec.CpuFeatureMask = New-Object VMware.Vim.VirtualMachineCpuIdInfoSpec[] (0)

$_this = Get-View -Id $VM.Id

$_this.ReconfigVM_Task($spec)

END

$ThisVMDevices[0] contains the exact same object as the above DeviceChange[0].

PS > $ThisVMDevices[0].ExtensionData.PciDevice

Id           : 0000:12:00.0

ClassId      : 3076

Bus          : 18

Slot         : 0

Function     : 0

VendorId     : 4319

SubVendorId  : 4319

VendorName   : Emulex Corporation

DeviceId     : -7680

SubDeviceId  : -1263

ParentBridge : 0000:10:02.0

DeviceName   : Emulex LightPulse LPe16000 PCIe Fibre Channel Adapter

PS >

END

When reviewing the documentation for VMware.Vim.VirtualPCIPassthroughDeviceBackingInfo.DeviceId here VMware vSphere 5.1 it was very convoluted as to what should be passed.

> The device ID of this PCI. You must use the device ID retrieved from the vSphere host (HostPciDevice.deviceId), converted as is to a string.

The recorded example provides `-1e00`

The PCI device provides `-7680`

The WebUI displays `E200`

I finally ran across this Python example: Add PCI passthrough device to a VM using pyvmomi · GitHub

After converting that syntax to the PowerShell equivalent, this is what worked:

$VMs = Get-VApp $LabInstanceName | Get-VM | Sort-Object | Where-Object { <#FILTER#> }

$ThisSkip = 0

foreach ($VM in $VMs) {

    Write-Host "Updating $VM"

    $ThisVMDevices = $ThisKitDevices | Select-Object -First 2 -Skip $ThisSkip

    # Add-PassthroughDevice -VM $VM -PassthroughDevice $ThisVMDevices -Confirm:$false

    # That's not working...

    $spec = New-Object VMware.Vim.VirtualMachineConfigSpec

    $spec.DeviceChange = New-Object VMware.Vim.VirtualDeviceConfigSpec[] (2)

    $spec.DeviceChange[0] = New-Object VMware.Vim.VirtualDeviceConfigSpec

    $spec.DeviceChange[0].Device = New-Object VMware.Vim.VirtualPCIPassthrough

    $spec.DeviceChange[0].Device.Backing = New-Object VMware.Vim.VirtualPCIPassthroughDeviceBackingInfo

    $spec.DeviceChange[0].Device.Backing.SystemId =                     $ThisVMDevices[0].ExtensionData.SystemId

    $spec.DeviceChange[0].Device.Backing.VendorId =                     $ThisVMDevices[0].ExtensionData.PciDevice.VendorId

    $spec.DeviceChange[0].Device.Backing.Id =                           $ThisVMDevices[0].ExtensionData.PciDevice.Id

    $spec.DeviceChange[0].Device.Backing.DeviceId = [Convert]::ToString($ThisVMDevices[0].ExtensionData.PciDevice.DeviceId, 16)

    $spec.DeviceChange[0].Device.Backing.DeviceName =                   $ThisVMDevices[0].ExtensionData.PciDevice.DeviceName

    $spec.DeviceChange[0].Device.DeviceInfo = New-Object VMware.Vim.Description

    $spec.DeviceChange[0].Device.DeviceInfo.Summary = 'New PCI device'

    $spec.DeviceChange[0].Device.DeviceInfo.Label = 'New PCI device'

    $spec.DeviceChange[0].Device.Key = -101

    $spec.DeviceChange[0].Operation = 'add'

    $spec.DeviceChange[1] = New-Object VMware.Vim.VirtualDeviceConfigSpec

    $spec.DeviceChange[1].Device = New-Object VMware.Vim.VirtualPCIPassthrough

    $spec.DeviceChange[1].Device.Backing = New-Object VMware.Vim.VirtualPCIPassthroughDeviceBackingInfo

    $spec.DeviceChange[1].Device.Backing.SystemId =                     $ThisVMDevices[1].ExtensionData.SystemId

    $spec.DeviceChange[1].Device.Backing.VendorId =                     $ThisVMDevices[1].ExtensionData.PciDevice.VendorId

    $spec.DeviceChange[1].Device.Backing.Id =                           $ThisVMDevices[1].ExtensionData.PciDevice.Id

    $spec.DeviceChange[1].Device.Backing.DeviceId = [Convert]::ToString($ThisVMDevices[1].ExtensionData.PciDevice.DeviceId, 16)

    $spec.DeviceChange[1].Device.Backing.DeviceName =                   $ThisVMDevices[1].ExtensionData.PciDevice.DeviceName

    $spec.DeviceChange[1].Device.DeviceInfo = New-Object VMware.Vim.Description

    $spec.DeviceChange[1].Device.DeviceInfo.Summary = 'New PCI device'

    $spec.DeviceChange[1].Device.DeviceInfo.Label = 'New PCI device'

    $spec.DeviceChange[1].Device.Key = -102

    $spec.DeviceChange[1].Operation = 'add'

    $spec.CpuFeatureMask = New-Object VMware.Vim.VirtualMachineCpuIdInfoSpec[] (0)

    $_this = Get-View -Id $VM.Id

    $_this.ReconfigVM_Task($spec)

    $ThisSkip += 2

}

## END Assign $ThisKitDevices to respective VMs

END

(I was copy/pasting directly from VSCode PS1 file to try to get syntax highlighting this time.)