5 Replies Latest reply on Feb 10, 2020 8:21 AM by NetAppJamesGriffith

    Is PowerCLI assigning the incorrect deviceId for PCI Passthrough?

    NetAppJamesGriffith Lurker

      Version stuffs:

       

      PS C:\project> $PSVersionTable

       

      Name                           Value

      ----                           -----

      PSVersion                      5.1.17763.771

      PSEdition                      Desktop

      PSCompatibleVersions           {1.0, 2.0, 3.0, 4.0...}

      BuildVersion                   10.0.17763.771

      CLRVersion                     4.0.30319.42000

      WSManStackVersion              3.0

      PSRemotingProtocolVersion      2.3

      SerializationVersion           1.1.0.1

       

       

      PS C:\project> Get-Module -ListAvailable | Where-Object { $_.CompanyName -like "VMware*" } | Select-Object ModuleType,Version,Name | Format-Table -AutoSize

       

      ModuleType Version         Name

      ---------- -------         ----

          Script 6.7.0.11233116  VMware.DeployAutomation

          Script 6.7.0.11233116  VMware.ImageBuilder

        Manifest 11.5.0.14912921 VMware.PowerCLI

          Script 6.7.0.14898114  VMware.Vim

          Script 11.5.0.14898113 VMware.VimAutomation.Cis.Core

          Script 11.0.0.10379994 VMware.VimAutomation.Cloud

          Script 11.5.0.14898112 VMware.VimAutomation.Common

          Script 11.5.0.14899560 VMware.VimAutomation.Core

          Script 6.5.4.7567193   VMware.VimAutomation.HA

          Script 11.5.0.14900247 VMware.VimAutomation.Hcx

          Script 7.10.0.14653756 VMware.VimAutomation.HorizonView

          Script 11.3.0.13990093 VMware.VimAutomation.License

          Script 11.5.0.14900141 VMware.VimAutomation.Nsxt

          Script 10.0.0.7893924  VMware.VimAutomation.PCloud

          Script 11.5.0.14898111 VMware.VimAutomation.Sdk

          Script 11.0.0.10380515 VMware.VimAutomation.Security

          Script 11.5.0.14899557 VMware.VimAutomation.Srm

          Script 11.5.0.14901686 VMware.VimAutomation.Storage

          Script 1.3.0.0         VMware.VimAutomation.StorageUtility

          Script 11.2.0.12483615 VMware.VimAutomation.Vds

          Script 11.5.0.14912923 VMware.VimAutomation.Vmc

          Script 10.0.0.7893921  VMware.VimAutomation.vROps

          Script 2.0.0.0         VMware.vSphereDSC

          Script 6.5.1.7862888   VMware.VumAutomation

       

       

      PS C:\project>

       

      #####

      #####

       

      First attempt using the most simple syntax:

       

      $VMs = Get-VApp $vApp | Get-VM | Sort-Object | ? { $_.Name -notmatch '.*CONSOLE.*|.*DC1|.*GW' }

       

      $kitBfc = $VMhost | Get-PassthroughDevice | Where-Object -Property Name -EQ 'Emulex LightPulse LPe16000 PCIe Fibre Channel Adapter' | Sort-Object -Property Uid | Select-Object -Skip 0 -First 8

      $kitAfc = $VMhost | Get-PassthroughDevice | Where-Object -Property Name -EQ 'Emulex LightPulse LPe16000 PCIe Fibre Channel Adapter' | Sort-Object -Property Uid | Select-Object -Skip 8 -First 8

       

      $skip = 0

       

      foreach ($VM in $VMs) {

          $VM.ExtensionData.Reload()

          $VM | Get-PassthroughDevice | Remove-PassthroughDevice

       

          $VM | Add-PassthroughDevice -PassthroughDevice $($kitBfc | Select-Object -Skip $skip -First 2)

      }

      <#

      RESULTING VMX

      -------

       

      pciPassthru0.id = "00000:031:00.0"

      pciPassthru0.deviceId = "0xffff"

      pciPassthru0.vendorId = "0x10df"

      pciPassthru0.systemId = "5dc21a50-2fb7-fe86-c9f1-4c5262353bbc"

      pciPassthru0.present = "TRUE"

       

      pciPassthru1.id = "00000:031:00.1"

      pciPassthru1.deviceId = "0xffff"

      pciPassthru1.vendorId = "0x10df"

      pciPassthru1.systemId = "5dc21a50-2fb7-fe86-c9f1-4c5262353bbc"

      pciPassthru1.present = "TRUE"

       

      WHEN ATTEMPTING TO START:

      The systemId does not match the current system or the deviceId, and the vendorId does not match the device currently at 31:0.0.

       

      #>

       

      #####

      #####

       

      Then attempted using the syntax from here: https://communities.vmware.com/thread/573657

       

      $VMs = Get-VApp $vApp | Get-VM | Sort-Object | ? { $_.Name -notmatch '.*CONSOLE.*|.*DC1|.*GW' }

       

      $kitBfc = $VMhost | Get-PassthroughDevice | Where-Object -Property Name -EQ 'Emulex LightPulse LPe16000 PCIe Fibre Channel Adapter' | Sort-Object -Property Uid | Select-Object -Skip 0 -First 8

      $kitAfc = $VMhost | Get-PassthroughDevice | Where-Object -Property Name -EQ 'Emulex LightPulse LPe16000 PCIe Fibre Channel Adapter' | Sort-Object -Property Uid | Select-Object -Skip 8 -First 8

       

      $skip = 0

       

       

      foreach ($VM in $VMs) {

          $VM.ExtensionData.Reload()

          $VM | Get-PassthroughDevice | Remove-PassthroughDevice

         

          # https://communities.vmware.com/thread/573657

       

          $currentFC = $kitBfc | Select-Object -Skip $skip -First 2

       

          $spec = New-Object VMware.Vim.VirtualMachineConfigSpec

          $spec.deviceChange = New-Object VMware.Vim.VirtualDeviceConfigSpec[] (2)

       

          $spec.deviceChange[0] = New-Object VMware.Vim.VirtualDeviceConfigSpec

          $spec.deviceChange[0].operation = 'add'

          $spec.deviceChange[0].device = New-Object VMware.Vim.VirtualPCIPassthrough

          $spec.deviceChange[0].device.key = "-100"

          $spec.deviceChange[0].device.backing = New-Object VMware.Vim.VirtualPCIPassthroughDeviceBackingInfo

          $spec.deviceChange[0].device.backing.deviceName = $currentFC[0].Name

          $spec.deviceChange[0].device.backing.id = $currentFC[0].ExtensionData.PciDevice.Id

          $spec.deviceChange[0].device.backing.deviceId = $currentFC[0].ExtensionData.PciDevice.DeviceId

          $spec.deviceChange[0].device.backing.systemId = $currentFC[0].ExtensionData.SystemId

          $spec.deviceChange[0].device.backing.vendorId = $currentFC[0].VendorId

          $spec.deviceChange[1] = New-Object VMware.Vim.VirtualDeviceConfigSpec

          $spec.deviceChange[1].operation = 'add'

          $spec.deviceChange[1].device = New-Object VMware.Vim.VirtualPCIPassthrough

          $spec.deviceChange[1].device.key = "-100"

          $spec.deviceChange[1].device.backing = New-Object VMware.Vim.VirtualPCIPassthroughDeviceBackingInfo

          $spec.deviceChange[1].device.backing.deviceName = $currentFC[1].Name

          $spec.deviceChange[1].device.backing.id = $currentFC[1].ExtensionData.PciDevice.Id

          $spec.deviceChange[1].device.backing.deviceId = $currentFC[1].ExtensionData.PciDevice.DeviceId

          $spec.deviceChange[1].device.backing.systemId = $currentFC[1].ExtensionData.SystemId

          $spec.deviceChange[1].device.backing.vendorId = $currentFC[1].VendorId

       

          $VM.ExtensionData.ReconfigVM_Task($spec)

       

       

          $skip = $skip + 2

          $VM.ExtensionData.Reload()

      }

       

      <#

      RESULTING VMX

      -------

       

      pciPassthru0.id = "00000:031:00.0"

      pciPassthru0.deviceId = "0x8980"

      pciPassthru0.vendorId = "0x10df"

      pciPassthru0.systemId = "5dc21a50-2fb7-fe86-c9f1-4c5262353bbc"

      pciPassthru0.present = "TRUE"

       

      pciPassthru1.id = "00000:031:00.1"

      pciPassthru1.deviceId = "0x8980"

      pciPassthru1.vendorId = "0x10df"

      pciPassthru1.systemId = "5dc21a50-2fb7-fe86-c9f1-4c5262353bbc"

      pciPassthru1.present = "TRUE"

       

      WHEN ATTEMPTING TO START:

      The systemId does not match the current system or the deviceId, and the vendorId does not match the device currently at 31:0.0.

       

      #>

       

      #####

      #####

       

      Finally assigned using HTML5 UI

       

      <#

       

      EXPECTED VMX

       

          pciPassthru0.id = "00000:031:00.0"

          pciPassthru0.deviceId = "0xe200"

          pciPassthru0.vendorId = "0x10df"

          pciPassthru0.systemId = "5dc21a50-2fb7-fe86-c9f1-4c5262353bbc"

          pciPassthru0.present = "TRUE"

         

          pciPassthru1.id = "00000:031:00.1"

          pciPassthru1.deviceId = "0xe200"

          pciPassthru1.vendorId = "0x10df"

          pciPassthru1.systemId = "5dc21a50-2fb7-fe86-c9f1-4c5262353bbc"

          pciPassthru1.present = "TRUE"

       

      NO ISSUE BOOTING

       

      #>

       

      What's going on here?

       

      Also, how can one add a code box to this editor?

      I tried <pre></pre> and <code></code>

        • 1. Re: Is PowerCLI assigning the incorrect deviceId for PCI Passthrough?
          LucD Guru
          User ModeratorsCommunity WarriorsvExpert

          Method 1 & 2 indeed look like a "feature".
          Not sure if it will show anything, but did you try using Code Capture while doing the action via the Web Client?

           

          I would, in any case, suggest opening an SR.

           

          You can copy HTML code directly in your reply.
          I normally get my syntax-coloured code from VisualStudio Code.

          Blog: http://lucd.info | Twitter: @LucD22 | PowerCLI Reference co-author: http://tinyurl.com/hkn4glz
          • 3. Re: Is PowerCLI assigning the incorrect deviceId for PCI Passthrough?
            NetAppJamesGriffith Lurker

            Looks like that's a 6.7 feature? We're running 6.5u3.

            • 4. Re: Is PowerCLI assigning the incorrect deviceId for PCI Passthrough?
              LucD Guru
              Community WarriorsUser ModeratorsvExpert

              You can use the HTML5 Fling in vSphere 6.5 afaik.

              Kyle did a post on that, see https://www.kmruddy.com/2018/powercli-code-capture/

              Blog: http://lucd.info | Twitter: @LucD22 | PowerCLI Reference co-author: http://tinyurl.com/hkn4glz
              • 5. Re: Is PowerCLI assigning the incorrect deviceId for PCI Passthrough?
                NetAppJamesGriffith Lurker

                Alright, I'm back after the Ops team updated VCSA to 6.7.

                 

                I hit a snag though in that mapping what the WebUI used for DeviceId did *not* align with *anything*

                 

                 

                #---------------ReconfigVM_Task---------------

                 

                $spec = New-Object VMware.Vim.VirtualMachineConfigSpec

                $spec.DeviceChange = New-Object VMware.Vim.VirtualDeviceConfigSpec[] (2)

                $spec.DeviceChange[0] = New-Object VMware.Vim.VirtualDeviceConfigSpec

                $spec.DeviceChange[0].Device = New-Object VMware.Vim.VirtualPCIPassthrough

                $spec.DeviceChange[0].Device.Backing = New-Object VMware.Vim.VirtualPCIPassthroughDeviceBackingInfo

                $spec.DeviceChange[0].Device.Backing.SystemId = '5d67cacd-3ea2-f27c-5400-001b21949eb0'                      # $ThisVMDevices[0].ExtensionData.SystemId

                $spec.DeviceChange[0].Device.Backing.VendorId = 4319                                                        # $ThisVMDevices[0].ExtensionData.PciDevice.VendorId

                $spec.DeviceChange[0].Device.Backing.Id = '0000:12:00.0'                                                    # $ThisVMDevices[0].ExtensionData.PciDevice.Id

                $spec.DeviceChange[0].Device.Backing.DeviceId = '-1e00'                                                     # $ThisVMDevices[0].ExtensionData.PciDevice.DeviceId ? '-7680'

                $spec.DeviceChange[0].Device.Backing.DeviceName = 'Emulex LightPulse LPe16000 PCIe Fibre Channel Adapter'   # $ThisVMDevices[0].ExtensionData.PciDevice.DeviceName

                $spec.DeviceChange[0].Device.DeviceInfo = New-Object VMware.Vim.Description

                $spec.DeviceChange[0].Device.DeviceInfo.Summary = 'New PCI device'

                $spec.DeviceChange[0].Device.DeviceInfo.Label = 'New PCI device'

                $spec.DeviceChange[0].Device.Key = -101

                $spec.DeviceChange[0].Operation = 'add'

                $spec.DeviceChange[1] = New-Object VMware.Vim.VirtualDeviceConfigSpec

                $spec.DeviceChange[1].Device = New-Object VMware.Vim.VirtualPCIPassthrough

                $spec.DeviceChange[1].Device.Backing = New-Object VMware.Vim.VirtualPCIPassthroughDeviceBackingInfo

                $spec.DeviceChange[1].Device.Backing.SystemId = '5d67cacd-3ea2-f27c-5400-001b21949eb0'

                $spec.DeviceChange[1].Device.Backing.VendorId = 4319

                $spec.DeviceChange[1].Device.Backing.Id = '0000:12:00.1'

                $spec.DeviceChange[1].Device.Backing.DeviceId = '-1e00'

                $spec.DeviceChange[1].Device.Backing.DeviceName = 'Emulex LightPulse LPe16000 PCIe Fibre Channel Adapter'

                $spec.DeviceChange[1].Device.DeviceInfo = New-Object VMware.Vim.Description

                $spec.DeviceChange[1].Device.DeviceInfo.Summary = 'New PCI device'

                $spec.DeviceChange[1].Device.DeviceInfo.Label = 'New PCI device'

                $spec.DeviceChange[1].Device.Key = -102

                $spec.DeviceChange[1].Operation = 'add'

                $spec.CpuFeatureMask = New-Object VMware.Vim.VirtualMachineCpuIdInfoSpec[] (0)

                $_this = Get-View -Id $VM.Id

                $_this.ReconfigVM_Task($spec)

                END

                 

                $ThisVMDevices[0] contains the exact same object as the above DeviceChange[0].

                 

                 

                PS > $ThisVMDevices[0].ExtensionData.PciDevice

                 


                Id           : 0000:12:00.0

                ClassId      : 3076

                Bus          : 18

                Slot         : 0

                Function     : 0

                VendorId     : 4319

                SubVendorId  : 4319

                VendorName   : Emulex Corporation

                DeviceId     : -7680

                SubDeviceId  : -1263

                ParentBridge : 0000:10:02.0

                DeviceName   : Emulex LightPulse LPe16000 PCIe Fibre Channel Adapter


                PS >

                END

                 

                When reviewing the documentation for VMware.Vim.VirtualPCIPassthroughDeviceBackingInfo.DeviceId here VMware vSphere 5.1 it was very convoluted as to what should be passed.

                > The device ID of this PCI. You must use the device ID retrieved from the vSphere host (HostPciDevice.deviceId), converted as is to a string.

                 

                The recorded example provides `-1e00`

                The PCI device provides `-7680`

                The WebUI displays `E200`

                 

                I finally ran across this Python example: Add PCI passthrough device to a VM using pyvmomi · GitHub

                 

                After converting that syntax to the PowerShell equivalent, this is what worked:

                 

                 

                $VMs = Get-VApp $LabInstanceName | Get-VM | Sort-Object | Where-Object { <#FILTER#> }

                 

                $ThisSkip = 0

                foreach ($VM in $VMs) {

                    Write-Host "Updating $VM"

                    $ThisVMDevices = $ThisKitDevices | Select-Object -First 2 -Skip $ThisSkip


                    # Add-PassthroughDevice -VM $VM -PassthroughDevice $ThisVMDevices -Confirm:$false

                    # That's not working...


                    $spec = New-Object VMware.Vim.VirtualMachineConfigSpec

                    $spec.DeviceChange = New-Object VMware.Vim.VirtualDeviceConfigSpec[] (2)

                    $spec.DeviceChange[0] = New-Object VMware.Vim.VirtualDeviceConfigSpec

                    $spec.DeviceChange[0].Device = New-Object VMware.Vim.VirtualPCIPassthrough

                    $spec.DeviceChange[0].Device.Backing = New-Object VMware.Vim.VirtualPCIPassthroughDeviceBackingInfo

                    $spec.DeviceChange[0].Device.Backing.SystemId =                     $ThisVMDevices[0].ExtensionData.SystemId

                    $spec.DeviceChange[0].Device.Backing.VendorId =                     $ThisVMDevices[0].ExtensionData.PciDevice.VendorId

                    $spec.DeviceChange[0].Device.Backing.Id =                           $ThisVMDevices[0].ExtensionData.PciDevice.Id

                    $spec.DeviceChange[0].Device.Backing.DeviceId = [Convert]::ToString($ThisVMDevices[0].ExtensionData.PciDevice.DeviceId, 16)

                    $spec.DeviceChange[0].Device.Backing.DeviceName =                   $ThisVMDevices[0].ExtensionData.PciDevice.DeviceName

                    $spec.DeviceChange[0].Device.DeviceInfo = New-Object VMware.Vim.Description

                    $spec.DeviceChange[0].Device.DeviceInfo.Summary = 'New PCI device'

                    $spec.DeviceChange[0].Device.DeviceInfo.Label = 'New PCI device'

                    $spec.DeviceChange[0].Device.Key = -101

                    $spec.DeviceChange[0].Operation = 'add'

                    $spec.DeviceChange[1] = New-Object VMware.Vim.VirtualDeviceConfigSpec

                    $spec.DeviceChange[1].Device = New-Object VMware.Vim.VirtualPCIPassthrough

                    $spec.DeviceChange[1].Device.Backing = New-Object VMware.Vim.VirtualPCIPassthroughDeviceBackingInfo

                    $spec.DeviceChange[1].Device.Backing.SystemId =                     $ThisVMDevices[1].ExtensionData.SystemId

                    $spec.DeviceChange[1].Device.Backing.VendorId =                     $ThisVMDevices[1].ExtensionData.PciDevice.VendorId

                    $spec.DeviceChange[1].Device.Backing.Id =                           $ThisVMDevices[1].ExtensionData.PciDevice.Id

                    $spec.DeviceChange[1].Device.Backing.DeviceId = [Convert]::ToString($ThisVMDevices[1].ExtensionData.PciDevice.DeviceId, 16)

                    $spec.DeviceChange[1].Device.Backing.DeviceName =                   $ThisVMDevices[1].ExtensionData.PciDevice.DeviceName

                    $spec.DeviceChange[1].Device.DeviceInfo = New-Object VMware.Vim.Description

                    $spec.DeviceChange[1].Device.DeviceInfo.Summary = 'New PCI device'

                    $spec.DeviceChange[1].Device.DeviceInfo.Label = 'New PCI device'

                    $spec.DeviceChange[1].Device.Key = -102

                    $spec.DeviceChange[1].Operation = 'add'

                    $spec.CpuFeatureMask = New-Object VMware.Vim.VirtualMachineCpuIdInfoSpec[] (0)

                    $_this = Get-View -Id $VM.Id

                    $_this.ReconfigVM_Task($spec)


                    $ThisSkip += 2

                }

                ## END Assign $ThisKitDevices to respective VMs

                 

                END

                 

                (I was copy/pasting directly from VSCode PS1 file to try to get syntax highlighting this time.)