VMware Cloud Community
Klean
Contributor
Contributor

Using Get-PassthroughDevice on hosts with Nvidia grid K1

I'm trying to automate the assignment of Nvidia Grid K1 GPUs via PCI passthrough for one of my vmware view pools. We are taking advantage of vdga for a 3D engineering class of about 30 seats and manually managing the cards via vcenter web console is a bit tedious. We are running vsphere and ESXI 5.5 and powercli version 5.8 release 1.

Looking at the example in the documentation for adding a passthrough device to a VM via powercli, I see that the command Add-PassthroughDevice is passed the an object that was had been returned from Get-PassthroughDevice. The problem that I'm running into is that the Get-Passthrough device will only return one object, the first GPU core, when there are four GPUs/Passthrough devices that can be assigned.

Would anyone have a suggestion as to how I would go about assigning the other three GPUs on bus 8, 9 and 10 via powercli? Is there something that I am overlooking?

PciPassthrough.JPG

(Get-EsxCli -VMHost "10.0.6.181").hardware.pci.list("0x0300") | select -Property Bus,CurrentOwner,DeviceName

BusCurrentOwnerDeviceName
7VM PassthruNVIDIAGRID K1
8VM PassthruNVIDIAGRID K1
9VM PassthruNVIDIAGRID K1
10VM PassthruNVIDIAGRID K1
16VMkernelG200eR2

Get-PassthroughDevice -VMHost 10.0.6.181 -Type pci | where Name -EQ "NVIDIAGRID K1"

BusClassIdDeviceIdFunctionSlotStateVendorIdVendorNameNameVMIdVMVMHostIdVMHostKeyExtensionDataClient
7768408200Active4318NVIDIA CorporationNVIDIAGRID K1HostSystem-host-1910.0.6.181VMware.Vim.VirtualMachinePciPassthroughInfoVMware.VimAutomation.ViCore.Impl.V1.VimClient
Reply
0 Kudos
4 Replies
tokemorell
Contributor
Contributor

Did you figure out how to assign the other Grid GPU's? I am in the same situation right now.

Reply
0 Kudos
Klean
Contributor
Contributor

I haven't found a solution using powershell. I don't think there is a way around the limitations of powercli. After posting this question, I did come across this post  Pcipassthru which suggests using Orchestrator. I have yet to configure orchestrator and test this solution for myself. I'd be really interested to hear whether this works for you or not.

Good luck

Reply
0 Kudos
medlalibi
Contributor
Contributor

Hi,

There no official solution. I used to have the same issue, and I fixed it using a makeshift PowerShell script. You may need to tweak it a bit for your needs. Please read the comments they are for you

[Console]::ResetColor()

. .\passtest.ps1

   $GPUsIdslist= "0000:0a:00.0", "0000:09:00.0", "0000:08:00.0", "0000:07:00.0"           # list the IDs of the PCI devices in the host starting from the bottom one. i am still working on making it

                                                                                                                                            a non static entry for those IDs to make them auto-generated for each host. I using the command you                                                                                                                                              provided in the original question to do so. Not yet ready sorry.

   $destHostList1 = "10.0.6.181","10.0.6.182","10.0.6.183","10.0.6.184"

                                                                                                                                       # Change depending on the ESXI hosts you have revers between list one and two this script hates using the                                                                                                                                           same hosts for two successive VMs. Don't ask me why. I have no idea.

    $destHostList2 = "10.0.6.184","10.0.6.183","10.0.6.182","10.0.6.181"

    $operationloop = $vm_nbr                     # fill in the number of VMs

    $Org_VMName = "VMtest"                     # fill in the common name between all VMs example: VMtest-01,VMtest-02,... (VMtest)

    $j=0

    $i=1

    while ($i -le $operationloop)

    {

        $destVMName = $Org_VMName + "-" + $i.ToString("00")   # + "-"

        $j=$j+1

        $VMName = $destVMName

        write-host -ForeGroundColor Yellow "`n############ Operation Nbr: $j  On: $VMName      ##########`n"

       # Adding PCi device GPU

        Get-VM $VMName | where { $_.PowerState –eq "PoweredOn" } | Stop-VM –confirm:$false

        Start-Sleep -s 5

        foreach ($vm in (get-vm $VMName)) {get-vmresourceconfiguration $vm | set-vmresourceconfiguration -MemReservationMB $vm.MemoryMB}

        $GPUID = "null"

        get-vm $VMName | get-passthroughdevice | remove-passthroughdevice -Confirm:$false

        $newvm = Get-vm $VMName

        $destHost = $newvm.VMHost.Name

        #get VMs info from the host

        $GpuConf=get-vmhost $destHost | get-vm | get-view

        $IdList = $GpuConf.config.hardware.device | ?{$_.Backing -is "VMware.Vim.VirtualPCIPassthroughDeviceBackingInfo"} | Select-Object  -Property @{N="Id";E={$_.Backing.Id}}

        $SlotLeft = 4 - $IdList.Id.Count

        "-----------------------------------------------------"

        Write-Host -ForeGroundColor Yellow "[DEBUG] The Host '$destHost' has  '$SlotLeft'  GPU slots left."

        "-----------------------------------------------------"

        foreach ($Idline in $GPUsIdslist)

        {

            if ($IdList -ne $null)

            {

            if (!($IdList.Id.Contains($Idline)))

            {

                $GPUID = $Idline

                #continue

            }

            }

            else

            {

               $GPUID = $GPUsIdslist[3]

            }

        }

        if ($GPUID.Equals("null"))

        {

            "-----------------------------------------------------"

            Write-Host -ForeGroundColor Red "`n[ERROR] The host '$destHost' has no free GPU. Please Migrate the VM '$VMName' to a host with available GPUs.`n"

            "-----------------------------------------------------"

            #$operationloop = $i

            continue

        }

        "-----------------------------------------------------"

        Write-Host -ForeGroundColor Green "[INFO] PCI Device with ID '$GPUID' from the host '$destHost' is going to be added to the VM '$destVMName'"

       

        "-----------------------------------------------------"

        add-uniquepcipassthroughdevice $VMName $GPUID $destHost

        Start-Sleep -s 3

        $g=get-view -viewtype VirtualMachine -filter @{"Name"=$VMName}

        $h=$g.config.hardware.device | ?{$_.Backing -like "*Pass*"}

        $h.backing

        $gpuvm = Get-VM $newvm

        $device = Get-PassthroughDevice -VM $gpuvm -Type Pci

        $devname = $device.Name

        $devid = $h.backing.Id

        "-----------------------------------------------------"

        Write-Host -ForeGroundColor Green "[INFO] PCI Device '$devname' with ID: ' $devid ' has been added to the VM '$newvm'"

       

        Write-Host -ForeGroundColor Green "#################################################################################`n"

       

        "-----------------------------------------------------"

        foreach ($vm in (get-vm $newvm)) {get-vmresourceconfiguration $vm | set-vmresourceconfiguration -MemReservationMB $vm.MemoryMB}

        Start-Sleep -s 2

        # Start the VM

        Start-VM $VMName

}

I hope this will help resolve your issue.

Mo

Reply
0 Kudos
amrit1711
Contributor
Contributor

Can some one please help in power cli to add specific vgpu device to a VM.

I would like give input for VM name and vGPU profile and script should be able to add it.@

Reply
0 Kudos