Tesla M60 Passthrough - VM won't power on once its added.

[Ryan425]

I'm trying to passthrough a Tesla M60 GPU on our Dell T640 certified server.  I got it to show up on the hardware list,  activated it as passthrough successfully.  I then go to the VM I want to use it and add it to the list of hardware to use.  Once I've done that, the VM won't power on.  Take off the M60, and it powers up fine.  I looked at the log and saw this line.

PCIPassthru: total number of pages needed (2109440) exceeds limit (917504), failing

Everyone online says I need to enable "Mapping I/O above 4GB" on the ESXi bios but we checked it and its already enabled.  So now I'm stumped on what to try next.  I click the button that says "Reserve all memory" in the VM settings but I don't understand what's happening here.

Log...

[Log dump removed by moderator]

[bluefirestorm]

There is an "Attach" at the lower right hand of the box. It is better to attach log files than pasting its content. Long log file text makes it a pain to navigate through the post. Appreciate if the original post can be edited with the log file attached instead.

The VM looks like is not using MMIO above 4GB.

Refer to this https://kb.vmware.com/s/article/2142307

These lines

firmware="efi"

pciPassthru.use64bitMMIO="TRUE"

pciPassthru.64bitMMIOSizeGB = "64"

needs to be added to the vmx configuration of the VM.

Note that having EFI as the virtual firmware will make the VM unbootable if it was created in virtual BIOS; so a recreation/reinstall of the guest OS VM would be required.

[scott28tt]

Moderator: Please use the Attach function in the bottom-right of the post creator/editor to upload logs, rather than pasting in thousands of lines of text.

[IRIX201110141]

Cant speak for the T640 but for R740 and M60.

On Dell EMC PowerEdge yx4x servers, ensure the below settings in System BIOS  are set:

• Memory Mapped I/O above 4 GB is set to enable
• Memory Mapped I/O Base is set to 512 GB

See https://downloads.dell.com/manuals/common/dell-emc-poweredge-nvidia-vmware-vsphere.pdf

NOTE: I only got it working when selecting the 12TB Setting for Memory Mapped.

Yes, i have the Nvidia Host Driver installed because its a VDI setup but i can switch between shared and dedicated Pass-trough.

My initial fault was picking up the wrong power cable when installing the card. We have preinstalled M60 Hosts (works out of the box) and pre-configured(needs to configure manually) and with only the base 75 watt of a PCIe slot the card wake up and is visible for the Host. But without the additional 225 Watt you will never get the card working.

Regards,
Joerg

[berndweyand]

R740 with P4/T4: got this working with 56TB-Setting (BIOS 2.8.1)

Ryan425​: you have to reserve all VM Memory otherwise the vm will not start

[IRIX201110141]

bewe​

Thx, i confirm that i also have the 56TB option selected in R740.  It was 12TB in the old R730 and i havent updated my docs as i see. Without changing that BIOS default a M60 will not work properly within the Server and ESXi/VM.

About the memory reservations...IIRC you get the warning during configuration or when try to powered on and it says clearly what to do.

Regards,
Joerg