Enabling onboard LSI SAS 2008 PCI passthrough crashes guest Win 2012 R2 vm

[TaoeDude]

ESXi version: 5.5u3
Motherboard: SuperMicro X10SRH-CF BIOS v 3.2 (latest)
OS VMs - WIndows 2012R2, with all mandatory updates installed

I'm using this ancient version of ESXi because it's the last one that SuperMicro asserts supports this motherboard.

Windows 2012R2 can successfully access various PCI passthrough controllers in PCI slots, but when I enable the onboard SAS 3008 controller, it won't boot.

But a Debian 10 vm starts fine and can talk to the 3008 without issue. So this is clearly related to Windows.

I'm a ESXi n00b. How do I go about debugging this? Which logs should I examine?

[dariusd]

What sort of crash is it?  Does the guest OS trigger a BSOD and stay running, or does the VM abruptly power off, with or without an error message, or do you get a "firmware exception" or similar error message early during boot (before the Windows logo even appears)?

Is the VM attempting to boot from the passed-through controller, or is it only to be used once the VM has booted?

The VM's folder will contain a file named vmware.log, and that will usually contain useful information for VM crashes and abrupt poweroffs.  (The files get rotated with each power-on, so if you have powered it on since the crash, look for some other vmware-*.log files which matches the time of the crash.)

If you could attach the vmware.log here, I'll take a look through it for you.

Thanks,

--

Darius

[TaoeDude]

Thanks for the reply, Dariusd.

There are no warning messages and no BSOD. When I power it up, the usual Windows 2012r2 pale blue rectangles appear, and the little circle of white lights starts spinning for a few seconds before they slow down and freeze. Nothing happens for about 10 seconds and then the vm powers down and the console goes black.

The vm is attempting to boot from a (virtual) ATA controller disc (the same happens if I use a virtual SCSI disc) which is virtualising the motherboards physical SATA controller.

I've flashed the SAS controller with the latest firmware that ESXi 5.5u3 supports, as well as updated the relevant vib in ESXi:

lsi-msgpt3                     14.00.04.00-1OEM.550.0.0.1391871

Having said that, if ESXi is passing the controller through to the guest I'd have thought the SAS firmware used should be the latest that the VM OS supports?

I attach the log.

[TaoeDude]

Actually, I think I may have found something:

2020-03-15T01:52:43.721Z| vcpu-2| I120: PCIPassthru: 01:00.0 tried to modify MSI-X vectors number 32-32, but maximum supported vector number is 31

https://kb.vmware.com/s/article/2032981

I'll let you know if this sorts out the problem.

[dariusd]

It definitely looks like you're on the right path there.  Any success?

[TaoeDude · Accepted solution]

Turned out to need pciPassthru0.maxMSIXvectors to be set to 31. See:

https://kb.vmware.com/s/article/2032981

Thanks for steering me in the right direction!