Upgraded to 6.7 u3 and can't passthrough on hardware version 11 or higher

[helsyeah]

I'm a bit baffled, I just upgraded one of my home-lab hosts (SM X10SRL-F +  Xeon E5-2630 V4) to ESXI 6.7 U3a from 6.0 U3

Now I can't get any VM that is at hardware version 11 or higher to boot with any PCI device setup in passthrough.

I've validated it across a number of VM's, the OS doesn't seem to matter. I have an older VM that is at hardware version 7 and it can do passthrough without issue, but when I upgrade the version to 11 or higher it no longer boots with a passthrough device.

I get the following error:

The virtual machine cannot be powered on because virtual nested paging is not compatible with PCI passthru. To power on the VM either disable VHV or remove the passthru device(s). If you wish to try this experimentally you can add vhv.allowPassthru=TRUE to your virtual machines config settings.

All VM's have hardware passthrough to the guess disabled. I have tried enabling it and setting vhv.enablepassthru = TRUE without luck.

I'm far from an esxi expert and my google skills are apparently failing me on this one as well. Anyone have any suggestions?

EDIT: I performed a clean install of ESXI 6.7 u3, re-added one of the VMs at hardware version 11 and passthrough worked. There appears to be something busted with the 6.0 configuration that was upgraded. I'll be doing a compare between the clean configuration and the old one to see if anything stands out.

[bluefirestorm]

Did use the right option?

You indicated you used vhv.enablepassthru = TRUE while the error message show

vhv.allowPassthru = TRUE

The virtual machine cannot be powered on because virtual nested paging is not compatible with PCI passthru. To power on the VM either disable VHV or remove the passthru device(s). If you wish to try this experimentally you can add vhv.allowPassthru=TRUE to your virtual machines config settings.

[helsyeah]

Did use the right option?

You indicated you used vhv.enablepassthru = TRUE while the error message show

vhv.allowPassthru = TRUE

I did copy/paste from the message, and technically it went into the VMX file as vhv.allowPassthru = "TRUE"

At this point, per my edit in the first post, there is something up with the upgraded configuration though since passthrough works on a "clean" 6.7 install. I've compared the "clean" 6.7 host configuration to the "upgraded" 6.7 configuration and nothing really stands out yet.

I suspect I'll just have to rebuild the host from scratch unless someone here has a brilliant idea what host configuration would cause all VM's at hardware version 11 and higher to think that vhv is enabled...

[bluefirestorm]

vhv.enable = "TRUE" is a vmx configuration although almost all vmx configuration settings can have a global default for all VMs if it is set at /etc/vmware/config. Other than that I can't think of any reason why the power up of the VM thinks that vhv is enabled.

Hardware version 11 exposes a lot of the Haswell chip features. Nested paging access/dirty bits also started with Haswell although I can't say with 100% certainty that version 11 enabled that feature. That might explain why the version 7 compatibility VM was OK (even if vhv is enabled) as nested paging feature won't be able as version 7 predates Haswell.