vSphere 7.0- Unable to perform PCI pass through with Nested Virtualization Enabled

[gskjelstad]

I have 2 HCL listed - VMWare servers that I recently upgraded to ESXi 7.0 from 6.5.

I'm attempting to get DDPIO working with some dedicated legacy SSL Accelerator cards (I know, I know, it's legacy support)

With ESXi 6.5, I was able to get around this problem by using the following syntax inside of the .vmx files and importing them.

vhv.enable = TRUE

hypervisor.cpuid.v0 = FALSE

However 7.0 doesn't seem to be respecting these flags and I continue to get the following:

PCI passthrough devices cannot be added when Nested Hardware-Assisted Virtualization is enabled. 

I'm not seeing anything in the release notes that would indicate that these customization would stop working..

I'm not expecting them to support the Guest or any of the passthrough configuration/functionality, but c'mon, I can't even power on the VM.

Before I jump in and open a ticket, does anyone have any direction on doing this on ESXi 7.0?

Thanks,

Garrett

Great Video on PCI Passthrough: PCI pass-through & Folding at Home on the VMware appliance (VMware vSphere ESXi 7) Jason Meers - You...

EDIT: 2020-06-09:

Added the

vhv.allowPassthru = TRUE

flag, and that didn't help either.

[Ardaneh]

Hi

According to the error that you have mentioned, You can not pass through a PCI device when the hardware virtualization feature of CPU is enabled, so you should check your VM configuration to be sure that "Expose hardware-assisted virtualization to the guest OS" is disabled.

I hope this could be helpful

[gskjelstad]

I think I didn't make it clear that I WANT to do that behavior, and that I could do it in previous versions. (yes in 6.7, they removed the feature, and require a flag...)

I'm wanting to enable passthrough as well as nested virtualization the same time in the most recent version as well.

[Ardaneh]

There is another tag in ESXi 6.7 u2, so I believe it can be used in vSphere7

If you didn't use this tag just give it a try:

vhv.allowPassthru = "TRUE"

I hope it could be helpful

Please share any solution if the problem was solved

[gskjelstad]

Adding in this advanced config flag didn't allow the VM to boot either. It continues to get the same error.

Any other ideas?

[christopherjef1]

I have this exact problem, had to rollback to 6.5 as that VM is critical

Here's a screenshot when I attempted to edit the VM config to remove and re-add while on 7 (which also didnt work)

[MarKol4]

Hi,

Did you find any solution?

Knowledge base article, sais "it is not recommended".

But some people used to run VMs with passthrough and nested virtualization succesfully in ESXi 6.7 and before. And now it is not recommended? Surely I can accept that it is not recommended (although it would be nice to know why). But not recommended does not mean forbidden or impossible, right?

Is there a way in latest ESXi 7.x to disable check described in kb article and and run VM with passthrough and nested virtualization together?

Thanks,

Marek.

[Khvastunov]

Hello.

For now we been able to bypass new checks for simltaneous use of nested virtualization and PCI passthrough with a little hack:

• power off VM in question
• remove VM from inventory
• edit .vmx file of this VM, you want to have both parameters there: vhv.allowPassthru = "true" and vhv.enable = "true"
• register VM

But be warned - we resulted in severe performance issues with this setup and now we consider downgrading to 6.5 

[SupportExperts]

hi gskjelstad

Did you find a solution? thanks

[bgwallace]

Team,

Does anyone know how we're supposed to enable VBS (which enables and then grays out the "Expose hardware assisted virtualization to the guest OS" setting) and then pass PCI devices like video cards to a VM which requires this setting to be disabled?  Am I missing something?

I tried adding the suggested vhv.allowPassthru and vhv.enable settings in 7.0 U3 but the VM won't register when using the UI (hangs indefinitely) and vim-cmd registers the VM but leaves it in an "invalid" status.

Did anyone find any solutions?  Any help much appreciated.  Seems like a major oversight on the VMWare Engineering team but maybe I'm missing something on how we're supposed to use both VBS and pass through PCI devices at the same time or if we're not why because it seems like we should be able to.

Thanks!

Brad

[vbondzio]

---

@bgwallace wrote:

(...)
Am I missing something?

---

https://docs.vmware.com/en/VMware-vSphere/7.0/com.vmware.vsphere.security.doc/GUID-8361F64B-DE1D-4FF...

Unsupported VMware Features on VBS

The following features are not supported in a virtual machine when VBS is enabled:

• Fault tolerance
• PCI passthrough
• Hot add of CPU or memory

This isn't an oversight, it is as a serious engineering challenge. VMware (lower case w) can't comment on potential future capabilities or when (if) they ship.

[stonic80]

For myself i was not able to do passthrough with k2200 video card until i put off Expose IOMMU to the guest OS in CPU option

[390843759]

Many thanks for your inspiration on that!

I had the same problem in vSphere 8, fixed doing that:

1) unregister vm esxi from vCenter

2) browse datastore containing vm esxi

3) In the vm .vmx file, add:

vhv.allowPassthru = "TRUE"

vhv.enable = "TRUE"

Good luck!