AMD EPYC nested virtualization Hyper-V

[kp3k]

I have an issue with nested virtualization on ESX server with AMD EPYC CPU.

On ESX is running VM with Windows server 2019 with Hyper-V and VM created in Hyper-V gen.2 is not working. Only black screen is showing.

The same configuration on Intel CPU is working fine.

VM gen.1 created on Hyper-V with AMD EPYC CPU is working too, but this VM gen.1 has Legacy boot, while VM gen.2 has UEFI

I've tried also Windows server 2022 but with same result.

Do you have some hint what can I try?

[fk1h8theseStupi]

I have the exact same issue, Gen2 VMs don't work on the AMD EPYC CPU; I believe its been a long-standing issue, would be good to have a definitive response from somebody as to why and how we can overcome this issue.

[nballfr]

Did you find a solution to this issue?

[bunsenbunner]

I spent the better part of a day trying to get this working and gave up (shortly after finding this thread and realizing it wasn't just me).  Nested Server 2022 with Hyper-V role on ESXi 8.0b with EPYC Rome 7302 CPU.  Same issue - Gen 1 VMs power on fine, but Gen 2 go to black screen with the equivalent of 1 CPU core pegged (25% VM utilization on a quad core Server 2022 VM according to Hyper-V manager).  The odd thing is that the Server 2022 host VM doesn't show any CPU activity close to 25%.

I found some references to SEV-ES being a possible culprit, but in vCenter 8 did not see any option to disable that encryption feature.

[nballfr]

Maybe you should try disabling SEV-ES in the BIOS/EFI.
Please let us know if it did make the trick.
Thanks

[dp213]

RE the SEV-ES option:

a) doesn't exist

b) disabling security same issue

[lamw]

JFYI - I've been able to reproduce the issue described here and I've reported this to Engineering. They are currently investigating the issue to understand what might be going on given G1 VM works fine but not G2. If we have more information to share, I'll be sure to reply back to this thread

[nballfr]

Thanks William for escalating this issue to Engineering as it is a major problem for us.
We are using lot of nested hypersvisors for testing purposes and now wish to use ESXi 8.0 on AMD.

Please check this thread:
https://communities.vmware.com/t5/VMware-Workstation-Player/Incompatible-AMD-V-on-Workstation-17-0-b...

Thank you.

[stanj]

Same issue I am seeing on vSphere ESXi 7.0.3 21424296 Enterprise Plus

GEN1 Nested Hyper-V VMs boot and load

GEN2 goes to a black screen.

We are supporting a customer that needs GEN2 to work. 

If not, many passible delays ahead. 

[lamw]

This will be fixed in a future update of ESXi 7.x/8.x. For now, only Gen1 will be functional when running an inner-VM inside of Hyper-V (running as VM)

JFYI - Nested Virtualization isn't officially supported by VMware 🙂

[stanj]

Thanks for the reply William.

Any idea what "future" may represent  - 1 month, 1 yr, 2 yrs?

It would be nice to have some idea in order to work with our customer on alternatives.

And yes, all of us know nested is a bad word at VMware, but with many labs now using it extensively for research (not necessarily production).

It would seem that it should be looked into more closely from an engineering and support stance?

thanks

[lamw]

We don't comment on dates/versions. As I said, this is fixed and will be available in a future update. 

[stanj]

Thanks ..

We will monitor the next rerelease.

i assume this would be noted in the release notes ?

[lamw]

I'm happy to share that latest vSphere 8.0 Update 1c resolves this issue! Even if you're not running a Hyper-V VM, if you are using Windows VBS, you could still see some performance degradations. For more details, please see https://williamlam.com/2023/07/vsphere-8-0-update-1c-resolves-windows-vbs-performance-hyper-v-genera... 

[nballfr]

Using nested vSphere 8.0 Update 1c (ESXi build-22088125 and vCenter) on AMD EPYC 7763 processor:

Failed to power on virtual machine. This host supports AMD-V, but the AMD-V implementation is incompatible with VMware ESX.

It immediately fails when starting any VM, whether it is Windows or Linux.
Not sure if it is related to this issue that Update 1c was supposed to fix.

This is driving me nuts: any ideas or workaround please?
Thank you.

[lamw]

Hi @nballfr - I assume you're running ESXi as a VM and you're attempting to power on a Windows or Linux VM? If so, can you provide the vmware.log for the outer (ESXi in VM) as well as the vmware.log for the Windows or Linux VM when you attempt to power those on? 

[stanj]

It would also be good to know if this is a nested Hyper-V VM or ESXi VM.

We see the issue with GEN2 Hyper-V VMs, nut on ESXi 7.

[nballfr]

That's right, running nested ESXi 8.0U2 attempting to power on a Windows or Linux VM on AMD processor.
Tried with EPYC 7763 and Ryzen 9 5950X, same issue on both processors!
Works just fine with nested ESXi 7.x on the same hardware/config

Please find the logs included as well as some screenshots.
Any help would be greatly appreciated.
Thank you.

VM Logs (vmware.log):
vmware.log 

ESXi Logs (hostd.log):
hostd.log

VM Config (w11-test.vmx):
w11-test.vmx

[stanj]

We are running ESXi 7.0 U2 on AMD EPCY 7452 processors

I have a VM running Windows Server 2019 with Hyper-V installed and have a GEN1 WIN10 VM and a GEN2 WIN10 VM.

I installed the WIN10 OS on the GEN1 nested VM without and issue and it boots up and runs

I tried installing the WIN10 OS on the GEN2 nested VM and it just boots to black screen..

[lamw]

This issue is fixed in both vSphere 8.0U1c and vSphere 7.0U3o - https://williamlam.com/2023/07/vsphere-8-0-update-1c-resolves-windows-vbs-performance-hyper-v-genera...