Enthusiast
Enthusiast

Nested virtualization on Workstation 14 PRO on AMD Ryzen

Jump to solution

Hello to all, I'm trying to build my own home lab for VMware solution testing, mainly to learn about the technologies and how to get arround them (install, configure, manage kind of stuff).

My setup is as follows:

CPU: AMD Ryzen 1700X, MB: Asrock X370 Taichi UEFI 4.80, 64 GB RAM, Windows 10 Pro with Workstation 14 pro installed atop of it (latest patches on both OS and workstation).

I am able to install a virtual ESXi host (6.5U2) on top of the setup and it runs OK. But when I create a VM on top of this virtual ESX when the VM starts and after it tries to boot, the VM crashes and just powers off. Sometimes it goes through boot OK and starts to load the iso, but then the whole virtual ESX crashes. I tried importing a VM from template to the virtual ESX, but this has the same results.

I tried a couple of UEFI version on the MB, different OS versions and Workstation versions, different OS iso files, but the results are still the same. I have another Intel based PC at home (old i5) and that one works OK and is able to run nested virtualization without any problem (same SW setup).

Is this some limitation of the architecture, or is nested virtualization simply not allowed on Ryzen CPUs? Any way arround it, other than selling this setup and buying a new one?

0 Kudos
1 Solution

Accepted Solutions
Enthusiast
Enthusiast

So to close this one. The answer after many tests done by me:
Nested virtualization on AMD Ryzen (1st gen) on Windows 10 is not supported as of now. Doesn't matter if the virtualization solution is Hyper-V, VMware Workstation, Virtual Box (the 3 I tried). Windows simply doesn't allow nested VMs running on AMD CPUs (something with errors in instruction forwarding).

The above was tested on my home PC and a couple (7 to be precise) of other 1st gen. Ryzens (1700X, 1700, 1600, 1500X) and none of them worked. I've even tested this out on a older AMD Bulldozer platform. The MBs were from Asus, Asrock and Gigabyte. The issues were the same or very similar on every system. Thus my conclusion that this will not work until MS enables this feature.

On the other hand, running an ESXi (6.7 U1) on bare metal does the trick, and nested VMs run without issues. I've successfully tested this with every setup that I had on my hand. So if anyone is looking to build a nested LAB on Ryzen -> go with bare metal ESXi.

After that you can enable GPU pass-through (AMD RX570 in my case) to a Windows VM along with a USB controller pass-through (I used a dedicated PCIe controller) for the mouse and keyboard and if everything works as intended, you can use the VM as a standard workstation and even game on it! Not suitable for professional gamers as there is some input lag, but for the occasional player it’s just fine.

View solution in original post

17 Replies
Enthusiast
Enthusiast

Hi,

Did you give the server enough memory?

0 Kudos
Enthusiast
Enthusiast

Hi, yes - ESXi 24 GB RAM, nested test VM 4 GB RAM.

0 Kudos
Enthusiast
Enthusiast

Guys, anyone else having this issue? I upgraded to Workstation 15 today, and still the same stuff. Anyone?

0 Kudos
Virtuoso
Virtuoso

With the market share of AMD among servers and desktops, add to that not many VMware Workstation users would be trying nested VMs, the chances of finding someone else having the same experience as you do now is pretty slim.

When the Ryzen CPUs came out, there were some posts of PSOD on ESXi installed bare-metal with Ryzen CPUs. Some workarounds were to disable multithreading (SMT at the BIOS); IIRC, the PSOD comes when certain number of threads were exceeded.

Understand that your case is slightly different as you are trying nested virtualisation but you could give disabling multithreading at the host machine a try.

0 Kudos
VMware Employee
VMware Employee

I couldn't reproduce your issue locally. Could you upload your test log via "Collect support data" ?

0 Kudos
Enthusiast
Enthusiast

So I've figured out what the issue is.

I was suspecting the platform (AMD) and solution (Workstation) incompatibility so I did a test with Hyper-V. And as it turns out, the nested virtualization didn't go even with this combination.

So after a MS support call, the official statement from them is - Windows 10 doesn't suppport nested virtualization on AMD Ryzen CPUs. So not a limitation of the CPU platform, rather than Windows not enabling this. I've found a couple of threads for this on the MS forums, and after more than 1.5 years MS is not able to provide this feature.

So sadly, right now this is a no-go option and I will have to re-think my home lab a bit. Maybe do a dual boot Win10 and ESXi.

Thanks to everyone for their responses.

0 Kudos
Leadership
Leadership

Hi,

Sorry, but that's the answer from Microsoft for their Hyper-V product, not for VMware Workstation.

VMware Workstation had nested virtualisation support years before Hyper-V had it.

In your case, I would supply the logs to Susie that she asked for.

She works at VMware and her answer suggests that she is not seeing the problem you have.

IOW have VMware at least look into it before you discard it as "not going to work".

--

Wil

| Author of Vimalin. The virtual machine Backup app for VMware Fusion, VMware Workstation and Player |
| More info at vimalin.com | Twitter @wilva
0 Kudos
Enthusiast
Enthusiast

Hello Susie,

I've attached the vmsupport bundle from workstation.

I kind of given up already, but maybe you will find something relevant Smiley Happy thank you.

0 Kudos
Enthusiast
Enthusiast

Hi.

Yes I know Workstation has support for nested virtualization. On my previous setup on a Intel CPU it worked flawlessly, but I had to upgrade and AMD was the best bang for the buck at the time. I rather think that this is a limitation of the AMD platform + Windows OS combination.

Anyway, the logs are here, so maybe Susie will find something there....fingers crossed Smiley Happy

0 Kudos
Contributor
Contributor

I also have the same issue. Tried both Workstation 14 and 15 and get the same error.

CPU: AMD Ryzen 1700X, MB: Asus Prime X370-Pro, 32 GB RAM, Windows 10 Pro

0 Kudos
Contributor
Contributor

Does VMware workstation work with the ryzen for nested virtualisation?

0 Kudos
Enthusiast
Enthusiast

So to close this one. The answer after many tests done by me:
Nested virtualization on AMD Ryzen (1st gen) on Windows 10 is not supported as of now. Doesn't matter if the virtualization solution is Hyper-V, VMware Workstation, Virtual Box (the 3 I tried). Windows simply doesn't allow nested VMs running on AMD CPUs (something with errors in instruction forwarding).

The above was tested on my home PC and a couple (7 to be precise) of other 1st gen. Ryzens (1700X, 1700, 1600, 1500X) and none of them worked. I've even tested this out on a older AMD Bulldozer platform. The MBs were from Asus, Asrock and Gigabyte. The issues were the same or very similar on every system. Thus my conclusion that this will not work until MS enables this feature.

On the other hand, running an ESXi (6.7 U1) on bare metal does the trick, and nested VMs run without issues. I've successfully tested this with every setup that I had on my hand. So if anyone is looking to build a nested LAB on Ryzen -> go with bare metal ESXi.

After that you can enable GPU pass-through (AMD RX570 in my case) to a Windows VM along with a USB controller pass-through (I used a dedicated PCIe controller) for the mouse and keyboard and if everything works as intended, you can use the VM as a standard workstation and even game on it! Not suitable for professional gamers as there is some input lag, but for the occasional player it’s just fine.

View solution in original post

Contributor
Contributor

do we know if this is limited to 1st gen ryzen? planning on upgrading to 3rd gen when its released so if 2nd gen isn't impacted hopefully 3rd gen will also be fine.

0 Kudos
Enthusiast
Enthusiast

I don't have the opportunity to test on a 2nd gen Ryzen as of now, so cannot tell. Maybe in the upcomming weeks.

0 Kudos
Contributor
Contributor

Just saw this post.  Very nice.  Wish i would have read this 2-weeks ago.  Anyway,  I would like to know what you did to get the bare-metal ESXi to work with your AMD Ryzen 1700.

My setup:

MSi B450-A-PRO motherboard, AMD Ryzen 1700 processor and 32 G RAM.  I have downloaded the ESXi free and tried 6.7.0 U2 ISO image and 6.5.0 U2 ISO image and both give me the same error saying, "no network adapters installed or no driver could be found."  Did you experience this issue when installing ESXi as a bare-metal on the AMD Ryzen configurations?

Thank you,

Brian

0 Kudos
Enthusiast
Enthusiast

Hi, I'm a little late on the response, so I guess you got this figured out already.

The MB you have has a Realtek GLAN chip, that is not nativelly supported on ESXi. You have to either find third party drivers for it (Installing Realtek Driver on ESXi 6.7 - Network Guy ), or you can buy an Intel NIC that should work for you most of the time right out of the box.

0 Kudos
Contributor
Contributor

I had a similar experience with Workstation 16. However, when using the latest version of VMware Player, it worked fine. However, now you have the Player limitations (no Virtual Network Manager, etc.).

0 Kudos