VMware Communities
gonace
Enthusiast
Enthusiast

Slow initialization of VMs on Ryzen Threadripper 3960X

Hi

I've just moved from an Intel i7-6700K to an AMD Ryzen Threadripper 3960X and have noticed that the initial startup process is extremely slow (5-10min) once it's started everything works fine and a reboot is as quick as before.

The same slow initialization happens when the VM is turned off or suspended, it takes 5-10min (extremely random) for the VM to get to the "Windows 10"-load screen.

Anyone having the same or similar issues or know a reason for this?

I'm running VMWare Workstation 15.5.1 and have attached the log.

2019-11-29T09:29:56.345+01:00| vmx| I125: vmm64-vcpus:   8

2019-11-29T09:36:35.583+01:00| vmx| I125: KHZEstimate 3792875

Best Regards,

Erik

32 Replies
Solus_VM
Enthusiast
Enthusiast

I have exactly the same problem. I moved from an i7 8700k to the 3960X with 128 GB of RAM and I use the latest Windows 10 Version (2004) with the latest version of VMware Workstation (15.5.6).

Booting my virtual machines takes a long time and when I switch between multiple VMs (tabs) at boot time in VMware Workstation, there is a delay there... When I start multiple virtual machines at the same time, then also the host system is almost unusable for about 2 or 3 minutes because it's so slow (at least when I, for example, open the explorer). Once the virtual machines are fully booted, everything is fine. A reboot is then also quick.

When I activate and use Hyper-V on my computer, the Windows VMs start up completely within 2 to 3 seconds... But I would rather use VMware Workstation, because I don't really like a few things about Hyper-V. Is there any solution to this problem? I've had it for over half a year now...

Reply
0 Kudos
gonace
Enthusiast
Enthusiast

Hi, I just wanted you to know that I've moved on, I'm now using Hyper-V as it's good enough alternative for me, and it does not "bug out" on a Threadripper as VMWare Workstations seems to. Guess VMWare does not think their Threadripper costumers are important since I've also sent in a bug report 3 months ago with no reply what so ever.

It's also kind of "funny" how a free open-source project like VirtualBox seems to work better on our platform than this (expensive) solution.

dariusd
VMware Employee
VMware Employee

Sorry we were unable to meet your needs.

Even though I may be too late to really assist, I have a "long shot" question: Do you happen to have the game Valorant by Riot Games installed on your host, and/or its Vanguard anti-cheat system?

Based upon your log, there is not much within our control that can go wrong at that point in the code, but we have seen other customers whose installations of Workstation have been adversely affected by the Vanguard anti-cheat system in a way quite similar to that shown in your log, and it is entirely possible that other 3rd-party anti-cheat or anti-malware solutions could also interfere in a similar way.  If Valorant isn't involved, I would check for other anti-cheat and/or anti-malware systems, particularly if the anti-malware system has an option for virtualization-based protection.

Thanks,

--

Darius

Reply
0 Kudos
gonace
Enthusiast
Enthusiast

Hi, no reply is too late in my mind. I've tried on a clean install of Windows 10 with no other applications (other than what is installed with Windows 10) installed. The problem is there even running a clean/fresh install of Windows 10.

But when installing Hyper-V and VirtualBox on the same install those Hypervisors run without problems.

Solus_VM
Enthusiast
Enthusiast

I can also confirm this. I've never installed any game on my computer as it is a workstation system. I've also tested it with a fresh Windows 10 installation and the problem was and is still present.

I upgraded to the Threadripper with 128 GB of RAM (256 GB was originally planned) because I wanted to run many virtual machines in parallel. That such a problem occurs is really annoying... As I said (and gonace too): With Hyper-V and other solutions there are no problems. But for various reasons I have to use VMware Workstation and would like to continue doing so.

I can't imagine that there aren't significantly more people who have the problem. Probably everyone with Threadripper 3960X, but I guess most of them don't contact VMware or post something in the forum. Maybe most of them may not even have any comparison between VMware Workstation and other software or other CPUs, I don't know... Anyway, I was very disappointed when I switched from an Intel Core i7 8700k to the Threadripper 3960X because I didn't have these problems with the significantly cheaper and less powerful i7. I hope that there is a fix for this problem soon.

Can you possibly run some own tests with this processor? After all, it's a CPU that I think is used a lot and this would be very helpful.

goverma
Contributor
Contributor

I am also experiencing Lag with a VM running Windows Server 2016 on a Ryzen 2700X on Windows 10 build 2004 with 64GB of ram.  The issue appears to be related with the number of CPU cores allocated to a particular VM.  When the VM is allocated 6+ cores the initialization slows down and becomes unusable after a point.  You then have to wait for the VM to unfreeze and then issue commands. 

Solus_VM
Enthusiast
Enthusiast

I can confirm that it is definitely related to the assigned CPU cores. If I start 4 VMs with only 2 cores each, they boot much faster (20 to 30 seconds) than if each VM has 4 cores (at least one minute). With 8 cores per VM it even takes several minutes until all VMs are booted and the host system is slow during this time. But you don't buy a CPU with 24 cores and 48 threads for a decent amount of money just to assign only 4 to 8 cores to the virtual machines in total...

Regardless of this, Hyper-V is much faster. Even if I start 5 or 10 virtual machines in parallel on Hyper-V, they all boot up in less than 5 seconds. I'm aware that Hyper-V technically is a type 1 hypervisor, but the difference should not be that extreme. Exactly the same VM (converted from VMware Workstation to Hyper-V) takes 20 seconds to boot with 8 CPU cores and 16 GB RAM on VMware Workstation, compared to only 2.5 to 3 seconds with Hyper-V. This makes Hyper-V 6 to 8 times faster than VMware Workstation. However, among other things, since I have many Linux VMs, I do not want to switch to Hyper-V.

When I start 4 VMs with 8 CPU cores per VM in parallel (Windows Server 2019 Guest OS on all of them), even opening the Windows Explorer on the host system takes 1-2 seconds. It takes another second until the disks and my folders are shown. I only use Samsung NVMe disks (Pro series) and this slow performance is only due to the VMs that are booting in the background. Switching between the screen/console of these VMs also takes almost a second while they're booting. Once they've booted up, that's no longer the case. Meanwhile, the host CPU load is only at about 10 to 15 %.

This problem must be with either VMware or the AMD virtualization technology. Currently, I rather assume that the problem lies with VMware Workstation.

goverma
Contributor
Contributor

Solus_VM​ Agreed, I don't have a processor that has that kind of hardware but I can say that the CPU usage makes no sense.  The usage is flat even though I'm performing a multi-threaded task in the VM.  When I tried the same tasks in Virtual Box the CPU usage looked much more irregular which is appropriate for the tasks being carried out on the VM.  Also when the tasks are being executed there is a delay of about a minute or two where the VM hangs and then resumes processing.  This is infuriating and means that it takes an inordinate amount of time to use Vmware.

My impression is that each virtualization program has to link up with the underlying AMD hardware.  At the end of the day those API calls to the AMD virtualization are likely similar because AMD provides the drivers.  If Hyper-V and Virtual Box work properly then the issue is with Vmware.  I plan to do more testing and will post more with these issues.

DominikWeglarz
Enthusiast
Enthusiast

Hi goverma​ !

I am planning to buy AMD Ryzen (3960x for desktop and AMD Ryzen 7 PRO 4750U for laptop) for nested virtualization (esxi hosts inside VMware Workstation) in near future but reading thread like this one makes me full of doubts.

Please give me any feedback if you will see any progress like helpful updates from VMware.

I will relay on your suggestion 😉

Thanks in advance

Reply
0 Kudos
Solus_VM
Enthusiast
Enthusiast

I've installed and tested VirtualBox on my computer today. Unfortunately, I'm experiencing the same problem here as with VMware Workstation. If I assign only 2 CPU cores to a virtual machine, it will boot up in less than 10 seconds. However, if I assign 8 CPU cores, it already takes about 20 to 30 seconds. I then assigned 20 CPUs to this virtual machine and even after 5 minutes of waiting, Windows was still not fully booted inside the virtual machine. So I now think that the problem isn't necessarily unique to VMware. It could also be AMD or Windows.

As I said: I have no problems with using Hyper-V, but this is a type 1 hypervisor. Even if I create a virtual machine in VMware Workstation on my physical computer, enable the AMD-V virtualization here, wait for that VM to fully boot (Windows 10) and then install VMware Workstation or Virtual Box there and create another VM inside that VM (nested virtualization), it will boot very quickly. This issue occurs only when I create virtual machines with a type 2 hypervisor like VMware workstation on my physical computer running Windows as host OS.

Interestingly, I have no problems with virtual machines running Linux (in my case Debian 10 without GUI) as guest OS. Even if I assign 16 CPU cores to these VMs (more isn't possible in VMware Workstation), they boot completely within a maximum of 5 seconds. In summary, this problem seems to only occur with Windows VMs and only when using a type 2 hypervisor virtualization software like VMware Workstation or Virtual Box.

In the next few days I'll install and test ESXi on a 2nd SSD of my computer. I assume that the Windows VMs will also boot up quickly there. The thought behind switching from the i7 8700k to the Threadripper 3960X was that I'd have better performance in terms of virtual machines. So I hope that I can somehow solve this problem, even though I'm running out of ideas. I'm also wondering why there is almost nothing to be found online about this issue. I'm glad that I at least found this forum post here.

gonace
Enthusiast
Enthusiast

Hi

I've been able to do the same tests as you, and it appears to behave in the same way. I've also tried ESXi without problems, so please let us know when you've tested ESXi Smiley Happy

hahakiki2010
VMware Employee
VMware Employee

@gonace,

Can you try Workstation 20H2 preview version and see if the issue still exists?

Thanks

Best Regards,

Yan

Reply
0 Kudos
gonace
Enthusiast
Enthusiast

Well, sadly that went even worse, first I inserted an NVMe drive with a fresh install of Windows 10 so nothing else is installed, I then installed "VMware Workstation Tech Preview 20H2" and moved a virtual machine (Windows 10) and another virtual machine (Windows Server 2019) but VMWare crashes as soon as I select one of these Virtual Machines.

xxxxxx.JPG

I then tried installing a new virtual machine that does not crash but it still takes as long as initializing the virtual machine even then so "wasted" around 2h.

Regards
Erik

Reply
0 Kudos
Solus_VM
Enthusiast
Enthusiast

I've already tested it with VMware Workstation 20H2 preview and unfortunately the same problem exists there. But since Virtual Box has the same issue, at least in my case (see my answer of July 30th in this thread), I now think the problem is more with Windows or AMD.

I installed ESXi 7 directly on my computer yesterday (bare metal). I didn't experience any performance issues there. The virtual machines (Windows Server 2019) fully booted up in less than 4 to 5 seconds. No matter if I assign only 4 or 8 CPUs and 16 GB of RAM to them or even up to 48 CPUs and 100 GB of RAM (single VM). I haven't noticed any differences regarding the allocated CPUs or the amount of RAM - everything runs very fast. I also started 4 VMs in parallel and each of them had 12 CPUs and 24 GB of RAM allocated. Again, after just 5 seconds all 4 VMs were completely booted and I could log on to Windows. The host CPU utilization was between 1 and 7 % while all 4 VMs were running. I recorded my screen during these tests and uploaded the videos (https://www.youtube.com/watch?v=d_unhtPf4E8 and https://www.youtube.com/watch?v=cVKMJFfxAdA). The text is in German, but I assume that you still know what's on the screen, because the structure of the ESXi UI is identical. I had also planned to record the issue that I'm experiencing in VMware Workstation, but so far I haven't been able to do so due to lack of time.

Because this issue didn't occur with ESXi and ESXi is a type 1 hypervisor, I assume that it's either Windows related or that there are general problems with type 2 hypervisors in combination with the AMD Threadripper CPU. Maybe drivers or something like this, I don't know... This would at least explain why the issue doesn't occur with Hyper-V, since Hyper-V is a type 1 hypervisor (the Windows OS of the host system runs in a VM, so to say).

To investigate this further, I'll install VMware Workstation under Ubuntu on my computer today. If the performance issues don't appear there, it's most likely due to Windows, I guess.

Solus_VM
Enthusiast
Enthusiast

I installed Ubuntu on my computer yesterday and used VMware Workstation there for further testing. The results were identical to my ESXi test results. My virtual machines booted very fast - no matter how many CPUs and how much RAM I assign to them. Again, I created 4 virtual machines with 12 CPUs and 24 GB RAM each (Windows Server 2019) and started them all simultaneously. All 4 virtual machines were booted after 6-7 seconds. When switching between the individual VM consoles (tabs) there were also no delays during the boot process. I've also recorded these tests. You can find the video at https://www.youtube.com/watch?v=eO1SeL8reZo.

After that I started Windows again and created 4 new virtual machines with Windows Server 2019 in VMware Workstation. Even when I started a single virtual machine with only 8 CPUs and 8 GB RAM, it took much longer to boot. In this case, it was 28 seconds and the virtual machine was freshly installed. The CPU utilization on the host system was 2-3 %.

After that, I booted the 4 virtual machines in parallel here as well, but initially with 8 GB RAM and only 2 CPUs per virtual machine. After 12 seconds, all 4 virtual machines were booted. After I increased the number of CPUs per VM from 2 to 8, the boot process took much longer. In this case, it took 1 minute and 42 seconds to start up the 4 virtual machines. During this time, the host CPU was used to a maximum of 14%, so this is not the reason. If I had assigned 12 CPUs per VM, the boot process would have taken even longer.

When switching between the VM consoles (tabs), a significant delay was noticeable during the boot process of the virtual machines. I also opened the Windows Explorer on the host system and it was very slow. The same applies to the Task Manager. Once the virtual machines were up and running, everything reacted very quickly again on the host system. I restarted all 4 virtual machines at the same time and it took only 10 seconds to boot them all. During this time Windows on the host system reacted quickly. The problem really only occurs when starting/initializing a virtual machine. You can find a video of my Windows tests at https://www.youtube.com/watch?v=1hySuuunsbo.

Since VMware Workstation on Ubuntu is of course a type 2 hypervisor, I'm quite sure that Windows is the problem. Maybe it's AMD (e.g. drivers or something like that), but I guess it's just Windows in combination with the AMD virtualization technology... Therefore I'll write a post in the Microsoft forum soon and hope for help there.

gonace
Enthusiast
Enthusiast

That sounds great, I was not able to watch your videos, youtube says "Video unavailable. This video is private.". can you please provide a link to that forum thread when you've created it?

Kind regards

Solus_VM
Enthusiast
Enthusiast

I actually uploaded the videos as "not listed", so they should be accessible via the links I posted. But now I've set this visibility setting again. Now the links should work, thanks for the hint Smiley Happy

DominikWeglarz
Enthusiast
Enthusiast

Solus_VM  do you have ability to check if VMs  (like Win7, Win8, Win10, Win2016 some Linux) that were created on Intel platform will run without any issue on Threadripper 3960X ?

Thanks in advance.

Reply
0 Kudos
gonace
Enthusiast
Enthusiast

I've tried running virtual machines that have been created on another workstation (running Intel, in this case, Xeon) and these are slow to initialize.

As far as I've seen it does not matter on what system the virtual machines have been created.