VMware Workstation Pro on RHEL9.3 slow VM guest RHEL8.9 performance

[orudie]

Hello. 

I am running VMware Workstation Pro 17.5.1 on RHEL9.3 with a single VM RHEL 8.9. The VM has 16 processors, 1 core per socket and 54.6 GB Memory (which is the maximum recommended amount in VM settings). I am having an issue with performance on RHEL 8.9 VM after about a day of running it. The performance is back to normal only after the VM is rebooted. RHEL 8.9 VM is running Pterodactyl gaming server panel which utilizes docker, php, and nginx.  I do not see any indication from the VM OS that it is lacking CPU (load average) or Memory resources. Also no errors or related warnings in /var/log/messages on neither the host or the VM. Are there any hypervisor settings that I need to check? In virtual machine settings under virtualization engine I see the following check boxes and they are all unchecked:

Virtualize Intel VT-x/EPT or AMD-V/RVI
Virtualize CPU Performance counters
Virtualize IOMMU (IO memory management unit)

The workstation hardware is
CPU: Intel Core i9-13900 
64 GB DDR5 memory
2 TB Samsung nvme disk. 

Any recommendation or advices are greatly apricated. Thanks in advance! 

[bluefirestorm]

On Windows 10/11 hosts, there has been a longstanding problem with 12th gen and newer Intel CPUs that only the E-cores are assigned to running VMs by the Intel Thread Director/Windows OS and generally results in slow VMs.

The only possible workaround that will apply for a Linux host will be to set Processor Affinity. Shut down the RHEL VM and add the following to the vmx configuration. This will prevent the VM from using the E-cores of the host.

As the i9-13900 has 8 P-cores with HT, and 16 E-cores, Processors16-31 are the E-cores. If HT is turned off, it would have to be 8-23 instead of 16-31.

Processor16.use = "FALSE"
Processor17.use = "FALSE"
Processor18.use = "FALSE"
Processor19.use = "FALSE"
Processor20.use = "FALSE"
Processor21.use = "FALSE"
Processor22.use = "FALSE"
Processor23.use = "FALSE"
Processor24.use = "FALSE"
Processor25.use = "FALSE"
Processor26.use = "FALSE"
Processor27.use = "FALSE"
Processor28.use = "FALSE"
Processor29.use = "FALSE"
Processor30.use = "FALSE"
Processor31.use = "FALSE"

As for the virtualisation settings in the VM, they can remain unchecked unless you have a specific use-case inside the VM.

Virtualize Intel VT-x/EPT or AMD-V/RVI (to run a VM inside a VM, or run things like Docker/WSL2 inside a Windows VM)
Virtualize CPU Performance counters (to measure performance of software, this require performance measuring software to do so, not even all software developers/testers need this to be checked).
Virtualize IOMMU (IO memory management unit) (for VBS inside Windows 10/11 VMs).

[orudie]

I am aware of the issue with E-cores and VMware workstation on 12+ gen intel CPUs. I disabled E-cores in Bios. However it did not solve the issue with performance. Strangely the issue exist only after about a day of powering on the VM. Restarting the applications does not improve performance. The only way I can get the performance back to normal is by restarting the VM. Then it is good for about a day, and after that the performance is really bad and everything is slow. 

Could it be because I am running the application on thin provisioned disk ? 

[bluefirestorm]

Considering the restart of VM puts it back to normal (instead of restarting the host), it is probably something within the VM.

You can narrow it down further instead of restarting the VM, restart the game server software, or PHP server/nginx.

From the looks of it, it might be the gaming server software if it runs normal for a day and slows down thereafter. I doubt if it is PHP/nginx itself as there will likely be thousands of web applications all over the world that run on PHP/nginx maybe even 24/7 without issues (be it on bare metal or virtualised in cloud).

[orudie]

Restarting game server software does not help. One thing I want to point out that before when I was running Windows 11 on the same computer with the same exact hardware with VMware workstation 17.5.1 pro and CentOS Stream 8 VM and the same game server software docker, php, nginx, Pterodactyl there was no issues with performance like I have now with RHEL9 host and RHEL8 VM.  Another difference on Windows host and Centos 8 VM was that I was using thick provisioned disks. And now with RHEL9 host and RHEL8 VM I am using thin provisioned disks. Therefore earlier today I created a new thick provisioned disk and copied the game server app files to the new disk. Will know tomorrow if it helped. 

[bluefirestorm]

You suddenly switched to using ESXi terminology of thin provisioned vs thick provisioned disks. On Workstation Pro it would be "preallocated" vs "non preallocated" and "split multiple files" vs "single file". Generally for Workstation Pro, it is better to have preallocated split into multiple files for reasons of performance, operational convenience and data recovery in case data corruption occurs. Operationally it is easier to backup/copy multiple slices of 2GB/4GB files than a single large 60GB file. If corruption occurs, higher chance of losing the entire virtual disk if it were single file vs just losing maybe only one slice of the virtual disk. Performance wise there is little difference now with very fast SSDs on host machines.

Preallocated disks tend to have more benefit for VMs that are disk write I/O intensive. But I doubt a game server would be disk write intensive. More likely 80-90% of the time it is serving up content which are likely already cached in memory. If it makes any difference with the switch of disk type, it might just be coincidence/chance.

It is better to understand the usage profile such as during the slowdown how many concurrent users were online, were they saving their games simultaneously, etc, etc. But those kind of discussions are way out of topic in this VMware community forum. Otherwise there is not much that can be tweaked within VM settings to squeeze every ounce of performance from the host and virtual hardware.

[orudie]

I used "thick" and "thin" terminology probably because I use vCenter at work. Anyway, you understood what I meant. Apparently migrating the app from "non preallocated" to "preallocated" disk did not help. I also did not think that "non preallocated" vs "preallocated" disks will make a difference in app performance. This is the reason why I chose to go with non preallocated disk when I created the new VM. Currently there are about 6000 files in the folder with the app. The size of the files vary from a few KB to 3 GB with a total size of 80GB. The topic here is slow performance of the app running in a VM in VMware workstation. It doesn't matter which app. What matters is the result of performance. And I am here to seek for the advice on how to solve the performance issue I ran into which I cannot figure out how to solve by myself. When the issue happens I constantly look at OS logs /var/log/messages, nginx logs, game server app logs, memory / cpu usage, and I do not see any indication that there is something wrong. 

[bluefirestorm]

I think it is better to understand how the game server itself is behaving. There is not really much that can be in the way in terms of VM settings.

[orudie · Accepted solution]

My issue with performance has been resolved after trying different things. 

Even though I'm not using qemu kvm, but installing some virtualization packages on my RHEL9 host mentioned on this Red Hat page greatly improved performance of my RHEL8 VM running on VMware Workstation.  I am guessing the main package which influenced stability in performance is libvirt but I also installed some other packages mentioned on Red Hat Page:

https://access.redhat.com/documentation/enus/red_hat_enterprise_linux/9/html/configuring_and_managin...

After installing the packages mentioned in the article I also ran virt-host-validate and added intel_iommu to kernal parameters:

Install the virtualization hypervisor packages, add intel_iommu to kernel followed by reboot. 

yum install qemu-kvm libvirt virt-install virt-viewer
grubby --update-kernel=ALL --args="intel_iommu=on"
reboot

After reboot my output of virt-host-validate looks like this. I do not care about the last check which says secure boot support. 

[root@bnrhlx:~ ]$ virt-host-validate
  QEMU: Checking for hardware virtualization                                 : PASS
  QEMU: Checking if device /dev/kvm exists                                   : PASS
  QEMU: Checking if device /dev/kvm is accessible                            : PASS
  QEMU: Checking if device /dev/vhost-net exists                             : PASS
  QEMU: Checking if device /dev/net/tun exists                               : PASS
  QEMU: Checking for cgroup 'cpu' controller support                         : PASS
  QEMU: Checking for cgroup 'cpuacct' controller support                     : PASS
  QEMU: Checking for cgroup 'cpuset' controller support                      : PASS
  QEMU: Checking for cgroup 'memory' controller support                      : PASS
  QEMU: Checking for cgroup 'devices' controller support                     : PASS
  QEMU: Checking for cgroup 'blkio' controller support                       : PASS
  QEMU: Checking for device assignment IOMMU support                         : PASS
  QEMU: Checking if IOMMU is enabled by kernel                               : PASS
  QEMU: Checking for secure guest support                                    : WARN (Unknown if this platform has Secure Guest support)
[root@bnrhlx:~ ]$

In addition I lowered the amount of memory for the VM on my 64G RHEL9 host from 54G to 52G. Moreover, I changed the VMs processor settings from 16 CPU and 1 core pre processor to 1 CPU and 8 cores per processor. 

In BIOS disabling hyper threading, turbo boost, and other CPU related advanced settings do not help improving performance, and only the other way around make performance a lot worse. The only thing which helps is disabled in BIOS in Advanced CPU settings are Processor E-Cores. 

All of this took days / weeks of experimenting with settings and trying different things to get the Pterodactyl app to run without degrading performance as I initially described in my post.

I hope someone will find this information interesting and useful  for others with similar setup.  

Regards!