CentOS 6 host becomes unusable when running Windows VMs

[FiZi]

I'm running CentOS 6.8 64-bit (2.6.32-642.13.1.el6.x86_64) on a Xeon E3-1230 V2 with 32GB of RAM and am using VMware Workstation Pro 12.5.2 build-4638234.

I've had this issue since 12.5 I'm pretty certain.

When I run my Windows 10 Guest, 2vCPUs, 4GB RAM, it will either at boot or a few hours/days of running idle start pegging the host CPU to the point where it comes unusable (some how 2vCPUs eat up the processing power of 8 cores).

Depending if I can get into the host or not the instant I shutdown the VM everything returns to normal. I have a smaller Debian Guest (1vCPU, 1GB RAM) which runs in the background with zero issue.

Thinking this issue is Windows 10 specific I just tried a Windows 7 install (2vCPU, 4GB RAM) and it immediately pegged the host and the Windows installer never got past the logo.

I honestly have no idea where to start on this other than Windows guests on a CentOS 6.8 host using Workstation 12.5 appear to have some kind of major issue.

[FiZi]

Attached is my vmware.log from the last time I ran the VM and it became unresponsive with 100% CPU usage.

I haven't been able to pick anything out yet.

[FiZi]

Best guess it appears to be a interrupts issue. I think the network card is generating interrupts that are bogging down the guest and host.

I tried recompiling my hosts NIC drivers via Linux* e1000e Base Driver Overview and Installation for Intel®...

I used their alternate method of compiling the driver since I have a 2.6 kernel on my host: "make CFLAGS_EXTRA=-DDISABLE_PCI_MSI install"

That appears to have made it possible for me to troubleshoot the guest but the VM still locks up hard constantly and spikes my hosts CPU while it does it.

[FiZi]

Believe i figured it out. One of the sticks of RAM in my server has gone bad. The VMs in VMware would push the RAM usage to a point where the bad stick would get used.

[FiZi]

While the bad stick of RAM was a problem it wasn't causing my Windows 10 VM to peg at 100% CPU usage.

I am still stuck on this issue.

Someone explain to me how a VM with only 2vCPUs can eat up all the processing power of a 4 core, 8 thread CPU? It seems like VMware is not respecting it's resource allocations.

The high CPU usage appears to trigger while the Windows 10 VM is idle in the back ground. I typically login once it's booted and then disconnect my RDP session. After it sits in the background for a while is when it decides to start using all of my hosts CPU.

Again 4 core, 8 thread host, 2vCPU Windows 10 VM.

[FiZi]

I am again back to looking at what I assume is a irregular number of interrupts being generated by my Windows 10 VM causing the issue.

The three threads you see in the hostCPU.png are for the same VMware Workstation VM that only has 2vCPUs in it.

[FiZi]

For anyone that finds this thread with a similar issue. It might be related to these:

https://communities.vmware.com/thread/551365

https://www.reddit.com/r/vmware/comments/53m0b9/vmware_workstation_125_freezescrashes/

Unfortunately I can't go back to 12.1.1 for testing because any 12.0/12.1 version fails to install on CentOS 6 due to a kernel change that occurred related to the VMware network driver that VMware didn't fix until 12.5 came out.

[FiZi]

Here are some logs.

vmware-test01.log

• 11:42 things started freezing, had to use vmrun to shut down the VM

I swapped the vNIC from a e1000e to a VMxnet3 and booted the VM again

vmware-test02.log

• 11:58 things start freezing, unfroze 11:59, dropped pings, etc
• 12:01 high network latency on VM, 10-25% interupts in Windows 10 VM
• 12:04 high network latency on VM, RDP timed out, 34% interupts when RDP came back for a second, was able to shutdown VM normally

Disabled 'Memory page trimming' on the VM

vmware-test03.log

• VM was instantly unresponsive on boot, eventually cleared up
• 12:30 RDP froze, all 8 threads on host pegged 100%, dropping pings to guest, stopped vm with vmrun

Added the following to the VMs .vmx file

sched.mem.pshare.enable = "FALSE"
prefvmx.minVmMemPct = "100"
mainMem.useNamedFile = "FALSE"

vmware-test04.log

• High ping/dropped packets to guest, unresponsive on boot, pegged host CPUs, never got into the VM it was just locked from boot, had to shutdown vm via vmrun

[wila]

Hi,

Not an answer, but as you are already in "test and try" mode, have a look at this topic, it might give you a few more ideas.

Tips for Improving Performance On Linux Host

--

Wil

[FiZi · Accepted solution]

Not a solution but a workaround that some people might be able to use.

First I tried the latest VMware Tools for ESX 6.5 from here: https://packages.vmware.com/tools/esx/6.5/windows/x64/index.html

This made the VM functional and not completely kill the host but guest performance was poor.

Then using the ELRepo (ELRepo : HomePage) I upgrade my kernel to 3.10.105-1 and have had two days with zero issues and performance appears to be normal again.

Best guess VMware Workstation is having issues with the 2.6.32 kernel available on CentOS 6. While I was upgrading to a 3.10 kernel CentOS released a new 2.6.32 kernel (2.6.32-642.15.1) which I did not get to test.