Subjective performance loss after upgrade to WS 6.5

[kzr1k9]

On a Windows 2003 server SP2 host, I run a minimum of 5 W2K3 SP2 guests supporting a System Center: Configuration Manager (SCCM 2007) environment. There is frequent large files (200MB - 2GB) exchanges between the guests which causes lots of I/O for the host.

One of the WS 6.5 performance improvement claim is ; "I/O performance - A new asynchronous I/O manager boosts performance on Windows hosts under heavy I/O loads.". Thinking that this could only be good, I upgraded from WS 6.0.4 to 6.5 and upgraded the VMware Tools on the guests.

Even after I defragmented the hosts VMDK files (some had more than 10,000 fragments) the perceived and subjective performance of the guests is that they are slower under 6.5 than they were under 6.0.4. When the guests are active doing disk I/O, programs within the guests definitely takes longer to start.

I have not run any benchmark to confirm/deny the above observations. Has anyone else observed the same behavior? That "Asynchronous I/O manager performance boosts", is there a setting I can check to verify whether it is active or not?

[tfrost]

I have found a real performance loss, not just subjective. After upgrading from 6.04 to 6.5 in an XP Host, XP Guest environment, my Finalbuilder run with a mix of 150 or so compilations, help builds and installer builds takes over twice as long as before. I would welcome some hints about what could be wrong before I uninstall 6.5 and drop back.

[Uxorious]

Indeed there is nothing subjective about it. I am seeing severe slowdown after upgrading to WS 6.5.

To verify things, I completely reinstalled a PC from scratch.

Dell XPS420 (quad core) with 4 GB RAM.

Vista Ultimate 64.

Workstation 6.5

Installed Ubuntu in VMWare (2 CPU).

Even booting the Ubuntu install disc from an ISO image takes forever.

After installing it boots rather slow, but seems to run ok ... for a while ... then all of a sudden it practically stands still.

Vista process manager shows the process taking very low CPU, and there isn't much IO going on either.

[JohnFLand]

My 2 bits: With Workstation 6.5, I found that I/O performance was horrible compared to WS 6 if I was using a Shared Folder mapped to a Guest drive letter.

For example, using Beyond Compare 3.0 in the Guest (Win XP Pro SP3, 1.2 GB RAM) to copy numerous small files (< 1 MB) from the Guest (running on an independent physical disk connected via eSATA) to the Host (Win XP Pro SP3, nominal 4 GB RAM, internal SATA drive connected as a Shared Folder and mapped to X:), each file was taking 3-5 seconds to copy. Stopping BC3, switching the destination path from the mapped Shared Folder to a network path to the same Shared Folder (same physical location on the Host) and running BC3, everything was back to "normal" copying speeds (numerous files per second). So somehow mapping a Shared Folder and using the mapped path (e.g., "X:\Host C") is significantly slower than just using the direct Shared Folder path (e.g., "
.host\Shared Folders\Host C").

I had a similar problem with Lotus Notes. I keep the Notes data files (about 25 GB) on the same drive as the VM, but outside of the VM. With WS 6, I was connecting fine to those data files as Shared Folders mapped to X:. Switching to WS 6.5, the performance was atrocious until I directed Notes to access its data files via the Shared Folders network path.

[Uxorious]

This is no fancy setup like that.

Like I said - even installing Ubuntu was crazy slow.

I just did some more experimenting, and for me it looks like SMP is the problem.

If I create a new VMWare system with a single CPU, then booting to BIOS is fast as hell.

If I change that same system to be 2 CPUs, then it takes maybe 20 seconds to get into the BIOS!

[Uxorious]

More testing, and it's starting to look like a bug...

Basically the problem happens if VMWare is configured with 2 CPUs, and the OS is running a low/idle priority program that uses 100% of all CPUs ... like distributed computing programs always running in the background.

How to reproduce:

Download the commandline version of Folding@Home

http://www.stanford.edu/group/pandegroup/folding/release/Folding@home-Win32-x86-620.zip

Extract it to as many individual directories as you have CPUs.

in each window, run it.

Default reply to the first 6 questions.

Reply "yes" to advanced options.

Default reply to the next 8 questions.

Machine ID: Enter a unique number for each of the windows.

Default reply to the remaining questions.

Now you will have full CPU utilization on all CPUs in the machine ... but since they are at low priority, you can't feel much impact.

Now in the VMWare client, a 1 CPU machine boots fast, but a 2 CPU machine takes forever.

The 2 CPU one will speed up after it gets going - but then at random times it will grind to a halt again!

[admin]

Are all of you running multi-CPU guests (i.e. 2 VCPUs in the guest VM)?

[admin]

,

thank you for the good bug report. We have duplicated it.

[admin]

Can everybody who has performance problems after upgrade to 6.5 add the following line to VM's config (.vmx) file:

monitor.usecHostedVSMPMaxSkew = 0

If it doesn't help please add the second line:

monitor.usecHostedVSMPYieldPoint = 0

[kzr1k9]

Unlike the other posters on this thread, my guests are all running single CPU and I do not use shared folders. The guests are running SCCM 2007 and they transfer OS images files (up to 3GB) files amongst them. With WS 6.0.5, the guest UI remained somewhat responsive while large I/O transfer was occuring between the guests. With WS 6.5, the guests are now quite "jerky" and discard many mouse actions ... and I get blank screens when I attempt to switch from guests to guests. I recommended my team peers to remain at WS 6.0.5 until WS 6.5 lives up to the I/O performance improvement claim or is on par with WS 6.0.5 in term of I/O performance.

[continuum]

Dmitriy

I tried your vmx-entries but they seem to have no effect

___________________________________

description of vmx-parameters:

VMware-liveCD:

[kzr1k9]

Additional info: I noticed that the VMwareUser.exe process consumes a significant amount of the CPU when I have a process performing lots of I/O. I currently have a running process to create a customized WinPE WIM and ISO files and the Task Manager of the W2K3 guest reports that VMwareuser.exe is consistently above 50% CPU with most of the time in the upper 90%.

By the time I wrote this, the WinPE creation process is now over but VMwareUser.exe is still pegged at >97%. I'm seriously considering moving back to WS 6.0.5.

[admin]

Continuum,

these entries are relevant only for SMP VMs and mostly in CPU-overcommitted cases (running more total VCPUs than the host has physical CPUs or running CPU-intensive apps on the host).

[admin]

Kzr1k9,

how do you transfer OS images files between guests? You don't use shared folders, do you use SMB over virtual networks?

[CLowe]

Here's another good one. Since upgrading to 6.5 my XP guest video has been very wierd. Specifically minimizing and maximizing windows CRAWLS. Opening and closing is ok but min/max is atrocious. I get the same effect opening the right click menu. I'd go back to 6.0.4 but I did not keep a copy of the install and I can't find it anywhere.

Any ideas on the video issue?

[kzr1k9]

System Center: Configuration Manager uses SMB to communicate between the servers and the guests are on Host-Only VMnet. The servers are spread across 4 VMnet (Host-Only).

[continuum]

>these entries are relevant only for SMP VMs and mostly in CPU-overcommitted cases

>(running more total VCPUs than the host has physical CPUs or running CPU-intensive apps on the host).

I only tried with VMs with 2 CPUs

___________________________________

description of vmx-parameters:

VMware-liveCD:

[topgunli]

It would be helpful if you can collect VMwareUser.exe log and upload then we can see if there is anything wrong.

To do that go to:

C:\Documents and Settings\All Users\Application Data\VMware\VMware ToolsThen open the tools.conf file.

Add two entries:

log = "TRUE"

log.file = "C:\vmtools.log"

Wait for a while, there should be one at c:\vmtools.log and possibly more than one

at c:\vmtools.log.<process id> where process id is just a number.

Once you are done, make sure to turn off logging by removing the two entries because they can take up a lot of space over time.

[kzr1k9]

I collected the log files and added them to a ZIP file which I'm uploading in this reply.

[topgunli]

From log file, looks like you were dragging some files around? Can you try this? From VM->Settings->Options->Guest Isolation, uncheck both 'Enable DnD' and 'Enable copy/paste', then log off/log on or restart guest, and to see if you still have the problem.