Subjective performance loss after upgrade to WS 6.5

[kzr1k9]

On a Windows 2003 server SP2 host, I run a minimum of 5 W2K3 SP2 guests supporting a System Center: Configuration Manager (SCCM 2007) environment. There is frequent large files (200MB - 2GB) exchanges between the guests which causes lots of I/O for the host.

One of the WS 6.5 performance improvement claim is ; "I/O performance - A new asynchronous I/O manager boosts performance on Windows hosts under heavy I/O loads.". Thinking that this could only be good, I upgraded from WS 6.0.4 to 6.5 and upgraded the VMware Tools on the guests.

Even after I defragmented the hosts VMDK files (some had more than 10,000 fragments) the perceived and subjective performance of the guests is that they are slower under 6.5 than they were under 6.0.4. When the guests are active doing disk I/O, programs within the guests definitely takes longer to start.

I have not run any benchmark to confirm/deny the above observations. Has anyone else observed the same behavior? That "Asynchronous I/O manager performance boosts", is there a setting I can check to verify whether it is active or not?

[kzr1k9]

I de-setected the 2 options you mentionned and I rebooted the guest (W2K3). The drag-and-drop of a file from the host (W2K3) to the guest does not cause VMwareUser.exe to go 100% CPU anymore. As expected since DnD is not enabled, the file is not copied to the guest.

[topgunli]

Then can you still reproduce the issue you reported? We already fixed the VMwareUser issue which will be out in next release.

[Uxorious]

So I tried applying the 2 configuration changes, and they help, but now I have other issues.

After applying the first of the entries, the 2 SMP machine boots nice and fast.

I can't see any difference when applying the 2nd entry.

Now if I boot the 2 SMP Ubuntu box, it boots fast, and runs pretty well.

However, if I run something on it that consumes 100% of those 2 CPU's (like compiling a massive codebase), everything else slows quite dramatically.

Windows task manager shows vmware-vmx taking 45-50% of the CPU (which makes sense since I have 4 physical CPUs) - but things are slow:

Opening a VMWare tab (for another VM) and clicking on "Edit virtual machine settings", it takes 5 seconds before the window opens.

Windows display updates in other Windows program sometimes positively crawl.

Regards,

-Jeppe

[Uxorious]

And now I'm seeing the VM only getting 25% in the Windows task manager (one of my 4 CPUs) even though it is a 2 SMP VM, and I have 2 ssh sessions running compiles that individually take all 25% as well.

[Scissor]

And now I'm seeing the VM only getting 25% in the Windows task manager (one of my 4 CPUs) even though it is a 2 SMP VM, and I have 2 ssh sessions running compiles that individually take all 25% as well.

Are you running anything CPU intensive on your host (SETI@Home, Watching a DVD Movie, encoding video/audio)?

[Uxorious]

>> And now I'm seeing the VM only getting 25% in the Windows task manager (one of my 4 CPUs)

>> even though it is a 2 SMP VM, and I have 2 ssh sessions running compiles that individually take

>> all 25% as well.

Are you running anything CPU intensive on your host (SETI@Home, Watching a DVD Movie, encoding video/audio)

I noticed this while running Folding@Home on all 4 CPUs (as in the report earlier in the thread).

However, all of these 4 processes are running at low/idle priority, so shouldn't VMWare win over them?

[Scissor]

I noticed this while running Folding@Home on all 4 CPUs (as in the report earlier in the thread).

However, all of these 4 processes are running at low/idle priority, so shouldn't VMWare win over them?

When you have a 2 vCPU Guest running, VMware needs to wait until 2 Host CPU cores are idle at the same time before it can service the Guest. If Folding@Home is running on all 4 of your cores (even at idle priority) it might take a while before 2 cores are idle at the same time. I would suggest either running only one or two copies of Folding@Home or changing your Guest to a single vCPU.

I think that compiling software tends to be Disk I/O bound anyway, so moving to a single vCPU on your Guest might be the better solution for you considering that you want to run Folding@Home on your Host.

If you are running several Guests (or several 2 vCPU Guests), then Host processor scheduling gets even more interesting.

[newbie93]

We (at least I) really appreciate answers/explanations like this. This type of information helps us to configure host/guest systems for more efficient operation. I hope that someday VMWare can publish (maybe there already is something?) a whitepaper or series of white papers all available in 1 area of the forum that describe things like knowning that 2 SMP guests need to wait for 2 CPU's to be idle "at the same time". I am sure there is a lot of info like this spread throughout the forums. I myself would find it very interesting reading. Please consider making replies like this sticky..

Thanks again

[Uxorious]

>> I noticed this while running Folding@Home on all 4 CPUs (as in the report earlier in the thread).

>> However, all of these 4 processes are running at low/idle priority, so shouldn't VMWare win over them?

When you have a 2 vCPU Guest running, VMware needs to wait until 2 Host CPU cores are idle

at the same time before it can service the Guest.

Huh?

Did VMWare 6.0 work like that too? (I never noticed performance issues with 6.0, so to me it looks like the behavior changed in a bad way and/or there is a bug in 6.5).

If Folding@Home is running on all 4 of your cores (even at idle priority) it might take a while before 2 cores are idle at the same time.

By definition, if I have a thread at low/idle priority, and you have a thread at normal priority, my thread will be idle immediately. (or as a minimum at the next scheduler interval).

I would suggest either running only one or two copies of Folding@Home or changing your Guest to a single vCPU.

I think that compiling software tends to be Disk I/O bound anyway,

Obviously I can stop running F@H completely, but I have other apps that from time to time will consume 100% CPU (at low priority).

Changing the Guest to single vCPU is not an option.

1) Compiling large projects take advantage of multiu CPUs by compiling several files simultaneously.

2) At least my compiles are not I/O bound (makes sense since source files and object files (without symbols) are rather small. But the CPU has to crunch on a lot of in-memory structures to optimize the code).

[Scissor]

.

[Uxorious]

Weird ... your reply shows up empty here...

[admin]

I de-setected the 2 options you mentionned and I rebooted the guest (W2K3). The drag-and-drop of a file from the host (W2K3) to the guest does not cause VMwareUser.exe to go 100% CPU anymore. As expected since DnD is not enabled, the file is not copied to the guest.

kzr1k9, did turning off DnD fix your performance problem? Are you still experiencing problems with VMwareUser using 100% of the CPU?

Thanks,

Jason

[Scissor]

I see that Workstation 6.5.1 was released today. I am curious if the new version resolves any of the slowness issues reported here?

[kzr1k9]

Sorry Jason for the delay. Turning off DnD fixed the VMwareUser problem with CPU at close to 100%. However since then, I reverted back to WS 6.0.5. The WS 6.5 claim of faster I/O performance is IMO bogus. Furthermore, the newly released WS 6.5.1 does not make any claim about fixing the 6.5 decreased I/O performance. So I will stick with WS 6.0.5 for a while longer.

[rsandri]

I run Vista x64 on a Dell D830 with 4 Gb Ram. I have applied both of these settings and seen a noticable improvement in performance. This seems to be the magical cure i have needed for a while. I had almost given up after tweaking everything possible just to have marginal performance. I routinely run 4 VMs (single vCPU) off of a single external USB drive. The guests are not very CPU intensive and mostly sit idle except when I am demoing my company's products. Anyway, I have shared folders enabled and have manually mapped a drive letter to Shared Folders.

The contents of my Config.ini file are below:

.encoding = "windows-1252"

prefvmx.useRecommendedLockedMemSize = "TRUE"

host.cpukHz = 2200000

host.noTSC = TRUE

ptsc.noTSC = TRUE

mks.noBeep = "TRUE"

prefvmx.minVmMemPct = "100"

mainMem.useNamedFile = "FALSE"

mainMem.partialLazySave = "TRUE"

mainMem.partialLazyRestore = "TRUE"

monitor.usecHostedVSMPMaxSkew = 0

monitor.usecHostedVSMPYieldPoint = 0