Terminal Server VM not scaling well

[sbarash]

Environment:

Host: HP DL380, 11 GB memory, Large Raid5 Array, 2 x Xeon 5120 Procs, 2 NICS - 1 for switch1 with Service Console, 1 for switch2 with VM1 & VM2. ESX 3.0.1 build 32039

VM1 - Win 2k3 SQL Server, 1 vcpu, 4 GB memory

VM2 - Win 2k3 Ent. Terminal Server, 1 vcpu, 4 GB memory - User's running MS Office 2k3 applications

Issue, Terminal Services scales well up to about 20 clients. When it gets up to about 30 clients, it gets very slow. With 35 its unusable. Client very unhappy.

What really bothers me is that when the terminal server gets slow, task manager CPU utilization and its Virtual Infastructure client CPU utilization get pegged at 100%, however Virtual Infastructure Client Host CPU utilization averages at about 30% across all four cores. So, it seems like VMWare is not able to scale and proved the VM2 guest with enough resources.

Have read and used the best practices (Performance Tuning Best Practices for ESX Server 3), however while VM2 is set to use 1 vcpu, its HAL is still set to multi. I will change to Uni if anyone thinks there's merit. Have tried 2 vcpu's per guest, setting affinity, and shares. Nothing seems to really make any difference. Currently, everything's at its defaults & one VCPU. Also, host memory, disk and network I/O & utilization look fine.

Other ideas -

\- Create another terminal services guest and setup round-robin dns to spread the load across two guests. But, I don't know if this would help.

\- Replace the CPU's with faster models - but if they are only averaging 30% when one of the guests is starved for CPU resources, then this seems unlikely to help.

Any comments or suggestions would be most welcome.

Thank You,

Stephen Barash

[wcrahen]

30% (probably slightly less than that) across all cores is 100% of a CPU, which is all you get with a single vCPU VM. When you up'ed it to 2 vCPUs you saw no difference at all? Any other apps besides Office? You might just be hitting the limit.

[Rumple]

Running a Multi hal against a uni processor will generally generate about 20% extra utilization as the core is trying to execure instructions against a non multi system so it gets a little silly.

Also, what kind of Disk I/O are you pulling with having that SQL server on the same disk. If the VM starts paging Memory out to disk and your did is running slow you are gonig to see some serious issues I think...

[sbarash]

That totally makes sense, but runs contrary to the best practices stuff that I've read - which is to only use one vcpu unless you use a multi-threaded application.

Why would / does ESX give only one core's worth of horsepower to each vcpu? Watching the performance graphs, clearly the load is being spread across all four cores.

If changing to 2 vcpu's helps, then would changing to 4 vcpu's make this guy scale even better?

Thanks,

Stephen

[sbarash]

Its funny that you mention paging, as while this terminal server only seems to eat about 2-3 gigs of its physical memory, its paging file floats from 2 - 3.5 GB. I guess that's the nature of Terminal Server & MS Office applications.

FWIW, average Disk Read rate is 41 KBps and average Disk Write rate is 370 KBps as reported by the host....

Thanks,

Stephen

[esiebert7625]

I would definitely correct the HAL as this can cause the problems you are experiencing. Once you have this corrected check your cpu usage and if it is still high then try adding a CPU.

Change HAL from multi to uni - http://www.vmware.com/community/thread.jspa?threadID=84611&messageID=646375#646375

This guide is for Citrix but part of it applies to TS also...

http://virtrix.blogspot.com/2007/03/vmware-best-practices-for-deploying.html

Also a good TS optimizing link...

http://www.microsoft.com/technet/prodtechnol/win2kts/maintain/optimize/tsappdev.mspx

[Jae_Ellers]

So these are 32-bit OS vers? 64-bit has been rumored to show some improvement, but I haven't tested it. I'd love to hear from someone who has.

30 users on a single vm is a bit much.

The recommendation for Citrix has been to scale by adding separate 1 vcpu systems.

Most postings I've seen before top out around 15-18, but things have been better since ESX 3. We frequently have around that on a heavily used admin server on PS4. It doesn't usually improve much by throwing vcpu at it.

The vms running as 1 vcpu machines can only impact one CPU each at a time. The cycles are not spread among the cpu except by expensive transfer from 1 cpu to the other.

You should have plenty of room to scale by simply adding another Citrix server.

You should definitely revert to uniproc HAL as well.

[CWedge]

Any particular reason you just wouldn't do VDI?