Citrix server slowness on ESXi hosts - any ideas???

[Ubuntu111]

I manage a farm that includes 10 Citrix Presentation Server 4.0 servers that all run very slowly at times. The slowness is usually a few second delay in the keystrokes appearing on the screen and hanging windows. These servers all are running Windows Server 2003 Standard SP2 and are fully patched. They are running on an ESX cluster that includes (2) ESXi hosts and (1) ESX 3.5 host. These ESX hosts are HP DL380 G5 boxes. There are about 300 users in the company. There are also other non-Citrix VMs on these ESX boxes. These other VMs include DCs, print servers, app servers, and file servers. They are also running Windows 2003 Stand SP2. There are about 30 VMs total (including the 10 Citrix desktop servers).

I am not sure why the Citrix servers seem to have sporadic latency issues with just about every app. The apps on these Citrix servers include Word 2003, Outlook 2003, and other lesser known apps. Users complain that keystrokes do not appear for a few seconds when replying to emails, creating Word documents, and typing in other apps. Also there are times where a window will hang before bringing up the next window or menu. When these apps are run locally they run much faster. Users connect to the Citrix desktop servers thru a Web Interface. They are then able to launch their published apps. We have about a 50/50 mix of fat clients and thin clients. Both types of workstations have the latency issues at times.

I have added a 2^nd vCPU to each Citrix server and each has 2GB or RAM. When I look at performance monitoring during peak activity all 10 of the Citrix servers they are normally running <50% CPU and RAM usage. So this tells me that the servers "should" have enough CPU and RAM resources. I have moved these Citrix servers to different ESX hosts and they still run slow on the other hosts. I have also updated the ICA clients on the workstations which did not seem to help. VMWare tools are also up to date.

Any ideas on what else I can check? Thanks.

[snootalope]

yeah, those are legite. I was seeing those files in different locations and sometimes different names when I was having cpu spike issue.

[Ubuntu111]

Below shows some VIC realtime CPU and network utilization stats during normal business hours. These are from the Citrix servers. How do these look to you guys? THANKS:)

[snootalope]

Looks to me like they're hardly being used.. our citrix servers have an average CPU usage during normal worklods of 55% - and that's only single vcpu's. what's your memory look like?

[J1mbo]

Personally I think we need to understand as much detail as possible about the networks between the affected clients and the problematic servers. Related to that, have you looked at packet loss rates between them? For example, on a known-good and an known-bad host run

ping -t -l 1536 hostname

for a while (say 15 mins) at a busy time, when you press crtl-break to abort it will give you loss rates.

HTH

[Ubuntu111]

Thanks for the reply. Here are the results from one of the client's workstations to a Citrix server at the datacenter. (about 15 miles away across a 5 mbps OptEMan circuit) This was a 20 minute ping test.

Ping statistics for 192.168.10.33

Packets: Sent = 1281, Received = 1281, Lost = 0 (0% loss),

Approximate round trip times in milli-seconds:

Minimum = 4ms, Maximum = 82ms, Average = 5ms

[Ubuntu111]

Memory is hardly being utilized at all. Normally under 20%. Thanks.