VMware Cloud Community
Ubuntu111
Contributor
Contributor

Citrix server slowness on ESXi hosts - any ideas???

I manage a farm that includes 10 Citrix Presentation Server 4.0 servers that all run very slowly at times. The slowness is usually a few second delay in the keystrokes appearing on the screen and hanging windows. These servers all are running Windows Server 2003 Standard SP2 and are fully patched. They are running on an ESX cluster that includes (2) ESXi hosts and (1) ESX 3.5 host. These ESX hosts are HP DL380 G5 boxes. There are about 300 users in the company. There are also other non-Citrix VMs on these ESX boxes. These other VMs include DCs, print servers, app servers, and file servers. They are also running Windows 2003 Stand SP2. There are about 30 VMs total (including the 10 Citrix desktop servers).

I am not sure why the Citrix servers seem to have sporadic latency issues with just about every app. The apps on these Citrix servers include Word 2003, Outlook 2003, and other lesser known apps. Users complain that keystrokes do not appear for a few seconds when replying to emails, creating Word documents, and typing in other apps. Also there are times where a window will hang before bringing up the next window or menu. When these apps are run locally they run much faster. Users connect to the Citrix desktop servers thru a Web Interface. They are then able to launch their published apps. We have about a 50/50 mix of fat clients and thin clients. Both types of workstations have the latency issues at times.

I have added a 2nd vCPU to each Citrix server and each has 2GB or RAM. When I look at performance monitoring during peak activity all 10 of the Citrix servers they are normally running <50% CPU and RAM usage. So this tells me that the servers "should" have enough CPU and RAM resources. I have moved these Citrix servers to different ESX hosts and they still run slow on the other hosts. I have also updated the ICA clients on the workstations which did not seem to help. VMWare tools are also up to date.

Any ideas on what else I can check? Thanks.

Reply
0 Kudos
25 Replies
mikepodoherty
Expert
Expert

Do you have the premium Citrix client license? If so, you should be able to gety EdgeSight from Citrix which can look at the performance of your hosts and help track down the issues.

Reply
0 Kudos
Ubuntu111
Contributor
Contributor

No Citrix Advanced Edition unfortunately

Reply
0 Kudos
snootalope
Contributor
Contributor

I ran into this with our PS4 servers as well. We only run about 15-20 users per vm, and they would become nearly unsuable for a short period of time. We have three HP ESX boxes with dual quadcore cpu's. I seemed to have fixed our issue by dropping down to one vcpu and giving each vm 4GB RAM (3GB is probably enough, but we've got plenty of RAM in our hosts). However, the biggest difference came when I built the VM's from scractch rather than using an imported image from a physical machine. Don't know if that helps ya at all, just thought I'd share since I went through the same thing..

Reply
0 Kudos
gary1012
Expert
Expert

Have you checked out any of the ESX/WTS/Citrix best practices docs?

Community Supported, Community Rewarded - Please consider marking questions answered and awarding points to the correct post. It helps us all.
Ubuntu111
Contributor
Contributor

I have load balancing set for up to 24 users per VM - I could up the RAM on each to 4GB - all VMs were built from scratch and not P2V'd - the DL380 G5s have dual quad core procs

Can you please provide me more info on the benefits as far as going back to 1 vCPU vs 2 vCPUs? I have read that ESXi 3.5 doesn't have the overhead re: processor scheduling of previous versions of ESX. Thanks.

Reply
0 Kudos
Ubuntu111
Contributor
Contributor

I did go thru that doc again recently. The one thing that I do not currently havethat is mentioned in that doc is the page file on a 2nd .vmdk. However, the Citrix servers currently only have one .vmdk. Should I add a 2nd .vmdk to each Citrix VM so that the page file is on a seperate .vmdk? Thanks.

Reply
0 Kudos
snootalope
Contributor
Contributor

Sorry, I can't tell you the advantages of 1vcpu vs 2vcpu. I can tell you that our citrix servers single vcpu percent is right about 50%. Watching the taskmgr, you'd think that the phone would be ringing off the hook cause it's constantly bouncing up to 80% or more. However, user sessions run just fine.

That is one thing I didn't do, move the page file. Left it right there on the system partition.

Reply
0 Kudos
gary1012
Expert
Expert

If you're not paging, then it won't buy you much creating a second vmdk disk. Are your Citrix clients local or remote? If remote, how is local performance?

Community Supported, Community Rewarded - Please consider marking questions answered and awarding points to the correct post. It helps us all.
Reply
0 Kudos
Ubuntu111
Contributor
Contributor

All Citrix clients are remote. The Citrix servers are located at our Data Center. We have about 10 offices throughout the state that connect to the Citrix servers over various types of circuits. e.g. T1s, fiber, etc. Local traffic is fine at all offices. Thanks.

Reply
0 Kudos
gary1012
Expert
Expert

Sorry, a bit of confusion over local. Try a ICA connection with desktop in the same location as the Citrix servers during peak access times. Are you seeing the keystroke latency from that local PC?

Community Supported, Community Rewarded - Please consider marking questions answered and awarding points to the correct post. It helps us all.
Reply
0 Kudos
J1mbo
Virtuoso
Virtuoso

Citrix is really sensitive to packet loss in my experience. We recently had a similar problem (with physical Citrix servers) which turned out to be a duplex mismatch at a sites router.

Reply
0 Kudos
CitrixSupport
Contributor
Contributor

There are a lot of good suggestions on this thread, checking on duplex mismatches, local data center testing, page file locations etc. If these have not help please send me your contact information and I'll get a case opened for you with Citrix Support so you can work with one of our engineers to help resolve this issue. mike.stringer@citrix.com

Managing Director, Citrix Support
Reply
0 Kudos
Ubuntu111
Contributor
Contributor

Thanks Mike. I will try the posted suggestions and let you guys know what the results are. Thanks.

Reply
0 Kudos
Ubuntu111
Contributor
Contributor

Good idea. I will try that. Thanks.

Reply
0 Kudos
Ubuntu111
Contributor
Contributor

I am just going thru some of the suggestions. Should auto-negotiate be set on the vnics or should I force it to 1000 Full? All vnics are showing as running at 1000 Full but are configured for auto-negotiate.

All of the vnics are connected to two Cisco 2960G switches. Unfortunately I don't have the login credentails for the switches to see if they are set to auto-negotiate. However, when I look at the CDP entry for the switch ports in Virtual Infrastructure Client they all show Full = TRUE.

THANKS.

Reply
0 Kudos
gary1012
Expert
Expert

I think the general recommendation for 1Gb is autonegotiate on both the ESX host and the switch ports.

Community Supported, Community Rewarded - Please consider marking questions answered and awarding points to the correct post. It helps us all.
Reply
0 Kudos
Ubuntu111
Contributor
Contributor

I am doing some additional monitoring and am noticing that the msiexec.exe process is launching on some of the Citrix servers when users logon. This process then takes up about 40-50% of the CPU resources for about 5-10 seconds before going down to 0%. It doesn't happen on all 10 of the Citrix servers but I did see it happen on two of them. What I did was log on the server once and open Task Manager. I then launched a 2nd session connecting to that same desktop server. That is when I saw that process appear. I was able to make it happen multiple times on two of those servers.

I thought that msiexec.exe (Windows Installer) only ran when an app was being installed. Why would it launch when logging on to a Citrix server? Thanks.

Reply
0 Kudos
snootalope
Contributor
Contributor

That's malware isn't it? Think I've seen that same problem before. However, that file can take different names but is always in the same location in the romaing profile. Check the user\application data\ directory of the roaming profile and delete that file.

Ubuntu111
Contributor
Contributor

Here is the location of msiexec.exe on one of the servers. It appears two instances installed. Are these the legitmate Windows Installer app? Thanks.

Reply
0 Kudos