Very Long User Login Times at Peak Periods

[six4rm]

Hi there,

We're having real issues with user logins at peak times within our View environment (08:30 to 09:30 in the morning) and it often takes up to 30 minutes for a single user to login. Outside of peak times it's absolutely fine 1-2 minutes tops.

My first instinct was a storage issue, either with the VDI desktops or the Roaming Profile server (also virtualised), but we're not seeing any storage latency within vSphere or our NetApp monitoring. We've got a trial version of vRealize Operations Manager with the Horizon View integration installed and this hasn't highlighted anything of interest.

I'm really struggling to get to the bottom of this. Clearly something is bottlenecked somewhere within our environment, but I haven't yet worked out where.

I've checked for storage latency, the fibre links between chassis to the switches to the storage, vNIC on the Roaming Profile server, host CPU & memory contention, Active Directory authentication doesn't show any issues and standard desktop users are OK.

Has anyone experienced anything similar and what steps did you take to pinpoint the issue?

As a background, we're running the following:

• 12 x ESXi 5.5 hosts (2 x Intel(R) Xeon(R) CPU E5-2670 0 @ 2.60GHz - 128GB RAM) - These hosts are HP blade servers and we're using the HP Gigabit switches in the back of the chassis, so inter-host connectivity is 1Gbps.
• 800 x Windows 7 based Linked Clone desktops (1 x vCPU - 1GB RAM)
• NetApp Storage (1 x FAS2040, 1 x FAS2240, an aggregate on each controller loaded to around 220 desktops per aggregate) using FC connectivity.

If I've missed any key information that someone might need to dig further or offer advice please let me know, I'm desperate to get this resolved and will be more than happy to provide further detail.

We're looking at taking an eval of a NetApp All Flash unit with 10Gbps connectivity and utilising NFS. I'm just worried that we'll implement this and the same bottleneck will exist elsewhere within our environment and we'll still be in the same situation.

I look forward to your thoughts!

###############################

EDIT:

I posted this thread a few weeks back, which touched on the issue. At the time I made the switch over to Persona Management for profile management instead of Roaming Profiles to see if this made a difference, it actually made it worse.

View 5.3 - Persona Management - High Writes on Profile Server

As I said, Chrome in our profiles is making some of them fairly large 100-300MB, but having spoken to other people this isn't particularly huge. My initial thoughts around these large profiles, but I'm now thinking the bottleneck lies elsewhere, possibly network.

[AlexeyKhudyakov]

Hi,

Do you have an antivirus software on Profiles Server?
And if it is so, have you tried to switch it off ?

[RyanGoldstein]

Sorry if you already are doing or have already check any of this, this is just what pops into my head when i hear slow profiles.

What is your file server OS?

If it's 2008 take a look at this, you are going to want to be running SMB 2.

https://www.citrix.com/blogs/2010/10/21/smb-tuning-for-xenapp-and-file-servers-on-windows-server-200...

Do you have the CIFS license on the NETAPP, can you use the NETAPP running CIFS and see if you have the same results, might be too difficult for a test.

You should remove chrome from your roaming profiles.  Chrome has a GPO that will put the chrome profile in a network location, i typically use the Documents folder and capture that with folder redirection.

[FIXED] Google Chrome user settings with roaming profiles and environmental variables | blog.samkend...

You could also take advantage of using a redirected app data folder in combination with roaming profile, this way the only the registry is roamed.

Folder Redirection Overview

In the past i didn't use appdata redirection then i did for 1 client, now i always do.

[six4rm]

First of all, thank you both so much for your replies.

AlexeyKhudyakov‌ - Good shout. We have Sophos running on our profile server, but it has been configured to have on-access scanning disabled. I must admit, I haven't tried totally disabling this on the server side, but I have on the desktop side. For the VDI we're using Sophos again, but with vShield. I first of all added in some exclusions for C:\Users, this made no difference, so as a test I powered off all Sophos Security VMs, again, it made no difference.

RyanGoldstein‌ - Now this is the type of nitty gritty I wanted to be delving in to! All the obvious vSphere stuff seems OK, so something like SMB would make perfect sense. I'm not going to get my hopes up yet, I've had too many "it must be this" moments already! We're running Windows 7 desktops, as previously mentioned, and the profile server is Windows Server 2008 R2. So I want to be setting all the SMB 2.0 registry values in that Citrix article? I've added the following:

TreatHostAsStableStorage (REG_DWORD) = 1

MaxThreadsPerQueue (REG_DWORD) = 64

Smb2 (REG_DWORD) = 1

We do have a CIFS license for our NetApp and I am tempted to shift the user profiles and redirected folders over to that, it would be interesting to do it for a subset of users and see how they get on.

As for Chrome, I redirect all user folders with the exception of AppData. I had some issues with a legacy application that calls Microsoft Word when I originally redirected AppData, so took the decision to leave it within the Roaming Profile. Chrome is damn annoying when it comes to Roaming Profiles. I'm using the GPO to move the profile into the Roaming AppData folder, hence the large profiles. Although the increased profile size is clearly not going to help matters, I don't think it's the only issue we have going on.

[AlexeyKhudyakov]

six4rm‌, please, try to completely disable antivirus on your's Profiles Server for test purpose.

I'm quite sure that it will be helpful.

[six4rm]

AlexeyKhudyakov‌ - Everything I read about disabling Sophos says to disable on-access scanning, which I've done already. So I think that proves it's not an AV issue.

The only other thing I could do is completely uninstall it.

[RyanGoldstein]

Those keys look right.

If you move the chrome data to my documents that will at least remove them from the equation.

[six4rm]

- I'm sorry to report that despite adding those registry values for SMB 2.0 to our profile server I'm still seeing long login times.

I don't know where else to look.

[kclark2907]

Hi

Are you able to some performance logs from the hosts or a batch capture of esxtop for the peak times?

It's not totally clear from your OP (unless I missed it ) but are you running your virtual infrastructure on the same physical hosts as the desktops ?

For VDI, redirected folders works much better than roaming profiles, you might want to have a look at that

Regards

Ken

[h3nkY]

Hi Ken,

DId you open ticket to VMware support when you saw Persona had longer time for user to login?

I have working issue with long login time.

Another possible cause is real-time anti-virus scan in the desktop (not in profile server).

[h3nkY]

Sorry I meant six4rm not Ken

[Gaurav_Baghla]

Hi Six4rm,

Where are the desktops stuck during logon.

Is your policy set that the computers are powered off and starts at user logon ??? I think below is something that might be useful.

Documentation for VMware Horizon 6 version 6.2