UEM hanging/delaying randomly.

[Sabian0309]

Running horizon 7.6 with UEM 9.5 and appvol 2.14.  Parent is windows 10 1607 LTSB, floating instant clones.  App data is not redirected, documents/favorites/desktop/etc.. are.

Occasionally i am seeing login delays when processing UEM.  On average UEM processes between 5-6 seconds.  However we have some users that it takes 25+ seconds occasionally with UEM not reporting why.

Example from logs:

2018-12-12 07:57:37.331 [INFO ] Config file '\\jfv-vm-fs2\UEM Configuration\general\Applications\Acrobat Reader.INI' added to DirectFlex cache

2018-12-12 07:57:56.269 [INFO ] Config file '\\jfv-vm-fs2\UEM Configuration\general\Applications\Adobe Acrobat.INI' added to DirectFlex cache

2018-12-12 08:39:13.444 [DEBUG] ImportRegistry::Import: Calling '"C:\Windows\REGEDIT.EXE" /S "C:\Users\test\AppData\Local\Temp\FLX6AB5.tmp"' (RPAL: l=0 (D/E), r=0)

2018-12-12 08:39:13.490 [DEBUG] Read 3 entries from profile archive (size: 151258; compressed: 31466; took 64 ms; largest file: 83810 bytes; slowest import took 0 ms)

2018-12-12 08:39:32.428 [DEBUG] Conditions: Condition set 'Microsoft Office 2013.xml' was previously evaluated to true

2018-12-12 14:02:29.110 [DEBUG] Read 25 entries from profile archive (size: 63987712; compressed: 7889959; took 427 ms; largest file: 26738688 bytes; slowest import took 154 ms)

2018-12-12 14:02:48.063 [DEBUG] Conditions: Check for endpoint name = false ('C81MK6V1' is not equal to 'LJ2359H2')

I can reproduce this by logging in over and over.  At times it happens, at times it does not although frequently enough it is easily reproducible.  It does not appear to be related to any specific pool or UEM setting.  It also isn't tied to any specific hosts, users, or type of client (thin or laptop), or version of horizon.

The UEM data is stored on a windows 2008 R2 server file share, which is a VM running off an all flash datastore on a Dell Compellent.  There are no indications the Compellent it is struggling to server the data, or on the file server.  This is as likely to happen during the morning login as it is when people leave for the day.

This appears to of been going on for a long while, this environment is just now getting healthy enough from the previous admin to where smaller issues like this can be tracked down.  But my googlefu is failing to find any possible solutions for this.

Any help would be appreciated.

Thank you,

Billy

[ijdemes]

Same here in both my lab and at the customer. Good to hear more and more positive confirmation.

[DEMdev]

Hi Raymond​,

Very happy to hear that! Thank you for confirming this, and for providing details on your setup.

[DEMdev]

Hey LukaszDziwisz, schepp, robsisk1972, Serg01, hschimpf, Rdiaz29,

Given the positive results with the workaround reported so far in this thread (and through some VMware-internal channels), and the fact that Sabian0309 just marked it as the correct answer , I just wanted to ping you, as you previously posted about seeing similar symptoms.

We're still (slowly...) working with Microsoft to get to the bottom of this, but at least the workaround seems to be doing its job in the meantime.

Many thanks to Sabian0309 and ijdemes for running all those tests!

[JohnTwilley]

I have added a Shutdown script to my environment, and it is working fine. (3,000 Instant Clone VMs)

Our Isilon NAS setup has Opt-Locks (Leasing) disabled at the Share level...which kinda helps. Although I've been told it needs to be completely disabled on the cluster.

I have not gone down to a packet level inspection (yet).

So I'm not sure that I had the exact same delay issues as all the others here.  

I do know however, that the posted workaround is fine, and causes no issues in my environment.

[schepp]

Hi UEMdev​

thanks for the info!

I've deployed the workaround at a customer and I am monitoring the effects right now.

Cheer!

[13erBlech]

Hi,

Unfortunately the shutdown script doesn't resove this problem in our environment.

Horizon view 7.10

Dem 9.11

Appvolumes 4.0.1

Windows 10 linked clones 1909

Any help would be appreciated

Thank you

[DEMdev]

Hi 13erBlech,

Unfortunately the shutdown script doesn't resove this problem in our environment.

Sorry to hear that. Can you provide a screenshot of your shutdown script configuration, and maybe also a DEM log file at log level DEBUG?

[13erBlech]

Hi DEMdev,

The shutdownscript is running as local computer GPO shutdown.

Script:

C:\Windows\System32\net.exe stop /y workstation

echo Power off was correct > C:\temp\log_script.txt

I have to search for a DEM debug log file, because it happens randomly and not all users have the debug level activated, perhaps I can find an old one.

[DEMdev]

Hi 13erBlech,

That looks good, thanks. Just wanted to make sure it was actually configured as a shutdown script; I've seen too many cases where people tried it as a logoff script .

[13erBlech]

Hi DEMdev,

I have to mention that there is a ticket about this issue and I discussed the debug log file with VMware support.

According to the support the 1789 error is not causing the delay and is not a problem because SID resolving is working.

Thanks

[DEMdev]

Hi 13erBlech,

Thanks for the log file – I'll take a look in a bit. Support just pinged me about an "oplock issue", and I asked them whether the customer from their case might be the one who'd just posted to the forum...

Added later: Let's pick this up through your ticket with support.

[sbingner]

It seems to me that DEM should be releasing even *read* locks on all these files as soon as it finishes running during the user logout, which should have alleviated this problem?  Perhaps I'm missing a nuance here...