BTW, the WPR trace is from the VDI (FlexEngine) side.
Thanks, Ivan! Downloading it as we speak – I'll have something to do during the Easter long weekend :-)
Does the delay occur consistently at logon? Would you be able to collect a ProcMon trace of a delayed logon?
No unfortunately it does not. Also, my colleague who does most with this has a week off. We are trying at creating a script to check all log files to see who has the gaps and at what time does this happen. Hoping to get some more info using that but it won't be until next week before we get that going.
And getting a procmon trace of a delayed logon is quite hard. Because we cannot reproduce it every time we should need to get that procmon running at every logon which stalls the logon and we are using linked clones so it ain't that easy to let procmon run at logon.
Well, it's good that your users aren't consistently confronted with slow logons, I guess :-) I fully understand the difficulty of catching random delays "in the act" – hope your log analysis can find some pattern after the fact.
I'm still lurking on this . So i just sent this on the case i am working on with this issue:
I’m open to suggestions, it does appear to be oplock related.
To recap, things that were done to the file server:
Set-SmbServerConfiguration -EnableLeasing $false
Set-SmbServerConfiguration -EnableOplocks $false
Name : OplockBreakWait
Type : REG_DWORD
Value : 10
However, this I see this on the file server when an issue happens:
Then 19 seconds later:
This matches the delay in the logs
2019-04-23 09:37:06.491 [DEBUG] Read 13 entries from profile archive (size: 66625536; compressed: 8536205; took 493 ms; largest file: 31457280 bytes; slowest import took 185 ms)
2019-04-23 09:37:25.428 [INFO ] Importing profile archive 'Internet Explorer.zip' (\\jfv-vm-fs2\UEM Profiles\XXXX\archives\Windows Settings\Internet Explorer.zip)
Now a normal session where this delay is not experienced(no oplock break request):
Does UEM do anything funny with exports where it would lock the configuration files as its processing? As seen in the previous post, the ini file has an oplock where a break is requested. I've captured earlier in the process, and see a fsctl_request_batch_oplock on the file that causes the delay from another session, guessing from an export of another user as shortly afterwards i see a similarly labeled .tmp then .zip being written:
Reading up on the batch uplock, it appears this is requested from the application
I was hoping you were still lurking on this thread, which is why I @'ed you in my previous post :-)
Do you see any change in those 18.9s delays if you tweak that OplockBreakWait registry value?
That OPLOCK BREAK in in the file server's ProcMon trace is interesting, as is the subsequent FSCTL_REQUEST_BATCH_OPLOCK (as UEM sure does not send that...) ijdemes, do you see that in your lab, too?
This last weekend i had an outage window where i tweaked the oplockbreakwait setting (and attempted to disable oplocking via powershell), however it did not alter the behavior. I know with SMB2+ many of the configurations within lanmanserver\parameters are no longer valid, so that may be the case.
I'll still poke around some more and see what i can figure out .
It looks like I missed your "batch oplock request during previous export" post yesterday... Fortunately, you'd also reported that to VMware support, who notified me :-)
UEM does not request any oplocks, but I'll try to reproduce this behavior.
I had the Storage Team disable OpLocks on our Isilon SMB shares for UEM. (Both User Profiles & Configuration)
So far, I really think it has helped! It is quick and easy from the Isilon management console. I'd share the commands...but I don't have them.
You can give them this from EMC Support:
For SMB related inquiries, please provide the following commands output:
isi smb shares view YourShareNameHere | grep –i oplock
You can test with the following Client registry setting to disable Opportunistic Locks from a client PC.
Turn Off Client requests for OpLocks for SMB
This is not what I’d like to use going forward, as it will turn off OpLocks for All SMB shares, not just the Isilon ones.
I see the same happening in my trace on the file server (didn't see it the first time, as I forgot to enable advanced logging ).
<click pictures for the full lines>
I did try some settings with Oplocks on the file server and client (VDI) side. But without any positive result so far.
I changed the settings you mentioned. Both the OplocksDisabled reg key and settings on the Windows file server, like OplocksDisabled and Leases. But both of these settings result in the same trace output. I still see Oplock messages.