DEM with Citrix, unable to start 2nd. published application, PicaVcHost.exe terminates prematurely

[Andreas_M]

We have an environment with Windows 10 VDIs and Citrix Virtual Apps & Desktops 2203 CU2 & 2303 (aka XenDesktop) and VMware DEM 10.5, and we have published application on this Win10 Systems (Note: no terminal servers, the published apps are on the Win10 machines)

In this environment we have the issue that end-users can only launch one published application, if a user tries to launch a second published application for its existing session nothing happens.

We have done a lot of tests and we figured out already some factors.

The issue is related to newer VMware DEM versions. We have this issue with DEM Version 10.5. Previously we had VMware DEM 10.1 without this issue. We tried already the newest VMware DEM 10.9 (aka 2303) version, but the issue still exists.

We also figured out the issue is some kind of a timing problem, users with a very small user profile that loads fasten than 6 seconds don’t have the issue, but users with a larger profile that need more than 6 seconds have this issue.

When the issue happens, we see the Citrix PicaVcHost.exe process terminates prematurely, instead of running continuously in the users session.

We noticed old DEM 10.1 was using the group policy client-side extension mode, newer DEM version is using the new service mode, we could imagine that the issue is related to design change, but this is only a guess.

We have this issue only with published application, but not with published desktops.

I can rule out other factors because I'm able to reproduce the issue in very simple vanilla test environment where nothing else than Win10, Citrix 2303 and DEM 2303 is installed.

Does anyone know the problem and its solution?

[DEMdev]

Hi @Andreas_M,

Thank you for the analysis and experiments you've done so far. And you're right: in DEM 2111 (10.4) we removed the Group Policy client-side extension. Since that version, the DEM agent runs inside FlexService during logon, regardless of whether the agent is configured via GPO or NoAD.xml.
That change does indeed affect the timing a little; the DEM agent now runs slightly later during the overall logon process.

Both DEM and Citrix components apply hooking mechanisms to intercept certain Windows calls; that can definitely be a source of interop issues. DEM uses this for DirectFlex, application blocking, and privilege elevation; does PicaVcHost.exe also terminate prematurely if you don't use those features? (Not that I would consider that a solution for the problem; just as a troubleshooting step.)
You could also configure PicaVcHost.exe as a process that should be excluded from DEM's DLL injection, and see whether that helps.

[Andreas_M]

Hi @DEMdev, thank you for your answer. 

I’m aware of DLL hooking. I have already implemented the DLLInjection.xml  for the new DEM version (and  the  Blacklist.xml for the old version) and I have included the relevant Citirx processes in this list, this is my list:

CdfSvc.exe|CseEngine.exe|CtxCeipSvc.exe|CtxGfx.exe|CtxLocalUserSrv.exe|CtxRdr.exe|CtxSvcHost.exe|GfxMgr.exe|PicaEuemRelay.exe|PicaSessionAgent.exe|PicaShell.exe|PicaSvc2.exe|PicaTwiHost.exe|PicaUserAgent.exe|ssonsvr.exe|VdaRedirector.exe|PicaVCHost.exe

I also disabled Citrix own hooking as explained in CTX107825 to make sure Citrix is not injecting something into the DEM processes, my reg key looks like this:

[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\CtxUvi]
"UviProcessExcludes"="FlexService.ex;FlexEngine.exe"

After I implemented this, I made a ProcMon.exe trace and verified that no 3. party hooks are loaded in the relevant DEM and Citrix processes. I would say we don't have a problem related to hooking.

[DEMdev]

Hi @Andreas_M,

OK, that pretty much rules out hooking being the culprit here. (Not important, but just in case: "FlexService.ex" is missing the final "e" in that UviProcessExcludes value.)

Let's take a different approach: Does the issue also occur if DEM doesn't really do anything during logon, except for taking some time? That is, if you only have a DEM logon task that performs something like timeout.exe 20, does the problem occur in that case too?

[Andreas_M]

Hi @DEMdev 

regarding the missing e in the FlexService.ex reg key, this is intentional, quote for the Citrix documentation:
There is a 14 character limit on the process names - this means to exclude an application named “badlongname.exe” you would add “badlongname.ex”. 

Regarding your approach with the 20 sec. sleep, I have done this test, I disabled all other DEM tasks, I created a simple 20 second sleep DEM task. Even with this simple sleep the issue occurred and PicaVcHost.exe terminates. When I decrease the sleep to 5 seconds the issue is gone and PicaVcHost.exe runs permanently. see attached screenshots. 

[DEMdev]

Hi @Andreas_M,

Thank you for testing that. I'd say that this would need to be looked at from the Citrix side, given that DEM isn't doing anything except for "delaying" the logon. If you (or Citrix) would like to loop us in for that, please open an SR with VMware Support and DM me the SR #.

And/or if you feel like performing yet another test: Does PicaVcHost also terminate if you don't use DEM but configure a standard Windows logon script (running synchronously) with that 20-second delay?

[Andreas_M]

Hi @DEMdev 

my colleague responsible for DEM is on vacation till Monday, I'll talk to him about opening a support case on Monday.
We have already opened a Citrix support case, but no results yet.

Regarding a test with a synchronous 20 second sleep logon script, I tried it, but unfortunately it runs only synchronous when I perform a logon to the console or via Citrix to the published desktop, but when I start a Citirx published application the application starts way before the sleep script is finished, this tells me the script runs asynchronous in this case and therefore it's not a valid repro of the problem.

[DEMdev]

Hi @Andreas_M,

Logon script: Ah, too bad. I assumed that a published app would only start once the (invisible) desktop session had completed its logon, but I (clearly 🙂 don't know anything about Citrix.

Monday will be fine for opening the support case, of course. I'll keep an eye on this thread in case there are any new developments before then.