App Volumes - App Stacks appearing after logon and connection errors when SSL Certificate validation is enabled

[DanVM99]

We are using Horizon View 7.2 and are currently testing with pools of linked clones. We are using App Volumes 2.12.1 (though the majority of stacks are based on the 2.12.0 template) to deploy our applications.

Initial testing of App Volumes (with SSL certificate checking enabled) produced a connection error each time we logged on 'Cannot contact App Volumes Manager - Virtualization is disabled'. We did find however that restarting the svservice and logging back on to that particular VM would allow Apps to attach... albeit with the behaviour described below.

When logging into our Win7 32bit pool with 10 App Stacks assigned we only see around 7 or 8 apps successfully attached upon initial logon. The remaining apps appear around 8 seconds later. We are eventually going to introduce UEM into the environment and the concern here is whether settings for the missing Apps would be applied or not. In an effort to try and resolve this issue we created the VolWaitTime regkey with a values of 5 and 10 seconds in successive tests. The result of which was only around half of the 10 apps were attached at logon and we had to wait a further 3-4 minutes before the missing stacks would successfully attach.

So, first question. Disabling SSL certificate validation prevents us from getting connection errors, but given we have successfully replaced the certificate on the App Vols Manager with an internal CA assigned cert that is trusted by our pool of desktops that have the Trusted CA cert in the Gold Image, why are we seeing connection errors with cert validation enabled? And why does restarting the svservice after the pool has finished composing get rid of these errors? Is the svservice starting prematurely for some reason?

Second question. Why are we only getting half our apps attached before the point of logon and subsequently seeing the remainder of the Apps burst in at a later point (either 8 seconds or 3 minutes with VolWaitTime key in place)?

Regularly trawling through both Manager and Agent logs doesn't reveal much as to why any of this is happening. We did however notice a 7 second pause after attempting to contact the AppVolumes Manager during the certificate validation phase regardless of whether validation is enabled or not.

Any insight into these issues would be greatly appreciated as we're really struggling to make a case for a successful implementation of AppVolumes for our VDI project.

[Ray_handels]

Will try to explain things one by one. If I'm missing something just ask the question .

We did however notice a 7 second pause after attempting to contact the AppVolumes Manager during the certificate validation phase regardless of whether validation is enabled or not.

We see the exact same thing. What it does in these 7 seconds is send a message to the Appvolumes manager stating which user is logging in, then sending a message to the VCenter to recompose and vcenter sends a message to appvolumes stating that it's good to go. Then and only then will the logon process start. First thing will be the writable, then all other appstacks.

We found out that the huge amount of groups users were member off (sometimes up to 200 groups) causes this sluggishness during recompose. It seems to check all groups a user is member of in AD which could take some time. Also, if you have multiple datastores and you have  "First mount volumes from a VMs local datastore when available." it checks all datastores to see if the appstacks are there what could also take up a few secs. We are at 8 seconds. Fastes we've seen is about 1,5 to 2 seconds with user having almost no memberships.

When logging into our Win7 32bit pool with 10 App Stacks assigned we only see around 7 or 8 apps successfully attached upon initial logon. The remaining apps appear around 8 seconds later. We are eventually going to introduce UEM into the environment and the concern here is whether settings for the missing Apps would be applied or not. In an effort to try and resolve this issue we created the VolWaitTime regkey with a values of 5 and 10 seconds in successive tests. The result of which was only around half of the 10 apps were attached at logon and we had to wait a further 3-4 minutes before the missing stacks would successfully attach.

There are a few things to take into account here. You have 2 registry keys, 1 is VolWaitTimeOut, the other one is VolDelayLoadTime. The first one is a key that you can set telling appvolumes not to wait indefinitely (which in Appvolumes case is 180 seconds) for all appstacks to be merged but just go ahead with logging in much quicker. We set this to 10. Downside is that merging is still taking place although the user already has his desktop available. For us this is no issue as we would like to have the desktop available ASAP.

The second one (VolDelayLoadTime) will wait for  a defined value to attach appstacks effectively speeding up the logon proces but will leave you with merging of appstacks after logon. If you have a writable this setting is useless as it will always wait for the writable to be attached and processed and then frees up the Windows logon process. Otherwise you'd end up with a corrupted profile in your writable.

Last but not least. I don't believe disabling or enabling certificate check will speed up or slow down logon behavior. To be honest I don't really see an extremely important purpose for it. As long as you have a certificate in place and use HTTPS it is encrypted anyway. And appvolumes is backend server, no need to access it externally.

[DanVM99]

Hi Ray, thanks for your quick response

Your description of the issue regarding the 7 second pause seems reasonable.  Just wanted to check, when you say that there is a message sent to the vCenter to recompose, do you mean a reconfigure of the virtual machine to attach the App Stacks the user has assigned to them?  If this is the case, as we have 10 App Stacks in our test it will take a while to attach them all.  We will try different amounts of App Stacks and see if it changes accordingly.

In regards to the certificate validation, we did a very rudimentary test of a theory we've had for a little while in that the Quickprep process is somehow interfering with our Trusted Root Certificate.  Our root certificate is present in our gold image but occasionally we notice that some machines will behave as if no root certificate is installed.  This usually is shown by programs asking for credentials when they normally wouldn't.  It may also explain why we are having issues connecting to the App Volumes manager when  certificate validation is enabled.  To test this we tried giving a certificate a friendly name in the gold image and then creating a pool from a snapshot that contained the friendly name.  When the pool had been composed, the friendly name had been erased.  Like I said it is a rudimentary test but shows that "something" is happening to the root certificate store during the creation of the pool.

Finally the issue with the all the apps not attaching prior to login seems to ignore the default timeout of VolWaitTimeOut as if the registry key is not present, it doesn't appear to wait the 3 minutes it should before logging a user in if not all app stacks are "ready".  By this I mean it will log in the user well before the 3 minutes is up and then the apps will appear around 3 minutes after logon.  I'm not sure what VMWare's criteria is in determining at what point everything is done and that the logon process should complete and the user should be presented with a desktop.  I'm guessing there are a variety of ways it could meet the necessary criteria but these details seem to not be included in the logs (unless we are reading the wrong ones), so it makes it difficult for us to troubleshoot.  We would prefer the user not be presented with a desktop prior to all the apps being present but have so far struggled to achieve this with any level of reliability.

It is also worth mentioning that we are not using writable volumes as the intention is currently go with UEM or Persona Management for user profiles.  We do use disposable disks though.

[Ray_handels]

Normally adding more appstacks won't make the reconfigure (yes, it attached the extra disks) take longer. It gets a command to change the machine settings and adding 1 or 10 disks for VCenter is just as fast. This used to be a problem in the first 6.0 version but has been fixed already.

It does however add up to the amount of time Appvolumes is busy merging the appstacks. This is what you would see in the logfile svservice.log in the Cloudvolumes\Agent directory.

To be honest I have heard more people complain about this. We try to get the user logged in as fast as possible and don't mind applications being added after logon. But this is the same behavior as we have on physical machines so our users don't mind. Maybe you could raise a support ticket with VMWare to check and see what they have to say.

[DanVM99]

Hi Ray,

Thanks for your insight into the attachment of the stacks.

We did some additional testing on Friday but didn't have anything concrete to post.  Some additional testing this morning appears to have shown some positive results in relation to the issue contacting the App Volumes manager.

With certificate validation enabled, we have not yet run into any connection errors by having a script that is initiated by the Horizon Quickprep process.  On the gold image we have created a folder that contains our root certificate, and a batch file that uses the certutil program to install it.  We've entered the location of the batch file into the QuickPrep post synchronisation script location (during pool creation in Horizon View) and the seems to have cured that problem, admittedly with only a small number of logons attempted.

If anyone else is having this issue, the batch file is only one line as follows

certutil -addstore -f -enterprise -user root <path to root certificate>\<certificate file name>

e.g. certutil -addstore -f -enterprise -user root C:\resources\root.cer

However we are still having the issue with being logged on prior to all application being available and have opened a support request with VMWare about this.  If we get a reliable fix, I'll post it here.

[fcardarelli]

Hi! I'm having the same issue with App Volumes 4.x, Have you found any workaround?

[HallJ0816]

Were you able to find a solution to appstacks not applying immediately?

[Ray_handels]

What do you mean by not Applying immediatly? Appstacks are being merged when logging in. You can add a registry key to wait for all Apptacks to be attached before releasing the login but if you have a very large application you could end up with a logon time of over 3 minutes (180 seconds is the default timeout and it will release login either way due to the long wait time).