Random Appstack Connection Issues

[jmatz135]

We have pools of Instant Clone VDI desktops and occasionally users will log in and get a Connection Error Unable to Contact App Volumes Manager.

These are pools of desktops with the same parent image so obviously they are all exactly the same and it works properly like 95% of the time, but 5% of the time it failing isn't really acceptable.

This is I guess the relevant portion of the logs:

[2017-09-25 15:13:18.105 UTC] [svservice:P848:T4388] OnStartShell: {Username} (NameSamCompatible)

[2017-09-25 15:13:18.105 UTC] [svservice:P848:T4388] Read registry value VolWaitTimeOut (value is 120)

[2017-09-25 15:13:18.106 UTC] [svservice:P848:T4388] OnStartShell: skipping scripts because filtering is inactive

[2017-09-25 15:13:18.106 UTC] [svservice:P848:T4388] Message: "Connection Error (Manager "{server_name}"):

Unable to contact App Volumes Manager." (hToken 0000000000000350)

[2017-09-25 15:13:18.106 UTC] [svservice:P848:T4388] RunExecutableAsUser: Path "C:\Program Files (x86)\CloudVolumes\Agent\svservice.exe"

[2017-09-25 15:13:18.106 UTC] [svservice:P848:T4388] RunExecutableAsUser: CommandLine svservice.exe message "Connection Error (Manager "{server_name}"):

Unable to contact App Volumes Manager."

CreateProcessWithTokenAndEnvBlockW() standard user or UAC turned off, continue...

[2017-09-25 15:13:18.108 UTC] [svservice:P848:T4388] Successfully launched (elevated privileges): svservice.exe message "Connection Error (Manager "{server_name}"):

Unable to contact App Volumes Manager." (wait 0 ms), pid=784 tid=4296

[2017-09-25 15:13:18.108 UTC] [svservice:P848:T4388] Successfully launched: svservice.exe message "Connection Error (Manager "{server_name}"):

Unable to contact App Volumes Manager." (wait 0 ms)

[2017-09-25 15:13:18.109 UTC] [svservice:P848:T3576] Waiting 0 second(s) for a new volume

[2017-09-25 15:13:18.109 UTC] [svservice:P848:T3576] Activate filtering (called by DelayActivateWorker)

[2017-09-25 15:13:18.110 UTC] [svservice:P848:T504] Preload volume event (startup): "\Device\HarddiskVolume2" GUID {b7497ec6-0527-11e7-8f44-806e6f6e6963} Hive  (1 logged in, SystemVolume 1, VolumeType 0)

[2017-09-25 15:13:18.110 UTC] [svservice:P848:T504] Sending reply to SVCMD_ID_NEW_VOLUME_PRE (Message 1, Size 24)

[2017-09-25 15:13:18.216 UTC] [svservice:P784:T4296] *** Started

One thing I did notice that was odd was the machine was last booted on 09/22 but the App Volumes server itself says it thinks it was last booted on 09/20.  Why would that happen?

We run two app volumes servers with a load balancer for the by the way.

[Ray_handels]

What version of Appvolumes are you using and what load balancer do you use?

We saw this exact same issue with the 2,9 agent we were using back then but 2.10 and 2.11 had somewhat of the same issues and our loadbalancer is an F5. To be honest I'm not fully convinced with how the F5 works. It seems to be a little bit flaky.

WIth the 2.12.1 version it works very well. We don't have users or machines that can't contact the Appvolumes managers.

[jmatz135]

We have an F5 load balancer and our app volumes management servers are 2.12.1 but we had to go back to using the 2.11 agent because of a different issue.  Sounds pretty much like the issue you had though.

[techguy129]

Please verify your F5 settings based on this doc.

https://devcentral.f5.com/Portals/0/userfiles/48273/F5-VMware%20App%20Volumes%20Configuration%20Guid...

Also, I remember reading somewhere, which I cannot find at the moment, that it was best to update the registry to include the app volumes servers along with the load balance address.

I've had no connection issues with 2.12.1 with my management servers behind our F5.

[jmatz135]

Yeah, everything is set up according to the documentation.  I have not added the individual appvolumes servers to the list in the registry though.  I'll have to try that out and see if that helps.

[VDINinja311]

jmatz135​

After upgrading from 2.11 to 2.12.1 we are seeing the exact same issues you are seeing, same error on the desktop side. In the beginning just a reset of the VM would make it check back into the AppVolumes Manager, but then they started timing out after only a few hours and would go back offline on the AppVolumes Manager. We had an entire linked clone desktop pool that ~50% of the pool was saying they were offline on the AppVolumes Manager. Also using an F5 LB for the AV Managers. No issues prior to the update. We have a SR open for the last week for this issue as we are now seeing it affecting another pool. Nothing but a few unrelated KB's being thrown our way to try.

Also seeing the same weird issues where we are seeing inconsistencies in the AppVolumes Manager almost like DB corruption type issues. I was watching a linked clone refresh after a user logged off and App Volumes thinks it saw our Master Image machine check into App Volumes like it was powered on. I verified it hadn't been powered on for ~7 days prior.

[Sravan_k]

Hi,

Please check the agent that you installed is taking to correct port number.

for example: "appvolmanager-ip:80" and make sure try to reach your manager from browser my typing address as "http:\\app-vol-manager-ip"

please provide me feedback so that based on that I can give you solution.

Thank you

[Sravan_k]

on this issue, I would say it's something happening at app stack's disk level, are you using the default app vol template [20GB]?

[VDINinja311]

Can confirm that we can browse to https://%F5LBVIP%:443 in the browser. Never had issues with that. F5LBVIP is our F5 Load Balanced vIP for App Volumes of which the Agent is configured on 443 on the desktop master image.

[VDINinja311]

No we use custom sized templates that were cloned from the default 20GB back on 2.11

[jmatz135]

We use custom sized disks cloned from the 2.12.1 disk.  Why would an issue with the disk cause an unable to contact the appvolumes server?  That doesn't really make sense.  It's not like the disks fail to apply, it literally times out trying to connect to the appvolumes server.  I mean even the logs themselves say:

"Connection Error (Manager "{server_name}"):

That doesn't seem to be a disk error.

Anyway, once in the machine, yes you can get to the site https://appvols_manager just fine.  In fact the logs on the server will see the logoff of the machine, just not the login.

[Sravan_k]

correct, app stack temple size is nothing to do in you case.

so, are you using port 80 or 443 for avm and cross check with the agent that when you configured on parent image, if you found agent is configured properly, then it may be issue with data base fragmentation [where you have to re-index].

please open case with vmware for confirmation and to review log's

[Sravan_k]

please open case with vmware to find the root cause, it looks like something going on app stack's

[jmatz135]

We are using port 443.  I highly doubt it is a database issue as it never even seems to be responded to by the App Volumes management servers on login thus it never would access the database.  Why the app volumes manager doesn't respond though is a mystery to me.

[VDINinja311]

jmatz135​

SR was entered on 9/22 for this issue. Just today were logs asked for, the entire first week was just them sending KB's over to try that didn't have anything to do with the issue. Going well so far 😕

We are each day getting more desktops that are staying offline in AppVolumes Manager after refresh operations when users log off. It was originally only affecting a single desktop pool, but is now affecting another pool (same snapshot of the same master image).

[Sravan_k]

Because when user is trying to log-in to VDI app vol agent is trying to attache app stacks due to database issue, the process is going to timeout and finally user is seeing AVM is not responding

[Sravan_k]

I sent you a message, please check it

Thank you.

[jmatz135]

That would imply that the app vols manager ever even was contacted by the agent at login though, which I don't see anything in the logs that would show that.  It's like the app vols agent tries to contact the manager at startup and when it can't it just turns itself off.

[Sravan_k]

how many users are affected by this? and what is your app volume manager and agent version?

Regards.

[jmatz135]

How many users are affected?  Well all of them could be, so a few hundred at the moment, but realistically only about 1-5% of users that login will have the issue.  That in my opinion is still way to high.  If you have 200 users login for the day and 2-10 users fail to get their appstacks, which in turn causes issues with their profile because UEM then captures their start menu/taskbar/application settings without applications.

App volumes manager 2.12.1 and agent 2.11 for the most part, but I have test pools with agent 2.12.1 and I believe I have seen it happen there as well.